MALT: Improving Reasoning with Multi-Agent LLM Training
Authors: Sumeet Ramesh Motwani
Chandler Smith
Rocktim Jyoti Das
Rafael Rafailov
Philip Torr
Ivan Laptev
Fabio Pizzati
Ronald Clark
Christian Schroeder de Witt
Venue: COLM 2025 (2025)
Keywords: reasoning
multi-agent systems
post-training
reinforcement learning
large language models
TL;DR: We introduce a multi-agent post-training approach to improve the reasoning and self-correction performance of a generator, verifier, and refinement model working together

Abstract: Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human or teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ultimately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with relative improvements of 15.66%, 7.42%, and 9.40%. It also generalizes to more challenging benchmarks, marking an early advance in multi-agent cooperative training.

Published as a conference paper at COLM 2025
MALT: Improving Reasoning with Multi-Agent LLM Training
Sumeet Ramesh Motwani', Chandler Smith?, Rocktim Jyoti Das Rafael Rafailov*, Ivan Laptev?, Philip Torr', Fabio Pizzati Ronald Clark', Christian Schroeder de Witt!
Universitty of Oxford Cooperatiive AI Foundation
MBZUAI
"Stanford University
Correspondence: sumeet. motwani@eng. w.. uk, csêrobots. ox.robtss.o.ac.uuk Project page: https://mulliaggetlinmrainnniiit com/
Abstract
Large Language Models (LLMs) often produce answers with a single chain- of-thought, which restricts their ability to explore reasoning paths or self- correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data genera- tion, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-trute data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ul- timately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with relative improve- ments of 15.66%, 7. 7.42%, and 9 9.40%. It also generalizes to more challenging benchmarks, marking an early advance in multi-agent cooperative training.
Introduction
Reasoning with Large Language Models (LLMs) is inherently challenging, particularly for tasks that require multi-step deductions, intermediate computations, or self-correction (Xiang et al., 2025). Recent work on multi-agent approaches -uchh as debate frameworks (Du et al., 2024) orchestrated problem-solving (Fourney et al., 2024)-2aah shown promise by assigning different parts of the reasoning process to separate models, allowing for refinement and increased deliberation at inference time (Snell et al., 2024). However, the underlying LLMs are typically the same base model pre-trained on data that lacks exposure to the specialized roles or meta-strategies involved in solving complex problems, which introduces distribution shift while reasoning at test-time (Xiang et al., 2025; Han et al., 2024). An open gap thus persists: How can we jointly train LLMs to specialize in multi-agent setting to improve reasoning? Such a gap persists due to several key obstacles: First, in an end-to-end supervised training approach, it is difficult propagate gradient signals through multiple discrete token-outputs. Second, there is limited role-specific labeled training data. Third, credit assignment is difficult in reinforcement learning with only sparse outcome rewards (Tumer & Agogino, 2007). Finally, it is important to determine what type of multi- agent setup is even useful to implement meta-strategies that can improve reasoning so that more inference compute can be spent efficiently.
In this paper, we address these challenges by introducing an intuitive strategy to jointly post-train specialized LLMs in venerate-verify--refne pipeline. This is analogous to how humans tackle complex tasks draftnng an initial answer, thoroughly verifying and cri- tiquing it, and refining the solution to match their specifications (Qian et al., 2024). We propose method that automatically generates large-scale, labeled data for each agent via multi-agent credit assignment approach. With this dataset, our post-training approach
1

---

Published as a conference paper at COLM 2025
enables role-conditioned models to learn from both positive and negative reasoning tra- jectories- providing a path to improve reasoning performance across a range of problems with trained multi-agent setups. We aim to address this critical gap with Multi-Agent LLM Training (MALT), a new post-training method that we apply to three models (a generator, verifier, and refiner) solving complex reasoning problems together.
MALT employs a sampling procedure that expands a search tree based on each model's outputs with an exponential branching factor, thereby producing large amounts of useful synthetic data. Role-specific data, particularly when augmented with rationales, has been shown in previous work to significantly improve performance (Zelikman al., 2022). However, this approach leads to a credit assignment problem where internal reasoning branches may be correct or incorrect and must be labeled solely based on final outcome rewards to post-train models. To address this, we propose value-iteration-aasee attribution strategy (Sutton & Barto, 2018). By analyzing only the outputs of the search tree, our method identifies which model introduced an error, enabling credit assignment without requiring additional multi-agent training data or an oracle policy. This eliminates the need for intervention in selecting trajectories, generating role-specific data, or designing value functions, and instead automatically produces reasoning traces from the search tree for post- training via supervised fine-tuning and preference optimization (Rafailov et al., 2023). MALT integrates all these steps into an intuitive joint-training procedure, providing performance improvements and serving as an early step toward unifying search and learning for multi- agent LLM systems. Our contributions are as follows:
We are the first to introduce Multi-Agent LLM Training (MALT) to cooperatively post-train a specialized generator, verifier, and refinement model on challenging reasoning tasks by leveraging search and subsequent fine-tuning.
We propose a search-tree expansion process and use value iteration technique to propagate outcome rewards to automatically attribute correct and incorrect reason- ing traces to individual agents. This synthetic data can be utilized for supervised fine-tuning and reinforcement learning in general multi-agent reasoning pipelines. We apply MALT to math and common sense reasoning questions from MATH, CSQA, and GSMSK, obtaining a relative boost of 15.66% 9.40% and 7.42% over a single-model baseline, also providing other comparisons and ablations.
We show how MALT on the benchmarks discussed above generalizes to far more challenging reasoning tasks such as GSM-Symbolie-hees MALT almost matches the performance of an 8.75x size model from the same series.
2 Related Work
Advanced Reasoning and Inference Compute: Multi-agent architectures have emerged as an effective way to handle complex reasoning by distributing problem-solving roles among distinct models (Cobbe et al., 2021; Xiang et al., 2025). Frameworks such as AgentQ (Putta et al., 2024) and AutoGen (Wu et al., 2024) produce solutions through guided search and compositional dialogue, but rely on a single underlying model for multiple tasks or omit a dedicated post-trained verification mechanism. Debate-style (Du et al., 2024) and orchestrated (Fourney et al., 2024) multi-agent methods enable increased deliberation at test-time with multiple interacting agents. However, the underlying models lack advanced training for role specification, which may lead to suboptimal performance or distribution shifts during test-time (Xiang et al., 2025). In contrast, single-agent approaches utilize techniques such as introspection (Qu et al., 2024) and self-critique (Saunders et al., 2022; Kumar et al., 2024) to detect their own errors, but they typically lack the mechanisms to correct those errors in single inference pass (Ye et al., 2024). Unlike others, we aim to learn specialized agents with dedicated functions, thanks to our novel credit assignment strategy and post-training procedure.
Training Data and Preference Optimization: Using synthetic data generation with breference-based training has emerged as important strategy for boosting LLM per- formance (Zelikman et al., 2022). (Setlur et al., 2024) demonstrate that training on correct and incorrect synthetic solutions, optimized with Direct Preference Optimization
2

---

Published as a conference paper at COLM 2025
(Rafailov et al., 2023, DPO), significantly improves math reasoning, performance. (Singh et al., 2024) surpass purely human-based approaches on challenging math and coding tasks by repeatedly generating, filtering, and tine-tuning with scalar feedback, while (Pang et al., 2024) demonstrate how preference signals applied across entire chains-of-thoughe refine intermediate reasoning steps. Our work unifies the aforementioned techniques to create multi-agent pipeline that orchestrates generator, verifier, and refinement model, leveraging search, synthetic data generation, and preference-basee post-training to enable robust multi-step reasoning. We discuss additional related work in Appendix A.7.
3 Preliminaries
Chain of Thought (CoT) An LLM's output for a question 9, with natural language rea- soning steps ($1,...,8n) followed by an answer a, can viewed as CoT process sampled from the distribution:
Pa(a 9) a pa(a Sim 9) II Pa (st,, 9) ds.
More generally for complex problems, (Xiang et al., 2025) provide meta-generalization that describes the true solution -eatttoinggeeatting process involving extra latent steps (z1,...,zy)
Pa (a Simu 1 <
pa(A, Z1:k 9)
II Pa(zz Zetz 9) dZ.
These meta-variables could capture iterative or corrective steps, unfolding a more adaptive, multi-stage reasoning trajectory. In multi-agent setup, consider process with three specialized models: a Generator G, Verifier V, and Refinement model R producing & and respectively. This can be mapped onto the meta-CoT setting, where
p(a ())
()) = pc(8 9) pv(v) 8,9) (R(a 8, 8,0,9) d(g,o).
Here, g and v, from the standpoint of standard (g, 4) datasets, act like latent meta-steps guiding more complex solution processes, akin to Z in meta-CoT. By explicitly modeling these roles during joint post-training (see Section 4), the multi-agent sequential deliberation process allows us to tackle problems beyond single-pass CoT setting.
Supervised Finetuning (SFT) Given train containing positive demonstrations (e.g. cor- rect generator outputs or useful verifier critiques), we can carry out SFT:
LSF (TT) = - BB
log nh (v:lyet))))
(x.y) (xy)) DSFT Inain
which is a simple and relatively successful technique to improve reasoning performance.
Direct Preference Optimisation (DPO) In SFT, order to improve our test set performance. However, using preference optimization also allows us to leverage negative samples, y < yt, where < means that y is preferred over yT. DPO (Rafailov et al., 2023) is one such method-a more efficient alternative to classical RLHF (Christiano et al., 2017)-becaus. instead of employing full reinforcement learning loop, directly optimizes the following contrastive objective:
exclusively positive samples in
x) - -Blog Thet (y+
Te
xx) Thet xx)
naly |
Loeo = -E (x,yx yx >~-DDPOOT
(6log
We assume Nuef is a reference policy, obtained through SFT. train is a dataset of triplets including both inputs and posítive positive/negativv negative outputs, and is the hyperparameter scaling reward signal strength. We provide a theoretical link between optimizing the DPO objective and the optimal RL policy in Appendix A.6.1.
3

---

Published as a conference paper at COLM 2025
Input question
q
Traiiing suention
Ganeritv srowers
IWW AANRR ROOCOU CAROORAE Airoo
D
SFT
G
G
56b
19" .99
GULT
UIIRER
Verifer
Dv
SFT+DPO M.O..S....Ch
V
(0
Retieer stowers
samoling
M
DR R
SFT+DPO
R
a Answer
SKe XXXXXXXXXXXXXX
Generating reasaning trajectories
Post-training
Inference
Figure 1: MALT Method Overview. Given an input, we consider a three-agent system composed of Generator for initial answer production, Verifier providing a critique, and Refinement Model integrating all intermediate reasoning steps into final output. For questions in the training set, we introduce a tree search and credit assignment process (Left) generate synthetic datasets with reasoning trajectory preference pairs for each model. These are used to post-train individual models (Center). During inference over the test-set, we perform three parallel sequential passes through the multi-agent setup, and return the final answer obtained via majority voting (Right).
4 Method: Multi-Agent LLM Training
In reasoning tasks, single LLM must handle all aspects of the solution generation process often leading to limited exploration, a lack of self-correction, and difficulty refining partial steps (Ye et al., 2024). However, the reasoning process can be broken down into system of decentralized LLMs, where each model has differing objectives and or partial observability. As described in Section 3, this represents a meta-CoT (Xiang et al., 2025) setting where intermediate outputs in the multi-agent setup can improve the overall reasoning process. Although in fully observable and cooperative cases, systems of LLM agents could technically be simulated by single centralized LLM, decompasition into separate heterogeneous those observed in decentralized multi-agent
LLM agents offers various benefits analogous learning (Boutilier, 1996; Schroeder de Witt et al., 2020). Decentralization factorizes large joint action spaces, allowing each agent to focus on smaller sub-tasks under its own partial observability, leading to more targeted exploration and clear credit assignment (Tan, 1993). Here, we present our methodology for a multi-agent setting consisting of sequential heterogeneous process where agents can be trained based on joint rewards.
4.1 Multi-Agent Inference Setting
We formulate our multi-agent inference setting as a collaborative reasoning framework designed to solve complex tasks. Let Q denote dataset of natural language questions, where each E Q represents a specific task instance. The objective is to generate prediction a E A for given input 9, where A is the set of all possible answers. During training, there exists a ground truth function f Q + A, where f()) = GT serves as the ideal prediction for evaluating a. Our framework consists of three specialized LLMs acting as distinct agents, each defined as a function:
1. Generator (G Q Pg + Oc): Produces an initial response to the question.
2. Verifier (V Oc x Q x Pv + Oy): Critiques the generated response for issues.
3. Refinement Model (R: Og x Oy Q x PR OR): Integrates feedback to improve
the final prediction.
Here, PG, Py, PR denote the set of prompts for each model, and OG, Oy, and Or are the sets of possible outputs for the generator, verifier, and refiner respectively. A is the set of possible answers extracted from the refiner's output with fixed deterministic function T.
4

---

Published as a conference paper at COLM 2025
Formally, we define the interaction between these agents as: go = G(g. P2) c;: vo = Veg.Po.
Pe. E Oy; = R(g, Pr. E Po) == T(ro) E..
This setup is reminiscent of how LLMs are used in production, where they receive initial prompts containing questions, feedback, and are then asked to refine answers (Wang et al., 2024). We demonstrate in Section 5 that this inference setting enhances performance compared to single-model approaches. The key insight, however, relies on leveraging this multi-agent inference setting to generate synthetic data that scales exponentially with respect branching factor. Below, we discuss out data generation and post-training setup.
4.2 Collecting Reasoning Trajectories
In standard single-LLM setups, single model simply produces an answer for each ques- tion q. By contrast, our multi-agent framework uses three specialized Senntt-renerator, Verifier, and Refiner--equentiallly To enable training of these agents, we need to capture how each agent's output contributes to the final prediction and whether the overall solu- tion is correct or incorrect. A reasoning trace leading contains [go, Vol,,,, where the multi-agent answer generation process will be:
a T(R(9, Pr, G(g, px), V(g, Por G(q, Pt)))).
During data generation, V, and R all share the same base model parameters; in the subsequent training phase (Section 4.3), each policy is updated independently to specialize in its respective role. An outcome reward function R A x A - {0, 1}, based on the ground truth in the training set, evaluates a to mark the trajectory as either correct (1) or incorrect (0). Specifically, for a predicted answer and ground truth aT define Ra, i, a aCT, and 0 otherwise.
To collect role-specific post-training data, we use < q, a > pairs from an initial training set Durain associated with each benchmark. Our solution is illustrated in Figure (left). The following sampling strategy is employed for all models, with branching factor of n. For each problem 9; E Draii in the training data, we sample completions {8ij G(q:))==1 from the generator. Then, for each G output gi.,, we produce verifications {Vijk ((i.f.:::) Finally, for each V output Vijjk, we generate refinements (Ti,kk Rgg, Pijkk 9:) 9))
This process results in n trajectories for each training example, totaling |Drain| n trajecto- ries. We use the outcome reward model R label the refinement outputs as correct (v) or incorrect (x). The exponential branching factor is very useful to collect large number of diverse training samples (and inference can be parallelized for efficiency while post-training will just rely on the fixed dataset collected).
To effectively utilize reward signals from the refinement model's outputs, we adopt a value iteration approach to propagate values backward through the reasoning chain. Specifically, we compute the expected value of each intermediate output (i.e. generator and verifier outputs) based on the values of downstream outputs from the refiner. This global pooling approach is useful because an intermediate step's utility depends on how it influences downstream outputs and final correctness (see Figure 1, Left). Partial solutions can still yield correct answers once refined, so simply providing binary labels is insufficient. By assigning each output an expected value, we better capture the influence the final reward. This also allows MALT to be generalized to settings where intermediate steps may not carry the same labels, and each intermediate output can now be credited in proportion its impact on the final solution.
Value Function Definitions We define the value of each refinement node Tijas by directly comparing its final answer a to the ground truth:
V(ripa)) R (T(((ITA))) (0.1).
5

---

Published as a conference paper at COLM 2025
The value of a V output Dij.k is then computed as the expected value of its child refinements:
V(Dijk) = IB; | (V(riRa))
((rizz))
Similarly, the value of a G output &ij is the expected value of its child verifier outputs:
(gg,) = Ex (V(Lfi))
2
V(Dijk)-
Each output state's empirical mean a Monte Carlo approximation (an unbiased sample average) of its true expected correctness, indicated by "z=" for each q. This process prop- agates reward signals from final outputs back through the tree to each intermediate state, capturing each output's overall utility.
Thresholding and Binarization To prepare the data for training (SFT and DPO), binarize values using a threshold of 0.5, aligning with majority voting principles. Nodes with values greater than 0.5 are labeled correct (V), and those with values less than or equal to 0.5 are labeled as incorrect (x). The intuition behind this strategy is discussed in more detail in Appendix A.6.1, where we demonstrate that ensures the policy's expected value monotonically increases. Formally, for each output state s (i.e. output from any model in the multi-agent setup), we define its label Vif V(s) > 0.5, and otherwise.
4.3 MALT Post-training
With each output assigned a value, the goal is to now use these for post-training the models (see Figure 1, Center). We first detail how training data for each model generated: Each refinement output Ti,jA,1 has associated value V(rijaz) {0,1}. We create preference pairs (r+, r) where r+ is correct refinement (V(r+) and r is an incorrect refinement (V(r-) 0) for the same verifier input Vijjk- Formally:
DR = ((+trr) rr, T E (riAaNlL v(rt)= =1 v(r = =0},
Dv = {(tt, ,0) vt E (Diik) 0+ = V 5 = =x].
For each verifier output Pijkkr we compute its value V(Di,,)) and binarize it as 6ijjk Vif ((tiik) 0.5, and 0 otherwise. Preference pairs for the verifier model are created by comparing outputs under the same generator output gt,; A similar process applies to the generator model, where generator outputs & are binarized based on their values (gi,), and preference pairs DG are created by comparing outputs under the same query 9i- This tree-expansion process leads to sufficiently large (see Sec. 5) and diverse dataset.
Refinement and Verifier Model Training Following the data generation process outlined above and the training setups in Section 3, we first perform SFT the G, V, R models with the questions and positive samples in DG, Dy, and DR respectively, updating each model's policy separately. For G, this is analogous 2022) with a specialized dataset. Next, on only the SFT updated V and R models, we apply DPO with question and chosen-rejected pairs in Dy and DR respectively. For the Generator, we opt for SFT-only because DPO does not improve performance (primarily because the base LLM - Llama 3.1 8B is already extensively post-trained on preferences as a generator (Grattafiori et al., 2024) and Sec. 5.3). By combining SFT with preference-based updates, we capture both "ideal" behaviors (through correct samples) and "relative" preferences (through
standard STaR post-training (Zelikman al.,
pairs). This allows to not only bootstrap reasoning based positive traces but also learn generalizable knowledge about useful reasoning trajectories (Chu et al., 2025). We depict method in Figure 1 and algorithm in Appendix A.1.

---

Published as a conference paper at COLM 2025
5
Experiments
Here, we outline the experimental details, including a description of the model, the bench- marks used for evaluation, and the training pipeline. We then present our main experimental results, followed by an empirical analysis and baseline comparisons, along with ablations.
5.1 Experimental Details
Benchmarks and Models We use 1.-ma---.-----nnttucce (Grattafiori et al., 2024), chosen for its open-source nature and balance between competitive baseline performance and size fitting in limited compute budget. We evaluate MALT and all baselines on three widely-used benchmarks: GSM8K (Cobbe et al., 2021), with 7,47k training examples and 1.32k test questions, focused on diverse grade school math problems. For more challenging mathematical reasoning questions, we use MATH (Hendrycks et al., 2021), with training and 5k test problems. MATH has proven to be consistently difficult for smaller language models, with Llama 3.1 8B performing around 49.50% test-accuracy.
Lastly, to extend the scope beyond mathematical tasks, we evaluate on Commonsense@A (CSQA) (Talmor et al., 2019) with 9.74k training examples and 1.22k dev-set questions. CSQA is a multiple-choice question answering dataset around commonsense reasoning problems and has been used similarly by prior work (Zelikman et al., 2022; 2024; Wei et al., 2023).
Baselines We compare MALT against eight baselines in 2 primary settings, all using Llama-3.1-BB. First, we implement the inference-only setting with (1) single-agent (SA) naive setting in which a single model is used as generator to provide an outupt, (2) multi-agent (MA) setting, where the pre-trained baseline model operates in sequential way as a generator, verifier, and refinement agent with the same prompts given to MALT post-trained models. Our second setting (equal training compute STaR baseline) with SFT all three models with the positive synthetic data. We also compare against equal inference compute multi-agent debate baseline (Du et al., 2024) with 3 agents over rounds.
MALT Procedure For MALT, we generate synthetic data for each benchmark separately using the tree-based approach (Algorithm 1) with branching factor of π 3, yielding 27 trajectories per question and approximately 2k-6k labeled pairs per model/benchng beecchaar.. Each final answer is compared against the ground truth in the training set to assign binary reward, which is then propagated to label the G, V, and R outputs. During this, each model has fixed role conditioning prompt template that is also used for baselines and for the post- trained models. We first train each role-specific -lama-3.1-8B-Instte labels with LoRA-based SFT and then with DPO to incorporate reasoning preference data for reinforcement learning (discussed in Section 4.3. We use LoRA adapter-based fine-tuning (Hu et al., 2021), reducing the computational load for post-training (see Appendix A.9 for more details). At inference, MALT follows simple sequential inference pass through the three heterogeneous models reasoning over questions in the test set. Training small models on long sequences of text can lead to instability caabbilityyhhllll /halluciinaiinnssssssssisss (Park et al., 2024), and thus MALT and baselines employ a three-vote self-consistency mechanism to mitigate this.
model on positive
5.2 Experimental Results
Our experimental results along with all the baselines are presented in Table 1, along with Ablations in Tables 4 and 5. Our results are averaged over four runs on random subsets of the large test-sets across seeds, and we report the standard deviation for all our core results.
Baselines comparison We present baseline results in Table 1. Baseline single agent scores on MATH, CSQA, and GSMSK are 49.50%, 74.50% and 84.25%, approximately in line with release (Grattafiori et al., 2024). These scores go up to 52.50%, 75.75% 86.75% and 53.50%, 79.00% 87.00% with single model majority voting and multi-agent majority voting respectively. Additionally, after applying STaR (SFT
scores reported in Llama-3.1-SB-Instruee
7

---

Published as a conference paper at COLM 2025
Test Accuracy (%) over 4 seeds 1
Inference-only
STaR Training
MALT
Benchmark
MALT
SA
SA+MV
MA
MA+MV
SA
SA+MV
MA
MA+MV
Llama-3.1-8h0 1-8--rrsttt
GSM8K
84.25 -2 86.75
84.75 2m
87.00
euc
81.75 was
84.75 2аи
80.00
a.
86.75 222
83.30
90.50
CSQA
74.50
75.75
77.50
79.00
76.25
78.75
75.50
76.00
77.50
81.50
MATH
49.50
52.50
51.75
are
53.50 +0
52.25 -
54.00
52.50
53.75
52.25
57.25
Qwen-2.5-1.5B-Bal
GSM8K
61.75
65.0
3.5
60.50
62.25
63.50
64.25
62.50 6:30
64.25
71.00
74.50
Table 1: Benchmark results. We compare MALT with baselines on three different benchmarks using baselines,
Comma-.1--B-Insttee and also evaluate GSM8K performance on 2.e.-..5-..5---a. we include different setups such as single agent (SA) and multi-agent (MA), both with and without equal inference compute based majority voting (MV). MALT outperforms all baselines across both models.
Method
MATH
CSQA
GSM8K
Method
Accuracy (%)
Inference-only SA
49.50
74.50
84.25
Base Model (Llama 3.1 8B Instruct)
71.75 :: 2.17
Inference-only SA
49.50
74.50
84.25
Base Model MV
73.75 + 2.38
Multi-Agent Debate
52.00
71.25
86.75
Multi-agent MV
75.25 + 2.38
MALT
57.25
81.50
90.50
STaR + MV
76.75 + 4.60
MALT (Llama 3.1 Instruct)
84.75 3.30
Table 2: Debate Baseline. MALT significantly outperforms an equal-inference spend multi- agent debate baseline (3 models debating for 3 rounds) across MATH, CSQA, and GSM8K.
Llama 3.1 70B Instruct
88.25 2.95
Table Generalization, MALT generalizes from GSM8K to GSM-Symbolic P1 (GSM-SP1).
positive synthetic CoTis, single-agent baselines achieve around 52.25%, 76.25%, and 81.75% on MATH, CSQA, and GSM8K, while the multi-agent STaR variant remains around 52.50% 75.50% and 80.00%. Self consistency results for STaR variants surpass untrained baselines but still underperform improvements obtained with MALT. We also evaluate Qwen-2.5- 1.5B-Base (Qwen et al., 2025) for a more diverse analysis. We observe that STaR training does not improve performance significantly beyond inference only baselines. However, DPO training for only the generator achieves 68.75% accuracy and works much better on non-instruction tuned (base) models. This allows MALT to offer even higher improvements.
MALT core results MALT with Llama-3.1-8D-Instreee (Table 1, right) achieves an accuracy of 57.25 25, 81.50% and 90.50% on MATH, CSQA, and GSMSK. Overall, MALT significantly outperforms all baselines, including all settings with supervised fine-tuned models. Over the untrained model's performance as a generator, MALT achieves relative improvements of 15.66% 9.40%, and 7 7.42% on MATH, CSQA, and GSM8K. This shows the efficacy of our search and attribution based data generation, post-training, and inference pipeline in MALT across benchmarks of varying difficulty. MALT on Qwen-2.5-1.55 2..5-1.11---B-.. when evaluated using GSM8K, achieves an accuracy of 74.50% and offers a relative improvement of 20.65% over the single agent baseline. We also compare MALT with an equal inference multi-agent debate baseline in Table 2, where it significantly outperforms across all three benchmarks.
Generalization to more difficult benchmarks To understand whether MALT on a small open model generalizes to challenging reasoning tasks, we evaluate the Llama-3.1-8) model trained with MALT across similar setup with increased complexity. We test our GSM8K- based MALT setup on GSM-Symbolic-PP (Mirzadeh et al., 2024), a complex and much more challenging reasoning benchmark. From results in Table 3, MALT on Llama-3.1-8B achieves an 84.75 3.30% accuracy, outperforming all baselines. Interestingly, MALT also yields performance comparable to the significantly larger Llama-3.1-77 3.1-70B, which results in 88.25 + 2.95%.
8

---

Published as a conference paper at COLM 2025
Incorrect Correct
Incorrect Incarrect
Correct - Correct
Correct - Incorrect
16
GSMBK
59
00 56
Acouney
CSQA
54
MATH
66
20
60
80
100
614
Percentage
Figure 2: Self-correction. MALT consistently increases number of correct answers by cor- recting previously incorrect answers at a much higher rate than introducing new mistakes compared to single-model baseline (MV@3).
Figure 3: Improvement turns. MALT demon- strates consistent improvements at each turn. Test accuracy at each turn from a sequential pass through the post-trained Generator, Verifier, and Refinement models (All with MV@3).
Gelf-consistency From the Llama 3.1 results in Table 1, MALT shows improved reasoning with majority-voting for self-consistency (Wang et al., 2022), but its performance without MV remained close to that of the multi-agent inference-only setting. Qualitatively, MALT solves questions unsolved by any baseline but suffers occasional hallucinations without voting. Thus, we use a small majority voting factor of and observe that celfisccns reliably yields a higher relative improvement for MALT over baselines. For instance, on MATH, self-consistency resulted in a relative improvement of 9,57% exceeding the 6% gains in other baselines. In Figure 2, we measure how often MALT flips an incorrect solution into correct one versus the opposite. We show that our post-trained generator-verffer--eee process results in strong self-improvement with low rate of mistakes introduced.
Improvement over turns We use GPT-4 to rate the correctness of each step produced by MALT (on Llama 3.1-8B-Instruct) on the test sets to analyze its performance at each turn. Figure 3 shows how performance evolves. For e.g,, the untrained baseline on MATH improves from 49.50% to 52.50% with majority voting. MALT, over its 3 reasoning and multi- agent deliberation turns, increases performance to 53.50% (turn Generator), 55.75% (turn 2 Verifier), and 57.25% (turn 3- Refiner). We observe similar increases across GSM8K and CSQA. This turn-wise improvement highlights how sequential verification and refinement are both very important for improving reasoning performance. We provide ablations to understand why generate-verify-refie is the most appropriate paradigm in Section 5.3.
Overall performance Across all benchmarks, MALT outperforms both zero-shot and fine-tuned baselines, closing gaps on problems previously unsolved by any baseline. Its multi-agent approach not only achieves higher average scores but also corrects systematic errors. For example, the verifier often successfully locates errors by redoing calculations and providing a precise critique, with the behaviors learnt automatically from synthetic data generated via our search process. Similarly, for CSQA, the verifier implicitly learns to focus on aspects of the problem overlooked by the generator, with examples of reasoning interactions provided in Appendix A.4. Additionally, MALT can be used on any frontier model since it does not rely on the presence of an oracle model for supervision.
5.3 Ablation studies
Untrained model ablations We test the usefulness of training each individual agent (Llama MALT setup) by replacing one with an untrained baseline and keeping the other two MALT agents. As seen in Table 4, this degrades performance on all benchmarks. For instance, using an untrained generator yields mean accuracies of 54.25% (MATH), 87.00% (GSM8K), and 78.75% (CSQA), notably lower than trained system, showing how all agents benefit from the MALT pipeline.
Importance of generate-verify-reiie Next, we measure the impact of ablating one role from our three-agent pipeline (Llama MALT setup) for simpler two-agent systems. We compare (i) generate-refine (G + R, skipping the verifier) and (ii) generate-verify (G + V, skipping the refiner). As shown in Table both setups underperform the full pipeline:

---

Published as a conference paper at COLM 2025
G
V
Poststraining
GSMBK
CSQA
MATH
Configuration
GSMSK
CSQA
MATH
G only
84.75
78.75
54,00
57.00
78.75
54.25
G+V V G G+R R
88.75
78.00
35.75
85.75
76.75
86.25
75,50
55.25
84.75
76.25
54.75
90.50
$1.50
57.25
G + + (Ouns)
90.50
81.50
57.25
Table 4:
Ablations with untrained models (MV@3). Combining untrained agents with trained ones shows that all LLMs perform best when cooperating with MALT agents.
Table 5: Performance of ablated multi-agent se- tups (MV@3). Our experiments show that all agents in the MALT pipeline are necessary to achieve the best results.
G+ R yields mean accuracies of 54.75% (MATH), 84.75% (GSM8K), and 76.25% (CSQA), while G + V yields 55.75% (MATH), 88.75% (GSM8K), and 78.00% (CSQA), showing that our specific multi-agent setup yields stronger results by spending inference for improved sequential reasoning. Importantly, we note that our three-model pipeline does not rigidly fix MALT's usability. It is, instead, a modular and flexible entry-point for multi-agent training. MALT subsumes the utility of single model learning from more data, while providing an opportunity to better use specialized models for improved reasoning. With our search and credit assignment strategy, MALT reduces the risk of reinforcing model's existing biases, providing diverse training signal for system-leve improvement that allows for generalization across difficult problems.
Effectiveness of DPO over only SFT As shown in Table 1, DPO improves performance beyond SFT alone (STaR) by using negative data-as observed in (Putta et al., 2024; Setlur et al., 2024). In particular, purely positive "rationales" can introduce spurious correlations and degrade SFT performance, which does indeed occur in our empirical and qualitative results; the contrastive training approach that DPO provides instead helps the model identify high-advantage reasoning steps to improve with higher sample-efficiency (Rafailov et al., 2024). For reasoning problems, SFT tends to memorize the data and rules, which is useful to bootstrap reasoning to a certain extent (Zelikman et al., 2022). However, our results indicate that this could degrade performance sometimes, and preference optimization methods (see Appendix A.6.2 for a theoretical analysis and A.9 for possible issues) exhibit better performance at learning generalizable knowledge for reasoning steps (Chu et al., 2025).
Discussion and Conclusion
We presented MALT, a novel post-training strategy dividing CoT reasoning among three specialized LLMs to tackle complex problems. MALT bridges the gap between prior multi- agent inference methods and fully-trained multi-agent systems by generating role-specific data using a tree-based sampling and credit assignment mechanism. Crucially, MALT utilizes the negative synthetic data to identify and correct flawed reasoning steps with LLM post-training, improving role-specific reasoning capabilities. Unlike standard single- LLM setups, our design closely mirrors how humans solve complex tasks or even use LtMs-attn a solution, critiquing errors, and finally refining the result. While we were unable to expand experiments to significantly larger models due to compute limitations, we have evaluated against comprehensive set of challenging datasets and benchmarks. Our vererratoo-vrriie---efeeeeee such as self-correction or chaining inference steps, but can be extended to automatically learnt roles obtained via search. We discuss this, along with future work such repeated rounds of refinement, online RL settings, and scaling the branching factor in Section A.8.
setup focuses on improving meta-strategies
Safety Our approach can be used not just to enhance the reasoning capabilities of LLM systems, but also address crucial open problems in the safety of multi-agent systems. Impor- tantly, MALT-trained systems of trusted small models could attain better task performance while retaining high degrees of trust, producing more powerful overseers within the Al control setting (Greenblatt et al., 2024). Another prominent application of our approach would be to train verifiers safety critics. This could scale up the settings such as OpenAl CriticGPT (McAleese et al., 2024) to any number of models, resulting in more powerful safety critics and allowing for the legibility of solutions to be improved.
10

---

Published as a conference paper at COLM 2025
References
Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In TARK, volume 96, pp. 195-210. Citeseer, 1996.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint ar Xio:2407.227888 2024
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In NeurIPS. Curran Associates, Inc., 2017.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv, 2021.
DeepSeek-A,, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, and Ziyi Gao. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https:///rxiv.crr org// 0112501..994..
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In ICML, 2024.
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: A generalist multi-agent system for solving complex tasks. Xiv, 2024.
Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the task or the annotator? investigation of annotator bias in natural language understanding datasets. In EMNLP, 2019.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Llama Team. The llama 3 herd of models. arXiv, 2024.
Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI Control: Improving Safety Despite Intentional Subversion. In ICML, 2024.
Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He.
Lim multi-agent systems: Challenges and open problems. arv, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS, 2021.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL, 2018.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lam- bert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. In NewriPS, 2024.
11

---

Published as a conference paper at COLM 2025
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning, wwwprriivv.ooraaaeeeee www.oogg////////ooooooo ab///bbbb/449.... 12917.
2024. URL https: //axxiv
Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. In AAAI, 2020.
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. LLM Critics Help Catch LLM Bugs. arXiv, 2024.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and
Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024. URL https: /ttp//axivvvvv///iv//eee///eevvt http:////aaxxv.or/aabssss/// wtt.oo/ba//./../22/2222
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In ACL Findings, 2024.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. Xrv, 2024.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative Agents for Software Development. In ACL, 2024.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. In NeurIPS, 2024.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, and Jiaxi Yang. Qwen2.5 technical report, 2025. URL https://arxiv www..orgg/bb//////2 15115.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to q²: Your language model is secretly a fuuncction. In COLM, 2024.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv, 2022.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. Torr, Mingfei Sun, and Shimon Whiteson. Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? arxiv, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arxiv, 2017.
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. Rl on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. In NeurlPS, 2024.
Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ram Pasunuru, Mrinmaya Sachan, Jason Weston, and Asli Celikyilmaz. The art of Ilm refinement: Ask, refine, and trust, 2023. URL https://aarxi wwtpexivvooggg/boaee wtgg/bbs//111..000 07961.
12

---

Published as a conference paper at COLM 2025
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl- Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. TMLR, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: introduction, 2nd edn.
adaptive computation and machine learning, 2018.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega: A question answering challenge targeting commonsense knowledge. In NAACL, 2019.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Kagan Tumer and Adrian Agogino. Distributed agent-based air traffic flow management. In Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS '07, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9788190426275.
Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. Understanding user experience in large language model interactions. arXiv, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in lms, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models. In NeurIPS, 2023.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen 11m applications via multi-agent conversation framework. In COLM, 2024.
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, Louis Castricato, Jan-Philipp Franken, Nick Haber, and Chelsea Finn. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though. Xiv, 2025.
Guangtong Yang, Chen Li, Yudong Yao, Ge Wang, and Yueyang Teng. Quasi-supervised learning for super-resolution pet. Computerized Medical Imaging and Graphics, 2024.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics language models: Part 2.2, how to learn from mistakes on grade-school math problems. arXio, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In NeurIPS, 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Good- man. Quiet-star: Language models can teach themselves to think before speaking. In COLM, 2024.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl, 2024. URL 2402. 19446.
13

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Algorithm
MALT orchestrates a three-agent system system-ccmmpiise comprising a Generator for initial answers, Verifier for critiques, and Refiner that integrates these steps into final response. Dur- ing training, we expand a multi-agent search tree for each question, labeling correct and incorrect branches via value iteration based credit assignment mechanism. This generates role-specific preference pairs for post-training each agent via supervised fine-tuning and preference optimization. Algorithm 1 provides complete description of the data collection and training pipeline for MALT.
Algorithm 1
Multi LLM Training and Synthetic Data Generation (MALT)
Require: Initial Dataset D, Models G, V, R, Branching factor GOAL: Trained models G', V", R'
Initialize datasets S;, Sv, Sg as empty sets
2: for q E DD do
3: Ag 4 {8] = G(q))}==
Generate n outputs from
4: for each 8j E AG do
5:
AV + {Uix = V(9.8)))
Generate n outputs from V
for each UjA E S: do
6:
AR 1 + (TiAJ RRq. gjr 1ia) )
7:
Generate outputs from R
i2 for each riAd A do
8:
9:
Compute V(rjxJ) = R (Tijanaatt
b. Reward for R output
10:
Add (9, Pjkk jkk. (Ti to Sr end for
11:
Compute () = 1 E ((rina) n1
12:
b Value V for V output
13:
Binarize 0/2- = I[V > 0.5]
14:
Add (9.8,. Pj.k, 91k) to Sv
15:
end for
16:
Compute (gg;) = Σx ((0jk)
b Value V for G output
11
17:
Binarize 8, = [[) > 0.5]
18: Add (9. 8j. 8j) to Sc
19: end for
20: end for
21: Training the Models
G' 4- SG)
b Fine-tune with supervised data
(v, ( Fix) ( (9,8) 0px) Sv, 8j = =}) Fine-tune on positive
VIST SFT
samples
24: V' + D-DOOVSSFT Sv)
Train V with DPO using preferences
25: RSFT < SFT R (33,)))) (1(18)=111
Fine-tune on positive samples
26: R' I'+ #'+DPO DDOO Sg)
Train with DPO using preferences
27: return G', V", R'
A.2 Experiments with GSM-Symbolic
GSM-Symbolic (Mirzadeh et al., 2024) is a synthetic extension of GSM8K that uses symbolic templates to produce varied instances (e.g., altering names, numerical values, or adding clauses). The "P1" variant adds extra clause per question, making the problems more challenging and exposing whether a model relies on shallow memorization or can genuinely handle additional reasoning steps. Testing MALT (Llama 3.1 8B Instruct model trained
14

---

Published as a conference paper at COLM 2025
the GSM8K training set) GSM-Symbolic P1 allows to understand whether the multi- agent setup allows for robust multi-step reasoning beyond the original data distribution. In such cases, even though the base model is much smaller compared to Llama 3.1 70B, it can find mistakes in subsequent iterations. Given that GSm-Symbolic tests for reasoning and penalizes memorization, 70B Instruct.
MALT achieving performance comparable Llama-3.1-
Method
Accuracy (%)
Base Model (Llama 3.1 8B Instruct)
Base Model + MV
73.75 + 2.38
Multi-agent + MV
75.25 2.38
STaR MV
76.75 4.60
MALT (Llama 3.1 8B Instruct)
84.75 + 3.30
Llama 3.1 70B Instruct
88.25 + 2.95
Table 6: Comparison with a larger model. MALT on Llama 3.1 8B Instruct outperforms significantly larger model (Llama 3.1 70B Instruct) on GSM-Symbolic P1, demonstrating important jump in reasoning performance challenging benchmarks. Results over 4 seeds for random subset of GSM-Symbolic PI.
A.3 Multi-Agent Debate Baseline
Based on (Du et al., 2024), we provide results for multi-agent debate baseline. In order to ensure fair comparison between MALT and the multi-agent debate baseline in terms of inference spend, we used baseline with 3 agents debating over rounds. We present our results in Table 7 and note that MALT clearly outperforms an equal inference compute multi-agent debate baseline across all three benchmarks.
Benchmarks
MATH
CSQA
GSMSK
Base Model
49.50
74.50
84.25
Multi-Agent Debate
52.00
71.25
86.75
MALT
57.25
81.50
90.50
Table 7: Debate. MALT significantly outperforms an equal-inference spend multi-agent debate baseline.
A.4 Qualitative Examples
In this section, we present three key qualitative examples: a synthetic training example for SFT and DPO, qualitative performance comparison between an untrained and trained verifier, and complete MALT setup demonstrating a backtracking and improvement reasoning cycle that generates correct answer.
GSMSK Synthetic Dataset Sample for the Refinement Model (GT Answer
System prompt
You are expert problem solver that refines math solutions based on feedback.
Input
Question: Marco looes trading cards. He has total of 500 cards on hand, and fourth of them are duplicales. Marco decides to trade one-fifth of these duplicates for completely new cards his friend Josh. How many new cards did Marco get?
Initial Answer: I'd be happy to help you solve this problem step by step.
Step 1: Identify the key information given in the question
Marco has total of 500 cards. A fourth of them are duplicates.
15

---

Published as a conference paper at COLM 2025
Step 2: Determine what the question asking us to find
'The question is asking us to find how many new cards Marco got from trading one-fifth of his duplicates with Josh.
Step 3: Plan our approach to solving the problem
First, we need to find the number of duplicates Marco has. Since fourth of his cards are duplicates,
can find the number of duplicates by multiplying 500 by 1/4. 'hhn, we need to find one-fifth of the number of duplicates to find out how many cards Marco traded with Josh. Finally, we'll 'lll subtract the number of cards traded from the total number of duplicates to find the remaining duplicates.
Step 4: Execute our plan
Number of duplicates 500 x 1/44- 125 One-fifth of the number of duplicates 125 1/5 25 Marco traded 25 cards with Josh.
Step 5: State our conclusion and double-check our answer
Marco got 25 new cards from trading with Josh.
Final Answer: 25
Rejected Output
Based the initial answer and the verification feedback, I refine the solution as follows:
Step 1: Identify the key information given in the question
*Marco has total of 500 cards. * A fourth of them are duplicates.
Step 2: Determine what the question is asking us to find
'The question is asking us to find how many new cards Marco got from trading one-fifth of his duplicates with Josh.
Step 3: Plan our approach solving the problem
First, we need to find the number of duplicates Marco has. Since fourth of his cards are duplicates, we can find the number of duplicates by multiplying 500 by 1/4. Then, we need to find one-fifth of the number of duplicates to find out how many cards Marco traded with Josh. Finally, we'll subtract the number of cards traded from the total number of duplicates to find the remaining duplicates.
Step 4: Execute our plan
Number of duplicates 500 x 1/4 125 One-fifth of the number of duplicates 125 x 1/5 = 25 Marco traded 25 cards with Josh. Since he traded one- fifth of his duplicates, we find the total number of duplicates he has left by adding the number of cards traded back to the number of duplicates: 125 25 150 To find the number of new cards Marco got, we subtract the number of duplicates he traded
from the total number of duplicates: 150 - 25 = 125
Step 5: State our conclusion and double-check our answer Marco got 125 new cards from trading with Josh.
Final Answer: Final Answer: 125
Accepted Output
Refined Solution:
Step 1: Identify the key information given in the question
*Maroo has a total of 500 cards. * A fourth of them are duplicates. * Marco trades one-fifth of his duplicates with Josh.
Step 2: Determine what the question is asking us to find
'The question asking us to find how many new cards Marco got from trading one-fifth of his duplicates with Josh.
Step 3: Plan our approach solving the problem
First, we need to find the number of duplicates Marco has. Since fourth of his cards are duplicates, we can find the number of duplicates by multiplying 500 by 1/4.* Then, we need to find one-fifth of the number of duplicates to find out how many cards Marco traded with Josh. Finally, we'll subtract the number of cards traded from the total number of duplicates to find the remaining duplicates. Step 4: Execute our plan
16

---

We are given that Billy has 12 sodas and 4 siblings (2 sisters and 2 brothers). To divide the total number of Published as a conference paper at COLM 2025
Number of duplicates 500 1/4 125 One-fifth the number duplicates
125 x 1/5 = 25 Marco traded 25 cards with Josh. Remaining duplicates = 125 25
=100
Step 5: State our conclusion and double-check our answer Maaco got 25 new cards from trading with Josh.
Final Answer: 25
Qualitative Example of an Untrained VS Trained Verifier
Question
Billy buys a 12-pack of soda from the store. he has twice as many brothers as sisters, and he has 2 sisters, how many sodas can he give to each of his siblings he wants to give out the entire 12-pack while giving each the same number?
Initial Answer
I'd be happy to help you solve this problem step by step.
*Step 1: Identify the key information given in the question.* **
Billy buys a 12-pack of soda. He has twice as many brothers as sisters. He has 2 sisters.
*Step 2: Determine what the question is asking us to find.** **
The question is asking us to find out how many sodas Billy can give to each of his siblings if he wants to give out the entire 12-pack while giving each the same number. *Step 3: Plan our approach to solving the problem.** **
Let's let be the number of siblings (brothers and sisters). We know that Billy has 2 sisters, so the number of brothers is 2 times the number of sisters, which is 2x. The total number of siblings the sum of the number of sisters and brothers, which is + 2x 3x. We are given that Billy has 12-pack of soda, so we can set up the equation: 12 2x
**Step 4: Execute our plan, showing all work clearly.*
To solve for x, we can combine like terms:
12 = 3x
Divide both sides by 3:
4=
So, Billy has 4 siblings in total (2 sisters and 2 brothers).
*Step 5: State conclusion and double-check our answer. **
Since Billy has 12 sodas and 4 siblings, he can give: 4 sodas to each of his siblings.
Final Answer: 3
Untrained Verifier
As verifier, will critically evaluate the initial answer provided above.
The initial answer states that Billy can give 3 sodas to each of his siblings. Let's verify the answer by performing the calculations. "Verification:
find out how many sodas each sibling get, sodas (12) by the number of siblings (4):
4
The initial answer is correct! Billy can indeed give 3 sodas to each of his siblings. **Feebback:**
The initial answer is mathematically correct. The calculations are accurate, and the solution well-explained. The answer also logically consistent with the given information. Well done!
17

---

Published as a conference paper at COLM 2025
However, it's always a good idea double-check the calculations to ensure accuracy. In this case, the verification step confirms that the initial answer is correct.
Trained Verifier
As verifier, I will carefully evaluate the initial answer provided above. The initial answer states that Billy can give sodas to each of his siblings. Let's double-check the calculations to ensure the answer is mathematically correct.
The initial answer claims that Billy has 4 siblings (2 sisters and 2 brothers). However, the question states that Billy has twice many brothers as sisters, and he has sisters. This implies that the number of brothers is times the number of sisters, which 2 brothers. Therefore, Billy actually has sisters and 4 brothers, making a total of 6 siblings.
The initial answer also claims that Billy can give sodas to 4 siblings, resulting in sodas per sibling, However, dividing 12 sodas among 6 siblings would actually give sodas per sibling, 3. "Conclusion:"*
The initial answer contains a calculation error. The correct number of siblings is 6, not 4. Therefore, Billy give sodas to each of his siblings, not 3. As a verifier, recommend that the initial answer be corrected to reflect the correct calculation. The correct answer is sodas per sibling.
A Complete Reasoning and Improvement Sequence Example with MALT on Commonsense(A
Question
He looked at the field of pumps, all slowly churning oil out of the what?
Choices
A: manual
B: street
C: restaurant
D: ground
E: service station
Initial Answer (Generator)
Reasoning steps: The context of the question is about pumps churning oil, which suggests location where oil is being dispensed or extracted. 2. A "manual" (A) does not typically relate to a location where oil is churning. 3. A "street" (B) general location, but it's not specific to oil churning. 4. A "restaurant" (C) is not associated with oil churning. 5. The "ground" (D) could be a possible location, but it's not the most specific or common location for oil churning. 6. A "service station" (E) is a common location where oil churning, as it's where cars are serviced and oil is dispensed.
Final Answer: E
Verifier
The initial answer is E: service station. This answer seems to be the most logical choice given the context of the question. The phrase "field of pumps" suggests a location where multiple pumps are present, and service station is common place where multiple pumps are used to dispense oil and other fuels.
18

---

Published as a conference paper at COLM 2025
However, the question asks what the pumps are churning oil out of, not where the pumps are located. This changes the interpretation of the question.
Considering the phrase "churning oil out of, it implies that the pumps are extracting or pumping oil from source. In this case, the most likely source of oil would be the ground, as oil is often extracted from the ground. Therefore, the correct answer is D: ground.
Refined Answer
Reasoning steps: The context of the question is about pumps churning oil, which suggests a location where oil is being extracted or dispensed. 2. The phrase "churning oil out of" implies that the pumps are extracting or pumping oil from a source. The most common source of oil is the ground, as it is often extracted from the earth. Therefore, the pumps are churning oil out of the ground. Final Answer: D
Ground Truth (not provided to any model)
D
A.5 Detailed Discussion
We now discuss key design choices. Simplicity with theoretical grounding -aaendiing-pattcllarry the assurance of trained policy with a monotonically increasing value-was a primary consideration. We opted for DPO over PPO-based methods due our focus on a straight- forward offline data generation process, treating our approach as an independent learning problem in centralized setting (Lerer et al., 2020) with a single iteration (the key difference being that our agent policies post-training differ). In this setting, DPO is more stable than PPO and requires less overhead. While PPO could use the value computed at each branch as a reward for post-training nodes (a promising future direction), it introduces significant computational complexity. Moving from offline to online RL with additional computational overhead might indeed improve performance.
Our value iteration method, when binarized, resembles global majority-based pooling: for given node and branch, the binary reward of the leaf nodes in the subtree determines the usefulness of the branch, analogous to binarizing values propagated through the tree. In contrast, local pooling computes the binary value of branch based only on the majority outcomes of its direct children, propagating this process to the leaf nodes. We also leave the choice between MCTS and an expansive tree-based sampling strategy as an open problem. Given our limited tree depth, tree-based sampling proved efficient, supported synthetic data generation with an exponential branching factor, and produces explainable outputs. Our dataset is collected offline, and individual models are trained on this synthetic data. While this approach works empirically, handling any new, ditto--distribution data would require iterative rollout and post-training methods.
Based on our empirical results and the modularity of our algorithmic approach, highly plausible that our method will scale to larger models and scenarios with many agents, thus laying the foundations for new saate-of-the-art Al agents based on systems of cooperative frontier models. Overall, our multi-agent system is currently composed of sequence of agents that start out with the same parameters and different prompts. MALT performs joint training to transform this into a heterogeneous agent setting, where agents with different parameters operate cooperatively. Exploring other multi-agent settings is an important direction for subsequent work.
19

---

Published as a conference paper at COLM 2025
A.6 Theoretical Justification for MALT
A.6.1 Credit Assignment Strategy
Here, we provide theoretical justification for why our framework, when updating the agent policies based on binarized pooled rewards with a threshold at 0.5, leads to policy improvements. We formalize MALT a three-step MDP, define the pooling operation through value iteration, and demonstrate how off-policy updates increase the expected reward.
The reasoning process in MALT is modeled as a three-step MDP over a set M questions (1 drawn from a distribution Q. For each question qi, the process begins at the initial state 5g = 9i, where an initial answer &i, is sampled from the generator policy πc(. so) for j = 1,..., H. The state then transitions to $1 = (qi, 8ij). - At this second state, critique Oijk is sampled from the verifier policy πt $1) for k == 1,...,,, leading to the state $2 (9: St, Pt.j&). Finally, at this state, refined answer Tijjk is sampled from the refiner policy π $2) for 1,..., n, and reward (ijau) is assigned: if Fijjkl is correct, otherwise. The joint policy is defined as = (TG, TV, TR), and the objective can now be expressed as:
J( = F [ Fg-noollo (so)
Eg (1s)) E ( (12) [R (5 r)]]
(0) Reasoning trajectories are collected offline under an initial policy z( = ( , TV RT yielding M n²² total samples. Through this tree-sampling method, values propagate backward using value iteration: Leaf nodes have ((TiAAz) = R(Ijaa) E 1), veri-
fier nodes compute (vija) - ((riAK)), and generator nodes estimate (gij,
1 # - Ei (Tiija)). The true value V (v) E (,) [R| v) is approximated by these Monte
Carlo estimates.
We note that our analysis rests on the coverage assumption, where for any relevant action (0) (e.g& g with (g) E(V"(g) over πG the initial policy satisfies πミ (g q) Il a > 0,
with analogous conditions for T. and πt
Independence holds across levels: refinements Yijau are i.i.d given (gt,, Vijx),, critiques Vijk are conditionally independent given 8i,j, answers 8t, are i..d. given 9;, and questions ;; are from Q. Moreover, our exponential branching factor n allows for sufficient sampling.
This allows to show that our value estimates are within certain bound of their true values with high probability. For a node v with m downstream refinements (e.g,, m = n? for 8i,)), the value estimate (ir 1 mi Em= R,, where each R, ~ Bernoulli()** (v)) provides an unbiased estimator of V (v). Hoeffding's 's inequality bounds the estimation error:
P(I((v) - Va (v)) 2 e) <
For generator nodes (m n²),, this becomes:
((IV(g.;)) - V" (g1,)) 2 2e) s 2exp(-2n2e?)
Thus, applying a union bound over all Ми generator nodes, we find that with probability at least 8, the estimation error for any given generator node for all questions satisfies
In
2M
(V - v** (8ix) S E, where 6 \
2n12
MALT binarizes node values using threshold of 0.5: DDv) if V(v) > 0.5, and 0 otherwise. This is analogous to majority-voting, where V(v) > 0.5 indicates that most refinements are correct, aligns with the Bernoulli decision boundary for binary rewards,
20

---

Published as a conference paper at COLM 2025
and balances misclassification costs for conservative updates. The updated policy π x(1) shifts probability mass toward these "high-value" nodes using Supervised Finetuning (SFT) and Direct Preference Optimization (DPO). For the refiner, SFT is applied for nodes where V(rij1,)) followed by DPO using preference pairs rr",)
Similarly for the verifier, SFT is followed by DPO its credit assigned preference pairs. For the generator, SFT performed using answers where 1 (g.,) These updates improve the joint policy by prioritizing actions that yield higher expected rewards under πτ(). In DPO objective based data collected
Section A.6.2, we discuss why policy optimizing offline under (0) is identical to the optimal RL. policy.
Finally, we provide intuitive explanation of our threshold used for credit assignment. In iterative settings, should be an adaptive factor increasing from 0.5 to However, in our offline setting, 0.5 balanced threshold to use for the following reasons:
Lower Thresholds (0 < 0.5): This allows for greater sample-efficienc as more branches labeled correct are used as part of training. However, it might introduce noise into the training process with samples that have low values being chosen as correct.
Higher Thresholds (0 > 0.5): This would allow for focus on actions leading to higher-value nodes, reducing variance. However, having too high would reduce sample efficiency.
Using 0.5 provides a balance suitable for a single iteration based on an offline generated dataset. By formalizing our value iteration approach and policy updates, we have shown how MALT increases the probability of selecting outputs leading to higher expected return from the system, and thus increases overall multi-agent performance.
A.6.2 Policy optimizing the DPO objective is identical to Optimal RL Policy
To support our claims in Appendix A.6.1, we leverage Theorem from (Putta et al., 2024) and Theorem 6.1 from (Setlur al., 2024), adjusted for our binarization setting:
Consider a policy that optimizes our objective over trajectories generated by a reference policy Tee. At each node (state) h,, preferences between actions during DPO are generated according to:
Theorem.
p(a > d; h1) -Q(hh.)))(Q(hhd))) Q(h..d)))
(1)
where:
a and dt are two win/loss actions node ht,
a)), E 1) is the binarized value function, representing the expected reward of action at state ht,
Then, the policy that optimizes the Direct Preference Optimization (DPO) objective is identical to the optimal RL policy:
πt" (a ht) a Teet(a ht:) exp ((hina)
(2)
where 8 is the DPO hyperparameter.
The proof for Theorem 1 in (Putta et al., 2024) shows that the policy π" approximates the optimal RL policy. That is, we can approximate the optimal RL policy if we generate preferences under the optimal value function (or an approximation thereof, iee our binarized version as shown below).
In our setting, since a a) {0, 1}, the exponential term simplifies to:
21

---

Published as a conference paper at COLM 2025
exp (8) when ah),,a) = 1,
when @(h), a) 0.
Therefore, the optimized policy becomes:
Keet(a h1)) exp )T) if Qh;,, a) = 1, if (h), =0 O.
π (a ht) a
(3)
Teet(a h1),
This means that the policy Λt increases the probability of selecting actions with Q a) =1 by constant factor relative the reference policy Tee By optimizing the DPO objective with these binarized preferences, we ensure that the policy increasingly favors actions leading to higher expected rewards, aligning with our credit assignment strategy described in Appendix A.6.1. This supports our claim of (approximate) monotonic improvement, as the policy updates move us closer to the optimal policy by consistently selecting actions associated with higher binarized values.
A.7 Additional Related Work
Strategic use of inference-timet compute can also boost accuracy. (Brown et al., 2024) demonstrates that repeated sampling from single model improves coverage, while (Snell et al., 2024) shows that iterative refinement can outperform naive scaling in certain tasks. Meanwhile, supervised fine-tuning (SFT) remains the backbone of LLM adaptation (Devlin et al., 2019; Howard & Ruder, 2018), but can demand large volumes of human-labeled data. Quasi-supervised methods (Yang et al., 2024) address data scarcity by providing supervision for intermediate steps. Finally, (Zelikman et al., 2022) and (Xiang et al., 2025) underscore the effectiveness of structured chain-of-thoughe prompting for complex tasks, while (Cobbe et al., 2021) confirm that an explicit verifier stage reduces errors on GSM8K. Our work unifies the aforementioned techniques to create multi-agent pipeline that orchestrates a generator, verifier, and refinement model, leveraging search, synthetic data generation, and preference-base
Inference Time Compute:
training to enable robust multi-step reasoning.
Multi- Tulli-TTur/ia RL Strategies: Recent work on multi-turn RL (Zhou et al., 2024) focuses on live model interactions over long horizons. MALT instead presents an offline training scheme, where models are post-trained on a preference dataset. Works such as (Shridhar et al., 2023) focus on allowing base LLMs to improve their answers with refinement strategies, and (Qu et al., 2024) further extend such work by allowing for repeated self-improvement turns with the same model. In contrast, MALT focuses on developing a framework where multiple models can reason together to improve overall system performance, and is focused on post-training each model in the system for its specialized role.
A.8
Future Directions
A.8.1 Improving Multi-Agent Systems
Our findings showcase the potential of multi-agent LLM systems optimized with fine-tuning and collaborative inference techniques. There are several future directions from this line of work: Using PPO (Schulman al., 2017) and the exact value propogated backward for each trajectory to update model weights, possibly in an online RL fashion, might produce strong results with additional computational overhead (Ivison et al., 2024). Moreover, we provide several levers around the number of models (where the three model setup can be used iteratively), controlling the branching factor for data generation, examining the effect of majority voting on more samples, changing the attribution threshold, or treating the attribution threshold as an adaptive parameter when iteratively training and rolling out from the multi-agent system (see Appendix A.9). Moreover, prompt-tuning strategies and different roles can be considered or distillation techniques. We note that these are all specific and interesting directions. However, they lie beyond the scope of this paper, where our
22

---

Published as a conference paper at COLM 2025
goal is to introduce new multi-agent post-training methodology and demonstrate strong empirical performance.
As for expanding beyond the Generate- Generate-Verify-RReiie Verify-Refine paradigm, it is possible for an orches- trator (human or automated) to either specify different roles or the number of rounds of interaction required. In case repeated interactions are required, the search tree can just be expanded in terms of its depth, and instead of an exponential production of steps, an MCTS process can be used to more efficiently prune our tree to collect synthetic data at each step. Roles can also be decided by a search process, where several proposed roles collaborate and are pruned until good combination is found for post-training, However, these steps remain beyond the scope of our work, where our goal was to introduce a multi-agent post-training framework that can be conveniently expanded in the near future.
A.8.2 Relation to DeepSeek R1 and LLM Mid-training
With the concurrent release of (DeepSeek-Al et al., 2025, DeepSeek R1) and similar reasoning models, an open gap remains: How do we generate cold-start data to enable better exploration of meta-strategies with reinforcement learning bused training of LLMs? MALT is not a competing, but complementary framework to such advances in single-model reasoning. Importantly, can be used in two specific ways: MALT can provide search trajectories that could be distilled into reasoning paths that can be used for instruction tuning
to prime the LLM for reinforcement learning (i.e. providing the LLM baseline level of exploration /self correction / /self refinement capabilities so that RL can be significantly more efficient). An alternative approach is to use PPO style online RL methods for multi-agent training with the same credit assignment strategy we have described, instead of collecting offline preferences. This is a viable direction in the presence of more computational resources for training and inference.
A.9 Additional Information
For SFT, we used LoRA with a learning rate multiplier of 0.1 and a batch size of 8 to avoid overfitting. For preference optimization, we used Direct Preference Optimization (DPO) with a preference tuning learning rate multiplier to 0.1, training beta parameter of 0.2, and adapter weight configured to 0.2. We varied the number of epochs between to 10 based on the size of the synthetic dataset for each model and leave a deeper exploration of hyperparameter configurations that could require significant amount of compute to future work. SFT training was often until convergence. DPO training did not necessarily converge by the end of all iterations. For the Generator, we find that SFT+DPO actually lowers performance (for e.g. 52.25% on MATH with SFT and 51 51.25% with SFT+DPO) because Llama 3.1 8n-Instt already underwent post-training with DPO on a very similar generator data distribution for benchmarks (Grattafiori et al., 2024), making DPO on similar distribution prone to overfitting, consistent with observations in (Setlur et al., 2024).
We keep our prompts the same for every baseline and trained model on a given benchmark. Our prompts use CoT and zero-shot prompting. We use temperature of 0.3 for Llama 3.1 8B Instruct since it was qualitatively good enough to prevent hallucinations and still led to diverse enough samples. MALT requires the presence of an initial training set containing question-answer pairs, which led to the use of MATH, CSQA, and GSM8K.
During inference for the data collection strategy, using an exponential branching factor does not add significant compute overhead because inference calls can be parallelized when sampling from a model with the exact same input. Moreover, during training, we obtain fixed dataset upon which LoRA fine-tuning can be conducted. LoRA adapters ensure that the model weights themselves aren't 't duplicated, thus requiring only minimal additional memory for the adapters themselves while the base models remain the same. For our ablation with only the Generator and Refinement model, we specify an empty verification to the Refinement model, requiring it to directly refine and improve the generated answer.
23

---

Published as a conference paper at COLM 2025
A.10 Limitations and Ethics Statement
We note that even at low temperatures, model performance on benchmarks often exhibits high variance. To address this within our computational constraints, we conducted evalua- tions on random subsets of test-sets across four seeds. While CommonsenseQA is known to contain many biased or incorrectly labelled questions (Geva al., 2019), we utilized it manner consistent with prior work.
A.11 Acknowledgments
We thank Aleks Petrov, Dulhan Jayalath, Xingyi Yang, Tala Aljaafari, Markian Rybchuk, Kalyan R, Milind Maiti, Divyansh Garg, and Lewis Hammond for their time and insightful discussions, SM dedicates this work to the memory of his grandmother, Mohini Motwani.
24


Reviews:
[{"review_id": "GfEdLxuom8", "reviewer": "Reviewer_Shaf", "summary": "This paper proposes MALT, a post-training method that assigns reasoning sub-tasks to three specialized agents: a generator, verifier, and refiner. It introduces a search-tree-based data generation process with value iteration for automatic credit assignment, enabling supervised fine-tuning and DPO without intermediate human labels. Experiments on GSM8K, MATH, and CSQA show performance gains over both single-agent and inference-only multi-agent baselines.\n\nWhile the system is well-engineered and results are promising, the overall novelty is moderate. The method largely builds upon existing work such as STaR, DPO, and multi-agent debate frameworks, integrating them into a coherent pipeline rather than proposing fundamentally new techniques.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1/ The method yields decent performance improvements across reasoning benchmarks and generalizes well to symbolic and out-of-domain tasks.\n2/ The value iteration credit assignment enables automated training of multi-agent systems without requiring human annotations or oracle models.", "reasons_to_reject": "1/ The core components—multi-agent design, DPO, synthetic search-based data—are all derived from prior work (e.g., STaR, AutoGen, Debate Agents), with MALT functioning as a well-engineered workflow rather than a methodology. Without majority voting, the improvement is also marginal compared to Star baseline (within std).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qJXYOwNlOJ", "reviewer": "Reviewer_rgPo", "summary": "This paper introduces MALT, a framework for improving a model's reasoning by jointly training 3 agents to generate, verify, and refine responses. Experiments show that MALT nontrivially improves performance across reasoning-based benchmarks. It serves as a solid proof of concept for the framework, while leaving ample room for future exploration.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is technically sound, well motivated, and novel. It introduces an interesting framework with many degrees of freedom built in, alongside solid experiments showing the usefulness of the framework.\n\nThe authors have addionally provided fair ablations showing the necessity of each part of the framework, and have provided good discussion around possible issues or questions.", "reasons_to_reject": "My main concerns are around slightly limited experiments - e.g. only 1 model being used for the framework and quirks that come with it. However, I would consider these concerns minor.\n\n- The generator is not trained with DPO and only SFT. A reasonable justification is given in A.10, but this might speak to something more fundamental in the method.\n- In table 1, SA STaR doesn't help the model much (GSM8k goes down; CSQA and MATH seem within noise). STaR MA+MV also doesnt see the gains that inference only MA+MV obtains. This is also consistent with the idea that Llama 3.1 has already undergone much training. However, this begs the question of when MALT is useful: is it only useful for such already post-trained models? What happens for other, less trained base policies, or more trained base policies?\n- A central feature is the tree-styled exponential branching factor. The authors mention in A.10 that future work includes ablating this factor, but I would prefer some experiments around this. \n- The noise levels in table 1 are quite high in some cases, which makes me slightly uncomfortable. However, it's interesting to note that the MALT tests have lower noise in comparison.", "questions_to_authors": "- Is it possible to repeat the core experiments with any other open source (small) model?\n- MALT in general implies that a final model deployed to production requires both MA and MV, as in table 1, MALT without MV performs about the same as inference-only MA, as noted by authors. What happens without even MA, using only the generator, in SA mode? \n- What happens as the number of generations for G, V, and R goes up or down? \n- Since the generator here uses only SFT and not DPO, doesn't it see significantly less data in training depending on the pass rate of the base policy? \n- Is this truly limited to reasoning? Certain justifications are given for negative feedback to improve reasoning, but I see no reason MALT could not also be used to improve soft-verifiable domains (even training against a reward model).", "extra_scores": {}}, {"review_id": "yKYVwVqDsd", "reviewer": "Reviewer_s2iA", "summary": "MALT, short for Multi-Agent LLM Training, is a post-training technique for improving the reasoning capabilities of LLMs. The technique builds on the idea that a \"division of labor\" for reasoning is useful, with one LLM agent responsible for generation of reasoning data, another which does verification/critique, and finally a refiner/editor that takes the critiques and produces a new solution, from which the answer is evaluated. The original element of this paper is the focus on training data instead of focusing on inference time. The authors propose using tree-search to explore reasoning paths, rewarding the paths that lead  to the ground truth answer, and using value iteration for credit assignment. The result can be converted into data for supervised fine-tuning on positive trajectories, or reinforcement learning on the preference pairs (chosen, rejected), for each agent.\n\nThe authors demonstrate that the method produces significant gains on standard math/common sense reasoning benchmarks, on the order of 7-16% over single-model baselines, and 3-5% over multi-agent baselines with majority voting. The ablation studies are convincing that the method produces meaningful gains on important benchmarks when used to post-train the Llama-3.1-8B-Instruct model. Of course, as is common in this field, it is an open question whether larger models will see comparable gains from this multi-step method. Nevertheless, I view this paper as an important contribution to the literature, demonstrating a novel technique that shows promise for improving LLM performance on a significant problem.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper demonstrates that the MALT technique gives significant gains on challenging reasoning benchmarks. The experiments are compelling, with strong baselines, including multi-agent without MALT and majority voting, as well as standard STaR. The technique makes intuitive sense, with a nice theoretical explanation of MALT as a three-step MDP in the appendix. As stated above, the empirical gains of the method are good, and the techniques make sense, being closely related to how humans solve problems. The authors also provide a full background of years worth of related material, and useful examples of the results of the method. Overall, I found the paper to be accessible and clear, with a strong principled approach and solid empirical results.", "reasons_to_reject": "I did not find any major reasons to reject this paper, for the reasons stated above. The major points which would strengthen the work, like increasing the dataset sizes and studying the effects of different branching factors and thresholds, are likely too difficult to perform with a standard compute budget. Although I am somewhat skeptical of multi-agent setups in practice, due to the associated complexity and costs, I can fully believe that this method will be useful beyond agentic systems, for example for training thinking models like DeepSeek R1, as the authors allude to.", "questions_to_authors": "1. I would love to better understand how the authors propose distillation of their trajectories towards training R1-style models. Does this mean selecting the highest value G-V-R path and training with these? Do the authors envision ways to use the lower-value paths to teach the model how to backtrack?\n\n2. Did the authors perform small-scale experiments with $\\theta \\ne 0.5$ or $n \\ne 3$? If so, did the results align with the intuitive explanations provided in the paper?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a new training paradigm rather than playing at inference time, it learns 3 specialized agents via a tree‑based, automatically labeled dataset and combines SFT with preference optimization. The novelty is integration into a unified, training‑time pipeline with principled credit assignment is, in my judgment, a substantive methodological advance that unlocks clear performance gains and good cross‑task generalization, although as reviewers pointed out they are not individually novel components.
The additional Qwen experiments alleviate the single‑model concern and demonstrate scalability to a weaker base policy. Remaining weaknesses compute cost of n³ sampling and lack of very‑large‑model results are genuine but do not, in my view, outweigh the contribution. Reviewer Shaf’s novelty objection is noted, and the rebuttal addresses fairness/ablation questions satisfactorily.

I agree that MALT’s novelty lies in how these components are orchestrated into a coherent, trainable multi-agent system with credit assignment. This is not trivial, prior work has not used value iteration over a reasoning search tree to train separate agents via outcome-based feedback. There is also a strong generalization to out-of-domain tasks on AIME, MMLU-Pro.

highly suggest the authors to include:
1. The promised branching‑factor ablation and expanded Qwen results.
2. Introduction foregrounds the training novelty more succinctly (per rgPo).
3. Compute/efficiency discussion is strengthened (runtime, GPU hours, scaling advice).

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept