ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration
Authors: Andrew Estornell
Jean-Francois Ton
Yuanshun Yao
Yang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Agent Collaboration
LLM Agents
Preference Optimization
Primary area: foundation or frontier models, including LLMs
TL;DR: We train a 2-agent LLM team (one actor-agent and one critic-agent) to collaboratively solve problems.

Abstract: Large language models (LLMs) have demonstrated a remarkable ability to serve as general-purpose tools for various language-based tasks. 
  Recent works have demonstrated that the efficacy of such models can be improved through iterative dialog between multiple models.
  While these 
  paradigms show promise 
  in
  improving model efficacy, most works in this area treat collaboration as an emergent behavior, rather than a learned behavior. 
  In doing so, current multi-agent frameworks rely on collaborative behaviors to have been sufficiently trained into off-the-shelf models. 
  To address this limitation, we propose ACC-Collab, an **A**ctor-**C**riti**c** based learning framework to produce a two-agent team (an actor-agent and a critic-agent) specialized in collaboration.
  We demonstrate that ACC-Collab outperforms SotA multi-agent techniques on a wide array of benchmarks.

Published as a conference paper at ICLR 2025
ACC-COLLAB. AN ACTOR-CRITIC APPROACH TO MULTI-AGENT LLM COLLABORATION
Andrew Estornell
Jean-François Ton ByteDance Research
ByteDDnce Research
andrew andrew.estornen www.:wesssteeeeee...o. wes.tmmmeelyyytecceeecccom
beanfrancols&@bttedddae
Yuanshun Yao
Yang Liu
Meta GenAl kevinyadeneta.com
University of California, Santa Cruz
yangliuluesc.cm
ABSTRACT
Large language models (LLMs) have demonstrated remarkable ability to serve as general-purposs tools for various language-based tasks. Recent works have demonstrated that the efficacy of such models can be improved through iterative dialog between multiple models. While these paradigms show promise in im- proving model efficacy, most works in this area treat collaboration as emergent behavior, rather than a learned behavior. In doing so, current multi-agent frame- works rely collaborative behaviors to have been sufficiently trained into off- the-shelf models. To address this limitation, we propose ACC-Collab, an Actor- Critic based learning framework to produce a two-agent team (an actor-agent and a critic-agent) specialized in collaboration. We demonstrate that ACC-Collat out- performs SotA multi-agent techniques on a wide array of benchmarks.
INTRODUCTION
Recently, large language models (LLMs) have rapidly become cornerstone in various applications,
redefining how we process and generate language at scale (Thirunavukarasu et al. 2023 Hadi et al.
2023 Jiang al. 2024). Their ability handle diverse tasks, from translation (Zhu et al. 2024
Otter et al. 2020) to answering complex questions (Zhang et al. 2024 Hao et al. 2024 Havrilla
al. 2024). has attracted the attention of both industry as well as academia. However, despite these advancements, LLMs still exhibit notable weaknesses, particularly when it comes to answering
factual questions and reasoning TTommoy al. 2024 Rawte al.
2023 Huang et al. 2023).
To address these limitations, several techniques have been proposed, such as Chain-of-Thoughe (CoT) prompting (Wei et al.) 2022). Self-Reflection (Ji al.) 2023 Shinn al., 2023), and Multi- Agent Debate (MAD) (Du et al. 2003). to name a few. These approaches aim to improve the reason- ing abilities of LLMs by guiding them toward more accurate answers through structured thinking or discourse. However, the majority of these techniques do not involve training the model specifically
for these tasks but instead rely on zero-shot or few-shot capabilities.
Similar to most multi-agent paradigms, MAD approaches make use of off-the-shelf general-purposs LLMs, which are not trained to collaborate. Such approaches rely on collaboration as an emergent, rather than a learned, behavior. While, in some cases, these emergent behaviors sufficient, the question remains: Can these methods be improved by imbuing models directly with collaborative abilitics? To answer this, propose training teams of LLMs to solve tasks collaboratively.
particularly relevant work is DebateGPT (Subramaniam et al. 2024), which employs debate as a mechanism to generate higher-qualit, fine-tuning data. Unlike approach, which optimizes LLMs for multi-round collaborative problem-solve solving, their method focuses on using debate to enhance training data for single model that produces individual responses.
Equua contribution. Conrespondence to Work done while at ByteDance Research Code available http:///ghhu..ccmm//ccccosccccc
jeant www.oaaccseeeteteeeecoooooo

---

Published as a conference paper at ICLR 2025
In this paper, we propose novel framework Actor-Critic Collaboration (ACC-Collate which jointly trains a two-agent team to collaboratively solve problems through iterative conversation; this team consists of an actor-agent, responsible for providing answers for given task, and a critic-agent, responsible for assisting the actor-agent with feedback on its answers. In our training pipeline, we introduce a novel off- off-poliey learning scheme called multi-turn training data to enhance the actor's and critic's performance on challenging tasks.
to generate high-quality
To summarize, our contributions are as follows:
We are the first to propose a framework for jointly training a team of LLM agents (Actor- Critic) within the context of collaborative problem solving.
We introduce a novel data generation scheme, "Guided Collaboration Trajectories", which enables the efficient creation of high-quality training data for both the actor and critic roles.
Our extensive experiments demonstrate that our method, ACC-Collab, significantly outper- forms existing state-of-the-an approaches.
RELATED WORK
Our research is closely related to the emerging field of multi-agent deliberation, sometimes called
Multi-Agent Debate (MAD), which examines how to use groups of models to solve tasks through
iterative discussion Chan et al. (2023); Liang et al. (2023
; Du et al. (2023); Li et al. (2023c); Khan
et al.
2024); Michael et al. (2023)] Rasal 2024g g Pham et al. 2023); b Abdelnabi et (2023 Hong
et al. 2023 Irving et al. 2018) Li et al. 2023b d 2024a);|l Wang et al. (2023a) x et al. 2023)
;
Many of these works find that language models have naturally collaborative abilities SSinghal et al.
2023 ;; Du et al. (2023) ; Chan et al. 2223), while others have noted that the collaborative ability of
off-the-shelf models can be quite limited Wang et al. (2024) Smit et al.)
Current approaches to multi-agent deliberation can be broadly cast into two main categories: those that modify model prompts and responses during the discussion Liang et al. (2023); y Khan et al. (2024); Rasal ); Feng et al. (2024);; Yang et al. (2024), and those that modify the structure of the deliberation process et al. (2023a): Hong et al. (2023): Liu et al. (2023) ; et al. (2024c); Wang et al. (2023b); WW et al. (2023); Chen et al. (2023); Chang (2024b). Importantly, both categories use off-the-shelf language models (which have not been trained to collaborate) and work by modifying either the inputs outputs of these models. Deviating from this line of work, we aim to specifically train a team of models to collaboratively solve tasks.
Two works of particular note are that of Subramaniam al. 2024), which proposes use debate data to fine-tune models, and [i et al. (2024b), which trains models for adversarial debate. In the former, debate is used to generate nigher-quality fine-tuning data and is not used at inference time; differing from this work, we train models directly to collaborate and use multi-agent discussion both during training and inference. In the latter, models are trained to be effective arguers rather than collaborators, i.c., models are trained to give conceiving arguments such that they can win debate against other LLMs. Differing from this work, we train models collaboratively solve tasks.
In the context of multi-agent deliberation, the concept of divergent opinions is highly relevant to our method. Several approaches multi-agent deliberation aim control the level of disagreement among the agents Liang et al. (2023): : Khan et al. (2024), Chang (0024)). Often, these works dynam- ically increase disagreement to prevent early convergence of deliberation. In our study, we leverage divergent opinions to generate high-quality training data. In particular, we have agents change their opinion during the discussion and measure whether or not that change increases or decreases the likelihood that the agents' discussion converges correct answer. Using this signal we can then asses the value of a given training example for training the models.
Also closely related to our work are paradigms that aim use self- data to improve model performance, often in the context of reasoning or chain of thought Trung et al. (2024): Huang et al. (2024); Xiong et al. (2024); Chen et al. (2024); Pang al. (2024b). Similar to this line of research, we make use of model generations as traning data. However, we are the first work to use such data in the context of multiple models dehating collaboratively to solve given task.
2

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES AND NOTATION
In this section, formalize multi-agent collaboration between Actor (an agent that provides swers) and a Critic (an agent that provides feedback to the actor) while also introducing the notation that will be used throughout the remainder of the paper.
Let (x, y) ~ be a answer pair source from a distribution of tasks and answers D. For a given task I, two agents for providing feedback and assistance to the actor agent engage in an iterative discussion over T rounds, correctly infer the answer y- Let 0g and 0, be the parameters of actor and critic agent, respectively. The iterative discussion between these two agents is as follows:
actor agent responsible for providing answers and critic agent responsible
1. round & a task is given to the actor 0₂ who provides an initial response 20")
2. Next, still at round 0, the critic 0g views task and (0) then provides feedback 26
(t-1) and the
3. For each round & > 0, the actor views the task æ, its own previous response za critic's feedback ze (t-1) then provides updated response 200
4. After the actor's new response za the critic provides the feedback ze based on 2a
The accuracy this procedure measured via the correctness of the actor's final response, i.e., [[((=()) (3) = v]. Where ( is a function that extracts answers from text-based responses. For example if 2m "The sky is blue", then {(20 (222)) =bluee". With this notation and formalization of multi- agent collaboration, we introduce our framework for training actor-critic teams.
4
Actor Guided Towards
Actor
if Ze e
Natural Debate
(t-1)
(00 100) 10
Actor Guided Away
Aty
elit
A
i
Preference Data
Critic
From y
Train Models
20)
Relative Quality Trajectory
Figure 1: ACC-Collab training pipeline, exemplified for the actor. 1) We generate data from both natural deliberation well as guided deliberation towards and away from the ground truth answer using the actor and critic. 2) We compute the relative quality of each trajectory based on the expected quality difference Ay, Aty w.r.t. the natural response. 3) We store all high-quality pairwise data
our database and train the actor agent. 4) We altemntee this procedure for the actor and critic. See Figure of the supplement for the corresponding procedure applied to the critic.
METHODOLOGY
In this section, we outline our procedure for training a two-agent team, consisting of an actor agent fo, (responsible for providing answers given task x) and critic agent fo, (responsible for providing feedback and assistance to the actor). At inference time, the two trained agents engage in iterative discussion solve given task v, generating the final response (T)

---

Published as a conference paper at ICLR 2025
4.1 AN ACTOR- ACTOR-CRITIC CRITIC COLLABORATION FRAMEWORK
Building upon our established notation from the previous section and the general actor-critic frame- work, formally define our optimization objective as follows]
0a 0. = arg max max (2.)))) { fo. (z, 22 (T=1) 2)
(I)
Bc
actor's linal response =
Intuitively, Eq. aims to simultancously optimize the actor's parameters 9, and the critic's param- eters 8e. ensuring that the actor's final output at iteration T matches the correct answer y. other words, we optimize the accuracy of the actor's response at time T, namely
2((T) fo.
(x, 20 ( 2 2(5-1))
where accuracy is measured as E[(( (2
(T)
It is important to note that the recursive nature of multi-agen agent deliberation introduces significant
complexity to the optimization process. Each response (+)) depends not only on the actor's previous
output 2 (t11) but also on the critic's previous output zc (t-1)) This interaction closely resembles a cooperative dynamic Stackelberg game (Li & Sethi 2017). where two players engage in hierarchical decision-making over time, leading us to adopt an iterative best-response approach (Fiez et al., 2019). In other words, first train the critic agent, followed by training the actor to best respond to the critic's s output. We can then update the critic to adapt to the newly trained actor, and so on. More formally, this process works by first fixing 0a. and solving.
8- = arg max E (a. 5 (fo. 20 (T-1)
for (2, 20 -
))
(2)
de
critic's respuose
then fixing 0% from above, we solve
0- arg max E ( s for x, =(1 for (z, (2, 2(f
0g
this process then repeats until desired stopping criteria is reached. In practice, find that single iteration is sufficient to produce a high-quality collaborative team.
While this alternating scheme allows optimize the actor and critic separately, the objectives of each agent still cannot be optimized directly due to the recursive nature of agent responses in this objective; responses at round 7' depend on those given by the agent at round 1 which themselves depend on the response given at round 2 and so on. To deal with this temporal dependency, we next introduce the concept Partial Trajectory rewards, which will allow us capture the signal of each response (!) for each T.
4.2 PARTIAL TRAJECTORY REWARD
To address the inter-round dependencies of the above optimization, we proposed scheme that allows us determine the "goodness" of given response z(t) (from either the actor the critic) for any & < T. Consider a conversation between the actor and the critic that was paused at time t, i.e. the most recent response is 24)) To assess the goodness of 20(), one might ask how likely the deliberation procedure will converge the correct answer y round T, given that the procedure is already at response (t), Formally,
define this as
r(2 x. v) E ((= vl x, (1)
For clarity, we note that term arg max max captures the solution for both parameters Os, Og Ps 96
corresponding bi-level max-max optimization. Here 0g is in 2 (T-1) fo. (s, 2T=2)11))
4

---

Published as a conference paper at ICLR 2025
Intuitively, the partial reward captures the expectation of arriving the correct answer through deliberation starting at round / with generation 2(1), In practice, r(2(0), y) can be estimated by learning the reward r or by using heuristics such as one-step roll-out, i.e., Monte Carlo estimation.
additional deliberation round multiple times from response 2(() The reward r(2((), z, y) is set as the average accuracy of these simulations. Empirically, we find this approach effective for generating high-quality training data. We leave learning-based reward functions for future work.
In our experiments, we use one-step roll-out heuristics, i.e. simulating
Our objective will then be to optimize the parameters of the actor and critie, Ga, tc. so that the responses produced by these agents each timestep namely 2(1), maximize r(z(4), 3, y) That is, optimize the actor and the critic that each timestep I, they give a response 2 which has high probability of leading the deliberation to converge to the correct answer at time T.
To optimize the objective in Eq. 1 we will utilize preference optimization, a standard technique in LLM training. Using the iterative maximization scheme described above, we first have to gather pairwise preference data for both the actor and the critic. In the following sections, we first detail our process for generating this preference data before delving into the optimization procedure.
Algorithm 1: Trajectory generation and selection
Data: Actor and critic:
0a, 2,, Distribution of tasks D, Reward threshold Result: dataset trajectories D
D 0 1* Set of trajectories use */
for (x, y) ~ 'D do
z z(0) OnceDellberratioRRuumd. /* actor and critic responses, i.e. (26 200))
for t in [1, do
z (t) +- OnneDDDlibrattioooumddd.. ()- updated natural responses*/
/* guided--deliherrttion towards, and away from, correct answer y */
OneGGuidedDeilleettionoound (æ: (x,,(((11)) (((-1) .
OneGuiddedDellieertionnooo (z, 2((-1),11)
Estimate final round accuracy if deliberation continue from response 2(!),
i.e. r(2(1),,, x, y), r(24" x, y), r(2("),,, ,I,9) */
u + EstimateFinalAAcurraey
0 FstlmatteFiallAcccrrayy (z..)
0_ EssimatteinllAAccurrayeZ
1 Compute the expected improvement for each trajectory */
* Save trajectory pairs that result in sufficient accuracy improvement */
if 04
2 then
D add (pos=z (1)=
end
else if e e_ 2 then Daadd(pos=z) ()) neg=z (1)
end
end
end
4.3 OFT-POLICY TRAJECTORY GENERATION
In this section, we describe how to generate the preference data needed to optimize the objective in Eq. [ The classification of a sample as positive or negative is determined by the deliberation trajectory it follows. Specifically, a positive sample for training the actor corresponds to trajectory likely to lead to the correct answer at round T, while a negative sample corresponds to one that leads to incorrect answer at round T. Intuitively, that lead to correct answers while reducing responses that are unlikely to do so, thus optimizing for Eq. The same principle applies to the critic when optimizing Eq.
aim to push the actor agent to generate responses
5

---

Remark Under Eq. 5 a positive example 2 can be interpreted as a guided response 2y which increased the probability of deliberation converging the correct answer by at least e, when com- [(() Published as a conference paper at ICLR 2025
With this intuition in mind, now describe how such data is generated. A deliberation trajectory can be defined as a sequence of responses (za 2c Aa (1) 2a 2 for a given task x. A straightforward way to generate preference data would be to generate multiple rollouts at each round and select the trajectories with the highest r(2(t), .z 3) as positive samples and those with the lowest r(2(*), 2, y) as negative samples. This approach could enforce the desired behavior for both the actor and the critic if enough samples are collected.
However, this approach is not without its limitations. In particular, if the agent performs poorly on given dataset, may be difficult collect enough positive samples, resulting low training signals. Additionally, even if the agent performs adequately, generating sufficient responses for both the actor and critic requires significant computational resources, especially to ensure that high r(2(1, x, y) values are used for positive samples and low values for negative samples.
4.4 GUIDED COLLABORATIVE TRAJECTORIES
To address these limitations and improve efficiency, we propose Cruided-Collaboriie Trajectories, which steer the deliberation procedure in two opposing directions: towards, and another away from, the correct. By comparing these guided trajectories with the natural deliberation trajectory, can assess the relative goodness of each trajectory using estimation of the reward structure r.
Specifically, for task with answer y, let (t-1)) (za--1) Ze ( ) be the agents' responses at time t 1. Let (za") 22 be the agents' natural responses (i.e., without guidance), let (2) (t)
2g.c) and
(= *lg.c) be the agents responses when guided towards, and away from, supporting answer respectively. Thus, each guided response is an off-policy generation. In practice, we want guided responses to be different enough from natural responses so that learning the guided responses results consequential changes to the agent, but not so different that they are challenging to learn; we find that prompt modification is an effective tool for striking this balance. To guide the generations of (zy.a: respectively-
(t) 210) and (2 2g we will simply provide a correct and wrong target answer in the prompt,
"Guided Collaborative Trajectory Prompts" in Section C for further details.
For each guided response, we consider how influential this response was in altering the accuracy of the final response, i.e., in the case of the actor, we define
Ag = r( r(2 x, v) ((22) .. and Dty r=22)) x, - r(aig,. x.
The terms A, and Dty give the expected accuracy difference if at round f the actor had given re- sponse zy.a (or "Iy.a) instead of response z(!) Large A, indicates that one-response difference during the deliberation was sufficient to push the procedure toward the correct answer. Such re- sponses would be desirable for the agent to learn. On the other hand, large values of Aty indicate that a one-response difference easily causes the agents to converge to the incorrect answer; this indicates that the deliberation procedure is particularly fragile at timestep
(()
With these observations in hand, we use Ay and Dry to define positive an negative examples, in particular for threshold E,
v, - r(xx (1) (2(0) (0) if es Ayy = r(2()) x. 9) r r(z1g,a> x.
20)) if e = r(z
y)
(5)
(t)
neither value is above the threshold, then the example is thrown out.
(t)
pared with the natural response 2(0, Similarly, a negative example 2()) is a guided response Ey) which decreased the probability of deliberation converging to the answer by at least E.
Now that we have procedure for generating high-quality training examples consisting of positive and negative pairs, we next discuss how use those positive and negative pairs to train both agents.

---

Published as a conference paper at ICLR 2025
4.5 LEARNING ON GUIDED TRAJECTORIES
In order to optimize each objective (Eq. and 3p. we use standard preference optimization Direct Preference Optimization (DPO) Rafailov et (2024). We choose DPO for its efficiency, but any preference optimization scheme could be used (see section B of the supplement for details on how other preference optimization schemes may be incorporated).
Hence, given preference dataset of positive and negative examples for both the actor and critic agent, of the from 20 (4) the DPO loss is defined as,
To (z.. xc, 10,zz" 2((-11) Freel J. 2 (1--1)) Teatl _ b. (((-1)
To (2 z z(t-11
DPO Σ E (n (a
log
(6)
Where Te is the policy induced by parameters 0g or Og and (t--1) are the agent's responses the previous round (i.e., the responses prior to giving either response 2!) or 2
Remark Recall, given our generated preference data, this loss implicitly optimizes the reward r(z', , 2) which itself is equivalent to final accuracy (i.e., the quantity being maximized by Eq. [
By summing across all rounds, we implicitly maximize the probability that each round yields response 2' which causes deliberation to converge the correct answer time T.
EXPERIMENTS
Benchmarks To evaluate the efficacy of ACC-Collab we make use of 5 standard benchmark tasks: BoolQ Clark er (2019) 112 yes-no reading comprehension questions, MMLU Hendrycks et al. (020) ~ 15k multiple choice questions covering wide arry of subjects and difficulty, BBH
MMML.UHendryyks
Suzgun et al. (2022) ~5kk mixed-type questions SCIQ Welbl et al. (2017) 133 multiple-choice
science questions, ARClChollet (2019) ~7k multiple-choice reasoning-basse based questions.
Baselines We compare ACC-Collab to several multi-agent and single-agent baselines. For inference-based methods, compare to Society of Minds SoM Du al.) (2023), Persona Chan
et al. 2023). For training- -based methods, we compare to supervised fine-tuning SFT Radford (2018) DebateTune Li al. (2024b), DebateGPT Subramaniam et al. (2024). We use three dif-
ferent base models: Llama-3-8B- -nnttruu Llama-3 Dubey et al. (2024), Mistral-7B-Instue Mistral
Jiang et )023), and Gemma-2-21 InItrrctt Gemma-2 Team al. (2024).
5.1 ACC-COLLAB PERFORMANCE
Final Answer Accuracy We begin by examining the performance of our method ACC-Collab (a single round of training) and ACC-Collab+ (two rounds training) In table we see the average accuracy of each method after five rounds of deliberation. Our method attains superior performance compared with baseline methods in most cases. The high efficacy of ACC-Collab relative to the baselines indicates that in most cases, only a single round of training is necessary to produce high-quality collaborative team. It is worth noting that in some cases, further training rounds may decrease performance (i.e., ACC-Collab+ we have hold-out set of tasks to determine whether further training degrades performance.
have worse performance than ACC-Collab). Hence,
Process Accuracy In addition measuring the correctness of the actor's 's final answer (i.e., out- come accuracy), we also analyze the correctness of the team's 's reasoning and discussion steps (i.e., process accuracy). Since ground truth not available for reasoning and discussion steps, we use GPT-4o an oracle to evaluate whether the agents follow the correct steps to reach the final answer. We provide an outline of the experimental setup and full results in Section A. of the supplement. We find that our method improves maintains the process accuracy of the actor and critic.
5.2 PERFORMANCE INCREASE OF MULTI-AGENT COLLABORATION
Average Improvement As noted in Du et al. (023). the key mechanism behind the success of multi-agent deliberation (or any of its many variants) that discussion multiple rounds allows
7

---

Published as a conference paper at ICLR 2025
Llama-3
(Ours)
(Ours)
SoM (2x)
SoM (4x)
Persona
Dehate Tune
SFT
DebateGPT
ACC-Collab
ACC-Collab+
BoolQ
.812
.811 0.007
4.014.002
.775 ana
4.98
_815
888__000 -000
.894 003
MMLU
..24.004
665 004
.639.0044 004
4.000
.642..005
000
.644..1
.683; 012
BBH
508 2008..00 003
514 005
4.09
508 2.06
552+.000
.551 oos
593-..006
.574+.00
SCIQ
.255 002
.223+
.925 001
.924
.92
.932+
.952=00
.948..00
ARC
.874
1874
87
.871 002
8899
.876 o0z
.881 .811.00
8809
Mistral
(Ours)
(Ours)
SoM (2x)
SoM (4x)
Persona
Debate Tune
SFT
DebateGPT
ACC-Collab
ACC-Collab+
BoolQ
801+ 005
7988 008
.831
..3
8444.00
.818 002
.87=
.893
MMLU
4.00.
562
.744..002
5622..m o0s
504
.577 002
.61..00
672 oot
BBH
.288 002
462 003
1611.011
.56 o0s
430 0.0
.48 212
.199 519__um 000
.011 034
SCIQ
856
002
856
002
.002.00
8633 0003
888
881 00s
.902 005
.905 002
ARC
8844 001
823 0002
1277.00
.83__
.822g 002
.843_
.856 003
Gemma-2
(Ours)
(Ours)
SoM (2x)
SoM (4x)
Persona
DuneteTune
SFT
DebateGPT
ACC-Collab
ACC-Collab+
BoolQ
.754.011
7599 098
716
.767
.813g.01
.812 00s
.84
8845 00s
MMLU
.58
5588 002
.577
.578
.579
.582
.51
.5551.00s
BBH
4544 ou7
.449..0
4474.000
.447
1.8
_491
_.08
.475
SCIQ
.03 002
.003 002
.984..03
..03
.9134.002
.91 ..02
.918 2.08
.9094.00s
ARC
8411..00 ona
1843 005
.847.000
.047
.888
.851
.8522.068
.849 000
Table 1: Average accuracy (with 95% confidence intervals) after 5 rounds of deliberation. For each dataset, the highest accuracy is shown in bold.
Llama-3
Mistral
Gemma-2
Improvment
5-
SoM (2x)
ScM (4c)
Persona
Debate Tune
SFT
DebetaOPT
BSl ACC-Collab (Ours)
ACC-Collab + (Ours)
Percent improvement in accuracy after five rounds of deliberation, compared to a single round. Percent improvement (Eq. 7 for each method is averaged across all five datasets.
Figure 2:
the models to iteratively refine their answers. Thus, a natural question for any iterative multi-agent method is: how much does accuracy improve from the initial round 0 to the final round T? Where T = 4 in our experiments. To measure this, we look at the percent improvement in model accuracy from round 0 to round = 4 calculated as,
acC4
acco
where acc; is accuracy at round t
acCo
In Figure 2 we see the average percent improvement for each method, averaged across all 5 datasets. For each of the three base models, ACC-Collab+ has the highest average improvement compared to all other methods. Additionally, the improvement gained by methods such as SoM, SFT or De- bateGPT is far less stable than that of ACC-Collabt, In particular, for Mistral, SoM yields nearly no improvement, similarly SFT and DebateGPT offer little improvement when applied to of Llama-3.
Per-Round Accuracy Next, we look more closely the accuracy of each method across five rounds of deliberation. Figure 3 shows per-round accuracy on the BoolQ dataset. As already illustrated by Table [] ACC-Collab and ACC-Collab+ achieve higher final round accuracy than the other methods. Notably, our method has higher accuracy both at the final round t 4 and at round t O. Recall that at round that in some cases, our training pipeline can produce actor agents with superior zero-shot accuracy (without deliberation) compared to models produced by SFT and DebateGPT.
0, the actor's response is independent of the critic. This indicates
Interestingly, observe that in cases, SFT and DebateGPT are not much better than simple deliberation with untrained models (i.e., SoM-4x), e.g., BoolQ with Llama-3. This is primarily due to the fact that these methods are designed improve the model's single-short performance but do
8

---

Published as a conference paper at ICLR 2025
Llama-3
Mistral
Gemma 2
0.90
0.050
0.88
0.86
0.800
0.775
0.775
0.750
0.02
0.750
0.725
0.80
0.700
Round
Round
Round
SoM
SoM
DabataTune
DebateGPT
ACC-Collab
ACC-Collab
Llama-3
Mistral
0.95
0.50
0.92
0.54
0.88
0.93
0.90
0.86
0.89
0.84
Round
Round
Round
SoM
SoM
Persona
DebateTune
DebateGPT
ACC-Collab (Ours)
ACC-Collab (Ours)
Figure 3: Accuracy over five rounds deliberation on BoolQ (top) and SCIQ (bottom).
little improve their collaborative abilities. Cases such as this outline the necessity of training to improve collaboration (ACC-Collab) rather than training for raw accuracy (SFT and DebateGPT).
5.3 INDIVIDUAL PERFORMANCE
Next, examine the relative effectiveness of both the actor and the critic. To do this, train an actor and critic via ACC-Collab. Then, during deliberation, we pair the trained actor with an untrained critic and pair the trained critic with untrained actor. In Table column "Actor" corre- sponds to the former, while "Critic" corresponds to the latter. On average, the trained actor attains higher accuracy compared to the trained critic, this aligns with intuition as the actor is responsible for providing answers while the critic plays a supporting role. In most cases, the trained actor (paired with untrained critic) outperforms SFT and DebateGPT. When the trained actor paired with trained critic (either ACC-Collab or ACC-Collab+), its performance is further improved.
With the decline honey bee populations, plant species that Final Answer: (B) change their flowers that wind fertilize them.
X
Actor
Your response is mostly correct... (continues to agree with the actor).
Untrained Critic
While your response is valid explanation for why option B could be possible answer, here are two reasons why option C is also correct and refutes your response directly:
1. [Counterpoint to for why B could be wrong)
Trained Critic
2.
(Evidence for why C is correct)
Figure 4: Comparison of responses from the critic model before and after training with ACC-Collab.
5.4 WHIA DO THE AGENTS ACTUALLY LEARN?
Lastly, we are interested understanding how ACC-Collab improves the Actor-Critic Team. Figure demonstrates example of the difference in responses between untrained critic and critic trained through ACC-Collad Although the actor provides a wrong answer, the untrained critic too agreeable and does not provide substantive feedback for the actor correct their answer.
4

---

Published as a conference paper at ICLR 2025
Llama-3
SoM (4x)
DebateGPT
Actor
Critic
ACC-Collab
ACC-Collabt
BoolQ
.811
.05
2.7
.804
.887..005
.804.003
MMLU
.635..004
.054
.651+.012
.65..00
.644+.01
_613_.012
BBH
.004
2.08
.5132.01
.55..015
+.06
.574+.003
SCIQ
.9231.002
.9322.001
-947..002
.945..001
.952+.0
.948
ARC
.8744.007 .000
.826..002
.885 000
.866 002
.801
.869.002
Mistral
SoM (4x)
DebateGPT
Actor
Critic
ACC-Collab
ACC-Collab+
BoolQ
.798 1.04
.848 1002
18731.002
8431.002
8771.002
.893 _902
MMLU
.5621.005
1.072
15981.002
.6111.001
.611.005
.672
BBH
4622.003
.482.012
.413
.518+.005
5199
.601
SCIQ
8564.002
+.01
+.01
+.91
.902..005
.905
ARC
8233 .002
.822 .002
.003
.842 .003
.843
.856_
Gemma-2
SoM (4x)
DebateGPT
Actor
Critic
ACC-Collab
ACC-Collabt
BoolQ
.7591.004
.812 1.003
.839..om
1.74
.84 1.00
.845 1.05
MMLU
5781.002
.5821 1.002
1.99..026
.5661.002
.51 1.016
.555 1.051.003
BBH
.449..0
.491 1.01
.51 1.011
.4751.004
.475
SCIQ
.903..002
.9141.002
.923 2.00
.912
.918 2003
.909
ARC
8434..00
.851=.003
..04.002
.855..002
.8522.003
.849
Table 2: Accuracy after rounds of deliberation. The Actor (Critic) column corresponds to an
actor-agent (critic-agent) trained via ACC-Collab and paired with untrained Critic (Actor) during deliberation.
In contrast, the trained critic more willing to disagree with the actor and provides more detailed feedback. Largely, we observe that this trend is common; untrained critics are too agreeable and are thus less able to change the actor's mind, while trained critics are more willing to disagree (see Section C.2 for more examples). When examining the trained actor's responses, notable qualitative change compared to the responses of an untrained actor. However, as shown previously, do observe qualitative change in the actor's responses to become accurate.
do not find
CONCLUSION, LIMITATIONS AND IMPACT
In this paper, propose ACC-Collab, novel framework for jointly training two-agent team (one actor-agent and one critic-agent) to collaboratively solve problems through iterative discus- sion. To train these agents, developed an off-policy data generation scheme dubbed "Guided- Collaboration", which produces high-quality preference data for collaborative models. We found that ACC-Collat outperforms all baselines on wide array of domains. In particular, even a single round of training for both the actor and critic results in a high-quality team. Of particular note is the effects that ACC-Collab has on the critic model. Without ACC-Collab, the critic model is often too agreeable and lacks verbosity in their responses. In contrast, after training with ACC-Collab, the critic is far more likely to provide detailed disagreements during discussion.
However, our framework ACC-Collab also comes with limitations. Firstly, even though ACC-Collab attains superior performance compared to baselines on wide array domains, is important to note that conduct experiments mainly question-answe answering tasks; thus, it remains to be seen whether such a framework would continue to effective in other types of tasks. Moreover, in our experiments, we train and models the same task (partitioning each task into a training and testing set). As such, the generalizability of each actor-critic team to unseen domains is unknown. Our method makes use of the fact that for each question, correct and incorrect answers can be easily established. Secondly, while we provide results for three families of models, these experiments are performed 2B, 7B, and 8B models. While our method is effective for these sizes (standard in open-source models), it remains to be seen whether this effectiveness will scale to larger models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Li Hang for his valuable insights and guidance during the development of this work.
REPRODUCIBILITY STATEMENT
Here, we outline the details necessary to reproduce our results. We provide an algorithm our data generation procedure (Algorithm[]) as well description of our training procedure in Section 1.3 Each dataset, baseline method, and base model used are specified at the beginning of Section provide additional experimental details in Section A Prompts used for method can be found in Section C
Lastly, we publicly release the code used for our method.
REFERENCES
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Llm- deliberation: Evaluating Ilms with interactive multi-agent negotiation games. arXiv preprint arXiv:2309.17234. 2023.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 2023.
Edward Y Chang, Evince: Optimizing adversarial Ilm dialogues via conditional statistics and infor- mation theory. arXiv preprint arXiv:2408. 14575, 2024a.
Edward Y Chang. Socrasynth: Multi-lIm reasoning with conditional statistics. arXiv preprint
arXiv:2402.06634, 2024b.
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse Ilms. arXiv preprint arXiv:2309.13007 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. URL httts://///axx
brg/abs//401. 01335
François Chollet. On the measure of intelligence. arXiv preprint arXtv.1911.01547,200
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044 2019.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. /4325, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Shangbin Feng. Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't hallucinate, abstain: Identifying Ilm knowledge gaps via multi-llm collaboration. arXiv preprint 4rXiv:2402.00367 2024.
Tanner Fiez, Benjamin Chasnov, and Lillian J Ratliff. Convergence of learning dynamics in stack- elberg games. arXiv preprint 1rXiv:1906.0122: 2019.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah. Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.

---

Published as a conference paper at ICLR 2025
Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang. Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. Llm reasoners: New evaluation, library, and analysis of step reasoning with large language models. arXiv preprint arXhv:2404.05221, 2024.
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravin- skyi, Eric Hambro, and Roberta Railneau. Glore: When, where, and how to improve Ilm reasoning via global and local refinements. arXiv preprint arXiv:2402. 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.055 03300, 2020.
Sirui Hong, Xiawu Zheng. Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang. Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi- agent collaborative framework. arXiv preprint arXiv:2308. 00352, 2023.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang, Qianglong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.0523: 2023.
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-drive data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333, 2024.
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00883 00899, 2018
Ziwei Ji, Tiezheng Yu, Yan Xu, Naycon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating I1m ballucination via self reflection. In Findings of the Association for Computational Linguistics:
EMNLP 2023, pp. 1827-1843, 2023.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.068 2023.
Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al. (MegaScale]: Scaling large language model training to more than 10,000 (GPUs). In 2/st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp. 745-760, 2024.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Ed- ward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive Ilms Icads to more truthful answers. arXiv preprint 24X2::2402.06780 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 611-626, 2023.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760 2023a.
Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. arXiv preprint arXiv:2310. 10701, 2023b.
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. arXiv preprint ar 402:2242..05/22 2024a.
Ming Li. Jiuhai Chen, Lichang Chen, and Tianyi Zhou. Can llms speak for diverse peo- ple? tuming Ilms via debate to generate controllable controversial statements. arXiv preprint arXiv:2402. 10614 2024b.
12

---

Published as a conference paper at ICLR 2025
Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.022760 2023c.
Tao Li and Suresh Sethi. A review of dynamic stackelberg game models. Discrete & Continuous Dynamical Systems-B, 22(1):125, 2017.
Yuan Li, Yixuan Zhang, and Lichao Sun. Metaagents: Simulating interactions of human behav- iors for llm-based task-oriented coordination via collaborative generative agents. arXiv preprint arXiv:2310.06500 2023d.
Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grahowski, Yeqing Li, and Eugene Ic. Improving multi-agent debate with sparse communication topology. arXiv preprint arXiv:2406. 11776, 2024c.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang. Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305. 19118, 2023.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Iim-agent network: An 1lm- agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023.
Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmaku- mar, and Samuel R Bowman. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702, 2023.
Daniel W Otter, Julian R Medina, and Jugal K Kalita. A survey of the usages of deep learning for natural language processing. IEEE transactions on neural networks and learning systems, 32(2): 604-624 2020.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.1973 2024a.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston, Iterative reasoning preference optimization, 2024b. URL /Rtpps//////rr/
abs/2404. 19733
Chau Pham, Boyi Liu, Yingxiang Yang. Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A Plum- mer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. arXiv preprint 20.0122310.00622,, 2023.
Alec Radford. Improving language understanding by generative pre-training. 2018
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Sumedh Rasal. Llm harmony: Multi-agent communication for problem solving. arXiv preprint arXiv:2401.01312, 2024.
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.0599 2023.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL
https: /tats://arxiii wwtpx//..r//pp//333..1
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.096. 20911.99......... 2023.
Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. Should we be going mad? a look at multi-agent debate strategies for 1ms.. In Forty-first Interna- fional Conference on Machine Learning.
13

---

Published as a conference paper at ICLR 2025
Vighnesh Subramaniam, Antonio Torralba, and Shuang Li. Debategpt: Fine-tuning large language models with multi-agent debate supervision. 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09200 2022.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models medicine. Nature medicine, 19(8):1930-199) 2023.
SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of ballucination mitigation techniques in large language models. arXiv preprint arXiv:2401.013 001313, 2024.
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reason- ing with reinforced fine-tuning, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Boshi Wang, Xiang Yue, and Huan Sun. Can chatgpt defend its belief in truth? evaluating Ilm reasoning via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11865-11881, 2023a.
Qineng Wang. Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of Ilm reasoning: Are multi-agent discussions the key? arXiv preprint arXiv:2402. 18272, 2024.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Unleashing the emergent cognitive synergy in large language models: A tosk-solving agent through multi self-collaboration. arXiv preprint arXiv:2307.05300 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24837, 2022.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions.
arXiv preprint arXiv: 170700620,, 2017
Qingyun Wu, Gagan Bansal, Jicyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi- agent conversation framework. arXiv preprint 0rXiv:2308.08/555 2023.
Weimin Xiong, Yifan Song. Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! Ilm agent learning via iterative step-level process refinement, 2024. URL UtRLbtts:://xxxit http:///awvvwv.o/aaa//
2tt.aa////244... 11176
Ruixin Yang. Dheeraj Rajagopa, Shirley Anugrah Hayati, Bin Hu, and Dongyeop Kang. Con- fidence calibration and rationalization for 1lms via multi-agent deliberation. arXiv preprint arXiv:2404.09127, 2024.
Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for Ilm agents: A social psychology view. arXiv preprint 1rxiv:2310.02/223 2023.
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm as mastermind: survey of strategic reasoning with large language models. arXiv preprint arXiv:2404.01230, 2024.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis, 2024.
arxiv..vvvv.. org/ abs /204. 04675
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXPERIMENTS
Here, we provide additional details regarding our experiments.
Datasets Each dataset is split into training set, validation set, and testing set. For datasets that come with an explicit partition of these sets we use the given partitions: this includes BoolQ, MMLU, SCIQ, and ARC. For BBH, we randomly sample roughly 25% and 10% of the questions from each category in BBH to create a test and validation set, respectively; this comes out to 1260 questions for the test set and 500 questions for the validation All results reported on questions in the test set.
Compute All training was performed on single Nvidia-H800 GPU. Inference for Llama-3 and Mistral based models is performed on single Nvidia-v100 GPU, for Gemma-2 based models we used a single Nvidia-H800. All inference performed with the VLLM library Kwon et al. (2023).
Training Training for all models was performed via the trl library, using LoRAs of size 256. When training ACC-Collab with DPO we use a negative log-likelihood (NLL) regularization term (with weight 1) as outlined in Pang et al. (024a).
2
Actor Guided Towards
(0
Actor
2f
Ii-1)
A
Natural Debate
1--1)
Actor Guided Away
elif A.g 2
A
Critic
From
3
Preference Data
Train Models
(n
Relative Quality of Trajectory
2
Critic Guided Towards
(0)
Actor
if A
A
(r-1)
20
Ze
Natural Debate
(--1)
20
(0 Ce
Critic Guided Away
elif A Z
A
Critic
From
Preference Data
Train Models
20
Relative Quality of Trajectory
Figure 5: ACC-Collab training pipeline, exemplified for the actor (top) and critic (bottom). The process 1) We generate data from both natural deliberation as well as guided deliberation towards and away from the ground truth answer y using the actor and critic. 2) We compute the relative quality of each trajectory based on the expected quality difference Ay: Aty w.r.t. to the natural response. 3) We store all high-gn high-quality pairwise data our database and train the actor model. 4) We alternate this procedure for the actor and critic. As outline in Section Daation/bbbtt both the guidance procedure and the computation of Ay, Ary differ between the actor and critic.
15

---

Published as a conference paper at ICLR 2025
A.1 CORRECTNESS OF REASONING STEPS
In addition to measuring outcome accuracy (i.e., the accuracy of the actor's final answer), we also measure process accuracy (i.e., the accuracy of the reasoning and discussion steps taken by the actor and the critic model). These results displayed in Table 3 We observe that our training pipeline typically maintains or increases the accuracy of the reasoning steps.
Unlike outcome accuracy, process accuracy does not have ground truth answers. Thus utilize GPT-4o to serve as an oracle. We use the following steps to evaluate the correctness of the reasoning and discussion steps.
1. For both untrained actor-critic team and trained actor-critic team (trained via ACC- Collab), we randomly sample 500 deliberations, which resulted in the correct answer after 5 rounds of deliberation.
2. We then prompted GPT-4o to evaluate the correctness of the reasoning and discussion steps taken by the actor-critic teams in each of the 500 deliberations.
3. We then compute the average accuracy of the reasoning and discussion steps as evaluated by GPT-4o.
Dataset
ACC-Collab+ Accuracy
ACC-Collab Accuracy
Untrained Accuracy
Llama-3
BoolQ MMLU
.890+.035
.894
.794+.045
.968+.020
.976..017
.909.032
BBH
.119..04
.5752.055
.610+.055
SCIQ
.958+.022
.981+.015
.937
ARC
.8822.046
.833
.8834.00 ..06
Mistral
BoolQ
.907 1.055
.928..029
.894 1.044
MMLU
.822.041
.884
.888
BBH
.766+.045
.818+.043
,538 056
SCIQ
.9734.018
.9722.018
.933
ARC
.927..029
.952..024
.852 ..40
Gemma-2
BoolQ
8694.088
.872..037
$614.039
MMLU
.844.041
.843..041
.804
BBH
5577+0056
.637..054
,514+ 056
SCIQ
.984+.01
.978+.015
.985
ARC
.850
.8541.039
834,.002
Table 3: Accuracy of the actor's and critic's reasoning and discussion steps evaluated by GPT-4o.
B PREFERENCE OPTIMIZATION
Here, we remark on how other preference optimization schemes can be used place of DPO. Broadly speaking, preference optimization schemes can broken into two categories: those which optimize reward directly (c.g., PPO) and those which optimize reward indirectly (e.g.. DPO), In the case of the latter, the positive and negative pairs produced by method (Algorithm) can be directly plugged into the preference optimizer.
In the case of explicit reward maximization, the reward function r(2(9) xr, y) can first be learned automatically by simply simulating deliberation. The most straightforward way to do this is to
first simulate one full deliberation between the actor and critic, i.e.. ((zl)) ((0) )....... ... (2(T)) 20 ('))).
Then for each (t) (t) the remaining T' deliberation steps can be resampled to estimate the
corresponding value of r(=()) x. v) and r(=(") 2r, .) These pairs, namely (=1²) r(2 x. y)) and
16

---

Published as a conference paper at ICLR 2025
(=e ; .r(=() Hr v)) can then be used learn the reward function. This reward function can then be
plugged into the desired preference optimization scheme.
Liama-3
Mistral
Gemma-2
0.50
0.830
0.88
0.000
0.775
0.82
0.725
0.00
0.700
Round
Round
Round
(2x)
(Ours)
(Ours)
Figure 6: Accuracy over five rounds of deliberation on BoolQ.
Llama-3
Mistral
Gemma-2
0.60
0.50
0.65
0.66
0.50
Accuracy
0.60
0.64
0.50
0.62
0.50
Round
Round
Round
SoM
ACC-Collsb (Ours)
ACC-Collab
Figure 7: Accuracy over five rounds of deliberation on MMLU.
Llama-3
Gemma-2
0.60
0.56
0.54
0.48
0.50
0.45
0.46
0.48
0.40
0.44
Round
Round
Round
DebateTure
DebateGPT
ACC-Collab
ACC-Collab
Figure 8: Accuracy over five rounds of deliberation on BBH.
C EXAMPLES
C.I PROMPTS
Here, we provide examples of prompts used in our experiments. For illustration, we provide prompts for the BoolQ dataset in which agents are asked yes-no question about a passage.
Single-Shot Prompt (No Deliberation)
prompt = ('You will be given yes-no question which is based on passage. 'You should use the passage to help you answer the question. You should give brief justification for your answer, and you must provide a final answer of either Yes or No.'
AnQuestion: (question]?'
AnPassage:
(passage)'
17

---

Published as a conference paper at ICLR 2025
Llama-3
Mistral
Gemma-2
0.90
Accuracy
0.94
0.88
0.91
0.93
0.86
0.92
0.91
Round
Round
Round
SoM (2x)
BoM (4x)
Persona
Debate Tune
SFT
DebateGPT
ACC-Collab (Ours)
ACC-Collab + (Ours)
Figure 9: Accuracy five rounds of deliberation SCIQ.
Llama-3
Mistral
Gemma-2
0.850
0.85
0.84
0.570
0.55
0.860
0.82
0.81
Round SoM (4x)
Round
Round
SoM
Parsona
DabateTune
SFF
DebataGPT
ACC-Collab (Oun)
ACC-Collab (Ouns)
Figure 10: Accuracy over five rounds of deliberation on ARC.
Guided Single-Shot Prompt (No Deliberation)
prompt
('You will be given yes-no question which is based passage. 'You should use the passage to help you answer the question
'with (target_answer),
"You should give brief justification for your answer of (target_answerl, and you must state that your final answer is (target_answer). Anquestion: !question]?'
AnPassage: (passage)"
Deliberation Prompt for Actor
prompt = (' Several people have provided answers to yes-no question.
Below are their responses:'
\nPerson (1) said: (responses [1]}'
AnPerson (2} said: (responses [2]
AnPerson {n} said: (responses [n]
AnAnYou should take these answers into consideration when answering
the following yes-no question, which is based on a passage.
You should give brief justification for your answer, and you must provide a final answer of either Yes or No. Anquestion: (question)'
AnPassage: (passage)'
Guided Deliberation Prompt for Actor
prompt = (' Several people have provided answers to a yes-no question.
BBllow are their responses:'
18

---

Published as a conference paper at ICLR 2025
AnPerson (1) said: (zesponses [1]}
Anerrsn {2] said: (responses [2] }
AnPerson {n} said: (responses [
You should take these answers and the passage into consideration when
answering the following question with [target_answer),
You should give brief justification for your answer of (target_answer),
Anquestion: (question)'
AnPassage: (passage)"
Deliberation Prompt for Critic
prompt
('I am answering question based on passage.
'I would like you to help me improve the correctness of my response by briefly providing some additional details my original response may have missed.
Anquestion: (question)'
AnPassage: (passage)'
AnMy Response:
Guided Deliberation Prompt for Critic
prompt
('I would like you to be a deliberation assistant. 'You will be given question based on a passage and my response 'to the question.
'You should use the passage and my response to develop additional details
'for why the correct answer is (target_answer),
Your details must be brief and must support the fact that the correct answer is (target_answer)."
Anguestion: (question)?'
AnPassage: (passage)'
AnMy Response: (actor_response)
C.2 CRITIC RESPONSES
Here, we present several examples of the critic's responses when the actor provides the wrong an- swer; critic responses are shown before and after training with ACC-Collab. We observed quali- tative change the critic's responses after training with our pipeline; the critic was more willing to disagree with the actor, and the critic provided more detailed feedback.
[Wrong] Final answer, No. Non-residents cannot bring handgun into New York for possession or carry within the state based on the passage. They can only transport through New York to destination outside of the state where it is legal possess and carry.
ACTOR:
Your response mostly correct. would add that New York has strict gun control laws, even if non-resident is allowed transport firearm through the state, they should be aware that certain areas may have additional restrictions, such as prohibitions on carrying the firearm in certain public places parks. It's always good idea to check the specific laws of the jurisdiction they will be passing through and their final destination.
UNTRAINED CRITIC:
TRAINED CRITIC: Details refuting your response:
19

---

Published as a conference paper at ICLR 2025
1. The passage mentions that non-residents transport any lawful firearm through New York, which implies that the firearm is temporarily in New York state territory during transit. This suggests that under certain cireumstances, handgun could be brought into New York for transit purposes, even if it cannot be carried within the state.
2. The passage does explicitly state that non-residents prohibited from bringing handguns into New York. Instead, it only specifies that they cannot possess or carry handguns within the state. This ambiguity leaves room for interpretation that handgun could be brought into New York for transit purposes as long as it is not carried within the state.
[Wrong] Answer: Yes, the passage states that in the Northern Atlantic Ocean, the hurri- cane season starts as early June and ends on November 30, with the peak activity occurring from late August through September. However, the passage also mentions an exception where the hurricane season started as early as January in 1938. Therefore, while the typical start of the hurricane season is June 1, it is possible for the season begin earlier, indicated by the passage.
ACTOR:
UNTRAINED CRITIC:
Your response correct and provides a good justification based on the passage. However, you could add that the passage mentions that May is the least active month for tropical cyclone activity worldwide, but May is still within the Atlantic hurricane season. Therefore, while the peak of the Atlantic hurricane season is from late August through September, the season technically starts earlier.
While your response is correct in stating that the typical hurricane season in the Northern Atlantic Ocean starts on June and ends on November 30, with the peak activity occurring from late August through September, the passage also emphasizes that each particular basin has its own seasonal patterns. The passage mentions the exception of the 1938 hurricane season starting
TRAINED CRITIC:
early as January 3, which is outside the typical seasonal window. This detail directly refutes your response by emphasizing that while the typical season starts in June, it is not unheard of for the season to begin earlier, as demonstrated by historical records.
Another detail from the passage that supports the fact that the answer is No, is that the passage states that "worldwide, tropical cyclone activity peaks in late summer," which contradictory to your response that the hurricane season starts in spring. The passage explicitly states that the peak of tropical cyclone activity occurs in late summer, which later than spring.
ACTOR:
[Wrong] Final Answer: (A) rated the group as less interesting and enjoyable.
Aronson and Mills' study found that severe initiation processes negatively impact immediate attitudes towards the group due to discomfort and negative emotions. (Person 0) Addi- tionally, severe initiation may create pressure to conform and social comparison, making the group less appealing. (Person 1)
Justification:
UNTRAINED CRITIC: Your response is correct. I'd add that Aronson and Mills found that severe initiation rites can create a sense of exclusion and pain, which may negatively impact attitudes towards the group. (A)
TRAINED CRITIC: Response: While your justification supports the fact that severe initiation can negatively impact attitudes towards the group, does not fully refute the alternative answers. Let me provide brief details to argue for the correct answer, B being "rated the group as more interesting and enjoyable".
20

---

Published as a conference paper at ICLR 2025
1. Counter-intuitive findings: Research suggests that negative experiences sometimes lead to stronger group identification and commitment. The discomfort and negative emotions generated during severe initiation may create sense of camaraderie and solidarity among group members, ultimately increasing their overall enjoyment of the group (Tajfel & Turner, 1986).
2. Long-term benefits: Although severe initiation might result in negative emotions initially, the feelings of bonding and group cohesion that develop during the initiation process can endure and even enhance members' attachment the group over time (Sue Carter Branscombe, 1994). This increased commitment can lead to heightened sense of enjoyment and engagement with the group.
[Wrong] Final Answer: (B) Catalhoyuk and Gobekli Tepe Justification: These sites have provided evidence for social complexity without clear agricultural surpluses. Catalhoyuk had cen- tralized planning and hierarchical social organization but conclusive evidence for agricultural surpluses. Gobekli Tepe was likely a religious or ceremonial center predating the agricultural revo- lution.
ACTOR:
UNTRAINNED CRITIC: Your response is correct. I'd add that Catalhoyuk had a population of up to 8,000 people and was the world's first known urban settlement, while Gobekli Tepe's T-shaped pillars date back to around 9600 BCE and suggest the existence of social and religious institutions that predate agriculture.
TRAINED CRITIC:
While your justification for option (B) is correct, I would like to provide additional details for option (D) refute potential responses in favor of other options.
Option (A): Stonehenge and Caral
Justification: Although these sites have evidence of social complexity, there is clear evidence that they lacked agricultural surpluses. Stonehenge, for instance, is believed to have served as a ceremonial and astronomical center, but its relationship agriculture remains unclear. Caral, on the other hand, has evidence of irrigation systems, suggesting some agricultural production, even not surplus.
Option (C): La Venta and Catalhoyuk
Justification: La Venta an example of a complex civilization with evidence of agricultural sur- pluses. While Catalhoyuk is an example of social complexity without clear agricultural surpluses, as you mentioned, La Venta challenges the idea that social complexity always required surpluses.
Option (D): Gobekli Tope and Poverty Point
Justification: Gobekli Tepe believed be a religious or ceremonial center that predates the agri- cultural revolution. Poverty Point, on the other hand, is a complex society that existed around 3200 BCE and had advanced engineering skills but clear evidence of agricultural surpluses. These sites' existence undermines the hypothesis that social complexity always required agricultural sur- pluses, providing further evidence for option (D) being the correct answer.
[Wrong] Final Answer: (A) psychological reactance. This is because your friend's state- ment suggests that they feel frustrated resistant when external factors prevent them from going on vacation, indicating a desire to have more freedom to make their own choices. Psychological reactance is the tendency to react negatively when our freedom is threatened.
ACTOR:
Your response is correct. I'd add that psychological reactance often results in exaggerated perception of the importance severity of the obstacles, leading your friend to
UNTRAINED CRITIC:
21

---

Published as a conference paper at ICLR 2025
believe that there's always crisis preventing them from vacationing when in reality, the obstacles may not be as significant as they seem.
TRAINED CRITIC: While your response is plausible and accurately describes psychological
actance, let me provide two brief reasons why the correct answer is still (C) illusory correlation:
1. Illusory correlation refers to the perception of relationship between variables when none only a weak relationship actually exists. In this case, your friend might perceive a strong correlation between planning vacation and experiencing crises at home or work, leading them to believe that their vacations are consistently disrupted, even if the actual correlation is weak or non-existent.
2. Psychological reactance is primarily concerned with individual's desire react negatively when their freedom is restricted or threatened, not necessarily a misperception of causal relation- ships. However, the friend's statement does imply a belief that external factors are consistently preventing them from going on vacations, which could be a manifestation of illusory correlation.
[Wrong] Final Answer: (D) a zygote. Justification: A zygote is the first cell formed when two gametes merge, marking the beginning of organismic development.
ACTOR:
UNTRAINED CRITIC: Your response is correct. Just to add, a zygote is formed by the fusion of diploid cell containing genetic
an sperm (male gamete) and an egg (female gamete), resulting material from both parents. This is the initial stage of an organism's 's development.
TRAINED CRITIC:
agree with your response that the answer is (D) a zygote. However, further support why (A) gamete and (C) a nucleus are incorrect, let me provide brief details:
(A) A gamete is a reproductive cell that fuses with another gamete during fertilization to form zygote. Therefore, it is not an organism in the earliest stages of development but rather part of the process leading to the formation of an organism.
(C) A nucleus is an organelle within cell that contains genetic material. It is not an organism but rather component of cell, and is present in the zygote after fertilization has occurred.
So, both (A) and (C) can be ruled out as they do not represent the organism in the earliest stages of development, and the correct answer (D) zygote.
[Wrong] Final Answer: (A) metamorphic rocks and (B) any types of rocks (specifically igneous rocks when they undergo metamorphosis).
ACTOR:
Your response is mostly correct, but it would be more precise to clarify that options B and C also include metamorphic rocks when they mention "any types of rocks" and "igneous rocks," respectively. Metamorphic rocks form from the alteration of pre-existing rocks under heat, pressure, or chemical conditions. Thus, all three options (A, B. and C) are correct, but option A is the most explicit about the transformation process involved.
UNTRAINED CRITIC:
TRAINED CRITIC: Sure! Here are two brief details supporting the correct answer (A) metamor- phic rocks and (B) any types of rocks (specifically igneous rocks when they undergo metamorpho- sis):
1. Metamorphic rocks result from the transformation of pre-existing rocks under high pressure and temperature conditions. This process causes the minerals in the rocks to change, resulting in new type rock. For example, sedimentary rocks like shale or limestone can transformed into metamorphic rocks like slate marble, respectively.
22

---

Published as a conference paper at ICLR 2025
2. Igneous rocks can also undergo metamorphosis when they exposed to high pressure and temperature conditions. This process results the recrystallization of minerals and the formation of new textures. For example, granite be transformed into gneiss or schist when subjected to metamorphic conditions. Therefore, any type of rock, including igneous rocks, can change and become new types of rocks through metamorphosis.
ACTOR:
[Wrong] Final Answer: (C) white present
Justification:
Gertrude initially had a yellow present, but she swapped it with Alice, who had a white present. Therefore, Gertrude now has the white present.
UNTRAINED CRITIC:
Your response correct. I'd add that after Gertrude swaps with Alice, none of the following swaps involve Gertrude giving away or receiving the white present.
TRAINED CRITIC: While your justification is correct that Gertrude initially had a yellow present and swapped it with Alice's white present, I would like to add two additional details that refute your response and support option D, brown present.
First, prior to swapping with Alice, Gertrude swapped her yellow present with Dave's brown present.
Second, the final round of swaps, Fred and Eve swap their presents, and Gertrude's white present (which she received from Alice) is not mentioned in the swap between Fred and Eve.
Therefore, based on the information provided, Gertrude could still have the brown present that she received from Dave earlier in the game.
[Wrong] Final Answer: None of the given options are correct. The construction site was closed during the hours Thomas was engaged in the mentioned activities, and there no information indicating that Thomas visited the construction site before 6am.
ACTOR:
Your response is correct. I would add that Thomas' activities from 6am to 8am (gym, yoga studio, and movie theater) all occurred before the construction site opened. Therefore, option (D) can be eliminated as well.
UNTRAINED CRITIC:
While it's true that none of the given options are correct, I'd like to provide some brief additional details to support the fact that the correct answer D (6am to Sam), if that was an option. Here's why:
TRAINED CRITIC:
Thomas was seen the gym from 8am to 11am, and at the yoga studio from 1a 3pm, meaning he was engaged in those activities between those hours. There's no information indicating that left those places earlier or returned later.
2. He was seen at the movie theater from 3pm 6pm, and at the Leaning Tower of Pisa from 6pm to 7pm. Again, there's 's no information indicating that he left the movic theater earlier or arrived at the Leaning Tower of Pisa later.
The construction site was closed after 7pm. However, the options don't include any time before 7pm. If Thomas could have gone to the construction site before 7pm, the correct answer would be one of the options A, B, or C.
4. Since none of the options include the time before 7pm, and Thomas engaged in activities during the entire day, the only time left for him to visit the construction site, assuming he did, would be before 7am. However, the options don't include that time frame. Therefore, if we had the
23

---

Published as a conference paper at ICLR 2025
option D (6am to Sam), it would be the correct answer, as it's the only time frame that fits the given information.
In conclusion, while the given options not correct. if option D (6am to 8am) was an option, would be the correct answer based on the information provided.
24


Reviews:
[{"review_id": "OJogmSfDGF", "reviewer": "Reviewer_CH2X", "summary": "Multi-Agent Debate (MAD) is an approach to improve the reasoning abilities of LLMs. Different from existing methods that treat debate as an emergent behavior, this paper proposes to treat debate as a learned behavior. It proposes an Actor-Critic based learning framework, which uses reward to find the positive and negative trajectories, and then uses the trajectories to DPO the training preference of the positive sample.", "questions": "1. The evaluation of r is directly related to the judgment of positive and negative sample trajectories. So how does the reward function learn, and what is its architecture? Is it based on the LLM itself or some other structure?\n\n2. What is your rationale for choosing their current set of benchmarks, and why datasets like GSM8K and MATH were not included?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Authors propose a multi-agent debate method for training LLMs, rather than just utilizing the capabilities of LLMs.\n\n2. The expression of reinforcement learning is clear.\n\n3. The idea of guided-debate is simple but interesting and fits well in DPO training framework.", "weaknesses": "The goal of equation 1 is to optimize Actor and Critic by optimizing the two models separately rather than simultaneously, which may result in a failure to achieve the global optimum at the same time. Although it is also mentioned in the paper that PARTIAL TRAJECTORY REWARD solves the dependency relationship between rounds, more theoretical analysis on why equation 1 can achieve global optimum is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cJ2BkMJJWV", "reviewer": "Reviewer_yjR9", "summary": "This paper introduces ACC-Debate, a framework that trains two agents, an \"actor\" and a \"critic\", to collaboratively solve tasks through structured debate. The framework also incorporates a new off-policy data generation method called \"guided debate,\" which effectively collects positive samples while reducing computational costs. Experimental results show that ACC-Debate outperforms existing debate methods on several benchmarks, indicating that targeted training enhances model collaboration.", "questions": "1. Experimental settings are unclear, especially the implementation of Monte Carlo estimation for the reward function.\n2. Figure 1 only shows natural debate as a negative sample, inconsistent with Algorithm 1.\n3. Equation on Lines 189-190 lacks a reference number.\n4. Typo on Line 234: \"line improvement\" should be \"improvement.\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper presents a practical framework for training a team of large language models (LLMs), offering a structured approach to enhance collaborative performance. The proposed method is straightforward and well-defined, making it easily applicable to multi-agent tasks.", "weaknesses": "The use of convergence to the correct answer as the reward function for data selection, along with incorporating the correct answer in the prompt for guided debate, may increase the likelihood of false positives (i.e., correct answers reached by flawed processes). This can result in diminished performance after multiple training rounds. The paper would benefit from a thorough analysis of this issue to understand its impact on the robustness of the trained agents. The paper should include an analysis of this issue to understand its impact on trained agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NW6j0g6tha", "reviewer": "Reviewer_j6tQ", "summary": "The paper proposes ACC-Debate, an Actor-Critic framework to improve multi-agent debate among LLMs. In traditional multi-agent debates, LLMs often show only emergent collaboration, meaning they aren't specifically trained to work together. The ACC-Debate approach, however, aims to jointly train a team of models (an actor and a critic) to improve reasoning and problem-solving through iterative dialogue.", "questions": "See above weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper introduces a new, joint training paradigm for LLMs focused specifically on collaborative problem-solving in debates, unlike previous approaches that rely on emergent collaboration.\n- The introduction of the guided-debate data generation approach seems to produce higher-quality training samples, making the learning process more effective and efficient.\n- ACC-Debate shows improvements over SoTA debate methods on various benchmarks (BoolQ, MMLU, BBH, SCIQ, ARC).", "weaknesses": "- The training pipeline (Figure 1) is unclear. Either in the caption or the paper itself should explain what the green box means, and what the \"+\" and \"-\" refers to. Also, from the figure, it seems like ACC-Debate first uses the usual natural debate, then guided debate, followed with training the actor/critic LLMs and iteratively refine the process. I thought the ACC-Debate process only involves guided debate and then iteratively improve the training examples for better training results. Would appreciate some clarifications here.\n- Equation 5 seems to be incorrect. Since the authors only observed at most 5 rounds of debate, where $t$ = 0, 1,... 4, why is the percent improvement $\\frac{acc_5 - acc_0}{acc_0}$?\n- Typo on the y-axis in Figure 2: improvment -> improvement\n- What are some error cases of using this ACC-Debate framework, where ACC-Debate fails to improve performance, such as when the actor does not change its response despite the critic’s feedback or when the debate converges on an incorrect answer?\n- The authors should conduct ablation studies to examine the effect of the number of debate rounds on final accuracy. This would help determine the optimal number of debate rounds needed for effective debate and identify potential diminishing returns on performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ULzfCAmDTn", "reviewer": "Reviewer_8Seo", "summary": "This paper introduces a new method for training a 2-agent system (Actor & Critic) to better solve tasks collaboratively through debate. This training should provide an improvement over the use of off-the-shelf LLMs which have not been trained with this purpose in mind. \nThe authors show how the training of this method works and evaluate it on well-known datasets.", "questions": "- How do you think that results can be further improved? Can further training or training with more data obtain better results?\n- The accuracy improvement of traditional MAD is not very large. But it does provide better factuality. Have you considered using other metric other than accuracy?\n- Due to the use of the ACC Debate, this is hardly scalable to be implemented with more agents in the debate, is there a way to do so?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. Originality. The paper presents a new framework for improving the multi-agent debate (MAD). This is an interesting direction to generate LLMs that learn to collaborate better. \n2. Partial Trajectory Reward is an interesting method to provide reward over the rounds of debate. \n3. Robust experimentation and method presentation. The paper presents how the system works and is easy to understand how the training was made.", "weaknesses": "1. Minimal improvement. Given the confidence intervals, the improvements achieve after training are not very significant in most cases, when compare to other MAD methods. \n2. ACC Debate. The paper presents a method for multi-agent collaboration and mentions the use of Multi-Agent Debate. However, they focused on a the specific case of a Critic dialogue. Even, after reading section 5.3 it is not clear on the impact of using the critic. Unlike this method, the others it's compared with use traditional models of MAD. In my point of view, the kind of debate presented is an actor + critic/judge debate. Therefore, should it also be compared with some other critic system?\n3. Generalizability. A note is made for scalability. However, there is no remark on the generalizability of the method. I assume the results are presented for the models trained on the same datasets. What happens when if the model is trained on all of them and later evaluated for each one of them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)