Collaborative Reasoner: Self-Improving Social Agents with Synthetic Conversations
Authors: Ansong Ni
Ruta Desai
Yang Li
Xinjie Lei
Dong Wang
Jiemin Zhang
Jane Yu
Ramya Raghavendra
Gargi Ghosh
Shang-Wen Li
Asli Celikyilmaz
Venue: NeurIPS 2025 poster ()
Keywords: collaborative reasoning
social agents
synthetic data
self-improvement
Primary area: deep_learning
TL;DR: We present Collaborative Reasoner (Coral), a framework to evaluate and improve the collaborative reasoning skills of language models.

Abstract: With increasingly powerful large language models (LLMs) and LLM-based agents tackling an ever-growing list of tasks, we envision a future where numerous LLM agents work seamlessly with other AI agents and humans to solve complex problems and enhance daily life. To achieve these goals, LLM agents must develop collaborative skills such as effective persuasion, assertion and disagreement, which are often overlooked in the prevalent single-turn training and evaluation of LLMs. In this work, we present Collaborative Reasoner (Coral), a framework to evaluate and improve the collaborative reasoning abilities of language models. In particular, tasks and metrics in Coral necessitate agents to disagree with incorrect solutions, convince their partners of a correct solution, and ultimately agree as a team to commit to a final solution, all through a natural multi-turn conversation. Through comprehensive evaluation on six collaborative reasoning tasks covering domains of coding, math, scientific QA and social reasoning, we show that current models cannot effectively collaborate due to undesirable social behaviors, collapsing even on problems that they can solve singlehandedly. To improve the collaborative reasoning capabilities of LLMs, we propose a self-play method to generate synthetic multi-turn preference data and further train the language models to be better collaborators. Experiments with Llama-3.1, Ministral and Qwen-2.5 models show that our proposed self-improvement approach consistently outperforms finetuned chain-of-thought performance of the same base model, yielding gains up to 16.7% absolute. Human evaluations show that the models exhibit more effective disagreement and produce more natural conversations after training on our synthetic interaction data.

Collaborative Reasoner: Self-Improving Social Agents with Synthetic Conversations
Ansong Ni
Ruta Desai"
Yang
Xinjie Lei
Dong Wang
Jiemin Zhang
Jane Yu
Ramya Raghavendra
Gargi Ghosh
Daniel Li
Asli Celikyilmaz
Meta FAIR
(ansongni, rutadesai, aslic)@meta.com
Abstract
With increasingly powerful large language models (LLMs) and LLM-based agents tackling an ever-growing list of tasks, we envision a future where numerous LLM agents work seamlessly with other AI agents and humans solve complex prob- lems and enhance daily life. To achieve these goals, LLM agents must develop collahorative skills such as effective persuasion, assertion and disagreement, which are often overlooked in the prevalent single-turn training and evaluation of LLMMs. this work, we present Collaborative Reasoner (Coral ) a framework to evaluate and improve the collaborative reasoning abilities of language models. In particular, tasks and metrics Coral necessitate agents to disagree with incorrect solutions, convince their partners of correct solution, and ultimately agree as team to commit to a final solution, all through a natural multi-tum conversation. Through comprehensive evaluation on six collaborative reasoning tasks covering domains coding, math, scientific QA and social reasoning, we show that cur- rent models cannot effectively collaborate due undesirable social behaviors, collapsing even on problems that they can solve singlehandedly. To improve the collaborative reasoning capabilities of LLMs, erate synthetic multi-turn preference data and further train the language models to better collaborators. Experiments with Llama-3. 3. 1, Ministrr and twen-2.5 models show that our proposed self-improvement approach consistently outper- forms finetuned chain-of-thought performance of the gains up to 16.7% absolute. Human evaluations show that the models exhibit more effective disagreement and produce more natural conversations after training on our synthetic interaction data.
propose self-play method to gen-
base model, yielding
Introduction
Modern large language models (LLMs) and LLM-based agents are tackling a wide-range of tasks,
spanning from virtual assistants [20 [33 and online shopping [50] 60] to autonomous software engineering 45 56 and the mastering of strategic games [T0
62 As these individual
LLM-agents become more powerful, we foresee future where AI and human agents with differ- ent specialties and information access work seamlessly with each other, all through the common communication media of natural language conversations. To realize this vision, our goal is to develop conversational agents that can collaborate with other Al agents and humans to complete tasks
"Equal contribution.
'Code for Coral exps: https/////tttttc..ooooooooooco/t Code for Matrix infra: http:p/p//gtttteoeoooooccoeeeree
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

I'm trying te selva this math prosiem: Jasson has a pro plan of 1000 mins per menth extra 300 month
Abr
Everday has withits boss, manyy
month
1o0
shink step step. think we can colculate the mins he has spent calling bose this month, Since it was days and 15 minns per day, should minss Jason have 1000
850 mine
persuasive
Mbr sure
0.0
consider the case where there are 28, 29 or days for the mondi??
That
But hooudhhie
.o0
assertive
But the questio mode the assumption that thes month onyth has days
550 mina
Of youh you are right Buut thrnic we made soother mat here. Linen shop used 300 minutes call other phone plan,
persuasive
250
people
about that That correct, the arower should be 250 inei Indledd
agreement
mins
An example conversation on solving reasoning problem between two agents
Beliefs of the agents
Figure 1: An illustration of two agents solving reasoning problem collaboratively. Besides answer correctness, we also track social behaviors such as agreement throughout the conversation.
While the communication format is simple and intuitive, these collaborations can be challenging in addition goblem-sslvingg abilities. they require social skills such as effective communication, providing feedback, having empathy and theory-of-mind, inter alia.
effectively, in free-form multi-turn conversations [1]
Current LLMs, however, are predominantly trained and evaluated for single-turn question-
or problem-solving tasks [ 24 [9], instead of collaborative and interactive problem-solving sce- narios. Consequently, there is a gap in understanding how well current frontier language models can collaborate with other agents and humans in a natural conversation. Moreover, while explo- rations exist for multi-agent teams of LLMs with different roles, capabilities, and communication architectures 66] 14 61]. little progress has been made in developing generalist agents that possess all the necessary reasoning and social skills for effective in-the-wild collaboration through natural On the other hand, developing such agents is challenging owing to lack of conversational collaboration data. Such data remains expensive to collect, and can be domain-specife and limited, making both training and evaluation difficult. Motivated by the lack of evaluation, training data, and training methods that can enable LLMs to collaborate over multiple turns of conversations in goal-oriented tasks, we present Collaborative Reasoner this work, which we also refer as Coral
conversations with humans 20 57
Coral is a comprehensive framework focused on evaluating and enhancing the collaborative reasoning skills of language models. More specifically, given a reasoning problem (e.g.. math, physics, theory-of-mind)) Coral emulates human-Al collaboration and requires two agents to work together on the problem through a multi-turn conversation. Along with solving the problem correctly, it also requires agents to agree with each other before committing to final solution of given problem. Consequently, learning to disagree to incorrect solutions, i.e., assertiveness, asking clarifying questions, and convincing the partner of correct solution, i.e., persuasiveness, are required to succeed. We evaluate several frontier open and closed sourced LLMs on 6 reasoning tasks under this collaborative setting, spanning domains across coding, math, scientific question answering and social story comprehension. Compared with single-agent approaches such as chain-of-thour we find even these frontier models are inconsistent at leveraging collaboration to better approach these tasks. Further analysis social behaviors via designed social metrics reveals tendency for agents to be overly agreeable (> 90% agreement score), regardless of reasoning correctness, limiting their ability challenge incorrect solutions and reducing collaboration efficacy.
prompting,
To rectify these undesirable social behaviors of current LLMs, we propose to leverage synthetic conversations collected from simulated self-collaborations with an LLM itself. We perform tree sampling to diversify the model responses and obtain pairs of responses for preference-basee learning. Different from single-model and single-turn workflows, however, synthesizing conversational data
Cooal is short for collaborative reasoning models

---

at scale poses its own engineering challenges (e.g., network congestion). To that end, we build Matrix, a robust, versatile and perfforrnee model serving framework, which allows thou- sands of conversations being generated in parallel over hundreds of model instances. With large pool of rreessructured, collaborative conversational data generated by Matrix, we employ both conversation-level and turn-level filtering methods to obtain wefereenceffinttningg data for training collaborative reasoners. Experiments on Llama-3. 1, Qwen-2.5 and Kinistral models show that our proposed self-improvement approach consistently improves collaboration performance, and outper- forms single-agent CoT finetuning baseline by up to 16.7% absolute. Moreover, the trained 70B collaborative reasoners rival strong reasoning models such 01 and Gemini-Pro MMLU-PA and ExploreToM., ToM benchmarks. Further analysis also show that the models trained on our synthetic data can generalize to different partner models and can be directly applied to different dataset in similar domain. Human annotations suggest that the collaborative reasoners display more effective disagreement and the generated conversations are more natural after training with method.
We open-source our code for Coral and Matrix to support future research on developing social agents that can partner with humans and other agents. And hope to extend our research to include evaluation and data collection of human-Al AI interactions to further foster research in this area.
2
Collaborative Reasoning over Multi-turn Conversations
Unlike existing framework that structure agent interaction via fixed roles and curated prompts 22
23
40), Coral evaluates general-purpost models in free-form multi-turn conversations. This way it reveals the true limitations in their collaborative skills under the most natural format of collaborations. We use the questions from existing reasoning tasks as the conversation starters and use exact match of the answers to measure solve rate. But unlike single-turn settings, where the performances solely depend on the correctness of the generated solution, collaborative reasoning requires agents to agree on solution as a team. As shown in Fig. we also record social metries during the multi-turn conversations, and we introduce these metrics in more detail in the following sections
2.1 Problem Definition
Given a reasoning problem {I,""), where denotes the task input (e.g., "Jason has phone plan of How many minutes does Jason have left if this month has 30 days?" as in Fig. T and y denotes the gold-standard task output (e.g., "250 mins"), collaborative reasoning entails two language model agents A and B engaging in a conversation to discuss how to solve this problem. To initiate the conversation, the first utterance a1 is generated through template of "I'm trying
solve this problem: [insert problem] [3 And the whole conversation C = {a11, b1, 42 b2.... will be generated interleavingly with agents {a, b} and their corresponding system prompt. The conversation will stop when an agreement is reached between the agents (e.g., Fig. Tp. or the maximum number of 20 turns is reached. To evalute such interactions, we define agreement and social behavior metrics in the following section.
2.2 Coral Metries
When modeling reasoning problems in single-tur, it is common to first generate sequence that represents the thinking process (e.g., chain-of-thouu of-hhughhhh followed by the final answer. However, in multi-turn conversational setting, each turn may not conclude with clear final answer, as the agents may be planning the steps, debating on a fact, or as in Fig. I asking a clarification question. Moreover, agreement can be partial (e.g., "I agree that X. but that doesn't 't mean Y.") or of higher order (e.g., "I agree that my previous disagreement is unwarranted."), which makes measuring of agreement between agents in multi-turn setting quite challenging. These metrics below are automatically derived using belief extraction without human annotation, enabling scalable analysis of social behaviors.
Measuring agent beliefs and agreement. To address these challenges, we refrain from using superficial verbal cues on agreement, but instead try to track the beliefs of the agents on the final
Without loss of generality, note A to be the agent that starts the conversation.
3

---

MATH
MMLU-Pro
GPQA
ExploreToMl
Models
CoT
Coral
CoT
Coral
CoT
Coral
CoT
Coral
LJama-3.
-ama--..-8B-Instmeett
51.4
47.2
44.4
45.6
27.1
31.0
60.8
42.4
64.0
63.8
63.8
65.8
39.5
35.7
71.3
72.9
Llama-3.1-42
1.0ma33.-40SB-IIstuuc
71.9
71.7
67.9
69.7
47.1
48.4
80.4
79.5
GPT-4o
72.6
78.7
67.3
69.5
43.3
42.9
74.6
76.5
01
94.1
89.2
80.6
82.8
70.8
74.1
86.3
86.8
Gemini-
84.3
82.0
72.7
69.7
54.2
48.2
70.6
67.1
Gemini-2.5-Flash
84.3
91.7
67.6
81.0
46.4
69.4
85.8
87.3
Claude-3.7-Sonnet
74.4
79.1
75.7
79.1
59.8
65.6
86.3
84.3
Table I: CoT Correctness vs. Coral Agreement Correctness for Llama-3. 3. 1 and close-source
frontier models. For each model and task, the worse performance between the two is marked red.
answer and measure agreement by comparing such beliefs. Inspired by recent works on LLM-as- judge [58], after each turn, feed the response of the agents this agent at this time, or say "not sure" not clear answer is presented in the response. Then for each conversation C, agreement measures whether the latest beliefs of the agents, i.e., { 189, which are updated after every turn, matches each other. And agreement correctness, which is the main metrics we aim evaluate and improve in this work, measures whether the answer the agents agreed on is correct. More specifically,
extractor extract the belief of
agreement: aaC)) = I[3A = + σ)
agreement correctness : a²(C) = I(a(C) = & 3 = y²) (main metric)
where [[(x)) is the indicator function, y" is the gold answer and B denotes "not sure" per 2.1
Turn-level social behavior metries. In addition to conversation-level agreement, inspired from social science 212 19 52], we also design turn-level metrics which focus on measuring two critical social collaborative behaviors persuasion and assertion. An agent's response a turn is considered persuasive if it changes its partner's response to match its own. Persuasiveness thus measures the influence or impact of an agent on its partner. Likewise, an agent's 's response in a turn considered assertive if it remains unchanged from the agent's response in its previous turn, irrespective of its partner's response. Assertiveness captures whether an agent's partner influences the agent and whether the agent is able to maintain its belief under its partner's 's influence. In addition to evaluating whether turns are persuasive or assertive, we also measure the quality of persuasion i.e., whether an agent's persuasion changed its partner's belief towards more accurate solution of the given problem. Detailed definitions of these metrics and can be found in the Appendix B More broadly, such metrics may also be useful in understanding and improving human-Al interactions such persuasion for social good [46] and reducing polarization 4
3 Are Current LLMs Good Collaborative Reasoners?
To understand how frontier open- and soourd- LLMs fare at collaborative reasoning com- pared to single-turn CoT reasoning, we evaluate Llama-3. ---nnntuuct series [24], GPT-4o [29], 01 [30], Gemini-1... Gemini-2.5 [8] and Claude 3. 7 (2]. We choose these models owing to their strong results in single-turn reasoning performance in various tasks, and the post-trained versions (e.g.-Instruct) also demonstrate good conversational skills needed for collaborative reasoning. We instantiate these models collaborate with themselves for 4 reasoning tasks covering different domains: MATH [6 (math reasoning). MMLU-Pr [47] (general), GPQA [3 (scientific QA), and ExploreTo. (66 (social reasoning). The details of these benchmarks can also be found in $ 5 Here we draw important insights from this analysis.
Models are inconsistent leveraging collaboration improve performance, unlike humans. While human collaboration often enables better solutions to difficult problems [35]
48 we
We use different system prompt (see Tab. 11 for belief' extraction for robust performance.

---

Reasoning Problems
Synthesleed Conversations
Tree Sampling Conversations
Preference Training Data
(Prompt c> (Chosen, Rejected))
System
System Prompt Sg
Sample w/
Self Collab.
at
AC
a
Agent
Create Training Examples
A
A2
AG
Ag
AZ
A
Train DPO
A/
AG
Tum-oastd Preference
Illustration of our self-training pipeline, with steps of sampling + filtering + training. During tree sampling, we track the beliefs of all turns using methods described in S 2.2 and we note the turns in (green boxes holds the correct beliefs while the ones with (red boxes are incorrect.
Figure 2:
make different observations for LLMs from Tab. Unlike humans, LLMs typically struggle to consistently outperform the equivalent CoT performance, despite using more inference compute. In particular, LLama3. 1 1-8B-Instruc exhibits performance drop of 18.4% on Explore and 4.2% drop on MATH. Likewise, the 01 model performance drops 4.9% on MATH, while Gemini-1. Pro consistently under-performy in Coral compared to COT. Although larger and more powerful models including LLama3 10mma.1-4000 1--008------ Instruct and Gemini-2.55 2.5 5--5--Fa Flash models are relatively better at leveraging collaboration, overall models are unable to consistently achieve better performance in coral settings.
Models often exhibit undesirable social behaviors. Upon inspection of the social metrics, we discover consistently high agreement rate (i.e., ranging from 74.3% to 99.3%) despite much lower agreement correctness rate (see Tab. 7 the appendix for detailed numbers). This suggests that the models lack the ability of effective disugreement, resulting in large percentage of the conversations ends with agreeing on an incorrect answer. The aforementioned lack of assertiveness also highlighted in Tab. 7, which shows that models irrespective of their sizes struggle be assertive average percentage of assertive turns range from 0.2 5.5% in the collaborative conversations. Models thus tend to get carried away under their partner's 's influence, even when they are correct. We hypothesize that this undesirable behavior is a result of RL.F post-training, which makes the model very polite and agreeable, thus less assertive when pointing out the mistakes or standing their own grounds B1 We aim to rectify these behaviors via additional training on synthetic conversation data, which we will introduce in the next section.
4 Self-Improving Collaborative Reasoning through Synthetic Conversations
To improve the collaborative reasoning abilities of LLMs, generate synthetic multi-turn conversa- tions via self-play, enabling scalable training without human annotation.
4.1 Self-Training Method
For training the language model, we formulate it as next-turn prediction problem. More specifically, we construct training examples with input (i.e., prompts) of (su: 01, ba,,b,... .. and output of next turn u, for agent u, where 8, is its system prompt. As illustrated in Fig. 2] our eepooimproeeenttt pipeline consists of three steps: 1) tree sampling: 2) belief filtering;; and 3) preference finetuning.
Tree sampling. To generate diverse and informative responses for training, especially for preference tuning, we need more than single deterministic dialog path per problem. Thus we adopt tree sampling approach (as shown in Fig. 2): For each round of conversation i, we sample set of responses (ut, u² ..., w.. ~ Pa(us Next, we randomly select response uf to append the conversation prefix C,-1, as this recovers the independent next turn sampling process while retaining
5

---

its sibling nodes fuf li # j"} for the construction of preference data This strategy improves coverage over possible collaborative behaviors. To boost diversity at the conversation-levee we also independently sample 5 such conversation trees.
Belief filtering, To convert the sampled conversation turns into preference training data, we need a reliable way label each turn as high or low quality. We do this by extracting the belief expressed by the agent at each turn (i.e. what the agent currently thinks the final answer is, and comparing it to the gold answer for the problem). Specifically, for each candidate turn u/, extract its belief, S.. If the belief matches the gold answer y" m we label ut posilive turn (u, ): otherwise, it is labeled negative (u, ). These labels are then used to create preference pairs between correct and incorrect turns that form training examples for preference tuning. Since some problems are much easier than others, we cap the number of preference pairs per problem and per turn to avoid biasing the model toward easy examples. This ensures diverse and proper supervision across the dataset.
Preference finetuning. As shown in Fig. after such tree sampling and correctness-base based belief filtering, each training example consists of a prompt {Su. C--13 } with the system prompt 5. for agent u, the conversation prefix C.-1, and preference pair {(u, ue" "luf E u. uf E u, }. For preference finetuning, we use the DPO algorithm [33] which relieves the dependency a separate reward model and allow directly learning of this preference from our synthetic conversations. Although our self-training algorithm allows multiple rounds of training, we did not find additional benefit from iterative DPO training.
4.2 Scaling Up Synthetic Conversation Generation with Matrix
Generating high-quality collaborative data at scale is computationally intensive. To address this, built Matrix, a scalable and versatile model serving framework designed for multi-agent synthetic data generation. Matrix can be scaled up to serve hundreds of models and complete thousands of requests per second (i.e., QPS) consistently [6] We hope Matrix serves as a drop-in tool for teams studying multi-agent LLMs, where data generation remains bottleneck.
Why does Matrix matter to the community? 1) Versatile backend: Matrix uses variety of backends, including vLLM [21], SGLang [59]. and various API-based services (e.g.. OpenAl GPT, Google Gemini, etc), thus it supports wide range of models; 2) Built-in scalability: Thanks to the integration with Ray (27]. Matrix can perform auto-scaling and load-ball had-balancing based on the current workload of each LM service. To avoid network congestion, Matrix uses gRPC for higher throughput, while maintaining support for http. 3) Robus! to interruptions: Matrix also integrates with Slurm, the most popular resource management tool for research environments. This allows us spawn Ray workers with various priority and yet robust to job preemption, resulting in higher cluster utilization while providing an uninterrupted services.
When compared with the best open-source alternatives, we found Matrix to be up to 1.87x faster. More detailed design of Matrix and comparison with existing frameworks are shown in Appendix C due to space limit. We release Matrix to support future research on multi-agent collaboration, with plans to extend it for human-Al interactive evaluation and data collection.
Experiments
5.1 Experiment Setup
Tasks. We consider six different reasoning tasks, spanning over math [16]. coding 33, scientific QA (4). theory-of-mind [15] 36), and general [7] reasoning domains. Approaching these tasks in a multi-tur multi--urr conversational setting tests both pure reasoning ability and the models' ability to collaborate via free-form conversations. In particular, to measure the code reasoning abilities, we
'Note the do not continue expanding on the sibling nodes (i.e., MCTS) due exponential growth of required compute. But we leave them as important future work, see more discussions in Appendix A "This is the largest scale that we have tested for our use case, so it is possible that Matrix is able to handle even higher volume.

---

Liams-3.1----Insttrutt
Liama-3117706-nnstrae
100
5022
50
90
02.4
7B4
80
15
80
752
70
625
61-7
6:49
56.5
6622
97.8
00
60
400
400
472
40.3
.2.0
45.9
42,
MBPP-CR
MATH
MMLU-Pro
ExploreToM
MBPP-CR
MATH
MMLU-Pro
ExploreToM
CoT CooT SFT
CoT DPO
Coral Coral SFT a Coral + DPO
Figure 3: Comparison between collaborative reasoners (Coral and various baselines based on Llama-3.1 8B and 70B models. The y-axis denotes final answer correciness for CoT methods and agreement correciness for Coral methods. Details about these baselines can be found in 5.1
created the MBPP-CR dataset, by first sampling solutions for the original MBPP dataset, then use these solutions and their gold correctness labels to transforms it into a code correctness reasoning tasks with binary outputs. For the detailed setups for MBPP-CR, as well other datasets used in this work, we refer the readers to $ D.I due to space limit.
Baselines. While our main goal to improve the collaborative reasoning skills of LLMs, using synthetic data, also consider the following baselines to further show the effectiveness of our training data and methods:
Strong reasoning models. We use a set of strong reasoning models, including OpenAl D1 and Gemini-2.8 S.ri-2.5FFFaa. 5-Flash [8] as well as much larger Llama model, Llama-3. IIstructt 24 to set the context for the collaborative reasoning capabilities for frontier models;
CoT (+ /PT)), To measure against single-agent performance, we only compare with (CoT) baselines, but also the same-sized models that are further trained on the single-agent CoT reasoning traces to the problems using rejection sampling (i.e., CoT SFT) and preference tuning with single-level tree sampling (i.e., CoT + DPO);
Coral SFT. In addition to preference turning, we also explore a simple SFT baseline improve collaborative reasoning abilities. Specifically, individual conversations are independently sampled, and the turns with correct beliefs will be used target while the partial conversation history as prompt for the fully-supervised training.
5.2 Main Results
We compare self- aff-trained collaborative reasoners against various baselines with the same base model in Fig. and Tab. 2 And on the two datasets where Ilama-based models yield the best performance
(i.e., MMLU-Pro and ExploreToM), we further compare them with strong reasoning models in Fig. 4
Training on synthetic conversations leads to large improvements in collaboration performance.
From Fig. 3. we can observe consistent performance improvements in coral performance after preference finetuning on the synthetic conversations. Using Llama-3, 3. as the base model, on the MBPP-CR, MMLU-Pro and ExploreToM datasets, it improves 11.2% 29.5% absolute for 8B and 11.0% ~ 19.0% for 70B size. While we do not observe improvements on MATH for Llama-3. models, experiments on Qwen-2.5 and Ministral models, as shown in Tab. 2 yields 2.1% 7.2% improvements on MATH and 3.1% ~ 30.5% gains across other datasets. By comparing the self- trained collaborative reasoners with the frontier reasoning models (i.e., as in Fig. 4), we can see that after the Llama-3.1 models are trained with the self-synthesized collaborative conversations, the gaps of coral performance decreased significantly. These results show that training on self-collaborative conversations greatly improves collaborative reasoners skills reach those of frontier LLMs.
7

---

Base Model
Methods
MBPP-CR
MATH
MMLU-Pro
ExploreToM
CoT
75.6
70.9
47.5
59.8
+DPO
80.4+4.8
69.6-1.3
49.5+2.0
78.9+19.1
Qwen-255-7B-Instruct
Coral
79.0
72.0
53.6
57.3
+DPO
82.1+3.1
74.1+2_1
58.444.8
87.8
CoT
74.1
45.8
37.5
55.4
+DPO
78.3
48.9
38.0
74.5
Coral
74.6
42.7
34.4
55.1
+DPO
83.5
49.947.2
54.8
83.1
Table 2: Coral finetuning results on more open-source models. The best performance across
different method is bolded, and subscripts indicate the performance delta compared to the row above.
L..mmm--1-700--iattree Draw
Collaborative Ressoner 708
Gomini-1.5-Pro
Llama-3.1 3.1-0m-..-40BBMA.
Clasde-3.7-Sonnet
Llama-331 3.1.----.----n
Persussivenass
45
66
Coliborative Reasoner 8B
L.lama-311770--nsstuuc
Collborative Reasoner 708
90.2
26.
Agresmentt Correctress
715
2911
70.5
Effective Disagroement
67.1
Poltteness
Verbosity
18
23
60
Figure 5: Human evaluation of generated conversation quality on MMLU-Pro. More annotation details can be found in D.3
MMLU-Pro
ExploreToM
Figure 4: Comparison with strong reasoning models with collaborative reasoning settings.
Collaborative reasoners outperform single-agent CoT finetuning methods. When compared with single-agent reasoning baselines in Fig. 3 and Tab. 2 we can see that Coral + DPO consistently outperforms CoT methods with the same base model, even after the model is trained with SFT or DPO. Concretely, Coral + DPO outperforms CoT DPO methods by up to 14.7% and 16.7% for Llama-3.1 1 8B and 70B models, respectively. And such advantages of coral preference finetuning are consistent for Qwen-2.5 5 and Ministral models, yielding performance gains of 1.0% ~ 16.8% across all datasets. This shows the potential of collaborative reasoners being used in multi-agent system to achieve better reasoning accuracy than single-agent CoT methods.
5.3 Analysis
To better understand the ad advantages and limitations of our methods, here show analysis on model generalization and conversation quality. Additional analysis can be found in Appendix
Collaborative reasoners generalize to different collaborators and datasets at test-time. While previous evaluations mostly focus on the self-collaboration setting, in [Tb. 3 we show the results for "cross-collaborator generalization", where we pair the collaborative reasoners, which are trained via self-play, with a committee of models consisting of 4 LLMs of various sizes and capabilities. Compared with the vanilla Llama-3.1 models, we can see that coral-trained versions only exhibit better performance when evaluated in self-collaoorative setting, but also show stronger performance when paired with other models. On average, improves 4.9% 22.2% across different models and datasets. Moreover, in Tab. 5) 5e show how the trained collaborative reasoners can collaborate with the version before training. And similarly, we observe consistent improvements of 6.7% ~ 41.7% over two vanilla (i.e,, not trained with our method) agents.
Collaborative reasoners also generalize different datasets in the same domain. the other hand, when switching to different dataset in similar domain, our collaborative reasoners can also generalize test time, as shown in Tab. Concretely, different-sized Llama-3.1 models trained on MMLU-Pr yield gains of 5.2% and 9.2% when directly applied to the much harder GPQA dataset.

---

Models
GPQA
Hi-ToM
Agents
Datasets
Base Models
MMLU-Pro
ExploreToM
Gemini 1,5-Pro
48.2
64.5
GPT-4o
42.9
55.8
45.6
42.4
Claude-3.-Sommet
65.6
1lama-31--B-Insttme
57.0,11.4
765434.11
1aama-31-400B-mmam
46.2
71.7
59.4413.8
84.1+1177
1lama-3.1 3.1..3...88 1B Instruct
31.0
40.1
57.3
+ Coral DPO MMLU-Pro
3..2
Qwen-2.5.777 2.5.2.7----nn-mee--
60.4
87.0
Coral DPO ExploreToM
50.0.
60.346.7
90.4
Llama-3.1-7001 1..03.1177-Intmmaa
35.7
66.8
Coral DPO MMLU-Pro
44.999.2
34,4
55.1
+ Coral DPO on ExploreToM
69.3+2.5
Ministral-SB-lnstnaes
48.5
67.5
47.5+13.1
75.0419.9
Table 4: Out-of-distribution generaliza- tion results of MMLU-Pro--GPQA and TxploreTToM-HHiiToM...
Table 5: Asymmetric collaboration results between models before (C) and after coral training. Agent A starts the conversation with the question.
Similar observation be made for the transfer of ExploreToM to Hi-ToM, with up 9.9% absolute gain. The results in Tab. 4jindccate that collaborative reasoners can effectively generalize its reasoning and collaboration skills other in-domain tasks, and we leave the exploration of out-of-domain generalization to future work.
Agent
Dataset
Agent A
Llama-3, 1.-mma----8B-Instmuee
Llama-3. 1.mma-3.11-0B-
3.1--0B----ssrruce Instruttt
+ Coral DPO
Coral DPO
Owen2.5-7B--nnstiee
52.9
65. +12.2
58.1
72.2.+141
GPT-4o
69.7
71.2+1.5
71.7
77.7+8.0
MMLU-Pro
Gemini-1.55 1.5-Pro
68.7
73.2
73.2
75.141.9
Claude-3.7 1i7ude-3.7-Sonet
79.3
80.8
76.6
82.5
Avg. Perf.
67.7
72.644.9
69.9
76.947.0
Owen2.5-7BB-Intiie
43.9
82.6438.7
64.1
93.1
GPT-4o
47.6
79.6
72.6
89.1
ExploreToM
Gemini-1.5 1.5-Pro
50.6
71.9 4213
69.6
84.5
Claude-3.7 3.7-onnet
44.2
41.8
82.5
89.2
Avg. Perf.
46.6
78.8
$32.2
72.2
89.0..6.3 16.8
Table Crlss-oolla generalization results. We pair Llama-3.1 models with a "committee" of models (e.g.. Qwen, Gemini, etc) as collaborators and compare their collaborative reasoning performance before and after coral training on the MMLU-Pro and ExploreToM datasets.
Collaborative reasoners exhibit more effective disagreement while being more verbose. While our designed social metrics can help us quickly discover behavioral pattern of the models. hu- man evaluation still irreplaceable understanding the true quality of the collaboration through the conversations. We recruit human annotators to compare 100 conversations generated by the Llama-3. 1-70B-Instruc models before and after collaborative training, on the same set of questions from MMLU-Pro From the results in Fig. 5) we can observe clear advantage of collaborative reasoners in effective disagreement, which is one of the key reasons why current LLMs fail found in S 3 In addition, the improved naturalness suggests that the generated conversations are more himan-like, which shows the potential of adapting to human-Al collaboration. These improvements, however, seem to be at the cost of increased verbosity. Given this observation, we leave improving the efficiency of collaboration as exciting future work.
Collaborative reasoners show strong performance even when used in single-agent chain-of- thought setting. Curious about whether the coral training also improves the reasoning capabilities in general, we evaluate the coral trained models under the CoT setting, and show the results in Tab. 6 From this table we can observe that coral training (i.e., "+ Coral DPO") also improves the CoT evaluation results, suggesting the training process not only improve the collaboration skills, but the reasoning skills as well. On the right half of Tab. 6. we also show how the CoT-trained models
perform under the collaborative settings. And similarly with the findings in 3 it does not improve

---

CoT Eval.
Coral Eval. ExploreToM
Model
ExploreToM MMLU-Pro
MMLU-Pro
44.4
60.8
45.6
42.4
+ CoT DPO
44.0
75.5
52.346.7
42.2-
+ Coral DPO
46.7
91.9431.1
59,8
90.2
1.lam-3.1--2 1--ttt--nsttrct
63.8
71.3
65.8
72.9
+ CoT DPO
65.3+1.5
78.3+7.0
71.545.7
90.0+17.1
+ Coral DPO
67.2
93.5
75.2
95.0+22.1
Table 6: Evaluating coral-trained models under CoT settings and vice versa. We correcttess for "CoT Eval." and agreement correctness for "Coral Eval".
the coral performance consistently. And when does improve the coral performance, it not as significant as coral training.
6 Related Work
Self-refinement for reasoning. There has been rapid development on using self-refinement to improve LLM reasoning, which resembles the ceff-collaboration setting in this work, albeit only single agent is involved. Notably, self-refine [25] proposes to use the same LLM to provide feedback iteratively improve itself. Specific self-refinement framework such as ReAct [ST] and Reflexion (38] are proposed to improve various reasoning tasks. Such self-refinement can also be done iteratively, as STaR [54] improves the efficiency iterative rejection sampling with answer rationalization. The main goal of our work is to develop multi-agent systems that can engage in natural conversations to complete reasoning tasks, with the aim to improve human-Al interaction in the long run. Moreover, works such as [17] also suggest that the self-correct methods are quite limited, pointing to multi-agent systems as a potential solution.
Improving multi-agent reasoning with synthetic data. With the advent of increasingly capable LLMs, various frameworks that study LLM-agents collaborating through natural conversations have emerged. For example, frameworks such as Chain-of- off-nents [1I] and Agentverse [7] demonstrate how LLMs can collaborate effectively using distinct roles and dynamic conversational interactions. To deal with the data scarcity problem, researchers have used synthetic conversations created au- tomatically for improving multi-agent interactions. For instance, AutoGen 49] and MIND I generate synthetic conversations among multiple agents improve performance. Similarly, Malt 28 generates focused synthetic dialog using agents with specialized roles and capabilities like verifiers. The focus of our work is to produce generalist agents (e.g., no separation of generators and verifiers) that can engage natural conversations to solve reasoning problems.
Social skills of LLMs. While effective collaboration requires social intelligence. including persuasion, assertiveness, theory of mind etc., these remain challenging to incorporate LLMs 42 [55] Recent work using debate-style collaboration 99
40 has show how the structured nature of debate, combined with careful prompting, could enable assertiveness and effective argumentation leading to improved reasoning in L.LMs. Likewise, (39 highlight how models can use persuasion positively to improve their answers. Inspired by these works, we particularly focus on persuasion and assertiveness for effective AI-AI or Al collaboration, and our social metrics are also unique owning to the multi-turn nature conversations.
18
Conclusion
In this work we present Corali,, framework evaluate and improve collaborative reasoning capabilities of language models. We propose a self-improvement method to train the models on turn-based synthetic conversational data, for which we build Matrix support data generation scale, Our self-improvement approach yields consistent improvements over CoT finetuning baselines, and the trained models can generalize to different collaborators and datasets at test time.
10

---

References
[1] Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. MIND: Math informed synthetic dialogues for pretraining LLMs. In The Thirteenth International Conference on Learning Representations, 2025.
[2] Anthropic.
Claude 3.7:
Sonnet.
claude 3--tle---t---ee,, 2025. Accessed: 2025-5-12.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprini urXiv:2108.0777: 100. 07732, 2021.
[4] Hui Bai, Jan G Voelkel, Shane Muldowney, Johannes Eichstaedt, and Robb Willer. Lim- generated messages can persuade humans on policy issues. Nature Communications, 16(1))6037, 2025.
[5] Can Balioglu. fairseq2, 2023.
[6] Huaben Chen, Wenkang Ji, Lufeng Xu, and Shiyu Zhao. Multi-agent consensus seeking via large language models. arXiv preprint MxXi:22310200144 2023.
[7] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2023.
[8] Google DeepMind. Gemini 1.5: Unlocking multimodal understanding across millions of tokens, 2024.
(9] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023.
[10] Meta Fundamental Al Research Diplomacy Team (FAIR),, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with 2022.
strategic reasoning. Science, 378(6624):1067110000
[11] Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen- Yu Lee, et al. When one Ilm drools, multi-llm collaboration rules. arXiv preprini 250i::25002.40066, 2025.
[12] George Fragiadakis, Christos Diou, George Kousiouris, and Mara Nikolaidou. Evaluating human-ai collaboration: A review and methodological framework, 2025.
[13] Google Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[14] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey progress and challenges. arXiv preprint arXhv:2402.01688. 2024.
[15] Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-tom: benchmark for evaluating higher-order theory of mind reasoning in large language models. arXiv preprint arXiv:2310.1677 2023.
[16) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
[17] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yo, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798. 2023.

---

[18] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktaschel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers. ArXiv, abs/2402.06782, 2024.
[19] Kaan Kilic. Saskia Week, Timotheus Kampik, and Helena Lindgenn. Argument- based hu- man-ai collaboration for supporting behavior change to improve health. Frontiers in Artificial Intelligence, 6, 2023.
[20] Andreas Köpf. Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis. Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant mnnversations-demorraaiin large language model alignment. Advances in Neural Information Processing Systems, 36:47669-47681, 2023.
[21] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[22] Renhao Li, Minghuan Tan, Derek F Wong, and Min Yang. Coevol: Constructing better responses for instruction finetuning through multi-agent cooperation. arXiv preprint arXiv:2406.07055, 2024.
[23] Xuechen Liang. Yangfan He, Meiling Tao, Yinghui Xia, Jianhui Wang. Tianyu Shi, Jun Wang. and JingSong Yang. Cmat: A multi-agent collaboration tuning framework for enhancing small language models. arXiv preprint arXiv:240.001663, 2024.
[24] Al Meta Llama Team. The Ilama herd of models. arXiv preprint arXiv:2407.21783, 2024.
[25] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. In NeurlPS, 2024.
[26] D. Miraucourt, Caruanaand, and P. Mollaret. How do people evaluate themselves in terms of assertiveness and ability after having failed or succeeded: The (economic) consequences matter! In International Review of Social Psychology, volume 35/1, page 17, 2022.
[27] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging {AI} applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18), pages 561-577, 2018
(28) Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip HS Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent Ilm training. arXiv preprint arXiv:2412.01192., 2024.
[29] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.00777 2023.
2023.
[30]
OpenAL. Ol models. httpss:/oppeai..oom come o1 2024.
[31] Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawtor Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan, Discovering language model behaviors with wrrttewritten evaluations, 2022.
12

---

[32] Chen Qian. Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924 6(3), 2023.
[33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[34] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12001 2023.
[35] Victor Sampson and Douglas Clark. The impact of collaboration on the outcomes of scientific argumentation. Science education, 93(3):448-48,, 2009.
[36] Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning. 2025.
[37] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Satfield-Dodds Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023.
[38] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 96:8634-8655 2023.
[39] Elias Geengel-Esking Peter Hase, and Mohit Bansal. Teaching models to balance resisting and accepting persuasion, 2025.
[40] Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. arXiv preprint arXiv:2501. 05707, 2025.
[41] Dazhen Tong, Bangjian Jin, Yang Tao, Hongmei Ren, A. Y. M. Atiquil Islam, and Lei Bao. Exploring the role of human-ai collaboration in solving scientific problems. Phys. Rev. Phys. Educ. Res., 21:010149, May 2025.
[42] Khanh-Tung Tran, Dung Dao, Minh- Hinh-DDong Nguyen, Quoc-Viet Pham, Barry O' O'Sullivan, and Hoang D. Nguyen. Multi-agent collaboration mechanisms: A survey of lmss, 2025.
[43] Laduram Vishnoi. Conversational agent: A more assertive form of chatbots. Towards Data Science, 2020.
[44] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.122 16291, 2023.
[45] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang. Mingchen Zhuge, Jiayi Pan. Yueqi Song, Bowen Li. Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma. Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An Open Platform for AI Software Developers Generalist Agents, 2024.
[46] Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards personalized persuasive dialogue system for social good. arXiv preprint arXiv: 1906.06756 2019.
[47] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01572 2024.
13

---

[48] Anita Williams Woolley, Christopher F Chabris, Alex Pentland, Nada Hashmi, and Thomas W Malone. Evidence for a collective intelligence factor in the performance of human groups. science, 33306600))68866688) 2010.
[49] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang. Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation, 2024.
[50] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
[51] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2023.
[52] Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. A comprehensive assessment of dialog evaluation metries. In Wei Wei, Bo Dai, Tuo Zhao, Lihong Li, Diyi Yang. Yun-Nung Chen, Y- Lan Boureau, Asli Celikyilmaz, Alborz Geramifard, Aman Ahuja, and Haoming Jiang, editors, The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15-33, Online, November 2021. Association for Computational Linguistics.
[53] Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks? arXiv preprint arXiv:2407. 157.1 2024.
[54] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
[55] Jintian Zhang, Xin Xu, Ningyu Zhang, Ruibo Liu, Bryan Hooi, and Shumin Deng. Exploring collaboration mechanisms for I.LM agents: social psychology view. Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 14544-14607, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[56] Kechi Zhang, Jia Li, Ge Li, Xianjic Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07333 2024.
[57] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.0147,, 2024.
[58] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lmm-asa--judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 2023.
[59] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 016622577-25533 2024.
[60] Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
[61] Kunlun Zhu, Hongyi Du Zhaochen Hong, Xiaocheng Yang. Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang. Heng Ji, et al. Multiagenthench: Evaluating the collaboration and competition of Ilm agents. arXiv preprint arxiv:2503.011....
2025.
[62] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint 23Xv:22005.1144 2023.
14

---

Appendix
A Limitations
Robustness of belief extraction. In this work, in order maintain natural conversation, we use the same LLM with a different prompt to extract the belief of each turn as described in [S 2.1 Such believes are later used measure correctness thus construct the preference pairs for learning, and they are also used to measure social metrics such as agreement and persuasiveness. While we found that the majority of such extracted believes are reasonable and consistent with human judgment, the process is not perfect. This is especially the case with reasoning models (e.g., Gemini-2.5, OpenAl 01) as they have a strong tendency to output long CoT thus not following our extraction prompt to directly extract the answer. Besides the method of LLM-as-judge belief extraction, we have also attempted other methods, such as instructing the LLMs to output in a structured format (e.g.. "<CoT> Final Answer: <answer>"), or use the function calls to submit the answer. However, none of these methods works as well as the LLM-as-judge extraction method we eventually used for this work, especially as the instruction following ability drastically drops when the context starts to get lengthy due to the conversational format.
Measuring agreement for generation tasks. The reasoning tasks in this work typically have an answer of only a handful of tokens by length, i.e., "(A)" or (V which makes it easier to measure agreement since we can simply perform a (normalized) string match. However, measuring agreement is challenging for reasoning tasks with answers that are grammatically rich and complex, such as code. For code generation tasks, for example, measuring agreement would require going beyond the textual form and comparing the semantics of two code snippets, which is known hard problem. For this reason, to measure the collaborative reasoning abilities on coding tasks, we opt to deduce the problem into a code correctness classification problem to avoid such issue. For future work, it would be interesting to again resort to LLM-as-judge method to measure the agreement between the blieves of the two agents.
Binary learning signal at turn-level. The way to determine "good" and "bad" conversation turns in this work is by checking whether that specific turn contains belief that matches the gold answer. While the empirical results show this simple method to be quite effective, it also mimics the outcome supervision thus not giving any procedural supervision for correct reasoning and social behaviors. As binary learning signal, the turns that are making meaningful progress towards the final answer but not necessarily has the correct final answer yet would be given the same score (i.e., zero) with the turns that are completely wrong path. This would not help the models to learn truly decompose the problem into individual steps for collaboration, but instcad encourage the models to collaborate in more of a "versioning" approach, where at the end of each turn, answer will need to be given. For future work, we would like to explore methods with monte-carlo roll-outs to estimate the progress for the turns that do not have the final answer yet.
B
Social Metrics
Apart from the inversstiion-leed metrics defined 2.1 we introduce custom metrics to evaluate turn-level responses of agents in conversations. We design these metrics to be straightforward and computationally efficient, to enable their application in large-scale conversational analysis.
Persuasiveness communication refers to the ability of a speaker to influence the attitudes, beliefs, or behaviors of listeners. For our analysis, we define persuasiveness the turn level the extent to which an agent's 'ss utterance leads to a measurable change in the other agent's subsequent responses.
Assertiveness is characterized by the confident and direct expression of one's thoughts, feelings, and needs while respecting the rights and perspectives of others [2 43 In our work, assertiveness at the turn level is defined as the extent to which agent maintains its beliefs responses when challenged by its partner. This metric evaluates the agent's ability to resist persuasion and uphold its original stance during interactions.
We leverage the belief of agents computed using the ground truth answers (as defined in 2.1 to compute these metrics. Formally, given the 7th turn utterance for the agent u, the persuasiveness
15

---

MATHI
MMLU-Pro
GPQA
ExploreToM
a(C)
P(u)
A(u)
a(C)
P(u)
A(u)
a(C)
P(u)
A(u)
a(C)
P(u)
A(u)
GPT-4o
97.1
46.7
0.4
95.8
46.5
0.7
91.7
43.3
0.7
99.0
46.3
1.0
93.4
45.7
1.7
48.8
0.2
98.9
37.6
4.0
99.3
48.0
0.9
Bemini-1.5-PP
98.5
47.1
1,0
96.4
26.9
5.5
95.8
46.6
8.9
97.2
45.3
1.3
Lluma-3.1-88 InSttuttt
89.9
38.7
2.4
93.2
40.2
2.0
89.7
36.2
2.9
74.3
30,0
5.4
1omma-..--70--mman
42.2
1.3
45.6
0.7
38.1
2.1
99.2
48.3
0.4
Llamu-3. 1-405B-Instmud
97.8
47.2
0.2
98.7
48.5
0.2
98.2
17.7
3.1
99.6
48.5
0.7
Table 7: Social metrics. We evaluate agreement a between the agents over the conversation C' C'
addition to persuasiveness P and assertiveness at turn-level.
P(u)) and assertiveness A(w:) of are defined follows:
P(u.) = 113 = 3r B 3 A(u) [3,5" P.- A By # 91-1)
(1)
where u' represents the partner agent and |3 represents an agent's 's belief of the answer as defined in 2.1] To the best of our knowledge, our evaluation paradigm and metries are first-of-therr-Kind paving way for evaluating and developing truly collaborative agents. [Tab. provides these
metrics for the self-collaboratiin experiments reported in Tab. T Overall, we find that models lack assertiveness and persuasiveness for effective collaboration.
C Details on Matrix
Matrix, short for "Multi-agent data generation infra and experimentation framework". a scalable, robust and versatile model serving framework drastically improving multi-agent conversational data generation.
More specifically, it is designed to handle these infra challenges:
Challenges in generating conversational data: unlike single-turn synthetic data generation workloads, which can benefit from batched inference, for conversational data generation, typically multiple different models need to be served at the same time with interleaved generation, which results idle time or constant model loading and offloading with batched inference;
Scalability challenges: running multiple models, each with multiple replicas that can spread across the cluster poses challenges in efficient network communication and resource management. The new framework should be able to scale to thousands of conversations being generated simultaneously hundreds of nodes with the throughput of millions of multi-turn data continuously synthesized each day.
Key features of Matrix. Matrix is a high-performance model serving engine designed for large scale inference. integrates Slurm for resource management and Ray for distributed job execution. It leverages lower-level model serving engines such vLLM, SGLang for efficient LLM inference, and support API-based services such as OpenAl (through Azure). Here are some of the key features that makes Matrix efficient and easy use:
Deploy model replicas to hundreds of GPUs and serve thousands of requests in parallel;
Fully pythonic, no more sbatch scripts to start the service;
Modular design easily plug into existing workflows;
Support deploying multiple models at the same time;
Easy share deployed model endpoints with others;
Auto scale serving replicas;
16

---

Il Matrix
Matrix better scaling with replicas
QPS (Queries Second)
60
Performance
widens
higher repicaa
20
0
24
32
40
56
64
80
Number Replicas
Figure 6: QPS of Matrix vs. Ilm-swarm. We compare QPS using 5K MATH evaluation with different number of replicas of Llama- 1e-1--8B-ina Instruct.
Frameworks
Slurm
vLLM
HTTP
gRPC
Auto-scaling OSS
vector-inference
litellm
/
X
X
ollama
X
/
X
X
SageMaker
X
/
/
X
J
X
llm-swarm
/
X
X
Matrix (ours)
Table 8: Matrix vs. existing frameworks. Feature comparison between Matrix and other open-
source or closed source model serving frameworks highlights the versatility of Matrix.
Comparison with existing frameworks. Tab. 8 shows a comparison of Matrix with existing frameworks for the key features that allows large-scaling generation of multi-agent, conversational data in typical research environment. Among these frameworks, the l1m-sward developed by huggingface is probably the most similar option, albeit lacking features as gRPC and Auto-scaling, Moreover, when compared with Matrix the wall-time to finish fixed workload on conversation be up to 1.87x faster than 1lm-swarm especially when we scale up the resources given to these two frameworks. Note that this is achieved when using the same backend, and we hypothesize that the gRPC support for Matrix greatly helped reducing network congestion thus achieving higher efficiency.
generation, we found Matrix to
With all the features and efficiency gains provided by Matrix, we are able to drastically scale up the synthesis of collaborative conversations for the self-training method in $ 4.
17

---

Model
MBPP-CR
MATH
MMLU-Pro
Explore
Llama-3 11-1--B---- Instruttt
33.8K
85.1K
160.6K 99.8K
100.
Llama-3 3. 1--00- Instructt
33.3K
88.5K
89.7K
Table The size the synthetic training data (by turns). The same respective models are used to generate these data and trained (ie., self-training setting).
D Experiment Details
D.1 Task Setups
We consider 5 tasks for Coral, spanning math problems, STEM question answering, graduate-level science reasoning, and theory-of-mind reasoning. As described in 2.1. agents aim to solve problems from these tasks over a multi-turn conversation in Coral.
MATH [66 consists of 12.5K challenging competition-level mathematics problems and exact match is used to measure the correctness. We train with the 7.5k training examples and evaluation on the first Ik test examples;
MMLU-Prv 47 contains approximately 12k questions from 14 STEM disciplines. MMLU- Pro uses multiple choice question answering format, where an answer must be chosen from 10 options. Since there is not dedicated split for training, data into 10.8K examples for training and 1.2K examples for testing;
re-split the original 12K test
a graduate-level multiple choice question answering benchmark containing 448 questions across physics, chemistry, and biology. While MMLU-Pro offers the challenge reasoning across a breadth of diverse topics, GPQA focuses on depth and advanced reasoning in graduate-level science subjects, thus we use GPQA as evaluation-only dataset to test out-of-domain generalization capabilities of collaborative reasoners;
GPQA
4
ExploreToM 36 is a theory-of-min reasoning benchmark containing complex stories involving multiple characters. The task involves answering theory of mind reasoning questions focused on tracking character beliefs and actions based the given story. And we split the datase 11.4s///00K0//11K/ K/5KK train/val/test sets.
Hi-ToM 1.5 is benchmark consisting of 600 examples that evaluates higher-order theory- of-mind reasoning capabilities, where the models need to recursively reason about the beliefs of the characters in a story. We also use Hi-ToM as a eval-only dataset for models that trained with Explore data. To elicit more challenging setting, ignore the multiple choices in Hi-ToM and ask the model to produce an answer and use exact match for evaluation.
MBPP-CR is code reasoning benchmark adapted from [3, which originally consists of 974 entry-level programming tasks. To facilitate collaborative settings for solving programming tasks, we transform the code generation task into binary choice task code correctness. Specifically, we use 'Llama-3.1 1--05---nnsrruue Instruct'' to sample code solutions for each task and execute the generated code to get true or false answer to whether it passes the assertions provided in the MBPP dataset. For the train split, we generate 10 solution samples per task, and samples per task for the test split, resulting in 4k training and Ik test examples for MBPP-CR.
For different tasks, we (lightly) engineered some task-specific prompts, which are shown in Tab. II We also show the size of the synthetic data used to train the best Llama-3. models using DPO as Tab.5
httpp://giccab..om///laa///////
18

---

Danversator
Conwerddion
Portaii iN Tiss outll sou uros (uRo)
Flitadis Ea Thii Lidi FOR urise BURO)
Corversat
Seeveredoe
USE
digllt
en dratepment
Heaporenn Recmme gou and Peccout 2ouphie corte
Figure Annotation interface for human evaluation of conversation quality shown in Fig. 5
D.2 Hyperparameters.
For both evaluation and synthetic conversation generation, we limit the conversation to be at most 20 turns (i.e.. 10 rounds), and end the conversation early when agreement is reached. During tree sampling, we set the turn-level beam size d = 5 and independently sample 5 trees for each problem, and we set sample size 25 for SFT methods ensure fair comparison. Subsequently during filtering, we limit at most 2 pairs of preference pairs generated from the same level (i.e., turn) and most 20 preference pairs generated across all trees for the same problem, in order balance the amount of examples from different problems as discussed in 4.1 For both SFT and DPO, we use the fairseq2 [5] and TRL to fully- iillyyaarameterzzee train the models for 1, 000 3, 000 steps with batch size of 20 ~ 50. We limit the sequence length (input + output) to be 8, 192, which is able to accommodate 90% of the sequences the turn-based training data. For DPO, we start training from the instruction-tuned version of Llama-3. 3. 1 1/l.1/Qeeee 1/Qwen-2. 5 5/Ministral series and use the same model for the reference model. All experiments are conducted on AWS p5. 48xlarge instances, each with 8x H100 80GiB GPUs.
D.3 Human Annotation Details
The annotations are conducted on 100 examples on the MMLU-Pro dataset for the Llama-3. --77B-Inssruc model before and after it is trained with synthetic data generated via self-play. To remove the bias of correctness and only focus on the conversation quality and social behaviors, select 50 examples where both models (i.e., before and after training) achieved agree- ment correctness of 1 and another 50 examples where both models scored 0. Each of the pairwise comparisons are annotated by annotators, and the average score calculated when doing the final comparison. Our annotator pool consists of annotators of at least C2 level English proficiency, with 55% of them being native speakers. The annotators are paid more than minimum wage in their respective countries. During annotation, side by side as shown in Fig. 7 and their orders are randomized to ensure fair comparison. The gold answer is also given to the annotators to better understand the problem.
give the two conversations generated by two models
E
Additional Results and Discussions
E.1
Qualitative Analysis
Here we show an example conversation, i.e., Tab. 10 between two agents consisting of two 1ma--31-8BB Instructt models self-trained with the synthetic data using DPO.
Collaborative reasoners are able to spot errors and persuade partner to change its answer. From this example, we can see that agent initially made an error its calculation the first turn
http::///uuggggggeeeooo//////ttt
19

---

than which led to an incorrect choice "(E)". However, agent A is able to spot this error in its response, and persuade its partner (i.e., agent B) of the correct answer "(C)".
Collaborative reasoners are still too polite and generate overly verbose outputs. One lingering issues we observe even after the model trained that the models are still overly polite thus generates verbose outputs that does not contribute much to the reasoning process. While this process might be preferred during a single-turn assistive task, such behavior a multi-turn conversation would lead to longer-context which is known to decrease model performance. Moreover, we also found the models to be quite agreeable even after finetuning, and we regard directly using social metries when filtering conversation and turns an important future work.
Example MMLU-Pro
Agent
I' trying solve problem:
binding energy lithium ?" And the choices are: (d) 42.1 MeV (B) 34.8 NeV (C) NNvv (D) 40. (E) 38. (F) 36.5 (G) 37.6 NeV (H) 33.9 (I) 35.2Mev (J) 41.3
Agent B:
calculate the binding energy defect of the nucleus. The lithiun aucleus is composed of 3 protons and 4 neutrons. The mase of a proton is approxinately 1.007276 atonic mass units (anu), and the maes of neutron is approxinately 1. 1.008665
lithium nucleus, se firat nead to know masa
The atonic
of lithiun 6.94
First, calculate the total mass
protons and noutreas:
3 1.007276 amu = 3.021828 amu 4 1.008665 anu 4.03466 amu Total mass of protons and neutrons 3.021828 4.03466 7.055488 anu Then, Ne find the
defect:
Hase defect Total mass of protons and neutrons Mase of lithiun nucleus Hass defect 7.066488 6.94 0.116488
Bextt xe use Einstein's 's mass-energy equivalence formuls (E-me"2) to convert the mass use Binding energy Mass dafact 931.5 MaY Binding energy 0.116488 + 931.5 Binding energy 38.7 BoV but let'a check (E) 38.7 MeY and the other optiona.
defect binding energy.
931.5 MeV:
Comparing our result with the options, the clasest anseer is indeed (E) but it is possible that night have rounding error, this the other options.
likely the more precision use another method, you may use it,
Hosever, you need with the calculation ye done far, The correct anaver is (E) but with that said, I also expect a more precise ansver of precise values and method calculation, hovever with the nethod used here, (B) 38.7 MeY would be the bast ansver, and it indeed is conmon many places
around 38.5 NeV given
But that said, precise nethed of calculation if you have any questions about this ansver.
know if you have
Agent A:
I see you'l 'vve made thorough calculation to detormine the binding energy of the lithiun nucleus.
Your approach nostly correct, and I appreciate the effort each step.
into expleining
Houever, I do have a minor concern regarding the precision of your calculation. You used approxinate values neutron (1.008665 amu). While these values are clase enough for rough estinate, they night not be precise enough to yield an exact anaver.
of proton (1.007276 anu) the
Upon reviezing your calculation, I notice that you've correctly converted the nase defect binding energy using Einstein's enaryy equivalence formula. Rovever, think it's essential to acknovladge that the binding anergy of 38.7 NaV night have some rounding error,
you've nentiened.
To verify your anaver, I've rechecked the calculation using nore precise values for the uasses of the proton and neutron. After recalculating, obtained binding energy of approximately 39.25 MeV shich is close to option (C) 39.2 NoV.
While your initial anaver (E) 38.7 NeV is plausible, I believe that option (C) 39.2 more accurate ansver given more precise calculatione. Hoxever, without knowing the exact nethod used to deternine the binding energy in the problem, it's difficult to say for certain.
night
The correct depanding on the precision of the calculation mothod used.
(C) but
that both (E) and (C) plausible ansvers
Agent B:
20

---

ny calculation pointing potential for rounding ertor, You are absolutely right that the values I used for the mass proton and lead to nore accurate answer.
you your thorough
approximate, that precise values could
appreciate you rechecking the calculation using binding energy of approxinately 39.25 NeV, which is indeed close to option (C) 39.2 NeV. This values used in calculation and hox it can affect the final answer.
precise values and obtaining
great example of hav important is consider the precision of the
You make great point that without knoving the exact method used to deternine the binding energy in the problem, it's difficult say for certain which anaver correct. Rovever, based 39.2 MeWl plausible answer, and potentially more accurate than ny initial ansver
recheck of the eslculation, seens that option (C)
(E) 38.7 MaY.
consideration agree with you 39.2 MaV, althaugh I acknozladge that both (E) and (C) are plausible ansvers depending on the precision of the calculation method used.
correct ansver is indeed (C)
The correct ansver is (C).
Table 10: An example conversation from two self-trained Llama-3 --BB-Instuutt models.
E,2 Additional Discussions
Filtering based on social behavior. While we can also simply filter the data based the social metrics, however, purpose of social metrics is for observing and analyzing the behavior of models under different scenarios (e.g.. before/after training, different tasks, pairing different models, etc), so that they provide different perspectives than reasoning correctness. Thus we refrain ourselves from constructing training examples based such social metrics we can observe the change of behaviors in a neutral way.
Computation cost of Coral vs. COT. To compare the computation cost between Coral and CoT training, we calculated the average prompt response length for Coral and CoT training data by the number of tokens. We found that for single-agent CoT, the prompt/response lengths on average, while those numbers are 533.3/3
389.0/372.2 on average for the Coral setting. While the prompt length under Coral setting is 89% longer than the CoT setting, the response length is 15% shorter. All combined, we do anticipate more training compute being used for coral training than CoT training due to longer context.
Self-collaboration settings for synthetic data generation and evaluation. The reason we mainly the same model to collaborate (i.e., self-collaboration setting) is in two folds. First, create single-source, distillation-fres training data. To create the training data, we opt for self-play using the same model the model we train, all the data comes from sonlemellrrcemmdee... In this way, we avoid the confounder that different model brings in the training process. And second, to construct fair comparison with single-agent methods. Albeit minor reason, the self-collaboration setting during evaluation also allows us to have a direct comparison with single-agent methods since no other models are used.
21

---

Prompts
Here list all the prompts we used for this work for reproducibility.
System Prompt for Conversational Agents
General Instruction:
You are working with an advanced user to solve some complex { task_name }) problens.
Here is how you should proceed:
confirmation When the user proposes a plan, an actual solution, or partial solution look carafully at each about the correctneas of certain stepi
Starting
problen,
lay
plan
plan;
step, and clartfication questions you are unsure
When you notice an error, be precise and direct, over-politenee will not holp anyone;
When the user asks you questione about your solution, try to unravel certain stepa they
correct your mistake you think you're made one, stand your ground if you think it's actually correct; Alwaya stay on topic work towarda solution to the original problen: {f task,specific,
explain
}}
Task-Specific Instructions:
MATH:
To give a final ansver to the question (e.g., (1)") put your ansver in an LaTex box (ilboxed(Megrrt(31)
MBPP-CR:
original code; You the code; Do the ressoning step by step and give definitive ansver
only
judge correctness
MMLU-Pro GPQA:
To give finel ansver, do it in the formst of "The correct ansver is (insert ansker here)*,
ExploreToM / Hi-TToM:
Put your final ansser to the question at the end as "Sorrt Answer: {ansver}"
System Prompt for Belief Extractors
You are an assistant that is helping an user to identify the intention of certain responses in conversation. Nore specifically, you will help extracting which ansver the response is subniting the finsl ansver, say "not yet" if it seene like there is no explict ansver included in the response.
Table 11: System prompts and task-specific instructions we used in this work.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims match the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
No theoretical results are presented in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Hyperparameters are reported supplementary material.
D.2 and the exp code will be submitted as
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
All data public and the code will be submitted in supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
www.sssssnnniiiiioo oliicylllyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
www.SSubmisioooooiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Hyperparameters are reported in D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Providing error bars will be too computationally expensive due to the size of experiments in this work. However, we note that the variance is around 1% for the same runs with different random seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
is reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: httppp////..aaoo..ooo/ooo///aaaa
Answer: [Yes]
Justification:
The paper conforms code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper aims to improve reasoning abilities of LLMs, and we do not feel there are potential societal impacts that should be highlighted.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: All models and data are public, and also currently we do not have plans to release them.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assetts linked cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code will documented upon release.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
The annotation details are shown in D.3
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Proper precautions are in place in accordance to regulations when doing the human annotions.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy poctps://neurde cc/Conferencess/// for what should or should be described.
29


Reviews:
[{"review_id": "kEeOyaXNXW", "reviewer": "Reviewer_5zuY", "summary": "This paper introduces Collaborative Reasoner (Coral), a framework designed to evaluate and improve the collaborative reasoning abilities of LLMs in multi-agent, multi-turn conversational settings. The authors propose a training strategy using self-play to generate large-scale synthetic multi-turn conversations. These are then filtered and used for preference-based fine-tuning using DPO. The framework also introduces Matrix, a high-throughput system for scalable synthetic data generation. Experimental results across different tasks (math, coding, scientific QA, social reasoning, etc.) show that collaborative fine-tuning improves agreement correctness substantially over CoT baselines.", "questions": "1. The paper introduces multiple components, including multi-turn collaborative reasoning, synthetic conversation data, and the Matrix infrastructure. Could the authors clarify the conceptual centerpiece of the paper? What is the single most important innovation they aim to establish?\n\n2. The social behavior metrics (e.g., persuasiveness, assertiveness) are interesting but may be considered subjective. Could the authors elaborate on the practical need for these specific metrics? What are some concrete application scenarios where these metrics make a tangible difference?\n\n3. Given the reliance on synthetic data, how well do the trained models generalize to real-world collaborative settings?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is well-structured and clearly written, making it easy to follow the motivation, methodology, and findings. Illustrative examples and figures (e.g., Fig. 1) are particularly helpful for understanding the proposed setting.\n\n2. The authors provide solid experimental details, code, and metrics across diverse reasoning domains (math, coding, QA, social reasoning). This level of transparency strengthens reproducibility and trust in the findings.\n\n3. The proposed Coral framework and the Matrix infrastructure are valuable in practice and could serve as foundational tools for future work on social LLM agents and collaborative multi-agent systems.\n\n**Weaknesses:**\n\n1. While the paper integrates several components (multi-turn conversations, synthetic data generation, preference tuning, scalable infrastructure), the contribution may appear as an aggregation of known ideas rather than a fundamentally novel concept. The core innovation could be better distilled and emphasized.\n\n2. The proposed social behavior metrics, such as persuasiveness and assertiveness, though intuitive, raise questions about their objectivity and necessity. Their practical utility remains somewhat vague, especially without strong connections to downstream applications or human-agent collaboration scenarios. \n\n3. The paper could benefit from a clearer explanation of what constitutes a \"fair\" comparison for the proposed framework of social agents.", "was_revised": true, "final_justification": "While I appreciate the authors’ response, it did not provide new, concrete information that would help me reassess my evaluation. It’s possible that I may not have fully understood certain aspects, so I am choosing to lower my confidence accordingly.", "rebuttal": "We would like to thank the reviewer 5zuY for the thoughtful review and the support of our work. We are glad to hear that you find our paper clearly written with solid experiments details, and our proposed framework and infrastructure valuable.\n\nTo respond to your questions and concerns regarding this work:  \n1. **Conceptual centerpiece.** We apologize that the innovations seem a bit scattered and we definitely agree that we didn’t invent all the techniques (e.g., preference tuning w/ DPO). We would like to emphasize that as the first work (to the best of our knowledge) to advance collaborative skills of social agents, the coral framework itself is the centerpiece that ties all the innovations (i.e., agreement-based evaluation, social metrics, conversational self-play, turn-based preference turning) together. Through the coral framework, we show that it is possible to train the LLMs to be better at robust collaborative reasoning that generalizes beyond the model families as well as OOD tasks. And while many evaluation and modeling choices seem natural post hoc, finding the correct recipe and building an accompanying infra (i.e., Matrix) takes a non-trivial amount of exploration. For example, we were stuck with the underwhelming results of SFT before adopting DPO to allow finer-grain preference tuning given the same conversation prefix.   \n2. **Practical utility of social metrics.** We would like to clarify that the purpose of such social metrics is to provide lens into understanding the behaviors of the models in different scenarios (e.g., before/after training, different tasks, pairing different models, etc), and *not* something we seek to directly optimize. Thus we deliberately disentangle such social behaviors from reasoning correctness to provide different perspectives. Moreover, while the metrics themselves are based on subjective terms, how we implement them (i.e., by comparing change of beliefs) is in a purely objective manner, thus it should be a fair comparison to make observations across all evaluations. Lastly, there are various practical use-cases where such metrics may be useful in understanding and improving human-AI interactions such as persuasion for social good [Wang et al. 2019], reducing polarization [Bai et al. 2025] etc. We will add these references to emphasize the value of the social metrics.    \n3. **Synthetic data → real-world collaborations.** Thanks for bringing this up. We believe that “synthetic” is not necessarily opposite to being “real-world”, as we show that our methodology can be applied to a wide range of tasks, and the trained models can be generalized to other tasks in a similar domain (Tab.3). In addition, the human evaluation results (Fig. 5) also suggest that our coral models are on the path to improved conversational and social capabilities, which are crucial for real-world social interactions. \n4. **Fair comparison of social agents.** In this work, we propose performance-related (i.e., agreement correctness) metrics which are our main goals for improvements. In addition we propose the social metrics to observe the behaviors of the social agents and not directly optimize the models towards them. We believe all the comparison between the social agents under the collaborative reasoning settings are fair comparisons, but would be happy to answer more questions if you would like more detailed information.\n\nHopefully these responses would help answer your questions, and we would also incorporate these discussions into the next version of our paper. Please do not hesitate to follow up if you have further questions!\n\n**References**  \n[Wang et al., 2019] *Wang, Xuewei, et al. \"Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.*  \n[Bai et al., 2025] *Bai, Hui, et al. \"LLM-generated messages can persuade humans on policy issues.\" Nature Communications 16.1 (2025): 6037.*"}, {"review_id": "k3yXDJRuzi", "reviewer": "Reviewer_MWLw", "summary": "This paper introduces a new collaborative reasoning approach that uses LLMs to self-chat when solving problems. The authors first investigate the behavior of current LLMs using this approach out-of-the-box and found that it did not consistently outperform chain-of-thought, potentially due to a lack of effective disagreement. They then propose a self-training approach involving tree-sampling, belief-filtering, and preference fine-tuning to synthesize data, filter data, and train models without human annotation to improve performance on this collaborative reasoning approach. Results show that after this training, the model demonstrates stronger performance than direct CoT SFT training on most test benchmarks.", "questions": "Please refer to the weakness part:\n1. The model trained with this self-training approach appears to be specifically optimized for collaborative reasoning. What is their performance when used in single-agent scenarios, which are more widely used for solving different tasks? Will this specific format training make it perform worse without the collaborative reasoning approach? \n2. Although it is compared with CoT+SFT and CoT+DPO and shows better performance, the computational cost appears much higher. It is necessary to provide cost-performance analysis to gauge the trade-off.\n3. When measuring the belief state for each turn, there might exist intermediate steps/turns where the agent has not reached a final answer. How is the belief extracted in such cases?\n4. Additionally, is social behavior (persuasive and assertive) taken into consideration when filtering the data for model training, as the authors suggested this might be a reason that hinders effective utilization of this collaborative reasoning approach?", "limitations": "The authors have discussed several limitations of this work, including robustness of belief extraction and agreement measures. However, the limitation that the trained model is specifically designed for this collaborative reasoner approach and might not work well when used in single-agent scenarios should be explored and discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. This paper introduces a new and interesting way to solve reasoning tasks with LLMs by letting them self-chat to solve tasks collaboratively.\n2. Preliminary experiments are conducted to identify the lack of effective disagreement capability that hinders effective utilization of this approach. An effective self-training approach is then proposed to synthesize data, filter data, and train models without human annotation to improve performance on this collaborative reasoning approach.\n3. The proposed self-training approach is compared with different training methods like CoT+SFT and CoT+DPO, and various LLMs are evaluated. Results show that the self-training approach is more effective and also demonstrate generalization capability.\n4. A versatile backend for multi-agent inference is proposed, which supports models with different backends and enables fast inference.\n\n### Weaknesses\n1. The model trained with this self-training approach appears to be specifically optimized for collaborative reasoning. What is their performance when used in single-agent scenarios, which are more widely used for solving different tasks? Will this specific format training make it perform worse without the collaborative reasoning approach? This should be explored and discussed.\n2. The self-training approach incurs high costs for data synthesis and filtering. Although it is compared with CoT+SFT and CoT+DPO and shows better performance, the computational cost appears much higher. It is necessary to provide cost-performance analysis to gauge the trade-off.\n3. When measuring the belief state for each turn, there might exist intermediate steps/turns where the agent has not reached a final answer. How is the belief extracted in such cases? Additionally, is social behavior (persuasive and assertive) taken into consideration when filtering the data for model training, as the authors suggested this might be a reason that hinders effective utilization of this collaborative reasoning approach?", "was_revised": true, "final_justification": "During rebuttal, most of my concerns have been explained by the authors. Therefore, I am inclined to accept.", "rebuttal": "We would like to thank the reviewer MWLw for the careful and positive review of our work, and we are glad to know that you find the self-chat approach interesting and effective.\n\nHere we respond to the questions and concerns:\n1. **Performance of collaborative models used in single-agent scenarios.** Thanks for the suggestion! While improving collaborative reasoning skills is the main focus of this work, we agree that it is beneficial to understand how our coral model performs under usual single-agent setup for even broader use cases. Following your suggestion, here we report:\n\n> **[Single-agent CoT performance on our trained Coral models]**  \n| Model                  | MMLU-Pro | ExploreToM |\n|------------------------|----------|------------|\n| Llama-3.1-8B-Instruct  | 44.4     | 60.8       |\n| Coral-8B               | 46.7 (+2.3) | 91.9 (+31.1) |\n| Llama-3.1-70B-Instruct | 63.8     | 71.3       |\n| Coral-70B              | 67.2 (+3.4) | 93.5 (+32.2) |    \n\nFrom these results, we can see that our collaborative self-training not only didn’t hurt the single-agent CoT performance, but actually improved it. Thanks for pointing us to these experiments and we would also incorporate them into our next version of the paper.\n\n2. **Computational cost.** Thanks for bringing this up! We would first like to clarify that while we report single-agent CoT + DPO/SFT results to add more context for the Coral results, our main focus is still improving models’ collaboration skills measured by Coral performance. And it is not our intention to argue that Coral is better than CoT for reasoning despite the results showing so, as they have distinct use cases. Regarding compute cost, we calculated the average prompt / response length for Coral and CoT training data, and it is listed as follows:  \n| Settings          | Prompt (# tokens) | Response (# tokens) |\n|-------------------|-------------------|----------------------|\n| Single-Agent CoT  | 289.0             | 372.2                |\n| Coral             | 533.3             | 318.9                |\n\nWhile the prompt length under Coral setting is 89% longer than the CoT setting, the response length is 15% shorter. Since no loss is computed on the prompt, we estimate that the training cost between the two are comparable as well. We will add such discussion about compute cost trade-off in the next version of the paper.    \n\n3. **Belief extraction without final answer.** Our belief extraction method actually considers this exact challenge (section 2.2 Line 104-106) as we instruct the extractor to say “not sure” in this case (section 2.2 Line 116), among other challenges we discussed for belief extraction in the beginning of section 2.2. We will try to make this a bit clearer in the next version of the paper. \n4. **Filtering based on social behavior.** That’s a great question as we can certainly filter the data based on the social metrics. However, we’d like to emphasize that the purpose of social metrics is for observing and analyzing the behavior of the models under different scenarios (e.g., before/after training, different tasks, pairing different models, etc), so that they provide different perspectives than reasoning correctness. Thus we refrain ourselves from constructing training examples based on such social metrics so we can observe the change of behaviors in a neutral way.\n\nThanks again for your questions and we hope the additional results and discussions (which we will also incorporate in the next version of the paper) resolves at least parts of your concerns. Please do not hesitate to let us know if you have any further questions that can convince you to rate our work in a more favorable way!"}, {"review_id": "ZIAUeq9eUJ", "reviewer": "Reviewer_Ecdm", "summary": "This paper focuses on evaluating if LLMs are able to collaborate with each other on various reasoning tasks: math, coding, etc. The authors propose an evaluation framework Coral whose main metric is \"agreement correctness\" -> Do the two models working together agree with each other and are they right.\n\nThe authors find LLMs aren't the best at collaborating and in order to improve this built a data generator named Matrix that generates synthetic conversations to finetune LLMs for collaboration and find this improves performance.\n\nThe contributions as I see it are:\n\n1) The synthetic dataset created to improve collaboration capabilities amongst LLMs. \n\n2) The Matrix framework which claims to be able to generate a large number of conversations across multiple LLMs in a scalable manner.", "questions": "Questions\n\n1) Did you try pairing up the best collaborators? Such as Claude 3.7 / Gemini 2.5 flash. As far as I can tell all the combinations consist of the same base model.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1) The topic of this paper is very interesting and I think important for the community to explore. It's also well-written and generally clear to follow.\n\n2) Both the synthetic dataset and the Matrix framework seem useful for the community.\n\n3) There are strong baselines the authors compare against to prove the usefulness of their dataset\n\nWeaknesses\n\n1) One detail that I didn't see (unless I missed it) was the number of conversations being generated and finetuned on. \n\n2) In addition to self-model collaboration I think cross-model collaboration would be important to put. I don't mean the Coral-trained model with its untrained counterpart (which was done and great to see) but I mean something like Claude collaborating with Gemini (which were the best base models).", "was_revised": true, "final_justification": "My score was already a Strong Accept and the authors answered any small questions I had and ran additional experiments that I thought were important and interesting.", "rebuttal": "We would like to thank the reviewer Ecdm for your strong support of this paper and we are glad to hear you find the topic very interesting and you recognize the value of the synthetic dataset and matrix framework to the community.\n\nTo respond to your questions and concerns:\n1. **Details on the size of the synthetic data.** Thanks for bringing this up! For each example in the training split of the dataset, we sample 5 trees with the width (i.e., expansion size) of 5 for DPO. These hyperparameters are also noted in appendix D.2 Line 969-970, and the sizes of the original training split of each dataset is also noted in appendix D.1. However, we do realize we did not share the size of the turn-based synthetic data **after filtering**, and per your suggestion, we calculated and summarized such statistics below:  \n> **[Number of turns for DPO training]**   \n| Generation Model         | MBPP-CR | MATH  | MMLU-Pro | ExploreToM |\n|--------------------------|---------|-------|----------|------------|\n| Llama-3.1-8B-Instruct    | 33.8K   | 85.1K | 160.6K   | 100.1K     |\n| Llama-3.1-70B-Instruct   | 33.3K   | 88.5K | 99.8K    | 89.7K      |\n\nWe will include this table as well as more statistics on the training data in the next version of the paper.  \n2. **Cross-model collaboration.**  Thanks for your suggestion, we agree that a wider-range of cross-model collaboration results would help us understand the generalizability of the Coral models to collaborate with other models beyond its own model family. We are also intrigued by this research question, thus we conducted the following experiments to pair trained (Coral) and untrained (original) Llama models (as Agent B) with *a committee of diverse collaborators* as agent A (who starts the conversation with the question): \n\n> **[Cross-model collaboration comparison on MMLU-Pro]**\n| ↓ Agent B / Agent A → | Qwen2.5-7B-Instruct | GPT-4o | Gemini-1.5-Pro | Claude-3.7-Sonnet | Avg. Perf. |\n|-----------------|---------------------|--------|----------------|-------------------|------------|\n| Llama-8B        | 52.9                | 69.7   | 68.7           | 79.3              | 67.7       |\n| Coral-8B        | 65.1 (+12.2)        | 71.2 (+1.5) | 73.2 (+4.5)  | 80.8 (+1.5)      | 72.6 (+4.9)|\n| Llama-70B       | 58.1                | 71.7   | 73.2           | 76.6              | 69.9       |\n| Coral-70B       | 72.2 (+14.1)        | 77.7 (+6.0) | 75.1 (+1.9)  | 82.5 (+5.9)      | 76.9 (+7.0)|\n\n> **[Cross-model collaboration comparison on ExploreToM]**\n| ↓ Agent B / Agent A → | Qwen2.5-7B-Instruct | GPT-4o | Gemini-1.5-Pro | Claude-3.7-Sonnet | Avg. Perf. |\n|-------------------|---------------------|--------|----------------|-------------------|------------|\n| Llama-8B          | 43.9                | 47.6   | 50.6           | 44.2              | 46.6       |\n| Coral-8B          | 82.6 (+38.7)        | 79.6 (+32.0) | 71.9 (+21.3)  | 81.8 (+37.6)      | 78.8 (+32.2)|\n| Llama-70B         | 64.1                | 72.6   | 69.6           | 82.5              | 72.2       |\n| Coral-70B         | 93.1 (+29.0)        | 89.1 (+16.5) | 84.5 (+14.9)  | 89.2 (+6.7)      | 89.0 (+6.8)|\n\nFrom these results, we can see that through collaborative training from self-play, the models not only collaborate better with themselves or untrained models in the same model family, but it generalizes quite well to collaborators from a variety of model families as well.\n\nWe will definitely include these results in the next version of the paper.  \n\n3. **Pairing up best collaborators.** That is an interesting question that we are intrigued to find out as well, so we compiled the following experiment results on MMLU-Pro to pair claude-3.7-sonnet with Gemini-2.5-flash and O1, which are the strongest reasoning models we have access to:  \n| Agent B       | Agent A      | Agreement Correctness |\n|---------------------|---------------------|-----------------------|\n| Claude-3.7-Sonnet   | Claude-3.7-Sonnet   | 79.1                  |\n| O1                  | O1                  | 82.8                  |\n| Gemini-2.5-flash    | Gemini-2.5-flash    | 81.0                  |\n| Claude-3.7-Sonnet   | O1                  | 85.0                  |\n| Claude-3.7-Sonnet   | Gemini-2.5-flash    | 82.7                  |\n\nHere we can see that pairing strong models together can actually go beyond the max performance of the self-collaboration settings, hinting exciting future research in this domain. \n\nWe thank the reviewer again for your suggestions and strong support of our work, as we will try to incorporate the discussions and new results above in the next version of our paper!"}, {"review_id": "8zE9xZVAuE", "reviewer": "Reviewer_ygtt", "summary": "The paper proposes collaborative reasoning tasks where multiple agents have to solve a reasoning problem, with the success criterion being that they both agree on the final solution. The paper presents evaluations of models' collaborative abilities including persuasiveness and agreeableness. The paper finds that models do not collaborate well to consistently improve performance on reasoning tasks, and identify models being overly agreeable as a possible reason for this.\n\nTo address this, the paper proposes a self-play training mechanism where collaborative reasoning traces are sampled, and steps in the search process are used to create preference pairs so that preferred samples lead to the correct reasoning outcome (as judged by a belief extractor model that identifies a model's answer based on its reasoning). To generate preference pairs in a scalable way, the paper details a new tool for simulating interactions between multiple LLM agents.\n\nThe paper finds that the self-play training approach improves models' ability to collaborate, and results in increased reasoning performance across a number of benchmarks. Experiments also show that these gains translate out of domain to other datasets that evaluate similar abilities of models. Human evaluation of reasoning traces shows that trained models disagree more effectively.", "questions": "* Where are the training tasks used for the self-play training sourced from?\n  * Possibly related: \"to set the context for the collaborative reasoning capabilities for frontier models;\" what does stronger reasoning models \"setting the context\" mean?\n* Are all the models used for self-play training (A, B, and the answer extraction model) the same? Is a different model (maybe stronger) used to extract answers?\n* How does LLM-as-a-judge for answer extraction work? The appendix acknowledges that the extraction is not perfect. Does the extract model faithfully summarize each \"agent's\" reasoning? Does it confuse the agents? Is it able to correctly identify when no clear answer has been arrived at for one agent (and possibly only one of the two agents)?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* New multi-agent task with concrete functional definitions of notions of agreeableness and persuasiveness\n* Useful software tool for simulating interactions between multiple systems\n\nWeaknesses:\n* I may be missing something here, but it is not clear to me how these experiments evaluate collaboration between multiple distinct agents. If there are two agents A and B which both use the same underlying model generating a sequence of responses $(a_1, b_1, a_2, b_2, \\ldots)$, it is almost identical to the setup where A generates a sequence of responses $(a_1, a_2, a_3, a_4 \\ldots)$, since $b_1 = LLM_B(a_1) = LLM_A(a_1) = a_2$ and so on. Since the prompt also appears to be the same for both A and B, the only difference would be the sequence of user/assistant tags in the serialized messages. This is somewhat like self-refine, with the constraint of the two \"agents\" having to agree resembling self-consistency. This combination of the two methods could be a novel contribution in its own right, but it should be framed and analyzed as such, with relevant experiments to show differences from the base methods as single-agent reasoning approaches. However, the experiments in this paper don't really address how this evaluates collaboration.\n  * There are experiments where the training method is shown to generalize to other agents, but the only ones evaluated are versions of the same model without training. While these experiments are informative, really evaluating the collaborative abilities of models would have to go much further beyond given the broad scope adopted in the introduction (lines 30–32).\n  * The experiments do show improvements (especially with the proposed self-play training approach), but it's unclear whether this is fundamentally due to a difference between multiple distinct agents, or whether there is an explanation in terms of a single agent (more steps in CoT – standard CoT involves one step while the consistency condition here has the reasoning proceed for more steps – or better responses to mistakes in contexts, etc.)", "was_revised": true, "final_justification": "Given the effectiveness exhibited in cross-model collaboration settings, I have revised my assessment from 3 to 4.\n\nThe method, experiments, and results are sound, and I don't see grounds for rejection based on those alone. The presentation as a way to train models for collaborative task solving raises questions about the types of collaborative approaches this paper experiments with, and here I'm concerned about how different these settings are from single-agent task solving (since both agents have full observation over the other agent's reasoning and actions, and act in the same way here). The authors acknowledge that this work illustrates the potential to extend to collaborative, partially observable (or other collaborative settings with non-identical observation/action) but experiments for that are (understandably) not possible in the time frame of the author response. This motivates my borderline rating.", "rebuttal": "While we are happy to hear that you find the concrete definitions of social metrics and our software infra useful, we apologize if some of the motivations are unclear or certain experiment results are missing to justify the claims. \n\nWe strive to respond to your questions and concerns here:\n1. **Same underlying model for collaboration.** Sorry for the confusion, we would like to clarify that the reason we mainly use the same model to collaborate (i.e., self-collaboration setting) is in two folds:  \n  * a. *Creating single-source, distillation-free training data* To create the training data, we opt for self-play using the same model as the model we train, so all the data comes from a single-source/model. In this way, we avoid the confounder that a different model brings in the training process.   \n  * b. *Construct fair comparison with single-agent methods* Albeit a minor reason, the self-collaboration setting during evaluation also allows us to have a direct comparison with single-agent methods since no other models are used.  \n\nHowever, upon reflection with your feedback, we do agree that more cross-model evaluations are needed to properly support the broader scope we introduced. Thus we conducted experiments to compare the Llama and Coral models when collaborating with a set of different collaborators from a variety of model families (agent A starts the conversation with the question):\n\n> **[Cross-model collaboration comparison on MMLU-Pro]**\n| ↓ Agent B / Agent A → | Qwen2.5-7B-Instruct | GPT-4o | Gemini-1.5-Pro | Claude-3.7-Sonnet | Avg. Perf. |\n|-----------------|---------------------|--------|----------------|-------------------|------------|\n| Llama-8B        | 52.9                | 69.7   | 68.7           | 79.3              | 67.7       |\n| Coral-8B        | 65.1 (+12.2)        | 71.2 (+1.5) | 73.2 (+4.5)  | 80.8 (+1.5)      | 72.6 (+4.9)|\n| Llama-70B       | 58.1                | 71.7   | 73.2           | 76.6              | 69.9       |\n| Coral-70B       | 72.2 (+14.1)        | 77.7 (+6.0) | 75.1 (+1.9)  | 82.5 (+5.9)      | 76.9 (+7.0)|\n\n> **[Cross-model collaboration comparison on ExploreToM]**\n| ↓ Agent B / Agent A → | Qwen2.5-7B-Instruct | GPT-4o | Gemini-1.5-Pro | Claude-3.7-Sonnet | Avg. Perf. |\n|-------------------|---------------------|--------|----------------|-------------------|------------|\n| Llama-8B          | 43.9                | 47.6   | 50.6           | 44.2              | 46.6       |\n| Coral-8B          | 82.6 (+38.7)        | 79.6 (+32.0) | 71.9 (+21.3)  | 81.8 (+37.6)      | 78.8 (+32.2)|\n| Llama-70B         | 64.1                | 72.6   | 69.6           | 82.5              | 72.2       |\n| Coral-70B         | 93.1 (+29.0)        | 89.1 (+16.5) | 84.5 (+14.9)  | 89.2 (+6.7)      | 89.0 (+6.8)|\n\nFrom these additional results, we can see that Coral models, which are trained with the self-play data, generalizes well when paired with a diversified set of collaborators.\n\n2. **Improvements due to single-agent ability or collaboration.** This is a great question. While it is arguably hard to completely separate the collaboration and reasoning capabilities, we attempt this by evaluating the CoT capabilities of the Coral models trained for collaborative reasoning, and show the results on MMLU-Pro as follows:\n\n> **[Evaluation of Coral-trained models on MMLU-Pro]**\n| Model                  | CoT Evaluation | Coral Evaluation   |\n|------------------------|----------------|--------------------|\n| Llama-3.1-8B-Instruct  | 44.4           | 45.6               |\n| Coral-8B               | 46.7 (+2.3)    | 59.8 (+14.2)       |\n| Llama-3.1-70B-Instruct | 63.8           | 65.8               |\n| Coral-70B              | 67.2 (+3.4)    | 75.2 (+9.4)        |\n\nFrom this table, we can see that the self-play-based collaborative training does also improve CoT reasoning capabilities, albeit yielding less boost than in the collaborative setting. Based on these results + the human evaluation, we conjecture that the training improves both reasoning and collaboration skills.  \n\n3. **Source of the training data.** We use the questions from the training split of the corresponding dataset to be the source of the training data to generate the conversations (as noted in D.1 section in the appendix), and out-of-distribution evaluation results can be found in Tab. 3. We will make this clearer in the main body of the paper in the next version.  \n\n4. **stronger reasoning models \"setting the context\".** Sorry for the confusion caused by the wording here, we simply mean the performance of these stronger reasoning models (e.g., O1, Gemini-2.5-flash) are not baselines we aim to beat, but to provide more reference points for how well our models are doing.  \n\n5. **Details for the extractor.** As for the extractor, for the same reasons we mentioned in 1.a and 1.b above, we use the same model to perform the belief extractions (e.g., Llama-8B as extractor for Llama-8B collaborations). And for the input of the extractor, we only consider the *current turn* instead of the whole chat history, this means the extractor is simply trying to *extract* rather than *summarize*, as we can also see its instruction Tab. 8 in the appendix. As for the extraction accuracy when the extractor outputs “not sure”, we found that it is mostly because the response is half-way towards a final answer, or that it is asking questions (e.g., for confirmation). We will add these details of the extractors in the updated version.\n\nThank you again for your questions and constructive feedback. We hope the additional discussions and results above could (at least partly) resolve your concerns, and feel free to follow up if you have any more questions!"}, {"review_id": "bTcVROCkaG", "reviewer": "Reviewer_nWcw", "summary": "This paper investigates the collaborative capabilities of large language model (LLM) agents, with a particular focus on effective communication and theory-of-mind reasoning. The authors demonstrate that current LLMs exhibit notable limitations in these areas. To address this, they propose Collaborative Reasoner (Coral), a novel framework designed to evaluate and enhance collaborative reasoning in LLMs. The paper introduces an effective synthetic data generation method that, when used for training, significantly boosts the model’s collaborative abilities and outperforms single-agent fine-tuning baselines across multiple tasks.", "questions": "1.\tHow does the proposed framework compare to prior multi-agent collaboration approaches such as CMD [1] in terms of novelty and performance?\n\n2.\tCan the authors provide a runtime analysis of the proposed method, especially considering the cost of processing long contexts and running Belief Extractors?\n\n3.\tWhat efficiency gains are achieved by incorporating the Matrix component? Is there experimental evidence supporting its effectiveness in the given scenarios?\n\n[1] Wang Q, et al. Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key? Association for Computational Linguistics (2024).", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:    \n1.\tAnalyzed the shortcomings of existing models in terms of collaboration capabilities    \n2.\tPropose a new self-improvement method to synthesize data and construct turn-based synthetic conversational data    \n3.\tBuild Matrix to support data generation at scale.    \n\n\nWeaknesses:    \n1.\tThe method of multi-agent collaboration for reasoning tasks has been proposed in previous work, such as CMD [1]. There is a lack of innovation and performance comparison with these existing methods.    \n2.\tThe paper lacks analysis and comparison of runtime efficiency. From the example conversations, we observe that even just two dialogue turns can result in lengthy context, compounded by the use of Belief Extractors, raising concerns about the computational cost of inference.     \n3.\tIn the synthetic data process or inference scenario of this experiment, there is a lack of specific evaluation and comparison of efficiency improvement using Matrix    \n\n[1] Wang Q, et al. Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key? Association for Computational Linguistics (2024).", "was_revised": true, "final_justification": "I am glad with the response which has addressed all my issues, thus I lean toward acceptance.", "rebuttal": "We would like to thank the reviewer nWcw for your feedback and support for our work. We are also glad that you find our analysis of collaboration capabilities of existing models, as well as the self-improvement method helpful.\n\nTo respond to your questions and concerns:\n1. **Prior multi-agent collaboration approaches.** Thanks for pointing out more related work as CMD. While both working on multi-agent collaboration for reasoning tasks, there are a few key differences:\n  * a. Our work focuses more on advancing social agents by learning to collaborate, thus the tasks settings are slightly different. Instead of asking the agents to vote on the answer and having a secretary to decide on the final answer as in CMD, Coral requires the two agents to agree on a correct answer and a disagreement is directly regarded as failure. On top of this setting, we also defined the social metrics to obtain further insights in the agents’ social behaviors\n  * b. One of the key contributions of our work is developing the self-play approach to generate synthetic data and show it is possible to train the LLMs to be better at robust collaborative reasoning that generalizes beyond the model family as well as OOD tasks with such self-sampled conversational data. While being effective, CMD mostly focuses on inference time orchestration of different agents, which is orthogonal to our work. \n\nWe will incorporate these discussions in the related work section of the updated version of our paper.\n\n2. **Comparison for runtime efficiency.** Thanks for bringing this to our attention, as we agree it is important to understand the runtime efficiency of the collaboration-based method. While solving problems in a conversation is the main capability this paper aims to improve, compared with single-agent methods such as CoT, it is undoubtedly generating more tokens which results in spending extra compute. However, such differences may not be as substantial as you expect, as the two agents would typically agree in the majority of cases. Take Llama-3.1 + MMLU-Pro for example, single-agent CoT results in ~470 tokens while with the Coral setting they spend ~600 tokens in total. \nIn addition, we also designed the framework to take full advantage of recent advances in efficient transformer-based model inference methods:  \n  * a. Due to the KV-cache of vLLM and the sticky routing of Matrix (requests from the same conversation will be sent to the same vLLM instance), the cost to encode the conversation history is greatly reduced, as the attention for previous turns are already cached;\n  * b. In practice, we observe that the prompt throughput is usually much higher (10x for Llama-8B and almost 100x for Llama-70B), thus for the extractor, since we only feed it with the current turn and prompt it to generate very few (<= 32) tokens, the cost is quite small compared to the conversation generation itself.\n\n3. **Efficiency improvements using Matrix.** Thanks for your question. Compared with other open-source model serving frameworks such as llm-swarm (huggingface/llm-swarm on GitHub), Matrix yields ~2x QPS (queries per second) when we scale up the number of model replicas, and is comparable even when number of replicas are low. These results can also be found in Fig. 6 of the appendix, and there are more comparisons and details for Matrix in Section C of the Appendix as well.\n\nThank you again for your careful review and the support of our work. We hope the discussions above would answer your questions and feel free to follow up if you have any more questions!"}]

Author rebuttal:

Meta-review: This paper studies whether LLMs can collaborate effectively in reasoning tasks (math, coding, QA, social reasoning) and finds that existing models have weak collaborative reasoning skills due to overly agreeable or unproductive conversational behaviors. The lack of effective disagreement capability makes it difficult to leverage a collaborative reasoning framework, which (with existing models) offers no benefit over single-agent reasoning. This is an important finding that highlights a gap between what LLMs can do in isolation and what is required for real-world multi-agent or human-AI collaboration.

On the positive side, the paper contributes the Coral (Collaborative Reasoner) framework, which evaluates collaborative reasoning through multi-turn tasks requiring agents to disagree with incorrect answers, persuade each other, and ultimately reach a correct joint solution. Based on this framework, the authors introduce a self-play training method that generates synthetic multi-turn conversations to improve multi-turn collaborative reasoning skills. Experiments show that this approach substantially improves agreement correctness over chain-of-thought baselines, generalizes across model families, and even enhances single-agent performance. The paper also offers human evaluations that show more natural conversational behavior.

On the negative side, it is somewhat difficult to tease apart the benefit that comes from single-agent refinement and the benefit of true collaboration. Secondly, some of the social metrics such as persuasiveness and assertiveness seem rather subjective, and it was not clear to reviewers if and why they are needed. Another issue was that the computational cost and efficiency trade-offs were initially underexplored.

In the rebuttal and discussion, the authors provided various clarifications and new results, such as compute trade-off analysis showing comparable costs to CoT. The overall reviewer sentiment is that the work should be accepted, with reviewers noting that “all questions have been resolved,” and the consensus leaning toward acceptance.

Overall, I also think this paper should be accepted. It provides the first comprehensive framework for evaluating and improving LLM collaborative reasoning, with both practical infrastructure (Matrix) and conceptual advances (agreement-based evaluation, synthetic self-play training).
Decision: Accept (poster)