Title: Collaborative Reasoner: Self-Improving Social Agents with Synthetic Conversations
Venue: NeurIPS 2025
Accepted: yes
Keywords: collaborative reasoning

social agents

synthetic data

self-improvement
Abstract: With increasingly powerful large language models (LLMs) and LLM-based agents tackling an ever-growing list of tasks, we envision a future where numerous LLM agents work seamlessly with other AI agents and humans to solve complex problems and enhance daily life. To achieve these goals, LLM agents must develop collaborative skills such as effective persuasion, assertion and disagreement, which are often overlooked in the prevalent single-turn training and evaluation of LLMs. In this work, we present Collaborative Reasoner (Coral), a framework to evaluate and improve the collaborative reasoning abilities of language models. In particular, tasks and metrics in Coral necessitate agents to disagree with incorrect solutions, convince their partners of a correct solution, and ultimately agree as a team to commit to a final solution, all through a natural multi-turn conversation. Through comprehensive evaluation on six collaborative reasoning tasks covering domains of coding, math, scientific QA and social reasoning, we show that current models cannot effectively collaborate due to undesirable social behaviors, collapsing even on problems that they can solve singlehandedly. To improve the collaborative reasoning capabilities of LLMs, we propose a self-play method to generate synthetic multi-turn preference data and further train the language models to be better collaborators. Experiments with Llama-3.1, Ministral and Qwen-2.5 models show that our proposed self-improvement approach consistently outperforms finetuned chain-of-thought performance of the same base model, yielding gains up to 16.7% absolute. Human evaluations show that the models exhibit more effective disagreement and produce more natural conversations after training on our synthetic interaction data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the collaborative capabilities of large language model (LLM) agents, with a particular focus on effective communication and theory-of-mind reasoning. The authors demonstrate that current LLMs exhibit notable limitations in these areas. To address this, they propose Collaborative Reasoner (Coral), a novel framework designed to evaluate and enhance collaborative reasoning in LLMs. The paper introduces an effective synthetic data generation method that, when used for training, significantly boosts the model’s collaborative abilities and outperforms single-agent fine-tuning baselines across multiple tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:    \n1.\tAnalyzed the shortcomings of existing models in terms of collaboration capabilities    \n2.\tPropose a new self-improvement method to synthesize data and construct turn-based synthetic conversational data    \n3.\tBuild Matrix to support data generation at scale.    \n\n\nWeaknesses:    \n1.\tThe method of multi-agent collaboration for reasoning tasks has been proposed in previous work, such as CMD [1]. There is a lack of innovation and performance comparison with these existing methods.    \n2.\tThe paper lacks analysis and comparison of runtime efficiency. From the example conversations, we observe that even just two dialogue turns can result in lengthy context, compounded by the use of Belief Extractors, raising concerns about the computational cost of inference.     \n3.\tIn the synthetic data process or inference scenario of this experiment, there is a lack of specific evaluation and comparison of efficiency improvement using Matrix    \n\n[1] Wang Q, et al. Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key? Association for Computational Linguistics (2024).\n\nQUESTIONS:\n1.\tHow does the proposed framework compare to prior multi-agent collaboration approaches such as CMD [1] in terms of novelty and performance?\n\n2.\tCan the authors provide a runtime analysis of the proposed method, especially considering the cost of processing long contexts and running Belief Extractors?\n\n3.\tWhat efficiency gains are achieved by incorporating the Matrix component? Is there experimental evidence supporting its effectiveness in the given scenarios?\n\n[1] Wang Q, et al. Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key? Association for Computational Linguistics (2024).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes collaborative reasoning tasks where multiple agents have to solve a reasoning problem, with the success criterion being that they both agree on the final solution. The paper presents evaluations of models' collaborative abilities including persuasiveness and agreeableness. The paper finds that models do not collaborate well to consistently improve performance on reasoning tasks, and identify models being overly agreeable as a possible reason for this.\n\nTo address this, the paper proposes a self-play training mechanism where collaborative reasoning traces are sampled, and steps in the search process are used to create preference pairs so that preferred samples lead to the correct reasoning outcome (as judged by a belief extractor model that identifies a model's answer based on its reasoning). To generate preference pairs in a scalable way, the paper details a new tool for simulating interactions between multiple LLM agents.\n\nThe paper finds that the self-play training approach improves models' ability to collaborate, and results in increased reasoning performance across a number of benchmarks. Experiments also show that these gains translate out of domain to other datasets that evaluate similar abilities of models. Human evaluation of reasoning traces shows that trained models disagree more effectively.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* New multi-agent task with concrete functional definitions of notions of agreeableness and persuasiveness\n* Useful software tool for simulating interactions between multiple systems\n\nWeaknesses:\n* I may be missing something here, but it is not clear to me how these experiments evaluate collaboration between multiple distinct agents. If there are two agents A and B which both use the same underlying model generating a sequence of responses $(a_1, b_1, a_2, b_2, \\ldots)$, it is almost identical to the setup where A generates a sequence of responses $(a_1, a_2, a_3, a_4 \\ldots)$, since $b_1 = LLM_B(a_1) = LLM_A(a_1) = a_2$ and so on. Since the prompt also appears to be the same for both A and B, the only difference would be the sequence of user/assistant tags in the serialized messages. This is somewhat like self-refine, with the constraint of the two \"agents\" having to agree resembling self-consistency. This combination of the two methods could be a novel contribution in its own right, but it should be framed and analyzed as such, with relevant experiments to show differences from the base methods as single-agent reasoning approaches. However, the experiments in this paper don't really address how this evaluates collaboration.\n  * There are experiments where the training method is shown to generalize to other agents, but the only ones evaluated are versions of the same model without training. While these experiments are informative, really evaluating the collaborative abilities of models would have to go much further beyond given the broad scope adopted in the introduction (lines 30–32).\n  * The experiments do show improvements (especially with the proposed self-play training approach), but it's unclear whether this is fundamentally due to a difference between multiple distinct agents, or whether there is an explanation in terms of a single agent (more steps in CoT – standard CoT involves one step while the consistency condition here has the reasoning proceed for more steps – or better responses to mistakes in contexts, etc.)\n\nQUESTIONS:\n* Where are the training tasks used for the self-play training sourced from?\n  * Possibly related: \"to set the context for the collaborative reasoning capabilities for frontier models;\" what does stronger reasoning models \"setting the context\" mean?\n* Are all the models used for self-play training (A, B, and the answer extraction model) the same? Is a different model (maybe stronger) used to extract answers?\n* How does LLM-as-a-judge for answer extraction work? The appendix acknowledges that the extraction is not perfect. Does the extract model faithfully summarize each \"agent's\" reasoning? Does it confuse the agents? Is it able to correctly identify when no clear answer has been arrived at for one agent (and possibly only one of the two agents)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on evaluating if LLMs are able to collaborate with each other on various reasoning tasks: math, coding, etc. The authors propose an evaluation framework Coral whose main metric is \"agreement correctness\" -> Do the two models working together agree with each other and are they right.\n\nThe authors find LLMs aren't the best at collaborating and in order to improve this built a data generator named Matrix that generates synthetic conversations to finetune LLMs for collaboration and find this improves performance.\n\nThe contributions as I see it are:\n\n1) The synthetic dataset created to improve collaboration capabilities amongst LLMs. \n\n2) The Matrix framework which claims to be able to generate a large number of conversations across multiple LLMs in a scalable manner.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1) The topic of this paper is very interesting and I think important for the community to explore. It's also well-written and generally clear to follow.\n\n2) Both the synthetic dataset and the Matrix framework seem useful for the community.\n\n3) There are strong baselines the authors compare against to prove the usefulness of their dataset\n\nWeaknesses\n\n1) One detail that I didn't see (unless I missed it) was the number of conversations being generated and finetuned on. \n\n2) In addition to self-model collaboration I think cross-model collaboration would be important to put. I don't mean the Coral-trained model with its untrained counterpart (which was done and great to see) but I mean something like Claude collaborating with Gemini (which were the best base models).\n\nQUESTIONS:\nQuestions\n\n1) Did you try pairing up the best collaborators? Such as Claude 3.7 / Gemini 2.5 flash. As far as I can tell all the combinations consist of the same base model.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new collaborative reasoning approach that uses LLMs to self-chat when solving problems. The authors first investigate the behavior of current LLMs using this approach out-of-the-box and found that it did not consistently outperform chain-of-thought, potentially due to a lack of effective disagreement. They then propose a self-training approach involving tree-sampling, belief-filtering, and preference fine-tuning to synthesize data, filter data, and train models without human annotation to improve performance on this collaborative reasoning approach. Results show that after this training, the model demonstrates stronger performance than direct CoT SFT training on most test benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. This paper introduces a new and interesting way to solve reasoning tasks with LLMs by letting them self-chat to solve tasks collaboratively.\n2. Preliminary experiments are conducted to identify the lack of effective disagreement capability that hinders effective utilization of this approach. An effective self-training approach is then proposed to synthesize data, filter data, and train models without human annotation to improve performance on this collaborative reasoning approach.\n3. The proposed self-training approach is compared with different training methods like CoT+SFT and CoT+DPO, and various LLMs are evaluated. Results show that the self-training approach is more effective and also demonstrate generalization capability.\n4. A versatile backend for multi-agent inference is proposed, which supports models with different backends and enables fast inference.\n\n### Weaknesses\n1. The model trained with this self-training approach appears to be specifically optimized for collaborative reasoning. What is their performance when used in single-agent scenarios, which are more widely used for solving different tasks? Will this specific format training make it perform worse without the collaborative reasoning approach? This should be explored and discussed.\n2. The self-training approach incurs high costs for data synthesis and filtering. Although it is compared with CoT+SFT and CoT+DPO and shows better performance, the computational cost appears much higher. It is necessary to provide cost-performance analysis to gauge the trade-off.\n3. When measuring the belief state for each turn, there might exist intermediate steps/turns where the agent has not reached a final answer. How is the belief extracted in such cases? Additionally, is social behavior (persuasive and assertive) taken into consideration when filtering the data for model training, as the authors suggested this might be a reason that hinders effective utilization of this collaborative reasoning approach?\n\nQUESTIONS:\nPlease refer to the weakness part:\n1. The model trained with this self-training approach appears to be specifically optimized for collaborative reasoning. What is their performance when used in single-agent scenarios, which are more widely used for solving different tasks? Will this specific format training make it perform worse without the collaborative reasoning approach? \n2. Although it is compared with CoT+SFT and CoT+DPO and shows better performance, the computational cost appears much higher. It is necessary to provide cost-performance analysis to gauge the trade-off.\n3. When measuring the belief state for each turn, there might exist intermediate steps/turns where the agent has not reached a final answer. How is the belief extracted in such cases?\n4. Additionally, is social behavior (persuasive and assertive) taken into consideration when filtering the data for model training, as the authors suggested this might be a reason that hinders effective utilization of this collaborative reasoning approach?\n\nLIMITATIONS:\nThe authors have discussed several limitations of this work, including robustness of belief extraction and agreement measures. However, the limitation that the trained model is specifically designed for this collaborative reasoner approach and might not work well when used in single-agent scenarios should be explored and discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Collaborative Reasoner (Coral), a framework designed to evaluate and improve the collaborative reasoning abilities of LLMs in multi-agent, multi-turn conversational settings. The authors propose a training strategy using self-play to generate large-scale synthetic multi-turn conversations. These are then filtered and used for preference-based fine-tuning using DPO. The framework also introduces Matrix, a high-throughput system for scalable synthetic data generation. Experimental results across different tasks (math, coding, scientific QA, social reasoning, etc.) show that collaborative fine-tuning improves agreement correctness substantially over CoT baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper is well-structured and clearly written, making it easy to follow the motivation, methodology, and findings. Illustrative examples and figures (e.g., Fig. 1) are particularly helpful for understanding the proposed setting.\n\n2. The authors provide solid experimental details, code, and metrics across diverse reasoning domains (math, coding, QA, social reasoning). This level of transparency strengthens reproducibility and trust in the findings.\n\n3. The proposed Coral framework and the Matrix infrastructure are valuable in practice and could serve as foundational tools for future work on social LLM agents and collaborative multi-agent systems.\n\n**Weaknesses:**\n\n1. While the paper integrates several components (multi-turn conversations, synthetic data generation, preference tuning, scalable infrastructure), the contribution may appear as an aggregation of known ideas rather than a fundamentally novel concept. The core innovation could be better distilled and emphasized.\n\n2. The proposed social behavior metrics, such as persuasiveness and assertiveness, though intuitive, raise questions about their objectivity and necessity. Their practical utility remains somewhat vague, especially without strong connections to downstream applications or human-agent collaboration scenarios. \n\n3. The paper could benefit from a clearer explanation of what constitutes a \"fair\" comparison for the proposed framework of social agents.\n\nQUESTIONS:\n1. The paper introduces multiple components, including multi-turn collaborative reasoning, synthetic conversation data, and the Matrix infrastructure. Could the authors clarify the conceptual centerpiece of the paper? What is the single most important innovation they aim to establish?\n\n2. The social behavior metrics (e.g., persuasiveness, assertiveness) are interesting but may be considered subjective. Could the authors elaborate on the practical need for these specific metrics? What are some concrete application scenarios where these metrics make a tangible difference?\n\n3. Given the reliance on synthetic data, how well do the trained models generalize to real-world collaborative settings?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-08"}