Title: Learning “Partner-Aware” Collaborators in Multi-Party Collaboration
Venue: NeurIPS 2025
Accepted: yes
Keywords: alignment

LLM agents

collaboration
Abstract: Large Language Models (LLMs) are increasingly being deployed in agentic settings where they act as collaborators with humans. Therefore, it is increasingly important to be able to evaluate their abilities to collaborate effectively in multi-turn, multi-party tasks. In this paper, we build on the AI alignment and “safe interruptability” literature to offer novel theoretical insights on collaborative behavior between LLM-driven *collaborator agents* and an *intervention agent*. Our goal is to learn an ideal “partner-aware” collaborator that increases the group’s common-ground (CG)—alignment on task-relevant propositions—by intelligently collecting information provided in *interventions* by a partner agent. We show how LLM agents trained using standard RLHF and related approaches are naturally inclined to ignore possibly well-meaning interventions, which makes increasing group common ground non-trivial in this setting. We employ a two-player Modified-Action MDP to examine this suboptimal behavior of standard AI agents, and propose **Interruptible Collaborative Roleplayer (ICR)**—a novel “partner-aware” learning algorithm to train CG-optimal collaborators. Experiments on multiple collaborative task environments show that ICR, on average, is more capable of promoting successful CG convergence and exploring more diverse solutions in such tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multi-party collaboration with LLM-based agents, especially how to make a “partner-aware” collaborator that can effectively interact with an intervention agent to improve common ground. The authors propose a novel framework called Interruptible Collaborative Roleplayer  which incorporates a Modified-Action MDP to explicitly model intervention effects, and introduces a counterfactual KL regularization to train agents to distinguish helpful vs misleading interventions. They validate the method on two collaborative tasks, showing ICR outperforms baselines like PPO, DPO, and BC in both task accuracy and CG.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper address very interesting and important problem of collaboration between LLM agents. It is pretty interesting and meaningful.\n\n2. I think the theoretical analysis is quite strong. Especially Theorem 3.2 about suboptimality of standard preference-aligned methods in MAMDP is very insightful.\n\n3. The proposed counterfactual KL objective is simple but clever way to encourage partner-awareness, and is well motivated by the collaborator’s dilemma discussion.\n\n4. Experiments are quite thorough.\n\n**Weaknesses**\n\n1. One limitation is experiments only use GPT-4o as the fixed intervention agent. It is not clear how well ICR generalize to different partner styles or to human interventions.\n\n2. Some of the metrics like CG gain are a bit ad-hoc, it will be stronger to also include human evaluations or more user-centered metrics.\n\n3. Paper mainly focus on only two domains (Wason, Weights), which are quite structured.\n\nQUESTIONS:\n1. Do you think it is possible to extend this ICR method to multi-round human-AI interactions where humans provide free-form interventions? Will the counterfactual prompt still work?\n\n2. How sensitive is the ICR performance to quality of initial expert trajectories? If the GPT-4o roleplay data contains subtle flaws, will it propagate?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper provides a theoretical prove of why conventional RL algorithms (such as DPO, PPO) cannot obtain optimal policy when LLM agent act as collaborator with a partner. Then the author introduce a new term in PPO loss that optimize the KL divergence between the policy model and counterfactual policy model, which can quantify the effective of intervention.

The evaluation is solid and the paper writing is good.

STRENGTHS AND WEAKNESSES:
Strengths:
The paper theoretically proves the theoretical upper limit of current RL algorithms in optimizing LLM collaborative scenarios, and proposes a new method to overcome this difficulty. Its theory, experiments and results are all excellent.

Weaknesses:
The paper is somewhat obscure and casual in its writing.

QUESTIONS:
Major concerns:
1. Could you provide the detail training metrics of ICR, e.g., total KL loss, the KL of policy model and counterfactual policy model.

Minor concerns:
1. line 267 has a typo "??".

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Interruptible Collaborative Roleplayer (ICR), a novel learning algorithm designed to train \"partner-aware\" agents capable of effective coordination in multi-agent collaborative tasks. Built upon the Modified-Action Markov Decision Process (MAMDP) framework, ICR introduces two core mechanisms: a counterfactual invariance objective and KL-divergence regularization, allowing the agent to differentiate between helpful and unhelpful interventions. This results in behavior that maintains logical consistency and task alignment even in the presence of diverse, and potentially disruptive, interventions. The paper conducts experiments across several collaborative environments, including a variant of the Wason Selection Task (DeliData) and the Weights Task, demonstrating that ICR significantly improves both task performance and common ground convergence when compared to baseline models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper addresses an underexplored problem—how to train agents that can effectively collaborate in the presence of dynamic and potentially interruptive interventions. \n2. The proposed ICR framework is conceptually innovative, integrating a counterfactual invariance objective and KL-divergence regularization into the MAMDP formulation. \n3. Empirical validation across on two collaborative tasks demonstrates the effectiveness of ICR.\n\n\n**Weaknesses**\n\n1. The paper uses a fixed GPT-4o model as the intervention agent and does not explicitly model the intervener’s intentions or goals, which limits the understanding of how the interventions affect the collaboration.\n\n2. The proposed method involves computationally intensive processes such as counterfactual state generation and KL-divergence calculations, but the paper lacks a detailed complexity analysis.\n\n3. The paper does not thoroughly evaluate the generalization ability of the proposed method across unseen tasks or unseen types of interventions (see Line 47).\n\nQUESTIONS:\n1. Adding real-world examples of settings involving a collaborator agent and an intervention agent will help understanding the significance of the research problem.\n2. As mentioned in Lines 127–128 and 139–142, a key difficulty in the proposed setting is that the intervention agent may simultaneously convey harmful and useful information. Yet in the experiments, GPT-4o is used as a fixed intervention agent. How is this diversity of intervention quality simulated in practice?\n3. Consider adding single-agent baselines to separately evaluate the capabilities of the collaborator and the intervener. If the intervention agent significantly outperforms the collaborator, does that imply most interventions are helpful? \n4. The ICR framework involves computationally heavy steps, particularly counterfactual state inference and KL-divergence computation. While mitigation strategies are briefly mentioned (Lines 254–258), could the authors provide a more detailed analysis of the training costs associated with counterfactual training?\n\nLIMITATIONS:\nThe authors discuss their potential limitations in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper frames multi-agent dialogues as a two-player Modified-Action MDP where a collaborator agent needs to decide whether to adopt suggestions from another intervention agent. It shows that standard reinforcement learning and preference alignment algorithms are sub-optimal because they treat those suggestions as static context rather than causal actions. To overcome this, the authors introduce Interrubptible Collaborative Roleplayer (ICR), which is a “partner-aware” learning algorithm to train CG-optimal collaborators with a counterfactual-invariance KL regularizer. Empirically, an 8B parameter ICR model outperforms behavior cloning, preference-based RL, on-policy RL on two benchmarks, DeliData Wason Card Selection and the Weights Task, under both language-free and language-rich settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This work tackles a timely and practically important problem—training LLM collaborators that can decide when to accept or ignore a partner’s suggestions in multi-agent dialogue.\\\n- The theoretical framing of mapping collaborator-intervention dynamics to a MAPDP and proving the sub-optimality of Bellman-optimal policies addresses a critical optimality gap.\\\n- The ICR algorithm introduces a simple counterfactual-invariance KL regularizer that consistently yields relative accuracy gains and stronger common-ground metrics across two tasks and two communication modes.\\\n\nWeakness:\n- All experiments use one trained Llama-8B collaborator with a GPT-4o partner, leaving open whether the gains persist with other model families, larger models, or human collaborators.\\\n- In full-press evaluation, the evaluations given by LLM-as-a-judge is not validated by agreements with human annotators.\\\n- The benchmark tasks are synthetic and tightly-scoped, so it is unclear whether the approach scales to complex real-world collaborations where the task state is mutable, the dialogue is long, or the success metrics are fuzzy rather than a single ground-truth label.\n\nQUESTIONS:\n1. The writing of the theoretical framing can be improved by clearly introducing notations in Sections 3 and 4. Some notations are introduced without explanation in the main text. There’s a reference typo at line 267. \n2. All experiments pair with a GPT-4o as intervention agent. It would strengthen the insight by testing at least one substantially different intervention style such as a weaker/stronger LLM or real humans. \n3. The LLM-as-a-judge evaluation should be validated to ensure it’s not biased. The authors may provide inter-annotator agreement between the model and human annotators on a sample or show that the ranking of systems is stable across different conditions. \n4. The counterfactual state is introduced by a single textual prefix (“The intervention agent’s suggestion will definitely not improve your performance…”). Model behavior may be sensitive to the exact wording of this cue. The authors could verify the effectiveness of the counterfactual intervention by (i) replacing the prefix with several semantically equivalent rephrasings and (ii) evaluating a null-intervention variant that simply removes the collaborator's utterance.\n\nLIMITATIONS:\nUnder the Broader Impact section, the authors explicitly state that they “do not clearly see any immediate negative impacts”, but partner-aware LLMs could be misused for collusive behaviour or persuasive manipulation of human teammates. A brief discussion of such risks and mitigation strategies, such as usage policies, red-team evaluations, or refusal triggers, would strengthen the discussion of potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper reframes multi-agent collaboration as a Modified-Action MDP where a teammate (the “intervention agent”) can inject suggestions into the state. It shows that standard RLHF/DPO learners are sub-optimal in this setting and proposes Interruptible Collaborative Roleplayer (ICR)—a lightweight PPO variant that adds a counterfactual-invariance KL penalty.  At each turn the collaborator is run twice: once on the real dialogue and once on a counterfactual prompt that asserts “the intervention will definitely not improve your performance.”  Minimising the KL between those two outputs—unless the suggestion actually raises task reward—teaches the model to ignore unhelpful advice while still adopting good ideas.  The authors provide a theoretical bound linking residual KL to sub-optimality and demonstrate large empirical gains on two synthetic reasoning games, with ICR outperforming the best baseline by 47 % on Weights (symbolic reasoning) and 14 % on common-ground build-up.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The method operationalises “safe interruptibility” by letting an agent disregard misleading advice while still learning from good tips.  Because it uses only prompt engineering and KL terms, it scales to larger models without exotic infrastructure with just a second forward pass plus a KL term; it can drop into any PPO-style fine-tuning pipeline.\n- Clear theoretical framing.  The Modified-Action MDP formalism and the bound pinpoint why naïve RLHF fails and how counterfactual KL closes the gap.\n\nWeaknesses:\n- The problem is somewhat niche.\n- The counterfactual world is invoked by a single sentence (“the intervention will not improve your performance”).  LLMs can still leak information from the suggestion, so invariance may be imperfect; there is no analysis of how well the prompt enforces LLMs to ignore the suggestion.\n- Missing simple baselines: The study doubles inference cost (two forward passes) yet omits a data-augmentation PPO baseline that would use the same budget (i.e. just add another example with the counterfactual prompt).  Likewise, there is a prompt-only baseline that praises every suggestion (PSO-INTENT) but no symmetric “be sceptical” prompt-only variant, leaving open whether part of ICR’s gain comes from the extra training rather than the objective itself.\n\nQUESTIONS:\n- Did you measure token-level differences between factual and counterfactual prompts across a variety of linguistic phrasings or noisy interventions?\n- Have you tried alternative counterfactual constructions (e.g., masking the suggestion entirely) and, if so, how do results change?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-12"}