Don’t lie to your friends: Learning what you know from collaborative self-play
Authors: Jacob Eisenstein
Reza Aghajani
Adam Fisch
Dheeru Dua
Fantine Huot
Mirella Lapata
Vicky Zayats
Jonathan Berant
Venue: COLM 2025 (2025)
Keywords: self-play
calibration
tool use
multiagent
TL;DR: Task-level reward in a multi-agent collaborative game can teach the agents about calibration and selective tool use

Abstract: To be helpful assistants, AI agents must be aware of their own capabilities and limitations. This includes knowing when to answer from parametric knowledge versus using tools, when to trust tool outputs, and when to abstain or hedge. Such capabilities are hard to teach through supervised fine-tuning because they require constructing examples that reflect the agent's specific capabilities. We therefore propose a radically new approach to teaching agents what they know: \emph{collaborative self-play}. We construct multi-agent collaborations in which the group is rewarded for collectively arriving at correct answers. The desired meta-knowledge emerges from the incentives built into the structure of the interaction. We focus on small societies of agents that have access to heterogeneous tools (corpus-specific retrieval), and therefore must collaborate to maximize their success with minimal effort. Experiments show that group-level rewards for multi-agent communities can induce policies that \emph{transfer} to improve tool use and selective prediction in single-agent scenarios.

Published as a conference paper at COLM 2025
Don't lie to your friends: Learning what you know from collaborative self-play
Jacob Eisenstein
Reza Aghajani
Adam Fisch
Dheeru Dua
Fantine Huot
Mirella Lapata
Vicky Zayats
Jonathan Berant
Google DeepMind jeisensteinggoogle.com
Abstract
To be helpful assistants, AT agents must be aware of their capabilities and limitations. This includes knowing when to answer from parametric knowledge versus using tools, when to trust tool outputs, and when to abstain or hedge. Such capabilities are hard to teach through supervised fine-tuning because they require constructing examples that reflect the agent's s specific capabilities. We therefore propose radically new approach teaching agents what they know: collaboratize self-play. We construct multi-agent collaborations in which the group is rewarded for collectively arriving at correct answers. The desired meta-knowledge emerges from the incentives built into the structure of the interaction. We focus small societies of agents that have access to heterogeneous tools (corpus-specific retrieval), and therefore must collaborate to maximize their success with minimal effort. Experiments show that group-level rewards for multi- agent communities can induce policies that trausfer to improve tool use and selective prediction in single-agent scenarios.
1 Introduction
While conversational assistants based language models (LMs) are having unprecedented success (Gemini Team, 2024; Llama Team, 2024; OpenAI, 2024; DeepSeek-AI, 2025), there is growing evidence that skills that are crucial for successful human-AT collaboration are still lacking. Compared with human speakers, LMs perform fewer grounding actions, such as asking clarification questions (Shaikh et al., 2024); they do not express uncertainty faithfully (Zhou et al., 2024; Yona et al., 2024; Stengel-Eskin et al., 2024); and they often use outputs from external tools incorrectly (Yoran al., 2024; Wu et al., 2024).
People build and use such skills when communicating in natural language to achieve collaborative goals (Grice, 1975). Such collaborations are more likely to be successful when participants are truthful about their knowledge, ask for help when needed, and use external resources when their knowledge is limited. This requires meta-cognitive capabilities such as estimating and communicating uncertainty, and learning to evaluate the reliability of provided information. But post-training procedures for endowing these skills onto LMs are devoid of this social, goal-oriented signal, relying instead on supervised fine-tuning curated examples (Zhang et al., 2023; Stengel- -skin- et al., 2024; Li et al., 2023; Yoran et al., 2024). This also entails that data collection needs to be carried out per skill.
In this work, we propose collaborative self-play (CSP) as a mechanism to teach language models to be more helpful, by constructing multi-agent environments where accomplishing goals relies on learning to be an efficient and effective communicator. Specifically, a small
"Core contributors: JE, RA, AF, JB.

---

Published as a conference paper at COLM 2025
Figure/ 1: Collaborative self-play for question answering. A small society of agents is asked to answer a question. Suc- cess requires cooperation: agents must use their unique re- sources and share not only information but also calibrated expressions of confidence. Here, Agent A receives the initial query from the user D), and asks for help from Agent B and C (2, who search 3 and then respond to Agent A b after which Agent A chooses final response C. Agent B does not have access to a relevant retriever, and therefore marks its prediction uncertain.
Frect SCF TMM
① PLSK
BASK
AHENGE b
#A02WEE2 O
C
① VSEUACC
a ASBANCI
Wikieeda
PuoMed
society of LMs, each with access to a different retrieval tool, is presented with a set of factoid questions. As shown in Figure 1, answering correctly requires the agents to know (i) when their parametric knowledge is reliable, (ii) when to express uncertainty to avoid misleading others, and (iii) how to appropriately use their retrieval tools. We hypothesize that training towards this multi-agent setting should encourage better tool use and hedging, which are useful skills even in single-agent scenarios,
To test this hypothesis, we instantiate the game illustrated in Figure and fine-tune language model on rollouts from the above multi-agent environment using Reinforced Self-Training (ReST; Gulcehre al., 2023). Specifically, we sample multi-agent interactions inttraactonsnsss from the environment and fine-tune on the rollouts that obtain the highest reward. Crucially, the reward defined only in terms of task completion and task effort; we want to achieve high accuracy while avoiding unnecessary tool calls. The actions of individual agents in the rollout are not prescribed or otherwise explicitly rewarded. All agents share parameters (they differ by their prompts and tools), and thus only a single model is trained and evaluated in a single-agent setup at test time. Empirically, on BioASQ (Krithara et al., 2023) and PopQA (Mallen et al., 2023), two factoid question answering benchmarks that benefit from corpus-specific retrieval, agents trained using CSP become get better at determining precisely when to search, while learning to expressing uncertainty in more calibrated manner. This advantage carries over to two unseen benchmarks.
In addition to these experiments, we provide a Game-theoree analysis that sheds light the conditions under which the optimal strategy for collaborative self-play aligns with the goals of calibrated confidence and tool use, providing concrete guidance for the design of collaborative self-play learning scenarios. To summarize the paper's 's contributions:
1. We propose a general framework based on collaborative self-play for teaching conversa- tional LM-based agents to be better collaborators when solving tasks.
In the context of aetrieallaugmmented QA, we empirically demonstrate that this frame- work teaches agents to be better selective tool users and to better express their uncertainty.
3. We provide game-theoretic analysis that identifies the conditions under which collabo- rative self-play can induce efficient tool use and calibrated question answering.
2 Related work
Multi-agent LLM systems. A growing body of work considers systems of multiple lan- guage models (for surveys, see Zhuge et al., 2023; Guo et al., 2024), particularly for inference- time deliberation and debate (Du et al., 2023; Chan et al., 2023; Chen et al., 2023; Khan et al., 2024). In contrast, we focus on the impact of multi-agent coordination at the training stage. Along these lines is work in which training examples for reasoning are distilled from multi-agent dialogues (Chen et al., 2024; Subramaniam et al., 2025). We do not work towards specific mdd-tas reasoning but rather we use multi-agent collaboration as a training environment to learn skills that should improve collaboration with humans.
Social learning. Our approach can be viewed in the context of social learning (Bandura, 1977), which considers the role of socialization in improving predictive performance and
2

---

Published as a conference paper at COLM 2025
An example multi-agent rollout, showing the #ASK, #SEARCH, #HEDGE, and #ANSWER actions, as well as how evidence is incorporated into the prompt. Agent A has no tools, so it asks the other agents for help. Agent B believes its retrieval tool can
Figure 2:
Q: What prinary iren the littell Tower?
*ekkee What wae the primary source of iron for the Biffel...
Q: What vas the prinary cource of iron for the Bitfel Tower? AGENT B:
SEARC What vas the painary source of Lron For the
Q: What the primary sourcw of iron for the Riffel Touer? RETRIEVAL: Forges Porpey near Hanny supplsed iron for...
help, it performs a search, which
returns useful information. Agent C
AGENT *ARSSERt Forges Poapey
does not have tool that it believes is
What prinary
iron
Biffel Towex?
applicable. It falls back to parametric knowledge, but marks its answer
AGENT c: SE The iron was Impozteed froo Australia
#HEDGE. Agent A receives the two an- swers, and selects the more confident one, which turns out to be correct.
Q: What the primary sourcw of iron for the Riffel Touer?
FAIEND Forgee de Paxpey
PRIEND_HFDOR: Thee fron wan ieported fron Australia
**ANSMERe* Forges Pompey
adaptability, where agents can learn new behaviors by observing and imitating others. collaboration among specialized For example, Yao et al. (2024) explored how multi-agent Clens experts can lead to mutual improvement on image classification tasks, where agents learn from each other to improve their predictions on image classes outside their respective domains of expertise through a group-wide knowledge distillation loss. Similarly, in a LM setting, Mohtashami et al. (2023) use teacher agents to compose instructions or exemplars for student agents, and Wang et al. (2024) use an Al judge to assign reward to the student's social performance. In contrast, we do not designate any agent teacher or judge, but rather construct mechanism in which pro-social behavior is required to get a high reward.
Related ideas are explored in the broader context of multi-agent reinforcement learning (Al- brecht et al., 2024), particularly with respect to learning to communicate (Foerster et al., 2016; Sukhbaatar et al., 2016). Here, however, our goal not to learn an effective multi-agent sys- tem, but rather to use the multi-agent context to learn strong single-agent policies - similar to the competitive games underlying GANs and adversarial domain adaptation (Goodfel- low et al., 2014; Ganin et al., 2016), but here in a cooperative setting that is furthermore grounded in conversational, natural language. In this respect we deviate from games that are cooperative but monglinguiitic (e.g., Bard et al., 2020) or which are linguistic but non- cooperative (e.g., Bakhtin et al., 2022). An additional distinction, which extends to games that are linguistic and at least partially cooperative (e.g., Liao et al., 2024; Xu et al., 2023), is that our goal is not to train language models to succeed in games, but to use games to teach more widely-applicable conversational skills.
Calibration and confidence. Language model confidence estimation is an active topic (e.g., Kadavath et al., 2022; Mielke et al., 2022; Yang et al., 2023; Lin et al., 2022; Stengel- et al., 2024; Quach et al., 2024; Mohri & Hashimoto, 2024), and prior work includes inference-time multi-agent debate (Du et al., 2023; Feng et al., 2024). While some papers claim that LLMs can be prompted to reveal their own uncertainty (Kadavath et al., 2022; Tian et al., 2023, e.g.)), other work raises serious doubts (e.g., Xiong et al., 2023; Kapoor et al., 2024), and in any case, these findings are generally restricted to parametric knowledge rather than the retrieval- augmented models that we consider here. More broadly, the decision to answer or abstain should be driven by two features that are often hard to know in advance: the likelihood that the agent can arrive at the correct answer, and the consequences of abstention vs being incor- rect. This view is aligned with Brengel-Eskin et al. (2024), who are also motivated by linguis- tic pragmatics. But rather than stipulating which action is pragmatically correct in a given context, we create self-play scenario in which pragmatic reasoning emerges in the solution to the collaborative game. Uncertainty is often decomposed into epistemic (due to lack of knowledge) and aleatoric (due to irreducible randomness; Hüllermeier & Waegeman, 2021). While the subtleties of this distinction are orthogonal to our contribution, we note that our fo- cus is on epistemic uncertainty with regard to (a) the model parameters, and (b) tool outputs, and not, for example, uncertainty about the intent behind the user's 's query (Min et al., 2020).
3

---

Published as a conference paper at COLM 2025
Game-theoretic dialogue. Prior work offers game-theoretic accounts of various dialogue phenomena, such as implicature (Parikh, 1991; Franke, 2009). We also build on game theory by analyzing the equilibria of a game-theoretic model of collaborative self-play. However, our goal is not to explain features of human conversation but to characterize the conditions under which our proposed mechanism induces normatively-desirable behavior (i.e,, calibrated expressions of confidence) from rational agents.
Social supervision from collaborative self-play
This section describes our framework for obtaining social supervision from collaborative self-play. By social supervision, we mean that the training signal emerges from the efficacy of interactions between agents, rather than from direct annotations of reference outputs of individual agents at each step of the rollout. The restriction to collaborative self-play is entailed by the use of single reward for the entire group of agents that all share the same model parameters, rather than distinct rewards and models per agent: all agents are playing together with the same objective to optimize a joint set of parameters.
Multi-agent orchestration. For each initial prompt or query X1 we define rollout as ((a), X1, ), (az, x2, X2).,...... (ar, xT, yr)), with a, E A indicating the active agent at turn and yt E V* indicating its input and output respectively. The initial agent A1 is selected from some initialization policy, and each subsequent agent is then determined by an orchestrator, based on the history of the rollout. The rollout continues until either we reach a terminal state, we reach f = T, or the rollout exhausts some maximum effort budget, which can be defined in terms of the cost of each action. The terminal state should return an output that can be scored.
The job of the orchestrator is to determine who speaks next: at each step f T, the orchestrator passes control to agent a, 11 (which can potentially be the same as at), and issues a prompt X1+1- The new prompt may incorporate tool outputs and communication from other agents. For example, if agent a uses a retrieval tool, the orchestrator will add the retrieval results the prompt 1 as evidence, while setting at+1 = a; If agent at poses a question, the orchestrator decides on new at+1 o at and constructs prompt x;-1 that includes this question. The orchestrator can then remember that the question was posed by a, and may then return control to this agent after a is finished - or it may pass control to another agent before returning to a, There are many possible orchestrators, each leading to different patterns of multi-agent interactions: for example, an orchestration policy may allow each agent to broadcast communication to all the others in an egalitarian society, or it may require that all communication flow through single hub node.
Actions and arguments. We now describe an instantiation of collaborative self-play that is designed for training agents to organically learn to use tools and give outputs. At each step, each agent's 'ss output is required to be an [action] and an [ar (argument]], where [action] is limited to the set ( (#ANSWER, #ASK, #HEDGE) } plus actions for each tool avail- able to the agent. This is relatively constrained compared to other implementations of tool use, in which the tool call can appear at any point in the generation (Schick et al., 2023; Mialon et al., 2023); such extensions can be considered in future work. After the [action] and [argument] are sampled from the policy, they are appended to the history, and the orchestrator is called to identify the next agent and its prompt.
When the sampled [action] is a tool, the orchestrator maintains control with the current agent and (potentially) adds evidence to the prompt. We focus on single type of #SEARCH tool, which obtains a set of near-neighbor retrievals and appends them to the prompt with the preceding tag RETRIEVAL, applied to other tools, such as calculators or code interpreters. The remaining actions (i.e., (#ANSWER, #ASK, #HEDGE)) )) pass control to other agents, as described below.
shown in Figure 2. A similar approach could be
Inter-agent communication and orchestration. Inter-agent communication is enabled by the actions #ASK, #ANSWER, and #HEDGE. If agent a; issues the #ASK action, then the orchestrator pushes 4, onto a stack and passes control to a new agent #;: * ap The argument to the ask action is appended to the prompt x; 1, as shown in Figure 2. Agents
4

---

Published as a conference paper at COLM 2025
Algorithm 1 Action Answer Reinforced Self-Training
Require: Initial model .Mo, reward threshold T, number of steps n,, and rollouts per query nr, number of ReST iterations T
Ensure: Trained model Mast
for f = 1 to I' da
Step 0: Define an empty training set (
query do
Step la: Generate rollouts 1 by applying model M,- to in multi-agent orchestration.
Step 1b: Partition the rollouts into action sequences (see $ 3), and let s,. indicate the action sequence with the maximum
mean reward among compatible rollouts. if the mean reward of S,, > r then
Step Ic: Among rollouts compatible with s.,, select the one with the highest reward, Pt. = arg max:(ny i Pig s.x). Step Id: Convert Pi.. into turn-level training examples and add them to D.
end
end for
Step 2: Train M, on training split of D for nt, steps.
Step 3: Select checkpoint with the smallest held-out log-likelihood dev split D.
end for return MM
relinquish control by issuing the actions #ANSWER and #HEDGE. The orchestrator can then return control to the agent at the top of the ask stack, or, in "broadcast" setting, can pass control to another agent. When control returns to At, its prompt is updated with the string arguments provided by the answering agent(s), with the preceding tag FRIEND.ANSWER or FRIEND_HEDGE (see Figure 2). the ask stack is empty when control-relinquishiin action is issued, the rollout has entered a terminal state.
We implement a broadcast version of the #ASK action. The action causes the ask stack to be updated S < a, o S; 1- Control then passes to each agent reachable from St- Conversely, a, issues a control-relinguishing action with ask stack S a o S', then control passes to agent with ask stack S, S'. We reach a terminal state when the initial agent d1 relin- quishes control and the stack ST is empty. Reward is then computed as effort-penalized task performance, i.e., we would like to return a correct answer, while performing as few tool calls as possible. Specifically, r(hr)= = Score(yr,y's - 5 Effort(hr), where Score is a metric like token-level Fr, effort is the number of search calls in the rollout, and is a prrramerrrrreer..
Training. Given a society, an orchestrator, and a reward function, we hope to learn language model policies that yield high expected reward. We apply Reinforced Self-Training (ReST), extension of supervised learning that is commonly used in post-training (e.g,, Gulcehre et al., 2023; Zhang et al., 2024). The idea is to generate rollouts, score them, train the good ones, and then iterate with the newly trained policy. However, the application of ReST to collaborative self-play requires adaptation to avoid fine-tuning on lucky guesses that do not lead to generally effective policies.
Specifically, we will focus first on identifying sequences of actions that reliably lead to good rewards, and then select the best rollout compatible with such action sequences (see Algorithm 1). Recall that a rollout is defined as p
Xt, Xt. )) with yt+ [z
now composed of an action 2, E Z and an optional argument 6,. An action sequence s = ($,, is a sequence of actions, s E Z (such as #ANSWER or #SEARCH). If for given rollout p we have 2 = 5; for all f, then we write p - s to indicate that p is compatible with s. can be coarsened into a set of action sequences {s), ..). Each action sequence is scored by the average reward of its compatible rollouts, and the best action sequence s* is selected. If this average reward is above some threshold, we then choose the best individual rollout 4" + s", and train on each step in this rollout. In ReST, training is done iteratively, and we run it for T epochs, where in each epoch rollouts are sampled from the mostly recently trained model.
Now a set of rollouts, {p1.P2....
Importantly, while we have a multi-agent environment, all agents share the same parameters and only differ in their tool access and input prompt, and thus we train single model. This is necessary because our goal is to use the multi-agent environment to elicit training data that will improve the model, but then have a single agent at test time.
5

---

Published as a conference paper at COLM 2025
Z2 =
Z2 2266 G
Table 1: Payoff matrix for the game de- scribed in Š 4, where each cell shows the reward for the two players, (2),
Zi S Z1 = G
(a- 8, 6)
(a1 8, x1) (B
(42, 8)
Analysis: An information-provision game
Our core intution is that the group can achieve high reward only by learning to calibrate its use of the #ANSWER and #HEDGE actions. To clarify the necessary conditions for this to work, we analyze the pure strategy equilibria of simplified two-player game, where an asker agent issues a question to two players with access to different retrieval tools, and each player has the choice between actions S
and G -heess-then- Agent generates the correct answer with probability a; when playing action Z; = S, and when playing action Z; G. When one player plays Z; S and the other player plays 2, =G the asker chooses the more confident answer (from player i); otherwise the asker picks one of the two answers at random. Both players receive a reward of 1 the question is answered correctly and zero otherwise; the search tool incurs cost of 5 > O. Table 1 shows the expected rewards under the joint distribution associated with the correctness probabilities
1,, 42, f and the random choice made by the asker, with & (A1 + 22). game to find the parametrizations under which there is a single equilibrium corresponding to our normative expectation of the agents' behavior under full information: i.e,, the use of the action should indicate that the agent is especially likely to provide a correct answer.
! We analyze this
Theorem 4.1. Assume without loss of generality that a1 > a > 02, {2), # B 6+6 6, and 41 * a2 + 25. Then the game in Table 1 has a unique pure stralegy equilibrium and only if at least one of two conditions is met: > a2 + 28 or 02 < & + 8. Specifically, the unique equilibrium is (S,G) if a1 > B++ and (G, G) otherwise. If neither condition is met, then (S, G) and (G, are both pure strategy equilibria.
The proof is in Appendix C, which also includes generalization to n-player games. The strategy space is illustrated in Figure 6 in Appendix D. The analysis sheds light on what the
game teach, and provides guidance game to achieve the desired results. We are most interested in settings where (S, G,G) is the unique equilibrium,
how to design the
that the players' actions informative of their relative accuracies. This will occur if the players have search tools that are very effective on some questions, (x) > B(x), with (x) and ((x) indicating the probability of answering correctly on given question x, and when the tools are complementary in the sense that a (x) > f(x)
then 41 (x) >> (x),,, so that it is clear which tool to use. If a2(x) = f(x) then the agents will learn to search precisely when it is better than guessing. In the next section we describe game that approximately meets these criteria.
Experimental setup
We evaluate whether collaborative self-play can teach agents when (a) they can answer confidently from parametric knowledge; (b) they have tool that is likely to yield helpful evidence; or (c) they cannot answer confidently and must therefore hedge.
Evaluation details. Our evaluation focuses practical special case of collaborative self-play (defined in S 3), using the Gemma2-9b base model (Gemma Team et al., 2024).
Helper agents: WIKI-BM25, which can search Wikipedia using BM-25 (Robertson & Zaragoza, 2009); PUBMED-GECKO, which can search PubMed abstracts using dense embeddings from Gecko (Lee et al., 2024). When using the #SEARCH tool, the agents must pass along the query verbatim. Helper agents cannot issue an #ASK action.
This game similar to the classical example of the provision of a public good (see, e.g,, Olson Jr, 1971; Osborne, 2004, Chapter 2), where the good is the information provided by the search tool. 2These conditional probability models are assumed to be learned by minimizing regret.

---

Published as a conference paper at COLM 2025
0.8
BioASQ
PopQA
Figure Mean Fj of the prompted model for each helper agent. The tool-augmented agents have comple- mentary knowledge, satisfying the conditions for collaborative self-play calibration described in $
/o search with search
0.6
0.51
4
0.4
0.37
0.57
0.2
0.16
0.13
0.16
pubmed
wiki
pubmed
wiki
Communication An asker agent that has no tool access passes query to the helper agents by issuing an #ASK action; their responses enter the evidence section of the asker's prompt as either #FRIEND_ANSNE or #FRIEND HEDGE. For #ASK and #SEARCH the argument is copied from the last turn, while for #ANSWER and #HEDGE is generated by the model.
Data: Short-answer questions from BioASQ (Krithara al., 2023) and PopQA (Mallen et al., 2023). BioASQ is manually-curated corpus of biomedical questions along with gold answers. PopQA is an open-domain question answering dataset with questions on entities of various levels of popularity from Wikipedia. We choose these two benchmarks guided by the analysis from 4, as they have long-tail questions that are less likely to be in the model's parameteric knowledge, and thus the advantage of tool use should be considerable for some questions.?
Out-of-distribution evaluation: We evaluate on the dev sets of two additional bench- marks: Natural Questions (NQ; Kwiatkowski al., 2019) and EntityQuestions (EntQ; Sciavolino et al., 2021). Both contain questions answerable from Wikipedia. However, NQ questions focus on common entities, so are likely answerable from parametric knowlege, while EntQ focuses
tail knowledge that is likely to require retrieval.
Reward: Effort-penalized F1 (comparing the set of tokens in the gold and predicted answers), with a penalty of d for each tool use call. The penalty serve as a tie-breaker between rollouts with the same F1.
to low value to
An example rollout in this setting is shown in Figure 2; the full prompts are in Appendix A.
Why does this implementation of collaborative self-play teach the capabilities enumerated above? As argued in $ 4, this setup will incentivize calibration of the #ANSWER and #HEDGE actions if the agents have complementary knowledge, because using the correct action is necessary to enable the asker to choose effectively between conflicting answers provided by the helpers. As shown in Figure 3, the agents do indeed have complementary knowledge, with significant gaps in F1 across the two datasets. Furthermore, the prompted Gemma2-9h model attends to the #ANSWER/ #ANSWER/#HEEGG #HEDGE distinction: in 80% of rollouts where receives #ANSWER and a #HEDGE from its helper agents, it passes along the more confident answer. There also an incentive for efficient tool use: if the agents can obtain an accurate answer parametrically, they will achieve the highest reward by doing so. They should make tool calls only if they are likely yield good evidence, or they will incur an effort penalty without improving their likelihood of correctness.
Models and baselines. We compare the following supervision strategies:
In-context learning (ICL). We use the base gemma2-9B model, with one-shot-per-action prompt shown in Appendix A.
Collaborative self-play ReST (CSP). We generate rollouts from the three-agent society and train as in Algorithm 1, with 0.1, n1 2000, n1 = 32.4 Hedging and search are learned only from inter-agent communication.
Deanonymized CSP ReST (CSP-DeAnon). Similar to CSP, but the asker agent knows the identities of the helpers. As an example, in the last turn of Figure we replace FRIEND_ANSWER with ANSENR (wiki), and FRIEND with SEIEND_HEDGS
'In PopQA, we filter out questions with higher than median annotated popularity. 4pilot experiments with 0.5, n, (1000,5000), n, n (24, 100) yielded broadly similar results.
7

---

Published as a conference paper at COLM 2025
task-level Fj 1
search rate
setting
agent
bioasq
popqa
nq
entq
bioasq
popqa
nq
entq
ICL
PUBMED
0.577
0.112
0.267
0.212
0.984
0.977
0,967
0.980
WIKI
0.423
0.338
0.407
0.521
0.972
0.995
0.978
0.994
CSP
PUBMED
0.568
0.235
0.377
0.329
0.228
0.191
0.104
0.126
WIKI
0.538
0.321
0.404
0.434
0.182
0.480
0.205
0.362
CSP-
PUBMED
0.576
0.234
0.386
0.349
0.251
0.091
0.059
0.052
DeAnon
WIKI
0.537
0.325
0.391
0.449
0.140
0.499
0.190
0.344
Act. Sup.
PUBMED
0.568
0.243
0.375
0.332
0.180
0.039
0.063
0.060
WIKI
0.542
0.293
0.395
0.373
0.136
0.188
0.092
0.131
Oracle
PUBMED
0.690
0.256
0,428
0.384
0.294
0.048
0.079
0.075
WIKI
0.606
0.427
0.537
0.619
0.175
0.248
0.209
0.379
Table 2: Task performance and effort. On in-distribution data (bioasq and popqa), col- laborative self-play (CSP) achieves higher or similar task-level F compared to in-context learning (ICL), despite using 2-5x fewer search calls. The action-supervised method is able to further reduce search rates by training directly on calibration-based objective, but does not improve the task-level Fl significantly. On EntQ, an OOD dataset, Action Supervision searches too little, reducing task performance.
(pubmed). This means that the asker can learn which helper is likely to give correct answer for a particular question (ignoring any of the helper's 's confidence markers). This reduces the incentive for the helper agents to learn hedge in CSP-DeAnon, which we predict will reduce the calibration of P ##ANSWER).
Action supervision. We generate rollouts from the prompted base model, with no inter- agent communication. We then construct silver label for the optimal action according to calibration-based objective. Specifically, for each question, we take the max-reward rollout that ended with #ANSWER. If the F > 0.5, we train on the steps of this rollout; other- wise, we take the highest-reward rollout that ended with #HEDGE, and train on its actions if F1 < 0.5, excluding the answer. Action supervision directly maps #ANSWER to high-Fi rollouts and #HEDGE to low-F, rollouts, while CSP-ReST relies on task completion alone.
The evaluation focuses on a single-agent deployment scenario with #ASK disabled, testing the ability of multi-agent supervision to transfer to single-agent policies. Thus, we run the model in single-agent setup, where the agent chooses between #ANSWER, #HEDGE, and #SEARCH. The goal is to use #ANSWER when it is possible to answer from parametric knowledge, #SEARCH when it is likely to be helpful, and #HEDGE when unsure.
Action Supervision is more similar to standard calibration strategies (e.g., Lin et al., 2022; Zhang et al., 2023), where teach the model to be confident when it is correct and unsure when is wrong. Its training is closely aligned with the evaluation procedure (see & 6), heuristically splitting #ANSWER vs #HEDGE on Fi 0.5. Conversely, CSP does not prescribe any specific actions, relying on the emergent training signal from task completion alone. Thus Action Supervision can be regarded as an upper bound on the calibration performance attainable by CSP on these tasks.
Inference cost to generate training data. Generating the training data required 64,000 rollouts per round of ReST training: 1000 queries per dataset, 32 rollouts per query, two datasets), with three five inference calls per rollout. Empirically, each ReST epoch produced < 2M input tokens. Over three epochs and with an approximate multiplier of 1.3 subword tokens per whitespace token, we estimate the total cost of replicating the training data generation at less than $150 US, based on third-party prices for gemma2-9b inference."
-eeiite output tokens, from < 180M whitespace-delimited
htttass//grro..co///rrcicc/////s US$0.20 per million tokens on August 2025.
8

---

Published as a conference paper at COLM 2025
(a) in-distribution
(b) out-of-distriuttio
Calibration of P(#SEARCH). For each method, held-out queries are sorted by P(#SEARCH); in 'shuffled' they are randomly shuffled. For the top x% of queries, retrieval is used; for the remainder, the model must respond parametrically. Well-calibrated tool users can achieve significant boosts in F from even rare use of #SEARCH.
Figure 4:
0.4
0.
0.35
0.3
Act. Sup. ICL
CSP shuffled
CSP-DeAnon
0.5
0.5
Fraction of searches
Fraction of searches
Results
We evaluate the learned agents on three dimensions: (1) whether they can reliably obtain accurate answers at minimal cost; (2) whether their use of the search tool is calibrated to its helpfulness; (3) whether their use of #ANSWER vs #HEDGE is calibrated to correctness. Here report results after three ReST epochs; for per-epoch results see Appendix E.
Task performance. To measure task performance, we run each helper agent (PubMED- Gecko and Wiki-BM25) on held-out questions from each dataset, and report token-level F.. Results are shown in Table 2. The most prominent distinction between the methods is that in-context learning (ICL) nearly always searches, while the three supervised methods search less than half of the time in most settings - usually much less. As a result, ICL yields poor performance when applied to an agent whose tool is not suited for the dataset (e.g.,
PubMed/ PubMed/PopQA) PA)QAA, with Fi gaps of more than 10 points from the other techniques. Among the supervised methods, Fi results are broadly similar, with Action Supervision searching at the lowest rate due to its calibration-based objective. However, this low search rate hurts Action Supervision on the EntQ dataset: on Wiki/Ento it trails CSP by 6.1 Fi. Conversely, ICL performs especially well on this dataset, because of its high search rate (> 99%). The deanonymization ablation has relatively little impact on task performance, except that CSP-DeAnon is able to reduce the search rate in mismatched settings.
The table also shows the performance exactly those instances where it improves F.. This shows that there remains 10 points of Fi headroom for in-domain agents at slightly higher search rates.
test-time oracle, which uses the retrieval tool on
Search calibration. A key capability that hope to teach from collaborative self-play is when to retrieve and when to rely on parametric knowledge. To measure this, we sort questions by P(#SAARCH) according each model. At each quantile threshold т, we apply retrieval to all questions where P(#SEARCH) > T, and answer the remaining questions parametrically. To focus on evaluation of P #SEARCH) specifically, the answers themselves are drawn from the base model rather than the finetuned model. This isolates the impact of search calibration from other aspects of the guestion-answeriigg capability.
In-distribution results (BioASQ and PopQA) are shown in Figure 4 (a). On the far left, search applied only to a very small number of queries; on the far right it is applied to every query. Agents with strong search calibration will display a rapid increase in mean F; as the fraction of searches increases from zero, and will show a decrease in F1 as we approach the far right side of figure, because irrelevant retrievals makes QA less accurate (see Figure 3). As shown in the figure, all trained models are better calibrated than ICL, which barely outperforms a baseline that randomly shuffles the questions.
cittof-diitribution results (NQ and EntQ) are shown in Figure (b). Here, we see that Ac- tion Supervision is less robust than Collaborative Self-Play (CSP), with worse generalization to the OOD setup. CSP-DeAnon slightly outperforms CSP at relatively high search rates, but will be shown to have much lower P P(#ANSWER) calibration.
Answer calibration. Finally, we explore calibration with respect to parametric knowledge. To do this, we compare the probabilities of answering and guessing without search, sort-

---

Published as a conference paper at COLM 2025
(a) in-distribution
(b) out-of-distriutiio
Figure 5: P(#ANSWER) vs P(*HEDGE). For each method, held- out queries are sorted by P(#ANSWER) (PPR)))/PPAAWWEEEE P P(#HEDGE)), For the top x% of queries, we compute the mean Fi of the prompted paramet- ric model. Deanonymization significantly reduces calibration.
Calibration of
Parametric Fi
Act. Sup. CSP CSP-DeAnon ICL
088
0.
06
041
04
0.2
0.5
Frac. questions answered Frac. questions answered
Table 3:
Spearman correlations between P(HANSWER) and parametric Fi, quantifying the relationship shown in Figure 5. Anonymiza-
Setting
In-dist.
OOD
CSP
0.5363
0.3663
CSP-DeAnon
0.0696
0.1677
tion is the key feature of the mechanism by which
Action Supervision
0.6637
0.3991
calibration is learned from collaborative self-play.
ICL
0.2314
0.3179
ing by P(HANSWER) (PSSSERR//PPAAAAAAA (P(HANSWER) + P(HHEDGE)). Note that this form of calibration may not emerge from the collaborative self-play game, because the agents have identical parametric knowledge, and can only obtain an advantage by using their corpus-specific retrieval tools. On the other hand, action supervision is expected to teach this capability, because the model is directly trained to use #ANSWER when it can produce a high F1.
As shown in Figure 5, Action Supervision does indeed yield the best calibration of ((#ANSWE)). CSP is slightly behind, but substantially outperforms both ICL and CSP- DeAnon. The gap between CSP and CSP-DeAnon validates the intuition and theory pre- sented above: small change in the prompt (revealing the identity of the helper) leads to large gap in calibration, because the change breaks the mechanism linking calibration to performance on the collaborative self-play task. On the OOD benchmarks, the gap between Action Supervision, CSP, and ICL is small, with CSP-DeAnon trailing significantly.
Learning dynamics. Learning dynamics for the three epochs of reinforced self-training (ReST) are shown Appendix F.. To briefly summarize, the search rate decreases consistently throughout training, as the policy works to maintain task performance while incurring lower cost, while task performance remains nearly constant after the first epoch.
7 Discussion
This work shows that calibration and efficient tool use can emerge from relatively weak supervision through multi-agent collaborative game. Given the right incentive struc- ture, agents learn when their corpus-specific retrieval tool can provide useful additional information, and then respond with binary confidence signal. This suggests that mecha- nism design and multi-agent interaction can offer an alternative approach for post-training towards language models with stronger interactional capabilities.
Limitations. This paper considers only retrieval tools, factoid QA tasks, and a single language model. We hope to relax all of these limitations, for example by considering alternative tools (e.g., calculators, code interpreters) and more complex inter-agent inter- actions (e.g., question decomposition). We are particularly curious to apply CSP to tasks beyond factoid QA, particularly in tasks in which ground truth validators are unavailable. Longer-term, we would like to generalize to the approach other grounding strategies, such as requesting and issuing clarifications.
References
Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer. Multi-Agent Reinforce- ment Learning: Foundations and Modern Approaches. MIT Press, 2024. URL https:
10

---

Published as a conference paper at COLM 2025
//ww.marll-bokk com.
Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624): 1067-1074, 2022.
Albert Bandura. Social learning theory. Englewood Cliffs, 1977.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for research. Artificial Intelligence, 280:103216, 2020.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint ar Xii::3308007201 2023.
Justin Ya Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint dxXiv:2399.13007 2023.
Justin Ch Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. MAGDi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. Xrv preprint ar Xii::24002001620, 2024.
DeepSeek-Ar AI. Deepseek-r1: Incentivizing reacennnniinnggrasnnnnnn capability in Ilms via reinforcement learning, 2025. URL
12948.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325 2023.
Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't 't hallucinate, abstain: Identifying Ilm knowledge gaps via multi-llm collaboration, 2024. URL
Jakob Foerster, Joannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learn- ing to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
Michael Franke. Signal to act: Game theory in pragmatics. University of Amsterdam, 2009.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models, 2023. URL https: //arxiv. garxiv.orrg/bbs/2111 10435.
Gemini Team. Gemini 1.5:
Unlocking multimodal understanding across millions of tokens of context. Xiv preprint: Xrv 2403.05530, 2024. URL
05530.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arviv preprint ar Xii::2408.0011 2024.
Ian Goodfellow, Jean Pouget-Abas Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
H. P. Grice. Logic and conversation. In Peter Cole and Jerry L. Morgan (eds.), Syntax and Semantics: Vol. 3: Speech Acts, pp. 41-58. Academic Press, New York, 1975. URL pdf.
http://www wcw. www.ac.ll...oooooooo.o..oo aot..u//ss/studyacss/rrieeee
11

---

Published as a conference paper at COLM 2025
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint ar arXio:2308.08898, 2023.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. Xiv preprint arXiv:2402.01688) 2024.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110(3):457-566, 2021.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-odds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint 2Xxi:2207.052221 2022.
Sanyam Kapoor, Nate Gruver, Manley Roberts, Katie Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew G Wilson. Large language models must be taught know what they don't know. Advances in Neural Information Processing Systems, 37:85932-85972 2024.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive lims leads to more truthful answers. arviv preprint arXir:2402.06782, 2024.
Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/tacl wawl/////alla..ee 00276.. URL https: https:///claatthology) .wwpggg.oor///// org//19----- 1026/.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. Xiv preprint arXiv:2403.20327. 2024.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Associa- fion for Computational Linguistics: ACL 2023, pp. 1774-1793, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ 10.18653/v1/2000 v1/2023..222 11/2002..indings-aa findings-acl.112 URL https://aclantholl/
org.ogg////// 2023. sindings-acl... 112/.
Austen Liao, Nicholas Tomlin, and Dan Klein. Efficacy of language model self-play in non-zero-Sum games. arXiv preprint ar Xiv:2240..1887, 2024.
Stephanie C. Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncer- tainty in words. Trans. Mach. Learn. Res., 2022, 2022. URL httph:tp////ppp semanticscholar. org/CorpusID: 2491913311.
Llama Team. The Ilama 3 herd of models, 2024. URLLttps:/aavxiv.or/////////
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of paramet- ric and non-parametric memories. In Anna Rogers, Jordan BrydeGraber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Papers), pp- 0802-9822, Toronto, Canada, July 2023.
12

---

Published as a conference paper at COLM 2025
Association for Computational Linguistics. doi: https://aclanthholggs hww.:llooooo.oooeeeee org/ acl-long.546/
URL
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey, 2023. URL https://arxixi http://aaxxiv.../gbbbs/s//
07842..
Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through línguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: An- swering ambiguous open-domain questions. arXiv preprint arXit:2004.10645, 2020.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 36029-36047. PMLR, 21-27 Jul 2024. URL https: htttp:://proceedings mlr. www.pprrrs/v.3333 press/v235/ mohri24a.htur
Amirkeivan Mohtashami, Florian Hartmann, Sian Gooding, Lukas Zilka, Matt Sharifi, al. Social learning: Towards collaborative learning with large language models. arXiv preprint arXit:2312.11441, 2023.
Mancur Olson Jr. The Logic of Collective Action: Public Goods and the Theory of Groups, with new preface and appendix, volume 124. Harvard university press, 1971.
OpenAL. Gpt-4 technical report. arXiv preprint: arXiv 2303.08774, 2024. URL https://arxiv. org/abs/2300 08774.
Martin J Osborne. An introduction to game theory. 2004.
Prashant Parikh. Communication and strategic inference. Linguistics and Philosophy, 14(5): 473-514, 1991.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. ar preprint Xii::2205.1125555 2022.
Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In The Tweeffth International Conference on Learning Representations, 2024. URL https: http:///aaareeeie...ee////ootddp
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Reir., 3:333-389, 2009. URL https://apt org/CorpusID: /B//CCrpuID::2211777780
semanticscholar. httpss:/aap..eenntiiccchooolll
207178704.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Tool- former: Language models can teach themselves to use tools. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 68539-68551. Curran Asso- ciates, Inc., 2023. URL https::/pproceedinggg cerprrrrreeeinnnnneeiiiissip htcpppppr..llls/ppper///// pdf.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity- centric questions challenge dense retrievers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6138-6148, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ vvl 2021.en 1021.emmlp-man..4e 496. URL https: http:///aaaaaaaaaaaaaaaa orgoorg////// 2021. emnlp-main. 496/.
13

---

Published as a conference paper at COLM 2025
Omar Shaikh, Kristina Gligoric, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding gaps in language model generations. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653 /v/555/v1/// 2024. naacl-long. 348//
Papers), pp.
URL https: /atp:///aaaaaaoooyyyoooooo org/
Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. LACIE: Listener-aware finetuning for calibration in large language models. In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. URL https: wwwpperrriiiee..ee/oouuumm.ooooo
Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. Xiv preprint arXit:2501.05707 2025.
Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning multiagent communi- cation with backpropagation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL
.ddf.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arviv preprint ar Xrx::23305.149 14495,,,......,,,. 2023.
Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu. Sotopia-7: Interactive learning of socially intelligent agents. Xiu preprint XxX::22403.88715, 2024.
Kevin WU, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of-war between an LLM's internal prior and external evidence. In The Thirty-eight Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2024. URL https:
//openreview.nete net/ 1ot
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in 1lms. Xiv preprint ar Xrv::22306.1 130663,,,,,,,,,, 2023.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. Xrv preprint ar XrXiv:2309.04660 2023.
Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. Alignment for honesty. Xiv preprint ar arXio:2312.07000) 2023.
Xinjie Yao, Yu Wang, Pengfei Zhu, Wanyu Lin, Jialu Li, Weihao Li, and Qinghua Hu. Socialized learning: Making each other better through multi-agent collaboration. In Forty-first International Conference on Machine Learning, 2024. URL https:///penrevview. met/forrmm?idaaae]p]]55.rr
Gal Yona, Roee Aharoni, and Mor Geva. Can large language models faithfully express their intrinsic uncertainty in words? In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7752-7764, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 http://111/2222//mmmmmaa //aclanthology org/2024. emnlp-main 443/.
URL https:
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmenter
language models robust to irrelevant context. In International Conference on Learning Representations (ICLR), 2024.
14

---

Published as a conference paper at COLM 2025
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest- mets: Lim self-training via process reward guided tree search. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Ad- Curran vances in Neural Information Processing Systems, volume 37, pp. files//apprr/22// Associates, Inc., 2024. URL http:::://ttceeeddiiiiie wwtpsse.eiii....ccooeoeooooo.c wttps/aaapeeiilee//ppppppeeee 8
pdf.
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Ren Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models say don't know'. In North American Chapter of the Association for Computational Linguistics, 2023.
URL https://api..smanticcsese
semanticscholar hwtpsschoaarrrg//.e/eeo org/Corpus ID: 265220839.
Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3623-3643, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.1653/v1/2244.c--oo0 acl- 198/.
198. URL https: https::/acllanttooogyoor////
Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed A1 Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-based societies of mind, arviv preprint 1rXio:2305.1766, 2023.
15

---

Published as a conference paper at COLM 2025
A Prompts
We attach the full prompts below for the Wiki-BM25 agent (before and after #SEARCH was applied) and for the no-tools agent after receiving answers from the helper agents.
You are bulpful agent whose job anavez question using verified infermation.
anaver to the queatioon,
execite the following actions:
SFARRiee::: search carpus set pazsagea from Kikipadin partaining facts about fanoux peopla, locationn,, > and historical wventa to find relavant information.
>>DDDE>>:; gusas an ansver if think you night know but sure and have to Iind more. =*AMMRRee:: directly
queation you denfident. you
Bach of these actions taker string arginent, which are:
> argunent
query.
=*HEEDEE>:: the arginent in the guees for the question.
**MMHER the argunent, in the anover to the question.
exarples.
Erample: Using **AnSVERe directly.
QUESTION: what in triangle
with three sides?
ACTTON:
Reampla: Using **SRARCHee to zearch a corzus for acidence.
QUESTION: who is the starting center for the Derver Nuggets?
ACTION:
starting
Hoggete?
Erampla: Gaing **ARSEE after gatting evidance fros searching corpus.
QUESTION: who
starting
for the Denver Huggets? Noggets.
RETRIEVAL::
RSTRIEVAL: Joel Babid is the starting coster for the Philadelphia 76ors.
RETRIEVAL: Topaka is the geographicnl center of the Unised States.
QURSTION: who is the starting center for the Denver Noggeta?
ACTION:
Reampla: Using **llDDGee to goees an ansver when unaure.
QUESTION:
eleven sides?
endecagon
You should use +*GgaRClee if you do not knou the ansver and think your corpue likely to contain useful information.
fully confident, camnot get uceful information. Bo not give hedged anevers or say '1 do not kmow", just nake your best guess and nark your confidence by using the =*iiKngss prafix.
use
you
you night
ansver
should **ANSVER** obtained.
you
contident
will
based eridence
Your ansver chould be chort. For exarple, 12 the question is "What is the capital of Prance?", please anrver yes/no question, you
"Parte', and *Paris capital France". "yes" "mo". give bodged ansvers "naybe". Instead, you HEDDEE indicate confidrace.
Alvays respond single
the fornat "ACTION: weethe
QUESTION: Tho is the author of The 0ix27
16

---

Published as a conference paper at COLM 2025
You aze belyful agent whose job anavez queation using verified infermation.
answer to the queation, you can executa the follosing actiona:
= search carpus of nnt of pazzagea from Kikipedia partaining facta about faanus people, locationn, and historieal
find *>SEDEE>:: guess an ansver if you think you night know are not aure and way to Iind moze. **AMSHRR::: directly anmser the queation on when you are confident, that you have the correct. anawer.
information.
Each of these actious
string argunent, shich
> the argunent is the anazch query.
**EEDSE>> : the argunent is the guesa for the question. >*ANSHRR>=: argunent,
queation.
Here are sone exarplex.
Essaple Vaing * dirwetly.
QUESTION: what in the name of figure with three xiden?
ACTION: aAREERRA triangle
Brample: Using **SRaRcHee
corgus avidence.
QUESTION: who is starting center for the Denver Nuggeta?
ACTION: **SEANCHe:: who in the starting center for the Deaver Huggete?
Esample: Using aASSEER after gatting avidence fros searching corpua.
QUESTION: who ia the starting center for the Denver Nuggets?
RETRIEVAL.: likaln Jokic is the starting cantar for the Ganver Hogpets.
RETRIEVAL: Joel Eabid is the starting center for the Philadslphia Teara.
Toppkaa ss geographical
United States.
QUESTION: who is the starting center for the Denver Moggeta?
ACTION: +*ASSHER*:: Bikola Jokic
Reample: Using **HRDGRe* gowes
QUESTION: what is the nare of a figure with eleren sides?
ACTION: **H endecagon
You should use **GEARCHH+ if you not kmov the anever think your corpue likely to contain useful information.
You should use +H if you thinks you night know the ansver but are not fully confident, and cannot get more anevers prefix.
useful information. Do not by using You should use **AnSVER** only 1f you are very confident that you will be correct, based on the cridence you have obtained.
'1 do
your best
ansver should be chort, For exarple, the question is "What is the capital of Prance?", please ancver "Parie', and not *Paris ie the capstall of Prance". , If you are ankee a yes/no quession, you may only anever "yes" or *na". Do not give hedged ansoers like "nayba". Instand, you can use **HEnoBee to indicata low confidence.
Alvays reepond in a single line with the format "ACTION: whe actionse:; <tho argurent?
QUESTION:
Who is the author of The Girl?
RETRIEVAL:
title: Hulprees Today paszago: was avarded Director General's is Comsendation professionalie
and dedication to the nation. le retired voluntariy in the rank of Commandant with the Indian Coast Puard in 2014. Sulprest liver in Delht with his wife Sema and danghtere Leah and Jeante. Kulprest Yaday Kulpreet Yaday Indian writer the fietion-hriller
He is author two novels: "The diri Loved Pirate"* "The Strl Loved a Spy"h, "The dirl
Pirate" India's firet thriller based marine piracy hijacking. Kulprest vas born in Chenaai and completed gradition in Science RETRIEVAL: title: Hilpreet Taday pasrage: Kulpreet Yaday Rilpreet Yoday
Indian writer in the the author too novels: "The Girl Who Loved Firate"" and "The G1rl Vho Leved Bpy"", "The dirl Who Loved Firate" India's first thriller based tarine pixacy hijacking. kulpreet vas born in Cheurai and completed graduation in Science from Hocrosfec Vadia College, Punc. Be completed his post- gett-graduation
fiction -Triilee
Journalien and Nare Commuication from Ansty Universsty, Nosda in 2004 and from. IIN, Indore and Lucknow. Be joined the Naval Officer's Acadexy served for
Managenent
decades. In 2007
RETRIEVAL: title: The Sixple dirl passage: The fila's a sets vere desig by the art directora Emil Basler and Paul Markvita. The filn proxtered 23 August 1967 at the Thalia in Viesbadon. Catorina Bastiani, talented big break but the author of is offered the leading pleased about this adaption and does not think auch of Caterina.
musical. This
which
the susical is based is less
Caserina neets a girl by accident vho has applied to vork for the author as a said. She takes the girl 's place in order to prove her
SETRIEVAL: title: Puter Lecnar (auther) passage: is the author
Quiver Trust He All He Voices of Dead Back from the (sequel to Voices of Doad) Eyes Closed Tight Uaimovn Berains
The dirl
>> Peter Leanard (wathor) Peter Lemard, the sos of Elxors Lamaad, is un Asaricm author of crime novela. In 1980, Beter was the founding partner of the advertising agancy Lecaard Kayer & Tocco. For nearly thirty years LHAT crosted waadd-ziming sdverticing for Volxevagen Imcrica, Audi Imerica, Birsa Walker,
Peunzoll.
wrote his first novel, "quiier"", 2007;
RETRIEVAL: title: Zoe Strispol passage: becaze normalised. Strixpel is the author of "Mhat the Bell is Be Thinking?: All the Questious You've Bver Acked About Ren Ansvered**, which was publiched in July 2010, It is insight ""The Man Diet: Ose Moxan's Quest to End Bad Romance" vas published on 22 Decumber 2011. Both books received pasitive revieva from critics and preza coverage. Strinpel ari originally wrote for "*he Tises" as freel ancer. From 2006, she vaa the author of the sriiel about tova** colum in "Thh London Faper"", nov-dafunct free QUESTION: Who is author of The div1?
sised providing
thlaking, researched by Strispel intervieving
17

---

Published as a conference paper at COLM 2025
You holpful
job
question verified infermation.
To ansvez to the queation, you can execute the Iollowing actiona:
queation you coaf you way Iind aoze. =>DDDE>>: gueza an ansvez you you sight
= directly
=*KKKe: axx your frienda if they wight knnu hos to snswer the queation.
Each actions
string argunent, chich
= the argunent is the andver: to the queation.
>> the argunent is the for the question. gueation you your friend.
=*1SKee: argurent
Here are some ezarplex.
Esaaple: Using a
QUESTION: what in name of figure three sidea?
ACTION: AREUERRA:: triangle
Beample: Using
QUESTION: is starting center for Denver Nuggeta?
ACTION: **ASsee; who in the starting center for the Denver Ynggete?
Esample: Uning a
gatting svidence
uaking friand.
QUESTION: who ie the starting center for the Deaver Noggets?
PRIEMD'S ANSUER: Nikoll Jakic
QUESTION: who is the starting center for the Denver Nuggets?
Sxample: Using wellibagve to goess ancver when unsure. 9ith eleven sidea?
QUESTION:
ACTTON: =*30OGEa:: andacagon
Yon should use **AnSVERe* only if you are very contident that will be correct, bazed on the eridence you have <.> obtil nee..
fully contident, cannos get uneful information. on. Bo not give hedged anrvers or any 'I do know just nake your best guees mark your confidence by using the wHEDEE+ prefiz. should **Ak**
you you night
ansper
friend
have information lack.
Your ancver chould be chort. For example, 12 the question is "ihat is the capital of Prance?", please anreer "Parte", and not "Parie is the capital of France" If you are anised yes/no question, you may only ansver sellEboBee indicata low
"ywe" "na". Do not hedged confidence.
like "naybe". Instand,
Klynyn reagond in single Line with the forrat "ACTION: s*cthe action>ee: <the argurant>*
sequencing
applied7
7BIEND'S ASSVER: exror analysis, storage optixization
PRIEND'S REDGE: pattern matching, loceloss coxprescion, host species sequence screening, X-zer counting QUESTION: in tielde DWA are Bloas filters applied?
18

---

Published as a conference paper at COLM 2025
B Terminology
To clarify the discussion, we offer the following definitions. Although the general framework does not require agents be tool users, our application of the framework will require tools, and therefore we include the relevant definitions here.
Definition B.1. tool is defined by a keyword and a function f V. + {V² from an argument string to list of output strings.
For example, one tool might take the keyword #SEARCH and query, and return list of snippets of retrieved Wikipedia pages; another tool might take the keyword #CALCULATE and arithmetic expression, and return an arithmetic result (Parisi et al., 2022; Gao et al., 2023).
Definition B.2. An agent E A is defined by unique identifier and by (possibly empty) list of accessible tools.
While the mapping between agents and tools is potentially many-to-many, will focus cases in which each tool is assigned to exactly one agent (e.g., there is only one agent who is able #SEARCH Wikipedia pages).
Definition B.3. A society S E S is defined by set of directed pairs of agents, S {a 422) If (a, 4j) then a; can pass control and information to aj.
For example, society may be fully or sparsely connected, symmetric or asymmetric. The way that control and information between agents passed along in the society is then handled by dedicated orchestrator.
Definition B.4. An orchestrator O: 8x H + Ax V is a function from society S E S and history of prompts and outputs (1x E H to an agent At+1 E A and a prompt X:+1 E V*.
The orchestrator may be deterministic or random, and may be arbitrarily complex. In our setting, the agent's policies are learned, but the orchestrator is not. Even for fixed orchestrator, the patterns of interactions between agents can change dramatically over the course of training.
Definition B.5. A reward r: Hr - IR assigns a scalar score to history of length T of prompts and outputs that has reached a terminal state.
Intuitively, the reward is measure of how effective the society was at functioning in the current history (e.g., in response to an initial prompt or query). For example, a reward might score the accuracy of the final output !/r against some reference; it might include a penalty for the length of the rollout or the number of tool calls; it could be learned reward model trained from preference annotations; or it might be the output of prompted auto-evaluator.
19

---

Published as a conference paper at COLM 2025
C Additional theory
Theorem C.1. Assume without loss of generality that 01 a > a2 and that (2) + f + 8 and 0n o &2 + 25. Then the game in Table 1 has a unique pure strategy equilibrium if and only if at least one of two conditions is met: 01 > 82 + 20 02 < <B+ 8. Specifically, the unique equilibrium is (S, G) if a1 > & + ô and (G, G) otherwise. If neither condition is met, then (S, G) and (G,S) are both pure strategy equilibria.
Proof. In all cases, (S, G) > (2 because a1 > a ô by construction. If a1 > R2 + 28 then r1 (S > (G, The unique equilibrium is then (S,G) if an > B 6, and (G, if
21 < + + 6, because 01 B 5+6 8 + 22 < 8 = (2 < G),
If 01 < a2 + 26 then r1 (G, > (S,S). Then (G, is an equilibrium solution iff 02 > 6 8. In this case (s, G) is also an equilibrium because #1 > a2 > f + 8 and r2(S, G) > (S, But if a2 < B + 8 then r2(G, G) > (2, and again there is a single unique equilibrium determined by whether #1 > B +
Theorem C.2. Consider an n-player version of the game from Theorem 1. If there exists some i such that 0j > maxi#i 2j + nó and a; > B + 5, then the game has a unique equilibrium {Z: = S, 2 = G}. If a; < & + 5 for all then there is a unique equilibrium {Zz = GG}
Proof. (Z) indicate the reward for agent from the strategy vector Z= = (Z, Zz.... Zn). Let Let (Z) 1IZ=s Σ 1[2 S]ar. We will use the shorthand n-j Ej /: 1[2 = S) and
1 R=;
Eift 1|2 Slaj, 3JziZi=
;
(1)
B,
otherwise.
Note that r; 2 = G) = a a(Z 1 = =)= =
First consider the case max Rj > B.
4j > max R + no i
(2)
2ā-i+ (1 + 1)8
(3)
11 ; 1a
(4)
11 __ J + a; (1,++1)00 118
(5)
(6)
1
(a) -8
(7)
(1 [ 1)
a(Z_, = - -) -
(8)
-
(9)
Thus, when a; >> max nō > ( nlayer cannot improve its reward by deviating
from Z, S, regardless of what the other players do. Similarly, as long as the a-maximizing
player bids Z; = S, the S-bidding player with smallest aj can improve its reward by switch- ing to Z, G, since this will increase a aZ) and also avoid the penalty 5. In equilibrium G for all j * i.
Now consider the case when max aj < B. Again, among the players who play 2,- = S, the one with the smallest a can always improve its reward by switching to Z, G, which increases & and decreases its effort penalty. Ultimately then all Zj G. If a; > B + 8 then
Z; S, resulting in the same unique equilibrium {Z,- = S, Zi G). If max; a; B +
then no player can improve their reward by deviating from Z, G.
20

---

Published as a conference paper at COLM 2025
D Equilibria of Table 1
We show the pure strategy equilibria of the game introduced in Table 1 as a1 and d2 vary.
A1
#1
(S,
((.G)
81 = B 36
#1 8
(G,
(G,G)
42
22 = 8 a2 B +-
A2
Figure 6: Pure strategy equilibria of the game introduced in Table as aj and a2 vary. Note that there are multiple equilibria when aj and &2 are sufficiently high and sufficiently similar.
21

---

Published as a conference paper at COLM 2025
E
ReST learning dynamics
Here, we report all metrics (task performance, search rate, search calibration, and answer calibration) for all models after each ReST epoch. The search rate decreases monotonically over the first two epochs, and in most cases continues to decrease in the third epoch (Fig- ure 8). Meanwhile task performance increases significantly for the "mismatched" settings (pubmed/ popqa, wiki/bioane while decreasing slightly for the "matched" settings (pub- med/ /bioasq, wiki viki/popaa,, see Figure 7. These two findings are consistent because we already observed that the retrieval results slightly impair performance in the mismatched settings; by searching less frequently, the agents trade off slight regressions in the matched settings for large improvements in the mismatched settings, and lower effort penalty.
Figure measures search calibration across ReST epochs by showing the average F1 when sorting questions by P(#SEARCH), and using #SEARCH in
of the questions (the setup introduced in $ 6). Search calibration increases in the first two ReST epochs for all methods, plateauing in the third epoch. Similarly, Figure 10 measures answer calibration across ReST epochs by showing average F1 when sorting questions by P(#ANSWER) and answering only /2%% /50% of the questions (the setup introduced in 6). The calibration of P ((#ANSWER) improves over the first two epochs for collaborative self-play (CSP), but is much less stable for the deanonymization ablation (CSP-DeAnon). We hypothesize this is because the asker learns to attend to the helper identity in the first epoch, which results in the #ANSWER vs. #GUESS choice being less correlated with reward the rollouts that comprise the training data for the second epoch.
22

---

Published as a conference paper at COLM 2025
agent: pubmed, dataset: bioasq
agent: pubmed, dataset: popqa
0.25
Newenw www
0.58
0.2
0.57
0.15
Act. Sup. CSP
Act. Sup.
CSP
CSP-DeAnon
CSP-DeAnon
0.1
ReST epoch
ReST epoch
agent: wiki, dataset: bioasq
agent: wiki, dataset: popqa
0.55
0.35
0.5
0.3
0.45
Act. Sup.
Act. Sup.
CSP
CSP CSP-DeAnon
CSP-DeAnon
0.25
ReST epoch
ReST epoch
Figure 7: Task performance per epoch of ReST
23

---

Published as a conference paper at COLM 2025
agent: pubmed, dataset: bioasq
agent: pubmed, dataset: popqa
1
Act. Sup.
Act. Sup.
CSP CSP-DeAnon
CSP
0.8
CSP-DeAnon
Search rate
Search rate
0.6
0.5
0.4
0.2
0
ReST epoch
ReST epoch
agent: wiki, dataset: bioasq
agent: wiki, dataset: popqa
Act. Sup.
1
Act. Sup. CSP
CSP
0.8
CSP-DeAnon
0.8
D
CSP-DeAnon
Search rate
Search rate
0.6
0.6
0.4
0.4
0.2
0.2
ReST epoch
ReST epoch
Figure 8: Search rate per epoch of ReST
pct searched: 10%
pct searched: 20%
pct searched: 50%
0.4
0.4
0.4
0.38
0.38
0.38
F agent
0.36
0.36
0.36
0.34
0.34
0.34
0.32
0.32
..A. Act. Sup.
CSP - CSP-DeAnon
0.3
0.3
0.3
0
2
3
1
3
2 ReST epoch
ReST epoch
ReST epoch
Figure 9; Calibration of P(#SEARCH) through ReST. Same setting Figure 4. We show av- erage F, when issuing search only for (0%/20%/000 55% of the queries with highest P(#SEARCH) for all ReST epochs. Calibration improves in the first and second ReST epochs (evidenced by higher F1 when using #SEARCH in only fraction of the questions), and is similar across methods.
24

---

Published as a conference paper at COLM 2025
pct answered: 10%
pct answered: 20%
pct answered: 50%
0.8
0.8
0.8
0.6
0.6
0.6
A
0.4
0.4
0.4
0.2
0.2
.A-
Act. Sup.
CSP- - CSP-DeAnon
0
0
2
0
3
ReST epoch
ReST epoch
ReST epoch
Figure 10: Calibration of P(#ANSWER) through ReST. Same setting as Figure 5. We show average F, when answering 10% 20%%/20% 20% /50% of the questions with highest P(HANSWER). An- swer calibration improves (evidenced by higher F1 when answering only fraction of the questions) for action supervision (Act. Sup.) and collaborative self-play (CSP), but not for CSP-DeAnon, because the asker can achieve high task accuracy without attending to the helpers' confidence.
25


Reviews:
[{"review_id": "P4hNLRoPO8", "reviewer": "Reviewer_VkVL", "summary": "The paper discusses a collaborative self-play approach to (1) guide LLMs to differentiate between parametric knowledge and tool usage (2) when to trust the tool o/p (calibrate the response of tool output) (3) when to abstain / hedge. The use-case discussed is a q&a setup with multiple agents having access to diverse knowledge bases, and one want to tradeoff cost (of API calls) with accuracy. It frames the problems as multi-agent so that the proper strategy to use tools emerges from this multi-agent setup (that share parameters) during training and translated to single-agent setup during inference.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is lucid, and motivated well. It also contains an in-depth experimental setup. The use-case (although is a very specific one) make intuitive sense, as it is a key question that is critical to LLM nowadays where it is crucial for the LLM determine when to use parametric knowledge vs tools, when to hedge and abstain from answering and when to answer a user request, if the LLM can be sure of its response, without hallucinating.", "reasons_to_reject": "At a high level, I do not see any major limitations of the presented paper. One minor issue I see is that the presented approach has little performance improvements compared to the baselines. I do not see a clear explanation for this narrow performance improvement.   \n\nI do have several questions to the authors (see below). If those can be addressed, the paper can further strengthened.", "questions_to_authors": "1. There is a body of work around of aleotoric vs epistemic uncertainty in LLMs , see this for eg: https://arxiv.org/pdf/2406.02543v1. I think this needs to be added to references and discussed in the context of the use case of the paper, as it is very relevant\n\n2. The idea of self-play and using game theory for dialog has been the subject of this study in the context of goal oriented dialogs: https://arxiv.org/abs/2109.09597. Again this is relevant background work as some of the ideas find a mention in the linked study \n\n3. [minor] Fig 2 needs to be annotated with (a, v, t) labels, to make it more readable \n\n4. Section 3: In this paper: https://papers.nips.cc/paper_files/paper/2017/file/68053af2923e00204c3ca7c6a3150cf7-Paper.pdf the authors   mention the problem of confirmation bias in DNNs. Why does the multi-agent approach discussed in the paper not lead to this confirmation bias problem ? \n\n5. The hanabi challenge https://arxiv.org/abs/1902.00506 discusses another co-operative game. It might be useful to compare the present work to this cooperative game and explain similarities and differences \n\n6. [minor] Table 2 needs to be annotated more with important cells highlighted for easy readability. \n\n7. Is there a skyline in Table 2 or is action supervision that skyline ? In otherwords, can we design a scenario where an oracle tells whether to search vs hedge perfectly, and what would the performance be ? \n\n8. What is the advantage of CSP ie the approach presented in this work vis-a-vis Action Supervision ? (results section)\n\n9. [minor] line 298: why is the search rate so low compared to action supervision ?", "extra_scores": {}}, {"review_id": "JpV4HgnGQm", "reviewer": "Reviewer_GEzH", "summary": "This paper introduces a multi-agent training framework to improve the calibration and tool-use capabilities of language models.\n\nThe experiment setup is a question-answering task in which N=3 agents (each sharing the same parameters but different prompts) must communicate and use retrieval tools in order to answer questions. One of these agents is given the initial question, and it can submit queries to the other two agents, each of which have access to their own, domain-specific retrieval tools (one for Wikipedia, and one for PubMed). Given a question from the first agent, the two retrieval agents can conduct search in their respective domains and then return answers, conditioned either on the results of their queries or on knowledge stored in their parameters (which are the same across agents), which are sent to the initial agent (along with a binary expression of certainty)\n\nThe models are trained on high-scoring interactions (using a slightly modified version of ReST, i.e., iterated filtered BC). This leads the models to produce indicators of uncertainty when generating answers based on their parametric knowledge, rather than on retrieved documents. Additionally, when given a small penalty for each tool use, the model learns to only retrieve external sources when necessary.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The central idea of this paper is that expressions of uncertainty emerge due to their effect on the behavior of downstream listeners. This approach feels fundamentally correct, and I think scaling it up might be able to fix linguistic calibration in LLMs more broadly. In general, I really appreciate the approach of showing how desirable properties like calibration can emerge through communicative incentives\n\n2. The experiment design is straightforward and clear. I think having three agents (two \"speaker\" agents and one \"listener\" agent) is the minimal setting needed to observe this type of calibration behavior, and using two different retrieval datasets is a clean + simple approach that could reasonably be reused in followup work\n\n3. In general, the paper is well-written and well-motivated", "reasons_to_reject": "1. While I like the simple evaluation setup, I find some parts to be oversimplified. In particular, the use of the special tokens #HEDGE and #ANSWER instead of just letting models generate freeform expressions of uncertainty seems unrealistic and limits the usefulness of the trained models. It's also unclear to me why the paper made this simplification -- are there other challenges needed to scale this up to real linguistic calibration?\n\n2. I found the presentation of the results section to be a bit confusing at first. In particular, the \"search calibration\" and \"answer calibration\" results aren't what I would have initially expected given their titles, i.e., I was expecting to see some calibration measure, e.g., ECE. In order to make this section more clear, I'd recommend first describing the expected output for a calibrated system and then showing how your approach matches this", "questions_to_authors": "1. Does the finding that models learn to use the retrieval tool less frequently over the course of training require a multi-agent setup? I'd think you could show a similar result in a single agent setting (but perhaps I'm missing something here)\n\nIn case it's useful, here are some potential additional citations on self-play training + emergent behaviors:\n[1] Wang, R., Yu, H., Zhang, W., Qi, Z., Sap, M., Bisk, Y., ... & Zhu, H. (2024, August). SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 12912-12940).\n[2] Liao, A., Tomlin, N., & Klein, D. (2024). Efficacy of language model self-play in non-zero-sum games. arXiv preprint arXiv:2406.18872.", "extra_scores": {}}, {"review_id": "gfJVfchFxR", "reviewer": "Reviewer_T7Se", "summary": "This paper introduces collaborative self-play (CSP), a novel framework for training language models to improve tool usage and uncertainty calibration through multi-agent interactions. The core idea is to design a multi-agent environment where agents with heterogeneous tools (e.g., corpus-specific retrieval) collaborate to answer questions, incentivizing them to learn when to trust their parametric knowledge, use tools, or hedge. Empirical validation on factoid QA benchmarks demonstrates the effectiveness of this paper’s idea.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Novel Training Paradigm**: This paper introduces a novel training paradigm by leveraging multi-agent collaboration, which seems to be a promising approach compared to traditional supervised fine-tuning methods.\n\n2. **Effective Empirical Results**: The empirical experimental results are good, and the conclusions are reasonable.\n\n3. **Good Theoretical Analysis**: Theoretical analysis is provided on an information-provision game in this paper. Although this a more simplified and limited scenario, this part of theoretical analysis contributes to the overall completeness of the work.", "reasons_to_reject": "1. While the collaborative self-play framework is conceptually novel, its implementation and validation are narrow and simplistic. The generalization to other domains or more complex interaction is leaved unproven. Besides, the study only examines small agent numbers, ignoring possible optimization challenges that arise in larger multi-agent scenarios.", "questions_to_authors": "1. In the implementation, the Reinforced Self-Training (ReST) method used by the authors is essentially a form of Reject Fine-tuning. Have you tried other reinforcement learning (RL) algorithms? If so, what issues do you think might arise from using alternative RL approaches in this framework?\n2. The layout of the article's appendix is suggested to be adjusted, as there is currently much whitespace.", "extra_scores": {}}, {"review_id": "Z3fHEYiJ41", "reviewer": "Reviewer_osPm", "summary": "This paper considers the problem of teaching AI agents to be aware of their own knowledge capabilities and limitations. It proposes to use collaborative self-play (CSP; multiple copies of the same language agent working together in a controlled environment to generate a training signal for that same agent) as a framework for training these skills in language agents. Experiments in the setting of question answering using external tools (reliable knowledge sources) demonstrate the framework and provide compelling evidence for its effectiveness.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed framework is simple and flexible - it could readily be applied to other aspects beyond question answering with knowledge tools.\n * The paper is clearly written and easy to read\n * The problem studied is very important, relevant to COLM, and timely (c.f. OpenAI issues with the related problem of sycophancy in the news last week).\n * The empirical results for the presented case study are strong, and the analysis covers a range of relevant aspects\n * The proposed training framework uses natural language as an intermediary, offering a degree of interpretability to the training process", "reasons_to_reject": "* The paper only presents results for one setting - question answering with knowledge retrieval tools\n * The use of self-play for generating a training signal is not new - not even for language agents - e.g. consider alpha zero, or the thread of work using Hanabi as a multi-agent benchmark for LLM collaboration and reasoning. This body of literature is completely missing in the discussion of related work. The paper needs to engage with this prior work to provide better context for the novelty of the current contribution.\n * The discussion of past studies on LLM confidence calibration is very brief. How does this paper shed new light on e.g. previous results that LLMs seem to have a degree of self-calibration in their own answer confidence? E.g. see Tian for a jumping off point in the literature [1]\n * Some details of the framework are not clearly explained (see Q's to authors below).\n * The CSP framework (Alg. 1) seems like it would be (very) inference intensive. A token level complexity analysis or even empirical results describing the token budget or compute cost scaling is missing from the paper, but is needed to provide a wholistic picture of the effectiveness of the method.\n * The current paper only considers one LLM - gemma2-9B (I believe).\n\n### References\n[1] Tian, Katherine, et al. \"Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.\" arXiv preprint arXiv:2305.14975 (2023).", "questions_to_authors": "* The nature of the orchestrator is not clear. Is this another language model? The same model as the self-play-ers? A hard-coded policy? \n * 'If the agent posers a question,...' (Ln 113) how is this determined? Regular expression parsing? Are you looking for certain literal token sequences (like `#ASK`?)\n * What is the middle circle operator on Line 146?\n * It seems like the most important immediate direction the current work could be extended is to add search-then-hedge and guess-then-answer actions - this would also teach the models when to trust their tools or their internal knowledge, right? Have you tried such experiments?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies how to train LLMs to (1) choose between relying on parametric knowledge or external tools, (2) decide whether to trust the outputs of these tools, and (3) determine when to abstain or hedge. The authors propose a novel training paradigm based on collaborative self-play, where one agent receives the question and, together with two other agents—each with access to a different external corpus—collaborates to provide answers.

The reviewers (VkVL, GEzH, osPm) generally agree that the paper is well-written and well-motivated. Reviewer T7Se highlights the novelty of using collaborative self-play to improve tool usage and uncertainty calibration. The reviewers also find the empirical results and analysis to be convincing (VkVL, GEzH, T7Se, osPm).

Similar to all the reviewers, I personally also like the idea of this paper to leverage the power of multi-agent collaboration. I would be more than happy to express my support for this work.
Decision: Accept