Incentivizing Truthful Language Models via Peer Elicitation Games
Authors: Baiting Chen
Tong Zhu
Jiale Han
Lexin Li
Gang Li
Xiaowu Dai
Venue: NeurIPS 2025 poster ()
Keywords: Large language models
Mechanism design
Nash equilibrium
Peer prediction
Primary area: theory

Abstract: Large Language Models (LLMs) have demonstrated strong generative capabilities but remain prone to inconsistencies and hallucinations. We introduce Peer Elicitation Games (PEG), a training-free, game-theoretic framework for aligning LLMs through a peer elicitation mechanism involving a generator and multiple discriminators instantiated from distinct base models. Discriminators interact in a peer evaluation setting, where utilities are computed using a determinant-based mutual information score that provably incentivizes truthful reporting without requiring ground-truth labels. We establish theoretical guarantees showing that each agent, via online learning, achieves sublinear regret in the sense their cumulative performance approaches that of the best fixed truthful strategy in hindsight. Moreover, we prove last-iterate convergence to a truthful Nash equilibrium, ensuring that the actual policies used by agents converge to stable and truthful behavior over time. Empirical evaluations across multiple benchmarks demonstrate significant improvements in factual accuracy. These results position PEG as a practical approach for eliciting truthful behavior from LLMs without supervision or fine-tuning.

Incentivizing Truthful Language Models via Peer Elicitation Games
Baiting Chen" UCLA brantchen@ucla. edu
Tong Zhu" UCLA toz015Qucla. edu
Jiale Han UCLA jialehanducla.ene edu
Lexin Li
Gang UCLA
Xiaowu Dai UCLA daix@ucla, edu
UC Berkeley lexinli @wwereele.eeeee edu
vli@ucla.edu
Abstract
Large Language Models (LLMs) have demonstrated strong generative capabilities but remain prone to inconsistencies and hallucinations. We introduce Peer Elicita- tion Games (PEG), a training-free game- thherethe framework for aligning LLMs through peer elicitation mechanism involving generator and multiple discrim- inators instantiated from distinet base models. Discriminators interact in peer evaluation setting, where utilities computed using deceeriinant-base mutual information score that provably incentivizes truthful reporting without requiring ground-truth labels. We establish thcoretical guarantees showing that each agent, via online learning, achieves sublinear regret in the sense their cumulative perfor- mance approaches that of the best fixed truthful strategy in hindsight. Moreover, we prove last-iterate convergence to truthful Nash equilibrium, ensuring that the actual policies used by agents converge stable and truthful behavior over time. Empirical evaluations across multiple benchmarks demonstrate significant improve- ments in factual accuracy. These results position PEG as a practical approach for eliciting truthful behavior from LLMs without supervision fine-tuning,
Introduction
LLMs have achieved remarkable progress in natural language generation, reasoning, and few-shot learning [62 D 37 32 Despite these advances, they remain fundamentally limited by hallucina- tion the generation of outputs that are inconsistent with previous responses or factually incorrect 25). common failure mode is inconsistency: LLMs may produce different answers to semantically equivalent prompts due to stochastic decoding or sensitivity to minor variations in 12 .nnother major challenge lack of truthfulness, where outputs are not semantically accurate or aligned with verifiable facts, even when the relevant knowledge is implicitly encoded in the model 3 17. These limitations are particularly concerning high-stakes domains such scientific discovery, education, and decision-making, where outputs must be both consistent and truthful [59] 255. This motivates a central question: How can we reliably elicit consistent and truthful behavior from LLMs?
phrasing [6] 19
To address this question, growing line of work has explored post-training alignment techniques, such supervised fine-tuning and reinforcement learning with human feedback 4 52
54
While
Equual contribution.
t Address for correspondence: Xiaowu Dai. Department of Statistics and Data Science, UCLA, 8125 Math Sciences Bldg #951554, Los Angeles, CA 90095, USA. Email: daix @ uuu...edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

these methods can be effective, they are typically computationally intensive, require extensive human annotation, and lack theoretical guarantees of truthful behavior [3 47 52). Their dependence on
model internals also limits their scalability and transferability across different L.MSs 9 61
An alternative line of research draws on game-theoretic frameworks that aim to improve LLM reliability through structured multi-agent interactions [13 27 These approaches offer the advantage of training-free alignment using only black-box access to LLMs. For example, the consensus game aligns a generator and a discriminator by rewarding agreement between their outputs 227. However, because the objective is based on mutual agreement, it can lead uninformative or collusive equilibria, where agents reinforce each other's 's responses even when those responses are factually incorrect.
We propose Peer Elicitation Games (PEG), training-free LLMss through structured peer evaluation. In PEG, a generator produces candidate responses to prompts, and multiple independently instantiated L.LMS act discriminators. Each discriminator assesses the generator's is output and is, in turn, evaluated by the other discriminators- who serve as peer referees. This mutual evaluation mechanism assigns utilities based on the level of agreement among discriminators, encouraging truthful reporting without relying on ground-truth labels. This incentive structure ensures that truthful behavior by each discriminator constitutes a Nash equilibrium, while discouraging collusion or uninformative consensus. To implement the framework, we apply the online mirror descent algorithm to iteratively update each discriminator's s policy, enabling the system to converge toward equilibrium through repeated, utility -riven interaction. The majority vote among discriminators is then returned to the generator, serving as feedback signal for improving future generations. The design of PEG PFG-introducing multiple LLM discriminators and rewarding them through peer evaluation-is inspired by renowned concepts in biology, including cognitive synergy [38, 55 and collective intelligence [64] 77,, where diverse agents, each holding partial or noisy information, can collectively arrive at judgments that surpass what any individual could achieve alone. PEG enables language models to self-organize toward truthful and stable outputs using only local incentives. This collective dynamic offers a scalable and supervision-fii approach to building more trustworthy LLM systems.
theorete framework for aligning
To summarize, the main contribution of our work is as follows:
We propose PEG, training-free free framework for eliciting truthful behavior from LLMs, without relying on ground-truth labels or model fine-tuning. The framework casts the interaction between generator and multiple heterogeneous discriminators as a multi-agent peer evaluation game as depicted in Figure
We provide theoretical guarantees showing that PEG promotes truthful reporting as a Nash equilibrium. Furthermore, when each discriminator updates its policy using online mirror descent, the system achieves sublinear regret and converges to a truthful Nash equilibrium over repeated interactions.
Through experiments on a range of benchmarks, including ARC, MMLU, and GPQA, we demonstrate that PEG improves factual accuracy by over 10% compared to existing methods. Additionally, our results show that smaller models (e.g., 7B parameters) using PEG can match or outperform much larger models (e.g.. 65B).
Related Work. This work relates to several lines of research. First, game-theoretic frameworks have been explored to improve reasoning and aligmment in LLMs through structured interactions (27), which promotes self-consistency by reconciling the outputs of a generator LLM and a discriminator LLM through game-theoretic interaction. Follow-up studies extend this perspective decoding (TT) federated learning with competing interests (65), and embodied tasks like vision-language navigation [67]. Our work builds on this foundation by introducing a multi-agent formulation with explicit peer interaction and a utility mechanism that promotes truthful reporting rather than mere agreement. Second, a growing body of research demonstrates that multi-agent systems of LLMS can collaborate through debate and cooperation to enhance factuality and task performance [6 13| 44). Our approach draws on this idea by coordinating multiple LLMs through structured peer evaluation. Third, our work connects to the literature on learning Nash equilibria in multi-agent systems, particularly no-regret learning frameworks such as Nash learning from human feedback 41 and direct preference optimization [58) We contribute to this literature by showing how no-regret learning dynamics converge to truthful equilibria in a structured, incentive-compatibl setting. Finally, PEG builds on peer prediction mechanisms
[13] Most relevant is the Consensus Game framework
learning in general-sum games [24]]
60 g
and preference-bel
[3] 39), which

---

Game (Single Agent)
Discriminstor: guees the cápital Pari align with the corretness label, I can recelve utility
Agreed but incorrect Answer
bry
Question: What is the capital of France? A. Beijing London C. Paris New York
provide Correct answer.
Peer Elicitation Game (Multi Agent)
Generator: think the capital of France is London!
Truthful and Correct Answer!
Multiple Diacriminatoos: "My utility based on muluel evaluetion with a random referee. The optimal strategy is to report truthfully without aligmment"
Figure 1: Comparison of the consensus game and PEG: when multiple discriminator LLMs indepen- dently evaluate the generator's output and are rewarded based on mutual agreement, their collective judgment aligns more closely with true answers.
We adapt these ideas to LLM agents by extending multi-task peer prediction mechanisms to a setting where multiple LLMs jointly evaluate responses via mutual scoring and incentive-aligned interaction.
have been extended to handle complex settings 63 46
68 49).
33
2
Methodology
This section introduces the key components of PEG including a formal definition of truthfulness and overview of the interactions between the generator and multiple discriminator agents.
2.1 Truthfulness as Incentive Compatibility
We adopt the concept of incentive compatibility (IC) from mechanism design 42 to define the truthfulness. IC ensures that each agent maximizes its utility by reporting its true private information. Specifically, let 6, denote the true private information of agent i, and r, a possible report. In our setting, the agents discriminators, each independently evaluating generated response. The true private information e E {0, 1} refers to the agent's 's truthful judgment on whether the response is factually correct (c, 1) incorrect (c, = 0). The report E {0, 1) is the label that the agent chooses to submit to the system based on (or possibly deviating from) this truthful judgment. Let c-s represent the truthful information of all other agents, The mechanism assigns utility u(rr, to agent based on its own report and the reports of all other agents. A mechanism said to be incentive compatible (IC) if, for all agents i, all ct C- and all possible reports Ti.
u,(c; c-+) 2 s-s).
(1)
The consensus game framework (27 involves one LLM generator and one LLM discriminator, aiming to improve consistency by rewarding agreement between agents. However, this structure does not guarantee truthfulness, as agents are rewarded only for agreement rather than accuracy and may benefit from reporting false but mutually agreeable outputs. For example, the generator produces an incorrect answer to question when tasked with generating a correct one. The discriminator, whose truthful evaluation would to mark the answer as incorrect (c; = 0), may instead receive higher utility for agreeing with the generator by reporting of truthful reporting is lower than that of misreporting, i.c., u,(c,, c_y) < u, c-1,, which creates a clear incentive for the discriminator to misreport. This violates the IC property defined in Eq. (.) highlighting a fundamental limitation of consensus-base approaches..
correct (r; = 1). In such a scenario, the utility
In contrast, recent studies have shown that multi-agent debates can better integrate the diverse perspectives of multiple models, leading to more accurate and reliable outputs [I0 [5]. Motivated by these findings, we propose PEG in which multiple discriminator agents independently evaluate each generated response. In this setup, each discriminator rewarded based on agreement with its peer discriminators. We show that when all other discriminators report truthfully, the best response for any individual discriminator is also to report truthfully- thus, truthful reporting becomes Nash equilibrium. That is, PEG achieves IC property defined in where agent improve its utility
3

---

by deviating unilaterally. Figure Iillustrates key distinction between our PEG and the consensus game, where the generator and a single discriminator are incentivized align with each other, which can lead to consistent but incorrect outputs. In contrast, PEG relies on independent mutual evaluations from multiple discriminators, promoting outputs that are not only consistent but also correct outputs.
2.2 Peer Elicitation Games (PEG)
We consider a generator and set of n discriminators D1. D D,. Each generator and discriminator maintains a probabilistic policy. At each round & E (1, 7, the system assigns set of tasks indexed by k 1,2,..., K,, where K2 is the number of tasks in round For each task (t, ke), the generator receives an input question Xt,,, along with correctness label Vi,ks E {0, 1}. and produces a response Yi,,, according to its probabilistic policy, The generator's policy conditional distribution over responses given the input and target label:
G (YE,, Xt,ks V1,,,) IP((i,,, Xt,ke Ve,ki
where f denotes the generator's parameters.
Each discriminator D, observes Xt,k, and the corresponding generator response Yi,,, and outputs a predicted label Vi,t, E [0 according its policy:
TD. (V,,, Xt.ke Y,,x)) PVVV,, Xe,ke Yikii de);
where Φi denotes the parameters of discriminator D;-
The goal is to generate responses that are both consistent and truthful. Our method achieves this by aligning the generator's output with the majority judgment of discriminators, while incentivizing truthful evaluation from discriminators through peer evaluation. The overview of our method is illustrated in Figure 2 The left branch (in red) corresponds the supervised alignment goal of imitating majority vote labels, while the right branch (in blue) depicts the PEG that ensures incentive compatibility among discriminators.
Step1: Response Generation and Discriminator Evaluation
Step2: Peer Elicitation Game
Correctnees Label
Question
Questions List
Assign K, Tasks
Generator
Answer
Step3: Policy Update
Majority
Judgement v.,
Step4: Generator Update
Goal: Align with Majority Vote
Goal: Incentive Compatibility
Figure 2: Overview of our method: multiple discriminators independently evaluate the response provided by the generator, while each discriminator is rewarded based mutual agreement with peers via PEG. This setup incentivizes truthful reporting for discriminators and aligns the generator without requiring ground-trute labels.
Step 1: Response Generation and Discriminator Evaluation. At each round, given an input ques- tion Xr.k, and a correctness label Vi,a, the generator produces a response Y,,,, Ta (-(--) Ve,,, ). Each discriminator independently evaluates the response based the question and outputs correct- ness report Vi..k. TD. (-)X,, Yr,.,)
To incentivize truthful reporting, the discriminator agents engage in peer clicitation game, where the utility each agent based on a mutual evaluation. Impor- tantly, this utility requires only reports from discriminators and does rely access to ground truth. Assume that round f, all discriminators are assigned K, tasks. For each task k2 E [.], discriminator i privately observes signal C. (0, 1} and submits report V Vi,kk, {0, 1). The task set {l,..., Ke} is arbitrarily partitioned into two disjoint subsets KJ,, and K2,1- For every pair of distinct agents * [n]), construct two 2 x 2 co-report matrices M and M for
Step 2:
Peer Elicitation Games.

---

each subset Ke, where l = 1,2. Each entry of M (c. (c,c)) counts the number of tasks ky Ke,, for which agents and j reported the pair (c, c.), i...
M (2, = Σ
((Vaae V.s.k)) (c,)) (c,dd))
The total payment to agent in round t is defined as the sum over all other agents of the product of the determinants of these matrices:
Ps,t Σ
det((M).)) det(
374
In our setting, the discriminators are rewarded based on mutual evaluation. Meanwhile, the generator incentivized to produce responses that align with the majority consensus among high-confidence discriminators. The utility functions are computed over a batch of K, tasks in round l. The overall utility functions are defined as:
"D: Vi Xede Ya.ke [pral,
(t)
Vi (1. (1. [(Vi. Vo, Venni].
UG
(t)
Ey a-aammaa--aa...m v.
where k; E {1,..., K. indexes the tasks in round 1, and Vr, denotes the majority vote label aggregated from the discriminator reports task (4,
Step 3: Policy Update. We assume that the policy space consists of conditional probability distri- butions over inputs, and focus a localized subset around reference policy π² (e.g.. truthful reporting strategy) [50]. This constraint reflects the intuition that although policies may adapt to optimize for high utility, they should not obviate too far from truthful behavior to maintain semantic consistency and interpretability. In practical terms, this can be enforced by initializing the agent with fine-tuned model that embodies л², and constraining updates to remain within a trust region 45 69]. Specifically, we define the local policy neighborhood as:
1eat {T(- input: 0) 11 || input: 0) -- +"(- -="-|input)]| < 6).
(3)
Our goal is to iteratively learn the policy that maximizes the utility function, aligning with the online learning framework that sequentially optimizes objective function by searching for its critical point 51]. Since each policy is a probability distribution, we adopt the Online Mirror Descent (OMD) algorithm with negative entropy the Bregman divergence, which naturally preserves the probabilistic structure of the policy and regularizes each update by penalizing large deviations from the previous policy [16). The update rules for the discriminators and generator are:
(e'+1) ( (Vi.,, Ne.kr Ye,s.) x D. (Vi..,, Ne,ke Ye,ee) exp D.
(t')
Te
V ce' (V
D.
( T
(1) (Ye,,) Xe,ke, Ve,,,) exp Nekke Ve,.,) x aG
T V To (X,,,, Vark
se'
(4)
Intuitively, these updates guide each agent adjust its policy to direction yields higher utility. After updating, the individual judgments (Vi,ek...... Vn,.,A,} } from all discriminators on task (t, ky) are aggregated via majority vote to form consensus label Vr,,, which serves as proxy for correctness.
Step 4: Generator Update. The utility of the generator is determined by whether its output aligns with the consensus label derived from the discriminators. Specifically, the generator receives utilies when its generated response matches the consensus label Vi This design enables learning without requiring supervised fine-tuning or access explicit ground trrth correctness labels.
Theoretical Guarantees
In this section, we present three main theoretical results: (i) the mechanism incentivizes dominantly truthful reporting in Section 3.1 (ii) both the generator and discriminators achieve sublinear regret
under online learning dynamics in Section 3.2, and (iii) last-iterate converges to a truthful Nash
equilibrium in Section 3.3 Here, last-iterate convergence means that the policy used in the final iteration converges to the equilibrium, rather than requiring averaging over past iterations (6].
5

---

3.1 Dominant Truthfulness
PEG satisfies the dominant truthfulness property, meaning that truthful reporting is a dominant strategy for each diseriminator, as it yields the highest expected utility regardless of the strategies chosen by other agents. This ensures that truthful behavior is consistently incentivized. We now formally state this property of PFG.
Lemma Let a be the number of agents (e.g., discriminators) and Ke be the number of tasks assigned in round defined in Section 2.2 When 2 2 and K, 2 4, under mild assumptions, PEG is dominantly truthful and satisfies IC in Eq. (T. That is, for every agent i, the truthful reporting strategy maximizes their expected payment regardless of the strategies chosen by other agents.
The proof of this lemma, which leverages ideas from [3 is provided in Appendix A.I The design of PEG is grounded in three principles to ensure this incentive guarantee. First, the use of determinant-ban
the determinant achieves its maximum value when agents report truthfully, making truthful reporting the most rewarding strategy. Second, the utility function acts as an unbiased estimator of the joint distribution of agent reports, allowing the mechanism to approximate the true distribution without estimation error. Finally, to prevent negative payments, the set of tasks divided into two disjoint subsets, and payments are computed using the product of determinants from these subsets. As both subsets serve unbiased estimators of the distribution of agent reports, their determinants are expected to have the same sign. As a result, the payment is always non-negative
utility ensures
coommation-moottonicce
3.2 Regret Analysis
Next, we show that both the generator and the discriminators can progressively improve their behavior such that their cumulative performance asymptotically approaches that of the best truthful policy by performing online policy updates as defined Eq. 44) in Section 2.2 This is formalized via the standard notion of nw-regret learning, which measures the cumulative difference between the utility obtained by the learned policy over time and the utility that would have been achieved by the best fixed policy in hindsight 66 [8] Since the generator and discriminators are updated independently, and their optimal strategy to report truthfully regardless of others, we follow the setting in [27] and define the regret for each discriminator and the generator
T
Regretp, D. (T)
=
W
(t.2') D.
(a
(mD))
- max
u D:
(t.e)) (Tb.),
='=1
1=1
K2
T
K
Regret (T) =
GG (IT(
(1.1' Ae)
(1))
max T
Σ v=1k==
GG
ke)
(Ta),
where (1') "D and "G
(t') denote the policies of discriminator Dj and the generator at iteration i', and
Th T are their respective best fixed policies in hindsight. Each uf (t.ke) (.) denotes the generator's
utility on task k, in round 1.
We introduce the following Assumption for theoretical guarantees. Assumption 1. The utility function up, (...) and ug (TG) satisfies the following:
(Part 1: Local concavity). There exists a neighborhood N around a truthful reference policy such that UDD(TD)) (TD,)) and ug (TG) is concave in TD, and aG for all D. TG E N.
(Part 2: Gradient boundedness). There exist constants M1, M2 > 0 such that for all n, and nG, the gradients are bounded in En-norm: ||VR,, RD. UD, (TD S M,, and || (aauC(T))) S M2.
Part 1 of Assumption I Jssumes that the utility function is locally concave with respect to the policy within restricted neighborhood around the truthful policy x², as defined in Section 2.2 This justified because π² is designed to be maxiiitima under PEG, and policy updates are constrained to stay close to x² in practice via deploying a fine-tuned model. Similar locality and curvature assumptions are standard in trust-region and online learning methods 20). Part 2 of Assumption [ requires that the gradients of the utility function are bounded. These assumptions are standard in the convex optimization literature such as [6] which also employ gradient-based optimization methods.

---

We now formalize the no-regret property of PEG. The following theorem shows that both the generator and each discriminator achieve sublinear regret when updated via mirror descent with appropriately chosen learning rates.
[ and set the learning rate in Eq. (4) for the generator as rf == Theorem 1. Under Assumption|
22DKa ((MPtt)) and for each discriminator D2 as rp == (m (.D.,,.
for 2 S T < 2 Then, the regrets of the generator and each discriminator are bounded by: Regret S
/2 V2-1 2 ) and Regret (T)) S
/2, M2 2 DKL( (lT) -T, respectively, where (-L(- denotes the KL. divergence be- Iween the optimal policy x² and the initial policy x 211,, and KT is the number of tasks at T iteration.
Theorem D guarantees that both the generator and each discriminator achieve a regret bound of O(VT), which is consistent with standard results in online convex optimization and learning theory le.g.. 6 30J. This sublinear regret implies that the policy update defined in Eq. 44 is Hannan consistent: their average regret vanishes T oc, meaning that each agent's 's average performance converges to that of the best fixed policy [28]. More specifically, the regret bound depends on three key factors: (1) the bound on the gradient norm of the utility functions; (2) the number of tasks K; and (3) the KL divergence between the initial policy and the optimal one, which measures how far the agent's 's starting point is from the target behavior. A smaller KL divergence leads to a tighter regret bound, as the learning trajectory begins closer to the optimal policy. Importantly. regret analysis does not imply convergence to the optimal policy, but ensures that the average utility gap to the best fixed policy vanishes as T + co. While regret guarantees are well studied in online learning, applying them a peer evaluation setting with multiple interdependent L.MMs novel. In this setting, agents influence each other only indirectly through their reported outputs, making this a theoretically grounded and promising direction for aligning L.MSs.
3.3 Last-Iterate Convergence
Our third main theoretical result concerns the convergence behavior of agents Nash equilibrium. Specifically, we establish last-iterate convergence, a stronger guarantee that ensures that the actual sequence of policies converges to a fixed point (6)
In our PEG setup, each discriminator interacts with others repeatedly, aiming to report truthful evaluations based on shared signals. These interactions can be naturally modeled continuous multi-agent game G = (N, II, {wr New)). where N' = {l...., n denotes the set of agents, II = IT= 11, is the joint space of stochastic policies, and u, II - IR is the utility function for agent i, reflecting agreement with peers. We assume each u, is continuous, differentiable in its own argument, and has Lipschitz continuous gradients.
Nash equilibrium in this setting corresponds to a stable configuration of policies where no discrimi- nator has an incentive to unilaterally deviate [4). Formally, joint policy n² (Ti, ...,...,.. mH E II is a Nash equilibrium if for every E N and any alternative policy मं E II,
(5) Theorem 2 (Last-iterate Convergence to Nash). Suppose each discriminator E N updates its policy using Eq. 44) with a decaying learning rate of order O(1/t"), for some p E (ž 1). Then, the sequence of joint policies ne converges almost surely to the unique Nash equilibrium x of PEG.
u m-.) 2 ui...
The proof of Theorem 2 provided in Appendix A.3] builds on the Robbins-Siegmund Lemma. It also inspired by the analysis in [16), though our setting is more specific-focuiigg on static games with perfect feedback. Theorem 2 establishee last-iterate convergence to the Nash equilibrium: the actual policies used by each discriminator at the end of training converge to the truthful equilibrium. This is in contrast to classical no-regret learning, which only ensures good average performance over time. Last-iterate convergence is particularly valuable in practice, as it guarantees that the learned policies are not just good on average but are inherently stable and reliable (6] [6] In our case, this means PEG reliably leads to consistent and truthful evaluations, effectively enabling smaller language models to coordinate and reach numan-aligned consensus without any supervised fine-tuning or distillation.
7

---

Table 1: Accuracy (%) of majority vote answers across benchmark datasets for each method. Bold indicates the best performance in each row.
Dataset
MI
ER-D
PEG
ARC-Easy
88.19
84.18
88.61 38.61
88.57
91.78
ARC-Challenge
77.01
70.68
78.03
77.52
87.01
MMLU
59.98
50.75
59.85
59.66
70.73
GPQA-MAN Main
18.08
9.15
16.29
16.52
22.54
4
Experiments
We evaluate PEG on question-ans -nswering (QA) tasks. In this setup, generator LLM agent produces answers, while a group of discriminator LLM agents independently evaluate each response, serving peers to promote both accuracy and consistency. Code for all experiments is available at https (gg.lbbb.cc//oo000/////rr2000000
4.1 Experiment Setup
Models. The main experiment employs the following models: SrepSeck-RI--istilll -wanilBBB Sicepseck-Qwen-75e
(deepseok-Llama-70) and Qwen/Qwen2.5- also serving as the
7B-Instruct (OQwen-7b) discriminators, with Dween/Qwen2.5-7B-Insteee generator. We consider Gemma-7B [53], Mistral-7B [2)]. Yi-Yi [66] and OpenChat-7BB [57] as candidate discriminators. Unless otherwise specified, set the learning rate 1) 0.1 for all experi- ments. The PEG mechanism between discriminators is run for 10 iterations for tasks. Discussions on different choices of learning rates and number of iterations provided in Appendix C.6 and C.7
Prompts.
We evaluate our models with zero-shoting following the format described in 21 By default, conditioning the PLM on (2r, correct) corresponds to the standard zero-shot prompt. Condi- tioning on (x, incorrect) uses the same structure, except that the phrase "Answer:" is replaced with "Incorrect Answer:" in the prompt C.I
Baselines. We incorporate several baseline methods, each representing different approach to generating and selecting responses. To evaluate against our method, we apply each baseline to obtain responses from the discriminators, and then compare their outputs using majority voting [6]]
Generative Ranking (G): A standard baseline that ranks candidate answers by the probabil- ity Pras(y) x, correct) and selects the top candidate [56).
Discriminative Ranking (D): This method employs discriminator an estimate P(corret x, y) and ranks responses accordingly (27].
Mutual Information Ranking (MI): This method reweights each candidate by the product of the forward and reverse likelihoods, PLa (3) 2, correct) PL x. y) (35).
Equilibrium Ranking Discriminator (ER-D): Based on the Consensus Game framework of (27), this method formulates the interaction between generator and discriminator as signaling game. The discriminator iteratively updates its estimates to maximize agreement with the generator. Each query-candidate pair (z., y) is reweighted by D (correct xr, y),
encouraging consistency in the final ranking.
Datasets. We conduct our evaluations using four diverse datasets: ARC-Easy, ARC-Challenge, Massive Multitask Language Understanding (MMLU) and Graduate-Level Google-Prov Proof Q&A Benchmark (GPQA) (21 [1 44).. Details the datasets are in Appendix C.2) Each dataset presents unique challenges, allowing to test PEG under various knowledge domains.
4.2 Experiment Results
PEG Improves Accuracy. Results Table show that PEG consistently outperforms all baselines across the evaluated datasets. Notably, it achieves more than a 10% improvement in accuracy on the most challenging benchmarks, ARC-Challenge and MMLU, compared to the strongest baseline. This performance gain is due to two key factors. First, PEG leverages the complementary strengths and

---

Table 2:
Accuracy (%) of each model before (D) and after applying the PEG mechanism (PEG)
across four benchmark datasets. Bold highlights the best PEG result per dataset.
ARC-Easy
ARC-Challenge
MMLU
GPQA
Model
PEG
D
PEG
PEG
OQwen-7B
90.43
90.89
82.91
85.04
62.72
68.19
16.07
22.77
dcepseek-Qwen-78
70.48
91.73
57.18
86.15
42.38
69.04
15.18
20.09
Jeapsrek-LLaMA-7B
76.30
91.44
59.83
46.23
68.71
17.41
diverse reasoning capabilities of multiple LLM discriminators. We provide illustrative example in Appendix 6aa to further validate that initial divergent outputs will become more consistent and accurate after applying PEG. Second, unlike sgreement-basee methods such MI and ER-D. PEG promotes truthful reporting through mutual information signals, which more effectively elicit the latent capabilities of the models. Table further supports this, showing that each individual discriminator consistently improves in accuracy following policy updates under PEG.
PEG Enables Coordination Among Heterogeneous Agents. Despite substantial differences in architecture and baseline accuracy, Table 2 shows that all discriminators benefit from PEG. Remarkably, even the weakest model, OQwen-7B, which initially has the lowest accuracy among all agents, becomes the most accurate after participating in PEG. This suggests that PEG encourages cross-agent learning, where each model learns align with truthful, high-confidence signals provided by its peers. Figure 3 reports the accuracy of each individual discriminator, as well as the majority vote, after applying PEG. The majority vote achieves notable accuracy gain over any single model, validating the benefits of collaborative learning among heterogeneous agents.
PBG docosock
Accuracy (40
ARC Easy
ARC Challenge
MMLU
GPOA- main
Figure 3: Accuracy comparison between original model outputs (D) and PEG majority vote answers.
Table 3: Accuracy (%) of individual discriminators before and after applying PEG in the discriminator setting.
Discriminators
OQwen-7B
deepseek--wen-7B
deepseek-Llama2-78
Gemma-7B
Mistral-7B
Original
82,91
57.18
59.83
69.32
70.26
After PEG
86.84
84.27
83.76
84.27
82.74
PEG with Varying Number of Discriminators. We further evaluate PEG on the ARC-Challenge dataset with extended discriminator settings: 5-discriminators (adding Gemma-7B 7B (53) and Mistral-
7B 22 and 7-discriminators (further adding Ai-Yi-9p [66] and OpenChat-7B (77). The results
consistently highlight the strong impact of PEG on both individual and collective performance. Notably, the weakest models with the lowest initial accuracy benefit the most from PEG. For example, in TablesBannd and Qwen-7B and LLaMA2-7B gain over 20% improvement after applying PEG, whereas initially stronger models such as OQwen-7B and Ai-Yi-9B exhibit only marginal gains. Nevertheless, all models, regardless of their initial performance, converge toward coordinated outcomes after PEG. These results demonstrate that agents indeed learn from one another and achieve coordination through PEG. Moreover, when varying the number of discriminators, PEG

---

consistently outperforms both initial discriminator majority vote D and ER-D, especially with 3 and discriminators, where achieves over 10% improvement, as shown Table
Table 4: Accuracy (%) of individual discriminators before and after applying PEG in the discriminator setting.
7 Discriminators OQwen-7B
Qwen-7B
Llama2-7B
Gemma-7B
Mistral-7B
Ai-Yi-9B
OpenChat-7B
Original
82.91
57.18
59.83
69.32
70.26
81,28
79.06
After PEG
83.85
75.30
73.08
77.61
79.06
81.62
82.99
Table 5: Overall accuracy (%) comparison of initial discriminator majority vote D, ER-D and PEG under 3-, 5-, and 7-discriminator settings. Bold indicates the best performance each row.
Setting
ER-D
PEG
3 Discriminators
70.68
77.52
87.01
5 Discriminators
71.71
76.32
86.75
7 Discriminators
76.50
81.54
81.97
Finally, observe slight decrease in accuracy when expanding to seven discriminators. This can be attributed to a mild violation of the conditional independence assumption (Assumption f in Appendix I. which ensures that the mutual information-bass utility remains informative and non- degenerate. Intuitively, when discriminators are highly similar, their outputs become redundant, i.e., observing one provides little new information beyond the others. As result, the mutual intitmaaiinoo based utilitty degenerates to zero, so truthfulness no longer maximizes the utility. Consequently. the learning dynamics may converge compatible (T). resulting in a reduction in accuracy. Overall, our results indicate that PEG remains robust with up five discriminators; see, Table As practical guideline, recommend using three to five heterogeneous LLMs, which provide a strong balance between performance, stability, and computational efficiency. Expanding beyond this should be done cautiously and only when sufficient diversity and independence among LLMs can be ensured.
suboptimal equilibrium where discriminators
not incentive-
Conclusions
In this paper, we propose a training-free, game-theorede framework for aligning LLMs through multi-agent peer elicitation game. Through mutual evaluations among agents, our peer elicitation game facilitates interactions between generator and multiple discriminators in a way that provably incentivizes truthful behavior. We theoretically show that truthful reporting is dominant strategy for each discriminator. Furthermore, using online mirror descent, each agent achieves sublinear regret, ensuring that its average performance approaches that of the best fixed truthful strategy. The agents' strategies also converge in the last iterate to a truthful Nash equilibrium. Empirically, our framework significantly improves factual accuracy across range of benchmarks and performs competitively with much larger models, highlighting practical direction for deploying lightweight models in coooure-constrainedd environments.
There are several promising directions for future research based on PEG. One is extend PEG to high-stakes settings such as medical decision support, scientific fact verification, and policy-relevant summarization, where truthful and consistent outputs are essential for safety and reliability. Another is to incorporate concepts from game theory and economics, such as reputation systems, repeated interactions, and budget-aware mechanisms, to further enhance alignment and robustness among LLM agents, particularly in open-ended or adversarial environments.
Acknowledgments and Disclosure of Funding
We would like to thank the area chair and four anonymous referees for constructive suggestions that improve the paper. X. Dai supported in part by NIH grant R01 142026, Merck Research Award, and a Hellman Fellowship Award.
10

---

References
[1] ACHIAM, J., ADLER, S., AGARWAL, S., AHMAD, L., AKKAYA, I,, ALEMAN, F. L.,
ALMEIDA, D., ALTENSCHMIDT, J., ALTMAN, S., ANADKAT, S., ET AL. Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08772 (2023).
[2] AGARWAL, A., MANDAL, D.,, PARKES, D. C., AND SHAH, N. Peer prediction with heteroge- neous users. ACM Transactions on Economics and Computation (2020), 1-34.
[3] AZARTA, A., AND MITCHELI, The internal state an LLM knows when it's lying. arXiv preprint arXiv:2304. 13734 (2023).
[4] BAI, Z., WANG, P., XIAO, T., HE, T., HAN, Z.,, ZHANG, Z., AND SHoU, M. Z. Hallucination of multimodal large language models: survey. arXiv preprint arXiv:2404. 18930 (2024).
[5] BUBECK, S., CESA-BIANCII, N., ET AL. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends@ Machine Learning 5, (2012), 1-122.
[6] CAL, Y., AND ZHENG, W. Doubly optimal no-regret learning in monotone games. In Interna- tional Conference on Machine Learning (2023), PMLR, pp. 3507-3524.
[7] CARRILLO, A., JIN, S., LI, L., AND ZHU, Y. consensus-based global optimization method for high dimensional machine learning problems. ESAIM: Control, Optimisation and Calculus of Variations 27 (2021), S5.
[8] CASGRAIN, P., NING, B., AND JAIMUNGAL, S. Deep Q-learning for Nash equilibria: Nash- DQN. Applied Mathematical Finance 29, (2022), 62-78.
[9] CASPER, S., DAVIES, X., SHI, C., GILBERT, T. K., SCHEURER, J., RANDO, J., FREEDMAN, R., KORBAK, T., LINDNER, D., FREIRE, P., ET AL. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research (2023).
[10] CHAN, C.-M., CHEN, W., SU, Y., YU, J., XUE, W., ZHANG, S., FU, J., AND LIU, Z. Chateval: Towards better llm-based evaluators through multi-agent debate. In The Twelfth International Conference Learning Representations (2024).
[11] CHEN, S., HAGRASS, O., AND KLUSOWSKI, J. M. Decoding game: On minimax optimality of heuristic text generation strategies. arXiv preprint arXiv:2410.0396 (2024).
[12] CHEN, Y., YUAN, L.. CUI, G., LIU, Z., AND J1, H. A close look into the calibration of pre-trained language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2023), pp. 1343-1361
[13] CHENG, P., HU, T., XU, H., ZHANG, Z., DAI, Y., HAN, L., LI, X., ET AL. Self-playing adversarial language game enhances Ilm reasoning. Advances in Neural Information Processing Systems 37 (2024), 126515-126543.
[14] CLARK, P., COWHEY, I., ETZIONI, O., KHOT, T., SABHARWAL, A., SCHOENICK, C., AND TAFIORD, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
(2018).
[15] DU, Y., LI, S., TORRALBA, A., TENENBAUM, J. B., AND MORDATCH, I. Improving factuality and reasoning language models through multiagent debate. In Forty-first International Conference on Machine Learning (2023).
[16] DUVOCELLE, B., MERTIKOPOULOS, P., STAUDIGL, M., AND VERMEULEN, D. Multiagent online learning in time-narying games. Mathematics of Operations Research 48, 2 (2023). 914-941.
[17] ELAZAR, Y., KASSNER, N., RAVFOGEL, S., RAVICHANDER, A., HOVY, E., SCHÖTZE, H., AND GOLDBERG, Y. Measuring and improving consistency pretrained language models. Transactions of the Association for Computational Linguistics 9 (2021), 1012-1031.

---

[18] GAO, X., LI, X., AND ZHANG, S. Online learning with non non-convex losses and non-stationary regret. In International Conference on Artificial Intelligence and Statistics (2018), PMLR. pp. 235-243.
[19] HAN, Z., HAO, Y., DONG, L., SUN, Y., AND WEI, F. Prototypical calibration for few- shot learning of language models. In The Eleventh International Conference on Learning Representations (2023).
[20] HAZAN, E. Introduction to online convex optimization. Foundations and Trends@ in Optimiza- tion 2, 3-4 (2016), 157-325.
[21] HENDRYCKS, D., BURNS, C., BASART, S., Zou, A., MAZEIKA, M., SONG, D., AND STEINHARDT, J. Measuring massive multitask language understanding. In International Conference on Learning Representations (2020).
[22] HOI, S. C., SAHOO, D., LU, J., AND ZHAO, P. Online learning: A comprehensive survey. Neurocomputing 459 (2021). 249-289.
[23] HOLT, C. A., AND ROTH, A. E. The nash equilibrium: perspective. Proceedings of the National Academy of Sciences 101, 12 (2004), 1999-40022
[24] HU, J., AND WELLMAN, M. P. Nash g-learning for general-sum stochastic games. Journal of Machine Learning Research 4, Nov (2003), 1039-1069.
[25] HUANG, L., YU, W., MA, W., ZIIOGG W., FENG, Z., WANG, H., CIEN,, Q., PENG, W., FENG, X., QIN, B., ET AL. A survey hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43, (2025), 1-55.
[26] HUANG, X., RUAN, W., HUANG, W., JIN, G., DONG, Y., WU, C.,, BENSALEM, S., MU, R., QI, Y., ZIIAO, X., ET AL. A survey of safety and trustworthiness of large language models through the lens of verification and validation. Artificial Intelligence Review 57, (2024), 175.
[27] JACOB, A. P., SHEN, Y., FARINA, G., AND ANDREAS, J. The consensus game: Language model generation via equilibrium search. In The Twelfth International Conference on Learning Representations (2023).
[28] JAFARI, A., GREENWALD, A., GONDEK, D., AND ERCAL, G. On no-regret learning, fictitious play, and nash equilibrium. In ICML (2001), vol. pp. 226-233.
[29] JIANG, A. Q.. SABLAYROLLES, A., MENSCII, A., BAMFORD, C., CHIAPLO,, D. S., DE LAS CASAS, D., BRESSAND, F., LENGYEL, G., LAMPLE, G., SAULNIER, L., LAVAUD, L. R., LACHAUX, M.-A., STOCK, P., SCAO. T. L., LAVRIL, T., WANG, T.,, LACROIX, T., AND SAYED, W. E. Mistral 7b. ArXiv abs/2310.06825 (2023).
[30] JONES, M.. NGUYEN, H., AND NGUYEN, T. An efficient algorithm for fair multi-agent multi- armed bandit with low regret. In Proceedings of the AAAI Conference on Artificial Intelligence (2023).
[31] KAPLAN, J., MCCANDLISH, S., HENIGHAN, T., BROWN, T. B., CHESS, B., CHILD, R., GRAY, S., RADFORD, A., WU, J., AND AMODEL, D. Scaling laws for neural language models. arXiv preprint wxXiv:2001.08366 (2020).
[32] KOIKE, R., KANEKO, M., AND OKAZAKI, N. Outfox: LIm-generated essay detection through in-context learning with adversarially generated examples. In Proceedings of the AAAI Conference on Artificial Intelligence (2024).
[33] KONG, Y. Dominantly truthful multi task peer prediction with a constant number of tasks. In Proceedings the Fourteenth Annual ACM-SIAM Symposhum on Discrete Algorithms (2020), SIAM, pp. 2398-2411.
[34] KREPS, D. M. Nash equilibrium. Game Theory. Springer, 1989 pp. 167-177.
[35] LI, J., AND JURAFSKY, D. Mutual information and diverse decoding improve neural machine translation. arXiv preprint
(2016).
12

---

[36] LIANG, T., HE, Z., JIAO, W., WANG, X., WANG, Y., WANG, R., YANG, Y.. SHI, S., AND Tu, Z. Encouraging divergent thinking in large language models through ageett debate. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing (Nov. 2024), Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds., Association for Computational Linguistics, pp. 17889-17904.
[37] LIGHTMAN, H., KOSARAJU, V., BURDA, Y., EDWARDS, H.. BAKER, B., LEE. T., LEIKE, J.,
SCHULMAN, J., SUTSKEVER, I., AND COBBE, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations (2024).
[38] LUPPI, A. I., MEDIANO, P. A., ROSAS, F. E., HOLLAND, N., FRYER, T. D., O'BRIEN, J. T., ROWE, J. B., MENON, D. K., BOR, D., AND STAMATAKIS, E. A. A synergistic core for human brain evolution and cognition. Nature Neuroscience 25, 6 (2022), 771-782.
[39] MILLER, N., RESNICK, P., AND ZECKHAUSER, R. Eliciting informative feedback: The peer-prediction method. Management Science 51, 9 (2005), 1359-1373.
[40] MITCHELL, E., NOHI, J., LI, S., ARMSTRONG, W., AGARWAL, A., LIU, P., FINN, C., AND MANNING, C. D. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Proceedings of the 2022 Conference Empirical Methods Natural Language Processing (2022), pp. 1754-1768.
[41] MUNOS, R., VALKO, M.. CALANDRIELLO, D., AZAR, M. G., ROWLAND, M., GUO, Z. D., TANG, Y., GEIST, M., MESNARD, T., FIEGEL, C., ET AL. Nash learning from human feedback. In International Conference on Machine Learning (2024), PMLR, pp. 36743-36766
[42] MYERSON, R. B. Incentive compatibility and the bargaining problem. Econometrica: journal of the Econometric Society (1979), 61-73.
[43] PRELEC, D. A Bayesian truth serum for subjective data. Science 306, 5695 (2004), 462-4642666 466.
[44] QIAN, C., LIU, W., LIU, H., CHEN, N., DANG, Y., LI, J., YANG, C., CHEN, W., SU, Y., CONG, X., XU, J., LI, D., LIU, Z., AND SUN, M. ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Bangkok, Thailand, Aug. 2024), L.-W. Ku, A. Martins, and V. Srikumar, Eds., Association for Computational Linguistics, pp. 15174-15186.
[45] QU, G., WIERMAN, A., AND LI, N. Scalable reinforcement learning of localized policies for multi-agent networked systems. In Learning for Dynamics and Control (2020), PMLR, pp. 266.
[46] RADANOVIC, G., AND FALTINGS, B. Incentives for truthful information elicitation of continu- ous signals. In Proceedings of the AAAI Conference on Artificial Intelligence (2014).
[47] RAFAILOV, R., SHARMA, A., MITCHELL, E., MANNING, C. D., ERMON, S., AND FINN, C. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems 36 (2023),
[48] REIN, D., HOU, B. L.,, STICKLAND, A. C., PETTY, J., PANG, R. Y., DIRANI, J., MICHAEL. J., AND BOWMAN, S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling (2024).
[49] SCHOENEBECK, G., AND Yu, F.-Y. Two strongly truthful mechanisms for three heterogeneous agents answering one question. ACM Transactions on Economics and Computation 10, 4 (2023), 1-26.
[50] SCHULMAN, J., LEVINE, S., ABBEEL, P., JORDAN, M., AND MORITZ, P. Trust region policy optimization. In International Conference on Machine Learning (2015), PMLR, pp. 1889-1897.
[51] SHALEV-SHWARTZ, S. Online learning and online convex optimization. Foundations and Trends in Machine Learning 4, (2012), 107-194.
[52] SUN, Z., SHEN, S., CAO, S., LIU, H., LI, C., SHEN, Y., GAN, C., GUT, L., WANG, Y.-X., YANG, Y., ET AL.. Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics ACL 2024 (2024), pp. 13088-13110.
13

---

[53] TEAM, G., MESNARD, T., HARDIN, C., DADASHI, R., BHUPATIRAJU, S., PATHAK, S., SIFRE, L., RIVIÈRE, M.. KALE, M. S., LOVE, J., ET AL. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08233 (2024).
[54] TONMOY, S., ZAMAN, S., JAIN, V., RANI, A., RAWTE, V., CHADIIA, A., AND DAS, A. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint 2 6 (2024).
[55] TONONI, G., SPORNS, O., AND EDELMAN, G. M. A measure for brain complexity: relating functional segregation and integration in the nervous system. Proceedings of the National Academy of Sciences 91, 11 (1994), 5033-5037.
[56] TOUVRON, H., LAVRIL, T., IZACARD, G., MARTINET, X., LACHAUX, M.-A., LACROIX, T., ROZIÈRE, B., GOYAL, N., HAMBRO, E., AZHAR, F., ET AL. Llama: Open and efficient foundation language models. arXiv preprint arX7 (2023).
[57] WANG, G., CHENG, S., ZHAN X., LI, X., SONG, S., AND LIU, Y. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309. 11235 (2023).
[58] WANG, M., MA, C.,, CHEN, Q., MENG, L.,, HAN, Y., XIAO, J., ZHANG, Z., HUo, J., Su, W. J., AND YANG, Y. Magnetic preference optimization: Achieving last-iterate convergence for language model alignment. arXiv preprint arXiv:2410. 16714 (2024).
[59] WANG, T., JIAO, X., ZHU, Y., CHEN, Z., HE, Y., CHU, X., Gao, J., WANG, Y., AND MA, L. Adaptive activation steering: A tuning-fres free Ilm truthfulness improvement method for diverse hallucinations categories. In Proceedings of the ACM on Web Conference 2025 (2025), pp. 2562-2578.
[60] WANG, X., AND SANDHOLM, T. Reinforcement learning to play an optimal nash equilibrium team markov games. Advances in Neural Information Processing Systems 15 (2002).
[61] WANG, X., WEI, J., SCHUURMANS, D., LE, Q. V., CIH,, E. H., NARANG, S., CHOWDHIERY, A., AND ZHOU, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations (2022).
[62] WEI, J., WANG, X., SCHUURMANS, D., BOSMA, M., XIA, F., CHI, E., LE, Q. V., ZHOU, D., AL. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824-24837.
[63] WITKOWSKI, J., AND PARKES, D. A robust bayesian truth serum for small populations. In Proceedings of the AAAI Conference on Artificial Intelligence (2012).
[64] WOOLLEY, A. W., CHABRIS, C. F., PENTLAND, A., HASHMI, N., AND MALONE, T. W. Evidence for collective intelligence factor in the performance of human groups. Science 330, 6004 (2010), 686-688.
[65] YOON, T., CHOUDHURY, S., AND LOIZOu, N. Multiplayer federated learning: Reaching equilibrium with less communication. arXiv preprint arXhv:2501.0866 (2025).
[66] YOUNG, A., CHEN, B., LI, C., HUANG, C., ZHANG, G., ZHANG, G., WANG, G., LI, H., ZHU, J., CHEN, J., ET AL. Yi: Open foundation models by 01, ai. arXiv preprint arXi::2403.046522 (2024).
[67] Yu, B., LIU, Y., HAN, L., KASAEI, H.,, LI, T., AND CAO, M. VIn-game: Vision-language equilibrium search for zero-shot semantic navigation. arXiv preprint arXiv:2411. (2024).
[68] ZHANG, P., AND CHEN, Y. Elicitability and knowledge-free elicitation with peer prediction. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems (2014), pp. 245-252.
[69] ZHANG, Y., QU, G., XU, P., LIN, Y., CHEN, Z.,, AND WIERMAN, A. Global convergence of localized policy iteration in networked multi-agent reinforcement learning. Proceedings of the ACM Measurement and Analysis of Computing Systems 7, 1 (2023), 1-51.
14

---

Appendix
Appendix A
contains the proofs of the theoretical results presented in this paper. Specifically,
Appendix A. presents the proof of Lemma [ Appendix A.2 provides the proof of Theorem [ and
Appendix A.3 3 includes the proof of Theorem 2 Appendix B provides the pscudocode of the PEG
algorithm. Appendix Coffers additional experimental details related to Section 4
A Proofs
A.1 Proof of Lemma
Formally, we consider a binary-choice setting, where each task consists of two possible outcomes, denoted as {0, 1}. There are agents. Each agent is assigned K binary-choice tasks, where for each task k, agent receives a private signal ci E {0. 1) These private signals for all agents are drawn from a joint unknown prior distribution U E A where 4/r, represents the set of all measurable distributions over {0, 1)"" For the same task, the private signals of different agents are correlated. For different tasks, the private signals of the same agent independent. In our setting, discriminators do not incur effort to acquire their private signals.
For multi-task peer evaluation mechanism, the agents do not know the specific realization of the prior distribution before receiving the private signals. After receiving the private signal c/ each agent is required to report rž, which may or may not reflect their true signal. The mechanism is designed to incentivize agents to truthfully report their signals.
Assumption 2. [A Priori Similar Tasks] We assume that all tasks are drawn from a common unknown prior distribution Un] such that U|/]] = Un) for all tasks
This assumption states that all tasks are fundamentally similar, implying that the signals across different tasks should follow the same joint distribution. Specifically, the signals ex for each task & are assumed be i.i.d.. While traditional single-task peer evaluation typically assumes a homogeneous prior, this multi-task setting allows for heterogeneity in agents' beliefs as long as the tasks themselves are similar. In our setting, this assumption implies that the problems are similar in the sense that the ground truths follow the same joint distribution within each batch at every round. We enforce this by ensuring that the same set of subjects and the same set of problems are presented each round.
Definition 1 (Strategy). Each agent i's strategy for reporting is mapping from her private signal k to a distribution over possible reports r. Formally, a strategy can be represented as a function Sk [0, + 1[), where St gives the probability distribution over the possible reports r, conditioned on receiving the private signal ci.
Every strategy S corresponds to a 2 x 2 transition matrix where 5 (c/, r.) is the probability that agent i reports rf given that she receives private signal d A strategy is truthful if the agent always reports r cf. Agent plays a truthful strategy if for every task k. St is an identity matrix.
Assumption 3 (Consistent Strategy). Each agent plays the same strategy St for all tasks.
This assumption reasonable because agents face structurally identical tasks drawn from the same distribution, with no task- lask-specific information to adapt to.
Assumption (Conditional Independence). We assume that agents' private signals C12 C2 C. are independent conditioning on ground truth. Since agents' strategies are independent, this also implies that agent's 's reports č1, ć2 C. are independent conditioning on ground truth.
In our case, the discriminators (agents) are instantiated independently and process inputs separately, so their reports naturally conditionally independent given the underlying truth.
Definition 2 (Informative Peer). Agent and agent j are considered each other's informative peers if the determinant of the joint distribution matrix over their private signals e, and ej is non-zero, Le., det,(Ue,))) # 0, where Uz,,, represents the joint prior distribution of the signals e; and cj. and Uer,z is expressed in its matrix form.
Definition 2 captures whether two agents are "informative peers" by examining the structure of their shared information. If their private signals are sufficiently correlated (as captured by a non-zero
15

---

determinant of the joint distribution matrix), they can serve as reliable references for each other in PEG.
Definition 3. Given two random variables X, Y which have the same support C, we define the determinant mutual information (DMI) between X and as
DMI(X: Y) = (dct(Uxxy)))
Lemma (Strict Information Monotonicity). For every two random variables X, with the same support C, when X' is less informative than X, i.e., X' is independent of conditioning X, it holds that
DMT(X'; Y) & DM)(X;
The inequality strict when det(Ux., (xx.) + and U x x is not permutation matrix.
Lemma's 3. Let / be the co-occurrence matrix formed by agents and j over the set of tasks Te, and define the ). Then the expectation of this score is unbiased estimator of the determinant mutual information between X, and y;
E Xx,X, [del(M?2) az det xx.xx))
where 0g = ( |C]! and U x, is the co-occurrence matrix of the random variables X, and X
The proofs of Lemmas and 3 follow the argument of Theorem 5.1 in 33 |Building on these results, we now present the proof of Lemma
Proof. Let agent truthfully report X, = X,, and let other agents report signals Xy. Consider the interaction between agent i and any peer agent j. The DMI between X; and X, satisfies:
DMI X3) & DMI(X;;
with equality if and only if X, permutation of X,. particular, if X, =x (agent reports truthfully), then:
DMI(X;; X2) S DMJ(X,; and the inequality is strict if X; is not permutation of X,, due to the strict information monotonicity of DMI. Therefore, the truthful strategy maximizes DMI against any truthfully reporting peer.
From Lemma know that the utility between agents i and j, based on co-occurrence matrix M satisfies:
Efde)(MM)) = a det (U x,,x,);
where ae is a constant depending on the number of tasks in round f, and U x, is the empirical joint distribution matrix. Therefore, the expected squared utility proportional to:
[E(det(MM)) 2 DMI2(X.. Xz),
and is maximized when X; X4, i.e., the agent reports truthfully. Since each agent's expected utility (aggregated over peers j * i) is weighted sum of DMI2 (X,; Xyy, and each term in the sum is maximized by truthful reporting, follows that truth-telling when all agents report truthfully, the resulting strategy profile forms equilibrium, and any deviation leads to a strictly lower expected utility unless the deviation is a permutation of the truth.
dominant strategy. Furthermore.
A.2 Proof of Theorem
To maintain notational clarity and avoid redundancy, we use t to replace the notation t' used in the main text Appendix A.2 and A.3
Definition 4 (Bregman divergence). Let Ω + R be differentiable and u-strongly convex (p 0) function with respect to a norm ||- that is, satisfy
p(x) 2 p(y) + (V:(y)) 3) + 1 21 Vz, y S.
The Bregman divergence centered in y E Q is the function Dx |l v) defined as
Dz(x v) p(z) p(v) (V)())) v).
16

---

Definition 5. Let f Ω - R. be a convex function, and let Dup be a Bregman divergence. The proximal mapping (or proximal step) for a point it with step size q > 0 is defined as:
(oxx)(nV((z)) 24) arg min (n (1x)) σ) + D)(x)laz))
sEn
The mirror descent algorithm is then given by the iterative update:
21+1 Prox, f
Lemma 4. Let To(y)>) be a policy and R(y) be utility function independent of t. Define the objective function:
J(0) (l) [R(y)]-
Then, the gradient of (0) with respect
[J) E ()) [R(y) log (o(l)))
Proof.
VoJ(0) Vo / no [o(y))) dy = VoTToylo)) dy
(b())) [oly|=)Vo (oglo)) R(y) dy
Bynao((k)) [R(y) V log mo (ylx)]
Corollary 3 (Gradient of utility Functions). Using Lemma f the gradients of the utility functions with respect to the discriminator and generator parameters are given by: For each discriminator D;:
Voo, UD. (Ira--
Σ dct( ddt((M Ven, log TD, (Fi 8kk) i
For the generator G:
V 0g UG Byg
[1( Voc log (o(Uu
When Ω = A is the set of full-support distributions over objects and the is set the negative entropy function, which 1-strongly convex with respect the norm || ll the corresponding Bregman divergence is the Kullback-Leible (KL) divergence.
Lemma' 5.
Lemma 6. Let x' = Proxx(9,2 x) be the proximal update. Then, all E Q we have:
(-9.9 - x') S -Dp((ulla') D.plull=)= De 1|x)).
By setting y x. this three-point inequality simplifies (-g, x') - -DD(<=ll>'') Dop(1111)).
Proof. The objective function of the proximal step problem is given by
h(z) == (g. 2) + (=(l))>) Q.
The first-order optimality conditions applied the solution x' are therefore
-Vh(x') E Na((x')
- Vp(x') v E Na(x')
(-g Vp(z') + Vp(z). x') s Vy EΩ
(-9. - x') < (Vp(=') Vp(x). Vy
The statement now follows from using the identity
(V<(x')- V x²) Dipl'l=')) (o(()(=) D=(1'1)
which can be checked directly from the definition of Bregman divergence.
17

---

Lemma 7. Let f : R be convex. Each step of the mirror descent algorithm satisfies
1 Dapu || x4) D (1e+1 m4).
[(m)) f(v) (Vf())), r++1) DD x++1) The
T
T)t
Proof. Using linear lower bound property convex functions, we can write f(x)) S (w) (V ( =z) ['v) + (V))), 24 Ie+1) - (VS(2)) x441)-
the other hand, from Lemma 6 applied to the mirror descent step (that is, for the choices g V f()) x' It+1. xy). we have
n VV(z)))) m+41) S - Dp(vllm(+1) + Do(lll=)) - Dp(#44allze)
Hence, dividing by n and plugging into the previous inequality, obtain the statement.
Lemma 8. Let ||- 11 be the norm with respect to which the DGF up is I-strongly convex, and - be the dual norm. If all functions Jx 92 + lR are convex, the regret of online mirror descent is bounded by
T R = Σ
T
min seR
((( +
7
[+(e()))
1=1
1=1
1=
In particular, assuming that all dual gradient norms are upper bounded by G, and setting
for 2' 2
7
M2.
find
V2
Rr S ((ll(( T
V2
Proof. Since functions /t are convex, we can use Lemma
1
fe(x.) &
D (al
Vo E Ω.
17s
1)
Using the Cauchy-Schwarz inequality, we can bound the right-hand side by
1 Th
D 1 D
f((z,) & f((x) + fexx))) || - z++1ll
(=
Tie
Using Young's inequality, as well the 1-strong convexity of the KL divergence, which implies
Du (ae+11l 2
||0141 - xel1?
1/
2//.
fe(x)) S f((x)+ Th 2 f+:)))))
1 (xlla(1te+) Dp(xllaa))= 1 || |1:0-2441122-
| 2x,1
2m
n
m
2m
Rearranging terms, obtain:
fe(x)) S f((x) IV[((())))
-
D (xllate)) 1 Do(ol()t))
mt
Summing over T:..,T::
Σ ( - ((())) S
(Do(ollxz) (xlleet)
m
18

---

The regret incurred by the algorithm is upper bound by the regret incurred in each of the intervals 2 < T < 2 Suppose the algorithm has been run until time 2' < T 2 Hence, the regret is upper bounded by
Regt 1 (M//
Do (x)(oi)
M221 + (V
(2D2(ollsn) 2M22
MM
[(22)
M MT -2 2/)
+
2
2M
1=0
(
1-1
= MM 2
[(V2) + > (v2)*) +
(T 21)
2M
1=0
1=0
In particular, since T < 2/+11
I-1
(x(( x1)
Regr S M
Σ (V2))
Σ (V2)))
+
2
2M22
(
M
(V2)* + Σ (V2)*
2
1==
1=0
(
(V2)
1=1
(V2)
-M/2D((a(la))
1
V/2
s (E 1
Lemma 9. Suppose the utility is defined over a set of K policies, and each policy updated independently via a no-regret algorithm with individual regret bounded by O(VT). Then, the total regret of the system with respect to the best fixed policy in hindsight is bounded by O(v Therefore, when both the generator and each discrininator maintain and update K independent policies, their regret bounds will incur an additional vK factor, yielding overall regret of O(VKT) per agent.
This result that the regret grows O(v KT) when updating and aggregating over K policies is classical result in online learning and multi-armed bandits, as established in [1. Therefore, by combining Lemmas g and [
we complete the proof of Theorem
A.3
Proof of Theorem 2
Lemma 10. If the utility function is concave and differentiable, its gradient is monotone.
Proof. By concavity, the first-order conditions for u at z and give:
u(y) < u(x) + Vu(x) (y - z), u(I) < u(y) + Vu(y) (I y)-
Adding these inequalities:
u(v) + u(z) < u(x) + ( + Vu(x) 1 + vuly) - v).
Simplifying:
(Vu(z) Vu(y)) (x y) 5o.
Thus, Vu is monotone:
(Vu(x)- - Vu(y)) (x - y) 50 Vs, X
19

---

Lemma 11. Let T14+1 be the update given by mirror descent:
T1.41 arg min atlll;
( m.) + Di TT,)).
where Di(:s)) ) is the Bregman divergence generated by a pusttrongy convex function h4. Then for any T E I1 we have
Lan 2
Di T(4+1) S Di(T,, TEa) The (VV((T)), Ti, Ritt -m²)
where Lj 0 is constant that bounds [T+]] due compactness II,
Proof. Let x = Tist x+ Ti,++. 2 n.. and 9i Vu,(T.). The mirror descent update is:
x arg min (-ne m.) + Di(m,; mllll;
By the first-order optimality condition for convex minimization over IT, we have:
(-7 + Vhy(ot) Vh,(z), --)) 0,
which gives:
(9,, 2 - x+) S (Vh((x+)- -
Using the three-point identity for Bregman divergence:
Vh,(2), - = Di(z,)) - D2(2x - Di(xx,xx))
substitute and rearrange:
Di(2, +) & Di(2,)) - The me z 2) + ng x) Di(xt,2)
Now apply Young's inequality:
1xt l012 + jon
T) (gi, x+ -x) -x)= S 2/1
2
By strong convexity, D,(x* 2) 2 ||1x* x1 So:
ne (ggi, x x) S Di(x (x) + 1/² 19. 112
2/1
Substituting back, the Dz(x+, .) term cancels, and obtain:
n 19:112
Di(z, S Di(z,æ) ne (e, xr) +
2/4
Let Li = p sup [u, (1) which is finite due to compactness of 11. Therefore:
Lani
D1(a), T.411) S Di(m) m.)) no (Vuu((T), x, Tx,)) +
2
Next we can move to the proof of Theorem
Proof. We define an energy function that tracks the distance to the Nash equilibrium using Bregman divergences. For each player N, D, denote the associated Bregman divergence:
Di(mi, == hy (m²²) ha(TTe)) (ha((Tr))) o m..e).
We define the total energy function as
E, = Σ Di(Ti m())-
iEN
Each player updates their strategy via mirror descent:
Tatt1 arg min (m---Vudl (m;) n5) + D..)) TTl,
20

---

With Lemma 11
we have:
Lin
Di(a), T(441) & ma,) - (VV(( Ri,t - n;) +
2
Summing over all players, we obtain:
E s E -) (Vu(T)) x²) + CT²
for some constant C > O, where (V,)) = (Vui(T),..... Vun (Te). Because the utility map Vu is strictly monotone, there exists μ such that
(Vu(Tz), =²> 1I|T= "²1|²
Substituting into the previous inequality yields:
E+1 & E. pmllm - 5 + Cni.
This recursion is in the form required by the Robbins-Siegnnn Lemma:
E S E. at + be:
where a, - μu |T 11 2 and b, CT with E be do since d (1/tt)) with 2p 1. Therefore, the Bobbins-Sieggaan Lemma implies that Eg converges almost surely to a finite random variable Esc. and E, n. n 112 < oc. Since E, ni < oc and Σ, 17 c0, || - T- almost surely.
B
PEG Algorithm
B.1 Algorithm
Algorithm 1 Two-Phase PEG Algorithm with Task Batches
Require: Dataset of questions (xx} (organized into batches of size 8 in the experiments); initial parameters 0g for generator; initial parameters {0 for discriminators; learning rate 7).
1: repeat
2: for each batch do
3: Generator samples answers: yk
(-(aa Uk))
4: (Step 1) Discriminator Updates:
5:
for each discriminators D; do
6:
(a) Discriminators provide judgments: Tik ~ TD, TD, (-|2k,, 9k).
7:
(b) Approximate gradients for each Dj using the log-derivative trick:
VeD: UDD
2
Σ
det(M))) det(M
Ven, log TD, (ral (
N
butches
((+1) (c) OMD update for discriminator policies: TD, ox T exp (n Vop, uD.))
8:
9:
end for
10:
(Step 2) Generator Update:
11:
(a) Compute majority vote ig for each question T&k in the batch.
12:
(b) Approximate gradient for the generator:
Vog UG FJ
Σ
10v = iz) Vog log TH (oyyelu vk).
N
batches
13:
(c) PEG update for generator policy: TG
((+1)
at exp (1 Vog
end for
15:
until all policy have been updated
16:
Output: final parameters Og and (0p, M
21

---

B.2 Illustrative example
Peer Elicitation Game (Multi Agent)
My judge to generators' answer is correct, correct.
Question List What Francs? Beijing
judge to generators' answer correct, correct ...
Desitaation
Which property of mineral can be determined just by looking C. weight
judge generators' correct, correct, ...
[.]]
Generator: think
[c, A ...
Based on mutual evaluation reward. need to report truthfully. My judge to generators' answer | correct, incorrect... ...]
Based on mutual evaluation reward. need to truthfully. My judge generators' anewer is correct, incorrect, ...]
Based on mutual evaluation reward. need to truthfully. My judge to generators' answer correct, correct,
After multipea round mutual evaluate, our judge to generators answer is coorrct.. correct.
Figure 4: An illustrative example of PEG's peer evaluation process: (1) The generator answers a list of questions. (2) Discriminators evaluate these answers, with some providing untruthful reports. (3) Determinant-basee utilitics penalize non- truthful discriminators, incentivizing them to align their future reports with the ground truth.
C
Experiments Details
C.1 Prompt
Generator Prompt
The following multiple choice questions (with answers) about Geology. Question: Which property of mineral can be determined just by looking at it?
A) luster
B) mass
C) weight
D) hardness
Answer:
22

---

Discriminator Prompt
You are an expert evaluator of questions about Geology. Determine the proposed answer is correct. Output ONLY A' or 'B'.
Question: Luster is the property of mineral can be determined just by looking at it. Proposed Answer: luster
Is this answer correct? Respond ONLY with: A. Correct B. Incorrect
Answer:
C.2 Datasets
Datasets from different domains are used in the experiments. Details summarized below.
ARC dataset [4] evaluates scientific reasoning, includes both the "easy" and "challenge" sets. consists of 7787 science questions, all nondiagrams, multiple choice (typically 4-way multiple choice). The experiment applies zero-shot setting for ARC, assessing how well the model navigates science-related questions requiring logical reasoning rather than memorized knowledge.
a benchmark with questions across humanities, STEM, and social sciences. It requires the model to demonstrate broad general knowledge. For MMLU, evaluate models following the format described in setting (21).
MMLU 21
is challenging dataset designed evaluate L.IM capabilities and scalable oversight mechanisms. consists of 448 multiple choice questions that cover biology, physics, and chemistry. These questions are intentionally designed to be high-quality and extremely difficult. The experiment will apply the zero-shot setting for GPQA.
GPQA 48
C.3
Computational Resources
Our experiments utilized the following hardware configurations:
Initial Policy Extraction:
GPU: single NVIDIA A100 GPU with 200 GB of memory.
CPU: AMD EPYC 7542 or 9654 processors.
Throughput: Approximately 2.2 iterations per second (it/s) for policy generation per question.
Estimated Runtime: Varies by dataset size, typically requiring 10 minutes for 1000 questions.
PEG Algorithm for Policy Update:
GPU: NVIDIA RTX 2080 Ti.
Runtime: Significantly faster, all datasets under 20 seconds.
23

---

C.4 The impact of batch size
93.0
89
92.3
88
92.0
Accuracy
91.5
86
91.0
85
batch 5ize
(a) ARC Easy
(h) ARC Challenge
72.0
24.0
71.5
23.5
23.0
71.0
22.5
70.5
22.0
69.5
20.5
69.0
20.0
(c)
MMLU
GPQA
Figure 5: Different Batch Size Effect on Majority Vote Accuracy
We conducted experiments four benchmark datasets by varying the discriminator update batch size from to 15 (with being minimum required form valid batch in this setup) to evaluate its impact majority vote accuracy. As shown in Figure 3] the accuracy remains relatively stable across batch sizes, though slight fluctuations can be observed depending on the dataset. For ARC-Easy, the accuracy is consistently high and robust to batch size, suggesting stable discriminator learning on simpler, more homogeneous questions. In contrast, the other datasets display more variability and slight decreasing trend batch size increases. This performance drop may due the greater diversity among questions within larger batches, making it harder for the discriminators to agree and learn from consistent utility signal. These findings align with Assumption which presumes that tasks within batch are drawn from a common underlying distribution. Overall, the PEG algorithm demonstrates robustness to batch size variation when the assumption of task similarity holds.
24

---

C.5
Sample Outputs
Question
Generator's
Ducriminntar #l (0Qwen TB)
Diciminator F2 (deepseck- Qwen-TB]
Diaciminatorr ideepseek- LLaM6-7B)
Majority Vode (91
determinn hen, doody organ aro artrtoro
Wh avvsur of loureee betuet
mouss na sopols
Corret
Connet
Saunoo 20g byshagen perveddc baikelort. glotti
An enter secrbbe sersi mase strte uattrr soygin that prodused?
addee coaessodd
Connec
Coowez
coe
eir tos no and convatte sonn, yettn salec spaio watur ahr spacit
Sha fermution beveodolo
Connec
Cenect
Cenect
Meurr to selirsaiteee
bounddrdee
Connec
Conwet
Coowet
snoniiingg thin?
Image the Mesger end the spoceovi proiide
ivseence furessher ad under tare produced evants
monemmont
Incenet
Aleotuplivnsare En oäter enowuaa per sameneatt whather armuz
sg sypoool like rosence lustend un smalla
Leiggo.
Connect
the resuurce in tiiie sleppor wherk occviry shove
(a) Initial Discriminator Responses
Question
Generator' Answer
Siscieiinttr #1
Dicokninatto
Gisckiinnttor ideepseaks
Mojarit vode
ror oder sl otee eoaiigeeegg
anologeus
Connec
Coneet
hearect
Suppose 20 gefliii tydirgge percudde bostted so?? complenny sie oben itoll iu cater and unclaaee aas. tniteal dtoctiles
oil coypum
20 necccaenno nather addvet
Connet
Conect
Iaceeet
no color cannat be swens pertt
aencee the beach ballion
AP ii geolage process lnsy shadtth luraatiiaatts
Mauni telersiiaicass
conweng bouundanoo
Corvect
Conwet
Coneet
Wil wiroospppoooooo shommm edutionch be be fiing frmngend noc wwr.ingigiggg hing?
Alrox senovves lngi Freechi
Corvect
Connet
Connect
Imaga fro the Vosseer and the Ga so epsoucrili provide esartice toither.
sllestad irceeeu meteriinccallon remowal sesuciin whithhr auma iecnuable
rephooe Hithe venerce bustr thamt vrstinnd the rovoree will thene sapper wilice ueivin shous www.reeeeeieeeeeeeccooo
(b) Updated Discriminator Responses via PEG
Figure A batch example from the ARC-Challenge dataset showing discriminator responses before and after PEG-based policy updates. The top table illustrates the initial disagreement among diseriminators, while the bottom table demonstrates improved convergence following utility-based updates. Red text indicates that the generator produced an incorrect answer. Red text indicates that the generator produced a incorrect answer. green arrow highlights cases where the majority vote correctly judged the validity of the answer.
25

---

(esurdo]shows a batch from the ARC- ERC-Chalee dataset that illustrates the discriminator responses before and after PEG-based policy updates. the top table 6a we observe noticeable disagreement among the three discriminators, reflecting their differing judgment capabilities across questions. For instance, Discriminator #3 (deepseek-LLLa)) (eepseek-LLaMA-7B) correctly identifies case that the other two fail to judge, while in other questions it falls behind the others. After applying PEG [6b] the discriminators show improved agreement. In most cases, they converge to the correct judgment, leading to a higher overall decision accuracy.
C.6 The impact of number of iterations
We conducted experiments on four benchmark datasets with {10, 20, 30, 40, 50) iterations using a fixed learning rate of 0.1 and a batch size of 8. From Table we observe that PEG's 's performance is stable and does not require many iterations to achieve strong performance, with only minimal changes in accuracy and a few questions. We attribute this result to the fact that our updates operate directly in the output policy space rather than modifying model parameters, allowing for faster convergence within a few iterations.
Table Accuracy (%) of PEG with different numbers of iterations across four benchmark datasets.
Iterations
10
20
30
40
50
ARC-Challenge
87.01
87.01
87.01
87.01
87.01
ARC-Easy
91.78
91.82
91.78
91.78
91.78
MMLU
70.78
70.81
70.81
70.79
70.81
GPQA
22.54
22.54
22.54
22.54
22.54
C.7 The impact of learning rate
We conducted experiments on four benchmark datasets using fixed number of 10 iterations with difference choices of learning rates in Table 7 Intuitively, smaller learning rates (e.g., S 0.1) yield stable performance without compromising convergence speed. In contrast, larger learning rates degrade performance, likely due to the fact that the updates are applied directly to the output distribution, which lies within bounded space [0, 1]. As result, overly aggressive updates may lead to oscillation or failure to converge.
Table 7: Accuracy (%) of PEG with different learning rates across four benchmark datasets.
Learning Rate
0.01
0.05
0.1
0.15
0.2
ARC-Challenge
87.01
87.01
87.01
86.50
80.68
ARC-Easy
91.82
91.82
91.82
91.65
88.44
MMLU
70.89
70.88
70.73
68.51
61.06
GPQA
22.54
22.54
22.54
22.32
19.42
C.8 Impact degree of initial disagreements
We conducted further analysis on the effectiveness our method by categorizing problems in each dataset into different levels of difficulty based on the initial level of disagreement among the discriminators. We report the accuracy for each method across these groups on both ARC-Challenge and MMLU in Table Our analysis reveals three key findings. First, PEG consistently improves accuracy across all levels of initial disagreement, suggesting that it enables agents to explore diverse judgments and learn from each other to reach better consensus. Second, a innuter-in finding that cases with initial disagreement among discriminators actually result in higher accuracy compared to those with full agreement. We believe this phenomenon occurs because agreement alone does not ensure correctness: all discriminators may still converge on an incorrect answer. In contrast, disagreement introduces diversity, increasing the likelihood that at least one agent is correct, and serves useful signal to trigger further exploration. Third, PEG achieves the highest accuracy
26

---

across all levels compared to other baselines methods, validating its effectiveness under different settings.
Table 8: Accuracy (%) under different initial agreement settings for ARC-Challenge and MMLU datasets. Bold indicates the best performance in each row.
Dataset
Setting
G
MI
ER-D
PEG
Agreement
64.31
73.85
72.79
84.10
ARC-Challenge
Disagreement
76.89
81.96
82.41
87.94
Agreement
50.54
60.11
60.17
66.93
MMLU
Disagreement
57.96
65.19
65.02
72.20
D Potential Social Impact
Our proposed peer elicitation game contributes to the broader goal of building trustworthy and accountable language models by explicitly incentivizing truthful behavior through multi-agent inter- action. By avoiding reliance on supervised fine-tuning and instead leveraging incentive- compaaibbe mechanisms, the method has the potential to enhance factual reliability in resource-constraiig safety-critical settings such as education, healthcare information retrieval, and scientific communi- cation. However, as with any mechanism involving agent- based interactions, care must be taken to ensure transparency in deployment and to prevent gaming of the system by adversarial agents.
or
27

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's 's main contributions and scope. All claims are supported by theoretical analysis and empirical results, with assumptions and limitations explicitly acknowledged. The content accurately reflects what is achieved in the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper explicitly discusses its limitations in Section
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides all necessary assumptions and complete proofs for each theoretical result, with details in Section and Appendix
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides comprehensive details necessary for reproducing the main experimental results, including dataset descriptions, model configurations, training procedures, and evaluation metrics Section and Appendix These disclosures are sufficient to support the main claims and conclusions.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The paper provides open access to the data and code in Section
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper clearly specifies all relevant training and testing details, including datasets. hyperparameters, optimizer types, and other configuration settings in Section and Appendix C
Justification:
7. Experiment statistical significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not report error bars or conduct formal statistical significance tests.
28

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The paper specifies the type of hardware used for experiments (e.g.. GPU
model) in Appendix C.3 along with relevant details such as batch size and runtime where applicable, providing sufficient information to estimate the computational resources required for reproduction.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research adheres to the NeurIPS Code of Ethics. It involves no human subjects, sensitive data, or harmful applications, and all datasets and methods are used responsibly and transparently.
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both positive and negative societal impacts Appendix
D
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve models or data with high risk of misuse; therefore, safeguards are not applicable this context.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets used in the paper, including datasets and codebases, are properly credited. Their licenses and terms of use are clearly acknowledged and fully respected.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not relsease new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper doesn't I involve crowd sourceing nor research with human subjects. Institutional review board (IRB) approvals or equivalent for research with human subjects
29

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper doesn't 't involve crowdsourcing research with human subjects
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this work doesn't 't involve LLMs as any important, original, or non-standard components.
30


Reviews:
[{"review_id": "mCCksHNHAE", "reviewer": "Reviewer_tcwC", "summary": "This paper proposes Peer Elicitation Games (PEG), a training-free, game-theoretic framework designed to incentivize truthful outputs from large language models (LLMs) without requiring access to ground-truth labels or fine-tuning. The PEG framework introduces multiple independently instantiated discriminator models that evaluate the output of a generator model. Discriminators are rewarded based on mutual agreement through a determinant-based mutual information scoring mechanism. The authors provide rigorous theoretical guarantees, including incentive compatibility (IC), sublinear regret, and last-iterate convergence to a truthful Nash equilibrium. Empirically, PEG improves factual accuracy across multiple QA benchmarks, showing significant gains even with relatively small models.", "questions": "1. In the introduction, the authors highlight PEG’s ability to incentivize truthful reporting without relying on ground-truth labels. Could the authors elaborate more on the disadvantages of using ground-truth labels in this context? For instance, is the goal primarily to reduce human annotation cost, or are there concerns about ground-truth availability, inconsistency, or domain adaptation?\n2. Could you add resource analysis, comparing the method with baseline methods?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper offers a comprehensive set of theoretical guarantees.\n2. The method demonstrates consistent performance gains across challenging QA tasks such as ARC-Challenge, MMLU, and GPQA.\n\nWeakness:\n1. The workflow of this method is not clear to me, the author claims that this is a training-free method, however it still updates the model using 'silver labels' from majority votes of discriminators? It should be more clear in the paper.\n2. It demands high computational and infrastructural resources. For each evaluation task, multiple discriminator models must be instantiated and run across multiple iterations. This includes maintaining several 7B-scale LLMs concurrently and performing multiple rounds of generation-evaluation-update cycles. In some settings, this might outweigh the resource costs of lightweight supervised tuning or distillation.\n3. The paper does not provide any quantitative analysis or comparison of computational costs or runtime relative to baselines. Without such comparison, it is difficult to assess the practical viability of PEG in real-world applications, especially in latency- or budget-sensitive settings.\n3. The paper does not explore how the number of PEG iterations or number of discriminators affects final performance.", "was_revised": true, "final_justification": "The paper was not clear enough on the fact that the updates are on distribution only. The authors' explanation makes me understand better the method.\n\nIn addition, the author adds more ablation results, e.g. number of iteration, number of discriminator.", "rebuttal": "Thank you for your insightful feedback on our manuscript. We appreciate your comments and have taken them into consideration to improve our paper. Below, we do our best to address all concerns adequately so that we could receive a better score.\n\n**W1: Clarification on PEG workflow and ''Training-Free\" claim**\n\nWe appreciate the reviewer’s concern and agree that a clearer explanation of the workflow is necessary. As illustrated in Figure 2 in our submission, the core procedure of PEG consists of four main steps: (1) Given an input question, the generator produces a candidate answer; (2) multiple discriminators independently evaluate this answer, each providing a binary judgment; (3) these judgments are aggregated into a scalar reward signal; (4) a parameterized distribution (policy) over candidate answers is updated using these rewards.\n\nImportantly, the parameters of the generator and discriminators remain entirely frozen throughout the process. All updates occur externally at the level of the output distribution. Thus, we describe PEG as “training-free” in the sense that it requires no gradient-based updates, no backpropagation through model parameters, and no fine-tuning or RLHF.\nIntuitively, the update rule in Equation 4 in our submission increases the probability of outputs that receive higher rewards without requiring access to model internals. Additionally, this approach is consistent with prior training-free paradigms (e.g., [1,2]) that optimize over output distributions rather than over model parameters. We will add these clarifications to Section 2.2 in the revised version to improve clarity.\n\n**W2, W3 \\& Q2: Clarification on computational cost and infrastructural resources, resource analysis compared to baselines**\n\nWe thank the reviewer for raising this important point. We respectfully clarify that our method does not require maintaining or re-evaluating multiple discriminators across iterations. In each batch, candidate answers are evaluated once by a fixed set of frozen discriminators. The policy update then occurs entirely outside the models at the level of the output distribution.\nAs a result, during the iterative process, we simply adjust the probability assigned to candidate outputs: there are no additional generations from the generator, no repeated calls to the discriminators, and no model training (e.g., no gradient updates or parameter tuning). This design ensures computational efficiency and maintains a training-free paradigm throughout the PEG procedure.\n\nTo be more specific, as detailed in Appendix C.3, the main computational cost arises during the initial policy extraction, i.e., generating candidate answers from the generator and evaluations from the discriminators. This cost scales linearly with the number of questions. In the subsequent PEG iterations, only lightweight computations (such as reward aggregation and softmax updates) are involved, and no additional infrastructure is required to coordinate multiple discriminators.\nThe runtime per iteration of the PEG mechanism for each discriminator is approximately 200 microseconds on average. By comparison, the Consensus Game baseline [1] executes around 40 microseconds per iteration. Notably, PEG processes batches of 8 questions per iteration, enabling efficient parallel updates across multiple inputs.\n\nImportantly, unlike traditional fine-tuning approaches which often require hours to days of compute time, our PEG mechanism is entirely training-free. The full fine-tuning methods generally require thousands of GPU hours and can cost anywhere from $10,000$ to over $35,000$ per run [3]. Even parameter-efficient fine-tuning (PEFT) techniques such as FinLoRA [4] require several hours to days to update model weights. In contrast, PEG operates without modifying model parameters or performing gradient updates, thereby completely eliminating the need for a training phase.\n\n**W4: How the number of PEG iterations or number of discriminators affects final performance**\n\nWe thank the reviewer for this valuable question. We conducted additional experiments to study how performance varies with the number of PEG iterations and the number of discriminators.\n\nFirst, we evaluated the accuracy of PEG with \\{10, 20, 30, 40, 50\\} iterations across all datasets, using a fixed learning rate of 0.1 and a batch size of 8. From Table 1, PEG converges within the first 10 iterations—subsequent iterations result in only minor adjustments on a few questions, which do not noticeably affect the overall accuracy. We attribute this result to the fact that our updates operate directly in the output policy space rather than modifying model parameters, allowing for faster convergence within a few iterations.\n\n**Table 1: Accuracy (%) of PEG with different numbers of iterations across four benchmark datasets.**\n\n| Dataset        |10| 20| 30| 40| 50|\n|----------------|--------|--------|--------|--------|--------|\n| ARC-Challenge  | 87.01  | 87.01  | 87.01  | 87.01  | 87.01  |\n| ARC-Easy       | 91.78  | 91.82  | 91.78  | 91.78  | 91.78  |\n| MMLU           | 70.78  | 70.81  | 70.81  | 70.79  | 70.81  |\n| GPQA           | 22.54  | 22.54  | 22.54  | 22.54  | 22.54  |\n\nWe also conduct additional experiments on the ARC challenge set with 5-discriminators (add Gemma-7B [5] and Mistral-7B [6]) and 7-discriminators (further add Ai-Yi-9B [7] and OpenChat-7B [8]). The results highlight the strong impact of PEG on both individual and collective performance. First, as shown in Tables 2 and 3, individual discriminators show consistent improvement after applying PEG. Secondly, in both the 5-discriminator and 7-discriminator settings, our PEG methods consistently outperforms majority vote and ER-D, as shown in Table 4. However, we observed a slight drop in PEG’s overall performance when adding more discriminators. We believe this may be due to a mild violation of the Conditional Independence assumption. The two added models (Ai-Yi-9B and OpenChat-7B [7,8]), while strong in general, may not align well with the existing set in terms of signal diversity or informativeness. This observation further reinforces that it’s not just the number of discriminators that matters, but their quality and compatibility with PEG’s theoretical assumptions. The results show if discriminators are chosen in alignment with our underlying assumptions, a larger number of discriminators is not necessarily required to achieve good performance. This suggests that the quality and relevance of the chosen discriminators are more critical than their sheer quantity for the effectiveness of our PEG mechanism.\n\n**Table 2: Accuracy (%) of individual discriminators before and after applying PEG in the 5-discriminator setting.**\n\n|5 Discriminators | OQwen  | deepseek-Qwen | deepseek-Llama2 | Gemma 7B | Mistral 7B |\n|------------------|--------|----------------|------------------|-----------|-------------|\n| After PEG|86.84| 84.27| 83.76| 84.27| 82.74 |\n| Original|82.91|57.18| 59.83| 69.32 | 70.26|\n\n**Table 3: Accuracy (%) of individual discriminators before and after applying PEG in the 7-discriminator setting.**\n\n| 7 Discriminators|OQwen|Qwen|Llama2|Gemma 7B|Mistral 7B|Ai Yi 9B|OpenChat 7B|\n|------------------|--------|--------|--------|-----------|-------------|-----------|---------------|\n| After PEG| 83.85|75.30|73.08|77.61|79.06| 81.62| 82.99|\n| Original| 82.91|57.18|59.83|69.32|70.26| 81.28| 79.06|\n\n**Table 4: Overall accuracy (%) comparison of PEG, Majority Vote, and ER-D under 3-, 5-, and 7-discriminator settings.**\n\n| Setting            | PEG     | Majority Vote | ER-D|\n|--------------------|---------|----------------|---------|\n| 3 Discriminators| 87.01| 70.68| 77.52|\n| 5 Discriminators| 86.75| 71.71| 76.32|\n| 7 Discriminators| 81.97| 76.50| 81.54|\n\n**Q1: Clarification on disadvantages of ground truth labels**\n\nThank you for the thoughtful question. Our motivation for avoiding ground-truth labels stems from these limitations:\n\nFirst, in many specialized domains such as biomedical, legal, or scientific reasoning, high-quality expert annotations are difficult or infeasible to obtain [9]. This makes ground-truth supervision impractical for real-world deployment and model adaptation in new tasks.\n\nSecond, even when annotations are available, they are often unreliable. For open-ended prompts, multiple plausible answers may exist, but ground-truth labels typically assume a single correct one, thus penalizing valid alternatives. In subjective tasks like summarization or ranking, labels are inherently ambiguous, and annotators frequently disagree due to personal preferences, leading to noisy supervision [10].  \n\nTo address these challenges, PEG replaces reliance on a fixed ground-truth label with feedback aggregated from a panel of random referees. This enables learning signals from each other rather than a single “golden” label, making the framework more scalable and robust.\n\n[1] Jacob, Athul Paul, et al. \"The consensus game: Language model generation via equilibrium search.\" ICLR  \n[2] Jacob, Athul Paul, et al. \"Modeling strong and human-like gameplay with KL-regularized search.\" ICML  \n[3] Liu, Aixin, et al. \"Deepseek-v3 technical report.\"  \n[4] Wang, Dannong, et al. \"FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasets.\"   \n[5] Team, Gemma, et al. \"Gemma: Open models based on gemini research and technology.\"   \n[6] Albert Q. Jiang, et al.  \"Mistral 7B.\"  \n[7] Young, Alex, et al. \"Yi: Open foundation models by 01. ai.\"  \n[8] Wang, Guan, et al. \"Openchat: Advancing open-source language models with mixed-quality data.\"   \n[9]  Zhang, Le, et al. \"Learning from multiple annotators for medical image segmentation.\" Pattern Recognition 138 (2023): 109400.  \n[10]  Davani, Aida Mostafazadeh, Mark Díaz, and Vinodkumar Prabhakaran. \"Dealing with disagreements: Looking beyond the majority vote in subjective annotations.\" Transactions of the Association for Computational Linguistics 10 (2022): 92-110."}, {"review_id": "otLiqkd9gb", "reviewer": "Reviewer_FbmF", "summary": "The paper proposes a game-theoretic post-training framework for inducing more truthful responses, by maximizing agreement with a committee of validators. The validators are incentivized to agree with each other. The paper proves three theoretical properties of this framework: (1) that the truthful reporting strategy is optimal; (2) that the updates have sub-linear regret with regard to the best fixed policy; (3) that the final iterate converges to the nash equilibrium. The paper also includes some empirical results on ARC, MMLU, and GPQA, using Qwen models.", "questions": "1. The core idea is of consistency among the validators. What prevents the validators from reaching consensus on an incorrect answer? \n1. Relatedly, how does the effectiveness of the approach relate to the \"hardness\" of the prompts? (i.e. are there prompts where the validators initially have high disagreement?)\n1. How many validators are needed? Does adding more validators induce a tradeoff between iterations to convergence and the cost of each iteration?\n1. Do the implementation use the learning rates from the theory?", "limitations": "Discussion of limitations is minimal.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n- an original, well-motivated, and theoretically-rigorous approach to a very important problem\n- strong theoretical guarantees\n- well-written paper\n\nWeaknesses\n\n- a few key questions remain unclear, at least to this reader - see below\n- possibly overclaims that the approach works \"without ... model fine-tuning\" - need more clarity on this\n- overclaims about comparisons with larger models (lines 334-342) by focusing on models which are significantly older and are not instruction tuned. Lack of instruction tuning is likely the primary reason that their performance is worse, given that OQwen outperforms these models even *without* PEG.", "was_revised": true, "final_justification": "I found this paper to be the most thought-provoking of my review batch. As noted in my review, it has several strengths:\n\n- an original, well-motivated, and theoretically-rigorous approach to a very important problem\n- strong theoretical guarantees\n- well-written paper\n\nI also appreciate the authors' work to resolve my questions and concerns. I recommend that it be accepted.", "rebuttal": "We extend our sincere appreciation for your valuable feedback and suggestions. Regarding your concerns, we would like to offer further clarification.\n\n**W2: Clarification on \"without ground-truth labels or model fine-tuning\" claim**\n\nThank you for pointing this out. When we state that PEG operates \"without relying on ground-truth labels or model fine-tuning,\" we mean the following:\n\nFirst, PEG does not require any  ground-truth labels. Instead, it leverages signals from a group LLM-based discriminators, which serve as referees to provide feedback. The optimization is driven by agreement among these models, rather than comparison to a gold standard.\n\nSecond, PEG does not involve any updating on the model parameters. All models—including both the generator and the discriminators—remain completely frozen throughout the process. PEG operates purely by adjusting a distribution over candidate outputs guided by external feedback, making it distinct from supervised fine-tuning or RLHF approaches.\n\nWe will revise our claims in the introduction to clarify that PEG relies on peer evaluation rather than ground truth and no model parameter throughout the process. \n\n**W3: Overclaiming due to comparison with older, non-instruction-tuned models**\n\nWe thank the reviewer for this valuable comment. We agree that some models used in our comparison are older and not instruction-tuned, which may partially explain their weaker performance. Our intention was not to overclaim, but to maintain consistency with prior work in consensus game [1], where these models were commonly adopted as standard baselines.\nWe will remove the text to avoid potential overstatement.\n\n**Q1: What prevents the validators from reaching consensus on an incorrect answer?**\n\nThank you for the insightful question. PEG relies on consensus among multiple discriminators as a proxy for truthful and high-quality outputs, without assuming that any single discriminator is always correct.\nFrom a theoretical perspective, our approach is inspired by the peer prediction framework, where heterogeneous agents are incentivized to provide informative and truthful signals under proper reward structure. \nFrom an empirical perspective, we observe that discriminators often select different correct answers in the early iterations. However, through PEG’s iterative aggregation and update process, these models implicitly “learn from each other” by reinforcing candidates that receive consistent support, while suppressing unstable or weakly supported ones. As a result, the output distribution gradually converges to more coherent and broadly agreed-upon answers, rather than amplifying isolated errors.\n\n**Q2: How does the effectiveness relate to prompt “hardness”? (initially have high disagreements)**\n\nThank you for raising this insightful question regarding the relationship between prompt \"hardness\" and the effectiveness of our approach. To address this, we conducted further analysis by categorizing prompts in each dataset into different levels of difficulty based on the initial level of disagreement among the discriminators (validators). This categorization serves as a proxy for prompt difficulty, with higher disagreement indicating greater uncertainty or ambiguity. We report the accuracy for each method across these groups on both ARC-Challenge and MMLU in Table 1.\n\nOur analysis reveals three key findings. First, PEG consistently improves accuracy across all levels of initial disagreement, suggesting that it enables agents to explore diverse judgments and learn from each other to reach better consensus. Second, a counter-intuitive finding is that cases with initial disagreement among discriminators actually result in higher accuracy compared to those with full agreement. We believe this phenomenon occurs because agreement alone does not ensure correctness: all discriminators may still converge on an incorrect answer. In contrast, disagreement introduces diversity, increasing the likelihood that at least one agent is correct, and serves as a useful signal to trigger further exploration. Third, PEG achieves the highest accuracy across all levels compared to other baselines methods. We will include this discussion in our revised version and leave the extension to settings with more than three discriminators for future work.\n\n**Table 1: Accuracy under different initial agreement settings for ARC-Challenge and MMLU datasets.**\n\n| Dataset | Setting |Initial | MI | ER-D| PEG |\n|----------------|--------------|---------|--------|--------|--------|\n| ARC-Challenge  | Agreement    | 64.31   | 73.85  | 72.79  | 84.10  |\n|                | Disagreement | 76.89   | 81.96  | 82.41  | 87.94  |\n| MMLU           | Agreement    | 50.54   | 60.11  | 60.17  | 66.93  |\n|                | Disagreement | 57.96   | 65.19  | 65.02  | 72.20  |\n\n**Q3: How many validators are needed, and is there a tradeoff between iterations to convergence and the cost of each iteration?**\n\nThank you for raising this important point regarding the number of validators and the associated cost-performance tradeoff. Theoretically, PEG only requires 2 discriminators(validators) to achieve dominant truthfulness. Empirically, We conduct additional experiments on the ARC challenge set with 5-discriminators (add Gemma-7B [2] and Mistral-7B [3]) and 7-discriminators (further add Ai-Yi-9B [4] and OpenChat-7B [5]). The results highlight the strong impact of PEG on both individual and collective performance. First, as shown in Tables 2 and 3, individual discriminators show consistent improvement after applying PEG. Secondly, in both the 5-discriminator and 7-discriminator settings, our PEG methods consistently outperforms majority vote and ER-D, as shown in Table 4. \n\nHowever, as shown in Table 4, we observe a slight drop in performance when increasing the number of discriminators, even though majority vote and consensus game methods show improvement. We attribute this to a potential violation of the PEG mechanism’s conditional independence assumption, which is critical for its theoretical guarantees and empirical effectiveness. Specifically, the two added models (Ai-Yi-9B and OpenChat-7B [4,5]), while strong in general, may not align well with the existing set in terms of signal diversity or informativeness. This observation suggests that the quality and compatibility of discriminators are more important than their sheer quantity. Additionally, introducing additional discriminators does not increase the number of iterations required for convergence, nor does it add overhead to each iteration. However, as performance becomes slightly more sensitive to the composition of the ensemble, a smaller, carefully curated set of discriminators that adheres to the conditional independence assumption can lead to superior and more stable performance.\n\n**Table 2: Accuracy (%) of individual discriminators before and after applying PEG in the 5-discriminator setting.**\n\n|5 Discriminators | OQwen | deepseek-Qwen| deepseek-Llama2| Gemma 7B | Mistral 7B |\n|------------------|--------|----------------|------------------|-----------|-------------|\n| After PEG|86.84| 84.27| 83.76| 84.27| 82.74 |\n| Original|82.91|57.18| 59.83| 69.32 | 70.26|\n\n**Table 3: Accuracy (%) of individual discriminators before and after applying PEG in the 7-discriminator setting.**\n\n| 7 Discriminators|OQwen|Qwen|Llama2|Gemma 7B|Mistral 7B|Ai Yi 9B|OpenChat 7B|\n|------------------|--------|--------|--------|-----------|-------------|-----------|---------------|\n| After PEG| 83.85|75.30|73.08|77.61|79.06| 81.62| 82.99|\n| Original| 82.91|57.18|59.83|69.32|70.26| 81.28| 79.06|\n\n**Table 4: Overall accuracy (%) comparison of PEG, Majority Vote, and ER-D under 3-, 5-, and 7-discriminator settings.**\n\n| Setting            | PEG     | Majority Vote | ER-D|\n|--------------------|---------|----------------|---------|\n| 3 Discriminators| 87.01%| 70.68%| 77.52%|\n| 5 Discriminators| 86.75%| 71.71%| 76.32%|\n| 7 Discriminators| 81.97%| 76.50% | 81.54%|\n\n**Q4: Clarification on learning rates in the experiments**\n\nThank you for pointing out the implementation details regarding the learning rate in our experiments. We clarify that the learning rate used in our theoretical analysis is adaptive and depends on a KL divergence term between the initial policy and the optimal policy. However, this term is difficult to estimate in practice. In our empirical studies, we find that the performance of PEG is robust to a range of learning rates. Specifically, we report accuracy results across four datasets using a fixed number of 10 iterations in Table 5. Intuitively, smaller learning rates (e.g., $\\leq 0.1$) yield stable performance without compromising convergence speed. In contrast, larger learning rates degrade performance, likely due to the fact that the updates are applied directly to the output distribution, which lies within a bounded space $[0, 1]$. As a result, overly aggressive updates may lead to oscillation or failure to converge. These findings support the practical stability of our approach under reasonable learning rate choices. We will include this experimental results and analysis in our revised version.\n\n**Table 5: Accuracy (%) of PEG with different learning rates across four benchmark datasets.**\n\n| Dataset       | 0.01|0.05|0.1|0.15|0.2|\n|---------------|-------|-------|-------|-------|-------|\n| ARC-Challenge |87.01|87.01|87.01|86.50|80.68|\n| ARC-Easy      |91.82|91.82|91.82|91.65|88.44|\n| MMLU          |70.89|70.88|70.73|68.51|61.06|\n| GPQA          |22.54| 22.54|22.54|22.32|19.42|\n\n[1] Jacob, Athul Paul, et al. \"The consensus game: Language model generation via equilibrium search.\" ICLR  \n[2] Team, Gemma, et al. \"Gemma: Open models based on gemini research and technology.\"   \n[3]  Albert Q. Jiang, et al.  \"Mistral 7B.\"  \n[4] Young, Alex, et al. \"Yi: Open foundation models by 01. ai.\"  \n[5] Wang, Guan, et al. \"Openchat: Advancing open-source language models with mixed-quality data.\""}, {"review_id": "Gvb559dmlg", "reviewer": "Reviewer_KmKz", "summary": "The paper uses a multi-agent system method to address the hallucination problem in language models. The traditional consensus game, whose rewards are based on agreement, falls into the problem of collusion, as the discriminator (the agent judging the outcome) has incentives to agree with the generator (the agent producing the outcome) anyway to receive a higher payment. This paper adopts a peer prediction mechanism, where multiple discriminators are hired to judging the outcome, and their reward are based on comparing each other's answer. In a multi-round dynamic game scenario, the LLM agents converge to a truthful Nash equilibrium with sub-linear regret. The paper also provides experiments to support their theoretical results.", "questions": "1. Please address my question in the weaknesses section. \n\n2. What is the covergence rate/speed of the PEG mechanism?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n\nI like this study. It brings me new insight into how peer prediction mechanisms and multi-agent systems can be applied in more trendy areas. The paper has a clear motivation and methodology. While I don't clearly examine the proof, I am more familiar with the DMI mechanism applied and believe this should work at a high level. The algorithmic and experimental results also strengthen the practicality of the study. \n\nWeaknesses: \n\nOne thing I would like the authors to discuss more clearly is the notion of \"truthfulness\" of the LLM agent. In the traditional game theory or mechanism design context, the notion is clear with no ambiguity. But for large language models, this needs more explanation. For example, there can be a problem of calibration. A LM can output \"I believe the content is 60% true\" with 80% probability and \"I believe the content is 40% percent true\" with 20% probability. Then, which one is the LM's true belief, and how do we define truthfulness for this LM?", "was_revised": true, "final_justification": "The author's rebuttal addresses my questions. While contrversy remains on what is truthfulness for LLM, I believe the paper is good enough for a broader discussion on NeurIPS. I will maintain my score.", "rebuttal": "We sincerely appreciate your acknowledgment of the quality, significance, and novelty of our work.\nRegarding your concerns, we would like to offer further clarification.\n\n**W1 and Q1: Clarification on the definition of “Truthfulness” in the LLM setting**\n\nWe thank the reviewer for raising this important and subtle point. As noted in our submission (Lines 93–102), we adopt the classical notion of incentive compatibility (IC) from mechanism design to define truthfulness. Under IC, an agent is said to be truthful if reporting its private information maximizes its expected utility under the mechanism:\n$u_i(c_i, c_{-i}) \\geq u_i(r_i, c_{-i}) \\quad \\forall r_i \\neq c_i,$\nwhere $c_i \\in \\\\{0,1\\\\}$ denotes the agent’s true private signal, $r_i$ it's the reported label, and $u_i$ it's the utility based on others' reports $c_{-i}$.\n\nWhen applied to LLMs, however, the concept of a “true belief” becomes less well-defined. LLM outputs are inherently probabilistic and often miscalibrated, meaning they may not faithfully reflect the model’s internal uncertainty or belief distribution. In our setting, we adopt a practical and operational notion of truthfulness: the report is treated as an output answer rather than a full belief distribution, and we assume the LLM reveals its \"true belief\" through greedy sampling, without additional exploration. For instance, if the model internally assigns 80\\% probability to a correct answer and 20\\% to an incorrect one, a truthful response would be the correct label. However, due to hallucinations or the stochastic nature of generation, the actual output may deviate from this internal belief, occasionally resulting in incorrect or noisy responses.\n\nReturning to the reviewer’s example: suppose the LLM outputs “I believe the content is 60\\% true” with 80\\% probability, and “I believe the content is 40\\% true” with 20\\% probability. In our framework, we do not treat the textual belief statement itself (e.g., \"60\\% true\") as the truthful signal. Instead, we focus on the correctness label associated with the most probable output—namely, the one with hither probability—as the truthful signal. That is, the model is considered truthful if it reports the option it most strongly believes to be correct. The goal of PEG is to encourage reporting behavior that aligns with the model’s strongest internal belief, ensuring that the output reflects its highest-confidence judgment of correctness.\n\nMore broadly, we agree that LLM truthfulness can be meaningfully extended to its belief distribution. In this view, truthfulness becomes closely related to calibration: a truthful LLM should produce outputs that accurately reflect its internal confidence or belief distribution. For example, if the model internally believes the answer is 60\\% likely to be correct and 40\\% incorrect, then this belief distribution can be treated as its true belief. Miscalibration, a systematic mismatch between internal confidence and the reported output—can result in deviations from incentive-compatible behavior.\nAdditionally, some prior works define LLM truthfulness more narrowly, equating it with factual correctness and treating any deviation (e.g., hallucination) as untruthfulness [1]. While this definition captures practical concerns, it does not incorporate ideas from mechanism design or incentive compatibility. The precise definition of LLM truthfulness remains an open question. We will include this discussion in our revised version.\n\n**Q2: Clarification on the covergence rate/speed of the PEG mechanism**\n\nWe thank the reviewer for raising this important theoretical question. PEG is implemented via policy mirror descent. Specifically, we use exponentiated gradient updates (akin to mirror descent in the simplex) to iteratively refine the distribution over candidate answers.Therefore, PEG has convergence guarantees in the form of a no-regret property over iterations. Specifically, this implies that the average reward across $T$ steps approaches that of the best fixed policy in hindsight at a rate of $\\mathcal{O}\\left(\\frac{1}{\\sqrt{T}}\\right)$. This result is consistent with standard findings in prior literature [2,3].\n\nIn terms of last-iterate convergence rate, we can observe that PEG typically converges and stabilizes within only 10 iterations in our experiments. However, providing theoretical guarantees for the last iterate remains challenging. Recent work such as [4] analyzes last-iterate linear convergence rates under magnetic mirror descent, but these results are restricted to two-player zero-sum games. We leave the extension of such convergence analysis to the multi-agent peer elicitation games used in PEG as a potential future direction.\n\nOnce again, we appreciate the reviewer's precious time. We are eager to engage in further discussions to clear out any confusion.\n\n[1] Azaria, Amos, and Tom Mitchell. \"The internal state of an LLM knows when it's lying.\" arXiv preprint arXiv:2304.13734 (2023).  \n[2] Beck, Amir, and Marc Teboulle. \"Mirror descent and nonlinear projected subgradient methods for convex optimization.\" Operations Research Letters 31.3 (2003): 167-175.  \n[3] Kiwiel, Krzysztof C. \"Proximal minimization methods with generalized Bregman functions.\" SIAM journal on control and optimization 35.4 (1997): 1142-1168.  \n[4] Sokota, Samuel, et al. \"A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games.\" arXiv preprint arXiv:2206.05825 (2022)."}, {"review_id": "Rb7sfx9O34", "reviewer": "Reviewer_6fgB", "summary": "The paper introduces **Peer Elicitation Games (PEG)**, a training-free multi-agent framework in which a generator LLM answers questions and multiple discriminator LLMs evaluate the responses using a determinant-based peer-prediction reward. The authors theoretically prove that truthful reporting is a dominant strategy, with sub-linear regret and last-iterate convergence. PEG yields over 10% accuracy improvement compared to existing methods only using 4 open-source 7B models.", "questions": "1. Can you report PEG accuracy for varying discriminators and note any saturation or instability?\n2. How does PEG accuracy change when each batch contains a mix of ARC-Challenge and MMLU items compared with single-domain batches?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### **Strengths**\n1. Paper provides formal, rigorous proofs establishing dominant truthfulness, sub-linear regret, and last-iterate convergence guarantees throughout.\n2. Truthful answers are elicited via pure black-box LLM calls, requiring no fine-tuning, labels, or RLHF, hence lightweight.\n3. Demonstrates accuracy gains on ARC, MMLU, GPQA using only four open 7B models.\n\n#### **Weaknesses**\n1. Guarantees rely on binary labels and conditionally independent signals, limiting confidence in complex real-world language settings.\n2. Experiments cover only multiple-choice QA, lacking tests on open-ended, factual, or safety tasks, limiting generality.\n3. No ablations explore adversarial, low-quality, or fewer discriminators, leaving robustness, scalability, and heterogeneity effects unverified.", "was_revised": true, "final_justification": "The authors addressed the main concerns of guarantees rely on binary labels and conditionally independent signals, and supplemented ablation test on number and quality of discriminators. Thus, I increased the quality score by 1. While for the originality score, I mistakenly chose a low score of 2, so in the revision, I would like to correct it to 3.", "rebuttal": "We thank the reviewer for the insightful and supportive comments. Below, we do our best to address the reviewer's questions adequately such that we could receive a better score.\n\n**W1: Theoretical assumptions on binary labels and conditionally independent signals**\n\nThanks for pointing out the validity of these assumptions in real-world language settings. We offer the following justifications to clarify our framework’s effectiveness:\n\nFirst, the generator is not restricted to a specific task type. PEG is compatible with open-ended tasks, as long as the generator can provide a set of candidate answers for evaluation. This makes the framework broadly applicable beyond multiple-choice QA.\n\nSecond, our theoretical discussion adopts binary feedback for clarity, as the discriminators provide binary judgments in our setting. However, the DMI mechanism is originally designed for multi-choice settings and still guarantees dominant truthfulness under standard conditions. Specifically, the key requirement is that the batch size must be at least $2C$, where $C$ denotes the number of choices per question.\n\nThird, the conditional independence assumption is used in the theoretical analysis to ensure that the mutual information–based reward does not degenerate to $0$. Intuitively, if agents' signals are highly correlated, then observing others provides little to no additional information, and the utility (measured via a form of mutual information) collapses toward zero. In practice, since we use heterogeneous discriminators, their outputs are diverse and rarely reach consensus in the beginning. This natural diversity prevents the reward signal from collapsing and keeps the system informative.\n\n**W2: Clarification on task generality beyond multiple-choice QA**\n\nThank you for the helpful suggestion. While our current experiments focus on multiple-choice QA tasks, PEG is not inherently limited to this setting. For any natural language task, we can use top-$k$ or other sampling methods to generate a candidate set, which PEG can then evaluate and refine. In this sense, the multiple-choice format is a practical instantiation of the candidate generation process, not a fundamental restriction of the framework. We have also discussed this in the limitations section and fully agree that evaluating PEG on broader task types is an important next step. We plan to include additional datasets covering factual, open-ended, and safety-critical tasks in future work.\n\n**W3 and Q1: Ablations on number and quality of discriminators**\n\nThank you for raising this valuable point regarding the number and quality of discriminators. To investigate these problems, we conduct additional experiments on the ARC challenge set with 5-discriminators (add Gemma-7B [1] and Mistral-7B [2]) and 7-discriminators (further add Ai-Yi-9B [3] and OpenChat-7B [4]). The results highlight the strong impact of PEG on both individual and collective performance. First, as shown in Tables 1 and 2, individual discriminators show consistent improvement after applying PEG. Secondly, in both the 5-discriminator and 7-discriminator settings, our PEG methods consistently outperforms majority vote and ER-D, as shown in Table 3. \n\nHowever, we observed a slight drop in PEG’s overall performance when moving to more discriminators. We believe this may be due to a mild violation of the Conditional Independence assumption. The two added models (Ai-Yi-9B and OpenChat-7B [3,4]), while strong in general, may not align well with the existing set in terms of signal diversity or informativeness. This observation further reinforces that it’s not just the number of discriminators that matters, but their quality and compatibility with PEG’s theoretical assumptions. The results show if discriminators are chosen in alignment with our underlying assumptions, a larger number of discriminators is not necessarily required to achieve good performance. This suggests that the quality and relevance of the chosen discriminators are more critical than their sheer quantity for the effectiveness of our PEG mechanism.\n\n**Table 1: Accuracy (%) of individual discriminators before and after applying PEG in the 5-discriminator setting.**\n\n|5 Discriminators | OQwen  | deepseek-Qwen | deepseek-Llama2 | Gemma 7B | Mistral 7B |\n|------------------|--------|----------------|------------------|-----------|-------------|\n| After PEG|86.84| 84.27| 83.76| 84.27| 82.74 |\n| Original|82.91|57.18| 59.83| 69.32 | 70.26|\n\n**Table 2: Accuracy (%) of individual discriminators before and after applying PEG in the 7-discriminator setting.**\n\n| 7 Discriminators|OQwen|Qwen|Llama2|Gemma 7B|Mistral 7B|Ai Yi 9B|OpenChat 7B|\n|------------------|--------|--------|--------|-----------|-------------|-----------|---------------|\n| After PEG| 83.85|75.30|73.08|77.61|79.06| 81.62| 82.99|\n| Original| 82.91|57.18|59.83|69.32|70.26| 81.28| 79.06|\n\n**Table 3: Overall accuracy (%) comparison of PEG, Majority Vote, and ER-D under 3-, 5-, and 7-discriminator settings.**\n\n| Setting            | PEG     | Majority Vote | ER-D|\n|--------------------|---------|----------------|---------|\n| 3 Discriminators| 87.01| 70.68| 77.52|\n| 5 Discriminators| 86.75| 71.71| 76.32|\n| 7 Discriminators| 81.97| 76.50| 81.54|\n\n**Q2: Effect of mixed-domain vs. single-domain batches**\n\nWe thank the reviewer for pointing out the possibility that each batch may contain a mix of ARC-Challenge and MMLU items.\nWe first clarify that our theoretical guarantee of dominant truthfulness relies on the assumption that tasks within each batch are similar. This homogeneity is important because the utility function aggregates feedback across multiple tasks in a batch; if the tasks are too heterogeneous, the computed reward may become ill-posed and fail to provide a meaningful learning signal for effective updates.\nFrom a theoretical perspective, this assumption ensures that agents can adopt a consistent reporting strategy across tasks within the batch. In contrast, if tasks vary significantly in nature or domain, agents may not be incentivized to follow a stable or truthful strategy. Empirically, we enforce this condition by constructing each batch using items from the same domain within a dataset.\n\nOnce again, we appreciate the reviewer's precious time. We are eager to engage in further discussions to clear out any confusion.\n\n[1] Team, Gemma, et al. \"Gemma: Open models based on gemini research and technology.\"  \n[2] Albert Q. Jiang, et al. \"Mistral 7B.\"  \n[3] Young, Alex, et al. \"Yi: Open foundation models by 01. ai.\"  \n[4] Wang, Guan, et al. \"Openchat: Advancing open-source language models with mixed-quality data.\""}]

Author rebuttal:

Meta-review: The paper introduces Peer Elicitation Games (PEG), a training-free, game-theoretic framework for improving truthfulness in LLM outputs. 
All reviewers see the paper as novel, theoretically rigorous, and relevant. After rebuttals and discussions, consensus is positive.

Strengths of the paper highlighted are:

- Novelty & theory: Original application of peer prediction to LLM truthfulness; rigorous proofs of IC, regret bounds, and convergence.
- Training-free paradigm: PEG updates only the output distribution, requiring no gradient updates or retraining.
- Empirical evidence: Improves accuracy across difficult QA benchmarks using small open-source models.

Of course the paper also comes with some weaknesses the reviewers pointed out, including:
- Computational Cost: Initially unclear; rebuttal clarified that PEG’s iteration cost is lightweight once candidates are generated, but infrastructure overhead still exists.
- Truthfulness Definition: Reviewer KmKz raised concerns about defining “truthfulness” for probabilistic LLMs; authors clarified they adopt a pragmatic definition (reporting the model’s most likely belief). This remains an open conceptual issue.

This submission offers a well-motivated, and theoretically solid framework for eliciting truthfulness from LLMs. While some assumptions (binary labels, conditional independence) limit the generality of the guarantees, the combination of theoretical rigor and empirical improvements makes PEG a significant contribution. Therefore based on the reviewers' feedbacks, I would recommend acceptance.
Decision: Accept (poster)