From Self-Check to Consensus: Bayesian Strategic Decoding in Large Language Models
Authors: Weitong Zhang
Chengqi Zang
Bernhard Kainz
Venue: NeurIPS 2025 poster ()
Keywords: Multi-agent System
Game Theory
Mechanism Design
Primary area: deep_learning
TL;DR: A game theory-based mechanism drives LLMs from self-check to consensus for improved accuracy.

Abstract: Large Language Models exhibit logical inconsistency across multi-turn inference processes, undermining correctness in complex inferential tasks. Challenges arise from ensuring that outputs align with both factual correctness and human intent. Approaches like single-agent reflection and multi-agent debate frequently prioritize consistency, but at the expense of accuracy. 
To address this problem, we propose a novel game-theoretic consensus mechanism that enables LLMs to self-check their outputs during the decoding stage of output generation. Our method models the decoding process as a multistage Bayesian Decoding Game, where strategic interactions dynamically converge to a consensus on the most reliable outputs without human feedback or additional training. Remarkably, our game design allows smaller models to outperform much larger models through game mechanisms (e.g., 78.1 LLaMA13B vs. 76.6 PaLM540B). As a model-agnostic method, our approach consistently improves even the latest models, enhancing DeepSeek-7B's performance on MMLU by 12.4%. Our framework effectively balances correctness and consistency, demonstrating that properly designed game-theoretic mechanisms can significantly enhance the self-verification capabilities of language models across various tasks and model architectures.

From Self-Check to Consensus: Bayesian Strategic Decoding in Large Language Models
Weitong Zhangf.
Chengqi Zangf
Bernhard Kainz
Imperial College London, UK University of Tokyo, JP, Friedrich-Alexandee University Srrangen-Nümmerg, GER weitong. 2hang200imperiil........mm ac uk
Abstract
Large Language Models exhibit logical inconsistency across multi-turn inference processes, undermining correctness in complex inferential tasks. Challenges arise from ensuring that outputs align with both factual correctness and human intent. Approaches like single-agent reflection and multi-agent debate frequently prioritize consistency, but the expense of accuracy. To address this problem, we propose a novel game-theoretic consensus mechanism that enables LLMs to self-check their outputs during the decoding stage of output generation. Our method models the decoding process as multistage Bayesian Decoding Game, where strategic interactions dynamically converge to a consensus the most reliable outputs with- out human feedback or additional training. Remarkably, our game design allows smaller models outperform much larger models through game mechanisms (e.g., 78.1 LLaMA13B vs. 76.6 PaLM540B). As model-agnostic method, our approach consistently improves even the latest models, enhancing DeepSeek-7BB perfor- mance on MMLU by 12.4%. Our framework effectively balances correctness and consistency, demonstrating that properly designed game-theoretic mechanisms can significantly enhance the self self-verification capabilities of language models across various tasks and model architectures.
1
Introduction
Large Language Models (LLMs)) have demonstrated extraordinary capabilities in tasks such as factual
question answering. fact-checking, and open-ended text generation [9] (54). Yet, this remarkable progress comes at hidden cost: as generative models grow in complexity and scale, they increasingly produce outputs that, while seemingly plausible, can be factually incorrect subtly misleading [46]. This issue, whether caused by the way models are optimized or by unintended hallucinations [4] 2].
leads to fundamental challenge: reasoning inconsistency. Models can give contradictory answers to related questions, which undermines their consistent correctness for coherent inference tasks.
Existing approaches attempt to optimize model outputs through human feedback (e.g., RLHF 17
16 57
However, human feedback is limited. It is hard interpret model behavior [58] and to evaluate the complex logic in Al-generated content [27). making it increasingly difficult for humans to Recent alternatives exhibit critical flaws: single-agent reflection produces inconsistent self-verification resulting for incorrect answers due to while multi-agent debates ultimately lead to collusive reinforcement, i.e., agents converging shared errors 220 64 41 In light of these challenges, are interested in the question:
follow and assess the reasoning of advanced models [10 38
confirmation bias
43
+ Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2: What is the capital of Swi Switaerland
Generator
Verifier
Self-Check to Consensus LLMs Beyond Human Evaluation Limitations
[Correctness]
[Confidence]
Rank4
A: Geneva
Rank4
Rank4; Geneva
Rank3
A: Basel
Rank2
Rank3; Basel
Rank2
A: Zurich
Rank1
Bank2: Bezn
Rank1 A: Bezn
Ronk3
Rank1; Zuzich
Figure Illustration of strategic decoding through a Bayesian Decoding Game (BDG) for the Switzerland capital query (Q.) with candidates (A.), with initial rankings (left) and convergence dynamics (right) between Generator (solid) and Verifier (dashed). The collusive Nash equilibrium validates 'Zurich' (incorrect) while rejecting 'Bem'' (correct) through -hee-the interactions based on choice patterns rather than factual correctness.
How can we enable LLLMs self-check through strategic interactions that overcome single-agent inconsistency and multi-agent collusion while surpassing human evaluation limits 1
Thus, we explore a game game-theoretic theoretic approach and introduce a Verifier, serving as a proxy for human judgment to systematically assess generators as outlined in Fig. [ The motivation for this approach threefold: (1) L.LMs are increasingly employed assist evaluating their own outputs, offering a more scalable alternative to solely relying on human feedback 1 57
49 (2) the flexibility to
adjust game-theorett objectives, such as utilities and policies between the generator and verifier, allows to analyze latent decoding consistency and legibility as a function [31, (27): and (3) in scenarios where human guidance is constrained, structured Al interactions can effectively elicit and refine latent knowledge, thereby enhancing model correctness, and generation consistency [15 63)
Focus of the paper. Realistically, neither models nor humans can be expected be perfectly correct. Thus, our work focuses on achieving consistency through systematic verification through strategic interaction between LLMs calibrated on correctness and confidence measures. We design a multi-step Bayesian Decoding Game with complex action spaces that enable generators and verifiers to iteratively refine their strategies. Through the proposed no-regret optimization, our framework drives agents toward an equilibrium that ensures both consistency and correctness. Thus, we define reliability as simultaneous consistency and correctness. Hence, our approach targets two types of outputs Consistencies: Cases where strategic interactions between agents converge to stable equilibria that reinforce both correctness Inconsistencies: Subtle errors that are revealed through
that are challenging for existing methods: (1) Equilibrium-Base
and consistency. (2) theneetteeretic dynamics, which may go undetected by human evaluation or standard automated checks.
We formulate these challenges as a multi-step Bayesian Decoding Game with complex action spaces (Fig.2) Here, generators and verifiers engage in strategic interactions: generators sample outputs based on latent model knowledge, while verifiers assess these outputs. To enhance the efficiency of this process, we improve upon traditional no-regret optimization through Markovian strategy updates and sesspaaatioon constraints, enabling faster convergence to optimal equilibrium while maintaining clear separation between correct and incorrect outputs.
2 A Bayesian Decoding Game (BDG)
While the ability of LLMs to self-check is crucial for reliable applications, existing verification frameworks lack formal guarantees against collusion. In this section, we develop the theoretical foun- dations of our BDG, focusing on how it overcomes the fundamental limitations of prior approaches through game-theoren theoretic principles. We first formalize the decoding process as signaling game from scratch, then introduce our key innovation: the o;-Separated Equilibrium, which enforces consistency self-checking by preventing collusive behaviors that plague existing consensus mechanisms.
"Single-agen agent reflection and multi-agent debates fail self-checking, please Appx

---

Single-agent Inconsistency
Multi-agent Collusion
BDG Strabegic Consensus
Env.
Signalling Generator Decoding Verifier Evaluation
Update
y1
Enx (Quettee)
(Queries)
(querias)
Gc
Gc
Yn
GC
GeIA
G₁
71
GC
Y1
Vena
Incosrect
Riuut 1ibritm
Yn
contradiction
axnors
contergaoce
(+) Communication Topolagins
(b) BDG Signalling Tree
Figure 2: Overview of the BDG under signaling game structure. (a) Communication Topologies: comparing Singlc-agent Inconsistency with internal contradictions, Multi-agent Collusion that rein- forces errors, and BDG Strategic Consensus achieving correct convergence through o-separation. (b) BDG Signalling Tree: Environment (Env) sends private signals to generator (Gc/1), who generates candidates (y.,,... Yn) for information transmission. The verifier makes judgments (C/I) without observing original signals, enabling strategic decoding through Markovian Strategy Update (dashed) until equilibrium.
2.1 Preliminaries: Modeling LLM Decoding Signaling Game
We first define LLM decoding as a signaling game. This formulation naturally captures the self-check process where LLM must both generate content and verify its correctness while maintaining strategic consistency. The simplest form of a signaling game [25] can described as follows: the generator receives signal (Correct Incorrect) and then takes strategy (choose answer implied by the signal from the candidate answer set) to transmit the signal information to the verifier. The verifier has to make judgment (Correct Incorrect) of the signal based the realized action of the generator. If the judgment matches the signal, both the generator and verifier receive utility 1, and otherwise O; in LLM decoding, the signaling game has been used to fine-tune LLMs to output the best possible answer(s) under equilibrium. For example, the Equilibrium Consensus Game (ECG) proposed by [3]] is pioncering work this problem, but like all existing consensus game frameworks, it fails to address a fundamental challenge: Collusion in a Nash Equilibrium.
Theorem 1. More than one (mixed) strategy Nash Equilibrium exists for this game.
Definition 1. Collusion O in competitive multi-agent game occurs when two or more agents cooperate covertly the disadvantage of others.
Collusion in Nash Equilibrium (NE). Thm. I. is both guarantce and curse; the existence of an cquilibrium ensures convergence, but the presence of multiple equilibria raises the risk of undesirable outcomes under collusion, where low-quality output may incorrectly align with successful verification. The proof and explanation are Appx. I.II This collusion problem directly parallels the failure modes observe multi-agent debate frameworks, where multiple LLMs converge incorrect answers in Appx. C
Example. (Fig. [ Continued) a signaling game, given the query "What the capital of Swirzerland??, one Collusive Nash Equilibrium be given by Correct signal, generator chooses "Zurich". ", verifier judges (Correct) signal, generator chooses "Bern", verifier judges [Incorrect] which means that the verifier makes judgments only conditioning the generator's choice pattern rather than factual correctness. Under this equilibrium, the more plausible but incorrect answer (Zurich) is validated while the correct answer (Ber)) is rejected.
Algorithmic collusion has been studied quite extensively in literature including 67 35 56] To avoid @onilmmmmooooooooonn collusion, BDG enforces a Separating Equilibrium condition for improved consistency.
Mixed strategies refer to probability distribution over actions rather committing to one action. A Separating Equilibrium (SE) 5 is a type of Perfect Bayesian Equilibrium (PBE) Appx. where agents with different types (signal) choose different strategies in equilibrium.

---

2.2 An Optimal Equilibrium for Decoding Games
Collusion Avoidance with Separating Equilibrium. To ensure that both the generator distinguishes between the correct and incorrect signal and the verifier verifies answers correctly, we design the BDG and convergence algorithm to constrain the equilibrium to be a Separating Equilibrium (SE).
Definition 2. (Decoding Game) The Decoding Game is variant of the signaling game in 2 and the utility is determined by the preference ordering of each player, O Pay,, i E [G,V). E {correct, incorrect) S where [ the cardinality of the candidate set y and Plyl is the set of all permulations of elements in Y. The utility can be defined
uG(TG, TV) =u wwv.aaaa TV)
Σ
1(0a Or 18)
2
BES
Strategies are given by ag(- 8, = A Afforreett incorrect)), and are actions. The preference ordering O, on 3 is determined by the actions given signal:
x)( E a()).ar( x, = Tv v) v) E
an xx 2 ac(ys () = ys to My. av(s x. 2 av a.
ys Zv Bj-
(2)
With an incorrect signal, the generator's strategy just maps to a reverse distribution of the correct signal and the utility is determined by the ordering resulting from the action distribution f collusion, BDG introduces a; o--Separteed condition:
To avoid
Definition 3. (a (o:-Separated Equilibrium (a:-SE) For both the generator and verifier, given constants OG, ov, the generator's 's equilibrium strategy said be og-separated, i E {G, V} if and only if ming,ey |aa(y; I, correct) ac(ys incorrecct) > ag. whereas for the verifier, we have ming.e ay (correct x, ys) av (incorrect x. v:)l > σv
Intuition. (Example 2. Continued) The aG separradde constraint enforces that the generator's strategies for different signals must maintain an L1 distance of at least aG. meaning its action distributions distinetly different when receiving different signals. Similarly, ay -spearrati ensures that the verifier's judgment probabilities maintain a clear quantitative distinction of at least dy for each candidate. It prevents the formation of arbitrary consensus based statistical patterns rather than 'factual' correctness.
2.3 BDG Optimization: No-regret Optimization for Equilibrium
No-Regred Optimization. Based on the Decoding Game in 22.2 we propose two strategy update
schedules to numerically achieve optimal convergence of -EE in Defi. 2. 3. The multiplicity of
SE leads to convergence to suboptimal outcomes, necessitating the definition of an initial strategy for each player. This prior is denoted as at x. v) and af
(1) (- s) following (31]
Through repeated interactions and iterative policy refinement, no-regret learning approximates equilibria in large games. Our cumulative regret is defined as:
Reg, (T) =
1 Σ Ui (Ti, us (TT)). T
(3)
where qर is the optimal hindsight strategy that maximizes this value. Rather than computing regret at each iteration, n. is selected based on the time-averaged strategies. In our game, global regret minimization achieved by minimizing regret locally within each information (C and I signal).
Markovian Strategy Update. The players update their strategy by learning the opponent's 's action in the previous period, which denote by Markovian strategy update schedule:
"The Generator G uses strategy To : S x X + A()) so that upon seeing (s, it selects probability distribution ac(- s) = (c(s,2) ( the Verifier V uses strategy πy X x ) - A(S) so that upon seeing (z, selects distribution av 22,)) TV av 2) E A(l correct, incorrect })

---

Ny (.yllayy
Eyav av (: "x,"") aNolt [x,s), s)=
avl. vo(.1a, x,30
ar av(-12,,)
(4)
E av (11
3 ja (ol 2.9) + Ao log of (v (0,6)
oa (y a, & exp
1/ (not) Ao
21 x) x exp av
to (v (.6) + Ay log a (e 2.
(6)
1/ (nvt)
Initial policies (. x. v). ag 2- ANV, aNG the normalized opponent's period t actior where M7, As, i E {G, V}. are the learning rate and stiffness hyperparameter and consistency bound. The two strategy update schedules we propose show satisfactory convergence properties, and the
stopping criteria: 1. Consistency: Og = Ov. 2. Collusion Avoidance: satisfy o;-SE in Defi. 3.
Theorem 2. A Markovian update schedule for a Decoding Game converges to an optimal di- Separated Equilibrium.
The proof can be found in Appx. E.2 Under BDG's utility and the design of the no-regret algorithm, our method reaches a,-SE 30 times faster than the current state-of-the-art [31] based on Average Recall update with accurate correctness alignment between the generator and verifier. Table 5 and Appx. H illustrate the difference in game design between BDG and ECG 31
2.4 BDG Analysis: Properties and Behavior
Equilibrium Properties. At oE with signal distribution P(correct, incorrect) (0.5, 0.5), we analyze the separation characteristics induced by our no-regret optimization. According to the environment, we label the a most preferred candidates as correct, and the rest as incorrect. We denote 2 the candidate in each group as Vi.C, Vs.1. respectively. For candidate set y with |J where mod? 0, we characterize degree of separation between 'correctness' and 'incorrectness' by the separation score:
ay (correct x. x, ( (correct x, :/g++)]-
At this measure quantifies the verifier's initial separation score between the least correct and least incorrect candidates under the prior. When this value is small (when the verifier is ambiguous about the correctness classification). the ambiguity is revealed and sorted through the preference fluctuation during the strategic interaction with the Markovian update, in contrast to the Average Recall update shown in Fig. 3 (right corner). Our separating constraint enforces that the equilibrium separation score is bounded by the same parameter as in Defi. under rational assumption, which is described by the following proposition:
Proposition 1 Under any signal distribution environment such that P(correct, incorrect) = (p. 1-p) s.t. p < 1 and the rationality condition that the equilibrium confidence scores is greater than 1/2 for correct candidates and less than 1/2 the incorrect candidates. the separation score is also bounded below by the same parameter in Defi
dr (correct x, (k)
azy (correct x. 9h+11 2 ov
(7)
if and only if the ox-separated condition is enforced. l is the least correct candidate and l + is the least incorrect candidate in equilibrium, determined by the candidate cardinality and signal distribution.
Prop. [exemplifies how o;-separated condition (Defi 3.) ensure that, in a binary decision-making environment, correct and incorrect candidates can be properly segregated. Our Markovian updates maintain this separation while ensuring convergence, as demonstrated below. However, there is no such guarantee based on the Average Recall update in Table. 5 and Nash equilibrium of Thm.
More details can be found in Appx. E.3 A comparison can be found in Fig. 3
Reliable Behavior. In this paper, we define reliability simultaneous consistency and correctness. The SEE in BDG prevents collusion through strategic separation which ensures reliable behaviors
'The normalization is essentially mapping from opponent's action space to player herself
5

---

of LLMs and agents. In equilibrium, we examine both correctness alignment between the generator and verifier and collusion prevention:
Intuition. (Intuition 2.2 Continued) equilibrium, reliable behavior emerges from two mecha- nisms: the strategic separation enforces strict preference ordering that prevents collusion, while the reliability measure ensures this preference translates to an optimal balance between strategic consistency and behavioral reliability.
We analyze policy entropy dynamics between BDG and ECG to understand the equilibrium be- haviorally. We evaluate convergence through policy entropy H (π) = - Σ T(*) log (+)), which measures agent strategy uncertainty. This metric captures both convergence efficiency (entropy reduction rate) and equilibrium stability (final entropy level) for generator and verifier policies. Fig. shows how BDG achieves reliable separation: the entropy trajectories show rapid stabilization after initial exploration, validating our game-theoree theoretti framework and theoretical guarantees.
Algorithm 1 BDG: Self-checking Mechanism for LLMs
Input: Query æ, Candidate set V, Thresholds da, da, 0 Output: Optimal ranking of candidates
1: Initialize Generator (LLMg))
Operates under partial information
2: ag (ylz,, correct) correct) + fum(ee),,corrett)
Sample with correct prompt b Inverse distribution for incorrect signal Verifying, asymmetric information structure
3: (ylæ, incorrect) < Aoverse(AAmmupllr,comee
4: Initialize Verifier (LLMy))
5:
( (conrect|xr, (r, y) < Fuur(correctjazz y)
Verify answer correct Verify if answer incorrect
6: (4 (incorrect)a x, y) + PLLM (incorreet æt, y)
7: Normalize ay to ensure (1) (correct|a: y) + ov (incorrect|æ, y)
8: t+]
9: while not converged do
10: /a Markovian Update (Eq. 4) Enhancing mutual information */
11:
Generator learns at (s|xr, y) through anv
Generator aligns with Verifier's judgment Verifier aligns with Generator's selection
12:
Verifier learns ag d (ylw,, through aNG
Strategy Update (Eq. 5-6) Strategic response under uncertainty (y(....v) (0) (- (1,9)) Σy' (. (N x. Σ [v..).))6
14:
av(s)x.))
15:
(t+1) 11+1) (vlæ, x exp 2G
In (A ( (vlane)
1/(nat)+
16:
(++1) (s|2, ( v) exp
(0 (ola, log av (slx.))
17(
17:
/ Separation Check (Def. 4) Addressing mutual uncertainty */
oc-condition +- minyey [|a (e+1) (v)l,, correct) ag (++1) (incorrect]z,, y)ll ov
(t+1) (ylær incorrect))
18:
19:
ov co-diniitoon minyey [+(()) (correct|z, y) dy
20:
if oc-condition and ov-codiii then
Prevents collusion in Nash Equilibrium
21:
1* Preference Alignment (Eq. 2) Establishing preference orderings */
22:
Og + Sort y by decreasing 4g
(t++1] (y/ı,, correct)
23:
Oy Sort )' by decreasing di
((+1) (correct(r,
24:
if Oc Ov then
Eq. and Thm.2
25:
return Candidate ranking based Og
Separating Equilibrium reached
26:
end if end if
27:
28:
<- 1<-++ 1++1
29:
end while
30:
return Best correct and reliable candidates ranking
Implementation. A
implementation is shown in Algm. [ with all theoretical concepts
introduced earlier: it initializes the generator and verifier with LLM probabilities (2.1). performs
Markovian strategy updates (2.3). and enforces o-separation conditions (Def. 3. to prevent collusion.
The preference alignment check corresponds to our utility definition (Eq. T. ensuring the game
converges to consensus that represents reliable self-checking rather than arbitrary agreement.

---

St. via Strengg Upoano
BCG. NE. via Avcrnge Rocall Uppkas
Figure 3: Convergence dynamics comparison between Separating Equilibrium (SE) and Nash Equilibrium (NE). We track generator (G, solid lines) and verifier (V, dashed lines) rankings for 10 answer candidates. (a) BDG's Markovian update achieves rapid convergence to SE within 100 iterations, with clear separation in rankings and consistent alignment between G and V. (b) ECG's Average-recall update [31 converges to NE but exhibits persistent oscillations and ranking ambiguity.
Velferr
Greenaee Falicy Verker Faleyy
with SCNI
234-
614-
11A
Netreipenn
(a) Convergence of BDG
(b) Convergence of ECG
Human Eva. wro BDG ECG
Figure 4: Left: Policy entropy dynamics of BDG vs ECG. (a) BDG exhibits initial exploration
followed by rapid stabilization, demonstrating efficient convergence to separating equilibrium. (b)
ECG shows continuous entropy decrease without stabilization, reflecting unstable agent interactions seen in Fig. 3 Right: Performance Comparison. Experts, Non-Experts, and Game- Theoretic Strategies (BDG and ECG) on time and accuracy. The evaluation is based on a user study (n=183) where participants classified LLMM-generated math solutions under three conditions (baseline, BDG- guided, ECG-guided), with expertise levels determined by a 150s temporal threshold.
BDG is game theory-based mechanism that drives LLM self-checking improve accuracy. Our model is a deliberate variant of traditional Bayesian/Signaling games, specifically adapted to address the mutual information between the generator and verifier LLM contexts. Unlike standard signaling games where only one agent faces uncertainty about types, in actual LLM context, both the generator and verifier operate under partial information about the "correctness" of outputs. Furthermore, the preference orderings in our model capture the iterative dynamics of LLM verification. where incorrect judgments must be systematically reversed from correct ones. In particular, standard signaling games assume asymmeíric information structure, whereas our formulation requires modeling mutual uncertainty between generator and verifier while maintaining game-theoretic foundations. This adaptation allows us to address these challenges of L.L self-check mechanisms that standard Bayesian game formulations do not fully capture.
Experiments
Focus and Setting. We aim answer the following questions: (1) What design choices enable decoding games to improve language generation performance? (2) To what extent does our BDG improve consistency? (3) To what extent does the BDG improve factual validity and reliability?
BDG focuses on improving the consistency and reliability of LLMs. However, consistency and reliability manifest themselves in various forms across different domains and dimensions, includ- ing correctness, truthfulness, factuality, valid reasoning, value alignment, among others. We first assess efficiency and reliability through multidimensional comparison with another game-theoretic method (31] and several variants. Then, we evaluate performance on a diverse set of L.MSs used for real-world tasks: MMLU [29]. ARC-Easy (E.), -Challenge (C.) [18), RACE-High (H.) 37 It
7

---

Table I: Comparison of LLMSs' inherent inconsistency (InC.9) and improvements (Imp.%) between Accuracies of (singlc-agent) G, (multi-agent) ECG, and (multi-agent) BDG, colored arrows in the table entries are assigned relative to the G bascline.
Domain
Model
G.tcc
InC. SaG. & VJ
RCG
Impp.%
BDG Arc.
Imsp S
LLsMA-7B
69.0%-
39,9
31,397
40.5
33.2%
MMEU
LLaMA-138
41.7
60.6%-
45.1
8.1%
46.9
12.5%
ARC-E.
LLaMA-7B
68.2
56,1% 1%-%
71.5
4.8%
75.3
10.4%
LLaMA- 13B
71.2
16.1%*
76.4
7.3%
78.1
9.7%
ARC-C.
LLaMA-7B
47.3
65.9%
58.3
23.2%
59.6
26.0%
LLaMA-13B
61.4
19.8%
RACE-H
LLaMA-7B
46.4
62.0%
56.4
21.5%
57.7
24.4%
LLaMA-138
58.8%
62.8
$1.1%
60.3
25.0%
Average
50.6
59.7%
59.0
18.25
60.1
20.2%
Table 2: Model consistency across different domains.
Table 3: Orthogonal enhancements.
Domain
Model
MI
SCD
ECG
BDG
BDG
Domain
Model
shot
few shot
LLaMA-7B
30.4
33.1
30,5
40.4
39.9
40.5
MMLU
LLaMA-138
41.7
41.8
41.7
41.9
45.1
46.9
Eibics Medical
PubMedQA
LLaMA-7B
71.45
71.89
LLaMA-13B
ARC-E.
LLaMA-7B
LLaMA-13B
71.2
68.2
71.5
68.8
69.5
73.0
65.0
$2.5
76.4
71.5
78.1
75.3
MMLU-M.
LLaMA-13R
LLaMA-7B
56.01
$1.35
52.90
58.85
ARC-C.
LLaMA-7B
47.3
47.4
56.5
42.7
58.3
59.6
Justice
LLaMA-13B
52.27
$3.15
LLaMA 138
51.9
52.1
59.3
48.5
61.4
62.2
Virtue
LLaMA-138
33.10
33.82
RACE-H.
LLaMA-7B 7B
138
47.9
46.3
48.4
58.9
53.1
55.1
62.8
56.4
60.3
Deontology
Utilitarianism
LLaMA-138
LLaMA-13B
65.35
52.41
66.75
53.01
Table 4: The reliability across domains with COT.
Table 5: ECG and BDG Comparison.
Domain
Model
Gready
Docoding Methods
Game-theeretic BOG
BDG
Criteria
Sestagy
BR-uphhee depis S0
BDG
BDG
laat-aand belidf up- dat h.,
Thm.
LLaMA-7B
10.8
14.7
13.4
15.0
15.1
15.8
GSMSK
L3MMA-13B
14.9
22.5
23.1
22.5
23.0
22.7
Comergence
NE
SE
TruhfalQA
LLaMA-7B
LLaMA-13B 13B
33.41
33.05
34.79
36.30
34.91
34.61
39.05
34.17
34,27
38.63
35.07
40.01
Update
Complesky
Average Recall
Ols')
O(vllge))
Mackevian
NIA
important to note that BDG is game-theoretic decoding strategy and not a deliberationn/rrainngg based method like a prover-verifier -game (PVG) (27). or contrastive-obeective based generation [2] Nevertheless, we demonstrate effectiveness through benchmarks reasoning task: GSM8K [9]. medical taks: PubMedQA [33], MMLU-MMdical (M.), and ethical scenarios, including justice, virtue, deontology and utilitarianism Ethics (28), that BDG yields reliable improvements and demonstrates synergistic potential across various scenarios.
Action Space in the Game. To define the action space in BDG, the generator selects from finite set of candidates y. For multiple-choice tasks, Y directly corresponds to the given options. For open-ended generative tasks, we construct y by sampling candidates from the LLM's distribution PL g, correct) using nucleus (30] and top-k [22] sampling methods. This standardized action space allows BDG and benchmarks to be applied consistently across different types of tasks while maintaining tractable strategy spaces.
Baselines and Models. For fair comparisons, following the setting and scores 31 use LLaMA models 622 (7B, 13B parameters) with 16-bit inference across all experiments unless otherwise specified. On multiple-choice datasets, employ single-agent and multi-agent patterns: Generative
Ranking (G): Ranks candidates by PuLM(y x., correct) following 9 62): Verifying Ranking (D):
Re-weights query-candidate pairs using dy (correct xr, y) based on Self-Contrastive Decoding
(1)
(SCD): Utilizes ag for reweighting candidates B1 42
ECG: Average Recall update with Nash cquilibrium verifier (z, 3) by ay. (correct x, y) B1: BDG: update qunry-candidate pairs based on Markovian Strategy with SE verifier (z, 3) by at (correct x. z.
Prompting. Unless otherwise specified, the condition for the Pt.. corresponds to the standard zero-shot prompt [31] 29). Furthermore, we combine chain-of-thoughe (CoT) (66), and few-shots setting (66] orthogonal analysis.
8

---

3.1 Evaluation on
Designs
Self-searching & Convergence Behavior. We compare searching behaviors of BDG with the most closely related method, the ECG [31], in the multiple-choice question answering (MCQA) task (18). Fig. 3 and 4 provide a visual case study. BDG demonstrates consistent and reliable convergence. Conversely, ECG exhibits prolonged and inconsistent searching behavior. Despite continuous shifts in candidate selections, ECG fails to achieve stable convergence with persistent disagreement between the generator and verifier. Tab. shows the improved convergence properties of the BDG over ECG.
shows
Game-emergent Inconsistencies. We quantified the degree of inconsistency during the decoding stage by analyzing the disagreement percentage between Generative (G) and Verifving Ranking (V). The game-theoretic ECG and BDG reveal inherent model inconsistencies (InC. following [3] with a 59.7% disagreement rate between them. In Tab. T G and V often yield conflicting results, indicating significant inherent inconsistencies during the decoding stage of generative models. These discrepancies can be effectively mitigated by approach, specifically during the decoding process, without the need for additional training. Tab. shows that BDG consistently outperforms both G and ECG, particularly in cases with higher disagreement rates. We achieve superior consistency with higher correctness with fewer updates in each case Fig. []]
Human vs. Heaee-Theoetti Detection We conducted user study (n=183) evaluating mathematical assessment under three conditions: unassisted baseline, BDG-guided verification, and ECG-guided verification. Performance metrics included solution accuracy and completion time, with participants stratified into expert/non-exper groups based on an empirically determined temporal threshold (150s).
Fig. Freveals significant performance disparities between experts and non-experts, quantitatively illus- trating human evaluation limitations as generation complexity increases. Game-theorele approaches. particularly BDG, enhance decoding effectively, maintaining accuracy while closely aligning with human intent. BDG consistently improves accuracy across non-experts and experts levels and sig- nificantly reduces sample identification time, outperforming unassisted baseline and ECG across multiple dimensions. This also suggest its effectiveness in bridging the expertise gap. Additional results about this finding are provided in Appx. K
3.2 Self-Check to Consensus Benchmarking: Across Domains with Smaller Models
With reasoning and comprehension tasks, we show superior performance compared to baselines and other game-theoretic methods in Tab. 2 due to the efficient alignment consistency. In a broader comparison, our zero-shot LLaMA-133 13B (78. 1, ARC-E.) outperforms larger models, PaLMM550B model (76.6) 14]. With latest DeepSeek-7B, generator is 44.07% on MMLJU, BDG achieves 49.52% over 46.20% with ECG. With more challenging reasoning and multitask understanding tasks, such as ARC-C, RACE-H, and MMLU, we achieve the best equilibrium decoding with fewer rounds and higher accuracy. Our LLaMA-13B (46.9, MMLU; 57.7, RACE-H.) outperforms zero-shot GPT-3-175B (37.7, MMLU) 29), LLaMA-65B (51.6, RACE-H.) 62]. and PaLM-540B (49.1, RACE-H.) 29
3.3 Orthogonal Enhancements for Robust Consistency and Consensus
Datasets in Tab. 3 involve challenging scenarios to test models' reasoning abilities. We use these benchmarks to study whether we can combine our approach with various orthogonal strategies. Based on game theory. BDG does not conflict with the computationally intensive game mechanism during training. nor does it conflict with CoT and few-shot variations. BDG shows enhanced performance in more challenging scenarios in Table 4 establishing highly novel direction in decoding research. Furthermore, it achieves broader accuracy and robustness across datasets, underscoring its adaptability and trustworthiness.
Discussion, Limitations, and Conclusion
Game Design over ECG. BDG and ECG share the common goal of aligning generative models with human intentions to improve output reliability, yet they differ significantly in their game design. BDG achieves substantial gains with reduced computational overhead. Appx. exxplore the distinct phases and transitions between these frameworks, highlighting BDG's scalability and its departure from training-intensive intensive frameworks.
6 HHw often the answers chosen by the Generator (G) and Verifier (V)

---

Robustness and Integrative Potential. BDG achieves consistent performance improvements across diverse domains, maintaining effectiveness even with lower-quality initial LLM outputs. The frame- work readily integrates with existing techniques such as self-consistency and CoT prompting, while offering fast equilibrium convergence and reliable verification.
Limitations. One potential limitation arises from the explicit specification of correctness consistency branches during the game process, as this alignment is primarily intended to match human intent with model outputs, similar to game-based approaches B1 277. Adding multi-metrics and multiple agents to achieve game- deliberation is possible.
Conclusion. We introduced the Bayesian Decoding Game, a game- thenrethe framework addressing limitations of single-agent inconsistency and multi-agent collusion in LLM self- verificeiiii Through separating equilibrium and Markovian updates, BDG enables smaller models to outperform larger ones without additional training. Our approach offers a practical advancement towards reliable LLM systems while demonstrating performance improvements across benchmarks.
A Acknowledgements W.Z. is supported by the JADS programme and the UKRI Centre for Doctoral Training in AI for Healthcare (EP/S023283/1). High-performance computing resources were provided by the Erlangen National High Performance Computing Center (NHR @ FAU) at Friedrich-Alexanderr (FAU), under the NHR projects b143dc and b 80dc. NHR is funded by federal and Bavarian state authorities, and NHR @ FAU hardware is partially funded by the German Research Foundation (DFG) 4400 19683. Additional support was received by the ERC project MIA-NORMAL 101083647, DFG 51281 19079, 513220538, and by the state of Bavaria (HTA). We also acknowledge the use of Isambard-Al National AI Research Resource (AIRR) 45 Isambard-Al is operated by the University of Bristol and is funded by the UK Government's DSIT via UKRI; and the Science and Technology Facilities Council [ST/AIRR/IAA-/1023]
Universität Erlangen-Nümbe
Potential Ethics Risks and Societal Impact
Our Bayesian Decoding Game (BDG) is a novel both the consistency and reliability of large language model outputs. By framing the decoding process as multistage signaling game between generator and verifier, BDG efficiently aligns model outputs with human intent while mitigating the trade-off between correctness and reliability. BDG ensures reliable and robust LLM outputs, offering scalable, training- free solution to the challenges of ambiguity and inconsistency in generative models.
framework that significantly enhances
References
[1] Y. Bai et al. Rlaif: Reinforcement learning from ai feedback. arXiv preprint arXiv:2304.03442 2023.
[2] Z. Bai, P. Wang, T. Xiao, He, Han. Zhang, and M. Z. Shou. Hallucination of multimodal large language models: A survey, arXiv preprint arXiv:2404. 18930, 2024.
[3] A. Bakhtin, D. J. Wu, A. Lerer, J. Gray, A. P. Jacob, G. Farina. A. H. Miller, and N. Brown. Mastering the game of no- press diplomacy via buguaazegll reinforcement learning and planning. arXiv preprint 10.5422200.05492, 2022.
[4] S. Banerjee, A. Agarwal, and S. Singla. Lims will always hallucinate, and we need to live with this. arXiv preprint ar 11t:::24990004466111 2024.
[5] J. Black, N. Hashimzade, and G. Myles. A dictionary of economics. Oxford University Press, USA, 2012.
[6] T. Bonjour, Aggarwal, and Bhargava. Information theoretic approach detect collusion multi agent games. In Uncertainty in Artificial Intelligence, pages 223-232. PMLR, 2022.
[7] N. Brown and T. Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418 424, 2018.
[8] N. Brown and T. Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456):885 890, 2019.
[9] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), pages 1877-1901 2020.
10

---

[10] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer. J. Rando, R. Freedman, T. Korbak, D. Lindner. P. Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.152111 2023.
[11] S. Casper, C. Ezell, C. Siegmann, N. Kolt, T. L. Curtis, B. Bucknall, A. Haupt, K. Wei, Scheurer, M. Hobbhahn, et al. Black-box access is insufficient for rigorous ai audits. In The 2024 ACM Conference on Fairness, Accountability. and Transparency, pages 2254-2272, 2024.
[12] J. C.-Y. Chen, S. Saha, and M. Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms, arXiv preprint arXiv:2309.1 13007, 2023.
[13] I.-K. Cho and D. M. Kreps. Signaling games and stable equilibria. The Quarterly Journal of Economics, 102(2):179-221, 1987.
[14] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Rescarch, 24(240):1-113, 2023.
[15] P. Christiano, A. Cotra, and Xu. Eliciting latent knowledge: How to tell if your eyes deceive you. Google Does, December, 2021.
[16] P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint
2018.
[17] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[18] P. Clark, Cowhey, O. Ezzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05457, 2018.
[19] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, al. Training verifiers to solve math word problems. arXiv preprint ar Xiv: 10.14 2021.
[20] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. 14325,, 2023.
[21] F. R. D. T. (FAIR),, A. Bakhtin, N. Brown, F. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 37866624):106-10777 2022.
[22] A. Fan, M. Lewis, and Y. Dauphin. Hierarchical neural story generation. arXiv preprint #rxiv:180- 805.04 2018.
[23] D. Fudenberg. Game Theory. MIT press, 1991.
[24] T. Gao, X. Yao, and Chen. Simese: Simple contrastive learning of sentence emheddings. arXiv preprint arXiv:2104.0889 2021.
[25] R. Gibbons et al. A primer in game theory. 1992.
[26] M. Hegazy. Diversity of thought elicits stronger reasoning capabilities in multi-agent debate frameworks. arXiv preprint arXiv:2410. 12853, 2024.
[27] Hendrik Kirchner. Y. Chen, H. Edwards. J. Leike, N. McAleese, and Y. Burda. Prover-verifier games improve legibility of Ilm outputs. arXiv e-prints, pages arXiv-2407, 2024.
[28] D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt. Aligning ai with shared human values. arXiv preprint arXiv: 02275, 2020.
[29] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0330 2020.
[30] A. Holtzman, J. Buys, L. Du, M. Forbes, and Choi. The curious case of neural text degeneration. arXiv preprint 1904v:1104.009990 009511,99111111 2019.
[31] A. P. Jacob, Y. Shen, G. Farina, and J. Andreas. The consensus game: Language model generation via equilibrium search. In The Twelfth International Conference Learning Representations, 2024.
11

---

[32] A. P. Jacob, D. J. Wu, G. Farina, A. Lerer. H. Hu, A. Bakhtin, J. Andreas, and N. Brown, Modeling strong and human-like gameplay with kl-regularized search. In International Conference Machine Learning. pages 9695-9728. PMLR, 2022.
[33] Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.061 46, 2019.
[34] A. Khan, J. Hughes, D. Valentine, L. Ruis, K. Sachan, A. Radhakrishnan, E. Grefenstette, S. R. Bowman, T. Rocktäschel, and E. Perez. Debating with more persuasive Ilms leads to more truthful answers. arXiv preprint wtw.:::24226666666 2024.
[35] Koirala and Laine. Algorithmic collusion a two-sided market: rideshare example. arXiv preprint arXiv:2405.02833 2024.
[36] A. Kori, B. Glocker, and F. Toni. Explaining image classification with visual debates. arXiv preprint #xXiv:2210009013 2022.
[37] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, Race: Large-scale reading comprehension dataset from examinations, arXiv preprint arXivv1704. 704,04683, 2017.
[38] Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint
2018.
[39] J. Li and D. Jurafsky. Mutual information and diverse decoding improve neural machine translation. arXiv preprint arXiv: www.:1601.0000 2016.
[40] L. L. Z. Chen, G. Chen, Y. Zhang. Y. Su, E. Xing, and K. Zhang. Confidence matters: Revisiting intrinsic self-correction capabilities of large language models. arXiv preprint arXiv:2402. 12563, 2024.
[41] X. Li, S. Wang, Zeng, Y. Wu, and Y. Yang. A survey on Ilm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9, 2024.
[42] X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto, L. Zettlemoyer, and M. Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022.
[43] X. Liang. Song, Z. Zheng, H. Wang, Q. Yu. X. R. Li, Y. Wang, Wang, Xiong, et al. Internal consistency and self-feedback in large language models: A survey. arXiv preprint arXiv:2407.14500 2024.
[44] T. Markov, C. Zhang, S. Agarwal, F. E. Nekoul, T. Lee, S. Adler, A. Jiang, and L. Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 15009-135 15018, 2023.
[45] S. McIntosh-Smith, S. Alam, and C. Woods. Isambard-ai: a leadership-class supercomputer optimised specifically for artificial intelligence. In Proceedings of the Cray User Group, pages 44- 54. 2024.
[46] I. R. McKenzie, A. Lyzhov, M. Pieler, A. Parrish, A. Mueller, A. Prabhu, E. McLean, A. Kirtland, A. Ross, A. Liu, et al. Inverse scaling: When bigger isn't better. arXiv preprint arXiv:2306 20440499900009799, 2023.
[47] C. Meister, T. Pimentel, G. Wiher, and R. Cotterell. Locally typical sampling. Transactions of the Association for Computational Linguistics, 11:102-121, 1211, 1211122222 2023.
[48] J. Mökander, J. Schuett, H. R. Kirk, and L. Floridi. Auditing large language models: a three-layered approach. Al and Ethics, pages 1-31, 2023.
[49] T. Mu, A. Helyar, J. Heidecke, J. Achiam, A. Vallone, I. Kivlichan, M. Lin, A. Beutel, J. Schulman, and L. Weng, Rule based rewards for language model safety, 2024.
[50] N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
[51] OpenAL. CharGPT-4: Version Opus- 2024.
[52] J. Perolat, B. De Vylder, D. Hennes, E. Tarassov, F. Strub, V. de Boer, P. Muller, J. T. Connor, N. Burch, T. Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990 996, 2022.
12

---

[53] K. Pillutla, S. Swayamdipta, R. Zellers. J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems. 34:4816 4828, 2021.
[54] A. RRafford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint MxXiv22103000222 2021.
[55] C. Rastogi, M. Tulio Ribeiro, King, H. Nori, and S. Amershi. Supporting human- ai collaboration in auditing Ilms with Ilms. In Proceedings of the 2023 AAAU/AMM Conference on AL, Ethics, and Society, pages 913 926, 2023.
[56] I. Sadoune, M. Joanis, and A. Lodi. Algorithmic collusion and the minimum price markov game. arXiv preprint arXiv:2407 233352, 2024.
[57] W. Saunders, C. Ych, J. Wu, S. Bills, L. Ouyang. J. Ward, and Leike. Self-critiquing models for assisting human evaluators. arXiv preprint UXXv:2206.05882 2022.
[58] C. Singh, J. P. Inala, M. Galley, R. Caruana, and J. Gao. Rethinking interpretability in the era of large language models. arXiv preprint 21X0:22402.11761 2024.
[59] Y. Su, T. Lan, Y. Wang. D. Yogatama, L. Kong, and N. Collier. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems. 35:21548-21561, 2022.
[60] Z. Tang, R. Wang, W. Chen, K. Wang, Y. Liu, T. Chen, and L. Lin. Towards causalgpt: A multi-agent approach for faithful knowledge reasoning via promoting causal consistency in Ilms. arXiv preprint arXiv:2308. 1123308.11144 11914, 2023.
[61] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201 100099. 2022.
[62] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B, Rozière. N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 139711 2023.
[63] M. Turpin, J. Michael, E. Perez, and S. Bowman. Language models don't 't always say what they think: faithful explanations in chain-of tf-thoughtt prompting. Advances in Neural Information Processing Systems, 36, 2024.
[64] Q Wang, Wang. Y. Su, Tong. and Y. Song. Rethinking bounds of Ilm reasoning: Are multi-agent discussions the key? arXiv preprint #xXi:2242..18777 2024.
[65] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, Narang, A. Chowdhery, and Zhou. Self-cons improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 11171, 2022.
[66] Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of -f-houg prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
[67] Z. Xu and W. Zhao. On mechanism underlying algorithmic collusion. arXiv preprint xrXiv:2409.01147, 2024.
13

---

Appendix Contents
A. Reproducibility Statement
B. Potential Ethics Risks and Societal Impact
C. Collusion in Single-agent Reflection and Multi-agent Debate
D. theoretc Formulation Supplementary
E. Proofs of Theorems
B.1 Proof of Thm.
B.2 Proof of Thm. 2
B.3 Proof of Prop.
E. Extended Related Work, Discussion, and Distinction
G. From Bayesian Decoding Game (BDG) to Prover-Verifie Game (PVG)
H. From Bayesian Decoding Game (BDG) Equilibrium Consensus Game (ECG)
I. Experiment Details
J. Searching & Convergence Behavior Supplementary
K. Human Evaluation
14

---

A Reproducibility Statement
We conducted our evaluations using widely recognized benchmarks such as ARC-Easy, ARC- Challenge, MMLU, and RACE. The experiments were performed using the open-source LLaMA 7B and 13B models. Key aspects the game, including update policies and initial strategies, are thoroughly detailed in both the main text and appendix to facilitate accurate replication of the results. experiments were conducted on NVIDIA A6000 and A100 GPUs, with runtimes ranging from 0.5 to 6 hours depending on the model size, task, and experimental settings. Further details on the game-theoretic mechanisms and specific design choices can be found in the methods section and following pseudocode model-agnoistic implementation.
Algorithm 2 Bayesian Decoding Game (BDG): Self-checking Mechanism for LLMs
Input: Query 2, Candidate set y. Thresholds dg. dv.
Output: Optimal ranking of candidates
1: Initialize Generator (LLMg):
b Operates under partial information
2: Gg comrect) PLLM(yla, corret),, correct)
Sample with correct prompt 3: ag (ylæ, incorrect) + Reverse((LLLMU/ correcttt) correct)) > Inverse distribution for incorrect signal
4: Initialize Verifier (LLMy):
Verifying, asymmetric information structure > Verify if answer correct Verify if answer is incorrect
5: ay (correct x. y) + Pulm(correct)a, y)
6: "v (incorrect) y) PalM (incorreetc v)
(1) 7: Normalize ay to ensure af (correct)r, + av (incorrect) z. =
(1)
8: +-
9: while not converged do 1* Markovian Update (Eq. 4) Enhancing mutual information
11: Generator learns a. (s) 2z) y) through aNv
Generator aligns with Verifier's judgment
(1) Verifier learns a (ylæx, 8) through aNG
D Verifier aligns with Generator's selection
13:
A* Strategy Update (Eq. 5-6) Strategic response under uncertainty */
x,y) Eyeov ((a,"") aN (2 E. (-1,))))))
(-r.w)
(-2.6)))
(y/ x, s) x exp ((+1)
ta (elx,))) log ald (gl
15:
ag
1/(nat))Ao
av 1) (s|z,y) y) x exp
te aa (x).))) + log a (sla.))
16:
1/(nvt)+Xv
17:
/ -epaprraiion Check (Def. 4) -Addressing mutual uncertainty */
ac conddittoon +- mingey/ ||a ((+1) (y correct) - "G
(t+1) (ylx, incorrect)| > aG
vv oy-coniition mingey ||a (t+1) (correct|z, y) av (++1) (incorrect 22 v)ll > σv
19:
20:
if ac-condition and cy-codiitoon then
& Prevents collusion in Nash Equilibrium
21:
1* Preference Alignment (Bq. 2) Establishhiig preference orderings */
(++1) (yl [r, correct)
22:
OG < Sort y by decreasing ag
23:
Oy Sort Y by decreasing ay
((+1) (correct])))))
24:
if G = Oy then return Candidate ranking based on OG
b Eq. 1 and Thm. Separating Equilibrium reached
25:
26:
end end if
28:
+ ++t+1 ++1
29:
end while
30:
return Best correct and reliable candidates ranking
15

---

B Potential Ethics Risks and Societal Impact
Our Bayesian Decoding Game (BDG) is novel game-theorets framework that significantly enhances both the consistency and reliability of large language model outputs. By framing the decoding process as multistage signaling game between generator and verifier, BDG efficiently aligns model outputs with human intent while mitigating the trade-off between correctness and reliability. BDG ensures reliable and robust LLM outputs, offering scalable, training-free solution to the challenges of ambiguity and inconsistency in generative models.
With the improvement of generation quality, automatic generation of fake news) that may be hard to distinguish from human-authored content. It might be worthwhile to augment current decoding techniques so that the generated outputs will also be watermarked without compromising their quality. More potential ethics risks and societal impact illustrated in Fig.
imagine more potent disinformation (e-g.,
Portent's Desoription: Ivv beon having persistent abcominal pain after meals, frequent diarrhea, blood in significant fovor
Exsluste Integhal.
tela
3. Integrate::
Valid:
disease. further
( 1
f
(colonsscops, imagingl and treatment with and infarmmatory medications
Eolloee (2- 48201- 4444000-
What the difirreniial disgrous based on the findings this witl scan?
Specious:
Specious:
Integrate was sieble 20ne
reveals globisstoms...
changes and reduction.
Evalste
f mido you
Specious: The deettifees meningloma..
(a) Decision Making
(b) Logicul Reasoning
(c) Visual Collaboration
Figure 5: Distinguishing different type of LLM outputs, particularly when human evaluation may overlook plausible errors. The three panels demonstrate how models can generate both accurate and reliable, and plausible but misleading responses.
16

---

C
Collusion in Single-agent Reflection and Multi-agent Debate
While both single-agent reflection and multi-agent debate approaches have shown promise in en- hancing LLM outputs, they face fundamental limitations that compromise their reliability. Our experiments reveal critical insights into these limitations, providing compelling motivation for our game-theoretic self-check mechanism. In this case,
The Collusion Problem in Multi-agent Debates. Prior work has demonstrated that multi-agent debating frameworks can improve reasoning and factual accuracy [20] [26] 60). However, our investigations reveal concerning dynamics when scaling these approaches. As shown in Figure 6 while increasing agent count initially improves accuracy on the MMLU dataset through cross- verification, this trend reverses beyond certain threshold. This phenomenon, which term "collusive reinforcement," occurs when multiple agents converge on shared inaccuracies or biases, creating false consensus that appears authoritative but remains factually incorrect. Essentially, agents begin to collude unintentionally by reinforcing each other's 's errors through mutual agreement rather than critical examination, resulting in a deceptive appearance of reliability through consensus.
Agent Accuracy Across Debate Rounds MMLU Dataset
0.620
0.62
0.60
589
0/588
0.592
Accuracy
0.583
0.58
0.77
573
0.56
0.565
0.50
Agent
Figure 6: Agent Accuracy Across 3 Debate Rounds on MMLU Dataset. The graph shows initial improvement as agent count increases from 1 to 5, followed by diminishing returns and eventual decline with further agent additions, demonstrating the emergence of collusive reinforcement.
This collusion effect is particularly evident in factual knowledge tasks where agents lack proper verification mechanisms. Without structured way to challenge potentially incorrect consensus, agents tend to reinforce rather than correct each other's errors. The computational cost additional agents further exacerbates this issue, creating situation where resources increase while accuracy plateaus or declines.
The Inconsistency Problem in Single-agent Reflection. Single-agent reflection approaches suffer from fundamental limitation: self-corruption. During reflection, LLMs frequently demonstrate inconsistency in their judgment processes, often agreeing with their previous outputs regardless of correctness. When a model attempts to verify its own statements, it creates a closed feedback loop where initial errors rationalized rather than corrected, forming self-reinforcing pattern of confirmation bias. This inherent tendency to justify rather than critically examine its own outputs renders single-agent reflection unreliable for detecting subtle errors, particularly in complex reasoning or factual verification tasks. The model essentially becomes trapped in its own reasoning patterns, unable to escape initial errors even through multiple reflection rounds.
The Need for Structured Strategic Verification. Our GSM8K experiments (Figure 7 further demonstrate that while adding debate rounds initially improves performance, benefits plateau after approximately three rounds. This indicates that simply increasing computational resources through more agents rounds does not address the fundamental verification problem.
17

---

Agents' Accuracy Across Various Rounds on GSM Dataset
Performance Line
Agent, Round (3,3)
0.89
3846
0.88
00.8
5882
00.86
00.8
0.84
0.83
0.82
0.81
0.80
5.0
4.5
3.5
3.0
Round
2.5
8
2.0
Agent
10
1.5
1.0
Figure 7: Agents' Accuracy Across Various Rounds GSM Dataset. The 3D visualization demon- strates that performance peaks at specific combinations agent count and debate rounds (marked by the red X), beyond which additional resources yield diminishing returns or performance degradation.
These findings highlight the need for structurally different approach to veff-verification. Rather than relying on consensus-based debate or single-agent reflection, we propose game-theorett framework that explicitly models the verification process as strategic interaction. By formalizing the self-check mechanism through Bayesian games, we can overcome the collusion problem inherent in multi-agent debates and the corruption problem in single-agent reflection, creating more reliable verification process that balances correctness and consistency.
Our BDG directly addresses these limitations by enforcing separating equilibrium condition that prevents collusive convergence, while maintaining the efficiency benefits of two-agent system. This allows for effective self-checking without the resource intensity of large multi-agent systems or the confirmation bias of single-agent reflection.
18

---

D Game-theoretic Formulation Supplementary
generative language model (LM) maps input output y according some distribution L x). Here, we do not impose restrictions on the form of input or output, as illustrated in Fig. 2 Instead, we address nautti-faceted problem involving question and set of answer candidates y generated by pre-trained language models on specific tasks. In the first stage, using this candidate set, we leverage generative LMs in two distinct ways:
Generatively, by supplying as input
1. prompt
2. the set of candidates },, and
3. a natural language prompt indicating that a correct or incorrect answer is desired. The LM may be thought of modeling distribution PLM(y) x, incorrect), where the token incorrect denotes the fact that the model was prompted to generate an incorrect answer.
Verifiably, by supplying as input
1. the same x and
2. possible candidate answer E V, together with
3. prompt indicating that correctness assessment (correct, incorrect] is sought. In this case, the language model acts as models distribution PLM(ss x, x, where s E [correct, incorrect).
The essence signaling game [25] that one player of certain type (the generator) takes action, to convey information to another player (the verifier) about her type; in the simplest setup, the final payoff depends on whether the verifier correctly judges the generator's type based on the generator's 's signal. Based on this intuition from game theory. [3I] design ECG, without formal definition of the game, Thus, for the first time, we provide comprehensive theorete formulation for generative model decoding, and propose improvements to address limitations.
Formally, the signaling game's 's components be defined as follows:
1. Players: Generator and Verifier;
2. Choice sets: Generator's 's choice (Correct, Incorrect), and the Verifier's choice set is Cv (Correct, Incorrect), based on the generator's choice y E y:
E CG V, with prompt p randomly drawn from
3. Payoff Function: ug uy 1p, s), where 1 equals if the correctness prompt matches the verification result, and 0 otherwise.
We are now ready state the fundamental concept of this signaling game, Perfect Bayesian Nash Equilibrium (PBNE) 3 We use the short form Perfect Bayesian Equilibrium (PBE) with the auxiliary definitions Defi and 1. PBE Definition.
Definition (Perfect Bayesian Equilibrium 23 A Perfect Bayesian Nash Equilibrium (PBE) is pair (s, b) of strategy profile and set of beliefs such that
1. is sequentially rational given beliefs and 2. is consistent with s.
Example 1. For generative model decoding, the generator's belief is given by its perceived probability distribution, P(/correct, incorrect)) = (P:, 1 - Pi). J. for each E Y of the verifier's judgment, and with its belief and type, the generator chooses mixed strategy that maximizes its utility, i.e., if the generator's 's type is correct, then its optimal mixed strategy would be allocating positive possibility only on y such that p:
p; and zero possibility to other y:-
In the decoding game did not precisely discuss belief is because the signal distribution is common prior and has been incorporated into the multiplier of the strategy update.
Definition 4. (Sequential Rutionality) A player is said to be sequentially rational if and only if. at each information set it is to move, it
19

---

maximizes their expected utility given their beliefs at the information set (and given that he is at the information set) even this information is precluded by their own strategy.
Definition 5. (Consistency on Path)
Given any (possibly mixed) strategy profile s, an information set is said to be on the path of play if Given any strategy
and only if the information set is reached with positive probability according profile and any information set on the path of play of s, the beliefs of player at I are said to be consistent with if and only if his beliefs derived using the Bayes rule and s.
20

---

E Proofs of Theorems
E.1 Proof of Theorem 1
Theorem
More than one (mixed) strategy Nash Equilibrium exists for this game.
Proof of Theorem
Suppose that the candidate set has options (can be extended to any cardinality ()), y1: 12, one equilibrium can be described as: If the environment sends correct/incorreet, the generator generates the probability distribution (1, 0)/ (0, 1) for (y1. 82) given their belief that verifier probabilistic judgment, (correct, incorrect), for 31, 82 is (1,
For the verifier, suppose its belief is that the environment chooses the state "correct" or "incorrect," and that the generator produces (y1, U2) with probabilities (1, 0) in the correct state and (0, 1) in the incorrect state. Given this belief, the verifier's best response
(1, 0), if observes V1
(correct, incorrect) = (0, 1), if observes y/2-
Together with the generator's 's strategy specified above, these beliefs and actions constitute a perfect Bayesian equilibrium (PBE) of the game.
second PBE can obtained by flipping every and in the strategy profile and in the corresponding beliefs; that is, the generator swaps the probabilities assigned to (1/1 U2) and the verifier swaps the outputs (1, 0) and (0, 1).
F,2 Proof of Theorem 2
Theorem 2. The Markovian update schedule will converge to an a-separated equilibrium of the Decoding Game.
Proof of Theorem 2.
We will show that the Markovian update schedule no-regret for correct generator, and when generator receives correct signal, she will automatically perform the reversed strategy; then, the Markovian update schedule converges for the correct signal, it automatically satisfies that the Markovian schedule will converge for the incorrect signal and thus to a o-separated equilibrium our Decoding Game.
(T) Reg:
Σ u4 (mi; _ u. (T
T
and in each t, the utility is the order matches and 0 if not; therefore, the schedule ensures that there exists T such that the period regret, defined by
regf = u; T.. (T, Us
(0) - 0 for &
(8)
Then the asymptotic regret converges to O, and we will show that the Markovian update schedule satisfies the condition above, will show this from generator's side and the verifier's proof follows the same steps.
The proof will follow 2 steps: firstly, we will prove a lemma that shows under Markovian schedule, once preference relation between any two candidates, y1 1/, aligns between the generator and verifier, then they will not diverge.
Lemma 1. Without loss of generality, we assume if aG (T") ( xr, x, 21. > a/ invariant.
ag (T (TT) xr, and ay (T') (s U2), then for all t > T", under Markovian update, this preference relation stays
(T" (s
This lemma obvious as the update is the exponential a linear combination with positive coeffi- cients of two actions, and both exponential and linear combination preserves monotonicity.
For the second step, we will prove lemma that ensures the Markovian update strict contraction mapping to ensure the preference ordering coincides in finite times.
21

---

Lemma 2. Then there exists finite round T such that for all > T" generator's 's and verifier's preference orderings coincide under the Markovian update schedule.
Proof of Theorem 2. At round t denote ge(y) ag 2, y) and the normalized verifier action as (y)
ag (v x, correct and et ez(y) dv (correct
(t)
Fix any pair of candidates y, z. Define the log-ratios
R (y. 2) :- In 9x(3)
(x(2)
R2 (y: =) =- In (y)
(2)
the generator's update is given by
(1+1(1) x ge(y) exp (- 2
A
04 =
A::= :
+ Ao
+ AG
not
NGt
so
R4 (v, at (v, 2) +
2 Be (v, 2).
(9)
the verifier's update is symmetric (write a/ 3/ for the verifier's coefficients)
(1+1( a, (v. 2) + R 2).
(10)
2
Let
D, R2 (y. R2 (y, 2)-
be a signed measure of "disagreement" for the pair y, if A Av T1G rJv, subtract get:
from
9
1
D411
a+
2 0
Rl
a
31
R c. Dt
(11)
where
C = αt
2
3/
+ AG
For unequal X, one gets two positive weights whose sum is below 1 and the same contraction argument goes through.
Assume mild regularisation: Ag, 1.2 Then 0 < < 1 for every t and ct 1, then from we have
11
Di+1 2) (i(,))
Dil,, 2
II
0
10000
-=
Hence for every pair (y, z),
lim R2 =) = lim R (y, =),
00000
t0
ie. (x(g) Ex(y)
for all y, z. Because convergence is by a strict contraction, there exists Tg.
9.(2)
(2(2)
such that for all f T
sign (R) 2)) sign (R) (y.z))
so the ordering of y versus 2 the same for generator and verifier from that round on. Since there finitely many pairs,
T == max Ty
00
8=z
and for every > T the entire rankings are identical. Therefore combing Lemma L. and Lemma
2. we verified condition g which shows that Markovian update schedule is indeed no-regret for the BDG.
22

---

E.3 Proof of Proposition
Proposition Under any signal distribution environment such that P(correct, incorrect)
(p. 1 p) s.t. < 1, if the equilibrium confidence scores is conditioned that the correct candidates are greater than } and the incorrect candidates is less than 2. then the separation score is also bounded below by the same parameter in Defi
ar (correct x. yk) a (correct 2, (k+11) 2 oV
(12)
if and only if the se-seereeeed condition satisfied. where the least correct candidate and k + 1 is the least incorrect candidate in equilibrium, determined by the candidate set cardinality and signal distribution.
Proof of Proposition 7 we first show that the o--separatedness implies separation score bound.
for any given environment such that the signal distribution given by P(correct, incorrect (p. 1 p) this proof applies, for simplicity, we will provide the proof only for P(p, 1 p) (0.5, 0.5), the only difference will be index the least correct and least incorrect candidate.
According to Defi. 3. the -e-saaartted condition, in equilibrium, the inequality below is satisfied
lay (correct x, Myy - a (incorrect y ) ov
and
(correct x, v,
o. (incorret 2, 12)
thus the inequality becomes
22 ai (correct x; v2) - ov
condition that the equilibrium confidence for correct candidate being greater than and the incorrect candidate being less than 2. thus we can remove the absolute value and get 2-a (correct x, .3y) - 1 > O, moreover, for at (correct 27 Vg +1). we have
1 a (correct x, vg++1) ov
adding the two inequalities together, divided by 2, we get
(correre x; yy) - a (correct x. (2,94+1) > σv
Conversely, we prove the implication in the opposite direction via contradiction: at termination, the d;-separation condition fails to hold, then the resulting separation score S must satisfy
S < c < ov. <v.
for some constant c strictly smaller than σv.
We first assume that for all candidates, the correct and incorrect confidence score is bounded above uniformly by some o' such that o' < dv, which is given by
la (correct xr y:) at (incorrect x, x2,4)l < o'
with the boldfaced rationality condition, we have that
2 ay (correct Ir yy - 1< o'
1 - 2- ai-rrete
Ir, x2, < <dd
adding the two equalities together we have that
ay (correct x. v/3 2- ai (correct x. u v 20' ai (correct x, - - (correct x, x,,4+1) o' < ov,
which showcases that without enforcing [3. under the boldfaced rationality condition, the separation could be bounded above by some constant less than σv.
23

---

F Extended Related Work, Discussion, and Distinction
E.1 Related Work
Multi-Agent Debate Systems. Previous work has explored mechanisms where multiple language model instances "debate" to refine and converge final answer [20] 12 34 [6]. is possible to categorize our method as major variant of this multi-agent debate in which the interaction occurs within a game- theoretie framework, rather than directly within the language models' outputs. This structured signaling game enables BDG to enhance the correctness of outputs without relying on human feedback, by dynamically optimizing the generation and verification processes. Additionally, this approach can resolve ambiguity, confusion, and low accuracy caused by inconsistencies, but not by poor reasoning. Conventional signaling game settings have been successfully deployed for Poker 7 [8 Stratego [52], Diplomacy [21
3 32 and L.M tasks 27 12. Building on these insights, we propose novel signaling game framework between a generator and verifier for systematic LLM output verification.
Single-Agent Strategies. Top-k sampling nucleus sampling 30), and typical sampling focus on generating high-confidence text but do not address the correctness of the outputs. Candidates were generated using these methods. Equilibrium-rankke to the initial distribution. In contrast, BDG integrates multistage signaling game that inherently balances correctness and consistency during the generation process. BDG can seamlessly combined with these strategies to enhance the reliability of generated text. Furthermore, single-agent ranking is a widely used approach to select the correct output from a set of candidates generated by language models. [61] use additional human annotations to train a ranking model for response filtering. [27] trains different provers and verifiers for increasing output legibility. Although our work also utilizes existing language models verifiers, BDG eliminates the need for additional training and does not impose specific assumptions on the structure of either the generator or verifier.
[3] applies an average- average-moving moving strategy
F.2 Discussion and limitations
Game Design over ECG. BDG and ECG share the common goal of aligning generative models with human intentions to improve output reliability, yet they differ significantly in their game design, achieving substantial gains with reduced computational overhead. While ECG utilizes moving- average updates to foster consensus, often leading to unstable and fluctuating equilibria, BDG employs structured Bayesian framework that drives interactions toward optimal equilibrium with greater stability. contrast, Prover- Verifier Games (PVGs) [27]. which contribute to ChatGPT ol (5T), RL-based alignment and focus adversarial training phases featured by RL and competitive dynamics. This requires intensive training and causes potential deviations from cooperative strategies.
Robustness and Integrative Potential. BDG achieves consistent performance improvements across diverse domains, maintaining effectiveness even with lower-quality initial L.M outputs. The frame- work readily integrates with existing techniques such as self-connistency and chain-of-thought prompting, while offering fast equilibrium convergence and reliable verification.
Balancing Correctness and Reliability. Reliability (55) tries to give an account of the prover model's failure modes and mense-making, whether the reasoning is correct or not. The resulting decoding can be arbitrarily complex [50]. In contrast, correctness allows to verify if given solution is correct, ignoring how the generator reasoned it to be reliable (consistent with the environment). Consequently, reliability requires model outputs that are coherent and consistent to human understanding [48). We show that it is possible to have both without sacrificing correctness for reliability [27). and especially in high-stakes settings reliability is as important as correctness 11
24

---

G From Training-free Bayesian Decoding Game (BDG) to RL-based Prover-Verifter Game(PVG)
PVG [27], structured as zero-sum games, encounter substantial challenges that undermine their efficacy in ensuring reliable outputs. The adversarial nature of zero-sum games inherently prioritizes winning over mutual consistency, which leads to strategic behavior focused on exploiting the opposing agent rather than achieving genuine correctness e.g. model collapse. This often results in provers generating outputs that are optimized to mislead the verifier rather than to align with factual truth, thus producing equilibria that favor strategic manipulation over accurate assessment. Such dynamics complicate the training process, requiring extensive tuning and computational resources without guaranteeing robust, interpretable results. Furthermore, the reliance on reinforcement learning in these systems falls short of effectively replacing human feedback, as the trained verifier cannot fully replicate the nuanced judgment required to evaluate complex or ambiguous output. These limitations fall into the misalignment between training objectives and practical needs, where models become adept at adversarial optimization but lack the reliability and consistency necessary for real- world application. However, Bayesian Decoding Game (BDG) can bring the advantages of the game and bridge the purposes with proper implementations, which bypass the extensive training and adversarial pitfalls by directly modeling output verification through probabilistic reasoning, enhancing the interpretability and reliability of the generated content without the dependencies on zero-sum competition.
Here, we consider the connections between the PVG and BDG, and before the comparison, we give a brief introduction of PVG.
We consider a scenario of problems with ground-truth answers (x, y) e X x Vr, where denotes a problem statement and y(4) denotes the mapping from the prompt x to ground-truth answer or set of answers. PVG has set the prover's type to be drawn from known distribution, which can be either honest or sneaky, to produce an answer 2 condition on the prompt x and type; such that the honest prover told produce correct answers and make sure the verifier recognize the correctness, whereas the sneaky prover is told to trick the verifier by providing wrong answers that are not easy to be detected; then, after receiving solution, the verifier verifies the answer based solely on the answer and prompt, and the output vzz, =) (0,1) estimates the correctness of the solution's 's answer. The output of the verifier is either (tee statement correct) or O(the statement incorrect).
More precisely, The game follows the procedure below:
1. The type of prover i. is randomly drawn from known distribution P(t), such that E 8), where h, a dictates honest and sneaky prover's type.
2. Given a problem statement, the prover generates proofs by maximizing the payoff func- tion(utility) of each round myopically, such that
z = argmax (v(2, z), 1t=n (z, 2) + 11= Loorect z)))
the function r r(c, q) satisfies that r(1, 1) r(0, 1) r(1,0) > r(0, 01), which corresponds to the fact that both prover prioritizing generating proofs that match their types (correct answer for honest and incorrect answer for sneaky), Vo r(v, 1) > r(v, 0) and then both want their answers to be accepted as correct by the verifer Vi r(1,4) )) > r(0,4).
3. the verifier provides verification v E {0, based solely on the proof x given that the utility is 1 if =) = (ax, =) and otherwise Theoretically, this is a simple Stackleherg game such that an equilibrium exists and before div- ing into the equilibrium strategy profile, an important definition from (27 needs to be mentioned.
Definition 6. (Checkability (27)) Prover x from model family H verifiable to verifier on the data distribution Pxy if and only if the prover always generates correct and convincing solutions, and no sneaky prover from the same model family H can fool the verifier into accepting incorrect solutions. In muthematical notations, the two properties are given by:
Completeness: Vx E X, 1l correct (I,, (x)) = v(x, x(x))
Soundness: Va' H. Vx E X, 1corect (z, (x)) = (z, n r O.
25

---

With definition 6 in hand, the equilibrium strategy profile is given by (the honest prover always provides checkable and correct proof, the sneaky prover always provides noncheckable and incorrect proef. the verifier can always verify the correctness of the given proof). For the neural networks to approximate the equilibrium strategies, (27 utilized reinforcement learning-basse algorithm to train the prover and the verifier.
RL-based PVG [27] can fit in part into the framework of our training-free BDG framework. As for the game-theoretic setting, PVG is zero sum verifier-lead Stackleberg game, the strategy update schedule must be modified to the utility defined [27]. Moreover, the verifier's strategy update cannot be achieved training- free as her utility only depends on the ground truth right/wrong of the candidate and thus needs be trained; but on the prover side, both honest and sneaky prover can update strategies pain-free free from the verifier's ' trained strategies.
Firstly, we define the strategy for verifier and prover the same way as BDG, such that given the environment signal, the prover generates probability distribution for set of answers, and the verifier always generates probability distribution of (correct, incorrect) for each of the answers. Also, make the same assumption that each player can observe the opponent's 's full strategy profile rather than the realized strategy; then, we are ready to highlight the difference in schedule update under the Markovian schedule, the condition where correct, we will abbreviate that as correct C, stays the same, such that because they want to align their strategies with the verifier
bp
(1++1) y
I. H) = av (C x.y)
SUp
(y xr C) + Ap log o (y x, C, ) 1/ (met) Xp
( QP x. H. O exp
However, for the sneaky prover, her utility is maximized when the verifier mistakens the correctness of the problem. Therefore, the optimal update schedule for the sneaky prover given updating toward normal distribution over the preference generated by the probability distribution of verifier's strategy. The reason for this update because, near the correct/incorrrettess boundary is where the verifier tends to make mistakes, such that
{ NN av) Ap log (2) (y xr, b,b b3))
y x. S) x exp (++1) ap
1/(p)) + Xp
For example, if there are 10 answer candidates, the verifier's preference from her strategy is given by Ys > 57 > > Ys > y1 > V22 > > V1 > Us. then N(y av) is given by
Normal Distribution Over N Discrete Answers Indexed by Permutation
0.25
0.20
Prebabiity
0.05
0.00
V3
57
V6
35
V1.
32
19
94
Index (Permutation of Cy. 03
26

---

H From Memoryless Bayesian Decoding Game (BDG) to Moving-average Equilibrium Consensus Game (ECG)
The moving average update schedule proposed by [31] requires both the generator and the verifier to keep track of the average strategy of the opponent addition to the strategy in the last round, while our Markovian framework allows the players to be memoryless. To better compare ECG with our update schedule, we provide a general, unifying framework called the History window schedule, where the player's belief is given by the average of past history strategies for the period n, and at the same time, this schedule retains a large part the initial policy for each round with a stiffness parameter A E V.. The belief is given by
(1+1)
(y x,
Σ
(T) ay
2,y)
(13)
by
(1+1)
(s x.
0 (y x,
Thus the strategy update is given by
G (y 8. + AG log ag ())) x; 8)
(1+1)
ag (x+ (y x, s) x exp
1/ (nGt) + A
20 ((tt))
(y (. + Av log a (s x.
ag (1+1 x) x exp
1/ (nvt)+
As it can be noted in 13 if take the update schedule coincides with ECG which requires the memory of the moving-average of full history, rather if we take = 1, the update schedule becomes fully memoryless and requires no memory of any past events other than last period's opponent strategy.
27

---

I Experiment Details
Baselines and Models. For the fair comparision following parameter models from the LLaMA family[62 and perform 16-bit inference for all our experiments. Since have multi-round optimization game and in order distinguish consensus/ zero-sum games, we define ours as a verifier rather than a verifier. Across the experiments, all the approaches and orthogonal techniques involved:
the same public 7B and 13B
Generative Ranking (G): The baseline 9 62 ranks every candidate y by PL | x, correct)
and picks the top candidate. This is the standard approach used in past work. Due to implemen- tation differences and non- con-public resourees, we report the existing scores in B1.
verifying Ranking (D): Following[31], this approach reweighs every query-candidate pair (x, v) (1) (correct x. v). Typically, this would surpass the performance of ordinary individuals, who might neglect to notice the ambiguity errors. And outstrip the generators that might trust the unreliable decoding.
by TV
Mutual Information Ranking (MI): The mutual-information based baseline reweights every candidate y by PLM(Y z, correct) (correcttt z, y) [39].
utilizes the contrastive-base generator #G to reweight every candidate y by "G (correct x. v). This
Self-Contrastive Decoding (SCD): The contrastive- based method 31 42
(1)
method achieves contrasting effect by comparing negative samples instead of employing verifier (in BDG/ verifier (in ECG).
Equilibrium Consensus Verifier (ECG): This approach is based on verifier कं. B1 It
reweighs every query-candidate pair (x, y) by T (correct x,y). This method, involving comprehensive policies and updates, serves as our main benchmark.
Bayesian Decoding Game (BDG): This approach utilizes our Bayesian Decoding Game-based verifier ... This approach reweighs every -uery-candidate pair (z, y) by (i-occerect 2,, y).
Orthogonal Techniques. Furthermore, BDG can combine chain-of-thoughh (CoT) 66 and few-shots setting [66] as orthogonal extra gains.
CoT [66] prompting enables language models to generate interme- diate reasoning steps, improving performance on complex tasks. By providing exemplars of reasoning chains, the model is guided to produce more coherent and accurate responses.
Chain-of-Thouge (CoT):
Few-Shot: Few-shot setting [66] involves providing the model with a small number of example input-output pairs within the prompt. This technique helps the model adapt to the task at hand without additional fine-tur tonintuingg improving its ability to generalize from limited data.
Hyperparameters. We set TDD AD and GG. ÅG with 0.1 compared to ECG. Experiments are run 5000 times with early stopping based on equilibrium convergence. BDG can usually converge by 500 iterations or less. The hyperparameters can be larger according to the tasks and initial model ability.
Extra Metries. Following
42
have
Diversity, This metric aggregates n-gram repetition rates:
unique n=grams(
DIV = II
n=2 total n-grams(Fwwm)
Models that score low for diversity are prone to repetition, while models that score high for diversity are lexically diverse.
MAUVE. MAUVE
5 measures the similarity between generated text and gold reference text.
approximates coherence by cosine similarity between the sentence embeddings of prompt pre and generated continuation Tront:
Coherence. (59
EMB (Fpre)
EMB (Fomt))
Teoont: COH
COH
Fpre)
|EBB| (Free)
EEB (T
where EMB(2) represents the pre-trained SimCSE embedding 24).
Human Evaluation. To further evaluate the quality of the generated text, we consider two critical aspects: correctness and confidence in reliability. More details can be found in the next section.
28

---

J Searching & Convergence Behavior Supplementary
Question:
Wlich oress evars duiing ostora sea [ean, canaoc Iddooggen Bors the enmosspeeeeooccom water? A Righ B lighhniing
C. presoue change tempevature Shange
Centaet
Phoaare Chanpe Alteruu selasiiy allectog ecgess
I: Wiv Disiltones bordosse' oogee
Algh Wonds (A) Goolo mote betó more ooges.
Demparatore Siex shbrww hoddy
(oD Wau-driin receese ogger
(9) posupppent argien
Ageh Whndl (I) Savwe wind caclange mntor oogges
It BoOosf Foow (8) invlvserrin riggrn fooll
0.72
4
0.62
0.60
50
100
200
300
Scarching
Searching
Model Presiton
Figure 8: BDG's game design quickly reaches equilibrium and consensus between the generator and verifier, typically within 100 epochs. In contrast, ECG requires significantly more epochs (3000 in this case) and exhibits continuous fluctuations (as shown in the lower right) before achieving consensus. (Zoom in for details.)
We first compare searching behaviors of BDG with the most closely related method, the ECG 31, in the multiple-choie question answering (MCQA) task [18]. provides a visual case study. BDG demonstrates a swift and consistent convergence in (b).
Conversely, the ECG, shown in (c), exhibits prolonged and inconsistent searching behavior. Despite continuous shifts in candidate selections, ECG fails achieve stable convergence with persistent disagreement between the generator and verifier. (d) and Tab highlights the enhanced and fast convergence properties of the BDG over the ECG.
29

---

K Human Evaluation
Setting, In this experiment, participants were tasked with evaluating the correctness of ten answers to high-school level multiple-choice mathematics problem generated by Large Language Model (LLM). Participants were instructed classify each answer correct, incorrect, ambiguous. The experiment was conducted in two stages:
In the first stage, participants were given minutes classify many answers as possible, and their results were recorded. In the second stage, participants were allowed to allocate their time freely complete the remaining classifications, and they asked record the time upon completion of their classifications. Below is the questionnaire we utilized for the experiment.
Each participant was randomly assigned three distinct problems, and the corresponding solutions classified under three conditions: without any hints, with BDG hint, and with ECG hint. The hints provided were rankings of the answers generated by the respective models (BDG and ECG). The assignment of different problems across the three conditions memorization and to control for potential confounding effects related to the content of the specific problem. Problems were drawn from pool of questions with similar difficulty levels, allowing for consistent observation of treatment effects across varying problem sets.
designed prevent
Samples. To better illustrate the experiment setting details, we provided the questionnaire interface, the instructions, and two cases set below.
Please read the instructions
Minutes (Mandatory)
Unlimted Time
#Question
Type
Ambiguous
(Seconds)
Ambiguoue
(Saconds)
wio
No
Soi No.(s)
Sol No./s)
Seconds
Sal No.te
Sal No
Seconds
No.
No.ss)
Sei
Seconds
No.
Seconds
No.
Seil No.
Sol No.
No.ts
No
Submit Questionnaire
30

---

Instructions
For task 1, you will judge the correctness of each solution under a mandatory 2-minute limit, and for the ones that you are unsure, label them ambiguous and record the result on the answer sheet provided. If you did not finish in 2 minutes, then record your incomplete result at the 2-minute cutoff and continue your task until finished, also remember to record your completion time.
For task 3, will provide reference (ranking of the solutions assist and accelerate your correctness judgement, however, always stick to your own judgement if there exists divergence. All other procedures the same task 1.
Question #
Random lected
On Cartesian coordinate plane, points (1,2) and (7, 4) are opposite vertices of square. What is the area of the square?
Choices
A. 20
B. 34
C. 40
D. /40
Task 1
Candidates
O.
"To find the area of the square, multiply the length of the diagonal, which is /40, by itself. This gives us 40, so the answer is C. 40."
"The distance between the points (1, 2) and (7, 4) /40. This is the diagonal of the square, so the area could either be 20 or 40. The answer might be A. 20."
2.
"The diagonal of the square is v40, and the area must be equal to this value. The answer is D. /40."
"First, calculate the diagonal between the points (1, 2) and (7, 4), which gives v40. Since this is the diagonal of the square, the area is (v2o)² 20. Therefore, the answer is 20."
2
4. "The area of the square is found by calculating the distance between the points. The diagonal is 440, so the area is likely / /40. The answer is D. /40."
5.
"First, find the distance between (1, and (7, 4) using the distance formula: d (77-1) 1)2 (4 2)² /40. This distance is the diagonal of the square. The of the square is then (v/0)² 20. Therefore, the correct answer is A. 20."
"Using the distance formula, the diagonal of the square is /40. Therefore, the area of the square is 20. The answer is A. 20."
"First, calculate the distance between the points (1, and (7,4), which gives the diagonal the square. The distance formula gives /(7 - 1)² + (4 2) /40. Since the diagonal of the square /40, the area is (v40)² / = 20. There- fore, the answer A. 20."
"Using the distance formula, the diagonal of the square found to be /40. The area of the square is 20, so the answer is A. 20."
"Calculate the diagonal of the square using the distance formula: d= /40. Since the diagonal /40, the area of the square simply 40. Therefore, the answer is 40."
Task
BDG Guidance [Will be hidden during evaluation]
0.
Consistent
"First, find the distance between (1, and (7, 4) using the distance formula: d ((7 - 1)2 (4 2y2 V40. This distance is the diagonal of the square. The area of the square is then (v40)² 20. Therefore, the correct answer is A. 20."
31

---

"First, calculate the diagonal between the points (1,2) and (7, 4), which gives /40. Since this is the diagonal of the square. the area is (va0)² answer is A. 20."
2 20. Therefore, the
2. "Using the distance formula, the diagonal of the square is v40. Therefore, the area of the square 20. The answer is 20."
3.
"Using the distance formula, the diagonal of the square is found to be /40. The area of the square is 20, so the answer is A. 20."
"The distance between the points (1, and (7, 4) /40. This is the diagonal of the square, the area could either be 20 40. The answer might be A. 20."
"First, calculate the distance between the points (1, and (7, 4), which gives the diagonal the square. The distance formula gives v(7 - 1)² + (4 2
/40. Since the diagonal of the square is /40, the area is (v40)2/2 2 = 20. There- fore, the answer A. 20."
"The diagonal the square is 400 and the area must be equal to this value. The answer is /40."
"The area of the square is found by calculating the distance between the points. The diagonal is / /40, so the area is likely /40. The answer D. /40."
8. "Calculate the diagonal of the square using the distance formula: d = /40. Since the diagonal is /40, the area of the square is simply 40. Therefore, the answer is C. 40."
9.
Inconsistent
"To find the area of the square, multiply the length of the diagonal, which is v 400, by itself. This gives us 40, so the answer is C. 40."
Task
ECG Guidance [Will be hidden during evaluation]
0.
Consistent
"First, find the distance between (1, and (7, 4) using the distance formula: d ((7- - 1)2 2 400. This distance the diagonal of the square. The
area of the square then (v 20. Therefore, the correct answer A. 20."
1.
"Using the distance formula, the diagonal of the square is /40. Therefore, the area of the square 20. The answer is A. 20."
"First, calculate the diagonal between the points (1,2) and (7, 4), which gives v40. Since this is the diagonal of the square, the area (V40)² 20. Therefore, the answer is A. 20."
2
3. "The distance between the points (1, 2) and (7, 4) is /40. This is the diagonal of the square, so the area could either be 20 or 40. The answer might be A. 20."
4.
"Using the distance formula, the diagonal of the square is found to be 440. The area the square 20, so the answer A. 20."
"Calculate the diagonal of the square using the distance formula: = 40. Since the diagonal is /40, the area of the square simply 40. Therefore, the answer is C. 40."
"First, calculate the distance between the points (1, 2) and 47, which gives the diagonal the square. The distance formula gives /(7 1)2 + (4 2)² /40. Since the diagonal of the square is /40, the area is (V40)² /2 = 20. There- fore, the answer A. 20."
"The diagonal the square is v 400 and the area must equal to this value. The answer is /40."
32

---

"To find the area of the square, we multiply the length of the diagonal, which is / 40, by itself. This gives us 40, the answer C. 40."
Inconsistent
"The area of the square is found by calculating the distance between the points. The diagonal is v40, so the area is likely v40. The answer is D. v40."
Question #
Random se- lected
The standard tip in restaurant is 15% of the bill before tax. Many people in California find the tip by doubling the sales tax, which is 8.25%. By how many cents does this exceed the standard tip bill of $60 before tax?
Choices
90
80
C. 70
6.
Task 1
Candidates
"The tax doubled 16.5% and the standard tip is 15%. The difference around 70 cents for $60 bill, so the answer is C. 70.
1.
"To find the tip, double the sales of 8. 8.25%, which gives 16.5%. The standard tip is 15%. and the difference on a S60 bill is 60 x (16.5% - 15%) = 90 cents. Therefore, the answer A. 90."
"Doubling the sales tax of 8.25% gives 16.5%. The difference between the standard tip and this new tip is around 60 cents for $60 bill. Therefore, the answer is D. 60."
3.
"Doubling the tax gives tip of 16.5%, and the difference from the standard 15% tip 90 cents. Therefore, the answer A. 90."
4.
"The standard tip is 15% of 60, which is 60 x 0.15 = 9 dollars. Doubling the sales tax of 8. 3.25% gives 16.5% and the tip based on this is 60 x 0.165 9.90 dollars. The difference is 9.90 9.00 = 90 cents. Therefore, the correct answer is A. 90."
5.
"Doubling the sales tax gives 16.5%, and the standard tip is 15%. This difference gives 60 x 0.5% = 30 cents. Therefore, the answer is A. 90."
6.
"The standard tip is 15%, or 9, and doubling the sales tax gives tip of 16.5%, or 9.90. The difference is 90 cents. The correct answer is 90."
"The difference between doubling the sales tax and using the standard tip is 90 cents for 60 bill, but it could also be 80 cents depending rounding. might be A. 90 or B. 80."
8.
"The standard tip is 15%, and doubling the tax gives different percentage. The difference could be either 60 or 90 cents. The answer might be D. 60."
9.
"Doubling the sales tax gives tip rate of 16.5%. while the standard tip is 15%. The difference for 60S bill is 90 cents. So the answer is A. 90."
Task
BDG Guidance [Will be hidden during evaluation]
Consistent
"The standard tip is 15% of 60, which is 60 x 0.15 = 9 dollars. Doubling the sales tax of 8.25% gives 16.5%, and the tip based this is 60 x 0.165 9.90 dollars. The difference 9.90 - 9.00 = 90 cents. Therefore, the correct answer is A. 90."
I.
"The standard tip is 15%, 9, and doubling the sales tax gives tip of 16.5%, or 9.90. The difference is 90 cents. The correct answer is 90."
33

---

"To find the tip, we double the sales tax of 8.25%, which gives us 16.5%. The standard tip is 15%, and the difference on $60 bill is 60 x (16.5% 15%) = 90 cents. Therefore, the answer A. 90."
"Doubling the tax gives a tip 16.5%, and the difference from the standard 15% tip is 90 cents. Therefore, the answer A. 90."
"Doubling the sales tax gives tip rate of 16.5%, while the standard tip is 15%. The difference for 60$ bill is 90 cents. So the answer is A. 90."
"The difference between doubling the sales and using the standard tip is 90 cents for 60 bill, but it could also be 80 cents depending on rounding. It might be or 80."
"Doubling the sales tax of 8.25% gives 16.5%. The difference between the standard tip and this new tip is around 60 cents for $60 bill. Therefore, the answer is D. 60."
7.
"Doubling the sales tax gives 16.5%, and the standard tip is 15%. This difference gives 60 x 0.5% 30 cents. Therefore, the answer is 90."
8.
"The tax is doubled to 16.5%. and the standard tip is 15%. The difference is around 70 cents for $60 bill, so the answer is 70. 9.
Inconsistent
"The standard tip is 15%, and doubling the tax gives different percentage. The difference could be either 60 or 90 cents. The answer might be D. 60."
Task
ECG Guidance [Will be hidden during evaluation]
Consistent
"The standard tip is 15%, 9, and doubling the sales tax gives tip of 16.5%, or 9.90. The difference is 90 cents. The correct answer is A. 90."
1.
"The standard tip is 15% of 60, which is 60 x 0.15 = 9 dollars. Doubling the sales tax of 8.25% gives 16.5%, and the tip based on this is 60 x 0.165 9.90 dollars. The difference is 9.90 - 9.00 = 90 cents. Therefore, the correct answer is A. 90."
2.
"To find the tip, double the sales of 8. 8.25% which gives 16.5%. The standard tip is 15%, and the difference on a $60 bill is 60 x (16.5% - 15%) = 90 cents. Therefore, the answer A. 90."
"Doubling the tax gives a tip 16.5%, and the difference from the standard 15% tip is 90 cents. Therefore, the answer is A. 90."
"Doubling the sales tax gives tip rate of 16.5%. while the standard tip is 15%. The difference for 60S bill is 90 cents. So the answer is A. 90."
5.
"The difference between doubling the sales tax and using the standard tip is 90 cents for 60 bill, but could also be 80 cents depending on rounding. might be A. 90 B. 80."
"Doubling the sales tax gives 16.5%, and the standard tip is 15%. This difference gives 60 x 0.5% 30 cents. Therefore, the answer is A. 90."
"The tax doubled 16.5%, and the standard tip is 15%. The difference is around 70 cents for $60 bill, so the answer is C. 70.
8. "The standard tip is 15%. and doubling the tax gives different percentage. The difference could be either 60 or 90 cents. The answer might be D. 60."
9.
Inconsistent
"Doubling the sales tax of 8.25% gives 16.5%. The difference between the standard tip and this new tip is around 60 cents for a $60 bill. Therefore, the answer is D. 60."
34

---

Results. To differentiate between expert and non-exper participants, threshold of 150 seconds was based on empirical observations of participant behavior. This threshold was corroborated by scatterplot that visually demonstrated the partitioning between experts and non-experts, supporting the appropriateness of the selected cut-off time for classification performance. From the 183 samples we collected, we have come to conclusions:
Firstly, human evaluation these LLM-generated solutions have instrinsic unreliability and am- biguity, especially for non-expert participants; under the 2-minute time constraint, there were 28 participants who scored an accuracy of 50% or below, even without time constraint, there were 10 participants in total whose accuracies were 60% or below.
Secondly, with the hint of BDG, observed that the participants in both the expert and the non- expert groups outperformed the participants without the hint or the ECG hint; more importantly, with the hint of BDG, the performance of the non-expert participants improved significantly with respect to the expert participants, whereas the ECG hint did not achieve this.
Thirdly, across problems, samples with BDG hint achieved the highest accuracies for 7/10 problems and fastest completion also for 7/10 problems. This result showcases the robustness of BDG hint also within the same problems.
Lastly, the BCG hint has also helped non-expert participants spot ambiguous solutions more accurately compared to non-expert participamts with the ECG hint or without the hint.
35

---

Generations.
Question.
Suppose the graph of y = f (x) includes the points (1,5), (2,3), and (3. Based only on this information, there those points (a, b) and (c, d), what is ab + od?
points that must be the graph of y fx() If we call
BDG.
"Given the points (1, (2, and (3, 1) the graph of y f(x). compute f(f(2)) = f(3) = 1 and f(((3) - f(1) = 5. So the points (2, and (3, 5) are on the graph of y - f(((x)) Therefore, ab + od 2 1133x3 17.
ECG.
"From f(z), we see that f(f(2) = 1 and f(f(3) = 5. Therefore, ab + cd = 17. The answer 17"
Question.
Positive integers and y have a product 56 and x < y. Seven times the reciprocal of the smaller integer plus 14 times the reciprocal of the larger integer equals 4. What is the value of
BDG.
"Let the integers be x and Since 56 and y. try small values. Testing 2 and y = satisfies 7 1 + 14. 1 4. Therefore, x 2, so the answer 2."
ECG.
"Since y 56, try small values of x like 2. This satisfies the equation, the answer is 2."
Question.
Six positive integers are written on the faces of a cube. Each vertex labeled with the product of the three numbers the faces adjacent the vertex. If the of the num- bers on the vertices is equal to 1001, then what is the sum of the numbers written on the faces?
BDG.
"The numbers the faces of the cube contribute the product each vertex. Given that the sum of the vertex products is 1001, and knowing the cube's 's geometry, we deduce that the sum of the numbers on the faces is 31."
FCG.
"Using the fact that the sum of the vertex products is 1001, and knowing there are six faces, we calculate that the sum of the face numbers is 31. So the answer is 31."
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state our paper's main contributions: intro- ducing novel game-theoretic approach (Bayesian Decoding Game) that enhances consistency and reliability during L.LM decoding. The claims about performance improvements (e.g., 78.1
LLLMMA3B vs 76.6 PaI.M540B) are supported by our experimental results in Section 3.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine include aspirational goals as motivation as long clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss limitations in Section 4 (Discussion and limitations), noting that our approach requires explicit specification of correctness consistency branches during the game process, and that this alignment primarily intended to match human intent with model outputs. We also acknowledge potential for further improvement by adding multi-metrics and multiple agents to achieve game-based deliberation.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "L.imitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
37

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The paper provides complete theoretical foundations and proofs for all theoretical claims. Theorems 1, 2, and Proposition are stated with clear assumptions and are accom- panied by rigorous proofs in Appendix sections A.3 (Ap__thm)) A.4 (App_thm3) and A.5 (App_prop!).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Section 3 provides detailed experimental setups, datasets used (MMLU, ARC-Easy, ARC- ARC-Challenge. RACE-High, etc.), model specifications (LLaMA-7B/13BB) and implementa- tion details. Additionally, we include detailed reproducibility statement in Appendix Section 1 with pseudocode implementation (Algorithm 1) of the Bayesian Decoding Game
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (e-g., with an open-source dataset instructions for how to construct the dataset).
38

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., to registered users), but should be possible for other researchers have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide model-agnostic pseudocode implementation in Algorithm (Appendix Section I) and detailed instructions for reproducing our results. The benchmarks we used (MMLU, ARC, RACE, etc.) are publicly available datasets, and we used open-source LLaMA models for our experiments. Complete implementation details are provided in Appendix Sections and 7.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips..c/ppullll
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips.cc/public/guu
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
experimental results for the new
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparame- ters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3 and Appendix Section provide comprehensive details of our experi- mental settings, including hyperparameters (7), Åp. JJ. Àg set to 0.1), model specifications (LLaMA 7B and 13B with 16-bit inference), datasets used, and evaluation metrics. We also specify hardware configurations (NVIDIA A6000 and A GPUs) and runtime information.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
39

---

Justification: Our experiments report statistical significance through comprehensive evaluations across multiple datasets and models. For our human evaluation (Section 3.1 and Appendix Section 9), we collected 183 samples with clear statistical differences between conditions. We ran our game-theoretic experiments 5000 times with early stopping based on equilibrium convergence to ensure robustness of results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report 1 I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates). If error bars reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In the Reproducibility Statement (Appendix Section 1), specify that experiments were conducted on NVIDIA A6000 and A100 GPUs, with runtimes ranging from 0.5 hours depending on model size, task, and experimental settings. We also note that BDG typically converges within 500 iterations or less, providing a clear indication of computational requirements.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpge////arrrpssccppppa//hhhhhe
Answer: [Yes]
Justification: Our research fully conforms to the NeurlPS Code Ethics. We have used publicly available models and datasets, appropriately credited prior work, and discuss potential ethical risks and societal impact in Appendix Section 2, including considerations about potential for misuse in disinformation scenarios.
Guidelines:
40

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss broader impacts Appendix Section 2 (Potential Ethics Risks and Societal Impact). Positive impacts include improved reliability and consistency of LLM outputs, which enhances trustworthiness. We also acknowledge potential negative impacts such as more convincing disinformation if the techniques misused, as illustrated in Figure 9.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, is legitimate to point that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper introduces decoding methodology rather than releasing new models or datasets. We utilize existing publicly available models (LLaMA) and benchmark datasets. Our method aims to improve reliability and reduce misinformation, serving safeguard itself rather than creating new risks that require additional protections.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
41

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit all existing assets used in our research. We cite the original papers for LLaMA, deepseek models and all benchmark datasets. All models and datasets used are publicly available resources with appropriate licenses for research purposes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The primary new asset introduced in our paper is the Bayesian Decoding Game (BDG) framework. This thoroughly documented through formal definitions, pseudocode (AI- gorithm in Appendix Section 1), and detailed explanations of the game-theoretic mechanisms. We provide comprehensive implementation details that allow for reproduction of our approach. Guidelines:
Justification:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicabll well as details about compensation (if any)?
Question:
Answer: [Yes]
Justification: Our paper includes a user study with 183 participants. Appendix Section (Human Evaluation) provides the complete experimental setup, including instructions given to participants, example questionnaires, and sereenshots of the interface used. We report detailed analysis of the results including methodology for expert/non-expert classification.
Guidelines:
42

---

for The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information the supplemental material is fine, but the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our human evaluation study involved minimal risk to participants they were only evaluating mathematical solutions. The study was conducted in accordance with our institution's s ethical guidelines for human subject research, with appropriate informed consent from all participants. The task involved no collection of personal or sensitive information.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it an important, original, or non- standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Our research focuses directly on improving LLM decoding methods, and we use LLaMA models (7B and 13B) as the foundation for our experiments. Section 3 clearly describes how these models were utilized in our Bayesian Decoding Game framework, including prompting methods, model configurations, and implementation details.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy what should or should not be described.
43


Reviews:
[{"review_id": "UZKUPM79bc", "reviewer": "Reviewer_YiqR", "summary": "This paper aims to address the issue of consistency or flip-flopping and correctness when large language models reason. The paper outlines the trade-off between inconsistency in single agent setting as well as a Nash equilibrium collapse in multi-agent setting where models converge to the incorrect answer. Therefore, in order to build consensus during inference, this paper proposes formulating this as a Bayesian Decoding Game and proposes an optimization algorithm to achieve this. The paper tests this hypothesis on a variety of natural language tasks and (older) language models.", "questions": "See weaknesses above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Addresses an important connection between consistency and correctness as two desirable properties of good reasoning systems.\n- The decoding method is fairly novel and outperforms the baselines\n\n**Weaknesses**\n- Missing citation to related work: Several works have addressed the issue of inconsistency in language models which the paper does not address or cite: \n    - https://arxiv.org/abs/2203.11171 (this paper also uses consensus to improve reasoning performance, so why not compare against it?)\n    - https://arxiv.org/abs/2311.08596\n    - https://arxiv.org/abs/2212.09251\n- Missing explanation of baselines: The experimental section of the paper does not outline the choice of baseline decoding methods. What do MI, D, SCD methods stand for? Why not use few-shot evaluation for reasoning tasks like GSM-8K where the reported numbers are exceptionally low and it is well-known that chain-of-thought sampling yields high improvement boosts. Why only include zero-shot and few-shot evaluation in Table 3 and why not report other decoding baselines apart from BDG.\n- Missing justification for choice and models. Clearly the models used in this paper are a few generations behind and obsolete, making some of the results not highly convincing. The paper fails to address this and show gains from their decoding method still persist with recent generation of models and with chain of thought sampling.", "was_revised": true, "final_justification": "I think the authors have mentioned to include the additional works in their related works section and sharpen their contributions and contrast. I also think in the camera-ready they should experiment with more recent models, but I think the paper can be accepted.", "rebuttal": "**We thank Reviewer YiqR for highlighting the connection between consistency and correctness, and for noting that our decoding method is novel and outperforms strong baselines. We address the reviewer’s concerns below by clarifying the positioning of our work and its distinction from the cited literature, and by reaffirming our key contributions.**\n\n---\n\n**1. On the relation to cited works**\n\nThe reviewer recommends comparing against three recent works [1,2,3]. While we appreciate the relevance of these papers to the general topic of reasoning and consistency, they differ substantially from our proposal in both goal and method.\n\nRather than sampling diverse rationales or probing sycophancy in multi-turn dialogue, we study *decoding-time strategic consensus* with formal game-theoretic guarantees. Concretely:\n\n- The **first paper** proposes a prompt + sampling strategy (sample-and-marginalize over CoT outputs). In contrast, our BDG modifies decoding itself via a two-player Bayesian game between a generator and verifier that provably converges to a *σ-separated equilibrium* (Definition 3, L107–L111). This equilibrium enforces strategic separation to prevent collusion and aligns verifier/generator preferences via Markovian updates (Eq. 5–6).\n\n- **BDG is formal, training-free, and model-agnostic.** It does not rely on specific prompting formats or auxiliary supervision, and can be composed with CoT/few-shot prompting orthogonally (see Table 3–4 and Appendix I). Our contribution goes beyond deliberation or contrastive-objective methods.\n\n- The **second and third papers** primarily *diagnose* LLM behaviors (e.g., inverse scaling, multi-turn sycophancy), but do not propose a decoding-time algorithm that enforces reliable output or prevents collusion. They offer no equilibrium or strategy formulation, nor convergence guarantees. Their contribution lies in behavioral analysis, not decoding design.\n\nWe will discuss these differences in detail in the Related Works section and will cite the provided references.\n\n**2. Clarification of baselines and experimental scope**\n\nWe define all baseline decoders in Appendix I (§I), including:  \n**G** (Generative ranking), **D** (Verifier-only ranking), **MI** (Mutual Information reweighting), **SCD** (Self-Contrastive Decoding), **ECG** (Equilibrium Consensus Game), and our proposed **BDG**.\n\nAll methods operate over the same candidate set, models, and decoding budget, ensuring fair comparison. BDG shows consistent improvements across MMLU, ARC-E/C, RACE-H, GSM8K, and TruthfulQA, often allowing LLaMA-13B to match or exceed much larger models like PaLM-540B (see Table 1, Table 2, §3.2).\n\n**3. On model choices and extensibility**\n\nBDG is explicitly model-agnostic. It only requires access to model likelihoods for reweighting and converges within a small number of steps (typically <100; see Fig. 3a). While we use LLaMA-7B/13B and DeepSeek-7B for comparison with baselines like ECG [31], our framework applies to newer models as well.\n\nIn fact, BDG improves DeepSeek-7B on MMLU from 44.07% to 49.52%, outperforming both its base and ECG variants (46.20%) — see §3.2 (L244–L246). Additional experiments on newer models will be included in the final version.\n\n**4. Our Core Contributions (beyond prior art)**\n\n- **Theoretical novelty**: We are the first to model LLM decoding as a *Bayesian signaling game* with a provably convergent equilibrium (Theorem 2, L128–L129), addressing multi-agent collusion and self-confirmation failures (Appendix C, Fig. 3).\n\n- **Decoding-time optimization**: Our Markovian update schedule defines no-regret learning between generator and verifier, ensuring preference alignment and stability (Fig. 4a).\n\n- **Training-free and plug-and-play**: BDG improves decoding without additional data, training, or architectural modification. Its consensus enforcement is robust to prompt format, model size, and domain (Tables 1–5).\n\n- **Separation over heuristics**: Unlike prior sampling or self-assessment heuristics, BDG’s correctness–consistency tradeoff is controlled analytically via *σ-separation* and preference alignment metrics (Definition 3; Proposition 1).\n\n---\n\nWe respectfully submit that the cited works operate under different assumptions, problem formulations, and technical mechanisms. BDG is a decoding-time game-theoretic framework with theoretical guarantees and consistent empirical benefits. We will clarify these distinctions more explicitly in the Related Work section and appreciate the reviewer’s feedback in helping us sharpen the description of our contributions.\n\n---\n\n**References**\n\n[1] Wang et al., \"Self-Consistency Improves Chain of Thought Reasoning in Language Models\", ICLR 2023. arXiv: 2203.11171  \n[2] Laben etal., \"Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment\", arXiv: 2311.08596  \n[3] Perez et al.\"Discovering Language Model Behaviors with Model-Written Evaluations\n\", ACL Findings 2023. arXiv: 2212.09251"}, {"review_id": "wN2QlKu3Gy", "reviewer": "Reviewer_Yhw4", "summary": "This paper proposes a game-theoretic decoding framework, called the Bayesian Decoding Game (BDG), to improve the self-verification capabilities of large language models (LLMs). The authors model decoding as a multi-stage interaction between a generator and a verifier, formalized as a signaling game, and introduce the concept of a $\\sigma$-separated equilibrium to prevent collusion and enforce both correctness and consistency. Unlike existing methods such as single-agent self-reflection and multi-agent debate, which either suffer from confirmation bias or collusive agreement, BDG dynamically refines decoding strategies through no-regret learning and Markovian updates, aligning generation and verification outputs. Experiments across multiple benchmarks (e.g., MMLU, ARC, GSM8K) show that BDG consistently improves both factual accuracy and consistency, even enabling smaller models like LLaMA-13B to outperform much larger models (e.g., PaLM-540B), without requiring additional training. The method also shows robustness across tasks involving reasoning, ethics, and medical question answering.", "questions": "1. Is the verifier and generator the same LLM? Have you tried using different LLMs for the verifier and generator?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. The paper introduces a new decoding strategy by framing LLM output generation as a multi-stage Bayesian signaling game between a generator and a verifier.\n2. BDG operates at inference time and does not require fine-tuning or retraining. This makes it highly practical and easy to integrate into existing LLM pipelines.\n3. The experimental results show that BDG yields consistent improvements across a variety of domains (e.g., ARC, MMLU, PubMedQA, Ethics) and question types (multiple choice and generative)\n\n**Weaknesses**:\n1. Although the method is theoretically sound, the practical convergence of the Markovian updates involves multiple hyperparameters (learning rate, stiffness, $\\sigma$-thresholds), and the paper does not deeply explore their sensitivity or robustness.\n2. The proposed method requires access to token probability, but the most performant LLMs are typically black-box models often with limited or no access to logits. It can not generalize to these black-box models. Therefore, some work has explored prompting language models to express uncertainty or confidence score in human language [2,3]. I think this paper should discuss and compare with this kind of method.\n3. For the open-ended generative tasks, the authors should compare their method with various Best-of-N selection methods (such as self-consistency [1], universal self-consistency [2], soft self-consistency [3]) to show the effectiveness and efficiency of their method. \n4. The method should be evaluated on more recent LLMs such as llama3 and qwen2.5 to show the generalization of their method.\n\n[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al., ICLR 2023)\n[2] Universal Self-Consistency for Large Language Models (Chen et al., ArXiv 2023)\n[3] Soft Self-Consistency Improves Language Models Agents (Wang et al., ACL 2024)", "was_revised": true, "final_justification": "I hope the author can make the changes as promised in further version. I keep my positive score.", "rebuttal": "**We thank Reviewer Yhw4 for their recognition of the theoretical soundness of our BDG formulation, its practicality as a plug-in decoding mechanism, and its consistent improvements across factual and reasoning tasks. Below we address each point raised.**\n\n---\n\n**1. Hyperparameter robustness and rationale for fixed settings**\n\nWe agree that further discussion of hyperparameter sensitivity would be valuable. Our primary focus in this work is the *framework-level design* of BDG as a decoding-time strategic mechanism that generalizes across models and domains. From this perspective, we intentionally avoid extensive task-specific tuning: excessive optimization of hyperparameters such as η, λ, or σ may improve local performance but reduce the generality and interpretability of the framework.\n\nIn our experiments, we use a single fixed setting across all benchmarks and models (see Appendix I), and observe consistent convergence within 50–100 iterations (Fig. 3a). As the Markovian updates (Eq. 5–6) are entropy-regularized, convergence speed may vary with η, but the equilibrium ranking and preference alignment (Definition 3) remain stable. To support this point quantitatively, we will include an ablation over η and λ in the final appendix.\n\n**2. Applicability to black-box models and relation to prompt-based methods**\n\nWe acknowledge that BDG requires access to token probabilities, which limits direct applicability to fully black-box APIs. However, we note that many black-box systems today are evaluated primarily via prompt engineering and meta-prompting techniques, which is an area orthogonal to our focus. BDG operates at the decoding layer and introduces an explicit multi-agent dynamic via policy updates; it does not rely on prompt design, instruction tuning, or post-hoc voting.\n\nWhile our goal is not to replace prompt-space heuristics, BDG can be adapted to use LLM-expressed confidence proxies or relative ranking signals (e.g., [2,3]) as substitutes for logits. We will discuss this adaptation path more explicitly in the limitations section.\n\nSpecifically, in the updated Limitations section and Appendix we will include the following:\n\n- A clarification that BDG’s core mechanism only requires *relative preference rankings* or *confidence differentials* over candidates from both agents (generator and verifier).\n- A note that many LLM APIs now allow structured self-evaluation or scoring (e.g., “On a scale from 1 to 5, how confident are you in this answer?”). These can be used to construct soft $\\sigma_i$-separation and entropy-regularized utility functions analogous to our token-based setting.\n- We will describe how BDG's Markovian belief updates (Eq. 5–6) can operate over *normalized confidence scores* instead of log-probabilities, which does not change the update logic.\n- We will include a pseudocode variant in Appendix J showing how the BDG belief update loop generalizes to black-box setups with prompt-accessible confidence outputs.\n\nThese changes demonstrate that BDG can be *trivially extended* to black-box LLMs using elicited confidence without modifying the core dynamics of our framework.\n\n**3. Comparison with Best-of-N and self-consistency variants**\n\nWe do not benchmark against Best-of-N variants (e.g., Self-Consistency [1], Universal Self-Consistency [2], Soft Self-Consistency [3]) because they follow a different design paradigm. These methods require large-scale sampling and rely on majority voting or heuristic filtering. In contrast:\n\n- BDG directly modifies the decoding procedure via a two-agent signaling game with provable convergence (Theorem 2, L128–L129), independent of CoT usage. Best-of-N variants cannot provide this level of independence and verifiability. \n- Our framework deterministically reweights a fixed candidate set through strategic interaction, achieving both consistency and correctness via equilibrium dynamics. We do not rely on post hoc aggregation as other methods do.\n\nWe acknowledge these methods in the Related Work section and will clarify how BDG can complement them, e.g., by using BDG to select among CoT samples or self-consistent rationales (Table 4, §3.3).\n\n**4. Evaluation on newer models**\n\nOur main experiments use open-weight models (LLaMA-7B/13B and DeepSeek-7B) to allow fair comparison against baseline decoders such as ECG [31], MI, and SCD. However, we emphasize that BDG is *model-agnostic*: it applies to any autoregressive LLM with scoring access and does not require retraining or internal architectural modification.\n\nWe demonstrate this with DeepSeek-7B on MMLU, where BDG improves performance from 44.07% (base) to 49.52%, surpassing ECG at 46.20% (§3.2, L244–L246). We are currently seeing similar results for BDG with LLaMA-3 and Qwen2.5 models, and these results will be included in the final appendix.\n\n**5. Mixing of Heterogeneous Models**\n\nWe did not mix different models by default, because doing so introduces asymmetry that dilutes the game‑theoretic meaning of the interaction, where differences in model capability and calibration can dominate outcomes rather than the strategic dynamics we study, thus changing the problem class to a more LLM-as-a-judge/student-teacher like topic. Thus, exploring controlled model asymmetry (e.g., pairing a specialized verifier with a domain‑specific generator under explicit calibration checks) is a valuable direction to investigate it as future work, and we thank Reviewer Yhw4 for the thoughtful suggestion.\n\n---\n\nWe appreciate the reviewer’s constructive feedback. BDG provides a decoding-time, model-agnostic mechanism that formalizes generator-verifier dynamics via Bayesian signaling games, offering provable convergence to a separating equilibrium. We will incorporate the suggested clarifications on robustness, black-box adaptation, and comparison with sampling-based approaches in the final version.\n\n---\n\n**References**  \n[1] Wang et al., \"Self-Consistency Improves Chain of Thought Reasoning in Language Models\", ICLR 2023. arXiv: 2203.11171\n\n[2] Chen et al., \"Universal Self-Consistency for Large Language Models\", arXiv 2023. arXiv: 2311.17311\n\n[3] Wang et al., \"Soft Self-Consistency Improves Language Model Agents\", ACL 2024. arXiv: 2402.13212"}, {"review_id": "Xn5gzkQHU9", "reviewer": "Reviewer_RFvm", "summary": "This work addresses the problem of logical inconsistency across multi-turn inference processes in LLMs. Since existing approaches like RLHF have limitations, the authors propose BDG, a novel game-theoretic consensus mechanism. The method models the decoding process as a multistage Bayesian Decoding Game with two LLMs: a generator and a verifier. The generator ranks candidate answers using strategy probabilities, while the verifier assesses each candidate with correctness judgment probabilities. The goal is to achieve reliability, defined as simultaneous consistency and correctness, by aligning these two rankings. The generator and verifier engage in strategic interactions, observing each other's strategies and updating them through Markovian strategy updates until they converge to the same ranking. However, multiple Nash equilibria exist, creating risks of collusive reinforcement where agents converge on shared errors. To prevent this collusion, BDG introduces the σi-separated equilibrium condition, which enforces strategic separation by maintaining a minimum distance between correct and incorrect candidate judgment probabilities. Through no-regret optimization with Markovian updates, the game is mathematically guaranteed to converge to an optimal σi-separated equilibrium (Theorm 2), ensuring both consistency and correctness while preventing collusive behaviors.", "questions": "- Experiments use LLaMA, which seems outdated. Is there a reason for not using LLaMA-2 or 3?\n\n> (L198) For open-ended generative tasks, we construct Y by sampling candidates from the LLM’s distribution P_LLM(y | q, correct) using nucleus [30] and top-k [22] sampling methods\n- Is y here a single token or can it be a token sequence?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- The paper is relatively well-structured\n- Performance improvements are substantial\n- Multiple iterations are required to decode a single answer. For example, Figure 3 shows BDG requires around 100 iterations for generator and verifier rankings to achieve clear separation and alignment. A comparison of inference counts with standard decoding strategies would strengthen the argument\n- Nearly all experimental tasks are limited to multiple-choice problems. GSM8K outputs numerical values with exact matching, lacking the feel of open-ended generation. Evaluation on tasks like code generation, machine translation, or summarization appears necessary", "was_revised": true, "final_justification": "Most of the concerns have been addressed in the rebuttal. Regarding the concern about outdated models, since the current evaluation is limited to relatively old models, I maintain my original score rather than raise it.", "rebuttal": "**We thank Reviewer RFvm for recognizing the clear structure of our work and the substantial improvements of BDG across tasks. Below, we respond to the specific concerns regarding inference cost and the range of evaluated tasks.**\n\n---\n\n**1. On iteration count and inference efficiency**\n\nWe agree that decoding efficiency is important for practical deployment and AI trustworthiness. While BDG introduces an iterative refinement process between generator and verifier, our framework is designed for rapid convergence:\n\n- In Figure 3a, convergence is typically achieved in fewer than 100 iterations, and often within 50 steps. Unlike methods that require sampling hundreds of completions (e.g., self-consistency), BDG deterministically reweights a fixed candidate set without additional calls to the language model after candidate generation (§3.2, L239–L246).\n- Importantly, the generator and verifier can be run in parallel during each update step, as their beliefs are updated synchronously (Eq. 4–6). Thus, wall-clock latency remains modest.\n- The policy entropy trajectory (Fig. 4a) shows that BDG stabilizes significantly faster than prior multi-agent methods like ECG, which exhibit oscillatory behavior and longer plateaus before convergence (Fig. 3b). BDG’s convergence criteria (Definition 3) also allow early stopping once preference alignment and separation are achieved.\n\nWe will clarify these design considerations in the final appendix.\n\n**2. On limited coverage of open-ended generation tasks**\n\nOur focus in this work is on structured and semi-structured QA tasks (e.g., MMLU, ARC, RACE, PubMedQA) that allow controlled evaluation of correctness and consistency. We agree that evaluating BDG in broader generation settings such as summarization or code synthesis are promising directions for future applications of our work. That said:\n\n- We already include arithmetic free-form QA in GSM8K (Table 4), where BDG improves over both ECG and SCD despite being applied in a zero-shot setup (§3.3, L251–L256). GSM8K outputs are not multiple choice, and BDG operates over sampled spans from the LLM distribution P(y|x).\n- Our method supports open-ended candidate generation via nucleus or top-k sampling (L198), as noted in §3.1. These candidate sets are then re-ranked via game-theoretic interaction, and are not constrained to classification-style outputs.\n- Our Proposition 1 (L142–L153) extends to continuous confidence estimates, making BDG compatible with scoring in generative domains, provided candidate answer sets are well-defined (e.g., sampled summaries, code snippets, or rationales).\n\nWe will clarify this generality in §3.1 and in the final appendix. We emphasize that BDG does not assume a task format: **it operates on any candidate set** where correctness likelihood can be estimated by a verifier.\n\n**3. Model recency**\n\nWe will include results on newer models (e.g., LLaMA-3 and Qwen2.5) in the final version. The current evaluations (LLaMA-13B, DeepSeek-7B) were selected for reproducibility and baseline availability. BDG remains model-agnostic and does not depend on architecture-specific features or training.\n\n**4. Clarification on L198 — is *y* a token or sequence?**\n\n*y* refers to a candidate **sequence**, i.e., a full output sampled from the LLM. This is clarified in L198: “we construct *Y* by sampling candidates from the LLM’s distribution P<sub>LLM</sub>(y | x, correct).” These y ∈ Y are complete answers (e.g., strings), not individual tokens.\n\n\n\n---\n\nWe appreciate the reviewer’s thoughtful concerns and will strengthen our discussion of inference cost, open-ended generalization, and future directions. BDG is designed to balance correctness and consistency via strategic interaction at inference time, without introducing high sampling cost or retraining. We believe this makes BDG a practical and extensible framework for both structured and open-ended LLM decoding."}, {"review_id": "0vHMqBzmWK", "reviewer": "Reviewer_sYwu", "summary": "The manuscript titled \"From Self-Check to Consensus: Bayesian Strategic Decoding in Large Language Models\" addresses the prevalent issue of logical inconsistency in Large Language Models (LLMs) during multi-turn inference processes. The authors identify that existing approaches, such as single-agent reflection and multi-agent debate, often compromise accuracy in favor of consistency. To overcome these limitations, the paper introduces a novel game-theoretic consensus mechanism named the Bayesian Decoding Game (BDG). This framework models the decoding process as a multistage Bayesian Decoding Game, facilitating strategic interactions between a Generator and a Verifier to achieve consensus on reliable outputs. The proposed method is model-agnostic, demonstrating that smaller models can outperform significantly larger counterparts without additional training. Experimental results across various benchmarks, including MMLU, ARC, and RACE, underscore the effectiveness of BDG in enhancing both consistency and correctness of LLM outputs. Additionally, a user study highlights BDG's potential in bridging the expertise gap by improving performance for both experts and non-experts.", "questions": "1. How sensitive is BDG to the choice of hyperparameters such as the learning rates (η) and stiffness parameters (γ)? Have the authors conducted any ablation studies to understand the impact of these parameters on performance?\n\n2. The paper mentions the possibility of incorporating multi-metrics and multiple agents to achieve game-based deliberation. Have the authors explored any preliminary results in this direction, and what challenges do they anticipate?\n\n3. Can BDG be efficiently integrated into real-time applications where rapid decoding is essential? If so, what optimizations are necessary to achieve this?", "limitations": "There is no potential negative societal impact.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The introduction of the Bayesian Decoding Game (BDG) as a game-theoretic framework is a novel contribution that effectively addresses the limitations of existing self-verification methods in LLMs. By framing the decoding process as a signaling game, the authors provide a structured mechanism for LLMs to self-check and reach consensus without relying heavily on human feedback.\n\n2. The paper offers a comprehensive theoretical foundation for BDG, including the formulation of the decoding process, the concept of Separating Equilibrium (SE), and the implementation of no-regret optimization through Markovian strategy updates. Theorems and propositions are clearly stated, demonstrating the framework's ability to prevent collusion and ensure reliable behavior.\n\n3. Extensive experiments across multiple benchmarks (e.g., MMLU, ARC-E, ARC-C, RACE-H) convincingly demonstrate BDG's superiority over baseline models and other game-theoretic methods like ECG. Notably, the ability of smaller models (e.g., LLaMA13B) to outperform larger models (e.g., PaLM540B) without additional training is a significant achievement.\n\n\nWeaknesses\n\n1. While the theoretical aspects of BDG are well-elaborated, certain sections of the manuscript are dense and may be challenging for readers unfamiliar with game theory or advanced machine learning concepts. Enhanced explanations or illustrative examples could improve accessibility.\n\n2. While BDG demonstrates improved convergence rates compared to ECG, the paper does not extensively discuss the computational overhead or scalability of BDG in extremely large-scale deployments. Insights into resource requirements and potential bottlenecks would be beneficial.\n\n3. The empirical evaluation primarily focuses on comparisons with ECG and baseline generative rankings. Including a wider range of existing self-verification and consistency-enhancing methods could further strengthen the evidence for BDG's superiority.", "was_revised": true, "final_justification": "The authors responded to my concerns about computational overhead and broader comparison. Although they did not fully address my concern, I still hold a positive attitude toward this paper. So I decided to maintain my original positive rating.", "rebuttal": "**We thank Reviewer sYwu for their positive evaluation of BDG’s theoretical contributions, its equilibrium structure, and the consistent empirical gains observed across strong baselines. Below we clarify several points raised, with emphasis on our design intentions and implementation choices.**\n\n---\n\n**1. On theoretical density and accessibility**\n\nWe appreciate the concern regarding theoretical accessibility. While §2.2–§2.3 introduces the core BDG framework formally, we emphasize that the manuscript integrates illustrative guidance throughout:\n\n- We provide intuition-first examples in §2.1 and Figure 1, as well as visual teasers in Figures 2a–2b to concretely motivate the generator–verifier game dynamics before formalizing the model.\n- Our narrative uses example-driven hooks across sections (e.g., policy ordering examples, belief mismatches) that directly correspond to the components of the signaling game.\n- Appendix I includes annotated pseudocode, and internal equation cross-references are used throughout to improve navigability between definition and application.\n\nOther reviewers (e.g., HSTc, SV6t) found the theoretical formulation rigorous and internally consistent. Our planned edits will preserve this formal integrity while expanding accessibility, for example, by relocating detailed proofs to the appendix, adding a notation table, and expanding comparative commentary on related frameworks (e.g., ECG, USC). These changes aim not to simplify the theory, but to better structure the entry points for different reader backgrounds.\n\n**2. On computational scalability and overhead**\n\nBDG was designed to operate with minimal decoding overhead. Rather than introducing new sampling or training steps, the generator and verifier engage in lightweight belief updates over a fixed candidate set. As described in §3.1 and Appendix I:\n\n- Candidate generation occurs once per query; all subsequent steps involve only softmax-based strategy updates (Eq. 4–6), using internal reweighting rather than additional model calls.\n- Generator and verifier operate in parallel and update synchronously, allowing for batch-efficient and hardware-parallelizable implementation.\n- Convergence is typically reached within 50–100 steps (Fig. 3a), and policy entropy stabilizes early (Fig. 4a), reducing unnecessary computation.\n\nWe acknowledge that the original version focused more on algorithmic detail than runtime profiling. For completeness, we will include decoding-call counts in the final appendix, and contrast BDG with sampling-heavy baselines like Self-Consistency. However, we stress that BDG’s inference-time design is already efficient by construction.\n\n**3. On broader comparison to self-verification and consistency methods**\n\nOur baselines include a representative spectrum of decoding-time techniques:\n\n- **Verifier-based ranking (D)**, **Mutual Information reweighting (MI)**, and **Self-Contrastive Decoding (SCD)**. These are all evaluated under the same budget (§3.1, Appendix I).\n- **ECG** [31], the most relevant game-theoretic baseline, is thoroughly compared in terms of convergence behavior (Fig. 3), entropy stability (Fig. 4), and disagreement rates (Table 2).\n- BDG consistently outperforms all baselines, even enabling smaller models like LLaMA13B to rival or exceed the performance of PaLM540B (Table 1).\n\nOur focus is on decoding-time strategies with no retraining and no auxiliary supervision. While training-time methods (e.g., deliberation fine-tuning, feedback alignment) exist, they lie outside the inference-time scope of BDG and would introduce orthogonal cost tradeoffs. That said, we will expand §4.1 to include a broader landscape comparison and discuss how BDG complements other approaches and vice versa.\n\n---\n\nWe appreciate the reviewer’s constructive suggestions and will revise the paper accordingly to improve clarity, expand evaluation context, and quantify inference cost. BDG provides a theoretically grounded, practically viable decoding strategy that addresses both correctness and consistency, bridging a critical gap in LLM self-verification."}]

Author rebuttal:

Meta-review: The paper introduced a new decoding method, which they refer to as a multistage Bayesian Decoding Game. The aim of the game is to decrease logical inconsistency and increase the change of arriving at the correct answer when using multi-agent inference to solve some task. Their method involves LLM-based generators and verifiers that refine their strategies over multiple iterations of the game.

The proposed method merges desires for consistency and correctness in the answers from AI systems. Empirically, it seems to work very well on the tasks that were evaluated. All reviewers expressed support for the value of the proposed method, especially since it is inference-time only and doesn't require any additional model training.

Reviewers did, however, note several weaknesses. Multiple reviewers expressed concerns about the baseline methods. Existing baselines were insufficiently documented in the main paper body. Also, more baselines (for example, a best-of-n) should have been considered. Evaluation was only conducted on tasks with simple numerical or classification-style answers, so we don't know what performance will be like on harder tasks such as summarization or code generation.

Overall, the paper offers a reasonably-sized contribution which I expect will be of interest to NeurIPS community. I strongly advise the authors to make the various textual changes recommended by reviewers (expanded references to related work, clearer description of baselines in main paper, more info on computational cost of inference of proposed method vs. other methods, improved accessibility for readers unfamiliar with game theory).
Decision: Accept (poster)