Title: From Self-Check to Consensus: Bayesian Strategic Decoding in Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-agent System

Game Theory

Mechanism Design
Abstract: Large Language Models exhibit logical inconsistency across multi-turn inference processes, undermining correctness in complex inferential tasks. Challenges arise from ensuring that outputs align with both factual correctness and human intent. Approaches like single-agent reflection and multi-agent debate frequently prioritize consistency, but at the expense of accuracy. 
To address this problem, we propose a novel game-theoretic consensus mechanism that enables LLMs to self-check their outputs during the decoding stage of output generation. Our method models the decoding process as a multistage Bayesian Decoding Game, where strategic interactions dynamically converge to a consensus on the most reliable outputs without human feedback or additional training. Remarkably, our game design allows smaller models to outperform much larger models through game mechanisms (e.g., 78.1 LLaMA13B vs. 76.6 PaLM540B). As a model-agnostic method, our approach consistently improves even the latest models, enhancing DeepSeek-7B's performance on MMLU by 12.4%. Our framework effectively balances correctness and consistency, demonstrating that properly designed game-theoretic mechanisms can significantly enhance the self-verification capabilities of language models across various tasks and model architectures.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript titled \"From Self-Check to Consensus: Bayesian Strategic Decoding in Large Language Models\" addresses the prevalent issue of logical inconsistency in Large Language Models (LLMs) during multi-turn inference processes. The authors identify that existing approaches, such as single-agent reflection and multi-agent debate, often compromise accuracy in favor of consistency. To overcome these limitations, the paper introduces a novel game-theoretic consensus mechanism named the Bayesian Decoding Game (BDG). This framework models the decoding process as a multistage Bayesian Decoding Game, facilitating strategic interactions between a Generator and a Verifier to achieve consensus on reliable outputs. The proposed method is model-agnostic, demonstrating that smaller models can outperform significantly larger counterparts without additional training. Experimental results across various benchmarks, including MMLU, ARC, and RACE, underscore the effectiveness of BDG in enhancing both consistency and correctness of LLM outputs. Additionally, a user study highlights BDG's potential in bridging the expertise gap by improving performance for both experts and non-experts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The introduction of the Bayesian Decoding Game (BDG) as a game-theoretic framework is a novel contribution that effectively addresses the limitations of existing self-verification methods in LLMs. By framing the decoding process as a signaling game, the authors provide a structured mechanism for LLMs to self-check and reach consensus without relying heavily on human feedback.\n\n2. The paper offers a comprehensive theoretical foundation for BDG, including the formulation of the decoding process, the concept of Separating Equilibrium (SE), and the implementation of no-regret optimization through Markovian strategy updates. Theorems and propositions are clearly stated, demonstrating the framework's ability to prevent collusion and ensure reliable behavior.\n\n3. Extensive experiments across multiple benchmarks (e.g., MMLU, ARC-E, ARC-C, RACE-H) convincingly demonstrate BDG's superiority over baseline models and other game-theoretic methods like ECG. Notably, the ability of smaller models (e.g., LLaMA13B) to outperform larger models (e.g., PaLM540B) without additional training is a significant achievement.\n\n\nWeaknesses\n\n1. While the theoretical aspects of BDG are well-elaborated, certain sections of the manuscript are dense and may be challenging for readers unfamiliar with game theory or advanced machine learning concepts. Enhanced explanations or illustrative examples could improve accessibility.\n\n2. While BDG demonstrates improved convergence rates compared to ECG, the paper does not extensively discuss the computational overhead or scalability of BDG in extremely large-scale deployments. Insights into resource requirements and potential bottlenecks would be beneficial.\n\n3. The empirical evaluation primarily focuses on comparisons with ECG and baseline generative rankings. Including a wider range of existing self-verification and consistency-enhancing methods could further strengthen the evidence for BDG's superiority.\n\nQUESTIONS:\n1. How sensitive is BDG to the choice of hyperparameters such as the learning rates (η) and stiffness parameters (γ)? Have the authors conducted any ablation studies to understand the impact of these parameters on performance?\n\n2. The paper mentions the possibility of incorporating multi-metrics and multiple agents to achieve game-based deliberation. Have the authors explored any preliminary results in this direction, and what challenges do they anticipate?\n\n3. Can BDG be efficiently integrated into real-time applications where rapid decoding is essential? If so, what optimizations are necessary to achieve this?\n\nLIMITATIONS:\nThere is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of logical inconsistency across multi-turn inference processes in LLMs. Since existing approaches like RLHF have limitations, the authors propose BDG, a novel game-theoretic consensus mechanism. The method models the decoding process as a multistage Bayesian Decoding Game with two LLMs: a generator and a verifier. The generator ranks candidate answers using strategy probabilities, while the verifier assesses each candidate with correctness judgment probabilities. The goal is to achieve reliability, defined as simultaneous consistency and correctness, by aligning these two rankings. The generator and verifier engage in strategic interactions, observing each other's strategies and updating them through Markovian strategy updates until they converge to the same ranking. However, multiple Nash equilibria exist, creating risks of collusive reinforcement where agents converge on shared errors. To prevent this collusion, BDG introduces the σi-separated equilibrium condition, which enforces strategic separation by maintaining a minimum distance between correct and incorrect candidate judgment probabilities. Through no-regret optimization with Markovian updates, the game is mathematically guaranteed to converge to an optimal σi-separated equilibrium (Theorm 2), ensuring both consistency and correctness while preventing collusive behaviors.\n\nSTRENGTHS AND WEAKNESSES:\n- The paper is relatively well-structured\n- Performance improvements are substantial\n- Multiple iterations are required to decode a single answer. For example, Figure 3 shows BDG requires around 100 iterations for generator and verifier rankings to achieve clear separation and alignment. A comparison of inference counts with standard decoding strategies would strengthen the argument\n- Nearly all experimental tasks are limited to multiple-choice problems. GSM8K outputs numerical values with exact matching, lacking the feel of open-ended generation. Evaluation on tasks like code generation, machine translation, or summarization appears necessary\n\nQUESTIONS:\n- Experiments use LLaMA, which seems outdated. Is there a reason for not using LLaMA-2 or 3?\n\n> (L198) For open-ended generative tasks, we construct Y by sampling candidates from the LLM’s distribution P_LLM(y | q, correct) using nucleus [30] and top-k [22] sampling methods\n- Is y here a single token or can it be a token sequence?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a game-theoretic decoding framework, called the Bayesian Decoding Game (BDG), to improve the self-verification capabilities of large language models (LLMs). The authors model decoding as a multi-stage interaction between a generator and a verifier, formalized as a signaling game, and introduce the concept of a $\\sigma$-separated equilibrium to prevent collusion and enforce both correctness and consistency. Unlike existing methods such as single-agent self-reflection and multi-agent debate, which either suffer from confirmation bias or collusive agreement, BDG dynamically refines decoding strategies through no-regret learning and Markovian updates, aligning generation and verification outputs. Experiments across multiple benchmarks (e.g., MMLU, ARC, GSM8K) show that BDG consistently improves both factual accuracy and consistency, even enabling smaller models like LLaMA-13B to outperform much larger models (e.g., PaLM-540B), without requiring additional training. The method also shows robustness across tasks involving reasoning, ethics, and medical question answering.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. The paper introduces a new decoding strategy by framing LLM output generation as a multi-stage Bayesian signaling game between a generator and a verifier.\n2. BDG operates at inference time and does not require fine-tuning or retraining. This makes it highly practical and easy to integrate into existing LLM pipelines.\n3. The experimental results show that BDG yields consistent improvements across a variety of domains (e.g., ARC, MMLU, PubMedQA, Ethics) and question types (multiple choice and generative)\n\n**Weaknesses**:\n1. Although the method is theoretically sound, the practical convergence of the Markovian updates involves multiple hyperparameters (learning rate, stiffness, $\\sigma$-thresholds), and the paper does not deeply explore their sensitivity or robustness.\n2. The proposed method requires access to token probability, but the most performant LLMs are typically black-box models often with limited or no access to logits. It can not generalize to these black-box models. Therefore, some work has explored prompting language models to express uncertainty or confidence score in human language [2,3]. I think this paper should discuss and compare with this kind of method.\n3. For the open-ended generative tasks, the authors should compare their method with various Best-of-N selection methods (such as self-consistency [1], universal self-consistency [2], soft self-consistency [3]) to show the effectiveness and efficiency of their method. \n4. The method should be evaluated on more recent LLMs such as llama3 and qwen2.5 to show the generalization of their method.\n\n[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al., ICLR 2023)\n[2] Universal Self-Consistency for Large Language Models (Chen et al., ArXiv 2023)\n[3] Soft Self-Consistency Improves Language Models Agents (Wang et al., ACL 2024)\n\nQUESTIONS:\n1. Is the verifier and generator the same LLM? Have you tried using different LLMs for the verifier and generator?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the issue of consistency or flip-flopping and correctness when large language models reason. The paper outlines the trade-off between inconsistency in single agent setting as well as a Nash equilibrium collapse in multi-agent setting where models converge to the incorrect answer. Therefore, in order to build consensus during inference, this paper proposes formulating this as a Bayesian Decoding Game and proposes an optimization algorithm to achieve this. The paper tests this hypothesis on a variety of natural language tasks and (older) language models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Addresses an important connection between consistency and correctness as two desirable properties of good reasoning systems.\n- The decoding method is fairly novel and outperforms the baselines\n\n**Weaknesses**\n- Missing citation to related work: Several works have addressed the issue of inconsistency in language models which the paper does not address or cite: \n    - https://arxiv.org/abs/2203.11171 (this paper also uses consensus to improve reasoning performance, so why not compare against it?)\n    - https://arxiv.org/abs/2311.08596\n    - https://arxiv.org/abs/2212.09251\n- Missing explanation of baselines: The experimental section of the paper does not outline the choice of baseline decoding methods. What do MI, D, SCD methods stand for? Why not use few-shot evaluation for reasoning tasks like GSM-8K where the reported numbers are exceptionally low and it is well-known that chain-of-thought sampling yields high improvement boosts. Why only include zero-shot and few-shot evaluation in Table 3 and why not report other decoding baselines apart from BDG.\n- Missing justification for choice and models. Clearly the models used in this paper are a few generations behind and obsolete, making some of the results not highly convincing. The paper fails to address this and show gains from their decoding method still persist with recent generation of models and with chain of thought sampling.\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-03"}