Kati8/ICLR_2023_2025 huggingface dataset:
authors: Athul Paul Jacob
authors: Yikang Shen
authors: Gabriele Farina
authors: Jacob Andreas
reviews: 
Comments: The paper introduces a new decoding algorithm called EQUILIBRIUM-RANKING, which addresses the challenge of reconciling different scoring procedures in language models (LMs) for question-answering and text-generation tasks. EQUILIBRIUM-RANKING is a game-theoretic approach, where a generator aims to communicate a correctness parameter to a discriminator through natural language sentences. By formulating language model decoding as an imperfect information sequential signaling game, the authors develop computational procedures to find approximate equilibria of the game. The experiment uses a set of QA tasks to evaluate EQUILIBRIUM-RANKING, including MMLU, ARC, RACE, HHH, and TruthfulQA, and a math benchmark GSM8K. Across these tasks, EQUILIBRIUM-RANKING consistently improves performance compared to existing LM decoding procedures.
Strengths: 1. The paper demonstrates originality by introducing a novel approach to decoding language models through the consensus game framework. The idea of casting language model decoding as a game-theoretic problem and seeking approximate equilibria is innovative and provides a fresh perspective on addressing the challenge of reconciling different scoring procedures. 

2. The authors develop robust computational techniques for finding approximate equilibria in the consensus game, ensuring reliable and practical implementation of the EQUILIBRIUM-RANKING algorithm. The paper also demonstrates the quality of the research through its extensive evaluation across a diverse range of tasks, showcasing consistent performance improvements over existing decoding procedures.

3. The authors provide a concise and coherent description of the EQUILIBRIUM-RANKING algorithm, making it accessible to readers. The clarity of the writing facilitates the comprehension and replication of the proposed approach. The paper also presents the results and performance improvements in a straightforward manner, enabling readers to grasp the significance of the findings.
Weaknesses: 1. To improve the clarity of the proposed method, it would be advantageous to include a running example that demonstrates the step-by-step process.

2. To enhance the quality of the paper, it is recommended to address minor typos and errors. For instance, in line 6 of the abstract, "a new, a training-free" should be corrected to "a new, training-free." Additionally, in section 2, line 6, "we may them" appears to be a typographical error and should be revised for clarity.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This work proposes a new answer reranking method for LLMs. The main objective is to reconcile the inconsistency between the model's response in the generator setting and its response in the discriminator setting. Specifically, this work designs a regularized imperfect-information sequential signaling game involving the generator and discriminator and finds the Nash equilibrium between them. To find the equilibrium, this work adopts the piKL algorithm proposed by Jacob et al., 2022. The overall method is a training-free method and can consistently outperform several baselines on a variety of QA tasks (also including open-ended and math reasoning tasks).
Strengths: 1. The inconsistent behavior of LLMs across different prompts is an annoying phenomenon. This paper proposes a principled method to reconcile such consistency via game theory.
2. The proposed method is evaluated on various tasks and consistently improves over popular baselines.
3. The proposed method is training-free, so it's applicable to even very large-scale models.
Weaknesses: 1. If we compare the proposed ER-based reranking method and previous baselines, there are actually two major differences. One is the process of finding the Nash equilibrium under the regularization part, the other is the combination of two normalized probabilities in Section 2.2. In the current evaluation, it is clear that the combination of these two can bring consistent improvement, but it's unclear what is the effect of each individual component. It would be helpful to have a baseline just combining the two normalized probabilities (e.g., via the MI approach).
2. This paper can benefit from showing some intrinsic analysis of the inconsistency of the generator and the discriminator. Right now, while the evaluation shows the effectiveness of the proposed approach, it is unclear how severe the problem is, and in how many cases finding the equilibrium can help.
3. I understand the piKL algorithm is from Jacobs et al., 2022, but I would still suggest the authors provide a slightly more detailed description in the appendix. The lack of related details makes it very hard to understand how important the hyper-parameters are, and if the method is sensitive to these values.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: In this paper, the authors present an interesting framework centered around a consensus game involving generators and discriminators within language models. They delve into the concept of equilibrium within this game and leverage the equilibrium states of the generator and discriminator to enhance output generation in language models. The efficacy of this approach is validated through rigorous experimentation across various question-answering (QA) tasks.
Strengths: 1. The game-theoretic formulation introduced for the interaction between generators and discriminators in language models is novel and interesting.
2. The experiments is comprehensive, covering an array of QA tasks, and the results presented are compelling.
3. The paper is generally well-written and easy to follow.
Weaknesses: 1. The paper could benefit from a more detailed explanation on the methodological. The process by which Equations (1) and (2) facilitate the attainment of the game's equilibrium is not sufficiently illuminated.
2. The reliance on the piKL algorithm for the learning dynamics of the consensus game may limit the methodological novelty of the paper.
3. The pursuit of a no-regret dynamic steers the system towards a coarse correlated equilibrium. The paper could be enriched by an exploration into the feasibility and potential advantages of converging to stronger equilibrium constructs, such as pure Nash or mixed Nash equilibria.
4. A notable performance drop is observed on the HHH dataset when transitioning from the LLaMA-7B to LLaMA-13B backbone model. An elaboration on this counterintuitive outcome would be beneficial.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good