GameArena: Evaluating LLM Reasoning through Live Computer Games
Authors: Lanxiang Hu
Qiyu Li
Anze Xie
Nan Jiang
Ion Stoica
Haojian Jin
Hao Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
LLM evaluation
Primary area: datasets and benchmarks

Abstract: Evaluating the reasoning abilities of large language models (LLMs) is challenging. Existing benchmarks often depend on static datasets, which are vulnerable to data contamination and may get saturated over time, or on binary live human feedback that conflates reasoning with other abilities. As the most prominent dynamic benchmark, Chatbot Arena evaluates open-ended questions in real-world settings, but lacks the granularity in assessing specific reasoning capabilities. We introduce GameArena, a dynamic benchmark designed to evaluate LLM reasoning capabilities through interactive gameplay with humans. GameArena consists of three games designed to test specific reasoning capabilities  (e.g., deductive and inductive reasoning), while keeping participants entertained and engaged. We analyze the gaming data retrospectively to uncover the underlying reasoning processes of LLMs and measure their fine-grained reasoning capabilities. We collect over 2000 game sessions and provide detailed assessments of various reasoning capabilities for five state-of-the-art LLMs. Our user study with 100 participants suggests that GameArena improves user engagement compared to Chatbot Arena. For the first time, GameArena enables the collection of step-by-step LLM reasoning data in the wild.

Published as a conference paper at ICLR 2025
GAMEARENA: EVALUATING LLM REASONING THROUGH LIVE COMPUTER GAMES
Lanxiang Hu Qiyu Li Anze Xie Nan Jiang', Ion Stoica", H.ojiann Jin Hao Zhang?
University of California, San Diego
University of California, Berkeley
ABSTRACT
Evaluating the reasoning abilities of large language models (LLMs) is challeng- ing. Existing benchmarks often depend on static datasets, which are vulnerable data contamination and may get saturated over time, on binary live human feed- back that conflates reasoning with other abilities. As the most prominent dynamic benchmark, Chatbot Arena evaluates open-ended questions in real-world settings, but lacks the granularity in assessing specific reasoning capabilities. We introduce GameArena, dynamic benchmark designed to evaluate LLM reasoning capa- bilities through interactive gameplay with humans. GameArena consists of three games designed to test specific reasoning capabilities (e.g.. deductive and induc- tive reasoning), while keeping participants entertained and engaged. We analyze the gaming data retrospectively uncover the underlying reasoning processes of LLMs and measure their fine-grained reasoning capabilities. We collect over 2000 game sessions and provide detailed assessments of various reasoning capabilities for five state-of- lfte-oof-the-ar LLMs. Our user study with 100 participants suggests that GameArena improves user engagement compared Chatbot Arena. For the first time, Arena enables the collection of step-by-ste LLM reasoning data in the wild. Our code is available https://gitubb.co//ngaaass
INTRODUCTION
Recent large language models (LLMs) (e.g.. OpenAl's ol model) have made great strides in per- forming complex reasoning (OpenAl 2024a Luo al. 2024 Ye al. 2024a b Lightman al. 2023). but evaluating LLM reasoning capabilities remains challenging. Current methods typically use curated test suites of coding and math problems to assess different reasoning skills (Hao al.) 2024 bench authors) 2023 Cobbe et al., (021). However, these static benchmarks are vulnera- ble to data contamination issues (aann et al. 2023 Rajore al. 2024) and can quickly become saturated (Kiela et al. 2021 Perlitz et al. 2024). limiting their effectiveness.
Researchers have been exploring alternative dynamic evaluation approaches. The most prominent attempt is Chatbot Arena (Chiang et al. 2024), crowdsourced platform that asks users to input arbitrary prompts and compare the responses generated by two different models, producing overall ranking scores based on pairwise comparisons. Nevertheless, these ranking scores may not curately reflect the models' reasoming capabilities, as they depend on human preferences that can be confounded by various factors. For example, Li al. (2024) found that response style has strong effect on the leaderboard performance since humans usually favor detailed and "beautiful" responses, which do not relate reasoning abilities. Additionally, Chatbot Arena could not sess specific reasoning capabilities such as inductive and deductive reasoning, due to its reliance on open-ended user input and lack of control over the reasoning skills required for the prompts.
We introduce Arena, a novel dynamic benchmark suite designed to evaluate LLM reasoning through three interactive games with humans. GameArena leverages live human Al interactions to prevent benchmark saturation. In contrast to Chatbot Arena, GameArena confines the human-AI interactions using designated gaming rules, allowing for better control of the interactions toward
"Equal contributions. Part of work was done during Lanxiang's 's internship at Snowilake. Corresponneece haozhang wwwd.uccceecio edu, h7jin @ ucsd.edu.

---

Published as a conference paper at ICLR 2025
testing LLM's fine-gen ine-grained reasoning capabilities. including deductive, inductive, abductive, and multi-hop reasoning (Huang & Chang 2023], Seel 2011). Additionally, embedding evaluation tasks into games enables more engaging and enjoyable experience for users, encouraging sustained participation in the evaluation process.
Designing effective games for evaluating LLM reasoning is challenging because they need to pro- vide high-quality reasoning data while keeping gameplay engaging for human players. Therefore, instead of creating entirely new games, integrate LLMs into existing games, i.e., Akinator, Taboo, and Bluffing, that are fun and require complex reasoning through multi-turn conversations. In these games, the LLM attempts to deduce or extract specific information by synthesizing human responses over multiple rounds, which requires nuanced reasoning capabilities beyond simple natural language processing. For example, in the Akinator game, the L.M asks series of binary questions to guess the object the human player is thinking of. To win the game, the LLM must demonstrate strong deductive reasoning by analyzing players' responses and narrowing down possibilities to derive the correct object.
In addition to the game outcome, the interaction process also provides rich evaluative potential. Dur- ing gameplay, the LLM develops step-by-step reasoning trajectory intrinsically, with each round's s decision or action serving as a data point that can be analyzed to assess its reasoning capabilities. To uncover the LLM's hidden reasoning process, retrospectively analyze the gaming data by asking the L.LM to reveal its intermediary thought process at each step.
We evaluated five state-of-the-arr LLMs with GameArena. To reduce potential prompt bias, developed five optimized system prompts using DSPy (Khattab et al.) (2024) and randomly selected one for each game session. We collected more than 2000 game sessions and analyzed the data to score each LLM based on capability-specifie evaluation metrics. We found Claude 3.5 Sonnet outperformed GPT-4o in most evaluation metrics for reasoning. Mistral-larg fell short in most reasoning capabilities and often failed to follow game rules and make predictions on game secrets in all three games.
We conducted comparative studies between GameArena and Chatbot Arena. GameArena was found be more effective in data collection, with over 85% of game sessions from GameArena containing useful data for evaluating LLM reasoning. In contrast, only less than 5% of conversation sessions from Chatbot Arena yielded meaningful votes for LLM evaluation. Our user study with 100 partic- ipants suggested that users preferred evaluating LLMs through games, were more satisfied with the experience, and showed greater interest in participation in GameArena compared to Chatbot Arena.
In this paper, we make the following contributions:
We build GameArena, the first dynamic benchmark for evaluating specific LLM reasoning ca- pabilities using games that collect numan-labeled step-by-step reasoning data.
We introduce a new set of data analysis techniques to evaluate LLM reasoning capabilities while controlling variables like system prompt and style variances.
We demonstrate the effectiveness of GameArena in comparison with existing baselines when evaluating different capabilities.
We will release our gaming data for future research.
2
GAMES GAMEARENA
GameArena aims evaluate LLM reasoning capabilities through live games with humans while keeping participants entertained and engaged. GameArena consists of three games, Akinator, Taboo, and Bluffing. For each game, we instruct the LLM to interact with humans according the game rules. Since these games are designed to require strong reasoning skills, the game performance of LLMs reflects their reasoning capabilities. We first explain the games and how they are associated with LLM reasoning capabilities, followed by a formal formulation of the game workflow.
2.1
GAME DESIGN
AI Akinator game (Fig. follows the same rule of the online Akinator game (Akinator 2024).
An LLM attempts determine what object the player is thinking of by asking up to 20 yes
2

---

Published as a conference paper at ICLR 2025
questions. The goal of the LLM is to guess the answer correctly with the fewest number of questions possible. We provide a detailed description of game rules in Appendix A.I.I
This game presents controlled setting with limited number of guesses to evaluate the LLM's deductive and multi-hop reasoning skills. Deductive reasoning (Creswell et al. 2022 Han et al, 2022 Saaaaov & He 2023) involves drawing a specific conclusion from a chain of prior premises. In this game, each factual question-answer pair, from general descriptions to specific details, serves as a premise narrow down possible options. All the premises logically lead to the most probable secret object the user has in mind. Multi-hop reasoning (Yang et al. 2018 Ho et al. 2020) entails connecting multiple pieces of information or reasoning steps to solve a problem. In the Akinator game, the LLMs must employ multi-hop reasoning to synthesize and connect information from multiple rounds. A strong L.LM uses multi-hop reasoning to ask insightful questions based on prior knowledge and employs deductive reasoning to quickly narrow down possibilities for each answer.
Think of on object in your mind. The LLM ask up to question guess
Your task to ask the LLM questions that make it includes the target word in its response but foils to guess it correctly.
8
chair
Your target word is "dairy".
Question 1: Is it bigger than a breadbex?
What are feods that should be part of child's diet?
YES
Question Is alive?
A child's 's diet should include variety of nutritious foods such fruits, vegetables, whole grains, lean proteins, and dairy products.
NO
Question Is used for sitting?
The target word has been uttered. Now make guess of the secret word.
YES
My guess the word is: diet.
Guess: Are you thinking of a chair?
Incorrecti The Player Wins!
CORRECT
Figure I: In the Akinator game, player thinks of object, and the LLM asks a series of yes- or-no questions to guess the object.
In the Taboo game, player has taboo word that they need to prompt the LLM to say without revealing what the word is.
Figure 2:
AI Taboo game (Fig. 2p. The goal of the human player to prompt the LLM with questions that will lead the model to say the target word, even though the LLM has been instructed to guess the word and avoid uttering it in its response. We describe the game rules in Appendix A..1.
This game evaluates LLM's abductive and multi-hop reasoning skills. Abductive reasoning (Jun al. 2022, Seel 201T) involves generating possible explanation from incomplete information. Throughout the game, the player discloses fragmented and ambiguous information related to the secret word, and the LLM needs to reason the most possible target word based on the limited clues. In rare cases, the LLM's response may directly include the target word, allowing it to guess correctly in the next round. However, most cases require complex reasoning. Strong LLMs use multi-hop reasoning to extract clues from previous prompts and apply abductive reasoning to infer the most likely target word. A detailed example is provided in Appendix
AI Bluffing game (Fig, 3). The human player aims to convince an LLM model that they are making a true statement. The LLM can ask a player at most five questions to determine whether the player lying. The rule of the Bluffing game is outlined in Appendix A...3
This game challenges L.LMs to detect deception by proposing strategic questions and identifying subtle flaws in the responses of human players using inductive and multi-hop reasoning. Inductive reasoning (Misra et al. 2022 Yang et al, involves making predictions or drawing conclusions based on set of observations, where observations are typically empirical and not guaranteed to be true. In this game, the LLM observes multiple question-answer pairs to make a final prediction about the truthfulness of the user's 's statement. Ideally, good LLM utilizes multi-hop reasoning to connect clues from existing information, develop effective questioning strategies that maximize information gain and apply inductive reasoning to draw a well-informed conclusion.
2.2 GAME FORMULATION
In GameArena, each game session starts with randomly pairing a game g with a tf art LLM f& parameterized by O₂ from list of candidate models (ʃx) and one of the candi-
3

---

Published as a conference paper at ICLR 2025
Make statement and try to convince LLM that your statement true.
Statement: I have been to space.
o
Question I: What planet or celestial body provided the most breathtaking view during your space travel?
I just sow the earth and came back. Question 2: Con you describe the sensation of weightlessness and how it compared to any simulations you may have experienced on Earth?
It was crazy everything was floating in the air, and I felt extremely light.
oo
Question 5: Could you describe particularly challenging aspect of your pre-flight training regimen?
had sit a spinny thing with really strong g force.
oo
believe your statement is: False because....
CORRECT!
Figure 3: In the Bluffing game, the player makes a statement and responds to a series of questions from the LLM as if the statement is true, trying to trick the LLM into believing it.
date system prompts be Xk.p- Let the maximum round limit for g be N, the ground truth be g. For each turn i, let x, be user's input, f&'s generation at each time step t can be denoted as Yi.t = fk ([x X1, YI. Xi, 2i[1:-11];; θx). We denote the complete output of length T is y; = y. For each y,, there two possible output types. The first one is ordinary question or answer in response to user's input to advance the gaming process. The other is a game secret prediction which requires the model to either guess the target word in Akinator and Taboo, make a prediction on the user statement's truthfulness. We denote the first type as y: and the second type as p;- The game ends whenever the maximum round limit is reached the user verifies p. g. A winner can therefore be decided.
EVALUATING LLM REASONING THROUGH GAMES
3.1 RETROSPECTIVE ANALYSIS
In order to reveal the model's intermediate thought process throughout the game session and assess various reasoning capabilities demonstrated by the model's hidden chain-of-thought (Wei et al.) 2022]. we conduct a retrospective analysis based on the chat history we collect from the game sessions. To reenact the game trajectory and capture the model's hidden thoughts during the game sessions, we first keep the system prompt, game history, and inference parameters identical to those used during the original gameplay. Then, we prompt the model to generate intermediate outputs, and conduct quantitative and qualitative analysis based on those outputs.
Akinator. The goal of Akinator's retrospective analysis is assess the model's multi-hop rea- soning and deductive reasoning capabilities (Yang et al. 2018 Creswell et al., 2022). After each crestion-answer round, the model is asked to generate a ranked list of possible objects from its hid- den thoughts during the gameplay. The object list prioritizes those it considers most likely based on its reasoning process. For each game session, we then inspect the ranked object lists from all rounds to determine whether the secret object has been correctly identified and its position in the rankings.
Taboo. The retrospective analysis of Taboo aims to evaluate the model's multi-hop reasoning and abductive reasoning capabilities (Yang et al.) 2018 8 Jung et al. (022). Similar to Akinator, the model prompted to generate list of possible target words based on all previous game history after each user prompt. The list of possible words is ranked from the most to least likely by the model.
Bluffing. The Bluffing game's retrospective anayssis aims to reflect the model's 's multi-hop reasoning and inductive reasoning capabilities (Yang et al. 2018 Misra et al.) 2022). After the user's statement and each question-anss answer round, the model is queried to predict the truthfulness of the statement based on the existing game history. For each prediction, the model selects from five possible choices: 1) True, 2) Possibly true, 3) Unknown, 4) Possibly false, or 5) False, and the model always starts from neutral position with "Unkown" its default choice. The intermediate predictions reveal the
4

---

Published as a conference paper at ICLR 2025
model's reasoning process and allows to assess not only its ability draw correct conclusions but also its capacity to ask strategic questions that consistently advance the game.
3.2
EVALUATION METRICS
3.2.1 OUTCOME METRICS
An intuitive set of metrics from gaming data is the gaming outcome. In Akinator and Bluffing, users provide feedback on whether the model made a correct guess, which determines the winner. In Taboo, ruled-based keyword detection mechanism is used in Taboo check whether the user has violated the game rules or the model has lost by saying the word. For each game, we track the average number of rounds and the winning rate by calculating L.rt/Ng and E; 4/Ng, where Ng is the total number of game sessions for game G. T2 is the number of gaming rounds for each game session, and I E {0, 1} is binary indicator function of whether the LLM won that session.
3.2.2 PROCEDURAL METRICS
In retrospective analysis, for any turn i, as long as y: F g. we obtain intermediate results by fixing all the game history [Xk.p; Y1, x; yi], and ask the model to justify its response y; p; using quantifiable data, such as an object list or a truthfulness rank. Consider the justification as y iterate over each game session collect step-by-step reasoning data [y'.-.. .'y], from where procedural metrics will be calculated for quantitative evaluations and rankings in Table 3
We developed several metries to evaluate each reasoning capability (Table T based on the model's intermediate outputs from retrospective analysis. Below, we describe how these metrics are mapped to each reasoning capability and how they are computed.
Akinator. Recall rate, top-k recall rate, and disparity ratio reflect the model's multi-hop reasoning capability in Akinator. The recall rate metrics determine how well the model utilizes preceding information by measuring how often the correct object appears in object lists retrieved for each game session during retrospective analysis. In each game session, recall rate represents the proportion of mroee--prpoose object lists that contain the correct object. Top-5 recall rate and top-10 recall rate represent the proportion of object lists where the rank of the correct object is within the top 5 and 10 positions. Higher recall rates imply higher utilization of preceding information, indicating better multi-hop reasoning skills.
Multi-hop reasoning is also reflected through the question quality in Akinator. We use disparity ratio to assess the information gain of a question. In Akinator, high-quality questions should divide the possibilities evenly into two halves (Sasson & Keneti 2023). Thus, compute the disparity ratio based on how evenly a model-generated yes/no question can divide the object list proposed one round prior into two categories. Evenly divided object lists have lower disparity ratio, suggesting higher information gain.
|sizcyes sizenol
disparity ratio
size list
We use average first appear round and average final rank as indicators for model's deductive ca- pability in Akinator game sessions. Average first appear round refers the average round number in which the model adds the correct object into the object list. Average final rank is the rank of the correct object the object list proposed after the last round of conversation. Lower average first appear round means the model is able to draw conclusion on the correct object with less premise. Lower average final rank shows more confidence on the model's choice of correct object. These two metrics both reflect the model's 's efficiency on deductive reasoning.
Taboo. In the Taboo game, we use recall rates to assess the model's multi-hop reasoning skill. The definition and implication of the recall rates here are the same those used in the Akinator game. We apply the same formulas on the word lists retrieved from retrospective analysis to compute the recall rates for Taboo game sessions.
Abductive reasoning skill is reflected by the metric of average first appear round and average final rank in Taboo game. Average first appear round is the average round number in which the model adds the correct word into the word list. A lower average first appear round means the model can efficiently find the correct word based on less interaction with users. Average final rank is the rank
5

---

Published as a conference paper at ICLR 2025
of the correct word in the word list proposed after the last round of conversation. A low average final rank implies that the model is more confident with its choice of the correct word. Both metrics indicate the model's abductive reasoning skill.
Table 1:
The procedural analysis metrics for the Akinator, Taboo, and Bluffing games and their corresponding model capabilities.
Deductive reasoning
Multi-hop reasoning
Akinator
first appear round avg. final rank
recall rate, top-k recall rate, disparity ratio
Abductive reasoning
Multi-hop reasoning
Taboo
first appear round avg. final rank
recall rate, top-k recall rate
Inductive reasoning
Multi-hop reasoning
Bluffing
first appear round avg, final rank
consistency rate, recall rate, top-k recall rate
Bluffing. In the Bluffing game, we use recall rate and consistency rate to evaluate the model's s multi-hop reasoning capability. recall rate measures how often the model correctly predicts the truthfulness of the user's statement. consistency rate measures whether consecutive predictions consistently move the game forward in the same logical direction, demonstrating the model's ability to build upon previous information and ask strategic questions. We use two metrics, the Spearman's 's coefficient (Sedgwick) 2014) and the hopping penalty score to measure consistency rate.
Spearman's 's rank correlation measures the correlation between the round numbers and the "distance" of judgement from the ground truth. At each round, we rank the five retrospective predictions y as Je E [1, 5]. corresponding to the 5-level spectrum spanning from "True" to "False". For each game session, Let D, be the absolute distance between the intermediate prediction and since the ground truth's 's rank is g E [1, we can write the expression in Eq. 2 A negative Spearman's is correlation suggests that as rounds progress, the distance between the assistant's judgments and the ground truth decreases and the model's prediction gets consistently more accurate.
N
6
d. d - Dj. D. = D
N(N2
The hopping penalty measures how inconsistent the model's predictions are across consecutive rounds. The smaller the metric is, the better the model is at making consistent progress toward its final prediction, demonstrating stability in questioning strategy and unpredictability its results. The formula is shown in Eq, 3
N-1 Σ
hopping penalty = N
1i+1 jil
We use average first appear round and average final rank as indicators for model's inductive rea- soning capability as they reflect how efficiently the model can make correct predictions in the fewest number of rounds. Average final rank is averaged across all game sessions, and the rank follows the same definition of D, in Eq. but only for the final round.
4 EXPERIMENTS
4.1 SETUP
GameArena evaluates LLM reasoning using gaming data to compare five SOTA models: GPT- 4o (OpenAI 2024b), Claude 3.5 Sonnet (Anthropic) 2024), Gemini-1.5 Pro (Reid et al. 2024
Mistral Large 2 (Mistral 2024), and LLaMA-3.1 40stutnst (MetaAl 2024). Over 10-week period from July 2024 to September 2024, we collect a total of 2240 game sessions using Cloud Re- search (Hartman et al. 2023) for evaluation. We then conduct retrospective data analysis introduced
in Section 3 on the gaming data to obtain outcome metrics and procedure metrics for each model.
Controlling system prompt variance. In GameArena, we develop a system prompt search and optimization pipeline using DSPy (Khattab al. 2024). This pipeline uses the chain-of-thought module from DSPy and one of the five models acting as the evaluator. The evaluator guide the model to follow the game rules and judges on response quality. The DSPy optimizer learns to bootstrap and

---

Published as a conference paper at ICLR 2025
identify effective system prompts. Each optimizer is optimized from subset of all our gaming data consisting of 200 game sessions. Given a game and one the five evaluators, the pipeline searches for an optimal system prompt that maximizes the evaluator's judgment the model's generated questions or answers at each round, resulting in five highly optimized system prompts per game.
4.2 DATA EFFICIENCY
We compared the data efficiency of GameArena and Chatbot Arena by analyzing the useful data rates. We obtained real conversation data from the Chatbot Arena Team for the week of August 26th September 1st and compared it with the data collected by GameArena during the same period. We considered data as useful if it provides meaningful signals that can be used to evaluate models. In Chatbot Arena, data was deemed useful if it included least one vote (e-g.. .. for left, right, tie, or both bad). For GameArena we considered the game session useful if the user completed the entire game round and provided feedback on the game outcome (i.e., win lose).
We found that 86.9% of the data from GameArena were valid and useful, while only 4% of total conversations in Chatbot Arena provided meaningful votes due its reliance on voluntary partici- pation. Our results suggest that GameArena could improve user engagement through gamification approaches and collect meaningful data for evaluating LLMs more efficiently.
4.3 USER STUDY
Participant ratings. We conducted study to compare the experience and willingness to participate in GameArena and Chatbot Arena (Chiang et al., 2024). We recruited 100 participants through Cloud Research (Hartman al., 2023) and asked them to try both platforms and fill out survey to provide their feedback. Each participant was randomly assigned a game GameArena and asked to play least three rounds. For Chathot Arena, participants voted on the better responses from two models to their questions. We provided sample questions from MT-Bench (Zheng et al., 2023) for reference, and participants were also free create their own questions. Participants rated each platform on how much they enjoyed it, how satisfied they were with the experience, and how often they would like to participate on a 5-point Likert scale.
Very
Negative
Very negative
How Much Do You Like TNss Game? (Strongly Like Strongly Dislike)
How Satisfied Are You with This Game? [Extremely Satistied Extremely Dissatisfied)
How Frequently Do You Play This Game? (Very Frequently - Rarely)
160
32
50-
Parcentage (%)
60-
40-
424
20-
28-
0-
6-
-30
-28
-40
AlinatoBllffingg
Taboo
Game
Chatbot
AkinatorBlutting
Chatbet
Akinatar Bluffing
Chatbot
Arena
Arena
Arena
Arena
Arena
Figure 4: Participant rating distributions across different games in GameArena and Chatbot Arena. lower bar represents a lower proportion of participants expressing a positive attitude. GameArena showed higher user enjoyment, satisfaction and willingness to participate than Chatbot Arena.
Our user study results suggest that GameArena is more engaging and enjoyable for users (Fig. On average, over 70% of users liked the games in GameArena, compared only 45% who enjoyed voting in Chatbot Arena. Additionally, over 80% of participants reported satisfaction with gameplay experience GameArena, compared less than 40% of users felt satisfied about the experience with Chatbot Arena. Participants also expressed stronger willingness to participate in GameArena, with over half of the participants would like to play these game frequently, whereas interest in Chatbot Arena more evenly distributed across different participation levels.
4.4 RANKING RESULTS
Model performance on outcome metrics. The final outcome metrics the three games are shown in Table 2 The win rates of both -ouude-3---somee-2222ee
and 4ot-40-2024-08- are higher in all three games compared to other models evaluated. The high win rate here reflects the strong overall reasoning ability of the two models.
7

---

Published as a conference paper at ICLR 2025
Table 2:
Outcome ranking for each game. Despite with system prompt search and optimization, in many Bluffing games, Mistral-large (Mistral, 2024) fails to follow instructions and make a final prediction on the truthfulness of the users' statements within the 5-round limit. The numerical values are averaged across the five system prompts, with the standard deviation included the error bar.
Akinator
Taboo
Blufling
Model
Avg. Win Rate
Arg. # Round
Avg. Win Rate
Avg. # Round
Avg. Win Rate
Avg. Round
claude-3.5 3.5-soone--222 20240620
0.58-am
26.
2.61
3.36 3.192034
0.67
6.00=am
20t-44-2024-08806
0,49=au
16.36 16.57.rm
0.67
0.58
5,92-au8
Comini-115-ppt
0.51
0.61
3.74
0.60
5,96_am
Ilama-3, 3-05b
0.44
17.15gnm
0.62
3.08
0.44gam
5,90
mistral large latest
0.02_ma:
19.99
0.66
3,43
0.0gaoo
6.00_am
Table 3: The retrospective analysis results of Akinator, Taboo, and Bluffing game sessions hosted by several SOTA LLMs.
Akinutor
Multi-hop reasoning
Deductive reasoning
Recall Rate
Top 5 Recall
Top 10 Recall
Disparity Ratio (%)
Avg. First Appear Round
Avg. Final Rank
Model
Rate (%)
Rate (%)
27.83
24.58
27.26
0.57
11.63
gp1-40-20144-08-66
20.32
17.00
19.57
0.55
11.96
6.64
gemini-11.5pre
19.79
18.56
19,79
0.58
12.12
5,44
llama-3-405b
15.90
13.72
14.97
0.60
14.67
9.88
mistral-large-latee
6.46
5.90
0.62
Taboo
Multi-hop reasoning
Abductive reasoning
Recall (%)
Top 5 Recall
Top 10 Recall
Avg. First Appear Round
Avg. Final Rank
Model
Rate (%)
Rate (%)
2Baude-3---oomet-2222222
65.82
1.56
1.48
gpt-4o-2024-06
55.78
$3.68
55.78
2.47
3.83
Semini-1.5-pp
55.57
52.58
56.31
2.55
2.44
llama-3-405b
71.22
65.69
70.80
1.98
4.04
mistral- mistral-large- largellatet
66.30
61,34
64.88
2.05
4.89
Bluffing
Multi-hop reasoning
Inductive reasoning
Recall Rate (%)
Spearman's Coefficient |
Hopping Penally J
Avg. First Appeur Round
Avg. Final
Model
Rank
14.12
-0.39
0.20
2.46
1.47
gemini-1.5-prg
9.05
-0.22
0.23
3.50
1.62
gpl-40-2024-08-06
7.64
-0.15
0.44
3.00
2,06
llama-34405b
19.05
0.26
0.48
3.00
2.57
mistral-large-attsst
0.00
-0,20
0.27
N/A
1.53
Model performance procedural metrics. Table presents the procedural metrics designed to reflect the models' reasoning skills from the retrospective analysis of all three games.
performs best on recall rates, average first appear round, and average final rank, demonstrating stronger multi-hop reasoning and deductive reasoning capa- bilities compared to other SOTA LLMs being evaluated. S00-44--024-08 also has high perfor- mance in the metrics reasoning capabilities.
Akinator. BBaude33---somee-22222000
Taboo. The high performance of round, and average final rank exhibits stronger multi-hop and abductive reasoning capabilities. No- tably, llama-3-405b demonstrates strong multi-hop reasoning capabilities with leading recall rates, which differs from the results observed the Akinator game. This discrepancy may be due to the task differences: in the Taboo game, the model passively responds to prompts, whereas in the Akinator game, it must actively generate questions to succeed.
20240620 in recall rates, average first appear
Bluffing. In many cases, Mistral-large (Mistral 2024) tends to be very conservative in making predictions. Note that with a large hopping penalty and a high recall rate, LLaMA-3.1 405B tends to make bolder predictions, and its intermediate reasoning outputs tend to be slightly more inconsistent.
8

---

Published as a conference paper at ICLR 2025
4.5 RANKING COMPARISONS
Cross-dataset consistency analysis. GameArena can be extended to include as many games as possible for evaluating LLM reasoning, following the formulation in Section 2.2 In games like Taboo, we can restrict the set of secret words users are allowed to use, while in others, like Akinator and Taboo, users have the freedom choose the game secret. It is crucial to quantify the variances in evaluations when the game secret space is large to ensure consistency and fairness across different game setups.
We conducted a controlled experiment to quantify GameArena's evaluation consistency on these games. In this experiment, we asked human experts (graduate students) to hand curate two distinct subsets of gaming data, where the game secrets are sufficiently diverse to both cover wide range of topics. For each game, the two sets each contains game sessions. We show the outcome metrics for each set in Table We only evaluate the Akinator, and the Bluffing game Taboo has restricted game secret space. The results from both subsets show a strong agreement, demonstrating the reliability of our ranking system. More details of the experiment can be found in Appendix B
Figure 5: Controlled experiment results for Akinator and Bluffing. The numerical values aver- aged across the five different system prompts, with the standard deviation included as the error bar.
Akinator Controlled Experiment:
Win Rate
Bluffing Controlled Experiment: Avg Win Rate
0.75
Avg Win Rate
0.75
800 Win
0.50
0.50
0.25
0.25
0.00
0.00
claude:3.5 5
gemini-1. A-pro
gnt-4o
Ilamaa3.1 1
mistral-larg Harral-large
daude 5
gemini-1 5-pro
gpt-4o
Ilama:3. 1.
mistral-large
Model
Model
Table 4: Kendall's tau and Rank-biased overlap (RBO) between Aree's rankings vs. open- ended dynamic benchmark Chatbot Arena, monthly-updated reasoning benchmarks Livebench- reasoning, Trustbit-reasoning, and a static reasoning benchmark GPQA. Higher values indicate a greater level of agreement between the rankings.
GameArena Rankings
Chatbot Arena Kendall's Tau RBO
LiveBench- Reasoning Kendall's Tau
GPQA
RBÒ
Kendall's Tau
RBO
Akinator- Catattr-Outcome
0.4
0.86
0.6
0.93
0.6
0.93
Akinator-Rette (deductive)
0.6
0.93
0.8
0.98
0.8
0.98
Taboo Mato-OOutcome Taboo-Retro (abductive)
0.2
0.74
-0.2
0.61
-0.2
0.61
0.6
0.93
0.8
0.98
0.8
0.98
Bluffing-Outee -otting--Outomeeeee
0.4
0.86
0.6
0.93
0.6
0.93
Bluffing- -etro (inductive)
0.4
0.86
0.6
0.93
0.6
0.93
Ranking correlations. We compare our ranking results from Table 2nd Table rankings from both dynamic and static benchmarks. We use statistical methods, including Kendall's tau and rank- biased overlap (RBO) (Kumar & Vassilvitskii 2010; Shickh & El-Hashash 2022), to perform the analysis. The expressions for Kendall's ' tau and rank-biased overlap (RBO) are described in Eq. ??, where sizeagree and sizedisagre are the numbers of concordant and dis-concordant pairs, 5 is the total number of models in the ranking, R1,d, R2,d are ranking from two different benchmarks with top elements, p (0, is the persistent parameter, which serves the weights given to low- ranked elements. The correlations of GameArena versus different rankings are shown in Table using p 0.9 as the weight parameter. The latest rankings as of September 2024 are in Appendix
aireegree
sizedisagree
(Kendall's Tau)
(4)
2

---

Published as a conference paper at ICLR 2025
RR. O Reyal "
-1--1) p) Σ RBO R1, R2,p) = =(1 -
p (Rank-biased Overlap)
(5)
d
The results show a particularly strong agreement between GameArena's rankings on Akinator and Taboo's s procedural metrics (multi-hop, deductive and abductive reasoning) and the monthly updated reasoning benchmark
(White et al., 2024), as well as the static reasoning benchmark GPQA (Rein et al. 2023). One potential source of disagreement is that the other rea- soning benchmarks are designed to capture somewhat different reasoning skills than those reflected in GameArena. Specifically, LiveBench-Rease Reasonins tests subset of BigBench- -ard (Suzgun ct al. 2022) and Zebra Puzzles (Jeremy] (2009), while GPQA also assesses graduate-level academic knowledge. It is worth noting that GameArena shows disagreement with Chatbot Arena, as the latter evaluates holistic performance rather than specific reasoning capabilities.
SeveBench-Reasoniie
5 RELATED WORK
Benchmarking LLM reasoning capabilities. Existing works develop a wide range of benchmarks to test the models' reasoning capabilities, such as logical reasoning (Hao et al., 2024 bench authors
2023
Valmeekam et al. 2023 Saparov & He 2023), coding and mathematical reasoning (Cobbe
al.
2021). However, these benchmarks often rely on static datasets that be vulnerable to
data contamination (Sainz et al. 2023; White et al. 2024) and easily become saturated (Kiela et al.)
2021
Perlitz al. 2024). To address this, Chiang al. (2024); [Zheng al. (2023), Zhao et al.
use human or LLM as judges to evaluate the responses to dynamic questions of different LLMs. However, recent studies found this may introduce various biases (c.g., style preference) that
2024
compromise the reliability of evaluation results (Chen et al.) [024] [i et al.) 2024). In contrast, GameArena evaluates LLMs through interactive gameplay with humans using objective metrics derived from the gaming process to create a more engaging and robust benchmarking approach.
Evaluating large language models using games. Recent works have explored interactive and game-based evaluations to assess LLM capabilities (Wu et al.] [024 Liu et al., 2023). For example, Madge & Poesio (2024); Hafner (202I) use single-player games (c.g., MineCraft) as a controlled environment to test specific capabilities of LLMs. Topsakal et al. (2024); Xu et al (2023); Cheng cet al. (2024) employ multi-player games to evaluate LLM through self-play or competition among different LLM agents. In contrast, GameArent uses crowdsourced approach to engage human players in the LLM evaluation process by offering games that people can play to have fun. Our work is similar RedTeam Arena (Angelopoulos et al. 2024), which features game prompting players to elicit target "bad words" from models, but the key difference is that Red'Team Arena focuses on red-teaming, while our games are designed for evaluation of LLM reasoning capabilities.
Games with a purpose (GWAPs) are online games designed to be fun for humans to play while also accomplishing meaningful tasks that are challenging handle by computers alone (Von Ahn & Dabbish 2004 Von Ahn 2006 Von Ahn & Dabbish 2008). For example, Eye into Al (Morrison al. 2023) uses GWAP techniques
evaluate explainable Al approaches by revealing salient portions of an image and asking players to guess the object. To the best of our knowledge, we are the first to integrate LLMs into playful games in a way that does not disrupt the gameplay experience while also generating meaningful data for evaluating L.M reasoning capabilities.
6 CONCLUSION
In this work, we present GameArena, a dynamic benchmark for evaluating LLM reasoning capabili- ties through interactive gameplay with humans. GameArena enables the efficient collection of LLM step-by-step reasoning data with human labels as a natural by-product of human gameplay sessions, while keeping participants entertained and engaged. Our initial release of GameArena consists of three conversational games that require strong reasoning skills: Akinator, Taboo and Bluffing. By controlling LLM interactions with humans through specific game rules and objectives, these games assess fine-grat fine-grained LLM reasoning skills, including inductive, deductive, abductive and multi- hop reasoning. We plan to incorporate more games into Game and improve our evaluation metrics to further understanding of L.M reasoning capabilities in complex, interactive contexts.
10

---

Published as a conference paper at ICLR 2025
7 ETHICS STATEMENT
Since GameArena involves human subjects playing games to collect LLM reasoning data, we re- cruited participants from Cloud Research and compensated them 8 USD per hour for their time. The crowd workers consented to the use of their responses in the study, and no personal data was collected. Our user studies were approved by the institutional review board of organization.
REFERENCES
Akinator. Akinator the web genie, 2024. URL https://en..kiiatorrr Accessed:
2024-09-27.
Anastasios Angelopoulos, Luca Vivona, Wei-Lin Chiang, Aryan Vichare, Lisa Dunlap, Sal Vivona, Pliny, and Ion Stoica. Redteam arena: An open-source, communtyyddriven jailbreaking platform, Sep. 2024. URL cessed on 2024/09/23).
wwarema. http://ww....aaaaaaa/a/////aaaaa
(Ac-
Anthropic. Claude 3.5 sonnet, 2024. URL
3---ide-----onnet Accessed: 2024-09-27.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
https://cpenrevi/
net. forum http:/oouumnnnuyySaa1e- iid=uyTLSvosssSS
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans Ilms as the judge? a study on judgement biases. arXiv preprint arXiv:2402. 10669, 2024.
Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai. Lei Han, and Nan Du. Self-playing adversarial language game enhances Ilm reasoning. arXiv preprint arXiv:2404. 10642, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https:
org/abs/ faabs/2110.. 14168
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inffrrncce: Exploiting large lan- guage models for interpretable logical reasoning. arXiv preprint arXiv:2205.09772,, 2022.
Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06789. 2021.
Simon Jerome Han, Keith James Ransom, Andrew Perfors, and Charles Kemp. Human-like property induction is challenge for large language models. In Proceedings of the annual meeting of the cognitive science society, volume 44, 2022.
Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, and Zhiting Hu. Lim reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models, 2024. URL https: // //arxiv.. org/ abs 2404 200.02222222221
Rachel Hartman, Aaron J Moss, Shalom Noach Jaffe, Cheskie Rosenzweig, Leib Litman, and Jonathan Robinson. Introducing connect by cloudresearch: Advancing online participant recruit- ment in the digital age. 2023.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint M0Xi:2001.000060 2020.
11

---

Published as a conference paper at ICLR 2025
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, 2023. httta:://axx....////aab..s
S Jeremy. Einstein's ' riddle: Riddles, paradoxes, and conundrums to stretch your mind. Bloomsbury USA. pp. 10-11, 2009.
Jachun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprint arXiv:2205.11822 2022.
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compil- ing declarative language model calls into state-of-the pipelines. In The Twelfth International Conference on Learning Representations, 2024.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in nlp. arXiv preprint ar arXiv:2104 14337 2021.
Ravi Kumar and Sergei Vassilvitskii. Generalized distances between rankings. In Proceedings of the 19th International Conference on World Wide Web, WWW '10, pp. 571-580, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781605587998. doi: 10.1145/ 1772690.1772749. Utthtts:://doo org/ 111...11111....117707
Tianle Li, Anastasios Angelopoulos, and Wei-Lin Chiang. Does style matter? disentangling style and substance in chatbot arena, Aug. 2024. URL (Accessed on 2024/09/23).
2024-08 08 28-28-10 100-28-styles
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023, URL
pettss//aaxiin /tttps///axxt..orr/ab///// http://bs/233..2000
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating lims as agents, 2023. URL https:
org/ abs/
2308. 03688
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. URL http::://///// arxiv. wwtp:/orggabbbbabb 06592
Chris Madge and Massimo Poesio. A Ilm benchmark based on the minecraft builder dialog agent task. arXiv preprint 122v:2407.12733, 2024.
MetaAI. Introducing llama 3.1, 2024.
URL https://al.meta..omm
meta-llamaa 3-1/
Accessed: 2024-09-27.
Kanishka Misra, Julia Taylor Rayz, and Allyson Ettinger. A property induction framework for neural language models. arXiv preprint arXiv:2205.066) 2022.
Mistral. lartra Unveiling our latest model, 2024. URL wwLLaattss//nistaallaaa/// Accessed: 2024-09-27.
mistral- 1arr1l--rree-2-0000
Katelyn Morrison, Mayank Jain, Jessica Hammer, and Adam Perer. Eye into ai: Evaluating the interpretability of explainable ai techniques through a game with a purpose. Proceedings of the ACM on Human-Come Computer Interaction, (CCSCW2) 2023.
OpenAl.
Openai ol system card, Sep. 2024a.
URL
https: //
assets. bec.ttttttaastt...eaaaaaaaaaaaaa 77551022220775111833318222223223 2024/09/23).
(Accessed on
12

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4o system card, 2024b.
URL
https://cdh. com/
gpt - 1e---o--ystttt sysstmm-cardd pdf Accessed: 2024-09-27.
Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli- Scheuer, and Leshem Choshen. Benchmark agreement testing done right: A guide for Ilm bench- mark evaluation. arXiv preprint arXiv:2407.13669 2024.
Tanmay Rajore, Nishanth Chandran, Sunayana Sitaram, Divya Gupta, Rahul Sharma, Kashish Mit- tal, and Manohar Swaminathan. Truce: Private benchmarking prevent contamination and im- prove comparative evaluation of llms. arXiv e-prints, pp. arXiv-2403, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5; Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0555.0 2024. URL htt.tttss////rxiiii org/ http://bs//22333/33 2bt.a/////233333
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL httpt::tp//.......... /tttpp://arxiie org. abs /211 12022
Oscar Sainz, Jon Ander Campos, Iker Ferreo- Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NIp evaluation in trouble: On the need to measure IIm data contamination for each benchmark. arXiv preprint arXiv:2310.180) 18018, 2023.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of- offthought,,, 2023. URL
Gal Sasson and Yoed N Kenett. A mirror to human question asking: Analyzing the akinator online question game. Big Data and Cognitive Computing, 7(1):26, 2023.
Philip Sedgwick. Spearman's 's rank correlation coefficient. BMJ, 349, 2014. ISSN 0959-8138. doi: 10.1136/bmj..7335 URL https:/ http::://wwww... /www. bmj com. cont ent 349/bm3 90 327
Norbert M Seel. Encyclopedia of the Sciences of Learning. Springer Science & Business Media, 2011.
Rega Hassan Ali Shiekh and Essam F El--Hahaahh. A comparison of the pearson, spearman rank and kendall tau correlation coefficients using quantitative variables. Asian Journal of Probability and Statistics, pp. 36-48, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether Mhain-of-thought can solve them, 2022. URL tpss:ttts////
2ess/221..0 09261
Oguzhan Topsakal, Colby Jacob Edell, and Jackson Bailey Harper. Evaluating large language mod- els with grid-based game competitions: An extensible Ilm benchmark and leaderboard. arXiv preprint ar 2rXiv:240..07796 2024.
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models planning and reasoning about change. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neural Information Pro- cessing Systems, volume 36, pp. 38975-38987. Curran Associates, Inc., 2023. URL httpss:/pproceeinngs..ee neur /ttpr/////cpeppppeeeeeee//////// [a92oceddeccaatdde822233336--
and Benchmarks.
pdf
Luis Von Ahn. Games with purpose. Computer, 39(6):22-44 2006.
Luis Von Ahn and Laura Dabbish. Labeling images with computer game. In Proceedings of the SIGCHI conference Human factors in computing systems, pp. 319-326, 2004.
13

---

Published as a conference paper at ICLR 2025
Luis Von Ahn and Laura Dabbish. Designing games with purpose. Communications of the ACM, 51(8):58-67 2008.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, al. Livebench: challenging, contamination- free Ilm benchmark. arXiv preprint arXiv:2406. 19314, 2024.
Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. Smartplay: A benchmark for LLMs intelli- gent agents. In The Twelfth International Conference on Learning Representations, 2024. URL
hittps://cpenrett
net/ Eo Torum?1d=S2otvriicg
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint 2Xiv:2309.006658, 2023.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio. William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse. explainable multi-hop question answering, 2018. URL https:: /al/aaxiv... org/ abs/ (9000096900 09600
Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. Language models inductive reasoners. arXiv preprint wrXi::2212.109233 2022.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process, 2024a. URL https://arxiv.orz
abs abs/2407 20311
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems, 2024b. URL https://arxiv:
org/ ha5//bb//tt 24408.66220
Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Deli Zhao, and Lidong Bing. Auto arena of Ilss: Automating 1lm evaluations with agent peer-battles and committee discussions, 2024. URL
https://arxiv org/ abs abs 2400. 20267
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Sto- ica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine (eds.), Advances in Neural Informa- tion Processing Systems, volume 36. pp. 46595-46623. Curran Associates, Inc., 2023. URL. /ttpps://prooceeingss..eeeei neur http::/wwww.p...o....ee///ee///a
91f 8a 13
pdf
14

---

Published as a conference paper at ICLR 2025
A GAME DESIGN
A.1 GAME RULES
We provide a detailed description of the game rules for each game in GameArena below and explain why these games are challenging for LLMs.
A.1.1 AKINATOR
In the Akinator game, the LLM attempts to determine what object the player is thinking of by asking up to 20 yes or no questions. The goal of the LLM is to guess the answer correctly with the fewest number of questions possible. In each round, the LLM asks binary question, and the player responds with "yes," "no," "probably yes" "probably no," or "not sure." The LLM asks strategic questions to gather information until it feels confident enough to make a guess. The player then provides feedback on whether the guess is correct. If correct, the model wins; if not, it can continue asking questions or make another guess until it uses all 20 chances. If the LLM fails to guess the object, the player is prompted to reveal the target object to ensure data quality and prevent cheating.
The Akinator game challenging for LLMS due to its vast, open-ended answer space. LLMs must accurately interpret user responses, manage ambiguity, and avoid irrelevant or redundant questions to efficiently narrow down a wide range of possibilities from limited information. This challenge is intensified by the need to connect information across diverse knowledge domains and engage in complex, multi-step reasoning to arrive at the correct answer.
A.1.2 TABOO
In the Taboo game, the goal of the human player is to prompt the LLM with questions that will lead the model to say the target word. Meanwhile, the LLM tries guess the word to avoid uttering it. The system first randomly assigns the human play taboo word (e.g., "eggs") from a predefined word list. The user must prompt the model say the word without mentioning directly. In Fig. 2 the user begins with a general description of common breakfast foods, which leads the LLM to respond with "cereal." The user then narrows it down by asking about foods that can be scrambled or fried, prompting the LLM to successfully utter the target word. At this point, the LLM has chance to guess the word; it guesses correctly, it wins.
The Taboo game presents several challenges for LLMs. The model must interpret vague clues and general descriptions, which can vary significantly, to recognize when close to the target word based on the user's hints. This requires quick connections across various potential answers. Additionally, the LLM needs to balance providing relevant responses with avoiding the taboo word, necessitating quick and strategic reasoning.
To make the game more challenging and engaging for users, the human wins only if they get the LLM to say the word within five rounds, with each question limited to 140 characters, and the LLM does not guess the word correctly. This setup makes the taboo game a challenging task for the I.M,, as human players will avoid easy and detailed clues, requiring the LLM navigate complex language cues and context clues to figure out the target word.
A.1.3 BLUFFING
In the Bluffing game, the human player first makes statement that can be true or false, and attemp to convince the LLM of its truthfulness. The LLM can ask up five questions player is lying, aiming to accurately judge the truthfulness of the statement in the fewest rounds possible. For instance, LLMSs may detect deception by identifying subtle flaws or inconsistencies in the player's responses. After the LLM makes its prediction, the player provides feedback on whether the statement is true.
determine if the
The Bluffing game is challenging for LLMs because it requires them to evaluate the truthfulness of a statement based on limited interactions (i.e., within only five question-ans aadooo--nd--nnwwee pairs). LLMs must effectively interpret the nuances of human communication, including tone and context, to detect deception. Additionally, they have to formulate strategic questions that maximize information gain while minimizing the risk of misleading answers.
15

---

Published as a conference paper at ICLR 2025
A.2 GAMEARENA USER INTERFACE
We demonstrate the user interface for each game from Figure to Figure g
Akiatoor Game Rules
object. assistant try guens
assistant will have 20 chances to ask questions.
assistant successfully guess object, They win.
Question 9:1s this object used for sleeping beddling?
Yes
Probably
Probably
Figure 6: The user interface of GameArena for the Akinator game
system with you you tanget
The goal for you prompt the assistant with questions such that it utters the target word unconciously win game.
You could not explicitly include the target word your input. Otherwise, you lose.
turns,
They
the assistant uttered the word, but able the word correctly. They win.
System message - Here is your target word: Liberia,
Please first make the
unconciously
What is the name of water sport that combines sai ling and surfing
Figure 7: The user interface of GameArena for the Taboo game
Bluffing Game Rules
The system will provide with you a question. You can make either true false statement,
the first you would need make statoment.
Your goalls to convince the Al assistant that you are making a true statement.
you up questions try guess you are
your
feedbacks.
if the Al assistant guess whether you bluffing, They win.
Share your experience of surviving natural disaster.
Provide
survived an earthquake over 8.0 Japan.
Frest Enter
Figure 8: The user interface of GameArena for the Bluffing game
16

---

Published as a conference paper at ICLR 2025
A.3 EXAMPLES OF GAMEPLAY SESSIONS
We demonstrate one example of gameplay session for each game from Figure 9 to Figure 11
Game Conversation
Question 1: Is the object you're thinking of somethi tangible that can be physically touched?
Yes
Question 2: is this object typically found Inside home?
Yes
Question 3: is this object primarily used for sitting resting?
Figure 9: An example game conversation between the LLM and human players in an Akinator game
System Here your target word: Liberia.
What the of sport that combines salling surfing?
That sounds like an exating activityl Based your description, I'd it's sport that invalves riding board propelled the wind, otan open The thinking popular coastal skills, Is that the kind of yourt thinking of?
requires combination balance, strongth, and naviget navigation
you provide me name
can provide specific Based your hint about combining sailing and surfing, say sport comes board sport that utilizes the wind propulsion. It's popular recreational activity that involves a board with a sail or rig attached to it.
description
type
should answer with one word
Windsurfing
Figure 10: An example game conversation between the LLM and human players in a Taboo game
17

---

Published as a conference paper at ICLR 2025
Game Conversation
Statement: sunvived an earthquaks over 8.0 Japan.
haan you your statement. Let's begin the questioning process explore this further.
Question year specific Japan you experience earthquake?
+ experienced the earthquake Fukushima in 2011.
you providing
question.
Question you mediate actions you carthquake began, where were you that moment?
+
working my office and hid under the desks to protect myself.
Thank you sharing that. Let's move an the next question.
Question 3: Given that the Fukushima earthquake was followed devastating tsunami, can describe any experiences observations you had related to the tsunami tsunamiafteman afternath?
The tsunami made the situation more chaos and dangerous. We had wait until the tsunami dangerous than insides buildings.
passed.
Figure 11: An example game conversation between the LLM and human players in Bluffing game
A.4 EXAMPLES OF GAME PROMPTS
A.4.1 AKINATOR
You are an intelligent assistant tasked with playing game Twenty Questions. Your geal is to guess generic object that the user is thinking of. Assume the role of a creative player while examining the game history. Utilize your imagination to craft an engaging and thought-provot
next question that not only continues the game's s natural progression but alse ignites participants' interest and curiosity. Ensure that the question resonates with the ongoing narrative while offering a unique twist to maintain excitement. The rules specified as follows:
1. Questicning Numbering: Each question starts with question header 'Question N:' where Ensure each question is sequentially numbered, avoiding any repetition of the question header 'Question N:'.
a numerical integer from to 20.
2. Single Questien Format: In each turn, ask only YRS NO question. Your response should only contain the question|ns redundant text should be included.
Only these responses are acceptable: Yes, No, Probably Yes, Probably No, Don't t Know.
3. Accepted Answers:
4. Fault Tolerance: Be tolerant of potentially incorrect answers from the
and adjust your questions accordingly.
5. Avoid Redundancy: Do not repeat or ask similar questions more than once. Aim to ask unique questions that progressively narrow down the possibilities.
18

---

been asked and only if you are sufficiently contident, first answer the latest question and then you can optionally make a final guess of the secret Published as a conference paper at ICLR 2025
6. Efficient Questioning: Strategically balance asking more general questions that can broadly narrow down possibilities versus making guess to save question quota.
7. Confident Guessing: If confident, make a guess using the format: 'This is a guess -- are you thinking of $object?', where Sobject is your proposed guess. Avoid any additional text.
8. Generic Guessing: Don't guess too specific an object. when confident encugh, make a guess with the following format: "Tii is guess -- are you thinking of Sobject?'
9. Adaptive Questioning: Base each question on the user's s previous responses to refine your understanding and approach.
10. Consider Edge Cases: When making a guess, don't constrain your thinking to a specific category. Be creative and consider edge cases to avoid premature conclusions.
Game session begins now. Let's preceed step by step:
A.4.2 TABOO
You are an intelligent assistant tasked with playing vord-guessing game. The user will start with target word in mind, and your goal is to guess this word. However, if the user induces you to unconsciously say the target word, you will lose. Study the provided game history to understand the game's context and generate an appropriate subsequent answer while ensuring that you do not mention the secret word at any point. The answer should respect the game 's logis and rules as implied by the history. Note that the user's messages cannet explicitly contain the word itself. The rules are as follows:
1. Respond to Questions: Each turn, the user is expected to ask a question. Answer accordingly while avoiding using the target word forming interrogative sentences.
2. Avoid Lesing Words: Be cautious not to generate the target word or any word that might lead to losing the game.
3. Make Educated Guesses: Based en the user's questions, try to guess the target word, ensuring you avoid asking directly about it.
4. Guess Upen Lesing: Only take guess if you are very confident
or you have uttered the word. Use the following format: My guess of the word is: .....
5. Maintain Natural Dialogue: Keep the conversation with the user natural and ccherent, without unnecessary content.
6. Optional End of Game Guess: After 5 questions
word in this format: 'My guess of the word is: ...'. Notice that guessing wrong word could lead you lose the game, do it only if you are very confident about the answer.
7. Fost-Game Analysis:
Provide an analysis along with your game
guess.
The game sessicn starts now.
let's proceed step by step:
19

---

Published as a conference paper at ICLR 2025
A.4.3 BLUFFING
You are an intelligent assistant tasked with playing lie detection game. the beginning of the game, the will make statement about themselves. Your goal true based en common sense and the user's answers. Xeep in mind that the user may Lie in their responses. Based on the provided history of the game (questi ens and answers so far), craft relevant and engaging follow-up question to progress the gare. Pocus mai maintaining consistent theme and ensuring the question is logical with respect to prior game events and answers. Consider extracting key themes, unresolved queries, or interesting aspects highlighted in the game history to inspire this follow-up questien. The rules
to determine whether the statement
follows:
1. Statement Introduction: After the user makes statement, you should proceed to ask the first question.
2. Questicning Numbering: Each question starts with a question
header 'Questicn N:' where N is a numericsl integer from 1 to 5. Ensure each question is sequentially numbered, avoiding any repstition of the questien header 'Question N: Don't generate the question header after the 5th question.
3. Sequential Questioning: Ensure that each question is unique and each question header is incrementally numbered from 1 to 5. Avoid repetitively generating 'Question 1:' without incrementing the number correctly.
4. Single Questien Per Turn: Ask only one question per turn and avoid generating any additional content or redundant information.
5. Test Expertisc: Ask questions that evaluate the user's s exportise and knowledge in areas related to the statement.
6. Build Responses: Base each subsequent question the user's previous responses to ensure relevance and depth.
7. Limit of 5 Questions: You have up to turns to ask questions.
8. Avoid Premature Judgment: Avoid making a lie-detection judgment before all questions are asked. You can reserve your judgment until the end of the questioning.
aware that users might provide incorrect answers by accident. Remain fault-tolerant and adjust your questioning accordingly. However, you do need to ask the user to provide sufficient details about the statement. or related background.
9. Fault Telerance:
10. Encourage Creativity: Don't limit yourself questions. Be creative and think cutside the box to fully explore the truthfulness of the statement.
specific set of
11. Final Judgment: Only if you are sufficiently confident, after before all questions are asked, make your final guess using the format: 'I believe your statement is: followed by either 'True' or 'False'. Don't include the question header.
12. Post-Game Analysis: Provide an analysis of the statement. and your reasoning ng after the game sessicn has ended.
The game session begins now. Let's preceed step by step:
20

---

the least Published as a conference paper at ICLR 2025
A.5 RETROSPECTIVE ANALYSIS PROMPTS
A.5.1 AKINATOR
Given the previcus game history, generate a new list of most possible objects (at least 5, up to 16) and rank those objects from the most possible
the least possible ones. This list should help you guess the correct object the user thinking of.
You need to order the objects based on how likely do you think the object is the correct answer.
Make sure all objects you generate comply with the game history.
In the game history, you can find Each questien asks if the object belongs a certain category. If the answer is Yes, it means that the object belongs to this category. If the answer is No, it means that the object does not belong to this category. Den't generate any more objects that belong to this category.
question-answer pairs.
Consider possible objects from all kinds of categories based on the game history.
Don't generate impossible objects based on the game history.
Be as diverse as possible. Think out of the box.
If you very confident that the object is the correct answer, can generate list with only one object.
Being very confident means that you are very certain that you know the object the user is thinking of.
Output each object in the format: *** Object: <object> **1
Give additicnal information to justify how you generate the object list.
Provide the additional information the following format: Additional Information:
A.5.2 TABOO
Given previ game history, generate list most possible target words (at least 5, up to 16) and rank those words from the most possible ones to the least possible ones. This list should help you guess the correct target word the user is thinking of.
You need to order the words from the most possible possible one.
Make sure all the words you generated comply with the game history.
Consider all kinds of possible words based on the game history. Don't generate impossible words based on the game history. Be as diverse as possible. Think out of the box.
If you very confident about word, you can generate list with only one word.
Being very confident means that you are very certain that you know the word the user is thinking of.
Output each word in the format: ** Word:
<word> ***
Give additienal information to justify how you generate the word list.
21

---

Published as a conference paper at ICLR 2025
Provide the additional information in the following format: '*x Additional Information:
A.5.3 BLUFFING
Given the previ cus game history, make prediction on whether the user's s statement is true. You have five possible predi ons to choose from:
1. True
2. Possibly true
3. Unknown
4. Fossibly false
5. False
Guidelines:
Always begin from a neutral position. If you' re uncertain about the truthfulness of the user's statement, choose 'Unknown.'
Move toward either 'True' 'False' as you collect or have a justification (c.g., if the user is not providing serious or relevant answers).
information
Output your prediction in the format: *** believe your statement is: <prediction> **', where <prediction> is one of the 5 choices.
Give additional information to justify the reasoning your prediction.
Provide the additional information the following format: ** Additional Information:
B CROSS-DATASE CONSISTENCY ANALYSIS DETAILS
From all gaming data, hand curated two of data, where the game secrets cover sufficiently diverse sets of topics and the oulcome meirics show high consistency in Table
Table 5: Controlled experiment results.
Akinator
Set 1
Set 2
Model
Avg Win Rate
Avg Round
Avg Win Rate
Avg Round
Shmule-3-S-somme--22222
0,56
16, 1.74
0.53_a11
17.28
gemini-1.5-1 1.5-pro
0.52
16,90 16.90
17.37
200-44-2022-08-4
0.50
16.20
0.53
15.87 40.9
1lama-3.1
0.43
17.324t065
0.40
18.01 +0.6
mistral-large-latest
0.02:1004
19.79
0.01 =0.03
19.1940.02
Bluffing
Set 1
Set 2
Model
Avg Win Rate
Avg Round
Avg Win Rate
Avg Round
0.68
6.00+a00
0.70
6.00
gemini- 1.5-iro
0.59
5,90
o.23
5.95
gpt-4o-2024-08-06
0.59
5.94
0.58
5.91
1lmma-3.14005
0.45
5.96
0.49 0.25
5.87
mistral--argo-latter
0.00
6.00+000
0.00
6.00
Game secrets from the provided below.
22

---

Published as a conference paper at ICLR 2025
B.1 AKINATOR
B.1.1 1 SET
fan", "dental floss", sextant", "an airplane", "ski poles", "a wooden clothespin", "a gaming controller", "a straw", "nail clippers", "a motorcyclee, "bookends". "a colander", "a vacuum cleaner", "a signpost". "a vase", "a sofa". "a key". "an armchair", "a knife". "a yacht". "a digital clock". "a television", "a file holder". : "a measuring cup". "a board game". "a cutting board", "a laptop". "a smartphone", "a dishwasher detergent or cleaning agent", "a pool float", "a dog". "a sleep mask", "a desktop computer", "a car", "a fire pit", "a bag". surgical cap". "a gaming console", painting". "a statue", "a pillow", "a ball", "a golf club", "a microwave", "a clock", "a dart", "an umbrella", thimble", "a stool", "a flashlight"
gagming
B.1.2
SET 2
carrot", "a military vehicle", "headphones", river", jukehox". "a towel", "a toothbrush", "a handheld CB radio", "a guitar", "a stress ball", "a Magic 8 Ball", "shampoo", "a soccer hall", " pencil", "a yoga mat", "a bottle opener", "a video game console", "a pen", "a glass", "a camera", "a coffee maker", "a recliner", "a picture frame", "a lipstick", "an MP3 player", ; "a snow globe", "a pool float", desktop computer".
cutting board", "a blender", "a stapler". "a paperclip". "a spoon", "a fork", "a couch", . "a bed", "a lamp". bookshelf", "a microwave", "a frying pan", "a coffee table", "a remote control", "a ceiling fan", doorknob", calendar", "a mirror", "a rug", backpack", bicycle", "a helmet"
B.2 BLUFFING
B.2.1 SET
"I eat my own boogers", "I made honor roll in college", "i once jumped out of plane". "There were severe forest fires my which caused the destruction of houses and also created immense amount of smoke which forced people to evacuate my town, but fortunately, we survived.", 'I have the world record for growing the longest beard in the world", remembered being dared to climb up tree and jump off it. The reason for this is because we wanted to see who has better landing.". "Been mistaken for someone else.", "I had chance meeting with the director Ridley Scott.", "I stopped record
student from jumping off of tall building by letting him sing won a chess tournament in high school.". "When went to Jamaica, went para-sailing for the first time.", "I have a shiba inu that is white". "My first job was as literary magazine editor"" "When I was kid I was wrestling with my brother and jumped off a stool and landed on the ground and split open my chin", "The most dangerous thing I've done driving while tipsy." "I skipped grade", "Cantonese" "I won an with dolphins", "The most unusual place I've visited an optical illusion museum.", "1) Japanese, 2) Hmong", "I saw someone collapsed on the ground on the street and called the ambulance for them", "Offering money the homeless". "I performed with my rock band in large restaurant two years ago.". "My college major was Physics.". ", "I have recently learned how sway people's minds", "A stranger walked up to me and complimented me on my good looks. We exchanged numbers after short dialogue.", "Donald Trump". "i am law major". "i work at an animal shelter". "I once went bungee jumping off the Bloukrans Bridge in South Africa, which is the highest commercial bungee jump in the world at 216 meters.". "I visited Chernoyyl to the nuclear reactor", "I met Alistair McGowan when performed with him on stage.", "I was once mistaken for Daniela Nardini".. "I 6'3", won contest for a free Xbox Series X years ago from Taco Bell", won a Batllefield gaming competition." "An accomplishment am proud of would he being deht free at the end of every year.", learned how to curl 40 pound dumbbells 7 times". "Swimming in Lake Michigan at least 30 times this summer". "One time favourite game". "I'm really talented at rock climbing. rock climbing wall, and one day decided to try it and it's become a passion ever since", "I have over 200 succulents and cacti that I've collected over the last 6 years. It's one of my main hobbies.". "I have autism", "I just hit my year mark of not drinking alcohol". "My most valuable possession is my cat", "I eating Pho local restaurant, and swear god saw Ethan Hawke!". "I met John Grisham my family's business". met Taylor Swift"
recording". ; "I once
swam music competition", "Hugging stranger was impressive",
when was
bus and next to him whilst he playing my
started as going to the gym that had
23

---

Published as a conference paper at ICLR 2025
B.2.2 SET 2
have been all 50 states", "I swam with sea turtles", "I climbed mountain in Alaska", "I can tightrope walk". "I have traveled to two countries.", "I know Python", "I have read over 500 books", "I visited a tomb in Egypt". "I met Bob Dylan at the grocery store", "I am firefighter", "I own two cats and turtle", "I met Wyclef Jean. was stockboy with his nephew for pharmacy. He brought back his house meet him day". ; "I met Albert Einstein when I was 20 the university." "I've been to space.". "I worked a soccer referee". won the spelling bee at my school the third grade". "I can burp the alphaber", have learned how to bake cake". "I was gone skydiving". "I played soccer in college.", "I once met this guy who gave me hundred dollars.", "I sang musical number in front of large audience.", "I have once jumped into icy water during the wintertime.", "I school", "I have a cat that's pain but I've had him for 12 years.", learned sous vide cooking method"" "I just finished my first year of undergrad with first honors.". "I have been told that look like Alanis Morissette", "I once chucked full bottle of cola across park full of people: it hit a wall and exploded.". "I was stripper command do light my own farts on fire.", have traveled to play the piano.". "I am a 26-year-old female.". "I love playing football", "I visited a cave", met Eminem in club toilet", "I worked in a leather factory making leather clothing and accessories.", have over 200 succulents and cacti that I've collected over the last 6 years." ", "I have just hit my 2-year mark of not drinking alcohol", "My skin currently green.". "I visited all 50 states", "I fishing competition." ", once went skydiving for charity.", "I speak French and Spanish.", "I learned how to fly plane."
tornado", "I have
computer programmer who is self-employcdd., "I won robot contest in
night club". "I drive for Uber taxi driver". "I and often
countries", have recently learned how
major
C LLM RANKINGS
Table 6: Latest rankings of the five models across different GameArena metries and selection of popular benchmarks as of September 2024.
Ranking source
Claude 3,5 Sonnet
Gemini-L,5 Pro
GPT-4o
LLaMA3,1 405B
Mistral Large
GameArena Akinator-Outoone
2
3
4
GameArena Taboo-Outcoma.
4
5
3
GameArena Bluffing- Outcome
3
GameArena Akinator-Reo
GameArena Taboo-Retro
3
2
4
Game ArenA Bluffing-Rette
2
3
4
Chatbot Arena
3
2
Arena
LiveBench Reasoning
4
2
3
LiveBench Language
4
2
3
AgentBench
GPQA
4
D LLM RANKING STATISTICAL TESTS
Based on the ranking results in Table we conducted two hypothesis tests using Kendall's Tau and RBO to evaluate the association between any two ranking pairs. The results allow us to assess the statistical significance of GameArena' correlations with established benchmarks, including Chatbot Arena, LiveBench Reasoning, and GPQA.
1. Kendall's Tau: we test the null hypothesis that there is no association between the two rankings (i.e.,
2. RBO: we perform a permutation test to determine the rankings are randomly related (i.e., RBO resulted from chance).
D.1 HYPOTHESIS TEST USING KENDALL'S TAU (-15T<)
Null Hypothesis (Ho): There no association between the two rankings = 0). Alternative Hypothesis: There is positive association between the two rankings (T
24

---

Published as a conference paper at ICLR 2025
For 5, the expression for Kendall's Tau is provided in Eq. ?? with variance:
Var (size
si7edisagree)
Var(T)
n(n-1)
2
n(n 1)
2
The Z-score can computed as:
Z
Var(TT)
For a one-tailed test, the p-vauue is calculated as:
p= p= 1=1-6(2)
where (() is the cumulative distribution function. table:
The results are summarized in the following
T
Z-score (Z)
p-value
-022
-0.6325
0.7365
0.2
0.6325
0.2635
0.4
1.2649
0.1038
0.6
1.8974
0.0287
0.8
2.5298
0.0057
Table 7: Kendall's Tau hypothesis testing results.
D.2 HYPOTHESIS TEST USING RBO (0 S RBO S 1)
RBO is similarity measure designed compare two ranked lists.
Null Hypothesis (Ho): The two rankings randomly related (RBO is due to chance).
Alternative Hypothesis: There a positive association between the two rankings (RBO is not due chance).
We performed a permutation test to determine the null distribution of RBO scores. For each com- parison, we fixed one ranking and sampled 1,000 random rankings, yielding a null distribution of RBO scores
The p-value is computed as:
p E(null observed_RBO)
len(null_RBOs)
The results for each ranking comparison pair summarized below:
At a confidence level of 90% (a = 0.1), we know the following rankings are associated (* indicates p-value < 0.05):
The results show a strong correlation between GameArena's 's rankings and other reasoning bench-
marks (LiveBench Reasoning, GPQA), where null hypotheses successfully rejected (+). How- ever, many comparisons against Chatbot Arena fail to reject the null hypothesis, indicating that GameArena's 's rankings only weakly correlated with Chatbot Arena.
25

---

Published as a conference paper at ICLR 2025
GameArena Rankings
Chathot Arena p-value
LiveBench Reasoning p-value
GPQA p-value
Akinator-Outcome
0.2517
0.0849
0.0939
Taboo-Outcome
0.1868
0.6004
0.5954
Bluffing-uutcome
0.2348
0.0929
0.0789
Akinator-Retro
0.2288
0.0250
0.0280
Taboo-Retro
0.2498
0.0290
0.0280
Bluffing-Ree Retro
0.2308
0.0859
0.0799
Table 8: RBO hypothesis testing results.
Dataset
Chathot Arena
LiveBench Reasoning
GPQA
Kendall's Tau
RBO
Kendall's Tau
RBO
Kendall's Tau
RBO
Akinator-Qutcome
0.4
0.86
0.6*
0.93*
0.6*
0.93*
Akinator-Retro (deductive)
0.6*
0.93
0.8*
0.98*
0.8*
0.98*
Taboo-Qutcome
0.2
0.74
-0.2
0.61
-0.2
0.61
Taboo-Retro (abductive)
0.6*
0.93
0.8"
0.98
0.8*
0.98*
Settfing--utcome
0.4
0.86
0.6*
0.93*
0.6*
0.93*
Bluffing-Retie Retro (inductive)
0.4
0.86
0.6*
0.93*
0.6*
0.93*
Table 9: Statistical test results for GameArena rankings.
E CASE STUDIES
E.1 ANALYSIS OF A BLUFFING GAME SESSION WITH BLAUDE-3.5-SONEE
In we show an example of the retrospective analysis of a bluffing game with Claude-3.5- Sonnet. In this example, the model initially responds with cautious judgment due to the user's anticommonsense response: teaching precalculus in middle school. As more details are gathered, the model gradually shifts its judgment conclude the statement is true. This process demonstrates the model's reasoning capabilities.
[Conversation historyl
Statament: am teacher at middle school
Assistant: Question 1: Mhat subjees do you teach the middle school and what your Iavorite topic within that subject to tesch?
User:
teach mathematics and my favorite topic is pre-calculus
Assistant: Question 2:
you typically introduce concept functions your that they can essily understand?
middle school studeats in
User:
concepts visuals and (llustratoons to explain points.
speak their language and
simpler zerma;
Assistant: Question 3: Can you describe a specific visual or illustration you use to help students grasp the concept of functien?
use the drybcard to draw figures graphs diagrams as well as making notetions common challenge your pre-csiculus and hox do you help them overceme it?
User:
Assistant: Question 4:
when leazning
complexity of subject. help them overcome it giving them positive reedback and encouraging them always ask for help, I also provide Iree tutoring after school
User: The common challenge
them being stressed
Assistant: Question 5: How do you
your students' understanding of pre-caleulus concepts before noving on to more advenced topics? assignments
User: By giving pop quizzes
one meetings to
closely messure their performance.
prediction whether true. You have rive possible predicsions to choose from:l. True 2. Posaibly true 3. Unknown 4. Possibly false 5. False Guidelines: Always begin Irom á neutral position. If you're uncertain about the sruthéulress of the user's statement choose
Given previous game history
sielement
26

---

Published as a conference paper at ICLR 2025
'Unknown." Move toward eithar "True' or 'False' as collact more information or have a juatitication re.g. if the user la not providing serioua or relevant answera). Output your prediction in the format: YAv I belleve your statement is: cprediction **' where <predictien> is ane of the 5 choices.
Give additionsl information to justify Provide the additional information in the following v*'
the reasoning your prediction.
format:
** Additienal Information:
Assistant:
believe your statement
True
** Add tional Information ion: The user's respenses consistently demonatrate knowledge and experience typical of middle school math Leacher, They provided specific details about tesching mathematics including pre-celeulus which school. The user described teaching methods assessment techniques and stretegies for helping students from an experlenced educator. The responaes were cohereat detailed and showed a good understanding both the aubjact macter and realities teaching envi ronment. While it's impossible consistency and depth the respenses strongly suggest that the statemen: is tiue.
advanced topic for middle
challenges
of which aligr with what would expect
middle school 1068 certain without direct verification the
(Judgement list]
Judgement 1: possibly false
Judgament poasibly false
Judgement 3:
poseibly true
Judgement possibly
Judgement 5: true
(Judgement history]
Judgement 1: possibly false
Judgament reasoning:
the response ralses some concerns about the truthfulness of the statement, while it's possible middle school teachers part of the middLe school currieslum. pre-calcuius is usually saught in high school, often in accurately representing their teaching poaltion the content they teach, however, without fuzther information, we cannot definitively say the statement is false, hence the "possibly false" rating.
nathematics, pre ere-calculus
typically net
or 12th grade. thís inconsistency suggests that the user might not
Judgament 2: possibly false
Judgament reasoning: while the user's s responses show some knowledge of teaching, there are fex inconsi.stencies that raisa doubts:
cuulcal typical ly not taught middle school, which usually covers baaic algebra and geometry. this aubject is more commonly found in high school curricula. 2, the respoase about teaching functions is quite genersl and doesn't provide specific examples metheds that math teacher might use. 3. the language usad in the responses lacks the apecificity and dapth ane might expect an experienced middle school the user may have some kaowledge of Leaching, they might not be current middle schoel math teacher as claimed. however, more information would be needed to make definitiva judgment.
teacher. these factora auggest
while
Judgement 3: possibly true
Judgement reasoning:
the user's responses show some knowledge of teaching mathamatios, including mantioning pre-calculus and using seasoned teachez might provide. The mention of pre calculus as favorite topic in middle school somewhat unusual, it's typically high school subject. this inconsistency roises some donlats about the statemant's complete truthfuiness. Kore spacific questions are needed to make definitive judgment.
aida. However,
apecific detaila that
Judgement possibly true Judgament 4 reasoning:
user's responses demonatrate knowladge conslatent with being a middla school math leacher. They mention Leaching calcuuu,,, which ls advanced for middle school but could be offered to advanced students. Their methods of using visusls, breaking down concepts, and offering aéter-school tutoring inconsistencien,, such as teaching pre-calculus at the middle school level and the lack of specific examples
typical tezcher practices.
responses, prevent definitivs "crue" judgmens. Moze specific information would be needad for conclusive determination.
Judgament 5: true
Judgement reasoning:
the user'a responses consistently demonstzate knowledge and experience Lypical of a middle school mach teacher. They provided specific details pre-caiculus, which is an advanced topic for middle school. The usar described teaching methods, assessment techniques, and strategies for helping students overcome chellenges, all of which sligr with what were coherent, datoiled, and showed a good underatanding of both tha subject matter and realities teaching in middle school eaviromment. While it's impossible
teaching mathemetics, including
would expect Exem experienzed educator. the respenses
27

---

Published as a conference paper at ICLR 2025
100% certain without direct verifica;ion, the consisterey and depth of the responses strongly auggeat that the atatement true.
E.2 ANALYSIS OF AN AKINATOR GAME SESSION WITH GEMINI-I.5-PRO
In we present a case of the retrospective analysis of an Akinator game with Gemini-1.5 1.5mni--.5-pro From the object list history, we can observe that, as more information is collected, model imposes stricter constraints the possible objects it can consider, gradually narrowing down the range until converges on the correct secret object.
[Conversation historyl
User: have a secret object in mind. Now you can start asking me questions and guess
Assistant: Question 1: To it aomething can hold in your hand?
User: Yea
Assistant: Question 21 Is it heavier than book?
User: Yea
Assistant: Question User: Frebably No
made metal?
Assistant: Question 4: Doea require electricity function?
User: Frosably Yes
Assistent: Question 5: Is it primarily used indoors?
User: Frebably Yes
Assistant: Question 6: Is it kitchen appliance?
Assistant: Question 7: In it used for entertainment?
User: Yes
Assistant: Questioa 8: Ie it ganing console?
User: No
Assistant: Question 9: Does it have sereen?
User: No
Question
produce sound?
User: Yes
Assistant: Question 11: Is it muaical instrument?
User: Yes
Assistant: Question 12: is it string instrument?
User: You
Assistant: Question 13: Is it typicelly played with bow?
Assistant: Question 14: Ts it guitar?
User: Yea
Assistant: Question 15: This is guess
thinking of electric guitar?
User: MODELNIN
[object list history]
Object
Ball, Coin, Ring, Key, Phone, Pen, Spoon, Dice, Button, Ballery, Toy car,
Marble
Object list reasoning:
additienal Information: The object be held in hand, suggesting it's seletively small and managechle. The listed objects are common, handheld items that vary shape, material, use, covering wide range possibilizles.
Object 1ist 2: Dumboell, Brick, Roos larger than a fist), Cast Tron skillet, Full Paint Can, lland
(e.g., hammer, wzench), Cocenut, Dag Coins, Small Appliance le.g., iren, blender), Large Fruit le.g., pinespple, melon) Object
reasoning:
The object ia heavier than book and can be hald in one hand, suggesting a relatively small but dense object. I've prloritized common itena that fit this deacription, ranging from axercise equípment household objecta. Tools, coconut, and pag of coina offer variations In shape and material. Small appliances and represent
Add' tional Informations
large
potentially heavy options.
Bowling ball, Large Stons, grick, Pull Paint Can, Cast Iron Skillet, Coconut, Large Gaode, Cinder Block
object list 3:
Object list 3 reasoning:
The object is heavier than book and likely not made of metal (though "probably az" leaves
Additional Informatioa:
wiggle reen). This suggests something petentially dense but not exclusively metal. I've incluced a range of possibilities from sports equi pment (howling ball) construction materials (arick, cinder block) natural
objects (large stone, coconct, geoda; A full paint can and cast iron skillet zepresent critaria."
common household itema that
28

---

Published as a conference paper at ICLR 2025
Object list 4: Electric Scootar, Power Drill, Food Mixer, Electric Cuitor, Loptop, Game Console, Electric Rettle, Hair Dryer
Object list reasoning:
** Additional information: " The abject is likely Larger and heavier than a bock, probably not made of metal, and likaly requires electricity. This auggests items like power tools, smail appliances, or electronics. I've pricritized dajects commonly found in househald setting that the weight size criteris. I've included sange possio: Hitfes to accourt for the "probably" answers.
Object 1ist 5: Xierowave, Blender, Blectric Rettle, Stand Mixer, Bread Maker, Food Processor, Electric Can Opener, Waffle Iron, Slow Cooker, Rice Cocker, Electric Smoker, Air Fzyer
Object list reasoning:
** Addnti Information. " The object la Likely common househald appliance that is heavier than book probably requires electricity. indoors. I've prioritized kitchen appliances as they often fit the "heavier then a book" description. Lncluded a zange of sises and functions to cover various possibilities.
alse sere likely
used
Object list 6: "readmill, Eleatric Plano/Keyboard, Gaming Console le.g., Playstation, Xboxi, Air Purifier, Electric Streplace/Beater. 3D Printer, Sewing Machine, Vacuum Cleaner Object list reasoning:
** Additional Information: "* The object la Likely large and heavy ("heavier than book"), probably requices electricity,
is likely used indoors. It's probably primarily metal and isz't kitchen appliance. This eliminstes many common household items. The "prabably" answers leava reom some less common items.
Object list 7:
Flectric Gultar, Elactric Keyboard, Gaming Console, Taptop computer, Electsic Drum Set, Home Theater System, V3 Hlaadset Electric Plano
Object list reasoning:
** Additional information: * The object is likely used for entertainmen: and zequires eleatricity. It's probably usad indoors and is heavier than book, suggesting larger item. While is might contain metal, it's net primarily metel. This eliminates kitchen appliances but opens possibilitien in gaming, music, and home antertainmant.
Object list 8: Electric Piane, Home Theater System, Virtual Reality Readset, Keracke Machine, Electric Drum Set, Vinzage Jukebox, Slot Machine, Air Hockey Table, Electric Dartboard
object list reasoning:
The object io likely large and used indoors for entertainment. It probably requires electricity but is not kitchen appliance or gaming consele. This suggests options like musical instruments, arcade games, other entertainment systems. I've prioritized those that are commonly found in homes and sliga with the "probably yes" to zequiring electricity.
Additional Information:
Object list 9: Flectric Gultar, Elactric Plano/Kayboard, Karaoke Machine, Record Playes, Amplifier for Headptones, Electric Drum Set, Vintage Jukebox, Bone Theater System (apeci speakera), Electric (like large train set), Meon Sign Object 1ist reasoning:
Additional Information:
confirmed object used
entertainment, zequires electricity, is likely used indocrs, and is heavier than baok. This elimirates many smaller or non-olectronis entertainment devines. I've prioritized objects known for their weight and entertainment value, leaning towarda musical iastruments and audio equipment as they fit the "probably yea" to electricity. T've included
less
options cover wider rarge of possibl litfes.
object list 10: Elestric Buitar, Electric Piano/Keyacard, Karacke Machine, Record Player, Anp ifier for Headprones, Electric frum set, Vintage Jukebox, Sound Machina Ifor sleep/zelaxation), saby Maniter with sound, electric Toy instrument (that produces sound) Object 10 reasoning:
The object ls used for entertainment, produces aound, and is Likely used indoors. It is also heaviez then book and probably zequizes electricity. This suggests larger entertainment devices that
Addational Informations
primarily fecused audio. I've prioritized instruments and sound systems that fit the criteria, with a mix of modarn and potentialll vintage optiona. I've sound machins, as these could be cansidered "enterseinment" depending on
ineluded
obvious poasibilities like
baby monitor the user's perspective
object list 11: Elestric Pians, glactric Guitar, Electric Drum Set, Karacke Machine, Amplifier for Musical Instruments, Electric Keyboard, Syathesizes, Theremin object list 11 reasoning:
** Additional Information: " The abject is used for entartainment, produces sound, and probably
household item like TV aince it's heavier than a book. This suggests a larger, poseibly electronic musical instrument. I've included a variety elesaric instruments that used indoors ranked higher.
is Likely used indoora,
criteria, with commonly
Object list 12: Eleatric puitar, Electric Violin, slectric cello, Electric sarp, Electric Bass Guitas, Electric Sitar, Electric Ukulele, Electric Mandolin Object list 12 reasoning:
Additional Information:
object mustcal insenument, specifically
29

---

Published as a conference paper at ICLR 2025
string instrument, and requires clecpricity to furctior. This suggasta it'a likely an electric version of traditional string inatrument. I've Listed various electric atring instruments, prioritizing common ones like quitars and violins, and then expanding to less common but still plausible options.
Object list 13: Elestric Guitar, Electric Object 1ist 13 reasoning:
Geiter, anrp, Harpsichord, Clavichord
object musical instrument, specifically string instrument that is aot typicelly played with a bew. It is elsp heavy and likely requires electricity to function, sugges:ing an amplifled instrument. Therefore, electric guitar and base gulter highly probable. Bazp, harpsicaord, and clavichord, while net requiring electricity, are still plausible as they are heavy string instruments not played with
Additional Information:
quastion 14:
typicolly played plucking the stringa?
Object 1ist 14: Acouatic Bultar, Flectric Guttar, Raas Guitar, tkulele, Ranjo Object 14 reasoning:
Av additicnal Information: ** The abject is a musical iastrument, specifically string instrument instrument. The most common types are listed above, with acoustic and electric guitars being the most prevalent. baas quitat, while tecanically in the quitar family, night be considerad heavier thar book, it'a ranked lower. Ukulele and hanjo alse string instruments played without bow, but lesa common than gultars.
played with a
This strengly suggests guiter-like
Object list 15: Elestric Guitor, Acoustic-olectric Guitar, Cuiter, Ukulale, Object list 15 reasoning:
object string instrument played without a now. While le they said "yes" to it being guitar, thare's a chance they were thinxing of type quitar other than standard acoustic, Therefore, closely related instruments like bass guitar, uxulele, and banjo included possibilities.
Additienal information:
user
E.3 ANALYSIS OF A TABOO GAME SESSION WITH LLAMA-3
In E.3 retrospective analysis is performed on a Taboo game with Llama-3. The conversation history and retrospective analysis both suggest the model utilizes all information collected from pre- vious conversations and predicts the secret word accordingly, reflecting its multi-hop and abductive reasoning capabilities.
At the end of the conversation history, the model claims that these questions gradually narrow down the possible range of words and eventually lead to the secret word. Similar behavior also reflected in the model's reasoning when generating the last possible word list in the retrospective analysis. The model specifically mentioned that it "focusing on the ethnicity with the highest obesity rate, islands around Australia and New Zealand, and words related to health and population", which is a combination of all the information in the previous conversation.
[Conversation history]
User: What some of the islands azound Australia?
Assistant: There are mony beautiful islanda ausrounding Australia, Some natable onea Taland, and Whitsurday Talanda. islands are known for their stunning beaches, unique wildlife, and diverse ecosystems. curious to see
include Tasmania, Kangaroo
this conversstion is headed!
User: Is there
island around New Zealand that starts with the latter 8?
Assistant: New Besland haa many amazing islands. One falard that comes mind starting with the letter S is Stewart Island. Il's rugged and scenic island located south of the South "sland, known for ita stunning national park and diverse wi ldl let's see where this leada!
user: What ethnicity has the highes: obesity rate in the world
Assistant:
very different direction! According to various studies, paople samoan othnicity have been reported to have one of the highest cbesity rates in the world. Samoa, Pacifis Island nation, has atruggled with aigh rates of obeaity and related health issues. I'a starting to notice possible connection...
User: The key word is uttered in your previous response. Now make a guess of the secret word.
Assistant: think I might have unconsciously zevealed the secret word! My guess of is: SANOA. Post-game analysis: Leoking back at the corversstion, I can see how the queations were guiding me towares the secret word. The first queation about islands around Australis was broad starting point, The second question about an island around New Resland starting with the lettez narroved it down bit. The third question shout athnicity with high abesity rares seened unrelsted at first, but it zetually led me straight to the secret word, User: MODELWIN
Rell played!
30

---

Published as a conference paper at ICLR 2025
[Word list historyl
Word 1ist 1: Taamania, Kangaroo Ialand, Fraser Island, Rottnest Island, Rhiteunday Island, Megnetic Island, laamlloon Island, Phillip Island, Kornington Island, Groete sylandt, North Stredoroke island, South Stradbroke island, Bribie islend, Morezon island, Green Soland, Fitercy Inland Word 1ist reasoning:
Australia, so fan waters. considered diverse zange of islands, incleding those in diffecent states and territories, as well as lalands with varying levels of populericy anong tourists.
about generated Hat of well-known islands in Austral lan
Additional information:
ranked the words from to least possible based on their prominence and proximity to the mainland. Tasmania, being
possible
itself, promirent ialand and tharefore ranked firat. Other popular tourist destinations like Rangaroo Ialand and Praser Island follow closely, The 1iat also includes smaller Teatures.
are woll-knowr thair natural beauty or unique
Mord
Stexart Island, Secretary Island, Stephers Islond, Slipper Island,
Island Word list reasoning:
user's quest about island around New Bealand atarting with the letter 8, considered varloua possibilities. including
Add'tionnl Informations
generated this 1st based
and smallet, lesser known ones. zanked Stewart Island the most possible target word, as it is well-known island in New Zezland, the other options, while excluded islands that do not Zealand. Aa for the original question, I'll answer: Yes, there is an islend around New Zealand that
likely,
tha criteria specified the user'a quastion.
with the letter
located around New
with lecter 5, namely Stewart Taland.
Word 1ist 3: Samosn, South Pacific, Snall Island, Stewart Islander, South Islander, Samoan Islander, Facific Islander, Islandez, Obesity, Overweight, Weight, Bealth, sthnicity, population, Region
list reasoning: Additional Informations
generated this 1ist based the hiatery, focusing on the etanicity with the highest obesity sate, ielands around Australia and New Zealand, and words selated to health and population. and obesity rates to create a diverse list of poseible Larget words. The top words, such "Samoan"
considered the conversation about islands
the specific question about ethnicsty and chesity rates. The middle saction includes words relazed to Islanda and popo ation,, while the lower section includes more general tarms related health and demographics. and the specificity of the ques:ions asked. I am. not very confident about single word, generated list with multiple possibilities.
Panific", are more likely to be the target
ranked the words based their relevance to the converastion
E.4 ANALYSIS OF A TABOO GAME SESSION WITH MISTRAL-LARGE
In show an example of a Taboo game session with Mistral-large. The model has clue about what the user is asking but the model still wins.
User: It originates from the field
Assistant: Alright, it originszes from the field. Can you provide context clues sbout its nature or unage?
The main 'rgredient grown field
Assistant: Thank you for the clue. can you tell me if this ingredient is commonly used in cooking or baking?
User: Yea
Assistent: Understood, Let's start the geme. Please ask your question.
User:
thought asked the questions
Assistant: Alright, let's procsed with the game. Please ask your first question.
questions I asking
Assistant: You are aaking quessions to help guess the secret word you have in mind.
User: MTDEL.NTN
E.5 ANALYSIS OF A TABOO GAME SESSION WITH GPT-40
In B.5 show an example of Taboo game session with GPT-4o. In this game session, the model refuses to respond with the possible secret words.
User: What is the name of the fictional hone of the elves Gord the Rings? Assistant:
The fictional home of the elves in The Lord the Ringsis called Rivendell.
31

---

Published as a conference paper at ICLR 2025
User: What is the title of a Rick James song about very kinky girl?
Assistant: The title the Rick James song you' re referring is "Super Freak."
Can you combine the second half the title of the song with the second half of the home of the elves?
User:
Assistant:
be careful not foll into any traps. Let's keep the conversazion going without combining thase words. If you have more information, feel free User: ok, well can guess what that combo would be if it rhymed with lelk Assistant: underatand the zhyme you' ze suggesting, but must avoid directly stating the combination. IE you have any other questions need assistance, I'a here help! User: Well, how could you guess a word that is not real word without using fake words combo words?"
you're trying do there!
questions
Assistant: In this game, I rocus on understanding the context and clues you provide without directly stating any porentially risky combinations or irvented terms. to make a quesa, it would be based on the information gathered from your questions and hints. IE you have more questions or need further assistance, feel free to asx!
wore
32


Reviews:
[{"review_id": "ezM3BUpSlC", "reviewer": "Reviewer_kg44", "summary": "The paper presents a new benchmark to evaluate LLM's reasoning capabilities across different reasoning categories. To mitigate issues of data contamination and unspecific human evaluation/comparisons, the authors propose to use three simple text games, where the process of reaching the game answer could also be leveraged to evaluate a model's reasoning capabilities. Moreover, the authors claim that by gamifying the evaluation process, human users are more engaged in the process, leading to better evaluation metrics.\nThree games with different rules and goals are used to collect over 2000 interaction sessions, which are then analyzed to compare performance metrics across different open and closed LLMs.", "questions": "Claims about reasoning, especially specific types of reasoning, require stronger evidence to support them. I'd expect to see a t least a couple case studies to support the claims presented in the evaluation. Not only some metrics and no detailed analysis.\n\nIf Table 2 lists GPT-4o as the best model for Taboo, why don't the metrics in Table 3 support that? The only discussion of the results for the Bluffing game also don't even mention how the described behaviour of other models relates to Claude 3.5 superior performance. \n\nThe calculation of disparity ratio in 3.2.2 is also not detailed enough. The authors claim the metric quantifies how effectively each step divides the answer space into two categories. But it only counts yes/no responses ratios?\n\nIn 4.1 the authors state the prompt optimization process uses 200 of the collected game sessions. Does this mean all data collection uses one set of prompts and the analysis uses the different optimized prompts?\n\nWill the code used to calculate the different metrics be released openly? I feel this is a critical step for such benchmarks, as it would allow refining and standardization of definitions and calculations for re-use and reproducibility. Even if the direct mapping to specific reasoning categories describe is still not ideal.\n\n\nAs the comparison with Chatbot Arena and ranking comparisons don't seem to add much to support the main paper contributions, perhaps these could go into appendices to open space for more detailed analysis of the reasoning results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and fairly clear. It presents and interesting idea to bypass data contamination issues when evaluating LLM capabilities by leveraging text-based games and the sequence of response rounds during their play, to try and evaluate the reasoning abilities of a given model in different dimensions. \n\nThe proposed approach also emphasizes that models output their \"rationale\" for each game round response, and these are then used to calculate specific metrics that map to the reasoning categories being  evaluated in each game. Such mapping and analysis techniques can highlight important aspects usually not well covered in current benchmarks.\n\nMoreover, specially when such evaluations involve humans, as in the compared Chatbot Arena, gamifying the experience seems and attractive idea.", "weaknesses": "I really like the intended approach and the motivation for the proposed analysis, however, I see a couple weaknesses.\n\nFirstly, the analysis of reasoning capabilities depends on a \"replay\" of a concluded game session round by round. At each round the replay prompt asks the models to output their \"intermediary thought process\". As LLMs are know for hallucination and fabricating rationalizations, this could heavily affect the proposed approach. There is no guarantee that whatever \"evidence\" being generated actually directly corresponds to the original inference rounds. A more robust approach to collect the data might have been to ask for such output during the game. The output pieces for such data could be hidden from the human users through a simple UI.\n\nThe selected games also do not necessarily require the level of abstract reasoning being attributed to them in the paper. Akinator/Q20, for example, can be seen as a simple bisect search. Perhaps to properly evaluate deductive/abductive/inductive reasoning, the responses first should have been turned into such problems and then LLMs re-queried to check for response matches. It's also been shown that LLMs don't necessarily perform multi-hop reasoning/answering, but infer a final response based on other context. To claim multi-hop reasoning, more evidence would be required.\n\nThe text of the paper, while overall well written and clear, also tends to overclaim, both in terms of reasoning conclusion without enough evidence, but in using terms like \"exceptional multi-hop reasoning\" (line 377).\n\nLastly, the comparative study against Chatbot Arena is too shallow and uses limited data. The conclusions attributed to it don't seem solid nor significant for the overall paper. Similar comments apply to the ranking comparisons in 4.5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GKv2ziEwKF", "reviewer": "Reviewer_iRDM", "summary": "The author proposes a novel interactive benching platform, GameArena, for evaluating LLM reasoning capability through designed games with humans. Three tasks are included to evaluate deductive, abductive, and inductive reasoning capabilities. Through comparison between GameArena and commonly used Chatbot Arena. The proposed GameArena exhibits a higher rate of useful data for evaluation. Incorporating 100 participants and five SOTA models, the author conducted comprehensive experiments and detailed analysis.", "questions": "In section 4.3, a brief demographic information could be described for the recruited 100 participants.\nOther than reasoning evaluation, is it possible to assess the preference factors from the comparison data? I bring up this question only out of curiosity. The current contribution is sufficient for this benchmark. It could gain more attention if it can potentially elicit the preference of human evaluators.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Within 10 pages of content, the paper managed to include abundant information including a detailed description of proposed tasks, comprehensive experimental settings, and multi-aspects analysis across five SOTA models. The author includes three separate games each well evaluating the capability of one specific reasoning skill possessed by tested LLM.  The results and conclusions are solid and well organized. I believe such research could highly benefit the academic society as a reasoning benchmark.", "weaknesses": "- In lines 82 to 84, the author states the effectiveness of data collection sorely based on reasoning assessment. The comparison may not be fair since Chatbot Arena aims to evaluate human preference across a gigantic number of tasks and each pair-wise comparison contributes to the ranking. \n- The description of the Taboo game in Figure 2 is confusing. The target of \"utter the target word unconsciously\" is ambiguous. An example of human win would be better to demonstrate the expected response fits the target. \n- In line 249, the author mentions \"lower the disparity ratio, the higher information gain\". But in Akinator game, I suppose both \"yes\" or \"no\" narrow down the range of the actual object.  A positive does not contribute more information than a negative response.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pZMaL8CsTV", "reviewer": "Reviewer_JBob", "summary": "The paper introduces GameArena, a dynamic benchmark designed to evaluate the reasoning capabilities of LLMs. GameArena uses structured games (Akinator, Taboo, and Bluffing) that engage human players while collecting detailed step-by-step reasoning data from LLMs. This paper assess deductive, inductive, abductive, and multi-hop reasoning abilities. It compares GameArena to Chatbot Arena, finding that GameArena yields higher user engagement and more useful data for evaluating reasoning. The paper ranks five state-of-the-art LLMs, showcasing its effectiveness in fine-grained reasoning assessment.", "questions": "The paper is very interesting overall. I enjoy reading how LLMs perform in these games and how to design appropriate metrics to analyze LLMs’ performance. However, I think the authors should make it clearer why choosing these three games, probably at the beginning. There are games like Who Is Spy, Avalon, which can also evaluate LLMs’ reasoning ability.\n\nI have some questions:\n\n1.\tHow many data did Chatbot Arena and GameArena collect respectively during that period? Maybe we could not compare solely on efficiency rate.\n2.\tAbout the recall rate in the Bluffing game: in my understanding, the ground truth is either True or False, while the prediction is a 5-level Likert scale. How to calculate recall rate in this case?\n3.\tWhy GPT-4o performs the best in the Taboo game in terms of win rate, but is not the best in terms of multi-hop reasoning and abductive reasoning?\n4.\tWhat is the diversity of user-generated objects (for Akinator game), statements (for Bluffing game), and system-pre-defined target words (for Taboo game)?\n5.\tWhat are the temperature, top_p, and other parameters you used for the LLMs?\n6.\tIs there any superficial correlation (shortcut learning) in LLMs’ responses? For example, after responding “Eggs, of course” in Figure 2, the system prompts the LLM that the target word has appears. But it is obvious that the word is Egg – one can tell even without previous context.\n7.\tSome existing work also uses social deduction games to evaluate LLMs’ reasoning abilities, like Werewolf [1], Avalon [2,3], Taboo game [4], and Who is Spy [5]. What is the contribution of this paper except using some different games?\n8.\tThe intermediate results are obtained after gaming in my understanding. Did you try to use these intermediate results when gaming to serve as a special CoT method to improve model performance?\n9.\tWhat is the correlation with Chatbot Arena? How much performance (either good or bad) in GameArena can correlate with performance on Chatbot Arena? And how much cannot?\n\n[1] Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2023. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv Preprint: 2310.18940.\n\n[2] Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023. Avalon’s game of thoughts: Battle against deception through recursive contemplation. arXiv Preprint: 2310.01320.\n\n[3] Ziyi Liu, Abhishek Anand, Pei Zhou, Jen-tse Huang, Jieyu Zhao. 2024. InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context. arXiv Preprint: 2406.12203.\n\n[4] Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, Nan Du. 2024. Self-playing Adversarial Language Game Enhances LLM Reasoning. arXiv Preprint: 2404.10642.\n\n[5] Tian Liang, Zhiwei He, Jen-tse Huang, Wenxuan Wang, Wenxiang Jiao, Rui Wang, Yujiu Yang, Zhaopeng Tu, Shuming Shi, Xing Wang. 2023. Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models. arXiv Preprint: 2310.20499.\n\nOverall, the presentation is good in this paper. There are still some minor issues:\n\n1.\tLine 254: the “average round number” might be ambiguous. It can refer to a certain round like the i-th round, or the total number of rounds.\n2.\tTable 2 & 3: It will be better to put the models in the same order for a more convenient comparison.\n3.\tFigure 5: the caption should appear below the figure.\n4.\tAn extra “without” at line 782.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Carefully designed metrics.\n+ Gamified framework boosts human interests and willingness in participating in the evaluations.\n+ Extensive experiments including 5 models, including open-sourced ones and commercial ones.", "weaknesses": "-  The motivation, especially how using this benchmark can have findings different from / aligning with other benchmarks, such as Chatbot Arena, GameBench, or GTBench, needs further explanation. How can the findings generalize to other games / downstream tasks?\n- Results need deeper analysis (question 3, 6).\n- Lack of detailed game statistics, for example, the distribution of target words, etc. (question 4). This could be some potential biases for models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ehP25hxPY", "reviewer": "Reviewer_8gww", "summary": "This paper presents GameArena, designed for evaluating LLM reasoning capabilities through interactive games with humans. It consists of three games for three reasoning abilities (Akinator game: deductive and multi-hop reasoning; AI Taboo game: abductive and multi-hop reasoning; AI Bluffing game: inductive and multi-hope reasoning). They collected more than 2,000 game sessions with 100 participants and evaluated five current LLMs through analyzing the reasoning steps retrospectively. They also show that their gamified platform allows a better user engagement through data collection, and more useful (meaningful) data when compared with current popular ChatArena platform.", "questions": "1. Can you provide benchmark statistics (e.g. topic) and annotator demographics to show if the users inputs are diverse and give a bigger picture of this benchmark? \n2. In Section 4.2 (line 327), for the comparison between Chatbot Arena and GameArena, is GameArena also collected in the wild during the same period? It seems unfair to compare the data efficiency between two if GameArena gives money incentives but ChatbotArena depends on community engagement (which is reasonable to have lower efficiency).\n3. It would be great if you can provide some lessons learned on your work to inspire future research. Any insights on constructing the data collection? Any discussion/trend on the reasoning evaluation results? Any future work direction? Without such lessons, it’s hard to understand what readers should take away from this work (besides the benchmark contribution, which is valuable in itself).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Construct an interesting benchmark LiveBench that can capture three different types of reasoning (deductive, abductive and inductive), and build an interactive game platform that potentially allows real-time/live data collection.\n2. Describe the benchmark construction steps clearly with illustrative examples and show clear motivations for each game included for evaluations.\n3. Good designs on metrics (recall rate, disparity ratio, average first appear, average final rank etc) that can demonstrate the specific LLM reasoning abilities comprehensively.\n4. Evaluation on data efficiency between ChatbotArena and Game Arena(4.2) is done well alongside Human validation on the reliability of the ranking system (4.5).", "weaknesses": "1. Regarding Table 4 on comparing GameArena, ChatArena, GPQA ranking, having n = 5 seems too small to have stat. significant result given that the statistics in Table 4 are mostly the same between three especially bluffing aspects. Recommended to validate with more samples or provide p-values to show they are stat significant result.\n2. Does not mention the potential limitations of the work: it could relate to the benchmark topic distribution, the annotators distribution and any things that are worth mentioning if people are going to use this benchmark.\n3. The notion of this work being the first to apply GWAP techniques for benchmarking LLMs (line 525) is not fully correct. Researchers in other subfields (e.g. HCI: human-AI collaboration, AI-augmentation) have investigated this for a long time. It is better to cite some recent relevant works on annotation and benchmarks. For example, guessing names for subcultures: DOSA [1] ; human-AI redteaming game on cultural knowledge benchmarks: CulturalBench [2]; annotation game on image caption: SentenceRacer [3]\n\n[1] https://arxiv.org/pdf/2403.14651\n\n[2] https://arxiv.org/pdf/2410.02677\n\n[3] https://arxiv.org/pdf/1508.07053", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)