Are Large Vision Language Models Good Game Players?
Authors: Xinyu Wang
Bohan Zhuang
Qi Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Large  Vision Language Models
LLMs
Evaluation
Game
Primary area: datasets and benchmarks

Abstract: Large Vision Language Models (LVLMs) have demonstrated remarkable abilities in understanding and reasoning about both visual and textual information. However, existing evaluation methods for LVLMs, primarily based on benchmarks like Visual Question Answering and image captioning, often fail to capture the full scope of LVLMs' capabilities. These benchmarks are limited by issues such as inadequate assessment of detailed visual perception, data contamination, and a lack of focus on multi-turn reasoning. To address these challenges, we propose LVLM-Playground, a game-based evaluation framework designed to provide a comprehensive assessment of LVLMs' cognitive and reasoning skills in structured environments. LVLM-Playground uses a set of games to evaluate LVLMs on four core tasks: Perceiving, Question Answering, Rule Following, and End-to-End Playing, with each target task designed to assess specific abilities, including visual perception, reasoning, decision-making, etc. Based on this framework, we conduct extensive experiments that explore the limitations of current LVLMs, such as handling long structured outputs and perceiving detailed and dense elements. Code and data are publicly available at https://github.com/xinke-wang/LVLM-Playground.

Published as a conference paper at ICLR 2025
ARE LARGE VISION LANGUAGE MODELS GOOD GAME PLAYERS?
Xinyu Wang', Bohan Zhuang2, Qi Wu' 'The University of Adelaide, Australia *Zhejiang University, China
qi. si.ww0lJeadelaidee edu
ABSTRACT
Large Vision Language Models (LVLMs) have demonstrated remarkable abili- ties in understanding and reasoning about both visual and textual information. However, existing evaluation methods for LVLMs, primarily based on bench- marks like Visual Question Answering and image captioning, often fail to cap- ture the full scope of LVLMs' capabilities. These benchmarks are limited by issues such as inadequate assessment of detailed visual perception, data contami- nation, and a lack of focus on multi-turn reasoning. To address these challenges, we propose LVLM-PIN Lly-llaygoound, a game-based evaluation framework designed to provide comprehensive assessment of LVLMs' cognitive and reasoning skills in structured environments. LVLM-
uses a set of games to evaluate LVLMs on four tasks: Perceiving, Question Answering, Rule Following, and End-to-End Playing, with each target task designed to assess specific abili- ties, including visual perception, reasoning, decision-making etc. Based this framework, we conduct extensive experiments that explore the limitations of cur- rent LVLMs, such as handling long structured outputs and perceiving detailed and dense elements. Code and data are publicly available at https://github.comm www.ctccc//iiiiiioo con/xinke
INTRODUCTION
Large Vision Language Models (LVLMs) (Liu al., 2024a; Bai al., 2023; Zhu al., 2024) have recently demonstrated remarkable capabilities in processing and generating visual and linguistic in- formation. These models extend the strengths of Large Language Models (LLMs) by incorporating the ability to understand and reason about visual data alongside textual inputs, exhibiting human- reasoning capabilities across variety of complex tasks that demand strong reasoning skills.
However, the current evaluation methods for LVLMs vary significantly across different studies, lack- ing a unified criterion. Most existing LVLM works are assessed using multimodal data established before the LVLM era, such as Visual Question Answering (VQA) (Antol et al., 2015), image cap- tioning (Chen et al., 2015), Optical Character Recognition (OCR) (Fu et al., 2024), etc. While these tasks have been instrumental in assessing model performance, they present several limitations:
Inadequate Assessment of Detail Perception. Existing evaluation tasks, such as generic VQA and image captioning, often fail effectively assess models' ability perceive and understand fine-grained visual details. For example, in some VQA datasets, questions can be correctly an- swered without referencing the accompanying images, relying solely text-based world knowl- edge (Chen et al., 2024a). This is because some of these questions are not even genuine visual dependent, and thus they might not fully evaluate the models' visual perception abilities.
Risk of Data Contamination. The extensive training of LVLMs vast amounts of data from diverse sources increases the risk of data contamination, where training data includes some of the crestion-answer pairs used for testing (Dong al., 2024; Zhang et al., 2024; Wei et al., 2023). Such overlap can lead to overestimated performance on tasks like VQA, as models may recall answers from their training data rather than genuinely understanding and reasoning about inputs.

---

Published as a conference paper at ICLR 2025
Limitations of Metrics. Existing metrics for evaluating LVLMs generative tasks such as CIDEr and CLIP Score, heavily rely on short, predefined captions. These metrics are not de- signed to evaluate the rich and varied outputs that LVLMs can produce, especially for open-ended questions without unique correct answers. This limitation forces many LVLM methods to prompt the models generate short answers to fit these metrics (Li et al., 2023), which can restrict the models' ability to showcase their full capabilities.
Inconsistent Prompts. LVLMs often employ specific prompts during training and inference, including zero-shot, few-shot, and Chain-of-Thought (CoT) prompting. These variations can sig- nificantly impact performance (Wei et al., 2022; Li et al., 2024b), making it difficult to attribute differences the models themselves rather than the prompts used.
Neglect of Multi-turn/ Lunti-turn/Long-coneeet Reasoning. Current evaluation benchmarks focus predom- inantly single-turn interactions, overlooking the importance of multi-turn reasoning and long- context understanding in LVLMs (Liu et al., 2024b). While real-world applications often require engaging in extended dialogues and maintaining context over multiple exchanges, these capabili-- ties are not adequately assessed by existing benchmarks.
Building upon the limitations of current benchmarks, there is a need for more comprehensive eval- uation frameworks that assess the full scope of LVLMs' capabilities. Games, with their structured environments, offer promising solution. They inherently require not only a detailed perception of dynamic game states but also the ability to formulate strategies, anticipate an opponent's moves, and adapt to new scenarios over multiple turns. This combination of visual understanding, long-term planning, and decision-making under constraints makes games particularly well-suited for evalu- ating LVLMs across wide range of cognitive and reasoning tasks. Given that even simple AI algorithms, such as Minimax and Monte Carlo Tree Search, can master games through optimiza- tion (Campbell et al., 2002), it raises the question: Can LVLMs generalize their advanced reasoning and perception skills to perform competitively in these structured, logic-baser environments?
To address this question, we propose game game-based evaluation framework LVLM-Playground to sess LVLMs' abilities in structured environments thoroughly. Unlike traditional benchmarks, games provide controlled yet dynamic setting that naturally tests perception, reasoning, decision-making and competing abilities. This framework directly addresses the key limitations of existing evalua- tions, outlined below:
Detailed Perception. Board games like Chess and Go require a precise perception of game states. Models must accurately interpret the positions and identities of pieces, processing Ane-grained visual details essential for strategic decision- making..
Unique Data. Game data is largely absent from current LVLM training sets, reducing the risk of contamination. Additionally, diverse environments and varying UI designs in games offer wide range of novel, procedurally generated scenarios.
Transparent Metrics. Games have clear, rule-based outcomes, such as winning or losing, pro- viding objective metrics for performance evaluation and eliminating the ambiguity often found in current benchmarks that rely on human-annotate answers.
Consistent Prompts. In games, the rules can serve as uniform prompt, clearly defining what the model should do. This ensures models are evaluated on level playing field, reducing variability and leading to more consistent, reproducible assessments.
Multi-turn Reasoning, Games naturally require multi-turn interactions and long-term strategic thinking. To perform well, a model must maintain context over several moves, plan ahead, adapt dynamically to changing situations, and anticipate the opponent's 's moves, making games an ideal testbed for assessing sustained reasoning over extended periods.
In summary, games offer an alternative to existing benchmarks by providing a structured, dy- namic, and comprehensive environment for evaluating LVLMs. In this paper, we propose LVLM- Playground, game-based evaluation framework that includes six unique games: Tic-Tac- Toe, Reversi, Minesweeper, Gomoku, Sudoku, and Chess. The contributions of this work are as follows:
We built LVL Plaa-gud from scratch, a comprehensive benchmark that integrates game UIs and AI opponents, enabling both online and offline interactions between LLMSs and games. The
2

---

Published as a conference paper at ICLR 2025
framework supports common interfaces for both commercial models, such as OpenAl API, and spen-source models, like those from the HuggingFace Transformers library. AVLM-Playgron includes multiple tasks with various settings, as well as automated evaluation mechanisms.
We systematically designed a comprehensive framework to quantify the abilities required for each game and task, which includes detailed metrics that assess performance across perception, reason- ing, decision-making, and ad adversary skills. Based on this, evaluated state-of-the-arr LVLMs, including both open-source models and commercial APIs, and generated detailed reports that highlight their strength and weaknesses under different gamplay settings.
Based the experimental results, we conducted an in-depth quantitative and qualitative analysis, uncovering key findings such as looping behavior in long structured outputs and poor performance in dense visual perception tasks.
RELATED WORK
2.1 EVALUATION FOR LVLMS
Evaluating LVLMs has proven challenging due to the need for benchmarks that comprehensively assess both language and visual modalities. Early evaluations (Liu et al., 2024a; Li et al., 2023; Alayrac et al., 2022) often focused on limited set of vision-language tasks, with benchmarks like VQAv2 (Goyal et al., 2017). VizWiz (Bigham et al., 2010), ScienceQA (Lu et al., 2022), and Text- based VQA (Singh al., 2019; Wang al., 2020) being the most commonly used. Other tasks, such as image captioning (Agrawal et al., 2019; Sharma et al., 2018) and image-text retrieval (Plummer al., 2015), were also employed evaluate -isually-conditioned language generation. However, without a unified evaluation framework, results were often reported on disparate datasets, compli- cating direct comparisons. In response, later works (Bai et al., 2023; Ye et al., 2024; Li et al., 2024a; Wang et al., 2024c) followed more consistent evaluation practices, relying on few widely adopted datasets and employing radar charts to visualize model performance across tasks. This approach provided clearer picture of each model's strengths and weaknesses, though they remained largely focused on VQA and related tasks. These benchmarks themselves were primarily designed for ear- lier classification- based VQA models, resulting in ground truths that are often short, lack diversity, and do not fully capture the complexities required for evaluating modern LVLMs
To address the limitations of earlier benchmarks, recent works have developed evaluation frame- works tailored for LVLMs to capture a broader range of multimodal understanding (Wu et al., 2024; Wang et al., 2024b; Lin et al., 2024; Ji et al., 2024). For example. ALMT-Bench (Ying et al., 2024) introduces 32 core tasks with 162 sub-tasks, covering areas like 3D perception and anomaly de- tection. Similarly, iit (Duan et al., 2024a) evaluates over 70 LVLMS across more than 20 benchmarks, unifying model comparisons. However, despite these expansions, many tasks still fall under VQA and its variants, limiting the diversity of benchmarked capabilities. Consequently, multiple benchmarks are often combined to assess a broader range of abilities, which increases the evaluation burden due the large number of datasets and tasks. In comparison, the proposed LVLM- Playground offers an affordable and efficient alternative, providing comprehensive assessments of multiple capabilities without the need for exhaustive testing across hundreds of sub-tasks.
2.2
LVLM FOR GAMES
AI has long been applied to games, with notable achievements like Deep Blue's victory over the world chess champion (Campbell et al., 2002) and AlphaGo's mastery of Go (Silver et al., 2016), while games like Atari have also served as benchmarks for evaluating AI algorithms such as re- inforcement learning (Schrittwieser et al., 2020). However, these models were trained for specific tasks with structured rules, often relying on non-visual inputs. contrast, games with rich visual environments and complex multimodal interactions present greater challenges. Recently, LVLMS have demonstrated strong reasoning capabilities by combining visual perception with language un- derstanding, making them well-suited to act as agents in game environments that demand both modalities for decision- making and interaction (Xu et al., 2024; Zhang et al., 2023). For exam- ple, CRADLE (Tan et al., 2024) introduces flexible framework where LVLMs interact with games and software using screenshots and keyboard/mousse inputs, enabling them to complete complex tasks in games like Red Dead Redemption 2 (RDR2) without relying built-in APIs. Similarly,
3

---

Published as a conference paper at ICLR 2025
the VARP agent framework (Chen et al., 2024b) applies vision- vision-language models to the action role- playing games like Black Myth: Wukong, using visual imputs for tasks such as combat and action planning. While these works focus on designing LVLM-based agent systems perform control tasks within game environments, they do not exploit games as a systematic platform for quantita- tively evaluating the models themselves. SmartPlay (Wu et al., 2023) and GAMA-Bent (Huang et al., 2024) are more closely aligned with our work, as they pioneer the use of games as systematic benchmark for evaluating the capabilities of LLMs. However, these benchmarks primarily focus on text-based models, converting all game states into text and evaluating models in text-only environ- ment. addition, there are several concurrent works that also leverage game- based environments to evaluate LLMs. GTBench (Duan et al., 2024b) explores the effects of different prompt engi- neering techniques, such as Chain-of-Thought and Tree-of-Thought, on the reasoning capabilities of LLMs in game scenarios, aiming to enhance their strategic decision-making processes. Simi- larly, GameBench (Costarelli et al., 2024) uses games as a platform to assess the strategic reasoning abilities of LLMs through various complex tasks. However, these benchmarks primarily focus on text-based models, converting all game states into textual descriptions and evaluating models a text-only environment. As result, they do not assess the visual perception capabilities multi- modal reasoning abilities required for real-world interactions. In comparison, the proposed LVLM- Playground provides more comprehensive evaluation framework by leveraging both visual and textual inputs, allowing for richer and more realistic assessment of LVLM capabilities in dynamic game environments.
LVLM-PLAYGROUND
3.1
OVERVIEW
Visual Inputs
Text Inputs
x
LVLMs
Perceiving
Rule Follow
E2E Playinng
Q&A
X
Make a Move
Next Movement
Outputs
Game Screenshots
Rule-based Task Prompts
Tic Tac Toe
Next Movement
Chess
Sudoku
Game States
Search- Based
Opponent
Game Simulators
Minimax Search)
Battle
Game-based Evaluators
Figure 1: Overview of LVLM- VlMM-Playrooudd LVLMs receive visual and textual inputs to perform
tasks such Perceiving, Q&A, Rule Following, and End-to-End Playing in game environments. Dashed lines indicate interactions with search-based opponent using algorithms like Minimax or Alpha-Beta pruning in competitive games.
Previous research has found that video game playing is associated with cognitive abilities such as spatial visualization, attention control, and visual search strategies (Latham et al., 2013; Han al., 201 The ability of individuals, even children, to quickly grasp game mechanics and consistently apply skills related attention, memory, and problem-solving during gameplay further reinforces this connection (Chaarani et al., 2022). Building upon these insights, WYLM-Playgroum lever- ages games systematically evaluate the diverse cognitive and reasoning abilities of LVL.Ms. As illustrated in Figure 1, LVL.M-Playrroum consists of pool of game simulators where LVLMs in- teract with visual inputs, such as game sereenshots, to interpret current game states. Along with task prompts based on game rules, including Perception, Question Answering, Rule Following, and End-to-End Playing (detailed in the following sections), the LVLM is expected to output either task- specific responses or the next move in the game. The selected move is then executed in the game simulator, updating the game state and generating new screenshots for further processing. com- petitive games, Al opponent is introduced, typically powered by search algorithms like Minimax
4

---

Published as a conference paper at ICLR 2025
TicTacToe
Reversi
Sudoku
Minesweoper
Gomoku
Chess
Figure 2: The LVLM-Playground comprises six different games, including Tic Tac Toe, Reversi, Sudoku, Minesweeper, Gomoku, and Chess.
ensure a challenging adversarial environment. The results of each battle task completion are fed into came-based evaluators, which assess performance based on specific game rules.
3.2 GAME SELECTION
A key consideration for building LYLM-Playground is selecting appropriate games. While re- cent works have explored using LVLMs in complex video games like RDR2, these games require low-latency, real-time actions, which current LVLMs struggle to handle due to high computational demands. a result, their application has been largely limited to simplified scenarios, such as fragmented combat sequences, rather than full gameplay. Additionally, using such large games as benchmarks increases implementation complexity and raises the barrier to usage. Therefore, AVLM-Playgrowe focuses lightweight, tum--aaeed board games that do not require strict real- time cetiiiionmmkiin. but feature clear, well-defined rules and significant challenges that demand strong reasoning abilities. Following these principles, selected six games (see Figure 2) that of- fer clear win/loss conditions or scoring systems, providing straightforward metrics for performance evaluation: Tic Tac Toe involves two players taking the objective of aligning three marks in row, either horizontally, vertically, or diagonally. Reversi features players taking turns placing discs on an 8 board. aiming to flip the opponent's dises and control the majority of the board by the game's 8 end. Sudoku requires players to fill a 9 x 9 grid with numbers, ensuring that each digit from to 9 appears exactly once every row, column, and 3 x 3 subgrid. Minesweeper challenges players to uncover squares on grid, using numerical clues to avoid hidden mines while revealing safe spots. Gomoku played on x 15 grid, where whether horizontally, vertically,
place their marks 3 3 grid, with
two players compete to be the first connect five stones in or diagonally. Chess pits two players against each other on a checkered board, where each piece moves according to specific rules, with the goal of checkmating the opponent's king.
3.3 ABILITIES
Games have long been used in psychology to assess and enhance cognitive abilities (Von der Heiden et al., 2019; Dale & Green, 2015), with numerous studies establishing a link between gameplay and cognitive development (Choi et al., 2020; Martinez al., 2023). Even the act of playing simple game engages multiple cognitive processes (Boot, 2015). Players must perceive the game state, reason through rules. and make decisions to choose the best moves. In competitive games, players also develop adversarial skills by anticipating and reacting to their opponent's strategies.
Intuitively, understand that different games present varying levels of difficulty. More complex games, like chess, require higher levels of reasoning and decision- games like Tic-Tac-Toe. This is due to factors such as larger game state spaces, more intricate rules, and a greater variety of pieces and possible moves. Building on these factors, and drawing inspiration from previous work in entertainment computing (Aponte et al., 2011: Fraser al., 2014), developed a framework to objectively quantify the demands each game places on the four key abilitics: Perception, Reasoning, Decision, and Adversary.
compared to simpler
Taking the relatively simple game of Tic-Tac-Toe (TTT) several key factors that influence the cognitive demands of the game. TTT is played on a x grid, with each cell being either marked as X, O, left empty, resulting in three possible states per cell. Ignoring turn-by-turn rules for simplicity, the total possible game states can reach up to 39 Additionally, there two distinct piece types, X and O, and the board size is 3. These
illustrative example, we first review
5

---

Published as a conference paper at ICLR 2025
Table 1: Ratings of the required abilities for each game across four key dimensions.
Ability
Tic Tac Toe
Reversi
Sudoku
Minesweeper
Gomoku
Chess
Perception
Reasoning
1
Decision
Adversary
13
*
N/A
N/A
****
factors contribute to the perceptual difficulty of the game. Specifically, as the number of possible game states (S), piece types (P),, and board size (N) increase, it becomes more challenging accu- rately perceive the overall game state. This leads to the formulation for quantifying the perception complexity as perception = Op (S) + 3, 1og (P) + "p [oogoo (o [N)]² where Opp Sp. 7pp are coefficients that weight the respective factors in determining the perception complexity.
Perceiving the game state is essential, but reasoning goes beyond mere observation. It involves deeper analysis, such as asking: "How many marks has my opponent placed?" "Which cells are critical to control?", "Should 1 focus on defense or plan an attack?". These considerations require players to process information and answer questions, moving beyond perception. To quantify rea- soning difficulty, we consider the number of possible game states (S), the average branching factor (B), which represents the number of available moves at each stage, and the uncertainty factor (U). For example, in TTT, B is 5, as the number of possible moves decreases from 9 as the game pro- gresses. U accounts for hidden or random information; games like TTT, where all information visible, U 0, but in games with hidden information like Minesweeper, U = 1 due to uncertainty. Thus, reasoning complexity is quantified as Preasoning = a, 1 + 3, (og + U, where ar, 3r. 7r are coefficients that weight the respective factors in determining the reasoning complexity.
Beyond perception and reasoning, strategic decision decision-making is essential, encompassing both short- term and long-term considerations. A move made early in the game can influence outcomes much later, linking decision complexity to the average game length (I.). In TTT, for instance, games range from to 9 moves, resulting in an average of 7. Additionally, decision-making involves resource management, where players must balance trade-offss, such as sacrificing one asset for a more valuable advantage. Although TTT involves managing only one type of resource (X or O), more complex games like chess require handling multiple types with varying functions, contributing to resource complexity (C). Finally, the average branch factor (B) still plays role, as more choices at each turn add to the decision makinng challenge. Therefore, we quantify decision- decision-making complexity as &decision = ard logio B) + Ba log ()) + T logio (C),, where ats, Sa. "aa are coefficients that weight the respective factors in determining the decision-making complexity.
For non-cooperative multiplayer games, the challenge extends beyond making optimal moves for oneself; it also involves anticipating and countering the opponent's strategies. Adversarial difficulty hard to quantify, as it depends not only on the player's actions but also on the opponent's level of skill and unpredictability. To simplify this, approximate the complexity by focusing the possible interactions between players, where each decision is influenced by potential responses from the opponent. We quantify this by still using the average branching factor (B) and the average game length (T.), but multiply them to reflect the steady increase in complexity as players react to each other's moves over the course of the game. Thus adversary complexity is approximated as 4 Padversary = L (B).
Through the above formulations, Playground. The resulting scores are normalized to a 0.5-5 scale, mapped to star rating system from half a star (>z) to five stars * enabling comparison of the relative difficulty between games (see Table 1). Due to space constraints, please refer to Appendix A for detailed calculations and human study validating the reasonability of the rating system.
quantify the abilities required for each game in LVLM-
3.4 TASKS
Evaluating models through E2E gameplay can leverage all the four abilities discussed in Section 3.3, yet it is unclear whether the challenges arise from issues in perceiving the game state, following the rules, or other factors. To address this, we break the evaluation into four distinct tasks: Perceiv-

---

Published as a conference paper at ICLR 2025
Game Screenshots
Percaiving
Q&A
Following
E2E Playing
X
0
there?
A3
:
%
Al
Key Abilities
P
P R
D
P
R
D
A
Models are evaluated on various games within the L.T.M-Playground framework, with each game consisting of four tasks: Perceiving, Q&A, Rule Following, and E2E Playing. Each task largets one more abilities, including Perception, Reasoning, Decision, and Adversary.
Figure 3:
ing. Question Answering, Rule Following, and E2E Playing. Each task targets one or more of the aforementioned abilities, allowing for more comprehensive assessment (see Figure 3).
Perceiving tests the model's 's ability to accurately capture detailed visual elements. The task requires the model to fully transcribe randomly generated visual game state into structured representations. For example, TTT, the model receives a screenshot of the board and must convert it into a 3 x matrix, where 0 represents "O", represents "X", and -1 denotes empty space.
Question Answering extends the perceiving task by evaluating the model's ability to apply rea- soning alongside perception. To align with common practices in recent Q&A-based evaluations, we adopt a multiple-choice format to help decouple instruction-following capability from percep- tion and reasoning abilities. This approach reduces the likelihood of models being penalized for deviating from the expected output format, which allows a more focused evaluation of their core capabilities. Each game includes set of tailored questions based on specific rules and scenarios, testing a range of abilities. For example, object counting (e.g.. "How many black pieces are on the board?" Gomoku), geometrical reasoning (e.g. "Which cell has been occupied by circular marks?" in Tic- Tic-Tac-Tooe Toe), and OCR-like tasks (e.g., "How many mines are adjacent to the cell at row b, column 1?" in Minesweeper). For each question, the answer is automatically generated by the game simulator, providing one correct answer along with multiple distractor options tailored to the specific game and question. Typically, four options are provided per question, although for specific question types, such as Yes/No questions, fewer options may be applied.
Rule Following tests the model's ability to internalize and apply game rules to identify valid moves based the current state. For example, in Chess, pawns move forward move in an "L"" shape. Similarly, in Reversi, a valid move must sandwich at least one of the oppo- nent's discs between two of the player's discs. While humans can quickly grasp game rules from a few lines of instructions, models rely on task prompts to internalize these rules, In this task, a random game state is presented, and the model selects the next valid move, which then executed in the game simulator to verify its compliance with the rules. This process also evaluates the model's in-context learning ability as it adapts to the game rules and makes informed decisions.
square, while knights
End-to-Ean Playing provides a comprehensive assessment of the model's competence, testing its ability to manage a game from start to finish. This task evaluates the model's overall proficiency across perception, reasoning, Gomoku, the model faces a search-based opponent that employs established strategies such as Min- imax search, requiring the LVLMs to make strategic decisions throughout the game. In single- player games like Minesweeper, the model must independently navigate the game, making informed choices based on the current state and available information. By simulating full gameplay, this task holistically measures the model's ' ability to understand and succeed in complex game environments.
mackiion-making... and adversarial play. In competitive games like
3.5 EVALUATIONS
To evaluate the models on Lly.M-aaygrouddd each task is assessed using specific metrics based on the task and game rules.
Perceiving. The perceiving task assesses the model's ability to convert a visual game state into structured matrix representation. Accuracy is measured by the proportion of cor-
7

---

Published as a conference paper at ICLR 2025
rectly identified elements compared to the ground truth, calculated using the formula Accp mxn Li=l ا L(P.) Gsy), where m and n denote the matrix dimensions, P.j represents the model's 's output for cell (i.j), and Gıj is the corresponding ground truth. I(Ps Giy) equals when the predicted value matches the ground truth.
Q&A. In the Q&A task, each question is presented in multiple-choice format with fixed set of candidate answers. The model's response is matched against the predefined options, and a response considered correct if exactly matches the ground truth answer.
Rule Following, The rule-following task evaluates the model's ability to apply game rules to de- termine valid moves. The board coordinates are specified in task prompts, allowing the LVLMs to output moves in alphanumeric formats such as Al or B3. The game simulator then verifies whether these proposed moves are valid according to the game rules.
E2E Playing. The E2E playing task evaluates the model's ability to play the game from start to finish. Similar to the rule-following task, the LVLM proposes the next move in alphanumeric format, which is executed in the game simulator. If there is an opponent, the model must consider their responses; otherwise, it plays independently. To prevent indefinite loops, if the model generates three invalid moves per gameplay, it is automatically declared a loss. Since most LVLMs struggle to complete a full game, performance was evaluated based on the number valid moves, partial game progression, and final outcomes. For example, in chess, this includes the number of valid moves made by the LVLM, the number of opponent pieces captured, and the final game result.
To provide comprehensive comparison of model performance, for each task by factoring in both the difficulty of each game and the specific abilities required for the task. As shown in Table 1, each game g G (2..92...... ... } is assigned star rating Sq.a for each ability a E A, which reflects the game's demand on that ability. In ad- dition, Figure illustrates the different combinations of abilities required by each task in T (Perceiving, ARlle Following, E2E Playing). The model's performance on each game-task pair represented by Mg.t- To aggregate these scores, we calculate the overall performance for each task using the formula OverallScore(t)- Eger Sa.t
compute an aggregated score
Evec Sg,0
4 EXPERIMENTS
To validate the effectiveness of the LlLM-Playrround benchmark and explore current LVLM limitations, we tested both cutting-edge commercial LVLM APIs, including GPT- 1ominni115-proo. and Claude-3.5-Sonnet as well as widely used open-source models like Qwen- 2-VL (Wang et al., 2024a), DeepSeek-VL. (Lu et al., 2024), Phi-3-VL (Abdin et al., 2024), LLaVA- 1.6 (Liu et al., 2023), and Intern-VL2 (Chen et al., 2024c). All models were evaluated under the same conditions, including identical settings for maximum new tokens and task prompts. For Per- ceiving, Question Answering, and Rule-Following tasks, we utilized the simulator to generate 2,000 samples for each, followed by offline evaluation. For the End-to-End playing task, we conducted online evaluations, running 100 gameplays per model. In the tables, bold values indicate the best performance for each task, underline values highlight the best among apen-source models, and blue values show performance below random baseline.
4o (op=-10-202400--06, Gemini-1.5-Pro
4.1 RESULTS AND FINDINGS
In this section, we present detailed task and game performance of the selected LVL.Ms. Based on these results, we highlight several key findings that may reveal the limitations of current LVLMs.
Finding I. Limits Complex Perception and Output. Table 2 shows LVLMs excel in simple games like Tic Tac Toe (e.g., Gemini: 0.994, Claude: 0.985), but struggle with complex boards like Gomoku and Chess. In Gomoku (15x15), models like Deepseek-vl-17 (0.010) and LLLVAA-16-11 (0) fail to output the required matrix, often due size errors (e.g., 15x16) or repetitive generation, while Qwen2-vl-7b (0.589) and Phi3-vl (0.593) perform better. In Chess (8x8), even top models (e.g., Claude: 0.554) show modest results, indicating challenges in perceiving dense details. This
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative results of the perceiving task across different LVLMs.
LVLMS
TicTTae
Reversi
Sudoku
Minesweeper
Gomoku
Chess
GPT-4o
0.709
0.195
0.122
0.148
0.013
0.271
Geminil.5 1.5-pro
0.994
0.882
0.723
0.580
0.583
0,473
Claude-3.5-somne
0.985
0.992
0.912
0.741
0.742
0.554
Qwen2-vl-7b
0.613
0.359
0.567
0.203
0.589
0.271
Deepseek-vl-7b
0.369
0,463
0.282
0.202
0.010
0.276
Phi3-vl
0,535
0.463
0.287
0,202
0.593
0.271
LLaVA-1.6-7b
0.302
0.477
0
0.188
0
0.209
Intern VL2-8b
0.489
0.178
0,405
0.343
0.416
0.242
Random
0.332
0.333
0.103
0.093
0.332
0.079
highlights LVLMs' dual limitations: poor recognition of complex visuals and difficulty generating long structured outputs.
Table 3: Quantitative results of the Q& task across different LVLMs.
LVLMs
TicTaaeToe
Reversi
Sudoku
Minesweeper
Gomoku
Chess
GPT-4o
0.812
0,440
0.430
0,436
0.351
0.485
Geminil.5-p. --5
0.843
0.524
0,471
0.420
0.361
0,414
Claude-3.5
0.821
0.694
0.694
0.778
0.467
0.656
Qwen2-vl-7b
0.553
0.409
0.407
0.546
0.296
0.441
Deepseck-vl-7b
0.326
0.319
0.260
0.357
0.265
0.396
Phi3-vl
0.477
0.354
0,313
0,470
0.278
0,281
LLaVA-1.6-7b
0.269
0.244
0.281
0.248
0.249
0.272
Intern VL.2-8b
0.600
0.334
0.365
0.457
0.280
0.393
Random
0.256
0.267
0.268
0.281
0.277
0.264
Finding 2. Perception Weakness from Vision-Language Misalignment. Table 2 shows GPT-4o un- derperforms compared to commercial peers like Geminil.55 .5-pro and -euude-3.--onnet in perceiving tasks (e.g., Gomoku: 0.013 vs. 0.742, Reversi: 0.195 vs. 0.992), often hallucinating that it cannot parse images. This suggests a misalignment between its visual and language abilities, hindering complex board recognition. Similarly. LLLVA--16-1 struggles across tasks, scoring zero Sudoku and Gomoku perception and falling below random baselines in Q&A. As an early model bridging vision and language with an adaptor, LLaVA's consistent poor performance highlights how such misalignment can impair both perception and reasoning in such tasks.
Table 4: Quantitative results of the rule-following task across different LVLMs.
LVLMs
TicTacToe
Reversi
Sudoku
Minesweeper
Gomoku
Chess
GPT-4o
0.895
0,070
0,462
0.333
0.413
0.509
Geminil.5-ppi
1.000
0.155
0.508
0.697
0.625
0.313
Claude-3.5-sonnet
0.874
0.255
0.721
0.308
0.506
0.133
Qwen2-vl-7b
0.690
0.174
0.238
0.570
0.517
0,440
Deepseck-vl-7b
0.521
0.091
0.212
0.587
0.509
0.419
Phi3-vl
0.461
0.145
0.235
0.567
0.483
0.370
LLaVA-1.6-7b
0.639
0.102
0.193
0.584
0.454
0.003
Intern VL.2-bb
0.706
0.170
0.206
0.583
0.510
0.378
Random
0.342
0.127
0.214
0.422
0.508
0.014
Finding 3. Poor Rule Comprehension in Complex Games. Table shows LVLMs excel in simple muleffollowing tasks like TicTacToe (e.g., Gemini: 1.000, GPT-4o: 0.895), but falter in complex

---

Published as a conference paper at ICLR 2025
games like Reversi, where most models (e.g.. GPT-4o: 0.070, Deepseek-vl-7b. 0.091) perform or below the random baseline (0.127). In Gomoku, performance hovers around random (e.g.. GPT- 4o: 0.413, LLaVA-1.6-7b- 0.454 0.508), with Gemini (0.625) exception. This suggests many LVLMs fail to grasp intricate rules, often producing random moves instead of informed deci- sions, especially in games requiring detailed perception and strategic understanding.
Table 5: Quantitative results of the E2E task across different LVLMs.
LVLMs
TicTaceoo
Reversi
Sudoku
Minesweeper
Gomoku
Chess
GPT-4o
18.8
94.9
9.2
82.8
55.5
70.8
Geminil
30.1
100.7
8.3
68.8
61.1
12.7
Claude-3.5-sonnet
31.0
70.0
22.5
68.7
66.7
32.6
Qwen2-vl-7b
20.0
40.0
1.1
30.2
10.0
27.9
Deepseek-vl-7b
14.9
40.0
1.5
36.3
10.0
10.0
Phi3-vl
20.0
40.0
1,9
49,8
30.0
10.0
LLaVA-L.6-7b
10.0
40.0
1.2
0.0
10.0
0.0
Intern VL228b
19.2
40.0
0.9
38.8
20.0
10.0
Finding 4. Stochastic Parrot Behavior in E2E Gameplay. Table 5 reveals that LVLMs struggle to complete full games in the E2E task, often exiting early due repeated in- valid moves despite scoring some intermedi- points. Prompted provide observations and strategies alongside movements, models frequently generate responses that sound plau- sible abbe-ddecriibig board states proposing plans-but fail to translate these into valid ac- tions. For instance, in Reversi, opensource models stagnate at the initial score (40), while commercial models advance further. This pattern suggests a "stochastic parrot" behav- ior: LVLMs mimic gameplay language without truly understanding rules or adapting strategies, reflecting gap between their linguistic fluency and practical game game-playing ability.
can ii Loctee
2.40
o2e
Figure 4: Game-weighted performance of L.V.Mss on four abilities in the LVL.M- VILM--Paaggruud.
Summary. Figure highlights LVLMs' performance across Perceiving, Q&A. Rule- Following, and E2E tasks in the LyLM-Playgoound benchmark. Commercial models generally excel in perception and rule comprehension, while open-source models show moderate progress but struggle with complex gameplay. Key limitations include poor handling of dense visuals, rule un- derstanding, and sustained play, revealing gaps in vision-language integration and output coherence. This underscores the need for further advancements in LVLMs to bridge these challenges effectively.
5 CONCLUSION
In this paper, we present I.I.M- Plaag-llayg, game-hased evaluation framework for LVLMS that incorporates six diverse games across four tasks, each targeting different core abilities. LVLM- Playground addresses several key limitations of current benchmarks, offering an effective alternative for analyzing and comparing LVLMs from multiple perspectives. Using this framework, ated both spen-source models and commercial APIs, uncovering several key findings that highlight the limitations of existing models. In summary, LVLM-Playgron provides novel solution for evaluating the perception, reasoning, and decision- makinn abilities LVLMs. We hope that our work will inspire further research and contribute new perspectives the community.
evalu-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Centre for Augmented Reasoning at the Australian Institute of Machine Learning.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally your phone. arXiv preprint arXiv:2404.14219 2024.
Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proc. IEEE Int. Conf. Comp. Vis., pp. 8948 8957, 2019.
Cran-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Proc. Advances Neural Inf. Process. Syst., 35:23716-2377 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proc. IEEE Int. Conf. Comp. Vis., pp- 2425 2433, 2015.
Maria- Virginia Aponte, Guillaume Levieux, and Stephane Natkin. Measuring the level of difficulty in single player video games. Entertainment Computing, 2(4):205-213, 2011.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time an- swers to visual questions. In Proc. Annu. ACM Symp. User Interface Softw. Technol., pp. 333-342, 2010.
Walter R Boot. Video games tools to achieve insight into cognitive processes, 2015.
Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134 (1-2):57-83, 2002.
Bader Chaarani, Joseph Ortigara, DeKang Yuan, Hannah Loso, Alexandra Potter, and Hugh P Gar- avan. Association video gaming with cognitive performance among children. JAMA network 2022.
open,
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang 7ang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20332 2024a.
Peng Chen, Pi Bu, Jun Song, Yuan Gao, and Bo Zheng. Can vlms play action role-playing games? take black myth wukong study case. arXiv preprint arXi::2409.12888 2024b.
Xinlei Chen, Hao Fang, Tsung-Yo Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zimick.. Microsoft coco captions: Data collection and evaluation server. arXiv preprint ar04v:/504.000 50.000055,, 2015.
Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic iisual-linguiste tasks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 24185- 24198, 2024c.
11

---

Published as a conference paper at ICLR 2025
Eunhye Choi, Suk-Ho Shin, Jen-Kwang Ryu, Kyu-In Jung, Shin- Young Kim, and Min-Hyeon Park. Commercial video games and cognitive functions: video game genres and modulating factors of cognitive enhancement. Behavioral and Brain Functions, 16:1-14, 2020.
Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. Gamebench: Evaluating strategic reasoning abilities of Ilm agents. arXiv preprini arXiv:2406. 1.6113.... 2024.
Gillian Dale and C Shawn Green. Video games and cognitive performance. In The video game debate, pp. 131-152. Routledge, 2015.
Yihong Dong. Xue Jiang, Huanyu Liu, Zhi Jin, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Proc. Annu. Meet. Assoc. Comput, Linguist., 2024.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vimevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:240.111611 2024a.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel- Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning limitations of Iims via game-theoretic evaluations. In Proc. Advances in Neural Inf. Process. Syst., 2024b.
James Fraser, Michael Katchabaw, and Robert E Mercer. A methodological approach to identifying and quantifying video game difficulty factors. Entertainment Computing, 5(4))441-449, 2014.
Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.00321, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 6904-6913, 2017.
Doug Hyun Han, Nicolas Bolo, Melissa A Daniels, Lynn Arenella, In Kyoon Lyoo, and Perry F Renshaw. Brain activity and desire for internet video game play. Comprehensive psychiatry, 52 (1):88 95, 2011.
Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang. Youliang Yuan, Wenx- iang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu. How far are we on the decision- making of llms? evaluating ilms' gaming ability in multi-agent environments. arXiv preprint arXiv:2403.1 11807, 2024.
Huishan Ji, Qingyi Si, Zheng Lin, and Weiping Wang. Towards flexible evaluation for generative visual question answering. arXiv preprint arXiv:2408. 00300, 2024.
Andrew J Latham, Lucy LM Patston, and Lynette J Tippett. The virtual brain: 30 years of video- game play and cognitive abilities. Frontiers in psychology, 4:629, 2013.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Proc. Int. Conf. Mach. Learn., pp. 19730-19742, 2023.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang. Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 26763-26773, 2024a.
Zhiyuan Li. Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. Proc. Int. Conf. Learn. Representations, 2024b.
Chenwei Lin, Hanjia Lyu, Xian Xu, and Jiebo Luo. Ins mmbennh: A comprehensive benchmark for evaluating Ivims' performance in insurance. arXiv preprint 10X 2024.
12

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Proc. Advances in Neural Inf. Process. Syst, 36, 2024a.
Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tumne
dataset for Ivims. arXiv preprint arXiv:2406. 11.400611888 1833, 2024b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.0552. 2024.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song- Long-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Proc. Advances in Neural Inf. Process. Syst., 35:2507-2522 2022.
Léa Martinez, Manuel Gimenes, and Eric Lambert. Video games and board games: Effects of playing practice on cognition. PLOS One, 18(3):e0283654, 2023.
Bryan A Plummer. Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to to-phrase correspondences for richer image- to-sentence models. In Proc. IEEE Int. Conf. Comp. Vis., pp. 2641-26499 2015.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839))604 609, 2020.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proc. Annu. Meet. Assoc. Comput. Linguist., pp. 2556-2565, 2018.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proc. IEEE Conf. Comp. Vis. Patl. Recogn. pp. 8317-8326, 2019.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Em powering foundation agents towards general computer control. arXiv preprint arXiv:2403.03186 2024.
Juliane M Von der Heiden, Beate Braun, Kai W Müller, and Boris Egloff. The association between video gaming and psychological functioning. Frontiers in psychology, 10:1731, 2019.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision language model's s perception of the world at any resolution. arXiv preprint 2xXi:2449..11991 2024a.
Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. arXiv preprint arXiv:2406.072388 2024b.
Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng. Canjie Luo. Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and liangwei Wang. On the general value of evidence, and bilingual scene-text visual question answering. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn.. pp. 10126- 10135, 2020.
13

---

Published as a conference paper at ICLR 2025
Xinyu Wang, Bohan Zhuang, and Qi Wu. ModaVerse: Efficiently transforming modalities with 1lms. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 26606-26616 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Proc. Ad- vances in Neural Inf. Process. Syst., 35:24824 01224224-4487,, 2022.
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang. Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. Skywork: more open bilingual foundation model. arXiv preprint arXiv:2310.19344 2023.
Qiucheng Wu, Handong Zhao, Michael Saxon, Trung Bui, William Yang Wang, Yang Zhang, and Shiyu Chang. Vsp: Assessing the dual challenges of perception and reasoning in spatial planning tasks for vlms. arXiv preprint arXiv:2407.01863. 2024.
Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. Smartplay: A benchmark for Ilms as intelligent agents. Proc. Int. Conf. Learn. Representations, 2023.
Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jiechuan Jiang. Zhiming Ding, and Börje F Karls- son. A survey game playing agents and large models: Methods, applications, and challenges. arXiv preprint ar arXiv:2403. 10249, 2024.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collabo- ration. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 13040-13051, 2024.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang. Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprchensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.1600 2024.
Chi Zhang, Penglin Cai, Yuhui Fu, Haoqi Yuan, and Zongqing Lu. Creative agents: Empowering agents with imagination for creative tasks. arXiv preprint arXiv:2312.02519, 2023.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large language model performance grade school arithmetic. arXiv preprint arXiv:2405.00332. 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. Proc. Int. Conf. Learn. Representations, 2024.
Appendix
In this appendix, we provide additional details that were not included in the main text due to page constraints. The appendix organized as follows:
Appendix A Calculation Ability Ratings
Weighting Coefficients
Game Rating Calculations
Human Study of Difficulty Ratings
Appendix B Detailed Game Settings
Perceiving
Question Answering
Rule Following
End-to-End Playing
Appendix C Search-based AI Opponents
Tic-Tac-Toe
Sudoku
14

---

Published as a conference paper at ICLR 2025
Reversi
Winesweeper
Gomoku
Chess
A CALCULATION OF ABILITY RATINGS
In section 3.2, we briefly introduced the calculation of ability ratings. Here, we provide detailed steps for calculating the rating values, including how base parameters such as the number of game states (S), branching factor (B), and game length (L) are determined for each game.
For simplicity, some parameters are estimated or mathematically derived, including game states that may NOT feasible in real gameplay.
A.1
WEIGHTING COFFFICIENTS
Here, we present the weighting coefficients used in the ability rating formulas Sec 3.2.
Perception: Op = 0.8, 3p = 1.5, 7/p 1.2
Reasoning: 0, 1.0, 3, = 1.0, / = 1.0
Decision: 0d 1.0, B4 = 1.0, 7 1.0
A.2
GAME RATING CALCULATIONS
Tic Tae Toe.
State Space Size (S) = 39. The Tic Tac Toe board is a 3 x 3 grid, with each cell having possible states (empty, X, O). Therefore, the total state space size is S = 3 19, 683.
Piece Types (P) There are two distinct piece types in the game, X and O. cells.
Board Size (N) 9: The board has 3
Average Branching Factor (B) = 5: Initially, the player has 9 possible moves (since all
cells are empty), and this number decreases as the game progresses. The average branching factor is approximated as B 9+1 5.
Game Length (L) 7: The shortest game ends in 5 moves, while the longest game can take up to moves. The average game length is approximately G = 5-2
Information Uncertainty Factor (U) O: Tic Tac Toe is perfect information game with no hidden states or randomness.
Resource Complexity (C) = 1: Players only manage one type of resource-their own pieces (X O).
Multiplayer (is multiplayer) True: Tic Tae Toe is two-player game.
Φ T = 0.8 log 1.5 log (2) + 1.2 (9) = 5.5057
To log (39) + (5) + 0 = 4.9931
T 1og (5) + log (7) 10g 1.5441
T llo (5) = 4.8928
Minesweeper.
State Space Size (S) 1081; Minesweeper
grid with 10 hidden mines. The total state space size is estimated considering all possible mine placements and grid states.
Piece Types (P) = 10: There are 10 possible piece types representing different grid states: numbers 1-8 (indicating adjacent mines), mines, and unrevealed cells.
15

---

Published as a conference paper at ICLR 2025
Board Size (N) = 81: The board contains 9 x 81 cells.
Average Branching Factor (B) 41: At the start of the game, all 81 cells are unopened. The average branching factor across the game is approximately BF 41.
Game Length (L) 50: On average, players perform about 50 actions in typical game. This includes opening cells and marking mines.
Information Uncertainty Factor (U)= = 1: Minesweeper is a game of hidden information, the mine locations initially unknown to the player, making it partially observable game.
Resource Complexity (C) Players primarily manage one resource: marking or open- ing cells, which represents the resource complexity.
Multiplayer (is multiplayer) False: Minesweeper is single-player game with no in- teraction between players.
Φ y = 0.8 log 1 ) + 1.5 logio (10) + 1.2 (l (81))² = 85.6423 83.6128
Φ log (10 logio (41) ) +
T logio (41) + (50) log1o(1) 3.3118
+ A is not available.
Gomoku.
State Space Size (S) = 3225, Gomoku is played 15 15 grid, where each of the 225 intersections can be either empty, occupied by a black stone, or occupied by a white stone.
Piece Types (P) There are two types of pieces Gomoku: black stones and white stones.
Board Size (N) = 225: The board consists of 15 x 15 225 intersections.
Average Branching Factor (B) = 113: The initial move offers 225 possible actions, and the
number decreases the game progresses. The average branching factor is approximately BF
225++1
113.
Game Length (L) = 45: While the maximum number of moves is 225, games typically end much sooner
player aligns five pieces in row. The average game length is around
L 45.
Information Uncertainty Factor (U) = O: Gomoku a perfect information game, mean- ing all information about the game state is visible to both players, with no hidden informa- tion or randomness.
Resource Complexity (C) 1: Each player manages single type of resource, their own pieces (black or white stones).
Multiplayer (is multiplayer) = True: Gomoku is a two-player competitive game where
players alternate turns, trying outmaneuver their opponent by forming five consecutive stones in a row.
$ 0.8 logio (222) + 1.5 logiu + 1.2 (225) 113.1861
Φ = logn0 (3 222 + logno + 0 = 109.4054
& = (113) + log1o (45) + log = 3.7063
& 45 log1 92.3885
Sudoku.
State Space Size (S) 2 9a1: Sudoku is played on a x 9 grid with each cell filled with digit between and 9.
Piece Types (P) 9: There nine different types of pieces, represented by the digits 1-9.
16

---

Published as a conference paper at ICLR 2025
Board Size (N) 81: The Sudoku board contains 9 9 81 cells.
Average Branching Factor =5 While each empty cell can theoretically hold any digit from I to 9, the Sudoku rules limit the options. On average, a player has around 5 possible choices per empty cell.
Game Length (L) = 50: Depending on the difficulty level, the game starts with some cells pre-filled. On average. player needs fill around 50 cells complete the puzzle.
Information Uncertainty Factor (U) O: Sudoku is a perfect information game, where all clues are visible on the board, and no hidden or random elements are involved.
Resource Complexity (C) = 1: Players only manage one resource, the digits to be filled into the empty cells.
Multiplayer (is multiplayer) = False: Sudoku is a single-player puzzle game, where play- ers work independently solve the puzzle without interacting with others.
+ S p = 0.8 logio (9 1.5 log 1 (9) + 1.2 (81)] 81.8902
Φ 1og10 + (0g10 (5) + 77.9926
e 1og10 + logio log10 (1) 2.3979 i ss not available.
Chess.
State Space Size (8) 13s1, Chess is played of which can be empty or occupied by one of 12 different types of pieces (6 types, each in two colors).
board with 64 squares, each
Piece Types (P)= 12: There are six distinct types of pieces in Chess (King, Queen, Rook, Bishop, Knight, Pawn), with each having two colors (black and white), for a total of 12 types.
Board Size (N) 64: The Chessboard contains 8 8 64 squares.
Average Branching Factor (B) 35: On average, a player has around 35 legal moves available per turn, though this can vary significantly depending on the game phase (opening, midgame, or endgame).
Game Length (L) = 80: A typical Chess game consists of about 40 moves per player, for total of approximately 80 moves.
Information Uncertainty Factor (U) 0: Chess is a perfect information game where all pieces and possible moves are visible to both players, and there is no element of randomness hidden information.
Resource Complexity (C) = 6: Each player manages six different types of pieces, each with unique movement rules and strategic roles, contributing to higher resource complexity.
Multiplayer (is multiplayer) True: Chess two- wlayer game, where players alternate turns, making it multiplayer, competitive environment.
Φ 0.8 lo 1.5 1.5 (12) 1.2 [log (64))² 75.6338
c = log (1364) + ( + 0 = 72.8364
& = log + log (80) + (6) 4.2253
$6 80 login 123.5254
Reversi.
State Space Size (S) = 364, The Reversi board is an 8 x 8 grid with 64 squares. Each square can be empty, occupied by a black disc, or a white disc.
Piece Types (P) 2: There are types of pieces in Reversi: black and white discs.
17

---

Published as a conference paper at ICLR 2025
Board Size (N) 64: The board contains 8 x 8 64 squares.
Average Branching Factor (B) 10: Players typically have around 10 possible moves per turn on average, although this can vary throughout the game. Initially, there are fewer available moves, while mid-game offers more opportumities.
Game Length (L) = 60: Reversi games generally last for about 60 moves in total, as the board has 64 squares, but not all squares are necessarily filled.
Information Uncertainty Factor (U) Reversi a perfect information game with no hidden information or randomness.
Resource Complexity (C) 1: Each player manages one type of disc (either black or white). There are no other resources to consider.
Multiplayer (is multiplayer) True: Reversi is two-player game, where players alter- nate turns placing their discs and flipping the opponent's 's discs.
p $ R = 0.8 (3 + 1.5 1o (2) + 1.2 (64)) = 34.0991
t 10 + logio (10) + 31.5358
Φ = (10) + log (60) + log,o(1) 2.7782
Φ 60 (10) 60.0
Next, we normalize the calculated ability scores to fit within 0.5 5-star range. This is done by first identifying the minimum and maximum raw scores for each ability dimension across all games:
Perception: d min 4 4.9795,
4 & 113.1861
Reasoning: R
$ R 4.9931,
& max
Φ 109.4054
Decision: @D = T = 1.3979, T D
max = $6 = 4.2253
min
Adversary: &4 & 4.8928,
$A4 max
$ 123.5254
Table 6 summarizes the normalized scores for each game across the four ability dimensions.
Game Tic Tac Toe
Perception
Reasoning
Decision-Making
Adversary
0.5
0.5
0.5
0.5
Minesweeper
3.86
5.0
3.08
N/A
Gomoku
5.0
4,22
3.57
3.83
Sudoku
3.70
1.82
1.92
N/A
Chess
3.45
2.87
5.0
5.0
Reversi
1.72
1.39
2.42
2.62
Table 6: Normalized ability scores for each game across the four dimensions.
We then map the normalized scores, after rounding, to the 5-star system shown in Table 1, providing comparison of the cognitive and reasoning demands across different games and abilities.
A.3 HUMAN STUDY OF DIFFICULTY RATINGS
To validate the reasonableness of the computed ratings, we conducted human-based evaluation by recruiting volunteers with varying levels of familiarity with the selected games. Each volunteer was asked to rank the relative difficulty of the six games based on their gameplay experiences. This ranking provided a comparative difficulty chain from the easiest to the most challenging game.
We then summed the normalized difficulty scores from Table 6 to generate overall difficulty score for each game. For games where certain ability scores were marked as N/A, such as Minesweeper and Sudoku in the Adversary dimension, we assigned minimal placeholder score of 0 to ensure consistent calculations across all games. This approach allowed us to rank the games based on their overall computed difficulty, producing difficulty chain follows:
18

---

Published as a conference paper at ICLR 2025
Nre age Mumeen Rarkking of Game Difficu
The the
Aniola
Chost
Figure Average Human Ranking of Game Difficulty
Tie Tac Toe Reversi < Sudoku Minesweeper Gomoku Chess.
For the human-based evaluation, the rankings from all 10 volunteers were averaged to create a consolidated ranking for each game. Each volunteer's ranking the overall difficulty, with the easiest game assigned a score of and the most challenging game assigned a score of We then calculated the average ranking for each game across all volunteers, as demonstrated in Figure 5. The comparison between the computed difficulty chain and the human- based rankings showed high degree of consistency, supporting the reasonableness of our computed difficulty scores. We also observed that the average human rankings for Sudoku, Minesweeper, and Gomoku were very close. This suggests that, while these games differ mechanics, they are perceived as similarly challenging. However, each of these games emphasizes different cognitive abilities which could explain the subtle differences in difficulty experienced by different players.
based on their perception of
To further validate the evaluation settings, particularly the difficulty levels of each game across the four dimensions (perception, reasoning, deeisionnmmainn adversary), we conducted an addi- tional human study involving 10 volunteers.
To ensure consistency and provide the volunteers with clear understanding of each task, we sam- pled 10 representative examples for each game from the LVLM- simulator, covering perceiving, cuuestion-nsseetnn and rule-following tasks. Volunteers were asked to perform these tasks themselves to gain firsthand sense of the difficulty of each task. Additionally, adversary skills required for each game, volunteers participated in competitive matches against the search-based AI opponent in the LVLM Playgoloooo Since each volunteer varied in their familiarity with the games, we provided detailed game rules and guidelines beforehand to help them understand the rules and goals of each game.
assess the
Finally, we asked the volunteers to rank the four capabilities we defined for each game. The aggre- gated feedback summarized follows.
Reversi
Sudoku
Minesweeper
Gomoku
Chess
Perception
1.00
2.20
4.60
4.50
4.80
3.90
Reasoning
1.00
2.10
3.90
4.70
4.50
4.80
Decision
1.20
2.50
2.30
4.70
4.90
5.40
Adversary 1.00
2.50
N/A
N/A
2.60
3.90
As shown the table, the human study of relative difficulty among different games demonstrates similar trend in the overall evaluation. Although the results do not perfectly align with the quantitative ratings of the required abilities for each game described the paper, the trends are consistent, further validating the robustness of our evaluation framework.
19

---

Published as a conference paper at ICLR 2025
B DETAILED GAME SETTINGS
In the main text, we provided brief introduction to the four tasks included in the LVLM- Playground. Here, we provide more details.
B. PERCEIVING
010
000
1
2
3
5
6
7
8
1
1
1
1
2
1
2
2 2
2
1
1 1
1 2 1
g
1 1
1
(a) Chess
(b) Gomoku
(c) Minesweeper
Mistakes 0/S
cre playing
X
X
4
X
O
5
X
Hayer Mile
1
5
(d) Reversi
(c) Sudoku
(f) Tictactoe
Figure 6: Screenshots of randomly generated game states. Since these are for perception purposes only, the game rules are not guaranteed, and the game states may never occur in real games.
As shown in Figure 6, the Perceiving task specifically designed evaluate the model's ability observe and interpret visual information from game boards, without following the actual rules of the games. The game simulator generates a wide variety of random game states, some of which may NEVER occur in real gameplay scenario. For example, in Figure 6a, the chessboard shows an unrealistic scenario where there are multiple queens of the same color, which not possible under standard chess rules. This randomness introduces complexity and variety, allowing the model to focus solely on its capacity to interpret the visual aspects of the game boards.
In general, the models asked to convert the image game state into structured format, such as a matrix to represent the game board. For example, in the Reversi game, the model is expeceted to output an 8 8 matrix where black pieces represented by 1, white pieces by 2, and empty spaces by 0, accurately capturing the current game state from the visual input. Below is an example of the 8 x 8 matrix representing the game state shown in Figure 6d:
20

---

Published as a conference paper at ICLR 2025
1
2
1
2
1
0
27
0
1
0
0
2
1
2
1
0
1
1
2
0
2
2
1
1
1
2
1
0
2
1
0
2
2
0
2
1
The Perceiving task, while simple for humans, requires models to achieve high precision in lo- cal perception. For humans, interpreting game board and translating into structured data is straightforward process, often done with little effort. However, for models, this task demands pre- cise handling of the visual input and may involve capabilities like OCR correctly interpret the elements on the board, particularly in games that include alphanumeric labels such as Sudoku or model's ability to accurately capture fine-grained
Minesweeper. Moreover, this task emphasizes details from the image and map them correctly into structured format. Small errors in perception can lead to significant inaccuracies the resulting matrix, which demonstrates the importance of the model's ability to perform precise local perception.
All models are provided with the same task prompt for each game, ensuring that they are operating under fair and controlled environment. This consistency allows for direct comparison of their performance on the same task, without any variations in the instructions they receive.
Below the task prompts provided to the models for each game:
Perceiving Task Prompt for Chess:
You are provided with an image of chessboard, and your task is to represent the current state of the game as an 8x8 matrix using the specified numerical format. Each type of chess piece, both black and white, is represented by a unique number:
- Empty squares: 0
White pieces: Pawn=1, Knight=2, Bishop=3, Rook=4, Queen=5, King=6
Bllack pieces: Pawn=-1, Knight=-2, Bishop=-3, Rook=-4, Queen=-5, King=-6
From the provided chessboard image, convert the visible board into this 8x8 matrix format. For example, typical chessboard configuration would be represented as follows:
2
-5
-6
-3
-
1
-1
-
0
0
0
0
0
Game State:
0
0
0
0
0
0
1
1
1
2
3
6
3
4
Ensure that your output strictly follows this matrix format with no devia- tions, based on the pieces shown in the image.
Perceiving Task Prompt for Gomoku: Gomoku Game State Task:
21

---

Published as a conference paper at ICLR 2025
Given screenshot of random 15x15 Gomoku board, please represent the current game state as a 15x15 matrix. Use to represent black stones,
represent white stones, and 0 to represent empty intersections. Example format:
0
0
Game State: 0
0
1
0
2
:
0
This represents simplified version of 15x15 board where 1 is black stone, is a white stone, and 0 is an empty space.
Perceiving Task Prompt for Minesweeper:
Minesweeper is logic-based puzzle game played on 8x8 grid. Each cell can either contain mine (represented by 9), it can be empty. Unrevealed cells should represented by -1. Cells that are revealed and contain adjacent mines are represented by O, while cells that revealed and show number from to 8 indicate how many adjacent mines surround the cell. Mines represented by the number Please strictly follow the format:
Game State: <boardmatrix>
where aboardmatrix> is an 8x8 matrix representing the game grid, with unrevealed cells -1, mines as 9, and numbers from to 8 indicating the number of adjacent mines. For example:
1
1
2
1
1
1
1
2
2
-1
1
2
2
3
4
3
1
1
2
2
2
2
2
Game State:
1
2
2
2
2
9
1
0
3
0
-1
-1
1
1
2
3
1
-1
1
-1
1
This example represents grid where some cells have been uncovered,
showing numbers indicating adjacent mines, unrevealed cells are repre- sented by -1, and mines are represented by the number
Perceiving Task Prompt for Reversi:
22

---

Published as a conference paper at ICLR 2025
Reversi (also known Othello) is a strategy board game played 8x8 grid, where two players take turns placing black and white pieces on the board. The goal is have opponent by the end of the game. A piece is placed on an empty square and must sandwich placed piece and another of the player's 's pieces in a horizontal, vertical, or diagonal line. The opponent's pieces in between player's ss color. Given a screenshot of the Reversi board, please represent the current state of the game using an 8x8 matrix. In this matrix, empty cells should be represented by 0, black pieces by 1, and white pieces by 2. Please strictly follow the format:
pieces of your color the board than your
more of the opponent's pieces between the newly
then flipped to the
Game State: <boa rdmatrix
where <boardbat rix> is an 8x8 matrix. For example:
0 0
0 0
0
0
0
0
0
0
0
0 000
0
0
0
2
0
0
Game State:
2000
0
0
0
0
0
1
1
0
0
represents Reversi board with few pieces already placed and empty cells (represented by 0).
Perceiving Task Prompt for Sudoku:
Sudoku is logic- based puzzle played on 9x9 grid, where the grid is subdivided into nine 3x3 subgrids. The goal is fill the grid that each row, each column, and each 3x3 subgrid contains all digits from 1 to 9 without repetition. Given screenshot of the Sudoku grid, please represent the current state of the puzzle using 9x9 matrix. In this matrix, an empty cell should be represented by O, and filled cells should contain their respective numbers (1-9). Please strictly follow the format:
Game State: <boardmatrix>
where www.rdbaattaa..oo..o............ is 9x9 matrix. For example:
5
3
0
7
0
0
07
6
1
9
5
0
0
0
0
0
0
8
0
6
0
0
Game State:
4
0
3
7
0
0
2
0
0
0
0
0
4
1
9
0
5
8
9
represents partially filled Sudoku grid with some cells empty (represented by 0).
Perceiving Task Prompt for Tic Tac Toe:
23

---

Published as a conference paper at ICLR 2025
Tic Tac Toe is game played 3x3 grid where players take turns placing X or O in the cells. Given a screenshot of the game board, please determine the current game state using a 3x3 matrix. In this matrix, an empty cell should be represented by -I, X should be represented by I, and O should be represented by 0. Please strictly follow the format:
Game State: <boardmatrix>>
where <boardmat is 3x3 matrix. For example:
-1
Game State:
-1
represents empty board.
As shown in the prompts above, each task specifies clear and structured output format that the models are expected to follow. For each game, the models must generate a matrix representing the current state of the game board. These matrices must conform to the exact specifications outlined in the task prompts, ensuring that the output is both precise and standardized.
To evaluate the model's performance on the Perceiving task, we compute the accuracy by compar- ing the model's predicted matrix with the ground truth matrix. Each task specifies a clear expected output format, typically matrix representing the current game state. Let P represent the predicted matrix and G represent the ground truth matrix, both of size m x n, where m and n are the dimen- sions of the specific game board. The accuracy be calculated using the following formula:
M
R
Accp
IPP Gt)
TI X 7
To ensure the responses are in the correct format, we apply a regex to extract the matrix from the model's output. If the model's response cannot be parsed by the regex does not adhere to the expected structure, is marked as invalid, and a zero accuracy score is assigned for that task. This approach ensures fair and consistent evaluation across all models.
B.2 QUESTION ANSWERING
The Question Answering task follows the structure of traditional VQA but is adapted to a game- based environment. key advantage of this setup is that specific game states can be easily obtained from the simulator, allowing for the automatic generation of question-answer pairs without manual annotation. This contrasts with traditional VQA, which requires significant human effort for data collection and labeling. Additionally. this approach enables detailed manipulation of the input, allowing us evaluate models on fine-grained perception and reasoning based on visual details. Unlike standard VQA, which often focuses on broader scene-level questions, LVLM -laM--Payground encourages models to engage with and reason about visual details.
We generate screenshots of randomly created game states, similar to the process used in the perceiv- ing task (as shown Figure 6). For each game, specific question pool designed, accompanied by APIs to retrieve the corresponding ground truth answers. These questions are crafted to assess the model's understanding of the game state and its reasoning based on visual input.
To address the common challenge in VQA evaluation of variability in response format from lan- guage models, we adopt a multiple-choice format for the Q&A task. Each question is paired with a predefined set of candidate answers, including correct answer and several distractors. This set- ting eliminates the reliance on free-form responses and simplifies the validation process, focusing the evaluation the model's 's reasoning and perception abilities rather than cdaructtoo-followingg
For each game, questions are designed with corresponding candidate answers, typically contain- ing four options, although fewer options may be provided for binary questions (e.g.. Yes/No). The
24

---

Published as a conference paper at ICLR 2025
multiple-choiee format ensures consistency tasks and enables direct comparison of model per- formance. The prompt structure includes clear instructions and examples of valid question-anseee answer pairs to guide the model in selecting the correct answer.
Q&A Task Prompt for Chess:
Chess is played on an 8x8 board with six types of pieces, including pawns, knights, bishops, rooks, queens, and kings, for both white and black play- ers. The board uses a coordinate system where columns are labeled "a" through "h" from left right, and rows are labeled "1" through "8" from bottom to top. For example, the bottom-left corner is "al" and the top-right corner is "h8". Please answer the following question based on the provided screenshot of the current game state:
(question)
Answer: <answer>
where <answer> should be one of A, B. D.
Q&A Task Prompt for Gomoku:
Gomoku is played on 15x15 grid, where black and white stones placed turns. The goal to place five consecutive stones a horizontal, ver- tical, or diagonal line. Please answer the following question based on the provided sereenshot of the current game state: (question)
Answer: <answer>
where <answer> should be one of A, B, o.
Q&A Task Prompt for Minesweeper:
Minesweeper is played on an 8x8 grid with exactly 10 mines. Each cell can either contain mine, be unrevealed, or show the number adjacent mines (from 0 to 8). The grid is labeled with rows A to H (top to bottom) and columns I to 8 (left to right). Please answer the following question based on the provided screenshot of the current game state: {question]
Answer: <answer>
where <answer> should be one of A, B, or D.
Q&A Task Prompt for Reversi:
Reversi (also known Othello) is played on 8x8 grid where two players take turns placing black and white pieces on the board. Please answer the following question based on the provided screenshot of the current game state:
{question)
Answer: <answer>
where <answer> should be one of A. B, or D.
Q&A Task Prompt for Sudoku:
25

---

Published as a conference paper at ICLR 2025
Sudoku is played on 9x9 grid, where each row, column, and 3x3 subgrid must contain the numbers to 9 exactly once. Please answer the following question based on the provided screenshot of the current game state: {question]
Answer: <answer> where <answer> should be one of A, C,C. D.
Q&A Task Prompt for Tic Tac Toe:
Tic Tac Toe is game played a 3x3 grid where two players take turns placing X or O in the cells. The goal is to form horizontal, vertical, or diagonal line with three of your own marks. The grid labeled with rows A to C and columns I to 3. Please answer the following question based on the provided screenshot of the current game state: {quest on
Answer: <answer>
where <answer> should be one of A, B, o.
As shown above, only responses that adhere to the specified format and contain the correct answer will contribute the accuracy. Otherwise, even if the answer is correct but presented in an unex- pected format, it might be marked as incorrect. This is reasonable, much like in human exams where failure to follow formatting rules can result in penalties, regardless of the correctness of the answer. Therefore, the ability to strictly follow instructions is crucial for the model, it reflects whether truly understands the given instructions.
For each game, have designed unique set of questions tailored to its specific dynamics and rules. Below, we outline the types of questions used each game and how they test different aspects of the model's perception and reasoning capabilities.
Chess Question Types.
Piece Color at Position Question: What is the color of the piece at column {col label], row {row label)?
Piece Name at Position
Question: What the piece at column (col label), row (row label]?
Count of Specific Pieces
Question: How many (piece.color) {picce_name}s are the board?
Count of Pieces in Row/Column Question: How many pieces are in row (row label)
OR Question:
How many pieces are in column {cbe_label}?
Comparison of White and Black Pieces Question: Which color has more pieces, white or black?
Comparison of Two Piece Types Question: Which has more, {piece 1 Lnmee}s {piece2.name)ss
Count of Edge Pieces
Question: How many pieces are on the edge of the board?
26

---

Published as a conference paper at ICLR 2025
More Empty Cells in Top or Bottom Half of the Board
Question: Which half of the board has more empty cells, top or bottom?
Gomoku Question Types.
Stone at Specific Position
Question: What is the stone at row (row.label column {col_idx}?
Count of Specific Stones
Question: How many 'Black' stones on the board?
OR
Question: How many 'White' stones the board?
Count of Stones in Row/Column Question: How many 'Black' stones are in row label??
OR
Question: How many 'White' stones are in column {col_idx}?
Winning Condition
Question: there winning line on the board?
Adjacent Stones
Question: How many adjacent stones are around row {row.label), column {dx]_idx}?
Count of Empty Cells
Question: How many empty cells are there on the board?
Maximum Consecutive Stones
Question: What is the maximum number of consecutive 'Black' stones in row [row ?
OR
Question: What is the maximum number of consecutive 'White' stones on any diagonal?
Count of Edge Stones Question: How many 'Black' stones are on the edge of the board?
OR
Question: How many 'White" stones are on the edge of the board?
Minesweeper Question Types.
Revealed Symbol at Specific Position
Question: What is the revealed number or symbol in row (row label), column (col label]?
Count of Revealed Mines
Question: How many revealed mines are there on the board?
Count of Revealed Cells
Question: How many revealed cells are there on the board?
Count of Revealed Cells in Row/Column
Question: How many revealed cells are there in row (row_label}?
OR
Question: How many revealed cells are there in column {col label)?
27

---

Published as a conference paper at ICLR 2025
Number of Adjacent Mines
Question: How many mines are adjacent to the cell at row frow_lbeel,, column {col label}?
Is There a Mine at a Specific Position
Question: Is there revealed mine at row (row_label} column {col_label]
Reversi Question Types.
Symbol Specific Position
Question: What is the symbol in row frow.label} column (col_label}?
Count of Specific Symbol
Question: How many 'Black' pieces are present on the board?
OR
Question: How many 'White" pieces are present on the board?
Count of Empty Cells
Question: How many empty cells are there on the board?
Count of Specific Symbol in Row/Column Question: How many 'Black' pieces are there in row {row label)?
OR Question:
How many 'Wiiee' pieces are there in column [col]label]?
Row with Most Pieces of Specific Symbol Question: Which row contains the most 'Black' pieces?
OR Question:
Which row contains the most 'White' pieces?
Comparison of Black and White Pieces Question: Which player has more pieces on the board, 'Black' 'White"?
Total Number Pieces in Row/Column
Question: How many pieces are there in total in row (row_label}?
OR
Question: How many pieces are there in total in column (col label]
Total Count of Black or White Pieces on the Board Question: How many 'Black' pieces are there in total on the board?
OR
Question: How many 'White' pieces are there in total the board?
Sudoku Question Types.
Symbol Specific Position
Question: What is the number in row (row). column (col)?
Count of Specific Number
Question: How many ((number's 's are present on the board?
Count of Empty Cells
Question: How many empty cells are there on the board?
28

---

Published as a conference paper at ICLR 2025
Count of Specific Number in Row/Column
Question: How many '(number)' 's are there in row {row}?
OR
Question:
How many '(number)'s 's are there in column {col}?
Count of Specific Number in Subgrid
Question: How many '(number''s are there in the subgrid starting at row (subgrid_start. row], column startrid
Sum Numbers in Row/Columm
Question: What is the sum of numbers in row {row}?
OR
Question:
What is the sum of numbers in column {col]?
Sum of Numbers in Subgrid
Question: What is the sum of numbers the subgrid starting at row (subgrid row). column sttatrod
Does Row/Column Contain Specific Number Question: Does row (row) contain the number [[uumber3"? OR
Question:
Does column {col} contain the number '(number]"?
Count of Empty Cells in Subgrid
Question: How many empty cells are there in the subgrid starting at row (subgrid start row), column (subgrid start col]?
Tic Tac Toe Question Types.
Symbol at Specific Position
Question: What the symbol row [row). column {
Count of Specific Symbol
Question: How many '(symbol)'s 's are present on the board?
Count of Empty Cells
Question: How many empty cells are there?
Count of Specific Symbol in Row/Column Question: How many '(symbol} 's are there in row (row)?
OR Question: How many '(symbol) 's are there in column {col}?
Winner of the Game Question: Did X or O win the game?
Count of Red/Blue Marks on the Board
Question: How many (color) marks are present on the board?
OR
Question: How many (color) marks are there in row {row]?
OR
Question: How many (color) marks are there in column {col}?
29

---

Published as a conference paper at ICLR 2025
010
000
1
2
3
4
5
6
7
8
1
1
1
1
1
1
1 1
1
2
1
(a) Chess
(b) Gomoku
(c) Minesweeper
Mistakes
You are playing as
X
6
X
(d) Reversi
(c) Sudoku
(f) Tictactoe
Figure 7: Visualization the Rule Following task, where the model is presented with a random game state and required to select valid In (a) Chess, (d) Reversi, (e) Sudoku, and (f) TacTac valid moves are highlighted in green rectangles or circles. In (b) Gomoku, valid moves represented by the empty intersections, and in (c) Minesweeper, valid moves correspond the uncovered (gray) cells.
As shown in the examples above, the question-answering task challenges the model's ability to gage with various question types that require reasoning and fine-grained perception. Unlike the pure visual perceiving task, these questions go further by testing the model's reasoning skills and multiple detailed analytical abilities. For example, some questions involve counting, such as de- termining how many pieces of certain type are present on board, while others require basic arithmetic calculations, such as summing numbers in rows or columns in Sudoku. There are also tasks that require OCR-like recognition, identifying symbols numbers at specific positions, as well as questions that demand geometrical shape recognition, such as identifying edge pieces or distinguishing between shapes like circles and rectangles. These capabilities, which usually require separate benchmarks to evaluate, are integrated within the proposed LVI MMMMllyground, offering comprehensive assessment of the model's ability to perceive, reason, and handle fine-graii ane-grained visual details.
B.3 RULE FOLLOWING
The Rule Following task evaluates the model's ability to internalize and execute game rules accu- rately, assessing whether the model can determine and perform valid moves in accordance with the specific rules of each game. Unlike the Perceiving and Q&A tasks, where the randomly generated game states may not strictly follow the game rules, the Rule Following task ensures that all randomly
30

---

Published as a conference paper at ICLR 2025
generated game states adhere to the rules of the game. Then, the model is tasked with selecting valid move based on the state.
In chess (Figure 7a), each piece has specific movement rules, such pawns moving square forward. These moves can be described using algebraic notation, where each square is identified by letter and number. For instance, valid pawn move might be from e2 to e4, indicating that the pawn moves two squares forward from its starting position.
In Gomoku (Figure 7b) and Minesweeper (Figure 7c), valid moves selecting empty intersections on the board or unmarked (gray) cells on the Minesweeper map.
In Reversi (Figure 7d), a valid move must sandwich least one of the opponent's dises between two of the player's s discs.
In Sudoku (Figure 7e), number must not be repeated within the same row, column, subgrid.
In Tic-Tac- Toe (Figure 7f), a move valid occupies an empty cell, as each cell can only filled once.
For each game, we provide detailed game rules through structured prompts to test the model's ability to internalize and apply them. Specifically, the rule-following prompts for each game follows:
Rule Following Task Prompt for Chess:
8x8 board following standard chess rules. Each piece moves according to its unique capabilities. The board uses algebraic coordinates with files labeled a through from left right and ranks labeled 1 through 8 from bottom to top (al at bottom-left, h8 at top-right), White moves first, followed by Black. Based on the current board state image, please choose one legal move for White and output it using Standard Algebraic Notation (SAN). For example, if White's pawn on e2 can move to e4, your answer should be e4; if White's knight gl can move f3, your answer should Nf3.
Chess is played
Please strictly follow the following format:
Movement: <move>
where <move> is the move in SAN (e.g., e4, NF3, 0-O).
Rule Following Task Prompt for Gomoku:
Gomoku is played on a 15x15 grid, where black and white stones are placed on the intersections of the grid. The objective is to place five consecutive stones in horizontal, vertical, starts with an empty board. The grid is labeled with columns A to O (left right) and rows to 15 (top to bottom). Each intersection can only be occupied by one stone, either black white. Based the board state, please find an empty intersection where you can place your next stone.
diagonal line. The game
Please strictly follow the following format:
Movement: <pos t tion>
where the <positions can be any combination of columns A to O and to 15, for example, A1, H8, or 015.
Rule Following Task Prompt for Minesweeper:
31

---

Published as a conference paper at ICLR 2025
Minesweeper is played on 8x8 grid with exactly 10 mines. Each cell can either contain mine, be unrevealed, or show the number adjacent mines (from 0 to 8). The goal is to reveal all cells that do not contain mines, without triggering any mines. You win when only the 10 mine cells remain unrevealed. The grid is labeled with rows A to H (top to bottom) and columns to 8 (left to right). Each cell can only be revealed once, and flagging is not allowed. Based the current board state image, please find a cell to reveal next.
Please strictly follow the following format:
Movement: <posite sposition> tion>
where the <position> can be any combination of rows A H and columns to 8, for example, A1, B5, or H8.
Rule Following Task Prompt for Reversi (Othello):
8x8 grid. Players take turns placing black and white pieces on the board. The grid starts with black pieces and two white pieces in the center (D4, D5, E4, E5). A valid move consists of placing a piece in such a way that it sandwiches or more of the opponent's pieces between the newly placed piece and another of the player's pieces in horizontal, vertical, or diagonal line. After placing the piece, all of the opponent's pieces in between are flipped to the player's color. The black player (you) goes first, followed by the white player (AI). The grid is labeled with rows to and columns 8. Based the current game state, please find valid position where you can place your next black piece and flip at least one of the opponent's white pieces.
Reversi (also known Othello) is played
Please strictly follow the format:
Movement: <position>
where the <position> can be any combination rows A and columns to 8, for example, A1, D4, or H8.
Rule Following Task Prompt for Sudoku:
Sudoku is played on a 9x9 grid, where each row, column, and 3x3 subgrid must contain the numbers I to 9 exactly once. The grid starts with the top-left corner as A1, where rows are labeled from A to I and columns numbered from 9. A valid move involves placing digit from to an empty cell, ensuring that the number does not already appear in the same row, column, 3x3 subgrid. Based on the current state of the Sudoku grid, please find a valid empty cell where you can place a digit and make valid move.
Please strictly follow the format:
Movement: <row><column <digit>
where <row> is A to (representing rows), <column> is I to 9 (repre- senting columns), and tit> is number between to For example, A1 5 means placing the digit in the top-left corner of the grid.
32

---

Published as a conference paper at ICLR 2025
Rule Following Task Prompt for Tic Tae Toe:
Tic Tac Toe is game played a 3x3 grid where two players take turns placing their respective marks, X or O, in the cells. The game begins with empty board, and the grid is labeled with rows A to C and columns to Each position on the grid can only be occupied by one mark at a time. Based on the current state of the board, your task is find empty cell where you can place your next stone. Please follow the format below when indicating your move:
Movement: <position>
The position should be combination of row (A C) and column (1 to 3), such as A1, B2, or C3.
B.4 END-TO-END PLAYING
E2E Playing task evaluates the model's capability to autonomously play game from start to finish, testing its ability to integrate perception, reasoning, and decision-making. As discussed in the main text, the model suggests the next move in alphanumeric format, which is then executed within a game simulator. For games involving an opponent, the model must respond to the adversarial actions and adapt its strategy accordingly.
Here, first provide the E2E Play task prompt for each game:
End-to-End Playing Task Prompt for Tic Tac Toe:
Tic Tac Toe is played on a 3x3 grid. Players take turns placing X or O in the cells. The goal is to be the first form an unbroken line of three marks horizontally, vertically, or diagonally. The game starts with an empty board, and the player goes first. The grid is labeled with rows to and columns to 3. You are playing O, aiming win by placing marks strategically. Each position can only be occupied by one mark, so do not choose spot that is already taken. Based the board state screenshots, please first observe the current situation, then carefully think and explain your strategy briefly, and finally output your movement for this status.
Please strictly follow the following format:
Observation:
<observation>
Strategy: <strategy> Movement: <position>>
Where the observation should briefly summarize the current situation, the strategy is brief explanation of how you plan to win the game, and the position can be any combination of rows to C and columns to 3, for example, A1, 2B, or C3.
End-to-Enn End Playing Task Prompt for Sudoku:
33

---

Published as a conference paper at ICLR 2025
Sudoku is logic-based puzzle played 9x9 grid, subdivided into nine 3x3 subgrids. The goal is to fill the grid so that each row, column, and 3x3 subgrid contains all digits from to 9 without repetition. The grid starts with some cells filled with numbers (clues), and you must fill in the remaining empty cells to bottom), and columns are numbered to 9 (left to right). Each move involves placing digit (1-9) in empty cell, ensuring row, column, or 3x3 subgrid. Based on the current board state screenshot, observe the situation, formulate strategy, and output single valid move to place a digit.
time. Rows are labeled A (top
repetition in its
Please strictly follow the format:
Observation: <observation>
Strategy: <strategy>
Movement: <row> <column> <digit>
Where <row> is A I, <column> is I to 9, and <digit> is 1 to 9. For example, A1 places 5 in the top-left cell.
End-to-End Playing Task Prompt for Reversi (Othello):
Reversi (also known as Othello) a strategy board game played on an 8x8 grid. Players take turns placing black and white pieces the board. The goal is to have more pieces of your color on the board than your opponent by the end of the game. A valid move must sandwich one or more of the opponent's 's pieces between the newly placed piece and another of your pieces in horizontal, vertical, or diagonal line, flipping the sandwiched pieces to your color.
The game starts with four pieces in the center: two black (at D4 and E5) and two white (at D5 and E4). The black player (you) goes first, followed by the white player (AI). The grid is labeled with rows A to H (top to bottom) and columns 1 to (left to right). You are playing as black, aiming maximize your pieces by placing them strategically.
Based the board state screenshots, please first observe the current situation, then carefully think and explain your strategy briefly, and finally output your movement for this status.
Please strictly follow the format:
Observation:
<observations
Strategy: <strategy>
Movement: <position>
Where the observation should briefly summarize the current situation, the strategy is brief explanation of how you plan maximize your pieces, and the position can be any combination of rows to H and columns to for example, A1, D4, or H8.
End-to-Eae tn- Playing Task Prompt for Minesweeper:
34

---

Published as a conference paper at ICLR 2025
Minesweeper is logic-based puzzle game played on 8x8 grid with exactly mines. Each cell can either contain mine, be unrevealed, or show the number of adjacent mines (from 0 to 8). The goal is to reveal all cells that do not contain mines without triggering any mines. You win when only the 10 mine cells remain unrevealed.
The grid is labeled with rows A to H (top bottom) and columns 1 to 8 (left to right). You can only reveal cells (e.g., A1), and flagging is not allowed.
Based on the current board state screenshot, please observe the situation, formulate strategy, and output move to reveal cell.
Please strictly follow the format:
Observation: <observation>
Strategy: <strategy>
Movement: <row> <column>
Where <observation> summarizes the current board state, <strategy> explains your reasoning, and <position> is a move in the format Al.
End-to-End Playing Task Prompt for Gomoku:
Gomoku played on 15x15 grid, where players take turns placing black white stones the intersections. The goal is to be the first to form an unbroken line of five stones horizontally, vertically, or diagonally.
The game starts with an empty board, and the black player (you) goes first, followed by the white player (AI). The grid is labeled with columns A to O (left to right) and rows to 15 (top to bottom). You are playing as black, aiming to win by placing stones strategically. Each intersection can only be occupied by one stone, so do not choose spot that is already taken.
Based the board state screenshots, please first observe the current situation, then carefully think and explain your strategy briefly, and finally output your movement for this status.
Please strictly follow the format:
Observation:
<observation>
Strategy: <strategy>
Movement: <position>
Where the observation should briefly summarize the current situation, the strategy is brief explanation of how you plan to win the game, and the position can any combination of columns A O and rows 15, for example, A1, H8, or 015.
End-to-Enn tod Playing Task Prompt for Chess:
35

---

Published as a conference paper at ICLR 2025
Chess is a strategy game played 8x8 board with 64 squares, using six types of pieces: pawns, knights, bishops, rooks, queens, and kings, for both white and black players. The game starts with standard initial position: white pieces on ranks and 2, black pieces on ranks 7 and 8, The board uses algebraic coordinates with files labeled through h from left to right and ranks labeled through 8 from bottom to top (a 1 at bottom-left, h8 at top-right). White moves first, followed by Black.
You playing White, aiming checkmate the Black king achieve a favorable position. Each move must follow standard chess rules and be expressed Standard Algebraic Notation (SAN), such as (pawn to e4), Nf3 (knight to f3), or O-O (kingside castling).
Based the board state screenshots, please first observe the current situation, then carefully think and explain your strategy briefly, and finally output your movement for this status.
Please strictly follow the following format:
Observation: <observation>
Strategy: <strategy>
Movement: <algebraic notation>
Where the observation should briefly summarize the current situation, the strategy is a brief explanation of how you plan to win or improve your position, and the position legal move SAN, for example, e4, Nf3, or O-O.
The scoring rules for the E2E Playing task are designed to reflect the model's performance in terms of efficiency, accuracy, and game outcome across different games. Below, we describe the scoring mechanism for each game based its implementation:
Tie Tae Toe: The score combines base score, calculated points per move made by the player (O), and an outcome bonus: 50 points for win, 20 points for tie, and 0 points for loss. The total score is:
50 if win
Score
(Nmoves x 10) B Bouteome:
B Bouteome
20 tie
0
loss
where Nmoves is the number of moves.
Sudoku: The score includes a base score of 2 points non-initial filled cell, correctness score of 10 points per correctly filled non-initial cell, and 1000-point bonus for winning. The total score is:
{
1000 if win
Score = (Nalled x 2) + (Noorrec X 10) + Bwin- Bwin
otherwise
where Nitled is the number of filled cells, and Noomect is the number of correct fills.
Reversi (Othello): The score consists of step score (10 points per move). piece bonus (20 points per Black piece), and an outcome bonus (1000 points for a win, 500 for a tie, for loss). The total score is:
1000
if win
Score = (Nmoves X 10) + (Nolack x 20) + Boutsome
Boutoome
500
if tie
0
if loss
where Nmoves is the number of moves, and Natack is the number of Black pieces.
36

---

Published as a conference paper at ICLR 2025
Minesweeper:
The score comprises step (10 points per move), reveal bonus (2 points per revealed non-mine cell), and a 1000-point bonus for winning. The total score is:
{o
[1000 win
Score = (Nmoves x 10) + (Nreveated x 2) + Bwin: Bwin
loss
where Nmoves is the number of moves, and Neveated is the number of revealed non-mine cells.
The score includes a base score (10 points per move by Black) and an outcome bonus (1000 points for win, 500 for tic, 0 for loss). The total score is:
Gomoku:
1000
if win
Score (Noovvs 10) + B Butcome,
Boutcome
500
if tic
0 if loss
where Noves is the number of moves.
Chess: The score combines a step score (10 points per White move), a piece bonus (5 times the value of captured Black pieces: Pawn=1, Knigh//Bishop=3, Rook=5, Queen=9), and an outcome bonus (1000 for a win, 500 for a tie, O for a loss). The total score is:
1000
if win
Score = moves x 10) + (V Vaapuured x 5) + Boutcome:
Bouteome
500
if
0
if loss
where Newves is the number of moves, and Veaptured the total value of captured Black pieces.
These scoring mechanisms provide a comprehensive evaluation of the model's performance in the E2E Playing task, balancing efficiency, strategic decision-makee making, and successful outcomes across diverse game types.
C SEARCH-BASER AI OPPONENTS
In adversarial games, effective decision-making is essential as players must anticipate and counter their opponent's moves. To facilitate this, many search-based methods have established solid foundation for developing Al opponents. Below are key approaches utilized in our framework:
Tic-Tac-Toe: The AI opponent employs the Minimax algorithm, which recursively eval- uates all possible moves to determine the optimal play. The algorithm considers both the Al's and the opponent's potential moves, aiming to maximize the Al's chances of win- ning while minimizing the opponent's chances. The evaluation function assesses the board state, returning positive scores for winning configurations and negative scores for losing ones. This allows the AI select the best possible move each turn, effectively making formidable opponent.
Sudoku: N/A
The AI opponent Reversi uses an Alpha-Bets pruning technique combined with scoring system to evaluate board states. The Al identifies valid moves and simulates their outcomes by temporarily applying the moves to a copy of the board. The Alpha-Beer Beta algorithm efficiently narrows down the search space, allowing the AI to maximize its score while minimizing the opponent's score. This approach enables the Al to make strategic decisions and adapt to the opponent's moves effectively.
Reversi:
Minesweeper: N/A
The Gomoku AI utilizes recursive evaluation strategy to determine the best move. It evaluates the board based potential winning patterns, assessing both its own and the opponent's configurations. The AI implements a depth-limited search to simulate possible moves, using evaluation function that assigns values based the length of contiguous pieces. This allows the Al to prioritize moves that advance its strategy while blocking the opponent's 's attempts connect five in
Gomoku:
37

---

Published as a conference paper at ICLR 2025
The Chess Al leverages the open-source Stockfish' engine, which is a lightweight chess engine that implements the Minimax algorithm with Alpha-Beta pruning. This li- brary enables the Al to evaluate board positions efficiently, considering multiple possible moves and their outcomes. The Stockfish engine uses a heuristic evaluation function to as- sess board states, allowing the Al make strategic decisions and respond to the opponent's moves effectively.
Chess:
hhtps://toctffsshhes.orr
38


Reviews:
[{"review_id": "suupoS9Q8W", "reviewer": "Reviewer_ciTF", "summary": "To address the lack of a comprehensive evaluation framework for assessing Large Vision-Language Models (LVLMs), this study proposes a framework called LVLM-Playground with game-base evaluation. This study collects six games: Tic-Tac-Toe, Reversi, Minesweeper, Gomoku, Sudoku, and Chess, to evaluate both powerful open-source and closed-source models across four tasks: Perceiving, Question Answering, Rule Following, and End-to-End (E2E) Playing. The framework provides insights into the performance shortcomings and iterative improvements of LVLMs. The study also finds that a small amount of data from games can enhance the model's reasoning abilities significantly.", "questions": "1. I have noticed that various models are currently unable to succeed under the E2E play setting. However, E2E play is a very important setting in LVLM-Playground. Would it be possible to introduce intermediate evaluation metrics to assess model performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This study collects multiple games to evaluate the multimodal large models' abilities in perception, reasoning, decision-making, and adversary, and it also visualizes the games.\n\n2. This study evaluates the top-performing models and presents multiple findings, providing guidance for future improvements of multimodal models.\n\n3. The authors proposed a set of metrics to evaluate the difficulty of test games in terms of Perception, Reasoning, Decision, and Adversary, which can aid in assessing game-based evaluations when selecting games in the future.", "weaknesses": "1. I noticed that in **Findings 6. Gameplay Data Could Enhance LVLMs' Reasoning During SFT**,\" the early work MiniGPT-4 was used as a base model. Since the SFT stage of MiniGPT-4 only used detail description data, it is difficult to prove that the performance gains are due to the VQA data from the game, rather than other factors like data format. Therefore, I believe that the experiments in Table 5 are not sufficient to fully support your conclusions. I suggest supplementing this with more robust base models, such as LLaVA-1.5 or LLaVA-one-vision.\n\n2. In my opinion, these types of games are all based on dense grid patterns, and the format used to output game states might be too challenging for current multimodal large models. This could potentially limit the evaluation of the model's decision-making abilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B3bm027yfq", "reviewer": "Reviewer_2f5P", "summary": "The primary focus of this paper is the evaluation of “LVLMs” (Large Vision Language Models).  The authors motivate the work by noting that prior benchmarks have three key shortcomings: (i) they fail to assess detailed visual reasoning, (ii) they suffer from data contamination and (iii) they fail to assess multi-turn reasoning.  In response, the authors introduce an evaluation framework called LVLM-Playground that uses 6 games (TicTacToe, Reversi, Sudoku, Minesweeper, Gomoku and Chess) to measure LVLM capabilities. Using these games, the authors assess model performance along 4 axes: perception (i.e., accurate inference of the board state), rule following (whether a given model takes legal moves), question answering (whether the model can answer questions about a game state), and gameplay.  Through their experiments, the authors demonstrate that while frontier models are best at perception, they are (somewhat surprisingly) weaker than open-source models on question-answering tasks. This latter result is investigated and attributed to the influence of post-training (via RLHF). All models completely fail to achieve reasonable gameplay performance across the game suite against strong opponents.", "questions": "1. I’m curious to better understand the motivation behind the authors’ choice of using \"unbeaten rate\" to measure gameplay performance? Given access to games simulators which enable repeated games, this seems like a very high variance estimator of capability. Moreover, it is tightly anchored to the choice of opponent implementation for each game.  Did the authors consider alternatives (for example, creating a pool of opponents at different thresholds and computing Elo ratings?)\n\n2. Related to the first question, did the authors consider evaluation against non-search based opponents? Taking Tic-Tac-Toe as an example, presumably minimax will produce optimal play, so the best that can be achieved is a sequence of draws?\n\nSuggestion: I don’t list this point as  a “weakness” since it is concurrent work, but I would suggest citing relevant pre-prints from the first half of 2024 exploring similar directions. Examples include (1) GTBench (Duan et al., 2024), which also concludes that LLMs struggle against search-based strategies (e.g. MCTS) in games like Tic-Tac-Toe. For this reason, they also include a “random agent” baseline to illustrate that the models have achieved a basic level of competency.  (2) GameBench (Costarelli et al., 2024), which evaluates LLMs across 9 game environments and uses visual input (for models that support this). \n\nReferences:\n- J. Duan et al., \"GTBench: Uncovering the strategic reasoning limitations of LLMs via game-theoretic evaluations.\" arXiv preprint arXiv:2402.12348 (2024)\n- Costarelli, Anthony, et al. \"GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents.\" arXiv preprint arXiv:2406.06613 (2024).\n\nMinor typos/suggestions:\n- Minor typo on L379 (missing parenthesis)\n- Claude “soonet” is presumably intended to be sonnet in table 4.1 (this typo appears in multiple various places in the paper)\n- This is a minor point, but I would suggest citing MuZero in the related work section. This is perhaps the canonical example of achieving very strong performance across multiple games (including Chess, which is used in the current submission) while also being capable of processing RGB input on Atari games by leveraging both learning and search.  Reference: Schrittwieser et al., “Mastering atari, go, chess and shogi by planning with a learned model” (Nature, 2020)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: Leaving aside concurrent work (mentioned below), this is the first work to extend simple game-based evaluations for LLMs to include a detailed evaluation of perception capabilities.\n\nSignificance: By introducing a visual component into simple game-based evaluation, the authors open up an interesting avenue of possibilities for efficient evaluation of model capabilities. Board games (such as chess in particular) are well-suited for benchmark generation since simulators provide automatic ground-truth and their large state space renders them relatively immune to contamination. The fact that frontier models struggle on the Chess perception task (Table 2) indicates that there is scope to guide future model development with this approach. As such, I think this work will provide a useful point of reference for the community. \n\nClarity: By highlighting challenges with previous benchmarks, the problem is clearly motivated. More broadly, the paper is clearly well-written, and the figures do a good job of communicating key information. Figure 4, for example, communicates the key takeaways of the paper in a compact way.\n\nQuality: Beyond simply using games to construct a benchmark, I think the construction of several game-related subtasks is particularly useful. In particular, examining model capabilities on each game with respect to perception, Q&A and rule following is a nice way to extract additional information beyond end-to-end game playing ability.", "weaknesses": "1. One concern relates to the scale of the overall contribution. As the authors note, there are previous efforts such as SmartPlay (Wu et al., 2023) and GAMA-Bench (Huang et al., 2024) that explore the use of games for evaluating LLMs. The SmartPlay work, for example, also proposes a suite of games and assesses the individual capabilities (such as instruction following, reasoning and planning). As the authors highlight, it is true that these works only operate in text-space (SmartPlay converts image input into text-based visual descriptions for games that require it, rather than feeding in RGB images to the models). This is an important distinction, particularly given the focus of the current submission on assessing detailed perception. Nevertheless, the proximity of prior work somewhat diminishes the contribution of this submission.\n\n2. A second concern relates to the breadth and depth of the analysis. On the one hand, I think the authors did an excellent job of conducting qualitative analysis (both in the form of the findings described in Section 4.1 and the observations contained within Section D of the Appendix).  On the other hand, I think the paper could have been improved by conducting more extensive quantitative analysis to provide further insight. To give a couple of concrete examples: (i) It would be interesting to see how perception performance on games like Gomoku and Chess varies as a function of board complexity (number of pieces on the board); (ii) It would be insightful to perform a manual categorization of a sample of errors on the question-answering task to inform the reader about the distribution of error types that are harming model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZmlQBqQ9u6", "reviewer": "Reviewer_giEs", "summary": "This paper proposes a new benchmark to evaluate large vision-language models in structured environments across 4 tasks in 6 games of different difficulties. This benchmark is designed to evaluate different abilities of vision-language models including perception, reasoning, decision-making and adversary.", "questions": "-\tThe reviewer appreciates that the authors conducted a human study on the difficulty rating, which mostly though not perfectly aligns with the difficulty rating defined by the authors. The reviewer wonders if the authors could also obtain human ratings for the four dimensions \n-\tFor finding 6, did the authors use a randomly sampled subset of VQA? Does this subset test for VLMs’ reasoning ability? If not, the authors could consider using a more suitable benchmark or revising their claim.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-motivated and well-written overall. \n- The idea of using structured game environments to evaluate vision-language models is somewhat novel. \n- Some designs of the benchmark are well thought out: for example, it includes games that test different abilities of the vision-language models and cover different difficulty levels and provides clear definitions of the complexity of each ability. \n- The paper is clearly written with sufficient technical details and effective illustrations.", "weaknesses": "The main area of improvement for this paper is the alignment between the experimental designs and the claims. The reviewer is mainly concerned about some of the evaluation task designs and whether they actually test vision-language models’ perception or reasoning abilities like the authors claim. \n- For example, the perceiving task prompts the models to output matrices that correspond to the game states, which not only tests for perception but also requires strong instruction-following ability. Similarly, while the Q&A is mostly designed to test for perception and reasoning (Figure 3), it asks the models to output answers in a particular format and therefore also requires strong instruction-following ability. \n-\tAs the authors discussed in finding 3, all commercial models fail to match the output format or refuse to answer in the Q&A task, which means this task doesn’t actually test their perception/reasoning ability as claimed. \n-\tIn the reviewer’s opinions, the tasks can be designed better to decouple the evaluations of perception/reasoning and instruction following abilities. \n-\tFor example, most VLM evaluation benchmarks adopt the multi-choice Q&A format when testing for their perception/reasoning abilities. And when the answer format is free-form, VLMEvalKit -- one of the most widely used evaluation frameworks of VLMs – uses an LLM as an answer extractor, putting more emphasis on the models’ perception/reasoning abilities rather than their instruction-following ability. \n-\tDue to the differences in task and evaluation designs, the results reported in this paper (e.g. Table 3) regarding these models’ perception and reasoning abilities deviate a lot from existing works. \n-\tThe authors also claim that gameplay data could enhance LVLM’s reasoning abilities but have only provided very limited evidence on a subset of one VQA benchmark. \n\nVLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models: https://arxiv.org/abs/2407.11691", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WrFuFEbvay", "reviewer": "Reviewer_iEip", "summary": "This paper presents LVLM-Playground, a game-based benchmark for evaluating Large Vision Language Models (LVLMs). Using six diverse games and four tasks (Perceiving, Question Answering, Rule Following, and End-to-End Playing), it assesses LVLMs' abilities beyond the limitations of traditional VQA benchmarks. Experiments reveal model weaknesses in handling long outputs, detailed perception, and instruction following (potentially due to RLHF), while also showing that gameplay data can improve reasoning during fine-tuning.", "questions": "The LVLM-Playground framework seems highly beneficial for the community. Do the authors plan to publicly release the code and data?\n\nCould the authors elaborate on the evaluation of the Q&A task? Are the methods I mentioned (few-shots, LLM-based evaluation, etc) relevant for overcoming the issue you mentioned?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Novel Benchmark: LVLM-Playground provides a novel, game-based approach to evaluating LVLMs, offering a more dynamic and comprehensive assessment than traditional benchmarks.\n\n2. Rigorous Methodology: The paper presents a well-structured and technically sound methodology, with detailed task formulations, evaluation metrics, and implementation of AI opponents.\n\n3. Significant Findings:  The benchmark reveals key limitations in current LVLMs, particularly in handling complex visual details and instruction following, while also suggesting gameplay data as a potential avenue for improvement.", "weaknesses": "Re. Finding 3 - I find it less sound. Maybe I am missing something, but -  Yes, VLMs produce long answers, and the expected answers are short, but this is an evaluation problem, not the model’s fault. There  are several ways to overcome it - few-shot prompts with short answers, parsing the answers with regex to a closed set of answers, a model-based LLM, and so on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)