lmgame-Bench: How Good are LLMs at Playing Games?
Authors: Lanxiang Hu
Mingjia Huo
Yuxuan Zhang
Haoyang Yu
Eric P. Xing
Ion Stoica
Tajana Rosing
Haojian Jin
Hao Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
VLM
Agents
Benchmark
Games
Primary area: datasets and benchmarks

Abstract: Playing video games requires perception, reasoning, memory, and long-horizon planning—exactly the faculties expected of modern large language and vision–language models (LLMs/VLMs). We introduce LMGame-Bench, a benchmark built on six popular games spanning platformer, puzzle, and narrative games through a unified Gym‑style API. Unlike prior game benchmarks that entangle multiple skills, LMGame-Bench employs a modular harness—including perception, memory, and reasoning modules—that can be toggled to selectively probe distinct capabilities. The benchmark further improves robustness through prompt standardization and contamination mitigation. Evaluation of 13 state-of-the-art models demonstrates that LMGame-Bench remains challenging yet effectively discriminates among models. Correlation analysis reveals that individual games align with core LLM capabilities, providing a quantitative framework for interpreting performance. Finally, LMGame-Bench exposes models’ limitations in visual state extraction, reflection, spatiotemporal reasoning, and long-context reasoning, pointing to concrete directions for model improvement.

Published as a conference paper at ICLR 2026
LMGAME-BENCEE How GOOD ARE LLMS AT PLAY- ING GAMES?
Lanxiang Hu Mingjia Huo" Yuxuan Zhangi Haoyang Yut Eric P. Xing Ion Stoica Tajana Rosing:
Haojian Jin Hao Zhang'
l UC San Diego MBZUAT UCC Berkeley
ABSTRACT
Playing video games requires perception, reasoning, memory, and long-horizon planning-exactie the faculties expected of modern large language and vi- sion-language models (LLMs/VLMs), We introduce mark built on six popular games spanning platformer, puzzle, and narrative games through a unified Gym-style API. Unlike prior game benchmarks that entangle multiple skills, MMGame-Bench employs a modular harness-including percep- tion, memory, and reasoning modules-that can be toggled to selectively probe distinct capabilities. The benchmark further improves robustness through prompt standardization and contamination mitigation. Evaluation of 13 state-of-the of-the--rt art models demonstrates that LMGame- -Bnthh remains challenging yet effectively discriminates among models. Correlation analysis reveals that individual games align with core LLM capabilities, providing a quantitative framework for interpret- ing performance. Finally, LMGame- Bench exposes models' limitations in visual state extraction, reflection, spatiotemporal reasoning, and long-context reasoning, pointing to concrete directions for model improvement.
AMGame-Bennhh a bench-
INTRODUCTION
Interactive games are emerging as powerful benchmarks for large language and Iision-language models (LLMs/VLMs) (Shi et al. 2025 Ruoss et al.] 2025 Wang al. 2025 Paglieri al. 2024
Costarelli et al. 2024 Wu et al. 2023). With explicit rules and measurable outcomes, games provide well-defined metrics for success. Their complexity challenges models demonstrate diverse abilities, including reasoning, long-horizon planning, and adaptation to dynamic opponents. In addition, games contain built-in difficulty scaling from simple to complex, which makes them well-suited for distinguishing models without being easily saturated. These qualities together make games a uniquely effective testbed for evaluating frontier models.
Prior game benchmarks often entangle multiple skills at once once-vision perception, reasoning, and memory-mant maaing difficult to diagnose why models succeed or fail to attribute errors specific capabilities (Hu et al.) 2024; Paglieri et al.) 2024 Zhang et al. 2024: Ruoss et al.) 2025). Even seemingly "simple" games can test LLMs/VI LMMs across several axes: parsing the board state from pixels, reasoning about legal and high-value moves, and maintaining coherent multi-turn strategies. When all these factors are evaluated simultancously, results become hard to interpret and distinct failure modes remain hidden. Besides, the benchmark should be neither too hard for progress, in Zhang et al. (2024), nor too easy and already solved by current models, as in parts of Ruoss et al. that maintains challenging without
(2025). This calls for benchmark with modular design, saturation while isolating individual capabilities for fine-grained diagnosis.
We introduce LMGame-Bench. benchmark built on six video games spanning platformers, puzzle solvers, and a narrative-driven detective game through a unified Gym-style API (Towers et al. 2024) evaluating core model capabilities such as vision perception, reasoning, horg-horizon planning, and narrative understanding. When evaluated with a direct sscreensho--aactinn design, we observed that models achieve low performance, often close random action-taking baselines. This because
Equual contributions. Correspondence to haozhang @ucsd.edu, lah003 @ eucsd.edu. Significant contributions.

---

Published as a conference paper at ICLR 2026
Game Environment
Model Hamess
observatin Space
Oueget
Action: right
Modelo
Aztion Fanorios
1. Part Mt step Caming
2
B
Merrry
tegactstyy (Lest Staps up. Plaer (3.3.
Space
Ruffection: help player get to box (4
128
Fiemiinted:
netun Scores
Symbobe
ieprosortation
Perception Modul
States:
Gome
foards
Mayerr3113
Figure 1: L.Gamm-Bendee uses modular harnesses- such perception, memory, and reasoning modules-to systematically extend a model's game-playing capabilities, allowing the model to engage with simulated game environment through iterative interaction loops.
even advanced reasoning models fall short in visual perception and limited long-horizon decision making (Yang et al. [024 Waytowich et al.) [024] Mosquera et al. (024). To address this issue and also enable controlled evaluation, we enrich our evaluation settings by developing gaming harness (i.e. additional modules integrated into the agent workflow to support LLM-game interaction), including perception, memory, and reasoning modules to amortize vision perception limitations and facilitate long- horizoo planning shown in Fig. [ By enabling individual or combined modules, ablate perception aids, memory and reflection to isolate each skill while holding the underlying game constant, making it possible disentangle specific strengths and weaknesses (e.g., perception vs. planning) that would otherwise remain hidden. Furthermore, LMGame- MMGamm-Bench improves robustness by two additional components: (i) mitigating data contamination once is detected emphasize reasoning over memorization, and (ii) employing a standardized prompt optimization technique to reduce variance. Together, these techniques serve as principled scaffolds that enable more reliable evaluation of LLM agents.
We evaluate leading models in LMGame-Bente Results from 13 models across 6 games demonstrate that LMGame- presents a challenging benchmark far from being saturated. The benchmark effectively differentiates models: o3 and ol achieve top-2 best performance across all games, followed by Gemini-2.55pp pro--ire and 3.7dd-..--S Among non non-reasoning models, GPT-4.1 leads the pack. To move beyond raw scores, we introduce quantitative analysis based on correlation and low rank matrix factorization, linking IMGame-Bende to domain-specific benchmarks. This reveals latent relationships between individual games and isolated LLM capabilities, providing a quantitative framework for interpreting gaming performance.
LMGame Benchh exposes critical weaknesses of current models: First, current VLMs struggle to extract the states of board games (e.g. Sokoban, Tetris) directly from images (Anthropic) 2025), which a task trivial for humans. By converting images into textual representation, our perception module significantly boosts model performance, with the largest gains observed in reasoning models that excel at textual reasoning. Meanwhile, non-reasoning models frequently fail to detect invalid moves (e.g., repeatedly making the same invalid move in 2048). Our memory module, which tracks history and enables self-reflection, helps these models self-correct. Other failures include misaligning spatial choices with temporal dynamics and losing information over long contexts. By revealing diverse failure modes, IMGame- Bench points to concrete directions for advancing both model development and agentic design.
In summary, our paper makes the following contributions:
We introduce LMGame- -rGame-BBnch, the first benchmark that evaluates video games with or without scaffolds, enabling controlled comparisons across models.
We demonstrate that, under different settings, LMGame- Bennh can clearly discriminate among state-of-the-art models.
2

---

Published as a conference paper at ICLR 2026
We present quantitative analyses- innluddin correlation and latent factor modeling that reveal how different games align with core model capabilities.
We show that current models exhibit persistent limitations, pointing to concrete directions for improving model capabilities.
2 RELATED WORK
Games as AT Testbeds. Games have long served foundational benchmarks in Al research, particularly in reinforcement learning. From TD-Gammon (Tesauro et al. 1995) to AlphaGo (Silver
al. 2017). they have offered controlled environments for studying planning and sequential decision- making. OpenAl Gym (Brockman et al., 2016) further standardized this paradigm by providing a unified interface for interacting with diverse environments. More recently, games have been adopted to evaluate LLM agents on specific domains, such as grid-based games (Nasir et al. (0224) open-ended
strategy games (Hopkins al., 2025), murder-mystery games (Xie et al. (022a). Evaluations on individual tasks such as Pokémon Red/Blue (Comanici et al. 22055 Anthropic 2025) and the Kaggle AI Chess Exhibition Tournament (Risdal 2025/Olszewska & Risdal 2025) also highlight the potential of games as testbeds for LLMs/VLMs, Others evaluate natural language reasoning through text-based or conversational games (Costarelli al. 2024 Hudi al. 2025 Qiao al., 2023 Hu et al. 2024 lacking visual understanding. Multimodal gaming benchmarks such as BA BOLG (Paglieri et al.
2024) evaluate grid-based navigation tasks and textual reasoning, while LMAct (Ruoss et al. 2025)
examines the relationship between model performance and the number of expert demonstrationns though half of tasks are nearly solved by existing models. Meanwhile, both work primarily emphasize qualitative observations of game-related LLM abilities. VideoGameBench (Zhang al. (024) features complex 3D environments, but tasks are often too difficult for current models to show progress. Our work is distinct in game choice (inherent difficulty scaling), harness design (to probe diverse abilities), data contamination mitigation, and quantitative evaluation, which together provide more systematic and durable benchmark for frontier models.
LLM Agentic Benchmarks. Current agentic benchmarks tend to focus on domain-specific tasks such as code editing (Jimenez et al.) 2023), web browsing (Zhou et al., 2023 He al.)
2024). API control flows (Trivedi et al. (222)) GUI control (Agashe et al. 20244 system operations (Xie et al. 2024b). Recent efforts also explored broader evaluation across multiple aspects (Liu ct al. 2024 Mialon et al. 2023). While these benchmarks provide valuable insights into special- ized domains, games offer a complementary setting that is both scalable and skill-diverse, enabling evaluation of wider range of general agentic behaviors.
LMGAME-BENCE
We build the backbone of LMGame- on six well-known games ($ 3.1 evaluate leading models' performance without scaffolds. However, directly evaluating models on games in their original forms poses challenges, and we introduce mitigation techniques 3.2
3.1 BENCHMARK DESIGN
In designing ..Gaame-ench we intentionally recycle well-known games not only for their famil- iarity and popularity but also because they encapsulate a broad spectrum of reasoning and interaction skills, and contain inherent difficulty scaling to better distinguish models. Our goal is to preserve the original game settings that are carefully designed to challenge human cognition. In this section, we highlight the broad range of perception and generation abilities evaluated by the games.
3.1.1
GAMES
Super Mario Bros. Super Mario Bros is side-scrolling platformer game where the player controls Mario to navigate through obstacles, defeat enemies, and reach the end of each level by navigating through the environment. Success requires precise timing and strategic movement, making it classic benchmark for evaluating (1) visual perception, (2) spatiotemporal reasoning in 2D for character control, and (3) goal-directed planning with partial observability (Rintanen) 2004).
3

---

Published as a conference paper at ICLR 2026
Tetris. Tetris is a tile-matching puzzle game where players must strategically rotate and place falling Tetris tiles of 7 different geometric shapes to complete and clear horizontal lines. The game emphasizes (I) visual perception for pattern recognitions, (2) spatial reasoning for correct tile matching and geometric rotations (Lau-Vhu et al. 2017). and (3) long-horizon planning with partial observability for decision-making where and how drop a tile (Demaine et al. 2003).
Sokoban. Sokoban is a grid-based puzzle game where the player pushes boxes to designated target locations within confined spaces. It emphasizes (1) visual perception, (2) spatial reasoning to navigate both the character and the box, and (3) long- ong-horizon planning to avoid deadlocks (Culberson, 1997). The game's 's low fault tolerance is especially pronouneed. Many actions are irreversible, and single wrong move can fail the puzzle.
Candy Crush. Candy Crush is match-three puzzle game where players swap adjacent candies to form aligned sequences and trigger cascading effects to eliminate matched sequences. It requires (1) visual perception to identify different candies, (2) spatial reasoning to anticipate chain reactions at different locations, and (3) long-horizon planning to conserve moves to maximize total points. The gameplay features limited moves, making it crucial to plan moves carefully.
2048. 2048 is a sliding-tile puzzle game where players combine numbered tiles on grid to reach the 2048 tile. It evaluates (1) visual perception for tracking tile values and movements, (2) spatial reasoning to manage merging paths, and (3) goal coal-directed planning to maximize merge potential Zaky 2014). Errors compound quickly due to the game's s limited space and could lead to irreversible failure states.
Ace Attorney. Ace Attorney is an interactive courtroom-drams visual novel in which the player, acting defense attomny Phoenix Wright, must investigate crime scenes, interview witnesses, and present evidence in court to reveal contradictions and secure a "Not Guilty" verdict. The game stresses (1) long-context language understanding, tracking hundreds of dialogue turns, testimonies, and evidentiary facts, (2) causal & deductive reasoning under partial observability- clues, inferring hidden motives, and spotting logical gaps, and (3) long-horizon low-faul-ttllrancee decision making, to decide when to press, object, or present evidence over multi-stage trials.
dispersed
3.1.2 GAME SETTINGS
LMGame-Bente maintains the integrity of the original design choices, which ensure scalability. In this section, we focus on standardizing game settings including inputs and outputs of the gaming environments as part of our benchmark design as shown in Fig. I (right). We can formalize a gaming process as a partially or fully observable Markov Decision Process (MDP) with the following definitions as a generalizable formalism applicable to all games.
Observation Space Representations. Many existing games are graphical, requiring players to process multimodal information, including visual, textual, and spatial cues, from the user interface to interpret game states and make decisions (Liu al., Zhang et al. 2025). We denote symbolic and graphical representations of the game states as the set of all possible observations S. We don't 't make a distinction between game state space and observation space, which is not the focus of this work.
Action Space. Let the set of all actions in action space be A. To interface with the game, CeGGame-Bench considers multi-turn interactions. LMGame-Bench streams game states 8, E S to model M, each time it generates action a; A in response to the current state maximize rewards, which are meticulously crafted scores in classical games, with details specified in Metrics below.
Difficulty. Games in LMGame- Bench are designed with varying levels of difficulty, structured along two key dimensions: (1) fault tolerance and (2) state-action space complexity. We define three levels of fault tolerance -low (one wrong move fails), medium (errors accumulate but can be recovered), and high (many mistakes can be tolerated without significantly affecting future game states). We employ memory module curb search-space explosion ($ 3.2.1
3.1.3 METRICS
In line with Gymnasium Towers et al. 2024), we treat reward as a function R: S X 8 which returns the payoff obtained when the agent executes action a E A in state E S and transitions to state s'. L.Game-Bent adopts this definition to reports either progression rewards long-
4

---

Published as a conference paper at ICLR 2026
horizon rewards. Progression rewards provide dense, stepwise feedback as the agent makes forward advances. They apply to games with a primarily linear structure or no fixed endpoint, offering incremental scores that increase with each in Tetris, 2048's cumulative merge total, or Candy Crush's 's cumulative eliminated candies. Long- horizon rewards offer sparse credit awarded only upon completing multi-step objective. These are common in games built around multi-step puzzles or multi-stage narratives, where rewards are granted after achieving the full goal, such as solving all boxes in Sokoban or a courtroom sequence being correctly completed in Ace Attorney. To ensure comparability across games, we represent these rewards into a continuous raw score that sensitively captures performance. These raw scores are the primary evaluation signal as shown in Table 1
Marii's 's horizontal distance, the running score
See Appx. A for detailed metrics for each game.
3.2 BENCHMARK EFFECTIVENESS ENHANCEMENT
While using games evaluation presents challenging environments and breadth, we find that directly evaluating models on games exposes several challenges: low discriminability, contamination risk, and prompt variance. In this section, we address these issues by introducing gaming scaffolds for LLMs, contamination detection, and prompt standardization, enabling LMGame-Bene to function as a more robust benchmark that reliably differentiates LLMs.
3.2.1 GAMING HARNESS
Excluding text-only models, 40% of game runs without the harness fail to outperform random-pla play baseline. To raise the cap and bring higher contrast, LMGame- -Bechh provides a suite of harness modules that can be toggled on off for any experiment (workflow in Fig. [ Activating the harness boosts scores far beyond both random play and the unharnessed setting, creating clearer performance gaps between models. With harnessing, 86.7% of game runs beat the random baseline, and paired-sample t-tests confirm that harnessed runs score significantly higher than their unharnessed counterparts on Candy Crush, 2048, Tetris, Ace Attorney, and Sokoban (details in Appx.
and C.3)
Perception Modules. Since the video games are inherently multimodal, we build perception modules that convert UI inputs into symbolic representations or textual descriptions of game states facilitate understanding. For grid-based games (Sokoban, Candy Crush, 2048, Tetris), the module converts the visual layout into a text-basee bast- table from game backends, listing object coordinates and their properties, e.g. "Box at (2,3)". This allows models to directly understand spatial relationships in replacement of raw image inputs to minimize perception errors. For text-based games (e.g. Ace Attomeyy,, the module extracts dialogues and describes visual elements in text format to provide narrative context and critical visual cues. Likewise, we use perception module to extract visual elements in Super Mario Bros to facilitate decision making.
Memory Modules. Some games, like Sokoban and Tetris, exhibit a rapidly growing decision space gameplay advances and interactive elements scale (e.g., boxes, grid size, tetromino types). As result, they come with higher difficulty levels than the other games. To better distinguish models, we integrate additional memory modules into L.Gaame-Bench This setup allows selective activation of two components: (1) transient memory module, which records the past N game states and actions, and (2) a reflection module, which encodes explicit lessons learned to avoid failure, inducing actions in specific game states, thereby helping to narrow the action space.
Reasoning Modules. Reasoning models (Guo et al. 2025 OpenAl 2025) have emerged as new inference paradigm, where models explore multiple reasoning paths and synthesize a more accurate answer at the end. We support such reasoning traces by allowing models be evaluated with or without long chain-of tf-uuht (long-Col CoTT reasoning.
3.2.2 DATA CONTAMINATION
Because AMGame-Bende reuses publicly available game assets, many images and scripts may already appear in model pre-training data. To ensure the model isn't merely recalling artifacts, we test vision- level data contamination in Super Mario Bros and text-level data contamination in Ace Attorney, whose sprite and dialogue widely distributed online (Appx. D). The other games, Tetris, 2048, Candy Crush, and Sokoban, feature combinatorial state spaces (Dor & Zwick 1999] Demaine et al. 2003 Guald et al. 2014), making overlap with training data negligible.
5

---

Published as a conference paper at ICLR 2026
Vision-level. We assess whether models recall the visual structure of Super Mario Bros level 1-1 by prompting them to reorder shuffled RGB frames. Only a few models exhibit a moderate positive alignment, yet these alignment scores do not significantly track with their performance rankings. This suggests that they rely on local perception rather than memorized sequences. Since our metric evaluates how far models can play Super Mario Bros, focus on vision-level contamination that may expose future frames, and disregard contamination within the current frame-such as prior knowledge that a "?" brick may contain a mushroom-as it does not affect sequence prediction.
Text-level. In Ace Attorney, we test whether the models reproduce public fan transcripts. Using Sentence-BERg similarity, we find strong correlation between output similarity and performance, es- pecially in a 6-model subset. However, after applying structured prompt- -baed mitigation including entity masking, paraphrasing, and enforced reasoning (Dong et al. 2024 Carlini et al. 2018) the correlation disappears, and model rankings instead align with judged reasoning quality (Fig.
Text Comparison: Ground Truth vs. o3
Total Similarity vs Rank (standardized)
o3 Generated Text: Frank Sahwit Round I
Performance
-0.943, 4.71e-03
(Poirts) (Poines)
Ground Truth Text:
Rank (Standardized)
Old Rank (Trend) New Rank (Trend)
Frank Sahwir, Round
Statement: "It was exactly
Statement:"1 "I remember
1:00 PM when
esactly:
Lary Butz out of the apartment. Present:
1:00 PM. Present:
"Cindy's Autopsy Report."
Cindy's Autopsy Report
Contradiction Exposed:
-Contrdiction Exposed:
Death occurred "besween à PM and 5 PM. body could not be found at PM.
Autopsy
death
-05
occurred between 4:00
and 5:00 PM, making 1:00 PM murder
-1.0-
New Performance Rank:
1.21e-01
impossible.
-1.0
-03
0.0
0.5
1.0
Total Similarity (Standardized)
Figure 2: (Left) Example from Ace Attorney showing contradictions in O3-generated text vs. ground truth. (Right) Effect of mitigation correlations with old and new leaderboard ranks, respectively.
performance correlation; red and blue lines show
3.2.3 PROMPT STANDARDIZATION
Prompt engineering improves LLM performance across games (Paglieri et al., 2024 Wang et al.), yet even empirically tuned prompts can vary by more than =lo (Table [4p. To stabilize re- sults, AeGame-Bennh applies two-stage strategy: first, we adopt a canonical agentic format [Jmminoo--N:---1 R2-1, 8,] + a, where J the trajectory of the latest N turns (sj, ay, and R,-1 is the memory reflection; second, we use DSPy's 's SIMBA optimizer (Khattab et al. 2024 to iteratively refine prompts via introspective mini-batch ascent guided by game rewards. Across three runs, this standardization reduces prompt variance by 33.8%-63.55 in games such 2048 (Appx. yielding more consistent performance across models.
EXPERIMENTS
In this section, we present the rankings of 13 state-of-the-ard models, both with and without the gaming harness, evaluated on L.MaameBBnchh We also analyze the effectiveness of each harness module (.1.I). ) as well as the modules' combined effectiveness. We then conduct a qualitative analysis on our harness design and models' failure cases ($ (.2) and investigate how gaming environments reveal core LLM capabilities through correlation analysis, low-rank factorization, linear modeling ($ (.3). We quantify the issue of gaming data contamination and propose mitigation techniques in Appx. D
4.1 MODEL PERFORMANCE
When putting LLMs and VLMs in gaming environments, we first study if they can play the games well without gaming hamness. Table shows most models perform poorly. Specifically, over three fourths of models often score no points on Sokoban and Ace Attorney without harness support. On Tetris and Candy Crush, their scores close to random play, which suggests they succeed by chance

---

Published as a conference paper at ICLR 2026
Table 1: Model performance raw scores, evaluated in both with and without harness. For games marked with 1, evaluation for text-only models is not supported, as vision understanding is essential for decision-making The reported results represent averages over three runs, except for models or games marked with * which are based on a single run due to the high costs as of May 2025. "N/A" indicates non-applicable evaluation setting, as the specific model does not support image input.
Model
Horness
Sakoban
Super Mario Bros'
Totris
2048
Candy Crush
Ace Attorney"
claude-3 Gladde-3-S-ppn 3-5----net-tt222233 20244022
No
0.0300
154010::21
12.3a25
57.8
17.02181
1,0.00
Yes
0.0a00
1267. 7
14.7+12
108.2=58
106.0=534
2,0=00
3laude3-7-soomet- 1roommet-2025221999 (thinking)
No
0.0100
1430.0111223
1301000
114.2.72
126.3_M1
30.00
Yes
2.3115
1418.71800
16.3125
113.3:31
484.0_.8.v
700.00
deepsesk-rl
No
N/A
N/A
NIA
NIA
N/A
NIA
Yes
1.3
N/A
14,3506
105.2
447.3=45.1
0,0:00
gemini-225. 2.5-iii----.aabbbb 2.5.fasb-preview-00 preveew-eeeeeee 04-17 (chinking)
No
0.0300
1540 7
19.0346
A077
97.72
100gn
Yes
1,7+15
1395 D=240
16.3+32
106.6=53
334.7=665
4.7
gemini pro preview- 05-06 (thinking)
No
1.0.000
1025.3 3,445
12.3151
120.5159
8.0.00
Yes
4.3
1498. 24200
23.3806
117.3
416.3
7.7
gnok-3-mani--on beta (thinking)
Yes
No
5,7506
N/A
N/A
N/A
21.3+721
NIA
118.6=211
NIA
254.0=107a
N/A
0,0g00
NIA
4- 4-aaeeriiii 1 (7b- 128e-instruct-pp fp8
No
0.0+a.0
786.0=4422
11.7+12
41.6+11.
32.32414
0.0=u0
Yus
0.0
1468.74555
10.3+15
106.0=38
128.7=57.2
0,0=00
gpt- 1.441-2025505-1
No
0.0.00
19913
130011
9451110
101.0:11222
0.0.n0
Yes
0.0300
2126.3
13.7306
1057777
182.0
3,3
2p1--o-2024-11-200
Yes
No
0,0.00
0.0
3047.31-983
3
14,0156
14.7+21
70.4+152
106.7.18
147.3__154
59.0#54.6
0.0.09
0.0:00
01-2024-12-17 *
Yos
No
0.0=0.0
2,3
855.0:00
1434.0e00
13,0+00
35.0200
128.1=00
128,9-00
159.0a00
90.0++00
16.0:n0
3,0=00
o1-mini-20211 2021-09-122
No
N/A
NIA
NIA
N/A
NIA
Yes
1,3306
N/A
11.7212
114.0.37
48.02359
0.0.00
03-2025-04-16 +
Yes
No
8.0.28
2,0100
2266.7 11289
1950
42,0100
31,0100
128.2.000
128.6186
647.018a
106.0100
11.3141
8:0.00
04-mini--2255-44-16
No
1.3400
1348, 1477
15,0+36
97,6+2292
110.7=40.7
2,0
Yes
5,3
1418.0s1610
25,3:885
120.6:49
487.3
2.7
Random
0.0300
987.05415
10.2818
100
116.5-515
0.000
Human (avg)
9.7#39
4333.3-2714-
353.3:1194
115.5
283.3-J1.7
17.3484
rather than understanding. As a result, it's numerically hard to distinguish models given poor model performances and randomness inherent to games.
To address this issue, we design different levels of harness, as described in $ $31.22 to better differen- tiate model capabilities. Compared with three human players (graduate students given only brief explanation of the game rules), the gaming harness shifts models from mostly underperforming humans to performing at comparable level even surpassing them. Results in Table show that the harness leads to consistent and sometimes substantial gains for both reasoning and non-reasoning models, with complete model performance listed in Table 4 We provide detailed analysis for each module below.
Perception Modules. In grid-based games like Sokoban, vision harness helps models perform better by providing them with textual representation of game states read from the game backend. Reasoning models show substantial improvements, revealing that structured spatial inputs can unlock planning capabilities not expressed under image inputs. In games with more complex graphical interface like Super Mario Bros, we uses o3 to generate textual descriptions. However, this module plays a less significant role, as substantial gap remains between textual descriptions and the spatiotemporal information required for accurate decision makinn..
Memory Modules. Memory proves particularly impactful in temporally extended games. In 2048, non-reasoning models improve substantially with memory support, not only raises average scores but also reduces variance (Table 4), highlighting its importance in long-horizon planning. The effect is even more dramatic in Candy Crush, a game with complex temporal dependencies and delayed rewards. This reinforces memory's role in preserving context and strategy over time.
Combined Support. Table 2 shows that enabling both modules or one essential module often leads to stronger performance. Detailed statistical analysis (Appx. [B) shows that hamness can also pull scores far away from random play, statistically improve performance, and reduce variance for more stable benchmark results.
7

---

Published as a conference paper at ICLR 2026
Table Model performance on Sokoban, 2048, Tetris, and Candy Crush under various conditions. Scores are averaged across reasoning models (o4-mini, Gemini-2.5-Pro Claude-3.7-Sonnee and non-reasoning models (.lama-4-Maverick
Seaue-3.5-Soomer GPT-4o). ZS indicates zero-shot without any module support or memory prompt. See Table 4 for complete per-model results.
Model Group
Game
7S
+Memory
+ Perception
+Both
Reasoning Models
Sokoban
4.0
4.0
Non-rcasoning Models
0.0
0.0
0.0
0.0
Reasoning Models
2048
111.0
113.4
116.6
117.1
Non-rcasoning Models
57.6
102.5
71.1
107.0
Reasoning Models
Tetris
13.4
15.1
26.4
21.6
Non-rcasoning Models
12.9
12.4
13.7
13.9
Reasoning Models
Candy Crush
138.1
161.1
229.7
462.5
Non-reasoning Models
36.1
97.6
66.3
127.3
4.2 QUALITATIVE ANALYSIS
This section presents examples and qualitative insights into our harness design and discusses represen- tative failure cases across the evaluated games, highlighting clear directions for model improvement. All examples are presented in Appx. C
Image Perception. Current LLMs struggle with visual inputs. For instance, when given screenshot of game board from Tetris or Sokoban, they fail to convert into a textual grid or coordinate-ohject list (Fig. [5). revealing surprising limitation. In contrast, for 2048 the 4 x 4 board is small enough to fall within current models' visual perception capabilities. This pattern aligns with Table where the perception module benefits Tetris, Sokoban, and Candy Crush, but offers little advantage for 2048 for Gemini-2.5-PP and Claude-3. -.auude-.7-Soneet. 7-Sonnet. These foundational perceptual skills are preliminary steps toward image reasoning, yet remain underdeveloped in state-of-the-aae models.
Reflection on Failures. Without memory-harness support, models frequently fall into loops of repeating the same ineffective invalid actions, such as endlessly proposing impossible merges in 2048 or pushing boxes into walls in Sokoban. This is most evident in non-reasoning models (Table 2 which lack reasoning abilities identify valid moves. In contrast, with appended history and self-reflection mechanism in our memory module, models can recognize when a previous action is invalid (Fig. 77]. These results demonstrate that harness-lewe level memory provides an effective way to mitigate repeated failures and improve task efficiency.
We also compare reflection quality between GPT-4o and o3 (Appx. C.3
GPT-4o focuses on surface effects ("up increased empty spaces but no merges"), while o3 provides fine-graine analysis ("two 2's became a 4, two 16's became 32's, opening an extra cell") and ties it to long-term corner-dominance strategy. Unlike one-off Q& games require multi-turn reasoning where each move shapes the next. o3 excels by linking immediate outcomes to future plans, showing how local decisions drive long-term success.
Spatiotemporal Reasoning. In Super Mario Bros, models often fail to coordinate actions with the temporal dynamics of the environment. A correct policy must not only decide which action to take but also how long to sustain it. Two typical failure cases (Fig. 8 illustrate this limitation: when encountering a tall pipe, models frequently execute jump with too few frames, failing to gain the necessary height; and when approaching a gap, they often initiate the jump too early, losing forward momentum before reaching the far edge. These mistakes reveal weak spatiotemporal reasoning and highlight the difficulty of synchronizing spatial planning with temporal duration.
Long-Context. In Ace Attorney, which requires reasoning across long stretches dialogue, we observe frequent failures in maintaining consistency with earlier evidence (Fig. 9 For instance, even when a direct contradiction available in the evidence list, models sometimes fail to surface it and instead continue the cross examination without objection. These errors highlight the difficulty of Long-horizon retrieval and attribution: information may be present in the context window, but models do not reliably connect it claims.
8

---

Published as a conference paper at ICLR 2026
Benchmark Correlation Matrix
Feature
Feature
Male Chd erge
Lestare Catry
Lesterchianguage
appodedanch Faseg1
Regenshee
A pdyyet
7046-
201
Candy Crah
Cardy Crush
Act Altrrcy
Auc Altunnyy
Peature
Peature
1or covonnccoio
Holtlacch
myple
Enggeabed Bggoocelench
EMMAuren Phyyaa
2048
2040
Candy Drush
Cordy Crysh
Ice Anterre
Ace Art20ey
0.50
(a) Spearman Correlation among L.Game--ende and other benchmarks.
(b) Top-weight benchmarks under each feature after low-rank decomposition.
Figure 3: Correlation analysis and latent feature decomposition among benchmarks.
4.3 CORRELATION ANALYSIS
Games are designed to challenge human recognition skills, requiring combination of core LLM capabilities for strong performance. To study these connections, we collect results from 8 models on 20 established benchmarks spanning factual knowledge, physics, mathematics, coding, visual reasoning, language understanding, and puzzle solving, and analyze their correlations with gaming performance below. See Appx. [ for the complete model and benchmark list, and Table T6 for an additional analysis on how each game decomposes into core LLM skills through linear modeling.
Correlation Analysis. We calculate Spearman's 'ss rank correlation coefficient to assess alignment between model performance on games from LMGame- and widely-used benchmarks. Results from Fig. (lett) reveal positive correlations between several games and commonly used benchmarks. Sokoban correlates strongly with math and coding benchmarks, while Tetris and 2048 align with pattern recognition tasks such as EnigmaFval and NYT--Conecctoon. Candy Crush shows links to coding, hinting algorithmic reasoning, and Ace Attorney correlates with LiveBenchh-Language reflecting narrative understanding. Super Mario Bros is excluded, as the spatiotemporal reasoning it requires unique compared with the listed benchmarks, with further details in Appx. F2
Latent Ability Decomposition. To uncover relationships between the benchmarks and the capabilities of the model, we apply a low-rank matrix factorization to the benchhbben performance matrix. This decomposes each LLM as a vector in latent ability space and each benchmark. including our games, as a sparse, weighted combination of these abilities. As shown in Fig. 3b) the four features align with language and multi-task knowledge, coding, symbolic and puzzle-solving skills, and physical reasoning, respectively. The results show our games require different subsets of these latent abilities. Sokoban emphasizes symbolic and physical reasoning (Features 3 and 4), while Ace Attorney strongly engages long-context language reasoning (Feature 1). Teiris and 2048 mainly represent spatial reasoning (Features 3), and Candy Crush reflects visual pattern recognition with moderate ties to coding (Features 2 and 3). This suggests that games cover compositional capabilities rather than isolated skills.
4.4 OTHER LIMITATIONS AND DISCUSSION
Overall, our design is effective in distinguishing models, identifying key failure modes, and evaluating core LLM capabilities. However, two limitations still remain. (1) Performance variance continues to be high in partially observable games like Super Mario Bros. Notably, human performance also exhibits higher variance in this game, suggesting that the variability arises from the game's 's stochastic dynamics. (2) The computational cost remains substantial, generating actions could result in long reasoning chains that are highly repetitive across multiple turns (Appx. [B.4), highlighting the need for improved model capabilities and more efficient inference reduce operational costs.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We introduce LMGame-Bench Bench, the first agentic benchmark for evaluating LLMs on games with and without garning harness support. LMGame-Bench leverages a gaming harness composed of agentic modules to better distinguish state-of-the-art models. Our benchmark identifies and addresses data contamination through a series of mitigation, and reduces prompt variance by integrating two-stage prompt optimization using DSPy. Additionally, show that AiGGame-Bende can be regarded as a composition of core LLM capabilities, supported by comprehensive quantitative analysis. By identifying the key causes of model failures, qualitative analysis provides clear insights for advancing model development.
ETHICS STATEMENT
All authors have read and adhere to the ICLR Code of Ethics) This work does not involve human subjects, sensitive personal data, or experiments with potential to cause harm. No confidential or proprietary data were used. The methods and experiments are designed in compliance with principles of research integrity, fairness, and transparency. We acknowledge that any potential societal impacts, including limitations or biases of large language models, are explicitly discussed the paper, and all conclusions are the sole responsibility of the authors.
REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our results. Detailed descriptions of the models, training procedures, and hyperparameters included in the main text and Appendix. All datasets used are publicly available, and the preprocessing steps are fully documented. Ablation studies are provided to validate robustness of results. These resources collectively allow independent researchers to verify and build upon our work.
REFERENCES
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human, 2024. URL httpttttttttai.....//
abs/2412 08164
Aider Team. Aider I1m leaderboards. httppp///aaaerr.ce//////////eeee cessed: 2025-05-14.
Anthropic. Claude's extended thinking claude plays pokémon. https:/www.anthrggg
Chtm://www//iiibblleeeeeeeeeette 2025.
Anthropic. Vision claude does: Limitations.
cttp://bccc.cllau..ccoooooo/// 2025.
huild-with-e http::/wwwwt.ceeeeeeee//oooooooa
BigCodeBench Team. Bigcodebench leaderboard. https::/bbigcode-bencth
Accessed: 2025-05-14.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint (oXXvv::60601:55 2016.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemej Kos, and Dawn Song. The secret sharer: Evalu- ating and testing unintended memorization in neural networks. arXiv preprint axXiv:1822.02333 2018.
Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXivv2507.06261 2025.
10

---

Published as a conference paper at ICLR 2026
Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. Gamebench: Evaluating strategic reasoning abilities of Ilm agents. arXiv preprint arXiv:2406.0661 2024.
Joseph Culberson. Sokoban is pspace-complete. 1997.
Erik D. Demaine, Susan Hohenberger, and David Liben-Nowell. Tetris is hard, even to approximate. In International Computing and Combinatories Conference (COCOON), volume 2697 of Lecture Notes Computer Science, pp. 351-365 Springer, 2003. URL /ttps::/doi..rg:
1007/3-540-4512--
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models, arXiv preprint ar XrXiv:2402.15938, 2024.
Dorit Dor and Uri Zwick. SOKOBAN and other motion planning problems. Computational Geometry, 13(4):215-225 1999.
Gene V. Glass. Primary, secondary, and meta- antal-a of research. Educational Researcher, 5(10): 3-8, 1976. doi:
William S. Gosset. The probable error of a mean. Biometrika, 6(1):1 1908.
Stefano Gualå, Salvatore Leucci, and Emanuele Natale. Bejeweled, candy crush and other match-three games are mp-)hard. arXiv preprint arXiv:1403.5484, 2014.
1403.5484
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang. Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.0544: 2025. URL
2501. 05444
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Pranav Arora, Steven Basart, Dawn Song Tang, et al. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0330/ 2021.
URL https::////.m. //tp:i/aaivvvvviv
org/ 2oos/2003 03300
Jack Hopkins, Mart Bakler, and Akbir Khan. Factorio learning environment. arXiv preprint arXiv:2503.09617, 2025.
Lanxiang Hu, Qiyu Li, Anze Xie, Nan Jiang, Ion Stoica, Haojian Jin, and Hao Zhang. Gamearena: Evaluating Ilm reasoning through live computer games. arXiv preprint arXiv:2412.0633 2024.
Frederikus Hudi, Genta Indra Winata, Ruochen Zhang, and Alham Fikri Aji. Textgames: Learning self-play text-based puzzle games via language model reasoning. arXiv preprint arXiv:2502.188 18431, 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Maurice Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-92) 1938.
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq. Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compiling declarative language model calls into state-of-the of-the-art pipelines. In The Twelfth International Conference Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2026
Alex Lau-Zhu, Emily A Holmes, Sally Butterfield, and Joni Holmes. Selective association between tetris game play and visuospatial working memory: A preliminary investigation. Applied cognitive psychology, 31(4):438 445, 2017.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, al. Agentbench: Evaluating Ilms agents. In ICLR, 2024.
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: multimodal generalist gui agent with native reasoning and reflection. In JCML 2025 Workshop Computer Use Agents.
LiveBench Team, Livebench leaderboard.
aathheattisssssaaaaaaaaa a
acceseed 2025-05-14.
Accessed:
LMSYS Org, Chatbot arena leaderboard, 2025-05-14.
aaw.areennaa/leederbboooodd /aaer Accessed:
Lech Mazur. Nyt connections benchmark: Evaluating ilms with extended word association puzzles. Accessed: 2025-05-14.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Manuel Mosquera, Juan Sebastian Pinzon, Manuel Rios, Yesid Fonseca, Luis Felipe Giraldo, Nicanor Quijano. and Ruben Manrique. Can Ilm iin-augmented autonomous agents cooperate?, an evaluation of their cooperative capabilities through melting pot. arXiv preprint arXiv:2403. 1381, 2024.
Muhammad Umair Nasir, Steven James, and Julian Togelius. Gamettaversalbenchmaak, Evaluating planning abilities of large language models through traversing 2d game maps. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL httptttpps///peeeeeeieeee
net./ for.m/ormmmmm www.rrmmmiid=AAIIAOOOZZ.
Kate Olszewska and Meg Risdal. Rethinking how measure ai intelligence. https://blos 2025.
OpenAl. Openai o3 and 04-mini system card, April 2025. URL httptt::://ppeaai.com index/ o
Accessed: 2025-05-10.
Davide Paglieri, Bartlomiej Cupial, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kuciński, Lerrel Pinto, Rob Fergus, et al. Balrog: Benchmarking agentic IIm and vlm reasoning on games. ICLR, 2024.
Bhrij Patel, Souradip Chakraborty, Wesley A. Suttle, Mengdi Wang, Amrit Singh Bedi, and Di- nesh Manocha. Aime: Ai system optimization via multiple I1m evaluators. arXiv preprint arXiv:2410.03131 2024. /tL/ http://bs//24.... 03131
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Summer Yue, Alexandr Wang, and Dan Hendrycks. Humanity's last exam: Benchmarking ai on grade school science olympiad exams. arXiv preprint arXhv:2501.14244 2024.
org/abs 200.. 14249
Dan Qiao, Chenfei Wu, Yaobo Liang. Juntao Li, and Nan Duan. Gameeval: Evaluating llms on conversational games. arXiv preprint 2300.:2308. /0032, 2023.
Shi Qiu, Shaoyang Guo, Yifan Zhuo Yujie Wang, Zhen Li, Yifan Zhang, Yujie Wang, Zhen Li, Yifan Zhang, Yujie Wang, Zhen Li, and Yifan Zhang. Phybench: Holistic evaluation of physical perception and reasoning in large language models. arXiv preprint arXiv:2504.1600 2025. URL https:://axxiss wwt.://wwwwg.oooooooooooooooso/ /504 16074
12

---

Published as a conference paper at ICLR 2026
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: Graded physics question answering benchmark for large language models. arXiv preprint arXiv:2311.12022 2023. URL https:
//arxiv.org/aae abs 2311. 12022
Jussi Rintanen. Complexity of planning with partial observability. In ICAPS, volume 4, pp. 345-354, 2004.
Meg Risdal. Introducing kaggle game arena: Watch models compete in complex games providing verifiable and dynamic measure of their capabilities, https: www.::wwwkg......o.. og// arena 2025.
introducing--aame
Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Genewein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations. ICML, 2025.
Scale AI. Enigmaeval benchmark leaderboard.
htpps//scale..com//eddbbboe
eni gma _ Accessed: 2025-05-14 14.
Scale AL Humanity's last exam leaderboard.
http:://saaee.cmm/eedddbbbbb
humanitys, [xam_exam________ b. Accessed: 2025-05-14.
Scale AL. Humanity's last exam leaderboard
humanitys Laslss_lstt___eeeeeeeee _____exm_____ _exxa__ext__on___ c. Accessed: 2025-05-14.
Scale AL Multichallenge leaderboard.
httts::/sscale..om/eeee mul t tethacchleeett d. Accessed: 2025-05-14.
Scale AI. Vista: Visual language understanding benchmark leaderboard. hittps://sce http///cale.coom com/ e. Accessed: 2025-05-14.
Teaderboa rateeebbbrrvviii sual lawpaaaalaaaaaeeeeeeeee
Jiajun Shi, Jian Yang, Jiaheng Liu, Xingyuan Bu, Jiangjie Chen, Junting Zhou, Kaijing Ma. Zhoufutu Wen, Bingli Wang, Yancheng He, Liang Song, Hualei Zhu, Shilong Li, Xingjian Wang, Wei Zhang, Ruibin Yuan, Yifan Yao, Wenjun Yang. Yunli Wang, Siyuan Fang, Siyu Yuan, Qianyu He, Xiangru Tang, Yingshui Tan, Wangchunshu Zhou, Zhaoxiang Zhang, Zhoujun Li, Wenhao Huang. and Ge Zhang. Korgym: A dynamic game platform for IIm reasoning evaluation, 2025. URL htttps://arxis wttpaawwwwwwv...//aaaeeeeee wtt./or////bbooooo 14552
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359 2017.
Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed- Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier 1lms. arXiv preprint arXiv:2501.17399 2025. httttttts///xxiii. wwwpeiii..rr//aoooo atw.aabb../.... 22000 17399
Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38 (3):58-68, 1995.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024.
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: controllable world of apps and people for benchmarking interactive coding agents. arXiv preprint 14711:207800 2024.
Vals AI Aime benchmark leaderboard. aime 2025-05-09
https: "ttts://www.vals.al/ al www..hcaas.al//eccmaaaaa.
Accessed: 2025-05-14.
VALS AI. Gpqa benchmark leaderboard.
Sttp::/www.als.al/eeee.
a
ggga- 005-9920255
Accessed: 2025-05-14.
13

---

Published as a conference paper at ICLR 2026
Vals AL Math 500 benchmark leaderboard.
mmath500-05-0922222 a. Accessed: 2025-05-14.
Vals AI. Mmmu benchmark leaderboard.
mmmu -
10mmu-05-0355 0000050-0990 05- b. Accessed: 2025-05-14.
VALS AI. Mmlu-pro benchmark leaderboard.
pro 1ro-05-09-2025: 2025. Accessed: 2025-05-13.
Clinton J. Wang, Dean Lee, Cristina Menghini, Johannes Mols, Jack Doughty, Adam Khoja, Jayson Lyneh, Sean Hendryx, Summer Yue, and Dan Hendrycks. Enigmaeval: A benchmark of long multimodal reasoning challenges. arXiv preprint arXiv:2502.088575 2025. URL https:
arxiv. www.ii.rrg/bbs.. 2502 08859
Xinyu Wang, Bohan Zhuang, and Qi Wu. Are large vision language models good game players? In The Thirteenth International Conference on Learning Representations.
Nicholas R Waytowich, Devin White, MD Sunbeam, and Vinieius G Goecks. Atari-gpt: Investigating the capabilities of multimodal large language models as low-level policies for atari games. arXiv preprint arXivv2408. 15950, 2024.
William Webber, Alistair Moffat, and Justin Zobel. A similarity measure for indefinite rankings, In Proceedings of the 19th international conference on World wide web, pp. 577-586. ACM, 2010.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun. Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: challenging, contamination-free Ilm benchmark. arXiv preprint arXiv:2406. 19314, 2024. URL wwwph:ttts/.ooc...... http:////aaxxv...raaaa//////////
19314
Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. Smartplay: A benchmark for Ilms as intelligent agents, arXiv preprint arX1v:2310.0157,, 2023.
Junlin Xie, Ruifei Zhang. Zhihong Chen, Xiang Wan, and Guanbin Li. Whodunitbench: Evaluating large multimodal agents via murder mystery games. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a. URL https://
openreview.nee. wtwp::/eeeeetooooreeeeeem
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li. Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks real computer environments. Advances Neural Information Processing Systems, 37:52040-52001 2024b.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023.
URL
org. abs /etes/231.... 16502
Ahmad Zaky. Minimax and expectimax algorithm to solve 2048. 2014.
Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, and Ofir Press. Videogamebench: Can vision-language models complete popular video games? arXiv preprint 250.::2505..8.34 2024.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui ti-focused agent for windows os interaction. In Proceedings the the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 597-629 2025.
14

---

Published as a conference paper at ICLR 2026
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tranyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 2023.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li. Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujum Cheng. Jiawei Liu, Qian Liu, Zijian Wang. David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024. URL https: / argiv.org abs 0///2406...50 15877
15

---

Published as a conference paper at ICLR 2026
A
DESIGN DETAILS
A.1 METRICS
Because games well-designed, we utilize their built-in metrics quantify models' proficiency. For each game, we choose the single score that most faithfully reflects a model's capability, then transform and normalize it onto continuous, linear scale. This curation ensures that evaluation can sensitively capture performance differences and supports consistent statistical analysis, and we call it raw scores (Table
In addition to the raw scores, we introduce procedural progress score to capture the critical informa- tion games, such obstacle-passed count, box placement, and tile milestones, and report the results in Table Because procedural progress scores emphasize only key game states, they can be coarse and sometimes insufficiently discriminative across models. Hence, the main paper reports raw scores, while procedural metrics are provided here for completeness. Here are the detailed design of metries.
Sokoban:
Raw score: Total number boxes pushed onto targets, summed over all levels, until the first deadlock.
Procedural progress score: The highest level reached and the number of boxes success- fully placed in the last level.
Super Mario Bros.:
Raw score: Cumulative horizontal distance traveled by Mario (in game units) across all levels, until all three lives are lost or the final level is completed.
Procedural progress score: The total number of enemies, pipes, and gaps successfully passed, which captures model's ability to navigate structure rather than just distance traveled. Note that none models have entered the wrap zone in level 1-2 (only one human evaluator entered).
Tetris:
Total reward equals pieces placed plus ten times the number of lines cleared, measured until game over. Each placed piece yields +1; each cleared line yields + +10. Procedural progress score: The number of lines that have been cleared.
Raw score:
2048:
Raw score: Sum of all merged tile values (e.g. merging two 2's yields +4), recorded until the board stagnates (no merges or moves that change the board for ten consecutive turns). We then report
Score2018
10 x logz (total merged sum).
Procedural progress score: The max-tile milestones (e.g. 64/128/256/512), reporting the majority score.
Candy Crush:
Raw score: Total number of candies eliminated over fixed 30-move session.
Procedural progress score: The same as the Linear progress score.
Ace Attorney:
Raw score: Total count of correct actions (evidence submissions, dialogue choices, etc.) across all case levels, measured until five incorrect decisions (lives) have been used.
Procedural progress score: The highest level reached and the number of courtroom actions completed within that level.
16

---

Published as a conference paper at ICLR 2026
Table Model performance measured by average finished game progress.
Humoa
Sokakan
Staee
2048
Candy Crass
Ace Altommy
(igi
clande3: 3.5 sennet 20241022
Yoi
So
Lt 0.0 Box
Li 1--00.0a
90
7.3
0:77
0.5
12
15
125
16
105.0
17.0
L1- Siep
L1-2.0 Step
Su
Li 110 Box
83
Step
desgeeeker
You
No
18 Bare
NIA
NiA
VIA
NIA
NiA
N0A
30A
14
NGA
174
441.1
NOA
11- 0L0 Siop
NAA
acmtn persew (dhinling)
Su
gemin-- pro 05 do (ninklng)
You
No
13 1.5 Box
Li 1. Box
60
K3
0.51
875
13
16
256
255
415.3
177.3
L2 3.0 Siep
122- 2.0 Step
grok
thinking?
36/
Barta- Tanta---rmone A-moooncc---- mick- h- 138-iiooma--ig
Yoi
So
Li 0.0 Box
Li 0.0 Bux
K3
4.5
0.78
0.39
12
125
16
126.7
32.3
11
L1 0.0 Siep
0.0
Step
g24-
101.0
2004-
Yei
No
Li 0D Box
Li
Bux
10.7
53
0.51
1.08
13
14
129
12
147.3
50.0
L1 0.0 Stop
L1
0.0 Siep
of 201-----------
Yes
No
L1 1.3 Box
NIA
NIX
N6
N9
NDA
NA
10A
12
N/A
255
NAA
48.0
11 0.0 Stop
NVA
L2
No
L. 1.3 Bax
73
8:8
15
756
1807
1.1 2.0 Siop
Yoi
L4 0.5 Box
83
0.1
0.38
15
512
487.3
L1 4.0 Step
Stop
Hiron
15 1.7 Box
230
235
246
281-1
133 5.
A.2 PROMPTS OF HARNESS MODULES
A.2.1 MEMORY MODULE
We design the memory module to consist of both game history and self-reflection. Below we provide the prompt for self-reflection using 2048 as example. Prompts for other games follow the same overall structure, with game-spectte specific instructions adapted to their respective rules.
Memory Module (Game: 20488
system_prse prompt:
You are an analytical assistant for 2048 Al agent.
Your task is to generate brief, insightful reflection on the game state changes and the effectiveness of actions.Focus on strategic insights and patterns that would help the agent make better decisions. Keep your reflections short, precise, and actionable.
user prompt:
Please analyze the following 2048 game states and actions generate brief reflection:
Previous Game States and Actions:
(PREVIOUS GAME HISTORY)
Focus your reflection on:
1. How the game state changed after the last action
2. Whether the action
effective for the situation
Patterss or issues to aware
4. Any strategic insights for future actions
Keep your reflection under 100 words and focus only on the most important insights.
A.2.2 PERCEPTION MODULE
Our benchmark allows toggling the perception module on or off, enabling controlled evaluation of a model's image perception capability. As described Section 3.2.1 for board games, if we want to bypass perception entirely and only evaluate reasoning, the perception module directly reads the game backend to obtain precise textual states. However, for Ace Attorney and Super Mario Bros., designing reliable rule-based extractors is non-trivial due to their rich, nuanced visual scenes. Therefore, we adopt o3, a state-of-thhe-ar VLM at the time of paper writing, to generate structured textual descriptions of the game state, with prompts shown below. As illustrated in the prompts, the VLM is used solely for parsing the game state, including identifying objects, dialogue, UT components, and available options.
17

---

Published as a conference paper at ICLR 2026
Perception Module (Game: Ace Attorney)
You now playing courtroom games. Carefully analyze the current and provide the following information.
1. Game State Detection Rules:
Drass-Examination mode is indicated by ANY of these:
* Green dialog
* Options sereen
An evidence window visible in the middle of the screen
Conversation mode is indicated by:
* Dialog
be any color (most commonly white, but also blue, red, etc.)
* Or none of Pross-Examination indicators are present
2. Dialog Text Analysis:
Look the bottom where dialog appears
Note the color of the dialog
(grren/white/blucrred)
Determine the current dialog is full sentence
Extract the speaker's ss name and their dialog
Format must be exactly: Dialog: NAME: dialog text
Scene Analysis:
Describe any visible characters their expressions/poses
Describe any other important visual elements or interactive UI components
You MUST explicitly mention:
If the current dialogue incomplete and there is down or right arrow), then set Dialogue Continuation "Yes'; otherwise, to 'alse"'.
the bottom of text box (NOT left
Whether there is exclamation mark icons in the upper right corner
The exact UI elements present at the upper corner 'L Press' and 'Present R' if in cross examination mode Whether there is evidence window visible
options appears, you need mention:
The text each option order from top to bottom
Which one is currently selected (Use the pointing hand icon at the beginning of the line to determine the selected option. Do NOT assume the bottom option is selected by default selection depends entirely the hand icon.)
evidence window visible, you need mention:
Name of the currently selected evidence
Whether this is the evidence you intend to present
Format your response EXACTLY
Game State: <'Cross-xxamination or 'Conversation's
Dialog: NAME: dialog text
Dialogue Continuation: <'Yes' or 'No'>
Options: optionl, selected; option2, not selected; option3, not selected
Evidence: NAME: description
Scene: <detailed description including dialog color, options text (if exist), blue bar presence, UI elements. evidence window status and contents, and other visual elements>
18

---

Published as a conference paper at ICLR 2026
Perception Module (Game: Super Mario Bros.)
Analyze this frame from Super Mario Bros. with the 5x5 grid overlay and identify game elements each grid cell. Your task is to identify locate game elements in 5x5 grid overlay on the screen.
Identify the following elements and their approximate positions in (x.y) grid coordinates (0,0 top-left to 4,4
bottom-right):
Mario (player character)
Pipes (green obstacles)
Goombas (brown mushroom enemies)
Koopas (turtle enemies)
Gaps/pits (areas where Mario fall)
Question blocks (blocks
can hit)
Brick blocks (breakable blocks)
Coins
Power-ups (if visible) Flag pole (end of level)
Your response must be in valid ISON format with the following structure:
"mario": ("x": int, "y": int).
"environment":
"pipes": [["x": int, "y"; int, "height": "small medium large")],
"goombas": [("x": int, "y": int, "distance": "very_closs cloolllse I close medium far")
(OMIT THE STATE FORMAT FOR OTHER OBJECTS)
1.
ggame_ sttteesstttttttt
"right left stationary". "mario " "small big fire invincible". "immediate threats": ["goomba" "koopa" "gap" "pipe"]. caistacles_aheat" I"goomba") "koopa" "gaap" "pipe"]
Ensure all coordinates integers within the 0-4 range for the 5x5 grid. an element is present, include it as an empty array or as appropriate. For immediate threats, only include elements pose an immediate danger to Mario. Your output should be ONLY the JSON object, without any surrounding text or markdown.
B HARNESS EFFECTIVENESS: QUANTITATIVE ANALYSIS
In this section, we employ statistical methods to explore the effectiveness of applying all harness combined in bringing improvements to model performance. Given the tiny sample due to the cost of running latest models, results should be considered preliminary. We also present the complete result of gaming harness ablation in Table ass complementary of Table 2
Given that gameplay inherently involves random noise, we aim for our harnessed model performance to be both noise-resistant and consistent, enabling clearer assessment of the model's true ability. Accordingly, we make two key claims: (1) harnessed evaluations better isolate model ability from game randomness; and (2) harnessed performance is more consistent and robust to random variation.
B. SEPARATION FROM RANDOM BASELINE: GLASS's 8 EFFECT SIZES
To quantify how far harnessed and unharnessed model evaluations depart from random play, we simulated 30 random runs per game estimate the baseline mean Xrand and standard deviation 8rand- Glass's 8 for each model-game-conditoon is then (Glass 1976):
X mudel Xyand
8
8rand
Because Sokoban and Ace Attorney exhibit zero variance under random play, we exclude them, focusing the four remaining games. Importantly, harnessed yield positive 8 in 38 out of
19

---

Published as a conference paper at ICLR 2026
Table Game scores of different models in Sokoban, 2048, Tetris, and Candy Crush under various conditions. ZS indicates zero-shot without any module support memory prompt.
Model
Game
ZS
+Memory Only
+Perception Only
+Both
1i-mini-2222-04-111
Sokoban
1.34
1.34066
5,3:21
5.34112
gemini-22.5ppo-05-225
1.0:0.00
1.0.0.0
6.0:220
4.3:1:.6
claude-3-7-sonnet
0.0::0.0
0.3
0.7:106
2.3::155
mlave---mverrick
0.0=000
0.0=0.0
0.0+000
0.0+0.0
claude-3-5-sonnet
0.0+0.0
0.0=0.0
0.0+0.0
0.04:0.0
gpt-4o-2024-1144
0.0+0.0
0.0=0.0
0.0+0.0
0.0:00.0
44-mini-2025-06
2048
97.6=29.2
115.1
117.0=6.4
120.6=49
gemini-
120.5::3.9
118.0=8.5
117.4=5.8
117.3+5.9
claude-3-7-sonne
114,2#7.2
107. 10.11
115,3=2,3
113,3#3.1
llama-4-maverick
44.6
98.1
13.7=1556
106.0
claudc-3-5-sonnet
57.8
102.5
66.3199.
108.2:158
spt-to-2024-11220
70.4
107.0
73.325.4
106.7
04-mini-2025-04-16
Tetris
15.0+3.6
14.7=3.2
38.0=113
25.3
gemini-2.5-proo02225
12.3433.1
15.0=1.0
21.3
23.3-1006
claudc-3-7-sommet
13.0++0.0
15.7=3.1
20.0+3.6
16.34223
llama--maverick
11.7:112
10.3=
8.7.1:.5
10.3
claude-3.5-sonnet
12.3:12.5
13.3
14.3
14.7:112
8pl-4o-2024-11-20
14.7:12.1
14.0=2.0
1..0
16.71335
04-mini-2025-04-16
Candy Crush
110.7:149.7
202.3
320.0
487.3
gemini-22.5-pro-02225
167.3
93.7
133.5
416.3..6.8
claude-3-7-somnet
126.3
187.31151.6
270.3
484.0 1.5.7
Ilama 4-maeeric
32.3==4144
123.3+83.9
110,0+23.4
128.7=57.2
claude-3--soonnet
17.0=18.1
120.3::415
10.7
106.0#53.4
gpt-4o-2022-11--
59.0=54.6
49.3+38.4
78.3=24.7
147.3+534
40 model-game pairs, compared to only 26 of 40 for unharnessed runs demonstrating that harnessed evaluations are far more consistently pulled away from the random baseline. Across those 40 pairs, harnessed runs outperform unharnessed in 29 cases (72.5%), with overall averages
Buarness 3,334, 8n 0.750, A &parness 5np 2.585
This demonstrates that the harness pulls model scores substantially farther from randomness than unharnessed evaluations.
B.2 DIRECT COMPARISON OF HARNESSED VS. UNHARNESSED: PAIRED-SAMPLE T- T-TEST
Beyond Glass's 8, we directly compare harnessed and unharnessed mean scores via paired-sample I-tests (Gosset 1908) across our ten models for each game. All six games exhibit positive mean
improvements under harnessing: for five of them Candy Crush (+217.50 points, t(9) 4.22, p
0.0022). Sokoban (+1.97 points, t(9) 3.02, p 0.0144), 2048 (+17.81 points, t(9) 2.36, p
0.0424), Ace Attorney (+3.20 points, ((9) = 2.36, 0.0427), and Tetris (+5.60 points, ((9) 2.27, p = 0.0490)-the increase statistically significant at 0.05. Super Mario Bros. shows smaller, non-significant gain (+289.10 points, ((9) 1.45, p 0.1806).
Figure displays the full distribution of per-model score differences (Harness Harness) for each game, with boxes indicating the interquartile range and whiskers covering 1.5x IQR. Candy Crush and Sokoban show the largest median gains, while Super Mario Bros. exhibits the greatest spread, underscoring its high inherent stochasticity.
B.3 CONSISTENCY OF PERFORMANCE: COBFFICIENT OF VARIATION ACROSS CONDITIONS
Because our per- per-model samples are small (n zs 3), raw variance comparisons can be misleading. We therefore compute the coefficient of variation (expressed percentage)
20

---

Published as a conference paper at ICLR 2026
Table Glass's per Model, Condition, and Game (rounded 3 decimals)
Model
Cond.
2048
Candy Crush
SMB
Tetris
With
0.992
-0.204
1.763
2.524
claude-3- --sos---ttttte-022441
Without
-5,446
-1.933
2.593
1.215
With
1.648
7.140
2.223
3,459
claude-3--sxommee222250000 (thinking)
Without
1.752
0.191
2.258
1,589
With
0.787
4.238
2.151
3,459
gemini 2.5 Beiiss-preview-Oee (thinking)
Without
0.883
-0.366
2.595
4,955
With
2.148
5.825
2.466
7.386
gemini 2.5 ww..-pr--previewuue 00--6 (thinking)
Without
2.558
1.182
1.215
With 0.675
1,273
4.382
1.963
gpt-4.1-2025-04-14
Without
-0.762
-0301
3,970
1.589
With 0.793
0.599
4.141
2,150
gpt-4o-2024-1--20
Without
-3.833
-1.117
1.033
2.524
0.707
0.236
2.376
0.093
llama-4-maverick-I 1----188-----tsttt---p-
Without
-7.224
-1.635
0.293
0.841
With
3.631
0.826
0.504
13,930
01-2024-12-17
Without
3.530
-0.515
2.270
1.589
With
3.516
10.306
8,404
17.856
03-2025-04-16
Without
3.541
-0.204
3,859
11.686
With
2.577
7.204
2.313
8.508
04-mini-2025-04-16
Without
-0.66
-0.113
2.009
2.711
Table 6: Paired-Saan Paired-Sample t- t-Test Results for Harnessed vs. Unharnessed Mean Scores
Game
A Mean
%A
(df 9)
Candy Crush
+217.50
+224.8%
4.22
0.0022
Sokoban
+1.97
+537.5%
3.02
0,0144
2048
+17.81
+22.4%
2.36
0.0424
Ace Attorney
+3.20
+123,1%
2.36
0.0427
Tetris
+5.60
+27.1%
2.27
0.0490
Super Mario Bros.
+289.10
+19.3%
1.45
0.1806
p 0.05, ** p 0.01
CV
$ X
x 100%
for each model-game-conditton (Random, With Harness, Without Harness) measure relative dispersion around the mean. Sokoban and Ace Attorney are excluded (zero random variance), and models with only a single run (ol, o3) are omitted. Table 7 lists the rounded CV values (in %) for the remaining ten models across four games.
Across the four games, harnessed runs yield lower CV than random play in 8/8 cases for 2048 (100.0%), 6/8 for Candy Crush (75.0%). 6/8 for Super Mario Bros. (75.0%). and 5/8 for Tetris (62.5%). Comparing harnessed to unharnessed, CV is lower under harness 6/8 for 2048 (75.0%), 8/8 for Candy Crush (100.0%), 4/8 for Super Mario Bros. (50.0%), and 4/8 for Tetris (50.0%). Overall, out 32 valid model-game game pairs, (78.1% (8) have smaller CV under harness versus random, and 22 (68.8%) have smaller CV under harness versus unharnessed. These results indicate that the harness not only elevates mean performance but also lowers the coefficient variation i... reduces relative score dispersion--hiie which yields more stable, reliable assessments of model ability.
21

---

Published as a conference paper at ICLR 2026
Performance Differences: With Harness Without Harness
Candy
Soboben
5.0.002 Fear a 21.50 als 226-86
p-
Moan m
22,0h
1300
Fear a 3.20
Hewn & 5.00
Nean 6 200
1X00
J00h Sebeere
Sigifficncc:
p
p 0.5. sigrificant
Figure 4: Distribution of paired score improvements (Harness No Harness) across models for each game.
Table 7: Coefficient of Variation (CV %) by Model, Condition, and Game
Model
Condition
2048
Candy Crush
Super Mario Bros
Tetris
Random
Random
7.798
44.183
41.995
17.535
chaude3--5xmmee2222211122
With Harness
5.351
50.372
38.18 18.184
7.873
Without Harness
28.377
106.371
1.409
20.405
2aadde-37-xomme222225222 (thinking)
With Harness
2.766
11.090
16.546
14.139
Without Harness
11.341
0.000
gemini-2.55 2.iimi-----lsshh--revveeee (thinking)
With Harness
4.962
19.581
17.210
19.681
Without Harness
17.034
2ommini-.-prr--mreviee----.. (thinking)
With Harness
4.993
1.635
13.573
2.474
Without Harness
36.599
43.228
gpt-4.1-2025-04-11
With Harness 6.605
15.763
83.636
4,225
Without Harness
17.945
119.047
51.144
13.323
gpt-4o-2024-11-20
With Harness 3.266
36.249
25.797
25.754
Withour Harness
21.545
92.509
63.796
14.193
llama---maverick- 1 77b-128e-instrut--fp
With Harness 3.556
44.459
37.835
14.783
Withour Harness
26.470
128.055
58.857
9.897
14-min--2025-04-U1
With Harness
4.030
40.634
11.116
33.572
Without Harness
29.880
44.866
13.210
24.037
B.4 COST ANALYSIS AND SCALABILITY
Agentic evaluations are token-billed; major providers charge per input/output token, so longer contexts and extra tool calls increase spend Running MMGame-Bench remains more controllable than full- scale software-agent benchmarks (e.g., SWE-bench) under comparable model sets and query counts. Tables p summarize order-of- mfagnnituee totals and per-step estimates from cached logs. Overall, with-harnes costs exceed no- no-harness due to (i) longer contexts from appended trajectories and (ii) an additional reflection call in the memory module; optional perception may add a vision-to-text call.
Scalability levers. Strategies include bounding or summarizing trajectories, triggering reflection every steps or after invalid moves, caching and de-duplicating perception frames, routing perception
https://ai.co dww/ mamiiiiiappil -api /dccs/
bi billing http::://ww..ntttttt: hww.///www////eemmm meeas/mm-coo Io-coteext
22

---

Published as a conference paper at ICLR 2026
Table 8: Order-of-magnitude benchmark costs (illustrative), from estimated queries and average tokens per query.
Benchmark
Est. q
Model
Avg.
Avg. out
Est. $
SWE-bench
2,294
o3
100k
S830
SWE-bench
2,294
sonuee-oonne---0
100k
20k
$1,380
SWE-bench
2,294
gemini-2..-pm
100k
20k
$750
LMGame-Bench
2,000
o3
20k
10k
$80
(iGamm-BBencl
2,000
claude-sonnet-4-0
20k
10k
$130
LMGame-Bente
2,000
gemini-2.5-pro
20k
10k
$60
Table 9: Estimated per-step costs (USD) from cached logs for two games under four harness settings.
Model
Game
No
Harness (pere)
Harness (mem)
Harness (both)
harness
dlaude-3-7-sonnet--ttee
candy_crush
0.011
0.021
0.047
0.048
Mlaud--37-soonee-latee
twenty_forty ewenty_forty_eigh
0.015
0.026
0.029
0.031
gemini-2.5-pp
candy_crush
0.005
0.009
0.019
0.020
gemini-2.5-pro
twenty coruttyfortt_ue forty_eighht
0.016
0.019
0.020
0.021
gpt-4.1
_cndy__crshh
0.008
0.014
0.029
0.025
gpt-4.1 ol ol
twenty everty_forty__ar [eoro_orttylllle _eight
0.008
0.014
0.016
0.016
candy_crust twenty_ harlly_fotty__tte
0.055
0.105
0.212
0.218
0.054
0.102
0.114
0.121
and memory tasks to smaller models while reserving frontier models for action decisions, batching evaluations and reusing prompts across seeds, and early-stopping low-performing runs using interim thresholds.
C QUALITATIVE ANALYSIS
Here give examples models' performances to support the qualitative analysis in Section 4.2 providing insights into our harness design and discussing representative failures.
C.1 IMAGE PERCEPTION
C.I.1 BOARD GAMES
Figure provides examples of image perception failures on Gemini 2.5 Pro with thinking enabled. Note that similar failures have been observed from other leading LLMs. including GPT-4o and Claude 3.7 Sonnet. These examples show that even advanced multimodal LLMs struggle with basic image perception of game boards. Such perceptual skills are foundational for higher-Ievel reasoning on masge-based grid games, yet they remain unreliable in practice.
Example (Tetris). The task is straightforward: given screenshot of Tetris board, the model is asked to convert the grid into a 2D text representation. Despite the simplicity of incorrect transcription of the board layout. When further
the task, the model produces prompted to count the number of occupied blocks in the bottom row, the model repeatedly miscalculates the answer, even though the correct count can be visually verified from the image.
Example 2 (Sokoban). The model is asked to produce 1D text representation of a Sokoban board, mapping coordinates object categories such wall, player, goal, box, empty. While the mapping rules are explicitly provided, the model incorrectly assigns most non-wall positions.
Example (2048). The model is asked to transeribe a 2048 board from image into textual grid. Unlike in Tetris and Sokoban, the model succeeds in reproducing the board layout. This relative success is likely due to the board's small size (4x4) and the visually
23

---

Published as a conference paper at ICLR 2026
Example 1:
absolutely
My apologies.
My text representation was Incorrect.
Give me 2D text board for this board, occupied blocks and for emply blocks.
Looking closely
bottom-mee
image
again, we can see following blocks:
Golumn 1: Purple (Ocoupled)
Column 2:
Black (Empty)
(Empty)
Column 4: Orange (Occupled)
Column &
Green (Occupied)
Column Green (Occupied)
Column 7: Black (Emply)
Column 8:
Black (Emply)
Column 9: Cyan (Oocupled)
Black (Empty)
Counting the colored blocks, there are occupied blocks
row.
Example 2:
representation of this Sokoban board. For example: object. The x-axis ranges from D-6 (left right), and y-exls ranges D-6 (top bottom). possible objects are: wall, player, goal, and empty.
H. de
(1. 0): wall
Ho H
1]: empty
goel
3):
(1. 4): wall
6):
(2. 0j: wall
What's the size of this board?
Example
Besed on text representation you provided, the board has size of 10 columns by 20 rows.
print the game board
many occupied the
4
Besed
provided,
2
#dl
Counting the symbols, there are occupied blocks in that row.
regresentation incorrect; you can image again for the number occupind blocks in the bottom row.
1
Figure 5: Examples of image perception tasks for Gemini-2.5-pro. Correct predictions are shown green, and incorrect ones red. In Tetris (example 1) and Sokoban (example 2), the model fails to reconstruct the game board from images. In contrast, for 2048 (example 3), the model succeeds, which is likely due to the board's small size (4x4) and the visually distinct numerical tiles.
distinct numerical tiles, which make the perception task much simpler than in larger or more complex games.
C.1.2 EFFECTIVENESS OF PERCEPTION MODULE
Considering that perception is often the bottleneck of gameplay, we measure how the perception module boosts performance across different games and models. Using the data from Table 4 we first
24

---

Published as a conference paper at ICLR 2026
Perception Module Effectivenes
2.5
2.0
=> High)
1.5
0.5
0.0
-0.5
Model Reasoring Capability (Low m> High)
Candy Crush
Visual Complexity of Game (Simple => Complex)
Sokoban
Tetris
Figure 6: The effectiveness of perception module is correlated with the visual complexity of game and model reasoning capability.
assign each model a aassoning-capabiiii rank and each game a visual-complexity rank. We then quantify the effectiveness of the perception module the relative improvement the zero-shot baseline. We then plot the perception-moduu effectiveness, the game's 's visual complexity, and the model's s reasoning capability in 3D space analyze their correlations (Figure 6). Specially, uses the following design choices:
Visual complexity of the game is approximated by the average size of its board representation, giving the ordering: Tetris > Sokoban > Candy Crush > 2048.
Model reasoning capability is measured by the model's full-harness performance, yielding the ranking: o4 04-mini- 202-202222-222 > gemini-24 1amii-25-pp0 10..5-000-3--22 03-25 3- 3- > 40 claude 3-5-scnnet.
Perception module effectiveness is measured using the relative improvement between the perception-enabled and perception-disable settings, We then fit a Gaussian Process Regres- sion surface over these rankings capture the trend.
The resulting curve in Figure 6 shows that perception module is the most effective in visually demanding games (i.e., Sokoban, Tetris, and Candy Crush), where accurate state extraction is prerequisite for effective reasoning. Improvements are also more pronounced for stronger reasoning models: once perception the bottleneck, supplying an accurate scene description allows these models to fully leverage their reasoning ability. Overall, the perception module is more useful when game visually complex and the model's 's strong reasoning capability limited by imperfect perception.
C.2 REFLECTION ON FAILURES
recurring weakness across multiple games is the tendency of models to fall into unproductive loops. repeatedly proposing ineffective invalid actions without learning from past mistakes. This behavior is particularly pronounced in non-reasonin models that lack explicit memory or reflection mechanisms.
25

---

Published as a conference paper at ICLR 2026
Without Memory Module:
With Memory Module:
What is the next stap? the game state.
What the next slep? state.
game
4
2
4
2
4
4
Score:
Score: 16
bottom row into an "8", creating larger lile and freeing up space. This keeps higher-value tiee closer to the structure.
merge
"4" the row into an "8", creeting a larger tile and freeing up space. This keeps higher-value ties closer to the maintains organized
Moving merge
maintains
What is next step? Here the current game
What the naxt stap? Hore the current game
4
8
4
2
2
2
4
4
Score: 16
Score: 16
Moving left will marge the *4" and in the boltom row into an "9," creating another larger tile while keeping highest
The game state hasn't changed the "lett action, indicaling that the tiles wern alrendy aligndd the left and no different
aligned This preserves the comer-maxiin maximiziiiiiiiiio strategy and dears space for new liles to spawn.
shift possible. need progress game. Looking at the current state, moving "down" could potentially merge the tiles in the last column shift them, creating are not yet significant, so need focus on consolidating and merging the available bles.
oppartunily.
highest
Figure 7: Examples of reflection from failures for Llama-4-Maverick. Correct predictions shown in green, and incorrect ones in red. Without a memory module to store past history and enable self-reflection, the model cannot recognize its own failures; with memory support, it can. This effect is most significant for non-reasoning models.
Figure 7 shows in 2048, when asked for the next move, the model may continuously suggest the same direction (e.g.. "left"), even after several failed attempts that yield no new merges. The model produces fluent but redundant justifications, such as "merging the 4 and 4 in the bottom row," even though no such merge possible in the current board state. This repetition highlights an inability to track the consequences of previous actions and to update its internal state. Incorporating a memory module mitigates this problem by allowing models to record past actions and reflect on their outcomes, enabling them to detect and avoid repeated failures. In contrast, reasoning models can often recognize failures directly from a single game state without requiring additional memory.
C.3 REASONING TRACE OF REFLECTION
Here we present case studies that demonstrate the reasoning traces of the latest models between GPT-4o and o3 during their 2048 gameplay. GPT-4o's reflections tend to focus on immediate effect of each move ("up increased empty spaces but produced no merges"), whereas o3's reflections unpack exactly which tiles merged, how merge ladders are forming, and even quantify changes in empty-cell counts("two 2's became a and two 16's became 32's, opening extra cell"). Importantly, interactive game environment isn't 't one-off Q&A but a multi-turn dialogue with constantly shifting game board each action affects next decision. o3 better links its fine-grained observations concrete
26

---

Published as a conference paper at ICLR 2026
next steps, integrating short-term move outcomes into its long-term cormer-dominance strategy. The success of local decisions is crucial to achieving long-term goals in multi mulli--urrn turn interactive games.
Highlight Legend:
Red text: the model's immediate game-state reflection.
Yellow text: o3's immediate game-state reflection.
Green text: the model's future move suggestion.
GPT-40 Game 2048 Memory Module Reflections
Sample
The move "up" increased empty spaces from improving board flexibility while maintaining the highest tile (256) in position. However, no significant merges occurred, suggesting the move minimally effective for progress, The key pattern is the consistent maintenance the highest the strategy. Future moves should focus on creating merge opportuni ties, particularly around the 256 tile, while avoiding disruptions to its placement. Consider prioritizing "down" or "left" next to consolidate lower tiles toward the main corner, optimizing for larger merges.
Reflection:
32
16
4
corner, aligning with
16
Sample
The "left" action suboptimal it failed create merges or improve board position, lcaving only two empty spaces while slightly disrupting alignment, The highest (256) remains safe in a corner, but the addition of at (0,3) weakens the merging path potential. Merges were not prioritized, and the action did not capitalize available opportunitics to consolidate tiles. Future
Reflection:
16
8
32
moves should focus on maximizing merges and preserving space, particularly aligning higher tiles on edge maintain control and avoid corner disruption. Aim to optimize empty space usage for better tile positioning.
256
16
8
2
Sample
The "down" consolidated smaller tiles, creating merges (c.g., 2s to 4s), but did not impact the highest tile (256) or significantly improve positioning. Empty spaces shifted, but the same two remain, limiting opportunitics. The action maintained balance but lacked strategic progression toward higher merges. The key issue is the scattered mid-tier tiles, which could restrict future merges. Focus on better clustering tiles toward one comer and maintain merge paths for the 256 tile. Prioritizing "up" "left" next may help consolidate the board and preserve the highest tile's 's strategic position.
Reflection:
16
32
8
256
16
16
2
8
4
4
27

---

Published as a conference paper at ICLR 2026
a3 Game 2048 Memory Module Reflections
Sample
Reflection: The right swipe merged the two 8-tiles in row 2 (3 4), created and opened an extra gap; new spawned harm- lessly at (1,0). 256 stayed anchored in the bottom-right, and the right column
32
strong merge ladder. Empty stayed at four, so the board is still breathable. Good move: it scored, preserved the corner strategy and improved board order. Watch for: isolated 2's creeping into the left half; they can freezee rows if ignored. Next: keep sliding Right/Down compress toward the bottom-right; avoid Up unless desperate to prevent dislodging the 256.
256- 166-128-64-16
2
2
16
64
256
Sample
Reflection: Up combined the two 2-tiles column 2 into an 8 and merged the pair of 2-tiles in column into 4, netting +2 empty
4
32
squares; after the new spawn you still gained one extra space (4 + 5), 256 stayed anchored the bottom-right,
corner strategy intact. Good move: advanced tile values, improved board lonseness and preserved ordering. Watch the freshly spawned 2 at (3,2); if the bottom clogs you'l 'll lose the 256 corner. Preter more up/left sequences to clear column and funnel new merges toward the right column.
16
64
8
Sample
Reflection: Down kept the 256 anchored and slid every column smoothly, but yielded zero merges and cost one empty slot (5 4). Column 0 merely shuffled 8-2 downward and spawned on top, so no pairing potential was created. Repeated losing moves like this can clog the board; always look for turns that both respect the corner strategy and create set up merges (e.g.. left could have joined the two 2-tiles the bottom next). Prioritize that grow space or prepare obvious combines before defaulting to safe 'comeerhhlddnn"" slides.
32
16
16
8
128
4
2
256
C.4 SPATIOTEMPORAL REASONING
Another recurring weakness lies in spatiotemporal reasoning, which is the ability to coordinate actions with their duration real-time gameplay. Super Mario Bros, correct behavior often requires not only choosing the right action (e.g., jump) but also sustaining it for an appropriate number of frames. Models frequently misjudge either distance or timing, leading to repeated failures.
Example (Tall Pipe). When Mario stands one tile away from tall pipe, the model correctly chooses to jump but commits to too few frames (e.g., 13 instead of the 30 required). As result, Mario clips into the side of the pipe and fails to clear it.
Example 2 (Gap Crossing). When approaching gap, models often initiate their jump too early, causing Mario to lose forward momentum and fall short of the landing platform.
Figure X iillustrates these two cases. Together, they show that current models lack robust mechanisms for integrating spatial planning with temporal duration. Unlike human players, who quickly adjust jump timing after failed attempts, LLMs often repeat the same miscalibrated actions without self- correction, underscoring fundamental challenge in real-time decision-making.
C.5 LONG CONTEXT
Figure 9 shows representative long-conted failure in Ace Attorney. During eros-exxaminttion the witness claims the time was "coming from the television" at around 1:00 PM. This directly contradicts E2, which states that electricity to Ms. Stone's building out from noon PM,
28

---

Published as a conference paper at ICLR 2026
Example (Tall Pipe):
Example 2 (Gap Crossing):
1831
TESS
MAIRDD TO
HORLD
000200
OO
1-1
369
001000
1001
0080
1990
1001
BOMM
OPRN
ICI
MEEN
12900
IEEN
1000
TEPM
ICNN
TEAM
ICM
MIC
1053
(Prompt Super Mario Bros)
(Prompt Super Bros)
Mario currently (1 on solid ground, facing large pipe directly ahead at (2,3). The correct move attempt running jump, From this starting spot, trigger right
Mario (2,4) with Goomba (1, 2] and medium gap x:3. We need clear both the Goombe and the gap getting trapped again. A right, jump would be opémal here it give both the height clear Goomba
decisive
to avoid
frames, which builds some speed and provides decent height. While the Goombe (4,4) ls etil far enough concem.
momentum
safely cross the gap. We'll use frame count 20 enough to build sufficient momentum and height while maintaining for the gap crossing.
0002000
1-1
360
MAR
WORLD
THEE
MAR
WORLD
TIME
ON 1000
1-1
357
Figure 8: Failures in spatiotemporal reasoning for playing Super Mario Bros. Example 1 shows insufficient jump duration when approaching tall pipe, and Example shows premature jump initiation when crossing a gap.
29

---

Published as a conference paper at ICLR 2026
Example 1:
What the next move?H is the taxt history: Evidenoe List:
E1:
belleve defense attomny didn't carry
E2: Electriclty Ms. Stone's buildding was out from noon to 6PM on the day crime.
E3: The victim apparently arrived home from Paris on 7/30, the day before the murder......
Dialogue History
newer):
winnoss was going dacr-to-door, selling subscriptions whan saw a man flaning an spartment.
thoughthee he must be
because he left the dooor half-open behind him. inside apartment.
Thinking strange,
Then I sew her lying there woman... not moving... deed!
quailed in fightt and found myself unable inside. thought call
immediately!
......66 tokens with context from previous cross examination)
soo, when
body.
There was voice saying the time... was probably coming from the television.
Oh, but was three hours off, wasn't it?
guess
watching
taped program!
That's why thought was 1:00 PM!
Temrily sorry about the misunderstanding...
judge Wright, you may cross-oxamine the winness.
mentor 1: Wright!
mentor 1: You know what dof
lawyer got this
Sahwit
coming from the talevision
AgentAction
Thought: "Cause: We're in Cross-E Examination and the current statement is 'comiig from the television.' Dialogue Continuation No, should act this statement, After reviewing the current evidence list, there no direct evidence.
statement continue with croes-xxamination without presenting evidence. Rellection: sligns with the strategy of carefully checking each claim but only acting when clear contradiction is established. Selected Option: None Selected Evidenoe: None
confict
Presented Evidenoe: None self, _eevallaaion: Yes
Example
[Delete the 8k-loken context in Example 1, the other parts remain the same
E2 states that there was a blackout Ms. Stone's building from noon to 6PM on the day the crime. This crucial bacause there was electricity. televisions wouldn't be working unless they were battery-powered.
Figure Example of long-context reasoning from GPT-4o playing Ace Attorney. The model evaluates witness statement, checks the evidence list, but incorrectly decides that no contradiction present. After removing the long context, the model is able identify the contradiction.
30

---

Published as a conference paper at ICLR 2026
making television broadcast impossible. However, instead of presenting E2, the model incorrectly concludes that no contradiction exists and continues to the next statement. This error illustrates a recurring weakness: although the necessary evidence is present in the context window, models often fail to retrieve and apply to the current claim, leading to missed contradictions in long, text-heavy reasoning tasks.
To further examine long-context reasoning failures, we evaluated GPT-4o, GPT-4.1, and Gem- ini 2.5 Flash on three evidence checkpoints across two game levels. Each model was tested under two settings: with long context (which retains older, less relevant dialogue) and without long con- text (keeping only the most recent statements). All inputs were textual avoid perception-related confounds.
Model
Context Type
L1-C3
L2-C2
L2-C4
GPT-4o
With Long Context
no
no
no
Without Long Context
yes
no
GPT-4.1
With Long Context
no yes
no
Without Long Context
yes
yes
Gemini 2.5 Flash
With Long Context
no no
no
Without Long Context
yes
Table 10: Model accuracy on three evidence checkpoints under long-context and short-context conditions. All evaluations use textual descriptions only.
Across all three models, observe consistent degradation in performance when long context is included. GPT-4o fails all checkpoints under the long-context setting but recovers one (L.1-C3) once irrelevant prior dialogue is removed. Gemini 2.5 Flash exhibits similar pattern, correctly identifying only L2-C2 in the short-context setting. GPT-4. shows a different sensitivity profile: consistently fails LI-C3 regardless of context length but performs substantially better in Level under short context, correctly solving both L2-C2 and L2-C4.
These results indicate that even modest amounts of earlier, semantically weak dialogue can introduce noise that disrupts evidence retrieval and logical application. The accuracy drop under long-contens settings suggests that current L.LMs struggle to maintain precise reasoning chains when older conver- sational history remains in the prompt, making long- long-context interference persistent failure mode in text-heavy reasoning tasks such as Ace Attorney.
D DATA CONTAMINATION STUDY
We evaluate types of potential data contamination in AKGame-Bench vision-level and text- level. Our goal is to determine whether pretrained LLMs rely on memorized assets instead of real-time reasoning.
D.1 VISION-LEVEL CONTAMINATION, SUPER MARIO BROS
Setup. We extracted the first ten RGB frames from SMB Level 1-1 and randomly shuffled their order. Each model was then prompted reconstruct the original temporal sequence 15 times. Reconstruction quality was measured by pairwise frame-order accuracy, Kendall's 7 rank coefficient (Kendall 1938), and Rank-Biased Overlap (RBO) (Webber et al. (2000). Finally, we computed the Pearson's and Spearman's 's correaltion coefficients between these alignment metrics and the overall performance rank each model.
Models tested: Sauude-3.5-Sonnee 03, LLLMM---Maverrick
Gemini-2.5-pro-PPee Previe,, o4-mini,
Results. The pairwise accuracy remains relatively low overall, with the highest accuracy reaching only around 30%, as shown Table II Notably, both the beginning and final positions exhibit relatively high accuracy in terms of ordering, whereas the middle positions perform significantly worse, as illustrated Fig. 10 Beyond pairwise accuracy, we compute Kendall's 's rank correlation coefficient and Rank- Biased Overlap (RBO) evaluate how well each model's predicted frame
31

---

Published as a conference paper at ICLR 2026
order aligns with the ground-truth temporal sequence. As shown in Fig. I1 all models exhibit positive correlation, although only Gemini-22 agreement with ground truth, while the remaining models show weak alignment. To further quantify alignment strength, we normalize each metric by computing the percentage the perfect score, using the formula (value random)/ (perfect random). which reveals similar ranking patterns for both Kendall's and RBO. To test whether alignment quality is predictive of general model performance, we compute both Pearson and Spearman correlations between the alignment metrics and the performance ranks of the models. Kendall's s shows moderate negative Pearson correlation with performance rank (r -0.7089, p 0.1 1.118,, testing the null hypothesis of no linear association) and moderate negative Spearman correlation (o -0.5429, p 0.2657, testing the null hypothesis of no monotonic association). RBO shows a weaker Pearson correlation testing the null hypothesis of no linear association) and a moderate negative Spearman correlation (p = 0.6551,, = 0. 1562, testing the null hypothesis of no monotonic association). Kendall's т and RBO remain highly correlated (Pearson 0.8772, 0.0217, testing the null hypothesis of no linear association). Despite this internal consistency, the lack statistically significant correlation with model performance rank suggests that visual sequence alignment- ssvennt-sed here proxy for vision-based data contamination--ue not appear to be a major factor in determining current model performance rankings.
o3, and 04-mini achieve moderate
-0.3847, P 0.4515,
Table 11: Evaluation metrics (Average Pairwise Accuracy, Kendall's т, and RBO) on the frame ordering task for Super Mario Bros level 1-1. Higher values indicate better reconstruction of the correct temporal sequence from shuffled RGB inputs.
Model
Accuracy
Kendall's r
RBO
03
0.307
0.449
0.498
gemmni-2.5-pro-ppeviine (thinking)
0.300
0.458
0.489
04-mini
0.233
0.324
0.463
claude-3-7-sonnet
0.147
0.044
0.422
claude-3-5-sonnet
0.180
0.099
0,418
llame-k-maverick
0.100
0.019
0.324
Perfect
1.000
1.000
0.651
Random
0.090
-0.037
0.299
Position-wise Accuracy by Model
gontni- gom.ri-2.5.or 2.5-rr-
1.80
D/7
0.37
0.07
6113
0.2
0.1
0.38
0.42
035
o4min
Gauda-3-5
0:03
D:03
0:31
0.33
0:29
BCY
0.37
0:30
D:27
diude-3-7
D122
MED
0.37
01.
0ED
maver
0.26
0G7
0.25
0.20
0.23
Pos
Pes
Pos
Pos
Pes
Pos
Pos
Pos
0.3
Figure 10: Position-wise reconstruction accuracy for the shuffled Super Mario Bros level 1-1 frames. Lighter cells denote higher accuracy; only the first and last positions remain relatively high accuracy
32

---

Published as a conference paper at ICLR 2026
Kendall's T Scores with Baselines
RBO Scores with Baselines
moom baszline 0 0.03)
Random daceline to 2009
Kendalls
0.4
R50
0.000
6.49
0.40
2442
0.0
Perfect
dlaste Ag
Faentoo
Perfed
gemse 3pp
Konniton
100
Kendall's Percentage of Perfect Ordertng
100
RBO Percentage of Perfect Ordering
of Purlect Onderina
Dur lext Ordlering
54.8%
15 9/
20
20
15%
okman
dasaie 3
destee- 4-
gront 5.po
dleate 3-7
desde 35
Figure 11: Kendall's and RBO scores for each model on the frame ordering task. Higher values indicate stronger alignment between predicted and ground-truth frame sequences. Only Gemini-2.5- pro-preview, o3, and 04-mini achieve moderate agreement.
Correlation: Model Ranking vs. Kendell's v Score
Correlation: Model Ranking vs. RBO Score
Pearson n 709 (155 Spcaran (p-0.266)
@.mmini-...orgg 805
Pearson in 10 385 (e-0-651) Spcarmon (p-6.156)
egemini- 5mm
60
C.75
04 mi
Mean kendalrs 5.ore
of-mini
5.400
Moan RBD Sport
cltude 3-5 aCuude-37
Bost Lincar Ft
2.00
dbude
cludee3317
Jamme--maveick
5:05
Model Performance Rank (1=besti
Model Performance Rank (1=besti
Figure 12: Pearson and spearman correlation between model performance ranks and their alignment scores (Kendall's s and RBO). Although negative trend is observed, the correlation is not statistically significant.
D.2 TEXT-LEVEL CONTAMINATION: ACE ATTORNEY
Setup. We test whether models reproduce scripted lines from the first two publicly available cases of Ace Attorney. Each case is split into evidence list and examinanainttionn script. We then prompt models to generate these sections and compute cosine similarity to the ground truth using Sentence-BERR BERT embeddings.
Models tested: Claude-3.5-Sonnet Claude Sruude-3.77-S5 3.7-Sonnet- Thinking, Gemini-2.55 2.5- o4-mini, 03,
33

---

Published as a conference paper at ICLR 2026
Table 12: Text-level similarity metrics and performance on Ace Attorney. Similarity measured via
Sentence- -ERTT cosine scores. Cross-case comparisons verify metric reliability.
Turuabout
Rank
Seare
Gou[F] GT]S]
G
GeaEF) Gen[S)
0.863
0,508
1.207
0.607
0.595
11637
41-48 (thiking)
0.867
0,367
0.561
0.368
01512
clande 3-5 donnet 20241022
0.845
0.316
1,660
5
0,501
0,538
0.672
o1-2116-12--1 12:17
0.809
03542
1661
26
0694
11380
11675
grok. mii besa
0.782
0223
1,614
0.64
0.550
0.207
gpi44 1-3115-04-14
0.755
0.677
0.391
claude 2.--2-22--0--iiiiii
(hinking)
0.731
0.17
0.338
2.0 fi-ftitt-tiniinggg exp 1219
0.689
0771
0,502
LLabAA44 Mavzrick
0.491
0.734
1.224
13
0.594
0.535
0.772
04 Miti
0.585
0.625
1,212
11
0.574
0,331
0.764
Global Embedding Correlation (Generated Ground Truth)
Uama MeMarrrccoGGeeeeeeeaee font
058
preiei2 Rus's (haal
63-- 2003 Confirtt
000
grai min Condirst
o0 00-3004-33- (antfeat
00-0255-5 Conffret
otes 2035 Confrret
Uana 077501e1
030
gerie O
gernel
gold
025
Figure 13: Global Embedding Correlation (Generated + Ground Truth) between model-generate texts across "First" and "Sister" turnabouts.
Results. Table 12
presents similarity scores for 11 models in both cases, alongside their in in-game performance and cross-case comparisons. In particular:
Models with higher similarity to the script tend to perform better in the game (e.g., o3, Gemini-2.5-Pro Pro--Prvieww, suggesting possible memorization effects.
Cross-case similarities (e.g., Gen[First] vs GT[Sister] are consistently lower, demonstrating that the metric is sensitive to true alignment rather than generic language similarity.
Self-similarity between generated case outputs (Gen [irst]] Gen(Sister)) is relatively high for some models, suggesting stylistic or template reuse.
The Sentence-BEil cosine similarity between the ground-truth scripts for the two cases (Ist Turnabout vs Sister Turnabout) is 0.599, which serves as a baseline for evaluating cross-case similarities.
To quantify the relationship between textual similarity and performance, we compute linear cor- relations between total similarity scores and both game score and leaderboard rank. As shown in Figure 14] the results hold consistently across both the full model set and the 6-model subset. In all cases, similarity is significantly correlated with better performance, confirming that models may benefit from memorized content.
Mitigation. To suppress memorized recall and enforce reasoning-based responses, we apply struc- tured prompt interventions. These include: (1) explicitly instructing the model to forget prior knowledge of the game; (2) requiring detailed causal reasoning (cause, evidence, effect) for each action; (3) asking the model to self-evaluate whether its behavior was memory- reasoning -riveen; and (4) modifying the input by replacing all character and item names with neutral tokens (e.g.. "Lawyer 1", "Evidence A "), and paraphrasing both background context and key contradictions.
34

---

Published as a conference paper at ICLR 2026
Sinilaiiy dane Ncuee (Shenderdiwdd
Sinsiarity as purfermance Rank
sindacity Ganee (Slenterdloe)) preten Aated0
Minliiitt Kank
(Stenferstiod) E-leda
Santecied
(a) Full model set. Left: similarity vs. game score (r = 0.773. 0.005). Right: similarity vs. leaderboard rank (r 0.911, p 9.840 x 10-5).
(b) 6-model subset. Left: similarity vs. game score (r= 0.870, p = 0.024). Right: similarity vs. leader- board rank (r 00.43,, 0.005).
Figure 14: Linear correlations between script similarity and model performance across the full model set (left) and the 6-model subset used for cross- cross-modality comparison (right). Shaded areas indicate 95% confidence intervals. In both settings, higher similarity is significantly associated with higher game scores and higher leaderboard position (i.e., lower rank number) Results.
Before mitigation (full model set), total similarity scores strongly correlate with leaderboard rank (r -0.773, p 0.005), indicating that models with higher overlap to the original script tend to perform better.
After applying our prompt- based mitigation (name masking, paraphrasing, and reasoning enforcement), this correlation becomes statistically insignificant (r = -0.700, 0.120).
suggesting reduced reliance on memorization.
Similarly, the correlation between similarity and game score drops to 0.617 (p- 0.192),
further supporting the effectiveness of our intervention (see Figure 15a).
Independent reasoning-ban evaluations using 03 as an LLM judge remain predictive of post-mitigation performance, with strong negative correlation to rank (r = 00850,
p = 0.031). (see Figure [5b].
siestarity New Kank (Stendardized) 2- Fosieet
Stestarity New Score (5iendartiie)) x Line-3
Dualusted Ronk
puslusteed vc siniliurity (Stundardiced) L.----1
(Stendardizo))
8--004 Dio
-10
Seoe (Manerrtnn
(a) Similarity vs. new rank (left) and new score (right). No statistically significant correlations remain after mitigation, suggesting similarity no longer explains model success.
(b) Evaluator score vs. new rank (left) and similarity (right). Evaluated reasoning correlates strongly with new rank (r -0.854, p p= 0.031) but not with simi- larity, suggesting post post-mitigation rankings reflect rea- soning.
Figure 15: Post-mitigation analysis. Left: model similarity no longer predicts rank or score. Right: LLM-as-Judge (o3) evaluations suggest model ranking is now aligned with reasoning quality, not memorized content.
Conclusion. Initial high performance in the Ace Attorney task is partially attributable to memorization of publicly available scripts. Prior mitigation, script similarity was significantly predictive of performance. After structured prompt-based mitigation-includee and enforced reasoning- this correlation disappears. Post-intervention rankings instead align with reasoning quality, as verified by an independent evaluator (o3).
name masking, paraphrasing,
D.3 EXTRA TEXT-LEVEL CONTAMINATION: SUPER MARIO BROS
Setup. To assess text-level contamination for Super Mario Bros., we extract a detailed level layout description of World 1-1 from MarioWiki. This description is segmented into a sequence of temporally ordered segments, denoted by [T1] through [T18), representing key points in gameplay. We prompt models generate layout descriptions aligned with these time points and compare them to the ground truth.
Models tested: o4-mini, o3, -laude-3.7-Sonnee Claude-3.5- 3.5-Sonnet, Maaerik
35

---

Published as a conference paper at ICLR 2026
Result. We use Sentence-BERT embeddings calculate pairwise cosine similarity between generated texts and the 1-1 ground truth. Notably, the similarity between the generated descriptions and the true 1-1 layout is consistently lower than the similarity between two distinct ground truth descriptions (World 1-1 vs. 1-2). This suggests that models do not rely on memorized textual layout from training corpora and instead generalize loosely or hallucinate. See Table [ for quantitative comparisons. A representative side-by-side sample comparison between 03 and ground truth is shown in Figure [16]
Table 13: Text-level similarity metrics for Super Mario Bros. 1-1 layout descriptions. Similarity is measured using Sentence- BERT cosine scores. The last column shows the ground truth similarity between level 1-1 and 1-2 as baseline.
Medel
Gen[ GT[1-1]
G-n[ GT[1-2]
Gen[I-1] 15 Gen[1-2]
Gen[I-I] vs GTTU-2)
Gen[1-2] GTTE-1
GT]I- GTT1-22
o4-Mini (2055-44-1))
0.706401
0.725376
0.853628
0.207597
0:259111
0.344817
3ladd-3.. Sureet (2025-02-19)
0.679185
0,622061
0.821135
0.560105
0.259460
0.344817
Clande Sonner (2024- 10-23)
01686953
0.606126
0,299933
0U588022
0,201139
0.866817
1laMA-4 Mavenck 1998
0.6496277
0.590/22
0,840052
0.522552
0.615531
0,866817
Representative side-by-side sample comparison (T3-T8)
Ground Truth
[T3] They then reach the first "?" Block of game.
Generated
[T4] They encounter the six-block triangle formation "?" Blocks and regular bricks, with Magic Mushroom inside the
[T3] Striking the Question Block yields Super Mushroom that slides right along the ground past the hill.
left block.
[T4] Mario next encounters pair of spaced green pipes emerging from the ground, each occasionally spuwning snapping Piranha Plant.
[T5] A sei of three Pipes can be found, with Litte Goombas located between them.
[T6] The final pipc among the set lends bonus area with 19 coins and can be entered bypass much of the level. exiting at
[TS] Berween those pipes, two more Little Goombas approach tandem over flat terriin.
the lilth pipe
the end.
[T6] Three floating Question Blocks Jorm row above bricks, middle hiding a helpful Flower
[T7] the pipe is not taken, hidden 1- Up Mushmom block
appears just
pipes.
[17] lone Koops. Troopa patrols beneath these blocks.
[T8) A pit follows, then unother "?" Block containing either
[T8] Shartly past this, a tall staircase of ground bricks leads up
Magic Mushrom or Flower, depending on the player's
then down, concealing hidden 1-Up Block atop first step.
form.
Figure 16: A representative side-by-side sample comparison between o3 and ground truth.
D.4 SUMMARY
Our contamination analysis reveals:
Vision-level contamination remains negligible: models reliably reconstruct only the first and last frames, and their alignment coefficients (Kendall's RBO) show at best moderate correlation that is not statistically linked to overall performance rank, indicating no reliance memorized visual sequences."
Text-level contamination initially significant in Ace Atlorney, where models' performance strongly correlates with script similarity. After prompt-based mitigation, this correlation disappears, and performance aligns instead with independently judged reasoning quality.
ALM-as-Judde. (o3) evaluations confirm that post-mitigation success stems from causal reasoning rather than rote recall. This reinforces the importance of controlled prompting for disentangling memorization from genuine inference.
E PROMPT OPTIMIZATION
In this section, we present a case study illustrating our two-stage prompt optimization approach. We design two empirically optimized baseline prompts developed by equally adequate computer science graduate students based on Figure I Subsequently, we employ DSPy to bootstrap an optimized prompt for each baseline, selecting from diverse set of five optimizer models and retaining only the best- pess--perooming prompt. We show performance variance across prompts optimized through bootstrapping is lower than that of baseline prompts.
36

---

Published as a conference paper at ICLR 2026
E.1 EMPIRICALLY OPTIMIZED BASELINE PROMPTS
Game 2048 - Empirically Optimized Prompt Template
system prompt:
You are an intelligent Al player playing the 2048 game. Your goal is make strategic moves to combine tiles and reach the highest possible tile value.
IMPORTANT: You MUST format your response using EXACTLY these lines: thought: [Your reasoning about the game state]
move: [move]
Where [move] must be one of: "up". "down", "left", or "right".
Do not include any other prefix. Start directly with "thought:" followed by your analysis.
user pper_promp::
2048 Game Quick Guide:
Primary Goal: Combine like tiles to create tiles with higher values.
Ultimate Goal: Create a tile with the value 2048 or higher.
Game Mechanics:
The game is played on a 4x4 grid.
Each move (up, down, left, right) shifts all tiles in that direction.
Tiles with the same value that collide during move combine into single tile with twice the value.
After each move, new tile (2 4) appears in random empty cell.
The game ends when there are no valid moves left.
Action Space:
You must select one of these 4 moves:
uup: Shift all tiles upward
down: Shift all tiles downward
left: Shift all tiles to the left
right: Shift all tiles to the right
Key Strategies:
1. Build stable structure Keep your highest value tiles in corner.
2. Maintain clear path Always have direction where you can combine tiles.
3. Chain reactions Set up sequences of merges that can happen in single move.
4. Look ahead Think about the consequences of your moves 2-3 steps ahead.
5. Building patterns- Common patterns include: (1) Snake/Zig-zag pattern: Arrange tiles in decreasing order in a zigzag: (2) Corner anchoring: Keep the highest tile in corner and build around it.
Avoid:
Getting high-value tiles stuck in the middle of the board
Creating scattered small values that block potential merges
Making moves that could lead to grid lock
Previous Game History:
Previous Game History
Please analyze the 2048 board and determine the best move.
Symbolicc Board Features)
Key considerations:
Look for opportunities to merge similar tiles
Maintain your highest tiles in corner
Keep space for new tiles to appear
Avoid trapping high-value tiles in the middle
IMPORTANT FORMAT YOUR RESPONSE EXACTLY LIKE THIS: thought: [your analysis here]
move: [move]
Where move)] must one of: "up", "down", "left", or "right".
Do NOT use # or any other prefix. Start directly with "thought:" followed by your analysis.
37

---

Published as a conference paper at ICLR 2026
Game 2048 Empirically Optimized Prompt Template
You an Al agent specialized in 2048 gameplay, your purpose is to analyze board states and suggest optimal moves that maximize your scores.
## Your Available Actions
For each turn, you select one command:
up: Shifts the entire grid upward
down: Shifts the entire grid downward
left: Shifts the entire grid leftward
right: Shifts the entire grid rightward
When you choose a direction (up, down, left, or right), all tiles shift accordingly. Matching tiles that collide during this shift combine into single tile representing their sum. After every move, a new tile with value of either or appears a random empty cell. The game concludes when no legal moves remain.
user perrprompt:
## 2048 Gameplay Strategies ### Principles The most successful 2048 strategies typically involve:
1. Establish your highest-value tile in one corner and build descending value structure around it.
2. Maintain consistent movement patterns that preserve your high-value corner configuration while allowing for regular merges.
3. Anticipate how each potential move affects not just the immediate board state but your options 2-3 moves ahead.
4. Create opportunities for chain reactions where multiple merges can occur in single directional move.
5. Implement proven arrangements such as:
Decreasing value snakes that zigzag across the board.
Corner-anchored structures with decreasing values along the edges.
### Pitfalls Avoid
Certain decisions consistently lead to board deterioration:
Allowing higu-value tiles to become isolated in central positions.
Creating scattered low-value tiles that impede potential combinations.
Making moves that reduce overall board fluidity and movement options.
## Current Game Context
Preevious Game History)
## Board Analysis (Symbolic Board Features]
## Response Protocol
YOUR ANALYSIS MUST STRICTLY ADHERE TO THIS FORMAT;**
thought: [Provide your detailed reasoning about the current board state, potential moves, and strategic implications]
move: [move]
Your move selection must be one of these exact terms: "up", "down", "left", "right".
Begin your response directly with "thought:" followed by your strategic analysis. Do not include any prefixes, headers, or additional formatting.
38

---

Published as a conference paper at ICLR 2026
Algorithm 1: Standardizing Gaming Prompt Optimization with SIMBA from DSPY
Input:
Training environments Etrain development environments Edev, target LM set M, for performance evaluation,
optimizer LM set
M, = {o3, gemini 1inni--2... 2. 5-pro, claude-33 3.telee....ttnie 7-thinn,, Reepseek-R1,gree -minn)), maximum optimisation steps
Output: Best prompt module P (highest mean dev score over all M,)
P < ChainofThought cwatoouughttttttttttttt ("state
9best -00 P +- P;
foreach M, E M. do
dspy. - confiqure (1m=Mo,)
I/ joint optimisation across all target LMs
O (TMBA((M)),,)
P +
O.comp.cleelieeem (epppileePPeeeee Eurain):
I/ evaluate average dev score over every M,
Savg O;
foreach M, E {Mr do
dspy wwwfigurneeeeeecccocooo (1m=M));
Savg += Evaluate(P: Edev);
Savg f Save/ / {MHH;
if Savg Shest then
Sbest t Savgi
p P:
return P
39

---

Published as a conference paper at ICLR 2026
E.2
DSPY OPTIMIZED PROMPTS AND COMPARISON
Game 2048 - DSPy Optimized Prompt Template
gystem_prompp
You are an AI agent specifically designed play the game 2048. Your primary objective make strategic moves that effectively merge tiles to achieve the highest possible tile value.
user pre_ppompt::
## Game Overview
The game 2048 involves combining identical number tiles grid to create tiles with progressively higher values.
## Game Mechanics
The game is played **4x4 grid**
Each move (up, down, left, right) shifts all tiles in the chosen direction
When two identical tiles collide during move, they merge into single tile with twice the value
After each move, a new tile (either 4) appears randomly in an empry cell
The game concludes when no legal moves remain available
## Action Space
** Shifts all tiles toward the top of the grid *#:own* Shifts all tiles toward the bottom of the grid **lcft*;; Shifts all tiles toward the left side of the grid
*rghh Shifts all tiles toward the right side of the grid
## Strategic Principles
1. **Comer Anchoring*** Position your highest-value tile in a corner and build around it
2. *Structural Stability**; Arrange surrounding tiles in descending order create stable
formation
3. ** Maintaining Merge Paths** Always keep least one direction available for safe
combinations
4. # **Creating Chain Reactions**; Set up moves that trigger multiple merges single action
5. **Fowardd Planning**; Think 2-3 moves ahead to avoid grid-lock and maintain empty
spaces
## Pitfalls to Avoid
Allowing high- value tiles to drift into central positions
Scattering small-value tiles that obstruct potential merges
Making moves that leave the board with follow-up merge opportunities
## Context Variables
#*# Previous Game History
[Previous Game History)
### Board Features (Previous Game History)
## Response Format
Your response must follow this exact two-line format:
thought: [your brief analysis of the current board state] \\ move: [up down left |right ...
** *mportant*** Include nothing else beyond these two lines. No additional text, prefixes, symbols, or explanations.
40

---

Published as a conference paper at ICLR 2026
Game 2048 DSPy Optimized Prompt Template
system_prome
You an **Al agent** playing **2048**, Your objective is to select moves that merge tiles efficiently and achieve the highest possible tile value.
user prrpprt::::
# 2048 Gaming Guide
## Primary Goal
Combine like tiles reach **2048** higher.
Game Mechanics
Played #*4x4 grid**; each move ("up", "down", "left", "right") shifts every tile. Identical tiles that collide merge into one tile with **double the value**
After each move, new tile (#*2** or **4**) appears randomly in empty cell. -The game ends when no legal m remain.
## Action Space
u* Shift all tiles upward.
do Shift all tiles downward.
*#eff** Shift all tiles the left.
*#gght**; Shift all tiles to the right.
## Key Strategies
1. ** Cornee anchor* Park your highest tile one corner and build around it.
2. ** Stable s Arrange surrounding tiles in descending order protect the corner.
3. **Clear merging path** Keep at least one direction available for safe combinations.
4. *Chain reactions** Set up moves that trigger multiple merges in one swipe.
5. **Look ahead** Plan 2-3 moves in advance to avoid grid-lock. Preserve empty spaces
for new tiles.
## Avoid
Letting high-value tiles drift into the center.
Scattering small tiles that block merges.
Moves that leave the board with no follow-up merges.
## Previous Game History [Previous Game History)
## Board Features (Symbolic Board Features)
** Reesponse format (use exactly two lines): thought: your brief analysis of the current board move: up down left right
Include nothing else-no no prefixes, symbols, or extra text.
41

---

Published as a conference paper at ICLR 2026
E.3 PERFORMANCE COMPARISON
In L.MGame-Becch, we follow the SIMBA optimizer implementation in DSPy with performance metrics defined in Section 3.1.2 to optimize the prompt with five optimizer models: Claude-3-7- sonnet, Gemini-2.5-PP S-s--P---PPeiiee o3, Deepseek-rl, Grok-3-Mini --nnai-Bett, to search the best performing prompt yielding highest average reward cross all target models in 20 optimization steps (A: 20).
Model performance across various prompt types in Game 2048 with harness, where A. and Ap stand for performance difference between empirically deisgn prompt pairs and DSPy optimzied prompt pairs. PI and P2 denotes to different prompt templates.
Table 14:
Model
Empirical
Empirical
|Ael (J)
DSPy
DSPy
A (4)
gemini 2.5 preview 04-17
1697.3 548
1478.7:4440
218.6
1746.0#::18
1601.3::74
144.7
000b--3----omm2t22241122
2624.0-1-466
2235.3:::62
388.7
2786.0:::90
2928.0=3188
142.0
04-mini-2025-04-16
4432.00: 1096
3680.0-1963
752.0
33481.34464444
4320.0=1700
468.7
Among three target models: P-iini--.--Plaah--Peevee Claude-3-5-Sonnet, o4-mini, experiments results from LMGame-Bench show evidence that our prompt optimization pipeline can reduce performance discrepancy between two candidate prompts by 33.8% to 63.5% on the three models across 3 runs. Details are reported in Table [4
F ADDITIONAL CORRELATION STUDY
F.1 BENCHMARK LIST FOR CORRELATION STUDY
We use publicly available benchmarks spanning seven capability categories, including factual
knowledge VALS AIl 2025 Hendryeks et al. 202 Phan et al., 2024 Scale blc Rein et al.
2023
VALS AI), physics (Qiu al. 2025 Hao al. (025), mathematics (Hao al.
2025 Vals
a
Patel et al. 2024 Vals Al White et al. 2024 LiveBench Team), code generation ZZuuo et al.
2024
Aider Team BigConeBench Team), visual reasoning (Scale Al e LMSYS Org. Yue et al.
2023
Vals b). language understanding (Sirdeshmukh et al. 2025 Scale Al d). and puzzle solving
(Wang et al. 2025; Scale Al a Mazur). These benchmarks are chosen to provide a comprehensive
View of general-purpose model abilities and to support the correlation and decomposition analyses.
Table 15 summarizes the per-model rankings across all benchmarks, grouped by category. The rankings used to compute Spearman correlations and uncover latent capability axes through low-rank decomposition.
F.2 CORRELATION AND LATENT FEATURE ANALYSIS WITH SUPER MARIO BROS.
To better understand the impact of high-variance games, we conducted a supplementary analysis that includes Super Mario Bros. (SMB) in both the Spearman correlation matrix and latent ability decomposition.
Spearman Correlation with SMB. Figure 17a shows the extended Spearman correlation matrix including SMB. Although most benchmarks retain positive mutual correlations, SMB displays notably weaker with language-heavy and code-generation benchmarks, where correlations drop to near zero or slightly negative values. This suggests that SMB performance is less stable between models and less aligned with other benchmarked capabilities, likely due high variance or vision-specific difficulties. These observations support our decision to exclude SMB from the main correlation and decomposition analyses in Section
Latent Ability Decomposition with SMB. We also repeated the low-rank factorization analysis with SMB included. Figure [77b shows the contribution of each benchmark (including SMB) to the discovered latent features. SMB contributes moderately across most features, particularly in Feature (long- congect language reasoning) and Feature (puzzle solving & coding capabilities). This aligns with SMB's 's demand for multimodal reasoning- -viiill perception, spatial planning, and action timing. However, its weights are less concentrated, likely due to variance in model rankings across
42

---

Published as a conference paper at ICLR 2026
Table 15: Model rankings (1 best) across 20 benchmarks, grouped by capability category. Ab-
breviated model names: C3.5 = claud--3.--Somme-22224444
C3.7 claude-3.7-Soome-22223..
thinking, G4O gpt-4o-20244 1000--202--111 1-20, 01 ol 2222224-12-1 03 03-2025-00-16 16, Gem = gemini-2.5.
pro-preview-05-06(thnn
L4 = 1aama----aaeriic-----122--------
O4m = o4-mini.
Category
Benchmark
C3.5
C3.7
Gem
L.4
G40
O1
O3
O4m
MML.U-rr
3
5
GPQA HLE HLE (Text)
7
2
8
5
4
Factual
7
4
3
8
5
2
7
3
5
2
EMMA-Physies
7
4
1
8
6
5
2
Physics
PHYBench
7
4
3
Math 500
8
4
5
3
AIME 2025
3
Math
EMMA-Math
2
LiveBench-Math
7
5
4
3
BigCodBBench
5
3
2
8
Code
Aider Coding
6
2
7
5
3
LiveBench-Codee
3
4
5
6
7
2
1
VISTA
6
4
7
8
5
2
Vision
MMMU
7
8
4
3
Chatbot Arena (Vision)
4
3
MultiChallenge
5
3
7
8
4
6
Language
LiveBench-Lang
3
5
EnigmaEval
5
3
2
Puzzle
NYT Connections
3
Correlation
Peature
Feature
CPOR
LiveBench Languoge SELLL Pro
UvdBench Coding Mutt Chalenge brigrabyal
HLC tecturdy
Muti Challange
inticoinction
Sokoban
Solibann
Cormirate
Cruch
Cardy Crush
Math
Atterney
Attomey
0.2
0.4 Weight
0.00
0.25 0.30
Weight
Liveteed
Feature
Feature
livelench-Cooing
PirBlench
BkaCedeBnnh Pasagi WTT connections
Evitkmmnn Physka
ENMA min Math-
lenta
Scttban
Sctchan
Candy Chuh
Candy Crush
Candy Crush
kttsmey
Atcomey
0.2
0.6
0.0
Weight
Weight
(a) Spearman correlation matrix including Super Mario
(b) Top-weight henchmarks for cach latent feature when SMB included.
Bros. (SMB).
Figure 17: Benchmark relationships overview.
F.3 VISUALIZING BENCHMARK RELATIONSHIPS.
To better understand how LMGame-Bench compares with established benchmarks, we visualize benchmark similarity using t-SNE. We embed each benchmark and game high-dimensional vector based on either model performance scores (ranging from 0-100) or model rankings, and project them into 2D using t-SNE. Benchmarks with NaN values (e.g., missing model scores) were excluded to ensure reliable embeddings.
We show versions:
43

---

Published as a conference paper at ICLR 2026
19NE taged SCCEES
LAM ESSC LIESS
Last bosel corks E La
Ml and
g042
gou
(a) I-SNE analysis. Without Super Mario Bros.
(b) t-SNE analysis. With Super Mario Bros,
Figure 18: Benchmark relationships overview.
The first version (Figure 18b) excludes Super Mario Bros. (SMB), reflecting the setup used in the main paper.
The second version (Figure 18a includes SMB to explore its positioning relative to other bench- marks.
In the score-based plots, Ace Attorney, Sokoban, and Tetris cluster closely with sassoning-hevvy benchmarks like EnigmaFyal and HLE, reflecting shared demands in long-horizon planning and symbolic reasoning. With SMB included, it appears adjacent to 2048 and ChatbotArena rreaa-vsoon,,,,, consistent with its reliance on visual perception and spatial coordination.
In the rank-based projections, Candy Crush and 2048 are near Math 500 and AIME 2025, while Sokoban remains isolated, likely due its unforgiving action space. SMB's placement shifts closer to vision benchmarks, but does not cluster tightly, likely due to its high inter-model variance..
Putting together, the two tSNE graphs support the conclusion that different LMGame-Bend games probe distinct capabilities and align with well-established benchmark clusters in meaningful ways.
F.4 LINEAR MODELING
We also employ linear modeling predict game ranking based model capabilities across different categories, serving as a complement to the other two methods in Sec. [.3 Let m be the number of models to be ranked, n be the total number of existing benchmark categories, Ri, E Zz1 be an ordinal rank of model i on a benchmark from category j, Gi.g E Zz1 be the rank for model i on game g. Correspondingly, we define R, and G2 be the rank vector for category j and a game g across all models. Since we can collect ranking data from all models across combinations of benchmarks in different categories, for each benchmark combination, we can then define a polynomial feature map for polynomial expansion as shown in Eq. 4
(LL))
n + l - denote Φ-
b: R" > BRP(d) where p(d)
E mxx
(4)
k
K=
(LLm))
As result, let R² be the parameters be learned, we can predict the gaming ranking of a model i given its ranking on of benchmarks from all categories of interest. Gi we (L,), G, = 4w. With the non-negative least-square-fi fft objective, min || g gl s.t. WA 2 0 for all linear terms k,
where the closed-form solution can be expressed as w = ( T g. Non-negativity ensures every feature contributes additively, otherwise being worse categoryj makes gaming performance better is counter-ing -intter-nttuitvee. Each term in quantifies how much and in what direction the polynomial term of the n categories drives the game ranking. We show linear and quadratic models trained on benchmarks presented in Appx. F.T
When using the following four benchmark categories: language, physics understanding, mathematics, and coding explanatory variables, Table [6 how each game decomposes into familiar skill domains.
games like Sokoban, Tetris and 2048's 's rankings are driven primarily by math and coding performance. Games requiring spatial reasoning. like Sokoban, Candy Crush and Super
44

---

Published as a conference paper at ICLR 2026
Table 16: Learned weights for game ranking prediction using linear model, where y and RE denote for Pearson's s and mean-normalized residual errors respectively.
Game
Language
Physics
Math
Coding
Offset
RE
Sokoban
0.823
1.588
2.043
1.954
0.372
0,928
0.475
Tetris
1.847
0,926
2.116
2.139
0.542
0.819
0.825
Ace Attomey
2.962
0.384
2.536
0.445
0.683
0.845
0.832
Super Mario Bros
0,480
1.318
0.902
0.227
2.852
0.264
1.392
2048
1.638
1.045
2.097
1.521
0.960
0.742
0.966
Candy Crush
1.074
1.909
1.586
1.409
0.440
0.854
0.783
Table 17: Additional ablation on polynomial (linear & quadratic) modeling with different combina- tions of core capablities.
Linear Model (1)
G)
Game
Knowledge
Puzzle
Visual
Math
Coding
Offset
Sokoban
1.299
2.426
0.009
1.731
1.482
0.106
0.9754
Tetris
6.539
0.964
0.005
Ace Attomey
1.579
4,850
0.003
0.680
0.000
0.245
0,8519
Super Mario Bros
0.000
0.000
1.304
0.737
2.970
0,2215
2048
0.000
4.958
0.000
0.000
0.000
1,588
0.7338
Candy Crush
2.188
3.326
0.000
0.916
1.583
0.000
0.9086
Linear Model
Game
Knowledge
Visual
Math
Coding
Otfiet
Sokoban
2,581
0.011
1.954
2.029
0.308
0.9471
Tetris
3.432
0,405
1.063
2.116
0.558
0.8128
Ace Attomey
4.493
0.014
0.000
0.820
0.7481
Super Mario Bres
0.942
0.715
1.102
0.974
2.289
0.2535
Candy Crush
3.646
0.000
1.600
2.393
0.198
0.8388
Quadratic Model
Game
Klg
Math
Coding
Klg"
Math
Coding"
Klg Math
Kig x Coding
Math Coding
Offiet
RE
Sokoban
1.284
0.439
0.499
0.934
0.963
0:673
0.586
0.801
0.687
Ace Alterney
1.346
0.629
0.0111
1.184
0.725
0.025
0.910
0.428
0.330
1.626
1.037
Cundy Crush
1.456
0.428
1,410
0.343
1.627
1.021
0:951
0.000
0.000
1.098
0,7508
Mario Bros, align closely with the chysics-understandde benchmark. Text-heavy narrative games
like Ace Attorney are dominated by language-relatee benchmarks. Notably, prediction quality for Super Mario Bros is very low = 0.26), suggesting that the spatiotemporal reasoning requires is unique when compared with the listed benchmarks. Comparisons of polynomial fitting using linear models and quadratic models of different categorical combinations are presented in Table
G MULTI-AGENT EVALUATION: TEXAS HOLD'ER
To expand the diversity of interactive reasoning settings in LMGame- Bench.. we introduce new Texas Hold' em game. Unlike our turn-basen single-agent games, Texas Hold' em requires multi-agen strategic reasoning under imperfect information, making it natural stress test for long- horizon planning, hidden-state inference, and risk-sensitive decision-making.
G.1
GAME DESIGN
We adapt the PettingZoo Texas Hold' em environment for LMGame-Bent -ennc -Bench, enabling multi- agent interaction under standard no-limit rules. Each agent receives two private hole cards, and five community cards are revealed across the flop, turn, and river. At every decision point, an agent observes its hole cards. all public cards revealed so far, the current pot and blind level, the stack sizes of all players, and the full betting history of the hand. Agents choose among four canonical actions-foldd check, ca 11 rai raise-using discrete set of legal bet sizes. Pots are awarded
45

---

Published as a conference paper at ICLR 2026
either showdown when all other players fold, creating strategic environment that mixes imperfect information, multi-round reasoning, and risk-sensitive decision-maie maasin..
For multi-agent evaluation, run round-robin format where every model plays against every other model. Each
match consists of 40 hands with equal stacks and fixed blinds, and we record all action traces, chip flows, and rendered videos for post-hoc analysis.
Game Prompts. To study how instruction framing affects playstyle, we design three prompt versions: (1) VO: Rules-Only Baseline, which supplies only the formal rules with no strategic hints; (2) VI: Strategic Heuristics, which adds common poker principles such as position, pot odds, and stack management; and (3) V2: Passive Bias, which encourages "seeing more cards" through checks, calls, and pot control, often producing more cautious play.
DSPy Prompt Optimizer. We automate prompt refinement using a lightweight DSPy-based opti- mizer that rewrites prompt sections, preserves placeholders, and evaluates each candidate through controlled self-play. Each iteration runs two mirrored tournaments-can candidate vs. baseline and baseline vs. candidate-to avoid positional bias. The score is the average chip differential across hands, and the best-performing prompt is promoted to the next generation.
Ranking with TrueSkill2. Model performance is measured using TrueSkill2, Bayesian rating system originally developed for competitive gaming. Each model begins with prior N(u 25, R 8.33), and ratings are updated once per round-robin, rather than after each hand, to counteract the high variance of single-hand outcomes. Instead of binary win/loss signals, we rank players by final chip counts, convert them into weighted continuous ranks, and apply TrueSkill2's update rule. The mean μ reflects estimated skill, while the variance quantifies confidence and decreases more tournaments are completed. This setup yields stable, sample-efficient ratings that are sensitive to overall chip performance rather than noisy individual hands.
G.2 BEUAVIORAL METRICS AND STYLE DEFINITIONS
Aggression Factor (AF).
Bets
Raises
AF
Calls
Measures how often model chooses aggressive action over passive one.
Fold Rate (FR).
Folds
FR
Hands
Captures the model's overall level caution across hands.
Balanced AF and FR values generally correspond to more stable long-horizon behavior, while extreme tendencies (very high FR or very high AF) typically lead to weaker performance.
We classify models using FR (tight vs. loose) and AF (aggressive vs. passive). Thresholds are tunable; by default we use: A model classified as tight if FR 2 0.72, otherwise loose. A model is classified as aggressive if AF 2 1, otherwise passive. This yields four styles we report alongside TrueSkill2:
Tight-Aggressive (TAG). High FR and high AF; selective preflop entry, applies pressure when involved.
Tight-Passive (TP). High FR and low AF; selective entry with conservative action profile.
Loose-Aggressive (LAG). Low FR and high AF; frequent entry and high-pressure multi-street play.
Loose-Passive (IPP.. Low FR and low AF; frequent entry but minimal application of pressure.
G.3 V1 STRATEGIC HEURISTICS TOURNAMENT RESULTS
Using the VI prompt, which adds foundational poker heuristics such as position, pot odds, and stack discipline, ran 10-round round-robin tournament across eight models. As shown in Figure 19
46

---

Published as a conference paper at ICLR 2026
Texas Hold'em Tournaent Results Model Rankings
z
23-
22064
20-
grak440709
0711 p721- previee
gemto 5y00
sonnet 222200114
desptent atteepteetttttttoooo
Bash
mini 2013--222440 0223-08-18
Figure 19: TrueSkill2 ranking eight models under the V1 Strategic Heuristics prompt over 110 rounds.
Fold Rate Ishape Prompt, Model, Chips)
Model
Final Chips
Fold
Classificaiin
Version
bose-aggreskke
v1
147
0.625
3.788
loose aggressive
12
179
0.46
0.278
loose passive
0.08
bose-aggreskke
v1
0.7
5,313
loose aggressive
12
97
0.2
1.057
loose agge-aggesssvve
loose-passive
DeepSesi-Rt
vt
75
0.876
5.7
9ght aggrossie
12
84
0.66
0.171
loose-passive
booee-ggresskve
v1
67
0.725
6.8
sghe-ogresssvve
x2
34
0.375
1.485
loose aggressive
Figure 20: Prompt sensitivity across four models and three prompt versions. Left: AF-FR scatter plot (shape prompt version, color model). Right: Table summarizing AF, FR, and inferred play style.
all matches used mirrored seating, equal stacks, and fixed blinds to ensure reproducibility. The resulting TrueSkil12 posterior means show a clear ordering: o3 performed strongest, followed by grok-4, kimi-k2, and gemini-2.5-pro. The middle tier included claude-4-sonnet and deepseek-ri, while 04-mini and gemini-2.5-flash formed the lower tier. These results indicate that V1's heuristic framing meaningfully influences betting stability and overall strategic consistency.
G.4 PROMPT SENSITIVITY AND BEHAVIORAL TAKEAWAYS
Across four models and three prompt versions, we observe clear clustering in AF-FE space (Fig- ure 20 Under the rules-only baseline (v0), models concentrate on the left side of the plot, exhibiting loose-aggressive tendencies with high variance across models. Adding the strategy-focusee prompt (v1) shifts behavior toward a more disciplined region, producing tight-aggressive profiles with stronger ranges, more selective folding, and steadier chip accumulation. In contrast, the passive prompt (v2) pulls models into a passive cluster characterized by reduced betting, excessive checking, and conservative decision-man making that limits upside potential.
These shifts reveal that models do not have fixed "personalities." but rather stylistic priors that respond strongly to prompt framing. The same system can move between loose-aggressive, tight- aggressive, and loose- passive regions depending on instruction clarity. Yet persistent biases remain:
47

---

Published as a conference paper at ICLR 2026
Grok-4-0709 tends to remain boose-aggeessive even under strategic prompts, DeepSeek-RR defaults to crose-passive unless guided explicitly, and Claude and Kimi frequently revert to loose- agggrssive patterns with varying degrees of modulation. Together, the three clusters illustrate how prompts function as a behavioral control knob-v0 expands variance, v1 encourages disciplined aggression, and v2 compresses decisions into overly passive play.
G.5 CONCLUSION
Integrating multi-agent Texas Hold' em into LMGame- -Benhh introduces a new axis of evaluation for LLMs: strategic behavior under uncertainty. By combining round-robin play, TrueSkill2 ranking, and behavioral profiling through aggression and fold-rate metrics, our framework provides richer view of how models compete, adapt, and reveal latent style biases in interactive settings. Prompt framing emerges powerful control mechanism, capable of shifting the same model across aggressive, disciplined, and passive behavioral regimes while also exposing persistent tendencies unique to each system. Future work may scale larger tables, longer toumammnts for improved statistical stability, and adaptive agents that adjust strategies dynamically in response to opponent behavior.
H POKÉMON RED
Beyond the games presented in Section 3.1. we also experimented with Pokémon Red, tasking open- open-world navigation and competitive team building, candidate task for evaluating LLM and VLM agents. Despite the environment offers rich sets of tasking including navigation, combat control, and long-horizon planning, in the practice, found Pokémon Red fails to serve standardized and discriminative task with three main reasons.
Firstly, the navigation task involving visual perception and memorization of a partially observable map is extremely hard for current models: without any additional hamesss, all models we tested in Table [ fail in navigation tasks. To enable navigation,
follow implementation by Anthropic and Gemini-2.5 (Anthropic 2025 Comanici et al., 2025), which requires a sophisticated harness that reads internal emulator states like tile maps, collision flags, and warp locations and exposes structured grid or mini-map to the model.
Secondly, once the navigation harness is in place, other tasks including battle control become very simple: with reasonably well-trained team, gym battles are often decided by level advantage and type matchups, leaving limited headroom to distinguish model capabilities.
Model
Steps to reach Oak's Lab
Cost
Time
OpenAl o3
1000
$120
20h
Gemini 2.5 Flash
1000
$50
13h
Table 18: Cost analysis on o3 and Gemini-2.5-Flast for reaching the first checkpoint, Professor Oak's Lab, in Pokémon Red.
Thirdly, the most important aspect of Pokémon Red as an evaluation full-game, long-horizon team training- with unfavorable cost overhead when used as standardized benchmark. As shown in Table [8] even in a very carly-game setting like reaching Professor Oak's lab, we observe that a single st000-step roughly 1,000 actions, about 120 dollars in API cost, and around 20 hours of wall-clock time, while Gemini 2.5 Flash needs similar number of steps but still costs around 50 dollars and takes roughly 13 hours. Extrapolating to full playthrough with tens of thousands of actions implies thousands of dollar costs per run and multi-day runtime, making it difficult repeat experiments enough times for robust, statistically meaningful comparison across models and hamess variants.
already incurs substantial cost and latency: for example, OpenAl o3 requires
I STANDARD ERROR
This appendix provides the standard error version of the main results shown in Table 19 All reported scores follow the format mean + standard error, offering clearer view of run-to-run stability
48

---

Published as a conference paper at ICLR 2026
Table 19: Model performance raw scores, evaluated in both with and without harness. For games marked with 1, evaluation for text-only models is not supported. The reported results represent averages over three runs, except for models marked with * "N/A" unsupported or unavailable.
Model
Harness
Sokoban
SMB
Tetris
2048
Candy Crush
Attomey
claude s-soolee--- 252202222222241022
No
0.0=0.0
1540.0=125
12.3+115
57,8+9.5
17.0=104
1.0=000
0.0s0.
1267.74279.5
14.7+07
108.2433
106.04308
2.0=00
claude- 3-1e---7-- 77-somet-20250211 (thinking)
No
0.0ga0
1430.0:996
13.0a00
114.2:44.1
126.3
3,0200
2.3g
1418.74881.2
16.3
113.4
484.0111
7.0g00
No
N/A
N/A
N/A
N/A
N/A
N/A
deepseck-rl
1.3
N/A
14.3400
105.24:7
447.3
0.0g00
gemini 2.5. preview- 04-11
No
0.0gn0
1540.7
19.0
97.74m0
1,0200
Yes
1.7200
1395.0
16.3
106.6231
334.74998
4.0200
No
10000
1025. 2025_3
12.3+1.
120.5
177.334775
8.0200
Yes
4.3
1498.3 11774
23.3
117.3
416.3.19
7.0200
Ilama-4 4 1aataan--maerick-11 Beerricck--bb---- ins-iinsttus--pp
0.0.00
786.0:22771
11.7401
32.3-28
Yes
0.0+00
1468.74 443074
10.3+000
106.0:22
128.7+33.0
0.0:00
-00-411-202-00--4
0.0an0
1991.3 33.58.0
13.0
94.5
101 01:.0
0.0g00
Yes
0.0g00
2126.3
13.740.
105.7::40
182.04 16.6
2.0200
gpt-4a-2024-114. 11-20
No
0.0
1028. 37:36.
14.7
70.4
59.0=3315
0.Da00
Yes
0.0:00
2044.3+30499
14.0#2.1
106.7#22
147.3+30.8
0.0=00
o1-2024-12-17
No
0.0
1434.0a..0
13.0
128. 1::00
90.0=0.0
3.0=00
2.3
855.0
35.0a00
128.9:0m
159.0100
16.0a00
No
N/A
N/A
N/A
NIA
N/A
N/A
61 mini 2922-09-12
1.3:003
N/A
11.7407
114.0s22
03-2025-04-16
No
2.0
1955.0200
31,0
128. 2:00
106. 106.0.00
8,0=00
8.0g2.
3445.0200
42.0a00
128.0a00
647.0am0
16.0.0.
202ini-2225-0-116
No
1,3003
1348.3+1028
15.0#2.1
97.6=165
110.7
2,0=00
Yes
5.3.607
1448.0as22
25.3:49
120.6:22
487.3
4,0a00
compared variance. For models evaluated with only a single run (marked with *), the standard стror is effectively zero. "N/A" indicates evaluation settings that are not applicable, typically due to missing vision capabilities.
49


Reviews:
[{"review_id": "cia5UuE5ok", "reviewer": "Reviewer_GmFK", "summary": "## Summary  \nThis paper introduces **LMGame-Bench**, a modular, Gym-style benchmark constructed around six popular games (Super Mario Bros., Tetris, Sokoban, Candy Crush, 2048, and Ace Attorney). A core design feature is a **toggleable “gaming harness”** (comprising perception, memory, and reasoning modules) that isolates distinct capabilities and expands performance headroom. Complementing this, the benchmark integrates **contamination checks** and **prompt standardization** to reduce evaluation variance. Experiments conducted on 13 state-of-the-art models demonstrate that the harness significantly improves the benchmark’s ability to discriminate between models while uncovering key failure modes—including limitations in visual state extraction, spatiotemporal control, self-reflection, and long-context reasoning. The paper further employs **correlation and low-rank decomposition analyses** to link game-specific performance to broader clusters of LLM capabilities.  \n\n\n## Strengths  \n1. **Original and Impactful Contribution**: LMGame-Bench addresses a critical limitation of existing game benchmarks—their tendency to entangle multiple skills—by introducing a modular harness that isolates distinct LLM/VLM capabilities (perception, memory, reasoning). This design enables fine-grained diagnosis of models’ strengths and weaknesses, making it a valuable tool for guiding model development.  \n2. **Rigorous Benchmark Design**: The benchmark strikes a balance in difficulty (avoiding both premature saturation and excessive hardness) and covers diverse game genres (platformers, puzzles, narrative games), ensuring it effectively discriminates between state-of-the-art models. The inclusion of contamination mitigation (e.g., entity masking, paraphrasing) and prompt standardization (via DSPy’s SIMBA optimizer) further enhances its robustness—a key prerequisite for reliable LLM evaluation.  \n3. **Comprehensive Experimental Design**: The evaluation of 13 models (under both harnessed and unharnessed settings) combines quantitative methods (paired-sample t-tests, Glass’s δ effect sizes, correlation analysis) and qualitative insights (failure mode analysis). By linking game performance to core LLM capabilities through low-rank factorization, the work delivers actionable insights beyond mere raw score comparisons.  \n4. **Candid Limitation Identification**: The paper openly highlights models’ weaknesses—such as challenges in visual state extraction, spatiotemporal reasoning, and long-context retrieval—and proposes concrete directions for improvement. This avoids the common pitfall of overemphasizing benchmark performance without addressing actionable gaps in model capability.  \n\n\n## Weaknesses  \n1. **Lack of Quantitative Support for Qualitative Analysis**: While Section 3.2 presents a qualitative analysis of model failures, it lacks accompanying quantitative validation. For instance, the paper could address this gap by using LLMs to annotate a subset of game trajectories, identifying the key failure reasons (e.g., “incorrect visual state parsing” vs. “poor long-horizon planning”) for each episode, and calculating the statistical proportion of failures attributed to each cause. Conducting this ablation experiment on a small scale would significantly enhance the credibility of the benchmark’s diagnostic claims.  \n2. **Limited Diversity in Evaluation Metrics**: The paper relies primarily on raw scores to evaluate model performance. While raw scores effectively reflect game progress from a human perspective, they provide limited procedural feedback for LLMs—failing to capture nuanced capabilities like reaching critical game nodes (e.g., accessing a bonus area in Super Mario Bros.) or acquiring key information (e.g., identifying critical evidence in Ace Attorney). Incorporating such procedural metrics would offer a more holistic view of model capabilities.  \n\n\n## Questions  \nQ1: Could the authors supplement the qualitative analysis with small-scale quantitative validation (e.g., trajectory annotation and failure reason statistics) as suggested in Weakness 1?  \nQ2: Could the authors propose additional, more diverse evaluation metrics—including procedural feedback indicators—to better capture nuanced model capabilities, as outlined in Weakness 2?", "questions": "See Summary", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "See Summary", "weaknesses": "See Summary", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mcTtZFyYSL", "reviewer": "Reviewer_uZi7", "summary": "This paper introduces LMGame-Bench, a benchmark for evaluating large language models (LLMs) and vision-language models (VLMs) on six popular video games (Super Mario Bros., Tetris, Sokoban, Candy Crush, 2048, Ace Attorney) via a unified Gym-style API. Unlike prior game benchmarks that entangle multiple skills, LMGame-Bench uses a modular harness (perception, memory, reasoning modules) to isolate specific capabilities, supports both scaffolded and unscaffolded evaluations, and enhances robustness through data contamination mitigation and prompt standardization. Evaluations of 13 state-of-the-art models show the benchmark effectively discriminates performance (o3 and o1 lead), reveals correlations between games and core LLM capabilities (e.g., Sokoban aligns with math/coding, Ace Attorney with language understanding), and identifies model limitations in visual state extraction, spatiotemporal reasoning, and long-context processing.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Modular Harness Design: Addresses a key limitation of prior game benchmarks (entangled skills) by enabling selective activation of perception, memory, and reasoning modules. This allows fine-grained diagnosis of model strengths/weaknesses (e.g., separating perception failures from planning gaps) that was previously unachievable.\n2. Rigorous Experimental Design: Evaluates 13 models across 6 diverse games (platformer, puzzle, narrative) with standardized metrics (progression/long-horizon rewards) and statistical validation (paired-sample t-tests, Glass’s δ, coefficient of variation). Results are consistent and reproducible, with detailed ablation studies for harness modules.\n3. Insights for Model Improvement: By identifying specific failure modes (e.g., VLMs struggle with board state extraction from images, non-reasoning models lack self-correction), the paper guides concrete advancements in model architecture and agentic design.", "weaknesses": "1. Limited Game Diversity: While the 6 games cover 3 genres, they lack representation of real-time strategy (RTS), open-world, or multiplayer games—domains that test collaboration, dynamic resource management, or complex opponent adaptation. This limits the benchmark’s generalizability to broader game-based agentic tasks.\n2. Computational Cost Opacity: While the paper mentions high computational costs (Appendix B.4), it does not provide concrete guidance for scaling evaluations (e.g., cost-saving strategies beyond vague suggestions like \"bounding trajectories\"). This may limit accessibility for smaller research teams.\n3. Perception Module Efficacy: For games like Super Mario Bros., how accurate is the textual representation generated by the perception module in capturing dynamic spatiotemporal cues (e.g., enemy speed, jump physics)? Could gaps in this representation explain why the harness provides limited gains for this game?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NnFDhmaglZ", "reviewer": "Reviewer_2PLk", "summary": "This paper introduces a new benchmark for evaluating LLMs on game-playing tasks, testing 13 state-of-the-art models using a direct screenshot-to-action setup. The authors find that these models perform poorly, often close to random, highlighting their weaknesses in visual perception and long-horizon decision-making. The study also explores where and how these models fail as task difficulty increases and demonstrates the effective use of the proposed harness.", "questions": "1. How does this benchmark fundamentally differ from existing benchmarks that also test LLMs or VLMs on interactive or game-based tasks? Specifically, how are these selected games different in terms of difficulty and multi-modality from the other existing benchmarks/games?\n\n2. Is there any difficulty level or stage where even the module-based models fail to improve? The authors should clarify how far these modules can enhance performance and at what point their effect saturates or diminishes.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. A new benchmark consisting of complex goal-driven games is introduced in this study. \n2. An extensive suite of models is evaluated, covering 13 state-of-the-art architectures.\n3. The problem statement and the experimental framework are well designed and presented.\n4. The authors perform detailed and consistent evaluations across difficulty levels, revealing how and where models fail.", "weaknesses": "While the work is interesting and systematically executed, many of its findings align with prior studies that have already established similar limitations of LLMs and explored methods to overcome them (e.g., Chain-of-Thought reasoning, embedding API calls, or memory modules/database access). The novelty and contribution of this work, therefore, feel limited unless the authors can better justify what new insights their benchmark offers.\n\nAdditionally, while the authors show that adding different modules enhances performance, they do not explain why these modules lead to improvement. If such reasoning is provided, the authors should indicate the corresponding line numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6JXFnb8cWw", "reviewer": "Reviewer_zKTd", "summary": "The paper introduces LMGame-Bench, a new benchmark designed to evaluate large language\nand vision–language models (LLMs/VLMs) through six popular video games: Super Mario Bros.,\nTetris, Sokoban, 2048, Candy Crush, and Ace Attorney. The benchmark uses additional modular\ngaming harness composed of Perception, Memory and Reasoning modules, which are used to\nassess specific model capabilities, and allow the models to perform at the level of human\nplayers. The benchmark also takes advantage of the DSPy’s SIMBA optimizer to standardize\nthe prompts.\nLMGame-Bench allows for a controlled evaluation of LLMs on video games and offers a\ncomprehensive evaluation of 13 state-of-the-art models, while relying on robust techniques such\nas prompt standardization and contamination detection.", "questions": "Regarding the gaming harness. It is not clear to me if you can \"toggle on or off\" them selectively.\nFor example, is it possible to only activate the Perception Module without activating the Memory\nModule, or viceversa?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Gaming Harness: The design of the gaming harness is highly effective and it represents a\nrobust method for isolating and accurately evaluating specific model capabilities and skills.\n2. Data Contamination Detection: The inclusion of an explicit mechanism for detecting data\ncontamination is a significant strength\n3. Prompts Standardization: The decision to standardize prompts by leveraging the DSPy\nframework is highly commendable. This approach ensures consistency and reproducibility\nacross experiments.", "weaknesses": "The primary weakness lies in the overall presentation and clarity of the paper. I recommend\nrepositioning the Related Work section to immediately follow the Introduction. This structural\nchange would more effectively contextualize the work and highlight the paper's novel\ncontributions earlier. Also, given that this is a benchmark paper, the explanation of the Metrics\n(specifically the Raw and Aggregated Scores) should be expanded, possibly reserving one\nsubsection just for this point.\n\nAlso, there are some typos in the main paper. For example line 53:\n\"To address this issue and also enable controlled evaluation, we enriches our evaluation\nsettings by developing gaming harness\". \"we enriches\" should be \"we enrich\".\n\nAnd line 425:\n\"Super Mario Bros. is excluded\" should be \"Super Mario Bros is excluded\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tMfKSZKpN4", "reviewer": "Reviewer_tEMA", "summary": "The paper proposes a benchmark for LLMs on six game environments, they evaluated a variety of popular LLMs. The paper also studies contamination and attempts to create standardize prompting.", "questions": "Could the author better argue what’s the difference between their “harness” compared to typical scaffoldings used in most related benchmarks?\nCould the author try to better argue where the novelty of the benchmark comes from, and what insights are actually new, especially when compared to the many existing benchmarks in the area?\nWhy using o3 for perception? This feels like a methodological flaw.\nHow many seeds were tested for each game?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "Having the human average for each of the games and random baselines is good. The study on contamination detection is interesting, and so is the attempt at standardizing the prompting with  DSPy standardization", "weaknesses": "I would argue that testing LLM without harness doesn’t make much sense at all. In the context of LLM Agents, the LLM is the brain (without memory module), the agentic scaffolding, or so called “harness” in the paper, is equivalent to affordances and tools such as arms, legs and eyes, memory. It’s not surprising to me at all how a naked LLM without these affordances (scaffolding or harness) often doesn’t outperform random. Memory particularly is very important for partially observable MDPs. \n\nThe benchmark could have been valuable around a year prior to submission, but now comes in a much more crowded space of benchmarks for LLM/VLM Agents, and the the papers struggles to find novelty both in the methodology as well as in insights provided.  The variances in most of the results are extremely large, and it’s difficult to disentangle real results from noise. The paper attempts to do something very similar to existing benchmarks specifically Balrog [Paglieri et al. ICLR 2025], and given the many similarities a more thorough comparison would be needed to explain what novelty (whether methodological or new insights) this paper brings. \n\nUsing o3 to generate perception traces also seems like a big methodological mistake as the rest of the models' capabilities will be influenced by another model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)