LLMs as Rules Oracles: Exploring Real-World Multimodal Reasoning in Tabletop Strategy Game Environments
Authors: Joseph J Peper
Sai Krishna Gandra
Yunxiang Zhang
Vaibhav Chennareddy
Shloki Jha
Ali Payani
Lu Wang
Venue: ICLR 2026 Poster (2026)
Keywords: game reasoning
multimodal qa
vision-language grounding
benchmark
situated reasoning
tabletop games
board games
Primary area: datasets and benchmarks

Abstract: We introduce **LudoBench**, a multimodal reasoning benchmark that evaluates whether vision-enabled large language models (LMs) can acquire, integrate, and reason over heterogeneous game knowledge in mainstream analog tabletop games. Unlike prior works that emphasize deep strategic mastery, LudoBench targets an initial reasoning challenge uninitiated gamers face: *correctly comprehending a new tabletop strategy game for the first time*.  

We examine whether, given a visual depiction of a tabletop scene and a corresponding ruleset, a model can correctly answer grounded questions about the pictured scenario. Concretely, LudoBench tests three cumulative situated game-comprehension capabilities: (1) *Environment Perception*, (2) *Heterogeneous Rules Integration*, and (3) *Short-horizon Optimization*, to progressively stress-test the foundational reasoning required for real-world game comprehension. 

Evaluating frontier LMs on three diverse strategy games, we find that even the strongest models achieve only ~68% accuracy on simple environment perception tasks and fall below 10% on situated multi-step comprehension puzzles that hobbyist gamers can routinely solve.
Our extensive failure analysis and knowledge-ablation experiments reveal that *models largely fail to comprehend rich cross-modal reference knowledge* and are subsequently unable to apply this knowledge to messy and unfamiliar situated environments. Our findings highlight the many steps remaining for current methods to succeed on complex multimodal reasoning in the real world.

Published as a conference paper at ICLR 2026
LLMS AS RULES ORACLES: EXPLORING REAL- WORLD MULTIMODAL REASONING IN TABLETOP STRATEGY GAME ENVIRONMENTS
Joseph J. Peper
Sai Krishna Gandra
Yunxiang Zhang*
Vaibhav Chennareddy!
Shloki Jha'
Ali Payani?
Lu Wang
University of Michigan
Cisco Research
ABSTRACT
We introduce LUDOBENCE"] multimodal reasoning benchmark that evaluates whether vision-enabled large language models (LMs) can acquire, integrate, and reason over heterogeneous game knowledge in mainstream analog tabletop games. Unlike prior works that emphasize deep strategic mastery, LUDOBENCH targets initial reasoning challenge uninitiated gamers face: correctly comprehending a new tabletop strategy game for the first time. We examine whether, given visual depiction of tabletop scene and a corresponding ruleset, a model can correctly answer grounded questions about the pictured scenario. Concretely, LUDOBENCII tests three cumulative situated copreeeomrennnnion capabilities: (1) Environment Perception, (2) Heterogeneous Rules Integration, and (3) Short- hhori-horizon Optimiza- tion, to progressively stress-test the foundational reasoning required for real-world game comprehension. Evaluating frontier LMs five diverse strategy games, we find that even the strongest models achieve only 766 accuracy on simple envi- ronment perception tasks and fall below 13% situated multi-step comprehen- sion puzzles that hobbyist gamers can routinely solve. Our extensive failure analy- sis and knowledge-ablation experiments reveal that models largely fail to compre- hend rich pross-modal reference knowledge and are subsequently unable to apply this knowledge to messy and unfamiliar situated environments. Our findings high- light the many steps remaining for current methods to succeed on complex mul- timodal reasoning the real world. Benchmark, leaderboard, and visualizer are available https: pettps:/huute httgss/ggggggiggegeeeeeee co/ co/spaces/// http//aanchhhaaa LudoBenchh
INTRODUCTION
Game reasoning prevalent area of research, spanning applications from strategic gameplay to human interaction (iilvrr et al. 2018 Lin al. 2024
Costarelli et al. 2024). Frontier LMs such as ol (OpenAl 2024c) and Gemini Pro Google 2025) perform well structured, simulator-
and rules-enforced digital environments, including games such as chess and Go (Zhang al. 2025 Guo et al., )024);; however, much of this progress utilizes rules-enforced or gym-style infrastruc- ture (Towers al., 2024 Brockman et al. 2016) that exposes clean environment, disallows illegal actions, and handles state progression. When this scaffold is removed, even strong models pro- pose invalid or nonsensical moves even for well-studied games such chess (Hwang al. 2025 suggesting incomplete internalization of game state and rules.
With this in mind, we pose the question: Can LMs serve as rules oracles in complex, real- world, gaming environments? This question is especially pertinent to the domain of tabletop strategy games, which are produced by the thousands each year and played primarily offline settings [Naghikhani 2024). with long, multimodal, and idiosyncratic rulebooks that notoriously
"Correspondence to jpeper @umi edu.. Ludoo-from from Latin lūdö, play", and the root of ludology, the scholarly study of games.

---

Published as a conference paper at ICLR 2026
impose substantial cognitive load an image a tabletop game scene and a corresponding external ruleset, can a model acquire dense cross-modal reference knowledge, retrieve and disambiguate the relevant clauses, parse iconogra- phy and diagrams, ground them in a cluttered spatial layout, and apply them correctly, without any legality checker or interactive affordances?
players (Heron al. 2018). For example, given only
To explore this challenge, we introduce LUDOBENCH, a multimodal game comprehension bench- mark explicitly designed to evaluate vision-capable LMs on situated game comprehension in con- ventional tabletop strategy games. Unlike traditional benchmarks centered on long-term optimiza- tion, LUDOBENCH emphasizes the ability learn and apply new rules, interpret dense, real-world game states, and make optimal tactical decisions, mirroring the foundational tasks faced in the first-time player setting. Specifically, the benchmark covers three cumulative competencies: (1) Visual Environment Perception, which involves identifying and interpreting key objects, compo- nents, layouts, and spatial relations within a physical tabletop game, (2) Heterogeneous Ruleset Integration, which requires retrieving, disambiguating, and applying information from complex, multimodal rulebooks to the current game state, and, to test comprehension, timization Puzzles, which challenge models to, for example, find the action sequence that maximizes the player's ss end-of-ture of-turn score.
(3) Short-Horizon Op-
To summarize, main contributions and findings are as follows:
We hand-curate and release LUDOBENCH, multimodal game comprehension bench- mark of 638 QA examples from five mainstream tabletop games that are varied in rule complexity and reasoning challenges, with each example pairing visual game state and the game ruleset with a question.
State-of-the-art LMs significantly under-perform humans on complex situated game comprehension: Testing nine frontier LMs, find that average model performance is only ~-63% on simple perception tasks (Tier 1), ~366 on game rules reasoning (Tier 2), and under 10% on optimization puzzles (Tier 3). In contrast, even Tier 3 puzzles are very solvable (~-22% accuracy) for hobbyist players.
LUDOBENCH surfaces diverse multimodal reasoning failures: Despite having full ac- cess to both the visual game state and rulebook, models frequently fail at scene parsing, rule retrieval, spatial manipulation, and temporal state tracking. Interestingly, although illustrated image-based rulebooks are expected offer richer guidance, mirroring how hu- mans learn from demonstrations, models often perform worse with these visual inputs than with plain-text rules, frequently overfitting examples and failing generalize. This re- veals core limitations in symbolic grounding and visual-contextual reasoning. We conduct extensive breakdown of these with annotated examples and case studies (Fig. [ Sec. Appx. D and
RELATED WORK
Game Reasoning with Large Language Models. Games have long been a central testbed for AI progress, especially in planning and decision-making. Reinforceeeent-earming systems such as AlphaGo and AlphaZero have demonstrated superhuman ability in
settings through extensive self-play and domain-specifit optimization (Silver al. 2016 2018). Recent ad- vances in LMs have enabled broad generalization across unfamiliar language, vision, and interaction tasks, often in zero-shot settings without fine-tuning (Brown et al. 2020 OpenAl 2023 Wang al.) 2023). The shift expands the scope of game reasoning: from executing within tightly scoped, rules- enforced environments acquiring and applying new knowledge unfamiliar settings. Building on this perspective, we examine how foundation models adapt to scenarios requiring rapid knowledge acquisition and environment comprehension.
Game Reasoning Benchmarking. Prior works have studied how LMs reason in text-based or programmatic (symbolic) game environments where rules and states are explicitly serialized (Hu et al. 2025a bl Lin et al. 2024| Costarelli et al.) 2024). These settings evaluate whether models can reason over explicit rule and state representations (c.g., "You are playing tic-tac-toe. The board has X at (1,1) and O (2,2)...)) "). In contrast, many real-world tabletop games present more vibrant,
2

---

Published as a conference paper at ICLR 2026
Game State
Multimodal Rulebook Knowledge
Scoring Rules
Placement Rules
A
Kingdom
territory is a group of matching
"When connecting domino another domino, horzzontally or verticolly, connecting squares the terrain type..."
Text
terrain squares connected
B
hortzontolly or vertically, score squares multippied the number of within it...
C
D
Image
1x3 =3
Tier How many crowns are pictured?
Tier How many points does the kingdom score? 3
Tier What new
most points?
Rationale
Tier 2: mines square 2 Wi 1 lake square 1 WT = (1x2) (1x1] = points.
Tier 3: Tile A has no W so scores points. Till B would score forest x square 1poins. Tile C seems to have potential given the existing lake however, tiles only connect orthogonally for scoring, the lake tiles will score independently (Ix1 + 1x1).
The swamp square
connect existing swamp square, netting swamp squares
W points.
Figure I: We illustrate examples (simplified, for presentation purposes) of questions from all three LUDOBENCH tiers. Tier (Perception) measures the ability to extract basic visual information from the game state; Tier 2 (Knowledge Integration) assesses how well the model applies game rules to situated questions; Tier 3 (Optimization) assesses the model's capacity to solve tactical puzzles akin to those frequently made during gameplay. Due to the situated nature of our task, models must integrate heterogenous knowledge across multimodal rulebooks and game states. Note: Figure provides extended gameplay examples and analyses of common challenges posed by LUDOBENCH.
cluttered scenes (e.g., an image of game with dozens or hundreds of diverse units and components on war-game map), making serialization nontrivial. Capturing these real-works nuances and the limitations they impose on current models is core aim of LUDOBENCH.
Beyond text- and cembolic-oompreeeniion settings, other work evaluates visual understanding by using board images schematic diagrams for common games such as Chess, Connect Four, and Tic-Tar Tac-Tor (Saha et al.) 2024; Wang et al. (025). These studies test whether models can align visual inputs with rule-consistent logic, but they still typically operate simple games with highly constrained layouts and minimal iconography. In contrast, LUDOBENCH targets visually rich, real- world scenes paired with full rulebooks, requiring models to parse heterogeneous components, trieve and disambiguate relevant rules, and make grounded decisions in unfamiliar environments.
Another related line of work probes core reasoning competencies via synthetic games with evolving rulesets: ARC tests visual abstraction over grid puzzles, while Baba Is AI, PuzzleVQA, and Algo- PuzzleVQA extend this to dynamic logic and fine- aneegra puzzle solving (Prize 2025 Cloos al.,
2025 Chia et al. 2024| Ghosal et al., 2024). These tasks isolate foundational reasoning competen- cles; however, they are often tiy-like not necessarily indicative of performance on real-world games. A complementary thread investigates long-context decision-making in interactive environ- ments (e-g., TexiQuests and Crafter) (Phan et al., 2025) Hafner] (022), often within rule-enforcing simulators with memory/tracking scaffolds. More broadly, many evaluations rely enforced structure, cleaned inputs, or oracle feedback (Costarelli et al. 2024: Lin et al. 2024 Topsakal al.) 2024 Balla et al. 2024), limiting applicability to analog tabletop settings. LUDOBENCH pairs intricate, real-world visual game states with often lengthy, heterogeneous rulebooks and situated questions, unifying these varied directions into a single, real-world setting and advancing tangible application for vision-enabled LMs: acting as multimodal rules oracles for offline play.
3

---

Published as a conference paper at ICLR 2026
Landbeape
Kingdomino
Carcassonne
Catan
Res Arcana
Pax Renaissance 2dd
Figure 2: Example game states from each game. Our benchmark requires models to perceive visual game states in sandhox environments, integrate rule knowledge, and perform situated reasoning.
Game
Rulebook
Diff.
Unique Game Properties
# Rules
Figs.
Kingdomino
4 pg.
1.2
tile-laying, spatial scoring, individual player areas shared tile-laying, dynamic board topol- ogy. position-coded roles
35
Carcassonne
8 pg.
1.9
39
30
Catan
16 pg.
2.3
network building, connectivity constraints, action chaining
44
Res Arcana
12 pg.
2.6
card card-based interactions, heavy symbol us- age, card orientation, action sequencing shared map. private cards/tableau, large numher of components, intricate ruleset
112
31
Pax Ren. (2e)
44 pg.
4.6
247
58
Table 1: Descriptions of the five benchmarked games, which are diverse in ruleset length and difficulty. In addition to game difficulty, we also roughly quantify the number of atomic rule clauses and supporting rulebook figures and illustrations. Difficulty is community-calculatie game com- plexity metric, ranging from (1 = casy, 5 = hard).
3 LUDOBENCH DETAILS
3.1 TASK FORMULATION
In this work, we explore the domain tabletop strategy games, with an emphasis on situated visual-language reasoning that faithfully reflects the experience of real-world gamers during first- time gameplay.
We formulate the task visual comprehension task with the following inputs: (1) game state G depicted by one or more images, (2) The game's 's corresponding ruleset R (often containing mix of text, diagrams, iconography, and worked examples), and (3) A natural-language query Q to probe the provided knowledge. An LM must then output an answer A which resolves Q given (G, R).
We define three game comprehension tiers of interest, emphasizing the cumulative levels of com- prehension needed to solve game reasoning tasks:
Tier 1 Environment Perception. These examples probe basic perception of G, such as item counts, distinguishing colors, identifying spatial relations, without requiring any rule knowledge; accurate scene parsing skills and generic gaming familiarity sufficient for success. The ruleset (R) is not needed for solving these problems.
Tier 2 Heterogeneous Rules Integration. The model must ground one or more rule clauses from R within the observed visual state assess game-specific information such as game state validity, current player score, or the valid actions player may take.
Tier 3 Short-Horizon Optimization. Beyond game state assessment and isolated comprehen- sion, in this tier the model must suggest or predict the optimal outcome from a given game state and provided constraint expressed in Q. Our emphasis is short-term optimization such cal- culating an optimal action sequence given tractable, short-horizon constraints (e.g., "what is the maximum points the player can score by the end of the current round"). Solving such problems
4

---

Published as a conference paper at ICLR 2026
necessitates comprehensive game understanding and requires models to formulate an internal world model capable of lightweight simulation.
3.2 BENCHMARK DETAILS
We hand-annotate 638 QA examples in the described task format across five mainstream tabletop gamnes-Kngddomig Carcassonne] CatarF) Res Arcana and Pax Renaissance (2e [eeeeccee for diversity in component density, ruleset length, and mechanical focus (Table In total, we have 100+ examples from each game and 200+ examples in each reasoning tier. All scenarios staged in Tabletop Simulator an interactive sandbox environment which affords free camera control and precise piece manipulation, enabling reproducible configurations and high-quality captures of visual game states (Appx.I).
&
As the first in-depth study of situated multi- modal game comprehension that pairs complex, multimodal rulesets with intricate visual tabletop game states, we prioritized fidelity and correct- ness through hand-crafted examples. Through this process, find that our situated visual states are effectively reusable scaffolds for expansion; likewise, find that rulebook worked exam- ples can be synthesized to quickly brainstorm meaningful new questions, topics, and scenario variants. We believe lightweight human-in-the- loop curation approaches offer a strong balance of quality and scale for this domain.
Game
Tier
Tier 2
Tier
Kingdomino Carcassonne
40
40
50
40
40
50
Catan
40
Res Arcana
40
40
50
Pax Ren.
40
40
57
Table 2: LUDOBENCH comprises 638 questions across five games (Tier 1: 200; Tier 2: 200; Tier 3: 238).
Example-Selection Heuristies. We use the following heuristics to ensure high-quality examples that are both easy to produce, and have verifiable ground-truth solution: (1) Deterministic, fully observable game states. The correct answer derivable from what visible in the provided game state images; hidden information and chance events excluded. This ensures that reasoning is grounded in visible evidence and that answers are reproducible by both humans and models. (2) Scoped opponent modeling. For multi-turn optimization tasks (Tier 3), we either explicitly assume no opponent interaction, or, clearly specify the expected opponent behavior: (e.g.. "on their turns, assume opponents will us to focus instead on assessing rules integration. (3) Single. well-defined solution. Each example crafted so that proper reasoning yields exactly one answer; ambiguous or probabilistic cases are avoided. This guarantees clear evaluation target and simplifies correctness verification during evaluation.
minimize your score"); this avoids ambiguity in outcomes, and allows
Annotation Process. We conduct three-stage annotation process to high-quality. First, we task a group of two gaming experts to produce and label QA examples using the Tabletop Simulator platform. Second, train and onboard three student annotators with hobbyist-level gaming experience to independently answers the questions (with the full rulebook available for reference). For every expert-hobbyist disagreement, resolve all conflicts to produce the final ground -tuth examples. We define annotator correctness as the proportion of examples where the original answer provided by either the expert or the hobbyist exactly matches the final resolved label in the benchmark dataset. We find that the Phase expert annotation yielded 98.3%, 97.4% and 87.2% correctness, respectively, for the examples from Tiers 1, 2, and 3 (i.e., this percentage of examples were not relabeled in subsequent rounds). Hobbyist performance accuracy (Phase 2) was 96.7 96.7%, 89.1% and 82.2% for Tiers 1, 2, and 3. Full details,
examples
then manually discuss and
https: http:////arrddaeeee.cc/////aaaaa ht http:////Bccaaaeeeeee.coooooorrr
-oaance-2nd-eiii

---

Published as a conference paper at ICLR 2026
100
Model
Haman (espett
90
Accurasy
60
Claude Gonnet
Gemire Pro
Geriiv Flach 6#77-0
Figure 3: Overall bier-by-tier game reasoning performance, averaged over combinations of games and rulebook modalities. Unlike humans, models struggle even with simple game state perception tasks (Tier 1), with further degradation when integrating rules knowledge into the questions (Tier 2), and single-digit accuracy multi-step comprehension puzzles (Tier 3).
including the annotator onboarding process and disagreement analysis case study can be found in Appendix
BENCHMARK EVALUATION
4.1 EVALUATION SETUP
We evaluate nine widely-used. multimodal-capable language models: OpenAl's 4o and its extended- context sibling 4.1; the cassoning-optimizer variants o1, o3, and 5.1: Google DeepMind's Gemini Pro 3, Gemini 2.5 Pro, and lower-latency Gemini 2.5 Flash; and Anthropic's hybrid reasoning model, Claude 4.5 Sonnet. Full details prompt construction, model configurations, and the evaluation setup are provided in Appendix(C
Prompt construction. For caching efficiency, the rulebook knowledge R is prepended the be- ginning of the input, followed by the game state image(s) G, then the natural-language query Q:
[Start Rules]
< (optional) zulebook knowledge> [End Rules]
<game-state (mage(s)> (s)>
<question + format specification:
All experiments are run with the OpenCompass VLMEVALKIT (Contributors) 2025). which unifies data parsing, batching, logging and answer extraction across models. Hyper-parameters are kept at their model defaults except noted here; we 1) raise max.tokens to each model's maximum, 2) impose a 300s timeout with up to five automatic retries for API-served models, and 3) omit the IMAGE-rulebook results for Claude 4.5 Sonner (Anthropic (025) on Pax Renaissance 2e and Catan, whose rulebook contents exceeds its token context window.
Evaluation Metries: We report the overall exact-match QA accuracy. By design, each example question specifies lightweight semi-structured or (for longer answers) JSON-style answer schema; model outputs are first normalized by a post-processing LLM step (GPT-4o) that maps free-form text into this schema, after which exact-match is computed against the structured ground truth. To ensure reliable automated parsing during evaluation, we manually verified this LM processing step, randomly sampled 100 model predictions. We find the processing step maps outputs to the expected structure with perfect accuracy.
Rulebook Format:
For each game consider three variants of the ruleset knowledge R:

---

Published as a conference paper at ICLR 2026
T1- Perception
T2 Rules Integration
T3
Short-horizon Optimization
0:36
0.52
00:22
0:755
0.35
0.62
0.7
0.20
0:31
0.35
0:30
0.17
0.37
6:72
5.02
0:03
D:1
0.26
10:66
0.50
0:36
0.00
kig0
8:43
0.55
0.00
0.20
885
952
0:32
0.30
2:27
6:73
503
0.02
0.12
0.24
9:00
0.50
0.50
0.04
No Aroone Hore
0:55
0.00
0:70
0.75
0.70
0.68
0.50
0.8
233
0:39
0:40
0:33
0:33
5.3
2:30
033
643
502
0:04
0:02
0:30
0:28
0.00
0.32
0.20
0.04
0.75
0.5
8:39
8:50
5:33
2:47
505
6:08
0:02
0.55
0.50
Pax Pooee
00.15
03.45
0.35
0:41
0.60
0.5
0.6
0.98
910
9:29
9:24
5.25
5:10
6:31
8.
5/3
8:33
0:12
18:22
0.59
0:32
Fox Nos Hoot
0.48
0.55
0:37
0.66
0.00
0.40
72
0.75
96
353
8:40
8:93
9:55
0.90
8:30
2:93
6/47
0:3
505
6:02
0.02
0:38
0:59
0.65
0.05
0:31
0.09
0:42
0:45
0:45
0.22
0.22
9:17
0.28
5:23
6/42
5:44
0:38
0:92
0.62
0.06
0.04
Carta. 1ikd
0:35
0:15
0:40
0.50
0.53
0:40
0.55
0.75
0.0
215
3:5
2:23
0.20
0:5
5.30
2:23
5/17
6:78
2.00
0:04
0:02
0:26
0:30
0.66
0.26
0.62
0.00
Caros. Image
0:40
0.38
0.57
0.55
0.33
030
3:30
5.13
0.20
5:20
5.15
2.04
0.04
0.08
0:30
0.04
0.06
Catan| Tlxt
0:45
0.33
031
0.7
0.22
0.62
221
020
5:35
0.31
2.03
0:25
0:43
0:28
0.00
0.30
0.00
Catan| Image
0:42
0.35
0.05
0.55
0.7
0:3
0:42
0.70
2.00
0:33
0:33
020
0:30
512
2.17
0.30
503
6:03
D:02
6.26
9:26
0.06
0.26
0:03
GPSS1
Genr Hhel
Gemee 2.
camee
Gamme
Figure 4: Accuracy breakdown. We display results of nine models on Tier 1-Tier tasks under three rulebook modalities (None, Image, Text) for each game (KingDomino, Res Arcana, Pax Renaissance 2e, Carcassonne, Catan). All models achieve their highest accuracy on Tier (Per- ception) and degrade sharply by Tier 3 (Optimization), confirming escalating and compounding task complexity. Note: the Pax Ren. and Catan image-modaling rulebook did not fit within the context window of Claude 4.5 Sonnet thus we do not report these results.
* IMAGE: High-resolution screenshots of the rulebook PDF are supplied in the prompt; all pages are human-legible. This full-fidelity representation captures the same rule informa- tion provided to our annotators.
*
TEXT: The text-only subset of the rulebook PDF is included in the prompt.
NONE: Only the game state G is provided; models can only use parametric knowledge.
4.2 OVERALL PERFORMANCE TRENDS
Figure reports tier-by-tier accuracy for every model, averaged over all games and rulebook con- ditions. First, we observe that even simple visual game perception tasks are challenging for frontier LMs, with average 63% performance on the Tier task. Across all systems, accuracy then degrades sharply, falling to 36% for Tier 2 (rules QA) and then to 8% for Tier 3 (Optimization), Of the models evaluated, we observe that Gemini Pro 3 is consistently the strongest, especially on Tier and Tier 2 reasoning. GPT-5.1 and o3 are also performant. We observe clear middle cluster Claude 4.5 Sonnet, GPT-4.1, 4o, ol, Gemini 2.5 Pro and Gemini Flash 2.5 with Flash 2.5 the weakest of the API-based frontier LMs. These results collectively highlight that even state-of-the- art multimodal LMs struggle to generalize across grounded reasoning tasks, indicating persistent limitations in visual parsing, rule grounding, and strategic planning.
Tier-specific Observations. Figure 4 provides detailed performance across all splits. This anal- ysis yields further takeaways, showing that:
(1) Environment perception (T1) is not yet solved - although some splits perform quite well (Gemini Pro averages 83.6% performance on Res Arcana Tier examples). Claude 4.5 Sonnet Tier 1 performance is as low as 30% (Carcassonne Img). We see frequent failures in small-object detection and counting under partial occlusion/clutter, yielding missed or miscounted gaming com- text/icon recognition and brittle layout-aware spatial-relation parsing, for example, when pieces sit on cards or are rotated.
ponents. Models also exhibit rrientation-depence
questions (T2) are nearly twice challenging as simple perception, having the highest accuracy of 75% by GPT-o3 (King Domino Image); while models often retrieve relevant rules correctly, we find they often misapply them due to flawed shallow game state comprehension, often suggesting invalid moves or making critical rule misinterprctations leading to faulty score calculations.
(2) Rules-comprehenteee
7

---

Published as a conference paper at ICLR 2026
GPT-4.1
GPT-S.1
0.6
Accuracy (S1
0.4
0.2
None
Image
Gemini 2.5 Flesh
Gemini Pro
Gemini 3 Pro
Claude 4.5 Sonnet
Nocurecy (60
0.4
0.0
Image
Image
Image
Figure 5: Rulebook modality analysis: overall accuracy for each model by rulebook modality, GPT and Claude families improve over the no-rulebook baseline with both image and text rulebooks; Gemini (except pro 3) improves with text but degrades with image. (Asterisks denote stat. sig. improvement vs. None; two-proportion z-test, two-sided; p .05, ** .005).
(3) Short-horizon optimization (T3) remains virtually unsolved, with only Gemini 3 Pro exceed- ing 20% performance on any split (Res Arcana Text). Figure 9 also shows that responses grow substantially longer in Tier indicating verbosity without commensurate reasoning gains. We see a systematic limitation models' ability to comprehend and then (validly) internally simulate even one two steps within complex, obbly--bseervable gaming environments.
ANALYSES
5.1 INFLUENCE OF RULEBOOK MODALITY
Figure aggregates model accuracy over games and tiers to isolate rule modality effects. Gemini Pro 3 performs strongly across modalities, reflecting both solid parametric knowledge and gains from text and image rulebooks. Gemini Pro 2.5 shows counterintuitive pattern - decreasing with image rulebooks yet improving with textual rulesets. All models benefit from textual knowledge, with Gemini Pro 2.5, o3 gaining nearly 7%. By contrast, image-modalit, rulebooks have mixed effects: GPT models benefit, whereas both Gemini models (except Pro 3) GPT-4. and Gemini Pro 2.5 were consistent across numerous examples during manual analysis; see Appendix
decrease. These trends for
E for details. In contrast humans who greatly benefit from visual structure current models still struggle to extract critical rule semantics from images, underscoring a need for stronger visual-document understanding.
5.2 BENCHMARK ORACLE SOLVABILITY
We assess benchmark head-room via plurality voting across all 27 (9 models 3 modalities) outputs (Fig. 77). Tier I: >90% of questions are solved by at least one system and 70% by plurality, con- firming task clarity. Tier 2: although most items remain solvable in aggregate (indicating models capable some knowledge integration), plurality accuracy slips to 50%, showing that robust rule grounding is still fragile 60% of Kingdomino questions and 50% overall remain unsolved; even the best plurality score (1 % on Pax Ren.) leaves large gap. For Kingdomino, explore this further (Fig. 6 Appx. DD and the combinatorial nature of Kingdomino (a) tile placement and (b) planning poses challenge for all models.
when the knowledge is present. Tier 3: solvability greatly decreases:
5.3 RULE RETRIEVAL ERROR ANALYSIS
To probe where game reasoning breaks down, analyse 30 Pax Renaissance Tier-2 questions under three input conditions- no rulebook, a text rulebook, and an image rulebook. For each, we annotate whether GPT-4o retrieves the relevant rule(s) and applies them correctly (Fig.
g We see rule
8

---

Published as a conference paper at ICLR 2026
retrieval climbs with richer input (20% 73% - 90%), yet application accuracy lags behind (17% + 64% - 56%).. In many stalled cases GPT-4o knows the rule but misreads the board (Appendix B.1 eπror images sometimes fix when the question hinges visual cues (Appendix B.2 This reveals a dual bottleneck: richer input almost eliminates grounding errors, but faithful rule execution remains elusive. Finally, the model often over-fits surface rulebook patterns, applying them rigidly and inappropriately. particularly on Tier-2 and Tier-3 tasks; Appendix catalogues these hallucinations and their inductive triggers.
Rt: "thhW wheeeooeeetnnng e domini tn snother demiine no. horbmetally or vertiaally loost 2 con connecting squirees have same tor rain type lono each domino)
Bdt Seariig Exomple
R5: Tile Drafting sest Manipulttion Esompple
Boamio SidecTion
R2: Scour lands of whoot he forests. lakes, gross sworpps, and mi miness
pleaan thok vdler aebiak zeelic
the
aliees piliiiiiing ing your kin gulon's
(Cpptionaallle
are
DEAN Eababbe
kingster's cingsoom moy
matching terroin that connected honusnt bally sertically (nok ciagorally"
$25
stote part
R7: "In score memiooryy count the sumber of conn eeting terniin sq coees and muiti pay than number by the nets number of crowis in indlin than terthrr..
D1: 'Mesple King". Colored tokens used risim dominow. This ralor motchun plaer's kingsom cnfor.
D2: TRk Domine' Arectangular game piece modeaal squam linn, shrwwnng terrain type crowes
RB: 'A players kinddom may not be more than 9x5 squares dimension."
Box
Pos.
Pos.
5
11.11
(2:1)
(3.1)
(4.1)
(5:11
(6:1)
A
E
(1,21
B
F
C
G
(1.4)
D
H
(1.5)
Pos.
Pos. 3
(1.6)
Pos. 2
Box
Type Mapping
Demonstrative Question
Arnwer
Cornect Annwer
Explaration
Canonical
rulebook groural
fwish, Idbas, swamps
helcs.
Spetial Enumenation Maniputation
"Hommany arique
full
placement options avaited rulos placedooterced
Lnique ways
Game State Perzeption Counting
"How maryy omply sofs ronsiis skayer' sarghhorn"
g
Model falls to correctly visualizs the Sxt kingdom (Rule 81 shown the game shite Perfaspe
model brunnd kinglom interectly vallo bound bindicated BBo
Scene Partiiooing
Aosame Etis piypers Burn phatt Hil they comently place?
CB
curment then are tile aad The model mocorectly scuress that the blue curnnt
un they shane 'mad Huseserr, thés inarrest- playyrs players only plaar the tike marted with mecple e ble CiRule
Misuspiiattone of Rules Constraints
'Wist the current soone the ploye's Vigdoo??
Bar audards a 2 bi swampx 2 1
Sgrassiands li swommp ig- 27
The modil the game state pretattiion correct, but fish apply the correct rule getting the curnent soore, it atsheeppet the terrain regions only simior loncs that are directly adjacett to the lands containing crowne. rather than connectidd terrans of
kind (Rule Rule Rule
Object Permanence dur Sequential Reasoning
"Iee the mecimmmm bingdoom shat green player avi
<ttle pontiions: Tik <Pas 12. TiE roPos
<tle preetinttnoonn: TikA <Pac 2n,
Medel downat understan plaeament sequenting (Rule has placed the frys Dle domino A Pusition 1 while reasoning about second the than invalitty places domino H Position inscedd Position 3. This platammnt and souting reflects only sacr invraed boch.
then forgets that
attar two placemonts linc siiictdd ald and posllans))"
Tik cPes 3-
Score:
Scome:34
Greedy Planning
"J/r the recommam bingdom srore live playe can achieve ofter owo elocemosts Sechude selecterfle prottiond!
<ttle posttions: Tile <Pos dig Tile .Poo Stoc.B
Tike <Pas
<tile prettions
Orignn 12 grostndo 2 Sivarox
H <Pos
Manels Stweei Mt
deseete 2
See Pown Thi Peettes
Plor The Portion selre,
dgproladta 24 of
We modill falk look freword and cherk dominos for selection.
instead foousing groadlly maximize the scores. Additinnaly by we nots that placing in Prottion 4 and 5 step proeiutes placemmnt into Profiions 1,2 and insteep and venue Selle
Figure 6: Illustration of common multi-modal game reasoning failures surfaced by LUDOBENCH. We provide key rule snippets from Kingdomino alongside a corresponding annotated game state, highlighting perception, rule application, and planning challenges. Ammotation/Rulebook references are bolded the explanation column. Note: highlighted annotations and labels (c.g., position IDs, rile boxes) used only for illustration clarity and are not part of the actual dataset inputs.

---

Published as a conference paper at ICLR 2026
Sabully Puailyy
concotiets
Modality: None
Modallty: Text
Modality Image
30
12
20
15
Figure Error breakdown heatmaps (rule retrieval vs. answer correctness) for 30 Pax Ren. Tier-2 questions un- der three input modalities: no rules, text rules, and image rules.
Benchmark solvability across games and bench- mark tiers. We aggregate 27 predictions per example (9 models x rulebook input variants).
Figure 7:
5.4 FURTHER ANALYSES
Due to the length of illustrated model outputs, provide full illustrated demonstrations for each game and additional qualitative analyses in the Appendices. These materials are designed to make model behavior auditable: they localize errors to (i) visual state recovery, (ii) rule retrieval and alignment, and (iii) multi-step application and planning under constraints.
Across models, (icons/cards/tokens are not reliably mapped to their semantics). brittle spatial reasoning (localiza- tion, adjacency, and orientation errors), and over-iteration in multi-step settings that derails initially correct logic. With image rulebooks, models sometimes pattern-match to visually similar rulebook examples rather than using diagrams as grounding evidence. Finally, many Tier-3 failures stem from missing intermediate state updates when multi-action moves require updating the board before ex- ecuting the next step.
observe recurring failure modes: weak grounding to rulebook images
Our extended analyses include:
Annotated Tier-3 puzzle walkthroughs that pinpoint where failures occur in the reasoning pipeline (state extraction vs rule grounding vs action selection), and how they cascade into incorrect plans (Appx. D
Rulebook modality comparison with side-by-side outputs under Text, Image, and None rulebook modalities, illustrating when additional context improves rule retrieval but still fails to guarantee correct situated application (Appx.
Rulebook hallucinations and overfitting patterns survey cataloging common error types (c.g., invented constraints, attribute transfer, and misread iconography), along with hypoth- esized triggers grounded in the provided state and rules (Appx.
When Claude Sonnet's "cautioos" reasoning helps analysis, highlighting cases where calibrated uncertainty and explicit verification reduce errors, well as where caution does not prevent state- tracktn failures (Appx. G
o3 behavior analysis, including overthinking, abstraction gaps, and brittle state tracking, which helps explain why strong general reasoning can still fail in long-horizon, component- heavy games (Appx. H).
Overall, LUDOBENCI offers a rich testbed for analyzing complex, situated multimodal game rea- soning, and the appendices provide concrete, game-specifir evidence to support these conclusions.
CONCLUSION
By introducing LUDOBENCII, we aim highlight the gap between current game reasoning research and the real-world knowledge comprehension faced in the tabletop strategy games setting. This benchmark highlights the need for further model development focused real-world situated game reasoning tasks, particularly in 'one-shot' learning scenarios where proper rule acquisition and game comprehension capabilities paramount.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Cisco Research. We thank David Jurgens, Rada Mihalcea, and Paramveer Dhillon for their insights and valuable discussions. We also thank the ARR and ICLR reviewers for their constructive feedback.
REFERENCES
Alibaba DAMO Academy. Qwen-vi 2.5 32b instruct.
http:://github.coom//e
Owen2. VL 2025. Accessed 20 May 2025.
Anthropic.
Introducing claude sonnet 4.5.
Slaude-sonnet-220 September 2025. Accessed: 2026-02-06.
Anthropic.
hEtps://www. bttss//ww..ntarriic.coo. claude-33 3- 3-7----nee- 2025. Accessed 26 July 2025.
Claude
3.7 sonnet.
Martin Balla, M. Long, George E. James Goodman, et al. Pytag: Tabletop games for multi-agent reinforcement learning. IEEE Conference on Games, 2024. 1XXi:22455.88123
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. URL /ttppt////// www.///gg//////eeeeeeeeeeeeeeeee hetp://..s//......../ 01540
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: / rxiv. org/ abs 2005. 14165
Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. Puz- zlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns, 2024. URL
org/ abs abs//403. 13315
Nathan Cloos, Meagan Jens, Michelangelo Naim, Yen-Ling Kuo, Ignacio Cases, Andrei Barbu, and Christopher J. Cueva. Baba is ai: Break the rules to beat the benchmark, 2025. URL https:
//arxiv.orq/abs 220pe//2407..0 13729
Google Cloud, Gemini 2.5 flash model card.
http://ooooooooleee...o/e/eeeeee http://cc/.oooeesseeeeem//////e/ 2025a. Accessed 20 May 2025.
Google Cloud. Gemini 2.5 pro model card. https://cloud. http:///......oooo.oooo//eeeeeee
/ggpeaatieeea///eeeee/oo///////p http:///moooeeeeemmmm/a/////0000 2025b. Accessed 20 May 2025.
OpenCompass Contributors. VImevalkit: An open-source evaluation toolkit for Ivims.
https:
www.thuu..coooppeeeeceete htg.at/b..ooooogeee..oo/aaa//aaa 2025. Accessed 20 May 2025.
Anthony Costarelli, Joshua Clymer, et al. Gamebench: Evaluating strategic reasoning abilities of Ilm agents. arXiv preprint arXiv:2406.. 06613, 2024.
Deepanway Ghosal, Vernon Toh Yan Han, Chia Yew Ken, and Soujanya Poria. Are language models puzzle prodigies? algorithmic puzzles unveil serious challenges in multimodal reasoning, 2024.
URL ltttps://axxiiv org/ abs/abbs..3 03864
Google.
Gemini
pro
preview:
Even
better
coding
per-
formance.
https://developeza wwogleblog.goooooooooooooooooooo www.///////////
2025.
Accessed 20
May 2025.
Google Cloud. Gemini 3 pro generative ai vertex ai. http:://csssssoscllloooo www..eo.o............oooooooo
http:///wwaaaaaaaaa.e.eeee////ea 02-066
Accessed: 2026-

---

Published as a conference paper at ICLR 2026
Hongyi Guo, Zhihan Liu, Yufeng Zhang, and Zhaoran Wang. Can large language models play games? a case study of a self-play play approach, 2024. UtLhttps::/ http//xxivv..o/ org/ absg//bbb//3333
05632
Danijar Hafner. Benchmarking the spectrum of agent capabilittes, 2022. URL httpp:////axx
org/ brg/abs/e abaabb/2/ aba/2109. 76780
Michael James Heron, Pauline Helen Belford, Hayley Reid, and Michael Crabb. Eighteen months of Meeple Like Us: An exploration into the state of board game accessibility. The Computer Games Journal, 7(2)75-955, 2018. doi: 10. 0177://440000-1--005560
URL https://lim:
springer. cwm wwwaicreeeei.0 100 940884 01 0056-9
Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric Xing, lon Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. benam. How good are llms at playing games?, 2025a. URL
https://arxis org/ abs 2505 15146
Lanxiang Hu, Qiyu Li, Anze Xie, Nan Jiang, lon Stoica, Haojian Jin, and Hao Zhang. Gamearena: Evaluating IIm reasoning through live computer games, 2025b. URL TRR./ttss://axxiv.orr
abs/2412.065 06394
Dongyoon Hwang, Hojoon Lee, Jaegul Choo, Dongmin Park, and Jongho Park. Can large lan-
guage models develop strategic reasoning? post-training insights from learning chess, 2025, URL
https://arxix: http/aaxxi....gaa/.. wwwp/rggg....ooooo../..../t..ast 2507 0077600776
Wenye Lin, Rohan Paul, et al. Beyond outcomes: Transparent assessment of Ilm reasoning in games.
arXiv preprint arXiv:2412.
2024.
Sourena Naghikhani. Beyond the box: A comprehensive market research of the board game industry, May 2024. ottp0///eooooooossrrcaaaaaaaaaaa
OpenAL. Gpt-5.1 model - gpt-5.1] Accessed: 2026-02-06.
OpenAl. Gpt-4 technical report, 2023.
OpenAL. GPT-4:: Openaï's real-time multimodal model. http:://openall.co//meee hello-gpt geo--ppt44//// 40/, 2024a. Accessed 20 May 2025.
OpenAl. GPT-4o system card. htttps:///dn..pp 2024b. Accessed 20 May 2025.
OpenAI. Introducing ol. bttps::/openai.ccom wttpppppaaa...coo//oooo//ttotttt 2024c. Accessed 20 May 2025.
OpenAl. GPT-4.1: Introducing gpt-4.1 in the api. https://ppenaa..com/ddee
1ptt4-17 2025a. Accessed 20 May 2025.
OpenAl. Introducing openai o3 and o4-mini. 2025b.
http:://opeaal.commineee
Lntroduciin--o---addd------
Long Phan, Mantas Mazeika, Andy Zou, and Dan Hendrycks. Textquests: text-based video games?, 2025. URL htLttttss:/
How good are llms at
ARC Prize. Arc-agi: The general intelligence benchmark. https://aroprize.ozz arc arc-agi 2025.
Soumadeep Saha, Saptarshi Saha, and Utpal Garain. Valued: Vision and logical understanding evaluation dataset. Data-centric Machine Learning Research, 2024. doi: 10.48550/arXiv23311..
12610.
David Silver, Aja Huang, Chris Maddison, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):4844488 2016. doi: 10.1038/nature 16961.
David Silver, Thomas Hubert, Julian Schritwwieser, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 140-6449)1114--1144 10.1 06//sciencee.a66444..
140-1144, 2018. doi:
12

---

Published as a conference paper at ICLR 2026
Oguzhan Topsakal, Mete Çişek, Yigit Kaya, el. Evaluating large language models with grid-based game competitions. arXiv preprint arXiv:2407.07796, 2024.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Vicente, An- drea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar Younis. Gymnasium: A standard interface for reinforcement learning environments, 2024. URL org/abs/2407. 17032
Guanzhi Wang, Yuqi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023.
16291
Xinyu Wang, Bohan Zhuang, and Qi Wu. Are large vision-language models good game players? In Proceedings of ICLR, 2025. arXiv:2503.02358
Yinqi Zhang, Xintian Han, Haolong Li, Kedi Chen, and Shaohui Lin. Complete chess games enable Ilm become chess master, 2025. URL TRLhttps:2 //wpxvaavvvv////////a//.aa./aiaa org/ abs 25002 1125501111111 2501.11181111
APPENDIX
A MODEL PREDICTION LENGTH ANALYSIS
Figure compares the overall model prediction lengths across tiers. We see Perception tasks (Tier 1) are notably shorter, averaging approximately 100 tokens. In contrast, integration (Tier 2) and optimization (Tier 3) notably increase prediction length, with Tier outputs averaging approximately 600 tokens. However, find that these additional prediction tokens do not simply correlate with improved performance, as evidenced in our extensive case studies.
examples with rules
Predicion (bo sca
10r
4.0
Figure 9: Prediction length vs example tier (log-scale). Models produce longer reasoning on the harder tiers, particularly tier although we observe that longer rationales do not result in improved performance.
13

---

Published as a conference paper at ICLR 2026
Renaissaucc
CENTER
Unteyy
5300
PAPALSTATES
OFTOMAN
MAMIUK
LOYOOO
Figure 10: Game state for the example question in Appendix B.1
B RULES RETRIEVAL CASE STUDY
B.1 MISINTERPRETATION GAME STATE DESPITE CORRECT RULE UNDERSTANDING
Question: How many blue concessions total are there on the map?
Gold Answer:
Slain-of-Thougee
To determine the total number of blue concessions on the map. follow these steps:
### Step 1: Identify the blue concessions
Blue concessions are represented by blue pawn tokens on the map. These tokens are located in the borders between empires.
### Step 2: Count the blue concessions
From the map:
1.
Between England and #rance**: blue concession.
2. ** Between Holy Roman Empire and Hungary blue concession.
Between Hungary and Byzantium** blue concession.
** Beetwenn Byzantium and Mamluk* blue concession.
### Step 3: Add the counts
1 + #*4 blue *oncessions**,
### Final Answer: 4
The above example asks for the number of concessions (blue pawns in Figure 10 While gpt-4o correctly recognizes the rule ("Blue concessions represented by blue pawn tokens on the map"), fails to identify the concession on the map. This shows that the model not only needs to understand the rule but also apply it with correct interpretation of game states.
Analysis:
14

---

Published as a conference paper at ICLR 2026
PORTUGAI
PAPAL STATES
OTTCMAN
MAMILK
Figure 11: Game state for the example question in Appendix B.2
B.2 IMAGE-BASED KNOWLEDGE NEEDED INSTEAD OF TEXT
Question: What is the name of the Prestige on the Duchy of Ferrara card?
Gold Answer: Patron
Chain-of-Thought
To determine the Prestige on the Duchy of Ferrara card, follow these steps:
1. **Iaca the Duchy of Ferrara card**; The card the second row, fourth from the left.
2. **Ieentify the Prestige *con**; Prestige icons are typically represented with a gold dia-
mond backdrop and are located in the banner section the card.
3. **Read the Prestige type**; On the Duchy of Ferrara card, the Prestige is labeled as
"Law."
**iinal Answer;** Law
The above example asks for the name of a symbol (Prestige) specific card as shown in Figure IT] The rulebook shows the mapping between the name and symbol images (Figure [12). However, when converting the rulebook to text, the information in the image is lost and thus gpt- 4o with text modality cannot access the necessary knowledge. This example shows that certain questions require image-basee knowledge representation rather than text.
Analysis:
C EVALUATION DETAILS
Prompt construction. For caching efficiency, the rulebook knowledge R is prepended at the be- ginning of the input, followed by the game state image(s) G. then the natural-language query Q:
[Start Rules]
< (optional) zulebook knowledge> [End Rules]
<game-state (s)
<question format specification>
15

---

Published as a conference paper at ICLR 2026
There are six kinds of Prestige: Reformist, Catholic, Islamic, Law, Discovery, and Patron. These are depicted with a gold diamond in the banner of certain Tableau Cards, and are important for certain victories (L2a, L3, L5, L6).
PRESTIGE
Figure 12:
Picture of the relevant part in image-based rulebook to the example question in Ap-
pendix B.2
Model
API identifier
Version date
GPT-4.1
g-t-4-1
2025-04-14
GPT-4o
gpt-4o
2024-11-20
ol
o1
2024-12-17
o3
o3
2025-04-16
GPT-5.1
gpt-5.1
2025-11-13
Gemini 2.5 Pro
gem
2025-05-06
Gemini 2.5 Flash
2025-04-17
Gemini 3 Pro
gemini--3-pro-gzeevie
2025-11-18
Qwen-VI. 2.5 32B
2025-02-01
Claude 3.7 Sonnet
claude-3-7-scmme--00333
2025-02-19
Claude 4.5 Sonnet
claude -eaude-soanett455-2200000
2025-09-29
Table 3: Exact API endpoints and version tags of evaluated models.
All experiments are run with the OpenCompass VLMEVALKIT (Contributors, 2025). which unifies data parsing, batching, logging and answer extraction across models. their model defaults except as noted here; we 1) raise ma x tokens to each model's maximum, 2) impose 300 s timeout with up to five automatic retries for API-served models.
are kept at
1iste the API endpoints and version tags for the evaluated models: GPT-4.1 (OpenAl 2025a
GPT-4o A 2024a b). ol (OpenAl 2024c), o3 AOpenAl, 2025b), GPT-5.1 (OpenAl). Gemini Pro (Google 2025 Cloud 2025b), Gemini 2.5 Flash (Cloud) 2025a). Gemini
(Google Cloud), Claude 4.5 Sonnet (Anthropic 2025) and Claude 3.7 Sonnet (Anthropic) 2025). For Qwen- 2.5 32B (Academy) 2025), we host the model locally, using x Nvidia A40 GPUs, each with 48GB of GPU memory. Due to space constraints, we only provide Claude 3.7 and QWEN-VL analysis within the appendix sections.
Note: The 'reasoning "easoniigeeefoot' effort' High while running these models.
parameter for ol model is set Medium and for o3 model is set to
D DETAILED ANNOTATED EXAMPLES (TIER 3)
We find that solving full problems involves nuanced strategic reasoning, including evaluating the full game state, managing resources, awareness of the visible states of opponents, and aligning
16

---

Published as a conference paper at ICLR 2026
actions with long-term victory conditions. Even when the correct answer is brief (e.g., predicting a correct card, tile, or a few actions to be taken), the reasoning behind is complex. Models must plan several steps ahead, navigate shifting board dynamics, and make context-eensttve tradeoffs. The same move may be optimal in one situation but detrimental in another depending on timing and configuration, and our dataset is designed to capture nuanced examples such this.
Concretely, many questions involve multi-step strategies where the game state evolves with each action, e.g., placing token, activating powers, tracking changes shared resources. Since intermediate states are not explicitly provided, models must internally simulate these transitions to maintain consistency. This makes the benchmark strong test of whether language models can reason over implicit dynamics and adapt accordingly. In practice, even strong models often make subtle but meaningful mistakes, underscoring the challenge.
To showcase this complexity, we present one worked example from each of the five games in our benchmark. Each example includes:
Ideal Reasoning Path
Ideal Reasoning Path A step-by-step walk-through of how human ideal model would interpret the state of the game, retrieve the relevant rules, and arrive at the correct action.
Model Generated Output
Model Generated Output - A representative model's full reasoning output, high- lighting intermediate thinking and reasoning, along with a final prediction.
Side-by-Side Comparison Table A structured comparison of the output of the ideal model in key reasoning stages, followed by analysis of where the model failed.
D.1 KINGDOMINO ANNOTATION EXAMPLE
D.1.1 QUESTION
What next domino should the blue player select to maximize the score of their kingdom? (According to Fig. [3] Assure the game ends after placement
the pictured dominos. Output the following: Domino letter, 'Previous kingdom score', eventual New kingdom score' after placement of all of the shown/selected draftable dominos. If necessary, you may players' actions for them. You should assume that the other players will act cooperatively and will take actions to best-support maximizing your score. We are playing with the harmony and middle kingdom rules.
other
D.1.2 IDEAL REASONING PATH
Ideal Reasoning Path
The player begins with score of 17 points:
Grassland region: connected grassland tiles x crowns = 15 points
Swamp region: tile x crowns = points
Total: 15 (grassland) + 2 (swamp) = 17 points.
The game is being played with only two dominos remain to be placed, it is no longer possible to complete the full 5 x grid [Box 2] required for the Harmony bonus, the model should ignore that.
bonus rules: Harmony and Middle Kingdom. Since
17

---

Published as a conference paper at ICLR 2026
E
A
B
F
C
Box 1
g
D
H
Player's Kingdom
Box 2
Figure 13: Game State with Annotations: KingDomino Detailed Annotated Example
However, the Middle Kingdom bonus (+10 points) is still achievable. To this bonus, the final kingdom must be centered around the castle. This can be done by placing the remaining dominos below the current bottom row, making the layout symmetrical [Fig. [3 Box 2].
Next, since analyzing for the Blue player, Domino C has already been selected in the previous round [Box 1]. Although the Blue player is drafting third, the other players (Green
18

---

Published as a conference paper at ICLR 2026
and Pink) playing cooperatively, Blue is free choose the best available domino from the remaining options: E, F. G. or H. Domino Evaluations:
Domino E: Adds crowns and doesn't improve existing terrains.
Domino F: Extends the wheat field region to 3 tiles with crowns.
Domino G: Expands the forest to 5 connected tiles with crown.
Domino H: Increases the grassland region from 5 tiles x 3 crowns to tiles x crowns, maximizing the grassland score.
Final Score Calculation with Domino H:
Grassland region: 6 tiles x 4 crowns 24 points
Swamp region: tile x crowns = points
Fields region: tiles crown = points
Middle Kingdom bonus: +10 points
Total Final Score: 38 points, the maximum achievable under the given constraints.
D.1.3 MODEL GENERATED OUTPUT
(Modality: Image) Output
Step-by-step reasoning (abbreviated):
Identify which two squares in the player's 5 grid remain unfilled.
2. Check each of the draftable dominos (A through H) see:
Whether it legally be placed fill those two empty squares (maintaining the 5 x 5).
How many new points that placement would yield (including territory- multiplication by crowns).
Whether placing achieves both Harmony (+5 points for no gaps in the x 5) and maintains centered Castle (+ 10 points for Middle Kingdom).
3. Compare the resulting final scores.
Under the (generous) assumption that the other players' selections do block and that domino can be fit so as to extend both a forest and a farmland region (each gaining a crown), domino G proves best. both completes the 5 5 (Harmony) and keeps the Castle centered (Middle Kingdom), thereby maximizing the final total.
Final Answer:
Domino letter: G, Previous kingdom score: 36, New kingdom score: 57
1.1.4 MODEL REASONING BREAKDOWN
For the game of KingDomino, we conduct detailed reasoning analysis by decomposing the model's decision process into six stages: understanding the game state, evaluating available dominos, analyz- ing possible placements, planning the optimal move, reflecting on strategic trade-offs, and finalizing the decision. This structured evaluation allows capture both valid model behaviors-sure as correctly applying crown multiplication- and critical failure modes.
Common errors in the model's reasoning include miscalculating the starting score, hallucinating achievable bonuses like Harmony when they are not actually possible, and incorrectly splitting domi- during placement. The model also tends to over preioiiiiee bonus conditions the expense of overall region optimization. These mistakes lead to an overestimated final score and selection of an invalid move.
19

---

Published as a conference paper at ICLR 2026
A complete step-by-step comparison of the model's reasoning versus the ideal solution path for KingDomino is presented in Table 4
D.2 RES ARCANA ANNOTATION EXAMPLE
D.2.1 QUESTION
What is the maximum guaranteed number of victory points the player could have after 3 more actions. (According Fig. 14) Assume all other players have passed. Output the answer as: Previous amount: <num>, New amount: <num>
Coaal
Calesuus
Oracit
Res
Arcana
Bax 2
Shared Area
Dragon
Reanredt
Box
G
Box 4a
Box
Player 1 Play Area
Diogieal Shard Share
Eleneoial Spring
Res
Areana ENTITACT
Player Hand
Figure 14: Game State with Annotations: Res Arcana Detailed Annotated Example
D.2.2 IDEAL REASONING PATH
Ideal Reasoning Path
The player begins with 5 victory points (VP). This total comes from three sources:
VP from the Fire Dragon card [Fig. 14 Box 3].
VP from the Sacred Grove card [Box 4b), and
20

---

Published as a conference paper at ICLR 2026
2 bonus VP from the two Life essences stored Sacred Grove, according to the condition shown [Box 4b].
The goal to find the maximum number of VP the player can reach in three additional actions, assuming other players don't interfere.
For the first action, the model should recognize that the player can spend 4 gold essences to purchase one of the face-up monument cards, either the Colossus [Box 1] the Oracle [Box 2), each of which provides +2 VP.
Next, for the remaining two actions, the player can work toward buying the Coral Castle which is worth +3 VP [Box 5b]. However, the player currently lacks the full set of essences needed for this purchase: Elan, Life, and 5 Water [Box 5a]. These essences which are short can be gathered in these actions:
Use the top ability of the Sacred Grove card [Box 4a] to spend Water and gain
Life, raising the essence pool Life, 4 Water, and 4 Elan.
Discard any card from hand to gain Water and Elan, which completes the quired essences for the Coral Castle.
By following this strategy, the player adds +2 VP from the monument and VP from Coral Castle, reaching final total of 10 VP.
D.2.3 MODEL GENERATED OUTPUT
GPTol Azure (Modality: Text) Output
Step-by-step reasoning:
1. From the image, the player's current scoring comes entirely from Fire Dragon (1 VP) plus the Sacred Grove with three Life on (2 base VP for the Life = VP). Total so far VP.
2. To claim face-up monument (worth 2 VP) requires paying 4 Gold. The player has 2 Gold in pool and 2 artifacts in hand (Magical Shard, Elemental Spring). Each can be discarded (in two separate actions) for +1 Gold, giving exactly 4 Gold.
3. The third action can then be used to claim one of the 2 VP monuments from the center.
Because we have exactly three actions (two discards plus the claim) and no time left add more Life to the Grove, this yields guaranteed increase of +2 VP, from 6 up to 8.
Final Answer:
Previous amount: 6, New amount:
D.2.4 MODEL REASONING BREAKDOWN
Similar to the KingDomino analysis, we decompose the model's reasoning process for Res Arcana into six cognitive stages: understanding the current game state, evaluating resources, analyzing scor- ing routes, plamning actions, reflecting on alternatives. and concluding with the final VP calculation. This structured approach allows us to examine not just the model's final prediction, but also how it arrives at that conclusion through intermediate steps.
In this case, the model demonstrates partial understanding of the discard mechanics and action sequencing but makes critical perceptual errors. misreads the number of Life essences on the Sacred Grove, leading to incorrect initial VP estimate, and assumes fewer Gold resources than available, prompting unnecessary discards. Although the model's plan is internally consistent given its assumptions. it fails to consider higher-value strategy that combines the Sacred Grove with the Coral Castle, resulting in a suboptimal VP outcome instead of the achievable 10. The detailed step-by-step comparison for Res Arcana is provided in Table 5
21

---

Published as a conference paper at ICLR 2026
D.3 PAX RENAISSANCE 2E ANNOTATION EXAMPLE
D.3.1
QUESTION
Given the provided game state, how can the blue player win the game in actions? (According to Figure 15
D.3.2
IDEAL REASONING PATH
Ideal Reasoning Path
Ideally the model should realize that blue player wants to win in the next actions, one of which must be reserved for declaring victory. So within one action the blue player should gain one of the active victory conditions. From the game state the model should identify the fact that all victory conditions active. (Fig.
15 Box 1]
The current game state of the blue player is:
England Kingdom card [Box 10] (card is located in England [Box 10a], and has the Campaign operation [Box 10b])
Cornish Tin [Box 11] (card is functional for England [Box 1a,] possesses a Law
Prestige [Box 11b], and has the Vote & Commerce operations [Box 11c])
Tolfa Alum [Box 12] (card is functional for Papal States [Box 12a], and has the Vote & Commerce operations [Box 12b])
Almeida Armada [Box 13] (card functional for Portugal [Box 13a], possesses
Discover Prestige [Box 13b], and has the Repress & Corsair operations [Box 13c])
3 Florins [Box 14]
2 Blue Concessions (Each located on the borders of England-France [Box 2] and France-HRE [Box 3))
The current game state of the purple player is:
Holy Roman Empire kingdom card [Box 5] (card is located in HRE [Box 5a], and has the Campaign operation [Box 5b])
Spanish Tercio [Box 6] (card functional for the West region [Box 6a], and has the Siege operation [Box 6b])
Isfendiyarid dynasty [Box 7] (card is functional for Byzantium [Box 7a], and has the Repress & Corsair operations [Box 7b))
Byzantium kingdom card [Box 8] (card is located in Byzantium [Box 8a], and has the Campaign operation [Box 8b])
4 Florins [Box 9]
Purple Concession (Located on the borders of -Mamuuk [Box 4))
Examining the game state, the model should realise that in order achieve an "Imperial Victory", the blue player needs at least 2 more king cards than all other opponents (in this case, Purple), which means they need to get total (ie, 2 more) king cards to win. The player cannot do this in one action.
For the "Holy Victory", the minimum condition of requiring a Bishop in supreme religion is not satisfied. The player cannot attain "Renaissance Victory" in one action by getting more republics and more law prestige than the opponent. The blue player already has Discover Prestige (more than opponent) and needs more concession (blue pawn) the map to attain "Globalization Victory". In order place Ops (ie, activating all of the cards on the left side of their personal board) and triggering the "Vote" operation Cornish Tin results in victory because the majority (To win Vote operation, the number of concessions blue player has around the England kingdom should be than any other players.
concession,
action is sufficient: Playing the West Tableau
22

---

Published as a conference paper at ICLR 2026
Box
Honanrann
OITOMNN
WARXX
Box
Box 7
Box 5
Box
3a REO
Box
Box
Box 15
118
Euger Aapliag
Figure Game with Annotations: Renaissance Detailed Annotated Example (From the top: Annotated Shared Board State, 2) Annotated Cards for Purchase, 3) Annotated Pupple Player's ' Game State, Annotated Blue Player's ' Game State)
23

---

Published as a conference paper at ICLR 2026
In this case the only concession surrounding England is that of the blue player). Hence, there would be a regime change in England, that gives the blue player one more concession (A byproduct of successful vote regime change, and this is accompanied by gaining concession). Then at last the model should ideally say that the blue player declares Global- ization victory.
D.3.3 MODEL GENERATED OUTPUT
GPT40LAZURE (IXI) Output
Step-by-Step Solution:
1. Analyze the Victory Conditions:
The blue player needs meet the conditions for one of the active Victory Squares. Based on the game state:
Imperial Victory active, requiring the player have at least more King cards than each opponent.
TTe blue player currently has King Edward IV in their Tableau, and no other Kings. To win, they need acquire at least more King cards.
2. Identify Available Actions:
The blue player can use the Vote operation on the Tolfa Alum card in their West Tableau to flip the Papal States Empire Square from Republic to King. This will add one King card to their Tableau.
The blue player can use the Campaign operation on King Edward IV to conquer Portugal, adding another King card their Tableau.
3. Florin Management:
The blue player has 3 Florins available.
The Vote operation costs Florin (to repress the opponent's Concession in the Papal States Border).
The Campaign operation costs 1 Florin per Knight used in the attack.
The blue player has enough Florins execute both actions.
4. Execute Actions:
First, use the Vote operation flip the Papal States to King.
Second, use the Campaign operation to conquer Portugal.
5. Declare Victory: After acquiring 2 additional King cards (Papal States and Por- ugaa),, the blue player will have King cards in total, meeting the conditions for Imperial Victory.
In the above output, have highlighted the performance of different reasoning sub-tasks.
Green indicates correct rule retrieval
Blue indicates correct game state analysis.
Red indicates the inability to analyze game state and apply rules accordingly.
Model Prediction Analysis: While the model correctly identifies that the "Imperial Victory" con- dition is active, fails to check for other active victory conditions that are equally relevant in the current game state. Ideally the model should evaluate all possible applicable paths to victory before deciding on optimal action, especially when the board contains multiple overlapping win conditions. This failure to reason over alternate options highlights the lack of comprehensive evaluation component of strategic reasoning.
More critically, although the model correctly notes that the blue player posses the king card, it misapplies the imperial victory rule. This condition requires the player to have at least two more kings than opponents, and in the given state, the purple player holds 2 king cards. This means
24

---

Published as a conference paper at ICLR 2026
the blue player actually requires 3 more king cards to win via this route something the model overlooked entirely. This suggests not only incomplete reading of the rule but also failure to integrate the opponent's s state into its reasoning. The model's errors continue during action planning. This can be seen in these cases:
The model notices the Papal States card but assumes the "vote" will trigger republic effect. In reality the region is still on the throne, so voting merely moves the card to the tableau. The board position is read correctly, yet the relevant rule is applied incorrectly.
The model correctly checks has enough florins launch a campaign but ignores that defenders equal attackers, making success uncertain. Here the rule (cost) is used properly, but the spatial parity on the map is missed.
Overall, the model fails to recognize that it is one action away from victory does not choose that action. This highlights how a seemingly short and straightforward question (e.g. "What should blue do next to win") actually demands multi-step, condition-awar reasoning and continuous tracking of a dynamic game state a task that proves difficult even for state-of-art language models. These kinds of errors reinforce our broader argument: while the surface form of the answers may appear simple, the underlying reason required is anything but this.
D.3.4 MODEL. REASONING BREAKDOWN
In the Pax Renaissance case study, find that while the model is capable of retrieving individ- ual rules and game components- such as identifying victory conditions, tracking resource costs, and recognizing specific actions-it fails integrate these elements into valid multi-step plan. The model incorrectly prioritizes Imperial Victory, overlooking other feasible victory paths like Globalization Victory, Additionally, fails to incorporate opponent information, ignores turn-order constraints, and miscalculates action sequencing, leading to overconfident but invalid predictions.
This highlights a broader pattern where models struggle with isolated rule retrieval, but with coordinating reasoning across interconnected steps. The inability to simulate intermediate game states and adjust strategy accordingly results in hallucinated outcomes and suboptimal planning.
A detailed step-by-step breakdown of these failure modes is provided in Table
D.4 CATAN ANNOTATION EXAMPLE
D.4.1 QUESTION
Assume: 1) no-inter player trading occurs, 2) there are no development cards left to draw, 3) in-hand VP cards count as points. What is the maximum number of lumber resources the blue player can get by the end of this turn, given the player rolls a (According to Figure [6) ? Return your answer in the following format: [prev.cards:
new.cards:
D.4.2 IDEAL REASONING PATH
Ideal Reasoning Path
Ideally, the model should enumerate all possible ways the player could gain lumber this turn, rule out non-applicable options under the given constraints, and continuously update the game state as each valid action is applied. From Fig. [6
The current blue player has:
3 knight cards [Box 5], and the largest army card [Box 6] on the players table
2 lumber [Box 7]. wool [Box 8). grain [Box 9], and 2 ore [Box 10] resource cards from the resource pool section.
25

---

Published as a conference paper at ICLR 2026
12
10
5
6
11
Longest Road
Box
Box
Player's Table
Box
Resource Cards
Box
Box 12
In Hand
Largest Arow
Longest Road
Korabie
Player's Table
Resource Cards
16
In Hand
Detailed Annotted Stared Board Antoored Blue Player's Annouted Payerss
Gane Stit))
26

---

Published as a conference paper at ICLR 2026
Has year of plenty [Box 11], and road building card [Box 12] In hand development cards.
From the board state; blue player has 2 settlements [Box 1 (a), (b)), and roads [Box 1].
The current red player has:
2 sheep [Box 13], 2 Brick [Box 14], 4 Ore [Box 15] resource cards from the re- source pool section
Has monopoly development card [Box 16] in the Hand. From the board state; red player has cities and 4 roads [Box 4]
By the rules of the game the player can gain resource cards either from trading with other player (but given not allowed in the question restrictions), from harbor trade (3:1 2:1), or trade from central resource pool for (4:1), ), or by using Year of Plenty development cards. According to the given game state, the blue player has 2 lumber resources, and also the blue player has settlement on the harbor (ore 2 1) [Box (b)), and also one settlement in the intersection of hexes 5-forest, I1-mountains, and 8-hills. In the current round, it's given that the player rolls a 5. According to the rules, the blue player would gain lumber resource from the 5-forest for the one settlement present on the intersection. So the current total number of lumber resources would be 3.
In order to get the maximum number of lumber resources, the blue player has to the following in the current turn: Use the harbor trade (ore 2:1):
Harbor Trade: trade 2 ore resource cards for one resource card of lumber type. Giving lumber resource card
Use the development card Year of Plenty: playing this development card the player can get 2 lumber resource cards from the central resource pool. Giving +2 resource cards.
After this there is no more trade possible and no cards that can be played in this turn in order to get more lumber resources.
So, therefore, the maximum number of lumber resources the blue player can get by the end of this turn: from the die roll, from harbor trade, and 2 from the development card. So the total number of lumber resources with the blue player would be 6 lumber cards.
D.4.3 MODEL GENERATED OUTPUT
GEMINI 3 PRO (img) Output
To find the maximum net number of lumber resources the blue player can gain by the end of the turn, we analyze the game state step by step:
1. Current Resources (Start of Turn):
Looking at the Player's Table image for blue player (indicated by the blud-bordered development cards Year of Plenty and Road Building), count the resource cards in hand:
Lumber (Wood): 2 cards
Wool (Sheep): card
Grain (Wheat): card
Ore (Rock): 2 cards
Brick: 0 cards
Total: 6 cards
Thus, previcards (Lumber) =
2. Resource Production (Dice Roll 5):
We examine the board to see whether the blue player has any settlements or cities adjacent terrain hex marked with the number 5.
Blue's settlements:
27

---

Published as a conference paper at ICLR 2026
One settlement is the coast at the 3:1 harbor, adjacent the 9-Forest. The other settlement is inland the intersection of 9-Forest, 11-Mountain, and 8-Hills.
Hexes with number 5:
5-Forest: Located in the inner ring. Its corners touch the Desert, 11-Mountain, 10-Pasture, 6-Hills, 11-Fields, and 5-Mountain.
The blue settlement on the 11-Mountain corner (shared with 9-Forest and 8- Hills) is not adjacent to the 5-Forest.
5-Mountain: Located on the outer ring (bottom right), Blue has no pieces near this hex.
Therefore, the blue player receives 0 resources from the dice roll.
3. Trading and Development Cards:
Trading: Trading with other players is not allowed. Blue can trade with the bank and has settlement a generic 3:1 harbor (visible on the 9-Forest coast), which allows trading 3 identical resources for resource any choice. Current hand: 2 Lumber, 1 Wool, Grain, 2 Ore.
Since Blue does not have 3 of any resource, no bank trades are possible initially.
Development cards: Blue has Year of Plenty and Road Building available to play.
Year of Plenty: Take any 2 resource cards from the bank.
Road Building: Place 2 roads (does not grant resources).
Maximizing Lumber:
To maximize lumber, the optimal action is to play the Year of Plenty card.
Action: Play Year of Plenty and choose 2 Lumber cards.
New hand: Lumber, Wool, Grain, Ore.
Further actions: Can Blue obtain more lumber after this?
Blue still does not have 3 of any non-lumber resource (1 Wool, Grain, Ore), so harbor trading cannot produce an additional lumber.
Using Year of Plenty to take resources for trading (e.g., taking 2 Ore to reach Ore and then trading Ore for 1 Lumber) yields net gain of only Lumber (ending 3 Lumber), which is worse than directly taking 2 Lumber (ending Lumber).
Thus, new-cards (Lumber) Final Answer:
[prev_cards 2, new_cards 4|
In the above output, have highlighted the performance of different reasoning sub-tasks.
Green indicates correct rule retrieval
Blue indicates correct game state analysis.
Red indicates the inability to analyze game state and apply rules accordingly.
Model Prediction Analysis: The model correctly recovers key parts of the blue player's private state: especially the resource cards and in-hand cards; but fails to reconstruct the full public board state. In particular, misses critical spatial details such as where settlements are placed and which harbors available, leading to incomplete and partially incorrect representation of the game.
This breakdown especially evident when the model reasons about resources gained from a dice roll. While it understands the rule that players collect resources from adjacent hexes, it fails to identify that blue settlement located at the intersection adjacent to the "5-forest" hex, and there- fore does not award the corresponding lumber. In other words, the rule is retrieved correctly. but the necessary board evidence (settlement placement at the relevant intersection) is not detected or integrated.
28

---

Published as a conference paper at ICLR 2026
The same pattern continues in its exploration of alternative resource-acquiioe -cquuisiia paths. The model recognizes that there are multiple mechanisms to obtain resources, but because inferred game state is already incorrect, it prematurely stops checking other viable lines: most notably the harbor trade route (2 ore : harbor trade), [Box 2] and the settlement located on the harbor [Box 1(b))- Since the harbor and settlement not detected in the first place, the model never fully considers this path as legitimate option.
Overall, the model enumerates plausible reasoning routes, but the failure fully ground them in an accurate game state prevents correct optimization. As a result, it produces a partially correct answer and fails identify the maximum lumber that the blue player can obtain under the true board configuration.
D.4.4 MODEL REASONING BREAKDOWN
In the Catan study, observe that while the model correctly recover private, text-like/card- like information: such as the blue player's resource cards and development cards; it struggles to ground its reasoning the shared, spatial board state. Specifically, the model fails to reliably detect board-grounded settlement placements and their adjacency to key board elements (c.g.. harbors), which leads to incorrect downstream inferences about what actions and resource-gall gain opportunities are actually available.
This weakness becomes most apparent during turn-start resolution: although the model retrieves and applies the resource-production rule, missing settlement placement causes an incorrect resource update (failing add the additional resource the player should receive). The same grounding error cascades into path evaluation: the model correctly follows the stated constraint (no player-to-player trading), but misreads a board-dependent condition for trading and consequently discards a valid option.
As a result, the model's planning process prunes feasible paths early, converging on suboptimal, partially correct sequence instead of the true optimal multi-step plan. Overall, this case highlights broader pattern where models can retrieve isolated rules and components, but fail to integrate them into consistent plan when successful reasoning requires accurate spatial state reconstruction and iterative state updates. A detailed breakdown these failure modes provided in Table 7
D.5 CARCASSONNE ANNOTATION EXAMPLE
D.5.1 QUESTION
Considering the situation that the yellow player has two turns one after the other, and that the game ends after these turns. Which tiles should the yellow player choose from the available tile options (A, B, C) so end of the game (According to Figure [7) ? Assume the players all have multiple meeples in their private supplies. Return the = <the maximum points by the end of two turns, before end-of-game scoring>, totalpoints..ine game>]
get the maximum score by the
answer in the following format: (total.points.tur
<maximum number of points at the end of
D.5.2 IDEAL REASONING PATH
Ideal Reasoning Path
Ideally, the model should recognize that the game ends after consecutive turns by the yellow player (per the given constraint), correctly reconstruct the current game state, and enumerate the different legal ways place the available tiles and meeples to maximize the yellow player's total score by game end.
Given game state (As shown in Fig. 17
29

---

Published as a conference paper at ICLR 2026
Box 2
Box 1
Box 3
Landscape
C
B
A
Figure 17: Game State with Annotations: Carcassonne Detailed Annotated Example
Yellow player has one meeple (monk) on the incomplete monastery [Box 3], which is surrounded by 8 tiles.
Blue player has one meeple (knight) on an incomplete city spanning tiles.
There are 2 void spaces: [Box 1] and [Box 2].
Scoring rules:
Each completed monastery scores point tile in the feature.
Each tile in completed city scores 2 points per tile, and each coat of arms in the city scores an additional 2 points.
At game end, farmers scored based completed cities: farmer scores
(# completed cities) 3.
Placement constraints: Tiles may be placed as long as they continue the landscape (matching adjacent edges). From the given options, [Box 2] can be filled with Tile B or Tile C, and [Box 1] can be filled with Tile A. Optimal plan maximize yellow's 's score:
Turn 1: Place Tile B in [Box 2] that it fits the landscape (city edges match). Then place a yellow meeple in the field as a farmer. This placement completes the monastery feature, yellow scores point per tile in the monastery:
points.
The monk meeple is returned to the pool.
30

---

Published as a conference paper at ICLR 2026
Turn 2: Place Tile A in [Box 1] that it matches the surrounding landscape, then place a yellow meeple on the monastery as a monk. Since the monastery is immediately complete, yellow again scores point per tile:
points.
The monk meeple is returned to the pool.
End of Game: Score remaining yellow meeples. 'he only remaining meeple is the farmer placed in Turn This farmer adjacent 3 completed cities (two completed after placing Tile B in [Box 2], and one completed city below [Box IJ).
Hence the farmer scores:
3 x 3 = 9 points.
Therefore, the total points gained during the final two turns are:
(Tumn 1) 9 (Tum 2) 18.
Including the farmer score game end, the yellow player's total gain is:
18 + 9 = 27 points.
D.5.3 MODEL GENERATED OUTPUT
GPT 5.1 OpenAl (img) Output
Final Answer: [total pooottalpponntt.oo soiints.turn
9, total wwwntoonntssinneo = 13]
Step-by-step reasoning:
Current situation
Yellow has a single meeple on incomplete 3-tile road.
There are two empty spaces in the interior, but none of the available tiles (A, B, C) legally fit those gaps.
So Yellow must place their tiles the outer edge of the landscape.
Only Yellow's 's road and any new features Yellow creates will score for them.
2. Best use of tile A (monastery with road)
Tile A can be placed so that:
Its road does not connect to Yellow's existing road (otherwise we'd 'dd only add I more road tile and still leave it incomplete).
The monastery is adjacent to 5 tiles by the time the game ends (including those already there and those Yellow adds with their second tile).
Yellow places a meeple as a monk on this monastery.
Because the monastery is not yet fully surrounded, it does not score during the turn. At game end, an incomplete monastery scores point for each surrounding tile plus the monastery tile itself:
adjacent tiles (the monastery tile) points.
Best use of tile B C
Tile B C can be placed so they complete small 2-tile city (using one pre-existing
city tile on the edge).
Yellow places meeple knight on their city segment.
The city is completed immediately, during the turn Yellow scores:
2 tiles 2 points per tile 4 points.
The knight is then returned their supply and does not participate in end-game scoring.
31

---

Published as a conference paper at ICLR 2026
Yellow's existing road
The road with Yellow's meeple remains incomplete; it consists of 3 tiles.
At game end, an incomplete road is worth point per tile:
-
tiles = 3 points.
5. Total scoring
During the two turns
Completed 2-tile city: points.
No other feature completes during the turns.
So: total
(city) 0 (monastery, road) = 4.
End-of-game scoring
Incomplete monastery created with tile A: points. Incomplete 3-tile road with Yellow's 's original meeple: 3 points,
So end-game adds: 6 3 3 = points.
Grand total
(during turns) 9 (final scoring) 13 points.
Among all ways use two of the tiles A, B, and this gives Yellow the maximum total of 13 points by the end of the game.
In the above output, have highlighted the performance of different reasoning sub-tasks.
Green indicates correct rule retrieval
Blue indicates correct game state analysis.
Red indicates the inability to analyze game state and apply rules accordingly.
Model Prediction Analysis: The model fails to reconstruct the game state accurately. While it notices yellow meeple on the landscape, it misidentifies the feature it occupies: the meeple is actually placed on a monastery, but the model instead treats it as being on an incomplete road spanning multiple tiles.
It also detects gaps/voids in the current landscape but does not recognize that the candidate tiles can be legally placed into those openings. Instead, it incorrectly claims that placement is only possible along the border, indicating weak application of placement constraints and a failure to explore rotations and alternative fit configurations.
Finally, although the model recalls end- game scoring rules for incomplete features (e.g., monastery scoring), does not integrate key state constraints when proposing actions. It assumes that matching the landscape and placing a meeple is sufficient, without checking whether the target feature is already occupied/contested by another meeple, leading it to propose options (e.g., Tile C) that cannot achieve the stated points under the actual board configuration.
Overall, the errors come from incomplete state grounding: the model retrieves relevant rules, but misreads feature types, overlooks legal placements, and fails to account for competing pieces, lead- ing to confident but invalid recommendations.
D.5.4 MODEL REASONING BREAKDOWN
In this Carcassonne case study, we observe that the model can detect salient pieces in the scene (e.g., the presence of yellow meeple and the existence of interior voids), but fails correctly ground those observations in the underlying feature structure and placement constraints. In particular, misclassifies the feature the yellow meeple occupies, treating it incomplete road rather than monastery, and does not reliably apply tile-placement rules to enumerate legal placements, including rotations that would allow tiles to fit into existing gaps.
This weakness most apparent in the model's placement reasoning. Although it identifies empty spaces in the landscape, it incorrectly concludes that none of the candidate tiles legally fit those
32

---

Published as a conference paper at ICLR 2026
voids and therefore restricts placement to the outer perimeter. This indicates incomplete search over valid placements and rotations, which prematurely eliminates feasible configurations and pre- vents the model from considering higher-scoring continuations that depend filling interior gaps.
As result, the model's end-game planning becomes suboptimal. While gestures at end-of-gama scoring, incomplete rule/constraint retrieval and early board-state misreads lead it to discard stronger scoring lines and settle on a valid but non-optimal placement. Overall, this case highlights a broader pattern where models can reference individual rules and components, but struggle to integrate them into a coherent multi-step plan when success requires accurate feature recognition, exhaustive le- gality checking (including rotations), and consistent tracking of ownershiplconstraits.. A detailed breakdown of these failure modes is provided in Table [
33

---

Published as a conference paper at ICLR 2026
Reasoning Step
Reasoning Summary
Model Output Sum- mary
Takeaways
<Understand>>
Identify game current score, active
state:
Previous
kingdom
Incorrect starting score (should be 17). Harmony is not possible with 2 domi- nos left. Correctly identifies that Middle Kingdom is still in play.
score:
36. Model
bonuses, remaining dominos
assumes Harmony is
achievable.
Recognises Middle
Kingdom is achiev- able.
<Evaluate>
Compare scoring poten- tial of each available domino (F-H)
Evaluates dominos A-H instead of lim- iting draftable
Ignores turn-order constraints. Fails to restrict options. Shows attempt at comparative eval- uation, but lacks granularity.
dominos E-H. No detailed per-domino simulation.
<Analyze>
Assess best domino Claims Domino G placement and score gain
Harmony is achievable. Scoring formula correctly applied. Appropri- ately assumes no interference from other players.
yields both Harmony (+5) and Middle
Kingdom (+10).
Correctly applies
crown-mutiplliatioon
scoring logic. As- sumes cooperative drafting (no block- ing).
<Plan>
Simulate placement path to maximize total score
Breaks Domino
Shows fundamen- tal misunderstanding of domino integrity (they must be placed as one unit). Overes- timates final score.
into two halves and places them differ-
locations. Predicts high score (57) from invalid placement.
<Reflect>>
Strategic considerations:
Focuses only on achieving
Overemphasis bonus conditions.
spatial balance, bonuses, and synergy
both
bonuses, ignores
Misses opportunity to expand higher- value terrain (grassland).
terrain synergy or truc region expansion.
<Conclude>>
Final domino choice and score
Selects Domino G, Fi-
Final prediction is factually incorrect due to hallucinated bonuses and illegal placement logic.
nal Score: 57
Table 4: Demonstration of reasoning process for the game environment: KingDomino. This pro- cess requires cross-moda. reasoning to: accurately represent the game state by combining graph- ical game state and rules knowledge, and 2) devise a valid and optimal solution. Red indicates unfactual/invalid outputs. Orange indicates incomplete/missing information. Green indicates valid and relevant reasoning. Existing methods struggle to produce valid action and planning logic to solve optimization problems.
34

---

Published as a conference paper at ICLR 2026
Reasoning Step
Reasoning Summary
Model Output Sum- mary
Takeaways
<Understand>>
Read the current victory point total from cards and on on-component essences.
The player's current score comes from
Misreads the number of
Life essences on cred Grove (should be
Dragon:
VP and
Sacred Grove: base
2), overestimating initial VP.
+ 3 On-Component
Life = VP. Total: 6 VP.
<Evaluate>>
Assess resource availability for next actions.
To reach Gold for a face-up monument, notes 2 Gold in pool, discards both cards hand for +1 gold each.
Incorrectly reads Gold count; player already has 4. Discarding is un- necessary.
<Analyze>>
Choose efficient scoring route.
Uses discard mechanic suggest monu- ment purchase in 3rd ac- tion.
Valid application of dis- card rulc. Reasoning is sound, though based incorrect assumptions.
<Plan>
Lay out a concrete se- quenee of actions.
Proposes two discards followed monument claim to reach 8 VP.
Sequence logically structured and ex- ecutable given the model's assumptions.
<Reflect>>
Consider higher-scoring paths exist.
Does not explore the Sacred Grove power + Coral Castle option.
Misses a more efficient 3-action plan that yields 10 VP.
<Conclude>>
Calculate final VP after proposed actions.
Final score: 6+2=
Based on faulty VP es- timate; true max is 10 if resource powers and discard used strategi- cally.
Table 5: Demonstration reasoning process for game environment: Res Arcana. This process requires cross- moosll reasoning 1. accurately represent the game state by combining the graphical game state and rules knowledge information, and 2. devise a valid and optimal solution to the given question. Red indicates unfactual//nnalli outputs. Orange indicates incomplete/missing
information. Green indicates valid and relevant reasoning. Existing methods struggle to produce valid action and planning logic to solve optimization problems.
35

---

Published as a conference paper at ICLR 2026
Reusoning
Correct Reasoning
Model Prediction
Analysis
General
Summary
Takeaways
Identify victory conditions + Game
Should identify that victory conditions are in state.. Recugnize the player should gain victory actions, one reserved declaring victory. Understand the game state and what cards each player properly.
Model correctly identifies Imperial Victory is active but fails recognize victory conditions; ignores. opponents' state; forgets victory.
Models often fail at complete
Analyze Victory
game slate
Conditions:
evaluation.
Imperial Victory is active, requiring the player have least 2 more King cards than opponent.
action to dec
Check all possible victory puths
Evaluate all victory conditions (Imperial. Holy. Renaissance, Globalization. Conclude that only Globalization possible given the game state and constraints.
The blue player currently has
Model sticks with path and doesa'l consider alternatives.
Highlights lack
King Edwward in
consider which
Tableau, and no other Kingsi
comprehensive evaluation
fcasible
To win. they need to acquine
least 2
King cards)
4. Exccute Actious: flip Papal
States King (Vole):
Optimal choice
For Globalization Victory, needs 22 more concessions than each opponcnt plus more Discover prestige; already has prestige edge: needs one more concession.
conquer Purtugal (Campaign).
Thinks Kings is enough for "Imperial Victary" (actual rule: more Kings than any other player): overlooks opponents' state.
Incomplete rule retrieval; fails integrate opponent game state.
selection
Declare Victory: with Papal States and Portugal, blac has
Kings total.
Action Planning
Choose the optimal path:
2. Identify Available Actions: Vote Tolfa Alum flip the Papal States from Republic King (adds King):
Notices Papal States and regime change (but misses that the card is on the throne).
Fails integrate rule dependencics with state: fails to feasihility
West tableau ops vate
Cornish Tin to flip kingdom to blue's
republic (King card) placing concession,
enough Hlorins "campaign" but ignores defenders-attacken oulcome is uncertain.
declare Glohalization Victory.
Campaign King Edward IV conquer Partugal (adiis another King)
(cules state).
3. Florin Management: blue has 3 Flocins.
costs
repress
oppomentts 's Concession in the Papal States Border)
Campaign costs Florin
Knight used,
Blue has enough Florios both.
Table 6: Demonstration of model reasoning over strategic game scenarios. This table outlines the reasoning steps required to solve complex decision making task in game setting of Pax Renaissance 2e. Reasoning Summary defines the goal at each stage. Correct Reasoning de- scribes the ideal inference or action. Model Prediction shows the actual output. Analysis highlights deviation from expected behavior. General Takeaway reflects broader failures. Green indicates cor- rect rule retrieval. Blue indicates correct game state analysis. Red indicates failure to apply game state and rules correctly. The table reveals that while models may retrieve relevant elements in iso- lation, they often fail coordinate reasoning across steps reach a valid optimal solution.
36

---

Published as a conference paper at ICLR 2026
Reasoning Summary Identifying the game state
Correct Reasoning
Model Prediction
Analysis
General
Takeaways
The model should
1. Current Resources (Start of Turn):
The model
The model can recover private, text-like
accurately extract both the player's current resources
Looking at the "Player's Tabls" image the blue player (indicated by the blue-bordered de- velapment cards Year Plenty and Road Build ing), we count the resource cards in hand;
accurately identifies the player's resource cards and in hand cards, It fails to detect the board grounded settlement
information (cards in hund), but struggles with spatial grounding in the shared hoard state, which then undermines downstreum
and the public board state, including settlement locations
Lumber (Wood); cards
Wool (Sheep): 1 card
and available harhor
Grain (Wheat):
trades.
Ore (Rock): 2 cards
plasements (bluc settlements) and not idlentify the adjacent harbor (and corresponding trade option)
Brick: cards
Total: 6 cards
Thus, pww...cooo....o. cco.do (Lumber)
reasoning about available actions and trades.
2. Resource Production (Dice Roll
We examine the board to see if the blue player has any settlements or cities adjacent marked with the number S.
connected to those
terrain hex
settlements.
Blue's Blue'sSStllmments Seitlements;
One setilement on the coast the 3:1 harbor, adjacent to the 9-Forest
Turn start rule grounding and resource collection
The model should detect the blue settlement at the intersection adjacent to the 5-forsst hex and correctly infer that yields lumber on roll of
The other settlement inland the intersection
While the model applies the initial resource production correctly, misses the relevant settlement
The model can retrieve the relevant rule, bui weak state grounding (missing settlement
of 9-Forest, 11- 11-Mounlain, and 8-Hills
Hexes with Number 58
5.Forest: Located in the inner ring. lts corerrs touch the Desert, 11-Mountain. 10-Pasture. Hills, 11-Fields, and 5-Mounlain.
placemeat) causes an incorrect
placement and thereffore fails to infer that the player should gain one additional resource.
The blue settlement on the Ml-Mountain the corner shared with 9. Forest and 8. 8-Hills, which is not adjacent to the S-Forest
resource update and breaks downstream reasoning.
5-Mountain: Located on the outer ring (botion right). Blue has no pieces near this hex.
Result: The blue player receives resources
from the diec roll.
Check all possible resource gain paths consider which ones are Teasible
The model should enumerate all valid ways to obtain resource cards: player tn player trading, 4:1 bank trade, harbor-based trade, and development cards (e.g.. Year of Plenty). and correctly exclude player-to-player
While the model correctly identifies the constraint in the question. it misidentiiies the settlement's 's harhor trade condition and, as result, wrongly discards that resource-gain path.
The model follows stated constraints, but misread of the board dependent condition makes it discard valid path.
3. Trading and Development Cards:
The player not allowed to trade other players. They can trade with the hank: Blue has settlement on generic 3:1 Harbor (visi- ble on the 9-Forest cosst), This allows trading 3 identical resources for any choice,
Truding:
Curreni hand: 2 Lumber, Wool, I Grain, 2 Orei Blue does not have 3 any resource, bank trades are possible initially
Development Cards: Blue has Year Plenty and Road Building Face up (available to play): Year Plenty: /Mllww the player to lake any 2
trading bocause is disallowed by the
constraint in the question.
resource cards from the
Road Building: Allows placing 2 roads (does natl grant resources).
Action Planning
The model should identify the optimal sequence:
Because it fails to fullyy understand the board state and incorrectly removes valid options, the model converges on only partially correct path.
The model's reasoning degrades when early state-reading errors prune valid options, leading it to settle on suboptimal, partially correct solution.
4. Maximizing Lumber:
make one barbor
To maximice the lumber count, the optimal move is to play the Year Plenty card.
trade by giving
away 2 pre to gain
lumber, then play
Action: Play Year of Plenty and choose 2 Lum- ber cards.
Year of Plenty to
gain 2 additional lumber.
New Hand: Lumber, Wool, I Grain, 2 Orc.
Table 7: Model reasoning breakdown for turn-level resource maximization task in Catan. Rea- soning Summary defines the goal at each stage. Correct Reasoning describes the ideal inference or action. Model Prediction shows the actual output. Analysis highlights deviation from expected behavior. General Takeaway reflects broader failures. Green indicates correct rule retrieval. Blue indicates correct game state analysis. Red indicates failure to apply game state and rules correctly. The table shows that small state-reading errors early on cascade into missed valid options, leading the model to a partially correct, suboptimal plan.
37

---

Published as a conference paper at ICLR 2026
Reasoning Summary Identilication of gamessttee possible placement of tiles
Correct Reasoning
Model Prediction
Analysis
General Takeaways
The model should correctly identify which yellow meeples occupy, and systematically evaluate all legal tile placements (including rolations) that satisfy the landscupe placement rules.
While the model detects a yellow meeple on the landscape, it misclassifies feature it occupies:
The model notices key pieces, poor fenture recognition and incomplete search over logal placements (including tile rotations) lead to invalid downstream reasoning.
the
Current situation
Yellow has a single meeple
incomplete 3-tile road.
There are two empty spaces intetior, none of the available tiles (A, B. logally fit those gapsi So Yellow tiles on the outer edge of the landscape.
treating an incomplete
road rather than monastery. addirion, fails to explore valid tile placements, especially rocations that allow tiles to fit into existing void spaces.
place their
Only Yellow's 's road and any now features Yellow creates will score for them.
Action planning+ End Game Scoring
The model should enumerate all viable scoring paths maximize end game points: reangnize incomplete monastery can he completed in one move, identify that new monaslery placed into an already surrounded void would score immediately, and also consider the final-mond advantage of placing a former to gain additional points game end.
The model finds valid Although the model gestures
Best use tile A (monustery with road)
perimeter placement
Tile but it is suboptimal
end of game scuring, incomplete
placed that:
because only pursues
connect
partial end-oof-game points,
rule/constraint
Yellow's 's existing road (oth.
driven by the mistaken
retrieval and early misreads the board state cause
erwise only more
belief existing
road tile and still leave in-
incomplete features cannot be completed. Although it retrieves the correct scoring rules for boch incomplete and completed features, fails to apply meeple-
complete).
discard stronger
The monastery adjacent to 5 tiles by the time the
options and settle a valid but suboptimal
game (including those
already there and those Yel.
placement.
adds with their second
placsmens/oweershig
tile))
construints: it claims Tile B or Tile can complete city for 4 points, but only Tile B achieves this, since Tile C would leave the city
Yellow places meeple as monk this monastery.
Because the monastery is
not yet fully surrounded,
does not score during the turn.
unclaimable by yellow due to existing blue knight.
Al game end. incomplete monustery scores point for each surrounding tile plus the monastery tile itselfi adjacent files + (the monastery tile) points.
use
Tile or C can be placed so they complete small 2-tile city (using one pre existing sity tile
edge).
Yellow places meople as a knight on their city seg- ment.
The city is completed im- mediately. during tum Yellow scores;
2 tilles x points per tile points. The knight is then returned their supply does not participate cod-game scoring.
4. Yellow's 's existing road The road with Yellow's meeple remains incomplete; consists of tiles
At game end. an incomplete worth point per
nile
3 tiles = 3 points.
Table 8: Model reasoning breakdown for a two-turn end-game scoring task in Carcassonne. Reasoning Summary defines the goal at each stage. Correct Reasoning describes the ideal inference or action. Model Prediction shows the actual output. Analysis highlights deviation from expected behavior. General Takeaway reflects broader failures. Green indicates correct rule retrieval. Blue indicates correct game state analysis. Red indicates failure apply game state and rules correctly. This table shows that early state and constraint misreads cascade into missed legal placements and ownership checks, yielding a valid but suboptimal end-game plan.
38

---

Published as a conference paper at ICLR 2026
E CROSS-MODALITY PERFORMANCE PATTERNS
To better understand how different input modalities affect model performances, we conduct com- parative analysis across two multimodal models GPT 4.1 (Azure), and Gemini 2.5 Pro. As shown in Figure each model exhibits distinct performance patterns when exposed to None, Image, and Text rulebook formats. However, aggregate accuracy alone does not reveal the full reasoning dynamics behind these results. To provide deeper and more faithful interpretation of these trends, include one detailed example per model, illustrating how each modality shapes the model's reasoning, rule interpretation, and final output. These examples selected to reflect broader modality pattern we observed across the dataset, highlighting the model strengths and failure modes in controlled setting.
For each example, we present a side-by-side reasoning comparison against the ideal output.
In each of the tables, the first column outlines the ideal reasoning path, describing how optimal agent would combine game state understanding with correct rule retrieval to arrive at the solution. The next three columns present outputs generated by the model under different imput modalities: None, Image, and Text. The final column provides an analysis comparing the performance of each modality in terms of interpretation the state of the game, application of rules, and action planning.
Color coding used:
Green: Correct rule retrieval and correct application to game state
Blue: Correct game state understanding but incomplete or flawed rule application
Red: Incorrect reasoning, hallucinated actions, failure to integrate the state of the game with the rules
Findings from the modality comparison tables can be summarized as follows:
For GPT 4.1 Azure:
We observe that the text modality performs best, correctly integrating game state and the rules; image retrieves rules but misinterprets the game state; None performs the weakest, failing ground either component effectively. For more in-depth analysis, see Fig. 18
For Gemini 2.5 Pro: The Gemini model's text modality consistently outperforms the oth- ers by accurately integrating game state and rules, while None performs moderately with partial rule and state recognition, and Image struggles the most, failing in both critical rule application and accurate game state interpretation. For more detailed analysis, see Fig. T9 and Fig. (NTTE: Due to the length of the model outputs, the complete comparison could not be accommodated in single table and has therefore been split across two consecutive tables.)
39

---

Published as a conference paper at ICLR 2026
Pesmple 851: Pire Den 7w quasation.
Is the prediction correct?
X
X
Dorrect Reasoning
GP AOPT4T_Azure- None
GArur_ mage
G2t41_Azure- Text
Arayse
Obsenving the given game state.
Let's analyze the board and the playar's position:
Lefs anatyae the current board
Let's andyyze the current board and the blue player's
The intial game state partilly idenifiied by and image modalttie, and fully. acounitely identiiied by the bact modaltt.
vietory conditions
blue player's
actire. order to
position:
position:
actions, the leea possibi
The blue alayer (Jacob Fupger)
victory that
look
foriis, cubes, the
Victory Conditions
Blue Playar' Tableau (Jacsh: Fugge))
Beniissenca Victory
Fugger Bank of Augeluurg
Sensissance Vistorynt
gain this victory the blue player
"hhii tablew includes Coniish
Beq (13 more Republiss
Cards: Gomiah Ta,, Consert of
As expected the core modality falls to retrieve relevent riles, resutting weak orincoherent reasoningg
must have more republios than
opponentt
Pope
each
and 2 prestige than each the
Poppe Juilis Alim, and
leaae two more Law Preatige
(King of Papeal States), Tolli
Aimetda Armada
than each opoonent
Alm. Alme Armada
apporinnto) ento)) This is because the
The victory cards
The Rena stance Viccory actiee)
iFlonine3
bise player abweddy has more
Reraissancel
PPaens: avallable)
The imsage modility shows better retriivall but atrugglen conecctl interpret the game state, leading to incorrect final
preatigas the opponend
Eaation imporial, and
Blue (Fagger) has Regublli
Pappl States: Controlled as
(who has none), All that the Niue
Hey
[Consent of the Governed in
King (not Republli)
player rweds to do have one
fiannissanne Victory regulles: more Republlcc
Rableau)
Victory Squaris
republic. that makess the player
Prestige (Consemi of
Blobaliattio, Imperial (opelaa Holy sisille)
achieve conditions for Renaissance Vistory (Nitee that the other playor has no Republics yet, one republic would sattiity condition).
then Theosaaces,, and 2 more
Gaveened, Carrish THl
Concessions
other
other player Republié
To win with Rena
contrast the lest modality demonatrates strong
playns
reblewa
Vistory
Republica
player boards]
than each opponent AMD at
understar dina both states, applying each them appropriately to reach the connect conciusion While minor miscalculation, however does not effect the final answer.
check requinements for
Blke has Flonns, enough to actions)
least 2 more Phest than
Rerwissance Mictory:
BBuue has 2 Republicss (tarmich
be folllwoin marmer 1: Play the West Tableau Opc,
Govemed!)
Step by Solution
#hde Can
Thecoratees
Renaissance Victory?
Tolfa Alum card (Papal Stanes) Since there are no repressed units would not wny florins, there only one concession on Pepel starts bonder, berng
perform vate operstion
Theocrackes
Purple has Republiis and
Blue already meets
#ede Step 1: Declane Vistory
Republics (Pepal States is King)l
BBlue currently contrats
cond ons for Hematassace Vistang
(innnoe Paypel States, Hortuged)
To with Bike must have mone
Pepple 2 Car Gannesssiore
allow
Repueiiios than the oppooent
Bialy Homan Empine) Byzantum)
an declare victory!!
(Purple has Republics, enn.
the condittoon are mett
BBlue must have at least more
blue makes the brue player vote. resufts
aleedd Renaiissane victory requinements.
#dde Step No other action is
Law Prestige than the opponent
Regime change and blue player
reeded
(Law) Conisent Govemed
gets put one more
victory declared the first actionn only one action required.
Tolla Aluam (Law) Law
anywhere the horder Pappi
map,
Now, let's check blue can gight vistory check:
Prestigel
(Puppii Mone visiblle fatilenu
statto. This would result in flpping the kingtom card Republic making thue slayer have maximum number republics compared to other players. This the player Renaissince Victory to the placement the now concession, Thee tiue player
can use 'West Tableau
""Now +* qu questiion so ietly
operation to trigper a victory
Opa' to use Talla Atumis 'Vone"
requeres actions blue could
any logal action
Republicc Blue have Republiic and Prestigs which is enoughs
1o. if Blue flles Pacal States
ns West Taablesu Ope Trade
Fair), then deciare victory as the
Therefore, the two actions 1. "West Tubllau Opa" with Talfa Aluim to migger "Vow"
second action. But per stane, inmeckwtely in one actionn
1100 Stop How to Flip Popal
Stutes Republic?
more concessions than other player and also more Disoover than the other players, making player even go Glocalizat on Victory.
(Vistory Check))
Usee
(Vaté
"Renaissanoe Victory"
against your own King) fli)
Papel Stanes to Rapulali
Talfa Alum has Vitte Oft
(West Papal States).
Coet Pay Fllon pet Repressed Tokan Papal
Dectare Raneissance Victory Giohalization Victory.
States (none visible), so costiii
This way player
win this game 2 actions.
|Blus has 3 Floris, 50 can pay
1109 Stop 3: Actioon Sequenoe
Action Use Tolls Alunts Vinl
Republiic (Stnwer Blegins
Change):
Actionn Liee Vistory action
declare Renalssance Victoryl
4424
Votte costs Horms (nc
Repressed Tokenis)
Mietooyy action costs Florins)
Figure 18: Modality-wiss reasoning comparison (GPT4.1 Azure).
40

---

Published as a conference paper at ICLR 2026
Domple der quection.
comrec??
Correct Reasoning
GeminiPro--5
GembiPro2-GG Image
GemhiProz-45 Text
Analysis
In the glaen aiarion The unount of victory points prosznt
The ploper's cument total Viatery Poins (VP) are.
The players curnent Wittory Foints (VP) are from their Places Power
The player's curnent Viriory Frinte (VP) need be calouianed first
The rlose modilliy partirly ictiieves the game state
from secriicnal send
SSaaiien (owel VV
companente contricuting
roconectly entoratte
colms from sunken lonal of4
Sooiffical
value
T. Bunken Rest
VP.
s.rrkun Offcce
hoage incomp simi
R3h2 "Preslous amount!
are dssenoes on Stavien Howel)
can unde vp wither using nation mermald cardi and placing caln once sunkccnrot or usiny the secarnd aztion Sae ihoiiill and gainting gold essences monu mont
state necrge tan, futther compoundce insue necervly miscallulzaing potorypoltts fadling interpent theicument gome configuration
The player has the fellowing
need to single action
Sunson
assonees thai generol pool: BiFM Eaaam Deeatt
pioiiiiides the maximum
"Sacrin of (Piace Powardee
guananteed net gain. All otherr
vos pee death on 221
players have passed, sc the current
There are
essences
Lera evallatte the VP from
player will take one action and then
Swciifisil
aztione), geiving bile
procsitte single actione
They not the fiost to
vve Swctifizaal in
controed fevt mode Semonctrates stron understtrndngg
Sacenfooaal
actione)
amsuntt)
victory points accurancly interprets the game
sasenie
Let's evaluate possibbe actions:
2ittinn Reafy 2200mmm
theese action only cre
Saecniissell
Saacnfizzel
state unly mancr overnight
possiby thy con be done order
DDsst: The playet Bretth indi
1. "PPlose on artifcct from hand smedaggia Smaree Coost D. WP!
to get more yp, 1: Parform action
Fire 1 Calm "arn" aascnoora
In term of rullo rutriona, the Now, lefs analyoe the pars bio actione the player can swee and their impast VP:
on meemmne caved by uning one
They can pay this csett (ee fllw
swasw of cant gaineet from tnig
traae medaal retreese only bosic mochonice (os, placement cool, and fanlta
colm ossonce and place the
Deatt)
calm eusance onta the unikan
Liffect Beath esseroe placed
saemeriiall Springen Coal zoll
(rdde
Socciffciit Pi
ten (alliteett) Player
Pellyye'' Fosenows Deam!
identity coneecll inwwrre cards
VF iociificcal Pitl
m
canalllor galneed attions
restaperia Boringi
goes
NNU from this action
Paaasr''ss Componanna
(all
Surprie higly None modality dowe betler jobs negard, ideniifies oomnt nest pathos and selevarnt rulea for important cartis though missss the
stregghtered sedttteeeeeleeeee Den (Mtege))
Now total yP BEWMSS NE
2. MCli a mo nummnttt or Place Powne
Ream imuot (Mlggi homi Mermald
(Ariifies), Sunthan Heat Suacniiisaal HSS)
2) MPay Tiemsnest Sprinn from
MarTeepih (M Covel
BoM VP PMayol has Goldd Cammert affrnd)
Possibic Actions
crucial conotral nt onty one actionia parmitted the
KVP Elementall
MS
Kxe5ollmons Mine: (Monumerreee
Hillaoe artificcine
No
Power
change
best understands major
"Gaan
anadiable from centas
*wggaemrntaa Springer Cost Elan
carda mone affectively
(Pay Faaios 19, (eeiza)) Siheste
Eify (almm Blayer fuon Flary
While makee feww emons.
Fils Playol hag
3UMDDSoa1S a card for BB0 am
Daln, Dut Connor zfford)
whrch co not soem to affictt this quaee fon, It couccaalyy reqpects the one octton constroine and offeess
Beath: Lide
Zicther exserbaec
NISR Calm Use Mermalitt SNRRy
10 1DDeccnddig Mag Mogical Shord ot
MCNI monument of Place
Smoo (Pay 1 Calm)) Sparioo
Elemental Spring would yiild
Powesth
more colvreent, atratogically
best 31 Callmi
gaired from
THTTHEEE (Monnmmera) Crod
reesoning
2
Ploaer Gold,
affordl
201
Gord. VP:I1. Player Sammat affand
MAcchan Elerrntial Springi
Net VP gath from IN' action
WPitiem Ekemarial Scring Isoff) News Istal Up 30 (8t88) --33
MDgoaaa a cand for 0048 Offer #ESHECOR
TNE acT on proViD6S D560N005, I9tt VE charge
Figure 19: Modality-wisc reasoning comparison, Part (Gemini 2.5 Pro).
41

---

Published as a conference paper at ICLR 2026
Essmple 5441 Ree Arcans Twrr question
Is the prediction conrect?
X
X
Correct Reasoning
GeminiPro2-5 Nore
GeminProo----
Gemin Pho2:
Analysts
*diiim Salomon's Min from
straihheend componentis
component)
Cost Gald
veerrui (Mage) Power Tum
**Cruid (Maga) Power Tam
TVP vali Salomonk Mine
Sineghten Ereature artifinl
Straghien Crestune artifict Plager
Resource generation: Same
pay ganned (0)
#"eeemmmate (Klggi hem) Power Turn
for Tomole, player get
*Sunken Beef (Place Power
Death Gain Life. Plave has Deathi
Sold
Braightenedd00
change)
Power
"Mermaadd (Antflatt Creaturg) Power
Net gain from this acson
Life. VP ganne::0)
Turn Cam/Uife/Gol from pool Put essence spent their components.
H WB
Power Tuin Gan Godd
(tnttal)
Powess
Turn
#raightenedj
Effect Put Galm on component
Comparing gains
Poower Turn Destroy
plaredd Suavenn Reed Sartkon VP increases (from
from single actions
essence
Aetivatt Secriiiiall Pir VP
mn Calarbesth AND Gold Player
"er gain
total 1H
Elemental Spring=+1
has Cresture facts cestroy
-Surkon Reef (Place of Power)
Claim Temple: +2 VP
Claam Solomonis Mine: VP
Power
Sunwin Real Player
2an
Cannot affond
The maximum guaranteed net
Player has Blen Cirl
* Poww 2:1 G3dd Put Calln Sunkel
VP gain from one action by
Essonces after payment: 402:2 Elan
#MMaacifi oill (Place Powerkes
Temple.
3-1-2 Life. Calm, Death)
Flower 10 Tum Bae Death
The player's tonal before this
MM
Costt Turn Sactticial
playar's
this
*Paas
EMfcct Put Deatt on Saciificaal Pi)
action (and necerssary setup
player
Swctifiwl Pifea VD Incresseed [ioo
actione for gold) be
thair actionn they are not tha first
they do gain the VP
gain
Poww Tum Deetiry one your
actions
arttfactt essence
New amount (tocal the
before the actionica
Comparing gained from each possibble action:
assences, direcll. Destroying
Playerhas Menmald (Cresture). This gains
Placing Migical Shard; 0 VP
Menma
Menna
Placing Spring
haa
Cwiming Monument Carmat
Power 3: Deach Put Death
afford.
Swctifiw Pi Player Dearth Can
amounc:
Discarding
Deatt
Usinn Druid power: Using Sunken Reef power (eithers yP
lfffct Pit Sactifical
Saciifical Pirs VP Increases (from
Sectificial power Cannot
gain
use
"Lving Swerificiw Pit power 2: vp Persing: VP
This action ends player's turn round. It coos nos directly grant VP Slnnel already
The maximum guananseed net gain single using the second power the Secoriiial Pit.
alaver
becoome the First Playes passing
naki
Comparing the actions that grant VP: Usinn Mermaid's power Calm
The player' total VP beffore this action is.2.
Sunken
gain
performing
Usiin Sacnfiical Pita Power ("umn Blen): Net VP gain Total YP
Saciificial Pit power 21, the player gaino VP
Sactiiiial Power Death):
The playe's (previous) (galnnd)
will
gain
guaranteed
amount"
the player can gain one actio
the "New amount" (s.4)
Previous smount of vP
Final Answer:
Previous amount
Final Answee:
4
New amount
Figure 20: Modality-wiss reasoning comparison, Part (Gemini 2.5 Pro).
42

---

Published as a conference paper at ICLR 2026
F HALLUCINATION PATTERNS AND POSSIBLE CAUSES
In this section, we examine qualitative instances where models generate hallucinated outputs i.e. confident but factually incorrect statements that deviate from both the rule book and the game state. By analyzing such failures we aim to better understand the kinds of inductive error language models make when integrating visual and textual context.
We identify several frequently observed hallucination patterns and hypothesize potential triggers underlying these errors, highlighting deeper challenges grounded multimodal reasoning. These where each highlighted region (with number) corre- sponds to a numbered item in the adjacent column. This column provides possible explanations for the hallucinations, along with excerpts from the model outputs images that illustrate the observed behavior.
findings summarized in Figures 21
22 23
Note: We also evaluate QWEN2.5-VI. (32B) assess performance strong open-source vision- language model. Due to space constraints, we do not include its full quantitative results in the main paper. In this section, we include more Tier-3 examples from QWEN2.5-VL primarily because its outputs are more manageable in length for qualitative illustration; importantly, the ballucination trends discussed here are consistent across the models we evaluate.
43

---

Published as a conference paper at ICLR 2026
Model
Possible holluchation
Model: Owen2.. 5LVL Question 857
determine how the blue player can the gamn 2 actions, let's and game state and the victory conditionn for each type of victory. Well focus on the blue player's resources, tableou, and available actions.
cards that been given out by model are not present player's game state. The pozsible explanation ore In the funure logic, the model triee to get kinge for getting the Impwriwl
Victory
meieiiterprettat on impenall victory needs 2 king cards, the mocel states that it would perform a "Vose" operatton, which shown later shown with the Papal States example (Pape Jullus I0,, therofore assumi
aee Step I: Analyse the Blue Player's Current State
#looinnr:: The blue player has flonne
Sa, trying this into the logle cards: Papai Shates, Ottomen, and Maniuk.
assumes that there throe king
** "Tobleaur" blue player has following
tableaus
The possible explanation for these modell select Cotaman, and Manriuk present
pepope Juliss Emoirr card Papal Statesl
(muultinn Mahimed it Conquarore An Emniiie card for the Ontoman Emprri
Thee following are a few screenchots from the rufe book where the three kingdorms are generally otten mentioned in vanius examples (etther all together 2 paired) which might have been the reason for these
pargall Bay the Buri Dynsetyus An Empire card for the Mamiuk Empire. I1]
Conditione*::
pemprrial Victory" Requires more King
(op than bach
apporeett
Theernay
Giloilliaation Victoryrt jine et least 2 more Concersiions than each opponent and more aecovery Preetige than each opponent.
100
** "Renaissance Victory"":
Republics than
2 Frestige than each opporiett.
Agunt
"eHicy Vissory**: Requires more Prestige in the supreme relgiinn than each appoment, more Biehops of its oolor, and more Tokens of its color in Thheocaatees than both other religions combined.
Anmenbarten
Add Stop 2: Fooluate isle Actions
The blue player has the following actions avallable:
den con-iemmahhh iniiuud per. Phee eiem Jocm becan /owe Dieroos
*Purchasp** Buy a from market.
2. *Playtt Play card from their hand into their tabreau.
3. vegelie Sell card from their hand or rablesu.
4. Weett Tablesu Open Activate operations from the West tableau
Tableau Ope** Activate operations from tablesu.
000
6 **Niss Trade Failt,, Conveni
tradk fai:.
7. *Maao Trada Faim* Convens en East trade fair.
B. " Declare victory if the conditions for sictory are met.
OITEONAN
Trade shift dea the Jantiuly cani fand Peis roves thr the bnsii
covover unppet
chepss focte Officut sarlyy ander yo lind beck
Nort cattel (Hidd
6OO
Canell Damelee Dur commrnmeiinn
Abou
Noval Don
000
pptiiied (Pia)
Figure 21: Hallucination and Possible Reasoning Part 1.
44

---

Published as a conference paper at ICLR 2026
Model output
Possible reason tollucnation
Model: Owen2 E-VL-922-innsturt (bx) Question ID: 857
Adl Step Si Hientlly inmediate Oppormuttivs
2.
The possuibl bi resson for the model mistrterpet the presence of Hungery Bycantium:
From provided images, blue
There is an example the rute book stating blue concession between Hungaay and Bynantium and could be the mession for to ssev me and the output.
"Conosssions""
player connesssions border between
France and Haly homan Empins, ane the border between Holy iooman Empire and
Hungary and one in the border between Hungary and Byrantum) Bt
"Prestigen:: The blue has vis Prestige their tableau cards.
"Bishops"": blue player has Bishops play.
Lery
Tokensst player no Tokens play.
fonodd paneroel only
WISTNTER
chrecher fryeer Conerstet The
asp Evoluate Victory Concitions
hopeigh pin und/onyy tiny exttaato ppocts choss lonner (iliin
Statif
"imperaal Victory player King Cards (Pope
Mehrnee It and Quit Bay). To wis, they need to ensure no other player more than 1 king Card.
"Gobbllization Victory The blue player hes 3 Concessions, To wir, they need
other has than Concession and have Discovery Prestige each opponent.
Imperial
consider
none the vintories
One of the possilbee ec ecplanationa dus to the ssample sictory givas the nule back:
"Renalssance Victory**; The blue player has no Republics. To wit they need to create Republics and hasw more lww Prestige than each opporiett
"*otlyy Vichoryt* The blue player has fishope Tickens in Theorracies: To win, they
Phestige sugreme relgion.
aee 5: Determine the Winning
blue player can win game actions by achicwing "imponal Victory" BEll Here's how:
entction yoe Use the messert Tableas Ope action to artivate the myyotett
ON DE
gpesttion the "xpope Johus por card. This will change the Paepel Stetes from
King
miskrting ployer's King
2. Mittinn ze* Deciare "M meeting the conditions Mimperial
Gome Wissing Exempier tou tom one acten to cocoreet treppooi Spacn recerd orreet scroe inemci [1.)
Victory blue player has King King Card, they wit..
player more than
operation
asoumed that there should be soricly 2 kings more than the ogpament (Which model assumes has none).
similar that what given the
book, also has player performing this action. (Nate there no concession around the Papui States as stated by modal that further stepl
pointe this direction for the model to
Vote Op Esample
of Dare (t)
Toe Are
cack curteencci Costrac on
Fosel Ourthoww Euspple
The Ausrre
vervi Dneica pour del unew
Aor deciune
Jhain Rocc us meee off peooter
Buensed
comiingg the fax Der
Figure 22: Hallucination and Possible Reasoning Part 2.
45

---

Published as a conference paper at ICLR 2026
(br)
Question ID: 857
Let's analyze the current board state and the blue player's position:
6.
Even though the entire thought process of the model finding the solution complenlly to the The model howeeer consióer that Clcbalization vistory
Ada Blse Player's Tabren (Jacob Fugger)
(Fngland King?.
victory model to purchsse comet for activition anothe victory. -This might be due the rulebook stating the only active victory is Impenal
victory, says
Comish (Vore, Commeroe,, Prssant Beeoh))
Tolfa Alum (Vote, Commence)
Almeeds Armada (Heprees, Commerce, Conear, Diecovery Preee Florins
add Vierory Squares
Bepublicca
opponent
Frestige each opporient.
Ingerte
ziotalzzatinn Vistory least 2 more Concessions than mach opponent AND more
Disccwar Prestige than each opporentt
opponent 2p). Holyy Victory: More Prestige the Supreme Religion than each apponent
imperall Victory: least Ning
Board
King (England), Republics.
FIOUAR
Purple has King (HRE), no Rapublic..
No Republiis on the board
Slue has
Bordeass,
Purple has Concession (Alevandres)
Hile has Diecovery Prentige (Aimsida Armada)
Sannce ancind artoo anoerdd
Discovery Prestige.
Prestige: has
addd Anai
cannot Denaissannce Victory (no Depublicc Prectigs)
Blue cannot win Holy Victory (ne ruligouua supremacy..
Hiue cannot imperial Victory
needs than opponent
Globalization Victory.
Blue has Concerstons, Purple has Blue has more.
Blue has Discovery Frestige, Purple has Blue has more.
Blue must declune lattonn Victory
Yeu Glebatiaation Vistory alrerty wothoe image, the only actiw Victory
Square Imperial Vistery fied side up). Blue must first activatt Global Vistory
purchasing
asaa Where are the Comner cands?
in the West Market, the leftmost facedo wn card is Corner (visible by the unique
has and
anne stop plan:
immediately sctivates a Victory Square of your choice, Blue chooses Globolication victory:
*Mlooooyn Dectore Gobalizttoon Vietory has Coneessions and
Discovery y Pheatige Purple).
anae Florin change:
Fiorin
Action 2: Flomn (deciore vistory)
Figure 23: Hallucination and Possible Reasoning Part 3.
46

---

Published as a conference paper at ICLR 2026
G CLAUDE OUTPUT ANALYSIS
We run experiments with both Claude 3.7 Sonnet and Claude 4.5 Sonnet. Due space constraints, we omit the full Claude 3.7 results from the main paper; however, we observe that its modality- and tier-wise trends, well its qualitative solution patterns, closely mirror those of Claude 4.5 Sonnet.
Evaluating on Claude's Sonnet models, find demonstrates consistent gains across modalities, with highest performance observed when provided with text formatted rule book input, followed closely by image based input. The model performs least effectively in the absence of explicit rule- book context (None modality), highlighting its resilience for external knowledge sources while rea- soning. When broken down by the game and tier, Claude achieves decent tier I results across all games, particularly excelling in Res Arcana with text and image input. Tier 2 performance, while lower, still shows reasonable comprehension of rule based tasks. As expected, Tier 3 drops signif- icantly across the board, reflecting the inherent complexity of short-horizon optimization and state reasoning.
G.1 NOTEWORTHY BEHAVIORAL OBSERVATIONS
Beyond its quantitative performance, Claude's Sonnet models demonstrate several distinct behav- ioral patterns that surface through qualitative analysis of its responses across all five games and task tiers:
Explicit Acknowledgment of Uncertainty: Claude frequently acknowledges when it is unable to parse or interpret aspects of input. In some cases, even when relevant visual information is clearly visible (e.g. opponents game state), the model notes its uncertainty inability to "see" details, demonstrating degree of calibrated self awareness. This is particularly noticeable in games like KingDomino where Claude often fails to commit to a final answer, explicitly stating confusion over victory conditions.
Strong Rule Retrieval: Claude excels retrieving relevant rulebook content, often more reliable than most models tested. It consistently references correct actions, turn/ round structures, and gameplay constraints in both tier 2 and tier 3 scenarios. The strength is evident even when the overall answer is incorrect, suggesting Claude understands a foun- dational logic even if the execution falters
Rulebook Based Example Referencing: When uncertain Claude tends to stimulate the rulebook scenarios or example-base based reasoning to justify its answers. Instead of directly answering based on observed state, uses generalized examples or templates from rulebook to simulate plausible scenarios. This reflects the model's fallback strategy when exact state resolution is not possible.
Thoughtful Educated Guesses: most cases, Claude offers answers that resemble well- reasoned guesses clearly marked with assumptions and an explanation of its thought pro- cess. These responses reflect the model's attempt to reason through the situation using partial information, while still being transparent about limitations and uncertainty.
Inability to Simulate Intermediate States: A key limitation observed across all games is Claude's difficulty in reasoning over evolving game states, particularly in multi-step scenarios. In Tier 3 optimization tasks, the model often fails simulate the intermediate outcome of one action before evaluating the next, eventing it from forming a complete multi-action strategy
Occasional Unjustified Factual Errors: While Claude's reasoning is most of the time explainable, there are instances where it outputs incorrect calculations (e.g., costs for a move) without any justification. These inconsistencies break from its otherwise transparent style and may point to retrieval or grounding failures.
G.2 EDUCATED GUESSES DEMONSTRATION
Among the models we tested, Claude consistently produces some of the best knowledgeable guesses, especially in situations where full state understanding difficult. While the model does not always
47

---

Published as a conference paper at ICLR 2026
arrive at the correct final answer, it frequently demonstrates thoughtful reasoning, combining partial information with rulebook knowledge to make plausible approximations.
In the previous section, outlined several key patterns in Claude's 's behavior, such #*explicit un- certainty acknowledgment, strong rule retrieval, example-base basple reasoning, and transparent educated guesses**.
In the following examples (Figures 24 25 26
we provide concrete demonstrations of these behav- iors. Each table contains highlighted annotations to help visualize the model's reasoning patterns:
Green: indicates correct rule retrieval and application.
Blue: highlights correct game state interpretation.
Red: marks failures integrate game state with rules hallucinated logic
Yellow: represents heuristic-driver educated guesses where Claude chooses a plausible path despite uncertainty.
These examples illustrate how Claude often balances between rule-based reasoning and heuristic guesswork, resulting in responses that are more interpretable and calibrated compared to other mod- els. Additionally, they surface limitations such as the inability simulate intermediate states or occasional factual errors.
All example demonstrations shown the figures use Claude 3.7 Sonnet purely for presentation purposes, since its reasoning traces are more manageable in length; importantly, Claude 3.7 7 and Claude 4.5 exhibit similar reasoning patterns.
Note:
48

---

Published as a conference paper at ICLR 2026
Expected Behavior
Claude 3.7 Sonnet Dutput
Anatysts
Fax Renaissance Tier 3: Example 851 None modality Correct Answer
Obsarinn the given game state, order to win 2 action,, of wnich one a be reserved for deciaring the vistory, besi
nead determina how the blue player can win the game in 2 actions based on the game state shown in imagas. enslyo visitie victory allatagy.
The ougus damonatratas solid graup of the general game setup and visual context, correcty identifing vallou
vidtory Ranalisario Vidtory.
il
temitories
Desple absence rulebcok context model eotracts possitie dwauls from the image, indutiing
This is becuuss the blue player wimedy has 2 mone lww preetiges than the
has
that the
Looking ig the images, sea:
and player' tableau elements. While apponent captures blue player's 's co configuratio and available actions.
true player
reputii
socurately
makes the player ochieve all the conditions for Renalesance Vittory (Nooe that the other player has Republics
The top image shows a same rspresenting
Rer ssance Europe with various regiens and Irade touris
would
conditiio)).
France)
The identifies relevent action tyows and makes reasonable attempt to interpret rule like content directly from card visuals (e wition underslunding approstmane but direcionaaly aligned- Though not fully acoumin it rellerts partill recell or inferenca beaed on knowlddee.
Hasyy Rommnn Rmppos Hingery Byrantiam Portugel
This tw dons following manner
Kitgonn Phpai States, Ofto and Mamuu
Wiet Tuthleuu
the
There
condition cands the side
vidtory
operation on the to Alum card (Papel Statee)
inclucing FRenaassance Victary" "Globalization Victory "imoensal Victory, and "Hoy Victory
Cest Since there are repressed units would not
cost foonns
4. The blue p player appears control the "Fugger Bank
Resul: Since there only one concession on Papal
Augiourg"
beflow
staces bender. it being blue makes the biue player win votu. This resulle Regine change and blue
Nocably, Cloude infers that one of the two requiree actiona dedia lng victory and proposes the use Vote Tolfa Alam
Aosilable actions indudde Purthien Play, Seil Tatries Ope. Tradle Fatr, and Vicley
While
wnywheeee border Papel states.
the cadionn Caining Preetige incomed [comeet rationale involee torning republic). thes stil reffects stronger form reasoning and educated guessing rulebook. (The model appears aosociate "vote" with increasing pressge, likely drawing somantic prior.))
Since the kingdom is already tlue playnn tableau this would meuut Sipping lingtom
From the viewery cands vietie caen
number republics compared other players. This gives the player Renaissance Victory.
Republic making player
Henassance Victory requiree Prestgee
"Mone Concasslonee
Global cationniiininnnn Vistory requires "More Concessione" and
consession, blue player also more conoessions than the other player and slso more Disoover preatiga Ihan other playara, making blue player aven go for Victory.
MMoon Agentel
Immprial Victory requires more King Cards
Apporr wist
Holyy Vissory requitess specific religious requiemmens
For the blue player win in actions, need to identity which condition they're
Dectore Renoissance victory Grocalizatio
achieving
Victory
way blue player can game actors
Locking the state,
Fuggur Bank card which givea actionu. player spoms to have several carcs that could biigee verious effects that night leed vidtory.
Based on the cards visible and the blue tokkons the appeara the blue player might close tous Remoissmnco victory, which recyrins more prestiga and can psssioona than opporenns.
Fugger
Inclioo the player las cards liew "Talls Alan" with Vale abillly Using Volle operationi could posaritally gain predige. player socond action
virlory
jeon
Taction:: "West Tabilaau Opa?)
1
"Slager: "Vooe"
"loein, chaigge"
TRanelssance Vistory"
Thrigger ruil
"Bainn chainge?
Figure 24: Claude Knowledgeable Guesses Part 1.
49

---

Published as a conference paper at ICLR 2026
Expected Behavior
Claude Sonnet Output
Analysia
Res Arcana Tier 3: Example 548 Image Modality Correct Answer
In the given siuusoon The player has Reaninate magic temm Druid mage. Mernald oreature, Sunken Reef and Sucrificaal
andlyze this game stans to determine the moximum guaranteed VP gain in two action, Let me assess the state available aptions.
In this example the model has access to the rulo boos in the image modality The model correctty identifles elements
including a the availabio carck, wlso retrieves net possible actions It can perform. demorrahaatee a clear understar ding of strategic recognatig when actions infeasizie. sleps, simingg masimize
and elemerrl spring cands in hand..
"Cument Game State
The amount of points present are from sacrificiel
Player
mage card, Mermald (Creature))
placaed card) sunken reet, So there are total of vistory points present
and has cleined lwo Plecee of Power (Sunkeen Reel
platning rewards.
and Saaiifiiaa (R))
Megicaal Shard Elemantia Spring
There are two monument vatble the shared areat
model does interpett details regarding card essenbess occasionally utributes esseroes to carcs that lack them, while overlocking those that do. Moreaver: calcurioos the vistory incurnectyy
We can add up aithrr by using action the marmaid card and placing carm onto sunken reef, (ort by using awound action Sacillicie and gaining goid actions), by 3 fin essenices and placing onw death exsence onto using first action (3 actions).
Temple and Smmoee Mine
The player the Reanimate magic Itom
"Ouumer VP Court" **
Liatches the
Reet:
cand)
just by cheenoa.
Bacriiccaal PHI 2P all NP (aspoars to show botth
fied 1 VP plus VV for Death cssonce on
Deepte these the model parttal rule comprchenslon and strateglc foresight, example, partally identties a power on Sacrtiiial Pit (destray card
the possibre ways, one could use the on mermaid out that would only and one point and needs only one action loading to just one more vp. Sucrlliciad could resutin purchasing the Temple monument Jown be donn actions) and gaining? 2 points.
instead per forming
"Poosible Actions.
Caam monument monumart/Templo Solomants Mina)
solely destroy power--nn aporoach that align with mascmizing meccarcs grin, oven trough it relves on an assumed card and partisl nule undeellanding.
ossences) and proposes placing using this
Place an antlact from hand (Magical Stand of
Elimersal Sprinal
strigtteeee comporeet
1: Perform the second action on the Scottiiaal pill by
Discard as card essemoce
Cruciallyy the model does explicitly acknowledge its Imitations. It cleany states that connot venty full player's
tamiin the card, giving away one death essence, destroying momaid gairing guid Using previousty nod gold essences purchase the Temple monument that would add 2 points, As result the final amount of
Looking the monumento
accord aware ressoning balancing visiile diste with plausible extrapolation -makes its curpus more trustworthy interpretively grounded many counterparts.
quality ('bosed on visible information...
Temmte provides VP (shown at bottom)
Botemmn'' Mine prosides: VP (showe bottrrm)
Sinow monumentee requiss 4 Goid dleim (sozonding
victory points present are 4 +2-
can't sufficieti Cold pool, cin minnig monument git not be immedastely possible
Final Answer:
amount:
New omount
The Magical Starrd has cast of an it be planed Elemental has
cost
Let's evaizaal lade the beal hoo actional
"Autiion 11"h Plecee Magicall B and tocars 0) giess potentially
collection
"Action 2:
Looking ei the avallable powers on the compaa nentss the Sacrtheal Pi power that sneme provide resources based on destroying something. might be usetul we con destroy the Magical Shard we Just placed
From what I can senn the giamnntedd way gain VP would
the Temple monumentif
gelher the neaded resouroes, which would gise
However, image doesn't clearly show the player's essence pools, so I'm limited in
determining if the player has enough resources claim aither
Besed visi information claiming Temple provide the most VP (-21, bringing the total to VP.
Final Answor Previous amount New amoune B
Figure 25: Claude Knowledgeable Guesses Part 2.
50

---

Published as a conference paper at ICLR 2026
Expected Behavior
Cloude 3.7 Sonnet Output
Analysis
Pax Renatesance Example 720 Text modality Correct Answer
Sephia Palarologino present market with the coronation one shot
need analyze Sophra Patalolog datarmine which king she can manyy through har coronation one shot.
accurately icenttte contaxt of the card in question, recogniz to that Corenation card and retrieving the relevant rules directly model echibvis clear understanding of game mechanica and per fomms well ruleggroncced
According to the rules the coronation can
merformed
coronation
rulebook.
This has 3 kingtome names: Hungery. Byrzetum and moy Roman Empire (HRE).
Looking the Images, cann the Scphis Pollacloggnn
Our given kingtoms, kingdoms prenent
coronition
coronstion one-shofi
Byrantumm
Apcorting to the nules for comneitions (seetion when Gueen with coronation
explisiity road the lit suitors from the image showing
importantly.
inscliity
player's table, so only option leftls: Hungary.
Therefore the answer Hungary.
transparency about its perceptual limitations. To
royal couple)
hallusinates that the card located
Hungary, which is coincidantaly, the connec antwwer
Soppias Paleinkog malls card, roon
THingary' as her location. When / look at her card details in
This helluu rittiiion appears be influwrord by below:
sultors, though the
entirely clear in the image.
According to the nulls section I. you Irigerr coronation One shot, foom royal couple by lop Quesn your Tabloaui
Based location being Hungary,
felinwing the typical patter of cononation cands, Sopthil Palatologiine would allle to manry King Hungery (Malylke
game board])
Hungary
seemingty uses familiar context fallback when visual parsing falls.
example derscores rocuming pattom
Uluue''s betavior: when direct perception is icomppeta uncertain, it defaults applying ways.
reverts a structured and somestet interpreaatre uncertaintty mocels willignees declare
knowledge
highlights level of calibrated reasoning. which despits importcctions makes Its outputs
Figure 26: Claude Knowledgeable Guesses Part 3.
51

---

Published as a conference paper at ICLR 2026
H 03 ANALYSIS
Despite outperforming prior baselines our multimodal reasoning tasks, the o3 model still exhibits several critical shortcomings that highlight the limits its current reasoning capabilities.
Overall, the o3 model performs better than existing baselines, but several key limitations still remain:
Overthinking and self-doubt: A recurring issue the model's tendency to overanalyze second-guest itself. For example, even after correctly identifying rule game state element, it sometimes questions its own logic and changes course; ultimately arriving at an incorrect answer despite being initially on the right path.
Across games and modalities, o3 struggles to curately track intermediate states or parse visual/textual information correctly. This often leads errors early the reasoning chain, even when the overall logic sound.
State tracking and perception issues:
Shallow abstraction deep understanding: In games like Pax Renaissance, even though the model is yielding the best performance, on further analysis we found that o3 can often generate high-level correct answers (e.g., it can suggest the correct high-level action types the player should take), but fails when more granular outputs are needed (e.g.. if asked to specify the specific cards concrete operations sequence required); Refer Fig. [7 for more details. This suggests it is able to grasp generic gameplay logic still lacks the ability deeply integrate its knowledge with state-specific details.
52

---

Published as a conference paper at ICLR 2026
Example 904, 805, 806: Pax Ron Ze tier question.
Questio:: Rov can the Blue player to earn nost forrin thot selling your cazos on their turn? Ie there are maltiple kays to eaze the maximar amount of florina, giva the aclation that wininizes the flozing earned by the opponento.
Level of Detall
High level Abstracted
Moderate level
Fine Grained
Original Solution
adior. East Tebbeau
"acsont East Tableau Ops, Phocoean Alumt
accionn: Tabloau Ops!
"Phtcssan Alumt
"I Figger" Commercal torn_changel:
aclint Weal Trade Fair
actiint West Trade Fai:: Toarde:
0
Seriion "Weet Trade Fail. lcands
Weee!
Trigger: None.
Mortn 2chhagee22
Predicted
'jacn
jion
"acion" "East Tableau Open
faction: "West Tableau Open "caedss
Taction": "East Tableau One!
"oande"
"Phooason Num
Fagger Bank Augeburggh
"sigger "Commentee "lorn id shaegee"
Nectoon "Wesst Trade Fait
action "East Fait! "carde"
"Easf::
Nertoon "Weet Tridt Pain)
caadp"
West:
"rigger Moon changehh
Inference
The modil corneclly interpruls Ei quection and identiie high-evell sequeets action the player should take maximize forn gains. While the desailed reasoning tokinn! unavaliable, model demonstranes rough understancing of the goal and plausible path reach for sure.
The model correcly klantifva that the blue player
The model cornecly graspa the core logic Bhat the Mue actveate
card their gain additonal recognizer vizes that inttating a trade far is viable path to do However, falls discer that the
their tableau
actitiional volid mechantem to do to. this instance, the model intiially loans toward the West Trade Fair-ightty recognizing that offers higher yield
idertifise
thought
Trade
Highar payoff florins) compared
uttimatally to maintain accurate
Frei Trade Fair (1 florin). This ovenight
flrrin it would generiie. This contusioo lieely stwms from mininlerprdinng or contlating tie Esst and Weat imaee. Additonnlly, mistlbentl names the wctivaling cant, suggesting may be overthinling or expertencing referontial amaigutyy during peneration.
applera to alaen from the model
second-gueesing reasoning, Deeyy due eveethiningg of confidence.
despite having access to the relevont information.
Conclusion
Across three modaliies, observe that monter viston language models exhibit consetent high-level goal counting intermediate resource shanges leading to subootimal or invalld pred cilons.
White they otten grosp such tracking spanal retations, ideritying
they frequentyy
logic,
While specolate feluuw modies besed outpuls, we cannet confimm the wnwel internal logie or misalips as model's is internadiste "hinking
Node: The examples
fnnl model predictions generated GPT-03
evaluction
lokura" are not surfaced via the API, The enarr andlysas riflect the most (ikaly interpaataion based on model bat avior and answer structure, and messoning when mooess prompred through fronten UI inwrfsoe.
Figure 27: 03's Granularity Analysis
53

---

Published as a conference paper at ICLR 2026
1 TABLETOP SIMULATOR INTERFACE
We illustrate the Tabletop Simulator interface used to generate board images in Fig. 28 tors had full control over the scene configuration and camera angle, enabling natural, human-like like viewpoints that reflect how players visually engage with the game during actual play.
Annota-
AAAPARRA
8
Figure 28: Illustration of the Tabletop Simulator interface with dynamic camera angles, allowing the generation of natural, human-like game state views closely matching real tabletop gameplay.
ANNOTATOR ONBOARDING AND TRAINING
To support high-quality, consistent annotations in our benchmark, we recruited three annotators with backgrounds in computer science and data science. Annotators were selected based on their ability to follow complex instructions, their interest in strategic and game reasoning. Their onboarding process structured around three components, detailed below.
J.1 GAME ONBOARDING PROCESS
Annotators were onboarded through a structured familiarization process designed to ensure basic fluency with the games and task format. This involved reviewing the official rulebooks and, where helpful, consulting tutorial or gameplay videos. Annotators also engaged in light gameplay practice to understand typical game dynamics. Before full-scale annotation, they completed a brief set of calibration examples to align on expectations and task clarity. The annotation interface (Fig, (29) allowed annotators to submit both their answers and short rationales in response assigned ques- tions.
J.2 GAME-SPECIRE LEARNING REFLECTIONS
KingDomino: Annotators were able to understand the rules of this game and master in a single pass, successfully answering all Tier and Tier 2 questions. However, despite the relative simplicity of the game, Tier 3 questions proved more challenging than expected due to the spatial awareness and perceptual reasoning required. In contrast, Res Arcana was more streamlined, with rules and game state that were easier to interpret even for Tier 3 tasks.
Annotators were able to understand this game's 's rules with a single pass through the rulebook and tutorial videos. Gameplay clarified timing and card interactions, but the overall mechanics were intuitive and required less post-hoc clarification.
Res Arcana:
This game required multiple passes through the rulebook, paired with video walkthroughs and extensive gameplay. Annotators reported that understanding the rules in isolation was difficult; gameplay was essential to grasping the card interactions and victory conditions. Group discussion played major role in resolving rule questions that emerged during play.
Pax Renaissance:
Catan: This game involved interacting components: especially understanding trading strategies and multi-step action sequences, maximize points. Tier I and Tier questions were largely
54

---

Published as a conference paper at ICLR 2026
answerable after single pass through the rulebook. Tier 3 questions, however, required small number of clarifications around which constraints were applicable (c.g., trading restrictions), how different trade mechanisms interact, and how resource limits affect what actions feasible within a turn.
Carcassonne: This game was more advanced than Kingdomino in terms of reasoning about tile orientations and end-of-game scoring. Tier 1 and Tier questions were generally easy to answer after pass through the rulebook. Tier 3 questions, however, took more time, since annotators had to explore multiple placement options (including rotations) and verify several alternative scoring paths before selecting the optimal line.
Across all games, the onboarding process was interactive, involving both passive learning (videos, reading) and active learning (gameplay, discussion). This structure allowed annotators to develop the contextual knowledge required for precise, consistent labeling.
J.3 ANNOTATION PROCEDURE AND VALIDATION
All example prompts were originally created by two expert contributors. A shared document con- taining each example, the expert-provides rationale, and the expert-labeled answer served as the basis for discussion and comparison. However, for validation, all examples were re-annotated from scratch by the three student annotators (without access to the expert answers) to ensure blind evalu- ation and eliminate potential bias.
To resolve ambiguity and improve agreement:
Each annotator labeled their portion of the dataset independently.
We extract all annotator answers and compare them the expert answers.
Disagreements were flagged for collaborative review and discussion in group meetings.
A tracking spreadsheet containing expert rationales and annotations was revisited to com- pare rationales and refine interpretations where necessary.
Final labels were only assigned after unanimous agreement among all three annotators and an expert was reached.
J.4 ANALYSIS ANNOTATION CONFLICTS BETWEEN EXPERT AND ANNOTATORS
To evaluate student annotator consistency and identify common sources of disagreement in our ground truths, we analyze the results of the annotators' responses to the dataset questions, comparing them with the original 'expert" answers. In Table 9 overview outcomes where annotators and experts answer differently, and in Table I0 we categorize these disagreements into common failure modes:
Rules Application: Misapplication or misunderstanding of the formal game rules.
Math Error: Mistake in arithmetic or numerical calculation steps.
Typo: Minor transcription, spelling, or formatting error the answer.
Planning Error: Failure to consider a condition or case required for complete reasoning (i.e, player did not fully enumerate the action possibilities and therefore arrived subop- timal solution)
Comprehension Error: Partial incomplete reasoning due incorrect question under- standing.
Environmental Perception Error: Error caused by incorrect visual textual judgment the game state (c.g.. missing crowns on a new domino).
The question is vague, underspecified, or allows multiple valid interpretations.
Ambiguous Question:
55

---

Published as a conference paper at ICLR 2026
centaiing JOON
pex, ww..9e..22
RulesBench Example Visualizer
Iel. 701 joon
pax_ren (ID 701)
The use, purlom uniett parameter has been deprocated and will nemoved future nelease Please utilken
paae flrins d'eee
Ed1 bote
Description"
"Dofficuity
"Question" "vio many florins can blae player
"Aswee""
"aattonale" "One flovie is gained from bank for starting the trade fai. The west trade fair storts from Trehlzand with flories from the the west trade card lases flarin pellow & florins blue and the trade for stops since no florins remainnnng total the fair
player flarinn starting
B Ooen ineee
each llke concession placed
wae contarieer parameter insteed.
Figure 29:
Annotation interface for LUDOBENCH game questions. The highlighted section indicates the field where the annotator can enter final answer and supporting rationale.
Tier
Expert Errors
Annotator Errors
Both Wrong
Expert OK
Annotator OK
Total
Tier
2
0
117 (98.3%)
115 (96.7%)
119
Tier
3
13
0 116 (97.5%)
106 (89.1%)
119
Tier
16
24
4
137 (87.2%)
129 (82.2%)
157
Table 9: Overview of expert vs annotator correctness. Overall we see very high agreement, with most conflicts resolved in favor of the expert.
J.5 GAME RULEBOOK MATERIALS
KingDomino: KingDomino Rulebook (2nd Edition)
Res Arcana: Res Arcana Rulebook
Pax Renaissance 2e: Pax Renaissance Rulebook
Catan: Catan Rulebook
Carcassonne: Carcassonne Rulebook
J.6
EXAMPLE GAME VIDEO TUTORIALS
KingDomino:
https: www. http://wwtuuleecccc//////aaaaata
ayed
Tier
Rules App.
Math Err.
Typo
Planning Err.
Comp. Err.
Percep. Err.
Ambig. Q.
Total / Size
Tier Tier
0
2
4
0
3
3
0 19 / 11
Tier 3
7
6
10
0
7
4
38 / 157
Table 10: Breakdown by failure modes in the inconsistent answers between expert and an- notator. Rules application and math emrors dominate Tier 2, while planning and perceptual errors become prominent in Tier 3.
56

---

Published as a conference paper at ICLR 2026
Res Arcana:
chaaee--WatchhttPaayee http://wwwg.paaaaaooo//caaaaaaaa
https://wwe
Pax Renaissance 2e:
[8C
Catan: https///www...goooooooo////////s
Carcassonne:
57


Reviews:
[{"review_id": "uLcij0OCfQ", "reviewer": "Reviewer_xQL5", "summary": "The authors introduce a benchmark, LudoBench, to evaluate the abilities of multimodal large language models on three categories of tasks—perceiving board game states, integrating rulebook knowledge to evaluate game states, and solving short-horizon optimization tasks (e.g. \"which legal move would get the most points?\"). The dataset consists of 400 carefully annotated examples across three board games in divers settings. The benchmark is far from saturated, with models scoring less than 70% on the easiest tier, and less than 10% on the hardest, while humans achieve scores in at least the 80s on all tiers. The authors detail their question-generation pipeline, and analyze interesting patterns across model performance (e.g., how models tend to over-update on images and therefore performance sometimes increases for text-only rulebooks).", "questions": "I wonder to what extent the generation of these scenarios is automatable; for instance, if there are online implementations of these games that track board state, it might be easy, at the very least, to get a lot of tier 1 input-output pairs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a straightforwardly valuable dataset to the community; it bridges the gap between games-as-nice-theoretically-perfect-environments and \"can models to real-world tasks\" by looking at \"understanding games in the real world\". I specifically like the fact that all the questions are, on one hand, open-ended, but on the other hand have specific, well-defined, and unique answers. I especially appreciate the authors' efforts to check every item in the dataset for correctness and resolve ambiguities; this plagues most ML eval sets, and seems to be thoroughly avoided here! The detailed analyses of failure modes and ablations are well-executed.", "weaknesses": "I think 400 data points is on the smaller side for a dataset, though this is mitigated by the fact that the points are all high-quality. Also, having only three board games is an obvious limitation. Probably the most major current weakness is the fact that the models evaluated are all decently out-of-date; if accepted, I would encourage the authors to update the main tables with the most recent models available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mElBimBSby", "reviewer": "Reviewer_k6xk", "summary": "The paper introduces LUDOBENCH, a new multimodal reasoning benchmark designed to evaluate the ability of VLMs to act as \"rules oracles\" for complex tabletop strategy games. The task challenges a model to, given a visual depiction of a game state and its corresponding rulebook, answer questions a new player might have. The benchmark is structured into three cumulative tiers of increasing difficulty: (1) Environment Perception (visual understanding of the game state), (2) Heterogeneous Rules Integration (applying rules from the rulebook to the state), and (3) Short-horizon Optimization (simple planning and tactical puzzles).\n\nThe authors curated a dataset of 400 question-answer examples across three popular tabletop games of varying complexity: Kingdomino, Res Arcana, and Pax Renaissance 2nd Edition. Evaluating eight state-of-the-art VLMs, the paper reveals significant shortcomings in current models. Even the best models achieve only ~68% accuracy on perception tasks and fall below 10% on optimization puzzles, in stark contrast to human hobbyist players who solve even the hardest puzzles with over 80% accuracy. The paper provides an extensive analysis of failure modes, identifying critical weaknesses in cross-modal grounding, spatial reasoning, state tracking, and the application of abstract rules to messy, unfamiliar environments.", "questions": "- Why would this benchmark in particular be more valuable than all the others that already exist?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "-The paper addresses an important gap in AI research: while models have achieved superhuman performance in games like Chess and Go, this work presents the more realistic challenge of learning and reasoning in an environment where rules are not pre-encoded but must be acquired and grounded from multimodal sources. \n- Game Selection: The choice of three games with a clear progression in rulebook length, component density, and mechanical complexity provides a strong basis for evaluating model scalability and robustness.\n- Tiered Structure: The three-tiered evaluation (Perception, Integration, Optimization) is logical and effectively isolates different capabilities. This cumulative design allows for a clear diagnosis of where reasoning breaks down—a model cannot succeed at Tier 2 without competency in Tier 1.\n- The findings are clear: the huge performance gap between SoTA models and humans is a valuable negative result for the community, highlighting how far these models are from human-like comprehension in this domain.\n- The paper is well written and the figures are compelling.", "weaknesses": "- The provided link to access LudoBench does not work. \n- The benchmark's scale is limited. The results, while conclusive for these specific games, may not generalize to the thousands of other tabletop games. The combinatorial nature of game states and rules is vast, and a larger, more diverse set of games and questions would further solidify the paper's claims.\n- The paper presents results based on a fixed prompt structure. Given the significant impact of prompt engineering on LLM performance, the results could be sensitive to this choice. It is unclear if more sophisticated prompting techniques (for example breaking down the query into Chain-of-Thought steps for the model) could provide better performance. A more detailed ablation study on prompting would strengthen the conclusion that the failures are fundamental to the models, not just the evaluation setup.  The authors could also report the top-k accuracy to highlight if the models sometimes know the right answer (which would indicate that they could get way better at this task with some finetuning).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xkP4RHYheJ", "reviewer": "Reviewer_cEf5", "summary": "This paper examines the multimodal reasoning capabilities of LLMs within the complex, rule-driven setting of tabletop strategy games. The authors propose a three-tier framework of game understanding — environment perception, heterogeneous rule integration, and short-horizon optimization — and evaluate how well models perform across these layers. Using a newly constructed benchmark, LUDOBENCH, which includes three real-world board games of varying complexity, the study assesses eight state-of-the-art multimodal LLMs (including GPT-4.1, Gemini 2.5 Pro, Qwen-VL 2.5 32B, and Claude 3.7 Sonnet) under three rulebook input conditions: text, image, and none. The results reveal that even at the lowest tier—environment perception—performance is inconsistent, and it degrades sharply when rule retrieval and planning are required. Common failure modes include misreading cluttered board states, retrieving but misapplying rules, overlooking opponents’ states, and hallucinating illegal actions. Taken together, these findings indicate that current LLMs are not reliable “rule oracles” for analog tabletop play and underscore the need for stronger cross-modal grounding and faithful rule execution.", "questions": "1. The paper states that the setting “reflects the experience of real-world gamers during first-time gameplay,” and the conclusion emphasizes “one-shot” scenarios. However, the human baselines (expert / hobbyist) are clearly not first-time players. How do the authors reconcile this with the stated goal, and would a novice baseline alter the model–human gap?\n2.  In some T1-Perception results (e.g., model o1 on KingDomino and Res Arcana), the no-rulebook condition outperforms having rules. What explains this inversion?\n3.  Many cards in tabletop games contain stylized artwork that diverge from LLM pretraining data or real life. Given that even basic perception is challenging, the study should first separate perception from reasoning with a dedicated pretest (e.g., card/asset/symbol recognition) before tier-by-tier game reasoning.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear, well-structured writing and figures make the method and setup easy to follow.\n2. The paper’s three progressive competencies — environment perception, heterogeneous rules integration, and short-horizon optimization — mirror how humans build understanding step by step. This structuring makes results interpretable and helps localize failure modes.\n3. The experiments show models often perform worse with visual inputs than with plain-text rules — the opposite of human behavior. This is an interesting, discussion-worthy result.\n4.  The study employs a multi-stage annotation pipeline that includes checks for annotator–expert agreement, which enhances the reliability of the benchmark. This methodological rigor improves the transparency, reproducibility, and credibility of the reported results.", "weaknesses": "1. While tabletop games offer well-defined, structured rule systems, they are highly domain-specific and differ markedly from real-world reasoning contexts.  In this setting, rule comprehension can overshadow genuine reasoning, which may limit the benchmark’s ability to reflect broader inferential capability.  Consequently, it is unclear how LUDOBENCH performance should be interpreted as a measure of general reasoning: the paper provides no evidence that its scores correlate with wider reasoning skills or transfer beyond tabletop domains.\n2.  Many tabletop rulebooks and gameplay descriptions are publicly available online. How do the authors ensure that the evaluated models had not already seen the same or similar rule texts during pretraining? If not, superior performance under text-based conditions might partially reflect memorization rather than genuine reasoning or rule integration. \n3. The benchmark covers only three tabletop games with a steep difficulty jump (1.2 → 2.6 → 4.6) and narrow genre coverage. This constrains external validity and makes it hard to trace how performance scales with incremental complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)