State Design Matters: How Representations Shape Dynamic Reasoning in Large Language Models
Authors: Annie Wong
Aske Plaat
Thomas Bäck
Niki van Stein
Anna V Kononova
Venue: Accepted by TMLR ()

Abstract: As large language models (LLMs) move from static reasoning tasks toward dynamic environments, their success depends on the ability to navigate and respond to an environment that changes as they interact at inference time. An underexplored factor in these settings is the representation of the state. Holding model parameters fixed, we systematically vary three key aspects: (1) state granularity (long form versus summary), (2) structure (natural language versus symbolic), and (3) spatial grounding (text-only versus images or textual map encodings) across sequential decision-making benchmarks. We find that trajectory summarisation improves performance by reducing noise and stabilising long-horizon reasoning. Second, natural language representations are the most robust across models, whereas structured encodings help mainly for models with strong code or structured output priors, such as JSON schemas. Third, while image-inputs show some benefit, text-based spatial encodings prove most effective. This advantage stems not from the spatial information itself, but from the act of construction, which compels the model to perform the spatial reasoning that static input does not elicit. Overall, we demonstrate that design choices for representing state are a decisive factor in performance, distinct from the availability of information itself. We note, however, that even with improved representations, current LLMs and VLMs remain brittle over long horizons, particularly when they must synthesise information to manage multiple subtasks to reach a goal.

Published in Transactions on Machine Learning Research (04/2026)
State Design Matters:
How Representations
Shape
Dynamic Reasoning in Large Language Models
Annie Wong
a. www.wonniiie...eeeenniii........
Leiden Institute of Advanced Computer Science, Leiden University
Aske Plaat
Leiden Instibute of Advanced Computer Science, Leiden University
Thomas Bäck
Leiden Institute of Advanced Computer Science, Leiden University
Niki van Stein
Leiden Institute of Advanced Computer Science, Leiden University
Anna V. Kononova
Leiden Institute of Advanced Computer Science, Leiden University
Reviewed on OpenReview: /t openrevew.. net/ formm? Bew.Hoooaaaiiiiggghooeeooz---eee
Abstract
As large language models (LLMs) move from static reasoning tasks toward dynamic envi- romments, their success depends on the ability to navigale and respond to an environment that changes as they interact inference time. An underexplored factor in these settings is the reprosentation of the statc. Holding model parameters fixed, systematically vary three key aspects: (1) state granularity (long form versus summary), (2) structure (natu- ral language versus symbolic), and (3) spatial grounding (text-only versus images or textual map encodings) across sequential decision- mabbiiinnmakingg benchmarks. We find that trajectory sum- marisation improves performance by reducing noise and stabilising long-horizon reasoning. Second, natural language representations are the most robust across models, whereas struc- tured encodings help mainly for models with strong code or structured output priors, such as JSON schemas. Third, while image-inputs show some benefit, tezt-based spotial encodings prove most effective. This advantage stems not from the spatial information itself, but from the act of construction, which compels the model to perform the spatial reasoning that static input does not clicit. Overall, we demonstrate that design choices for represcnting state are a decisive factor in performance, distinct from the availability of information itself. We note, however, that even with improved representations, current LLMs and VLMs remain brittle over long horizons, particularly when they must synthesisc information to manage multiple subtasks to reach goal.
Our code available at Sttp:e///aaayull.cco///attttrtee
1 Introduction
Benchmarks for dynamic, multi-stop reasoning have rapidly developed in recent years, reflecting a broader shift Loward evaluating large language models (LLMs) in interactive settings where they act as controllers PPalleri et al.] 2025 Wu et al.) 2024b). Yet evaluation practice remains dominated by static, single-turn benchmarks, such as question answering, coding, and math word problems (Brown et al. 2020 Chowdhery et al. 2023 Wei et al. 2022b). Strong performance on these tasks has fueled claims that reasoning abilities
emerge with scale Wei et al. 2022a However, whether such behaviours constitute genuine reasoning

---

Published in Transactions on Machine Learning Research (04/2026)
Axis
Representation
Granularity
Long-form
Summary
Step 1: agent at (5,5),
Agent at (5,4);
observes bird at (5,3).
bird step south; fish steps east.
Step 2: agent moves south to (5,4),
observes fish (7,4).
Structure
DictList
Matrix
Natural Language
("A": "B": (C)
[[2, 1,0],
Peg has disks 1, and O. Pegs are empty
[-1,-1,-1],
[-1.-1,-11]
Spatial grounding
Text IIage
Text+VoT
Peg has disks 2, 1, and O. Pegs and C are empty.
Map (Top-Down View):
Row1:
Row2:
Row3:
Row4:
R B M
A Agent
R Robot (message; secret document)
B Ball (goal; orb)
M Mage (enemy; "wizard")
Figure State representation variants evaluated our study. The design space is organised along three axes: gramularity (long-form trajectories vs. summaries), structure (natural language vs. concise symbolic encodings), and spatial grounding (text-only vs. explicit grounding via images structured text). These axes capture key prompt-level design choices that influence how the environment state is presented to the model, enabling systematic study of how trajectory compression, structural constraints, and spatial grounding affect long- horizon decision-making
remains contested (Mitchell 2025). Crucially, static benchmarks decouple reasoning from interaction. They not test whether a model can revise its beliefs and decisions as the environment changes response to actions, which core requirement. of autonomous agents (Wong el al. 2025).
In sequential decision-making, it well established that agent performance depends on the quality of the state representation, and that poorly designed encodings can impair decision making and control (Lesort et a.. 2018 Dynamic LLM benchmarks instantiate the same representational problem: the agent must repeatedly condition its actions on an evolving state expressed through text. Yet, despite the growing use of LLMs as controllers, these representational trade-offs have received little systematic study.
At inference time, L.IM-based agents receive the environment. state through an externally constructed prompt. Although dcscriptive natural language aligns with common pretraining data, introduces three distinet challenges interactions lengthen. We discuss (1) granularity, (2) structure and (3) modality of the prompt representation.
The first challenge is verbosity. Natural language descriptions rapidly expand the token count of the prompt, causing performance to suller when critical information is lost in the middle of long contexts, phenomenon where models fail to uniformly attend to all parts of a lengthy input (Liu et al. 2024). To mitigate this, previous work has employed context compression via summarisation (Li et al.) 2023). However, for summari- sation to be effective, task-critical information must remain preserved; in dynamic, morghorizon settings, omitting distorting relevant details can alter downstream decisions and compound errors over future steps.
2

---

Published in Transactions on Machine Learning Research (04/2026)
The second challenge is ambignity and lack of structure. Natural language is expressive but often imprecise for state tracking: values may be implicit, ordering may vary, and the same state can be paraphrased in many ways. This forces the model to infer how the environment changed. One alternative is to impose encodings (e.g., dictionarics, lists, or matrices) that make state variables explicit and easier to updale consistently after each action. For instance, consider the Tower of Hanoi state described in text as 'Peg A has disks 2, I, and 0 on it, from bottom top. Peg B and Peg C are both emply"; ; this can be encoded as a dictionary, 'A': [2,1,0], 'B": []. 'C": B] Such representations can improve planning in some settings (Hu et al., 2024). though evidence for consistent benefits across tasks and model families remains mixed (Sullivan & Elsayed 2025).
structure through symbolic,
The third challenge is spatial grounding. When environments require spatial reasoning, natural language may obscure geometric relations under verbose descriptions. While multimodal LLMs can consume visual inputs (Masry et al., 2022 Lu et al. 2023 Yue et al., 2024), spatial reasoning remains underexplored (Shiri) et al. 2024) and gains are inconsistent (Paglieri et al. 2025 Yuc al.) 2024). "Visualization-of "haiaiiiattoooof-Toonnng"" Thought" (VoT), a prompling method that elicits spatial reasoning by having models visualize intermediate reasoning traces, suggests that pixel inputs are not always necessary: structured text layouts such as ASCII grids can be sufficient for spatial tasks (Wu et al. 2024a).
Together, these challenges motivate systematic investigation of how prompt-level state design influences hori-norizon control: whether compressing trajectories, imposing structure, or adding spatial grounding improves performance, and under what task and model conditions these choices help or hurt. Therefore, we ask the following research question:
How does inference-time state representation, along the ares of granularity (long-form trajectories versus summaries), structure (natural language versus symbolic encodings), and spatial grounding (lezl-only rep- resentations versus explicit spatial grounding via images or textual map encodings), affect performance on dynamic reasoning tasks?
We evaluate these design choices multi-step, interactive environments where the state evolves in response lo the agent's actions and success requires long-horizon planning, memory; and spatial reasoning, We consider range of open-source LLMs spanning small to large scales, focusing in-context learning with frozen parameters and gradient updates or fine-tuning.
Our main contributions are as follows:
Using a summary of the interaction history, rather than the full history, can improve lori-horizon decision-making by removing noise, helping the agent to focus on task-relovant information. Perfor- mance effects are task- and model-dependent. Summarisation tends to help when the task-relevant state can be captured by compact abstraction, the summary is accurate, and contains key infor- mation for action selection.
Across tasks, natural language is the most robust state representation; when structured encodings help, gains are largely confined to models with stronger exposure to code structured outputs, which can reliably exploit JSON-like schemas, whereas other models fail to correctly interpret the structured encodings.
Adding spatial representation to the prompt can improve performance, and among variants Visual- ization of Thought is the most consistently effective (Wu et al. 2024a). However, these gains depend on sufficient model capability to construct faithful maps. Moreover, we find that VoT primarily helps by inducing step-by-step spatial reasoning rather than by supplying a richer state description.
The rest of our paper is organised as follows. Section 2 provides the background, Section 3 details the methodology, Section presents the results, and Section 5 concludes with discussion of the findings.
3

---

Published in Transactions on Machine Learning Research (04/2026)
2 Background
2.1 State representation
In sequential pecision-making, a state representation specifies how the environment is encoded for a policy, with the goal retaining task-rclovant information while discarding irrelevant variation. In particular, a good state representation should satisly four criteria (Lesort et al. 2018 Böhıner el al. 2015):
1. The state is Markovian: the state summarizes all information necessary for the agent to select an action based on the current state alonc.
2. The state representation contains sufficient information to assess the quality (value) of the current state and judge which actions are likely to lead to successful outcome.
3. The representation supports generalisation to unseen states with similar future outcomes.
The representation remains low dimensional to enable efficient decision-making
Whether a given state encoding actually meets these goals cannot be inferred from its format; must tested by evaluating the performance of a controller that uses (Lesort al. 2018). Since multiple encodings can work for the same task, there is no single best representation; the right choice depends on the task and the controller.
We next summarise state representations used in prior work, organised by textual encodings and spatial grounding.
2.2 State granularity
Prior work in reinforcement learning suggests that compressing observations into language based state rep- resentations can improve generalisation and interpretability, and that representation design can matter even when the model is held fixed (Rahman & Xue 2024 Echchahed & Castro 2025
Closely related to our work, recent research in dynamic routing games studies natural language state rep- resentations along three axes: action informativeness, reward informativeness, and the degree of natural language compression (Goodycar et al.) 2025). The study demonstrates that restricting information gram-- larity improves stability. Specilically, they lind that agents reach game-theoreta equilibrium more effectively when provided with history summaries, regret feedback, and limited visibility of others' actions. However, the study is limited to natural language, & single task, and one language model. In contrast, we study how gramularity, structure, and modality interact across multiple tasks and different model architectures.
2.3 Structure and symbolic representations
A common way to improve reasoning is by eliciting intermediate reasoning steps Pllat et al. 2026 as Chain-of-Thought (Wii et al. 2022b), Self-Refine (Madaan et al., 2023), and Reflexion (Shinn al. 2023 While this can improve performance, it often becomes verbose over long horizons and may add redundancy ambiguity. A complementary line of work uses structured or symbolic encodings instead. Methods such ofain-of-Symbol represent state in symbolic forms, reducing token usage and improving planning in settings Hu et al.] 2024).. Related approaches use symbolic structure for verification or tool use (Chen) et al. 2024b Pan et al.] 2023). or switch between symbolic and natural language formats depending on the
subproblem (Han et al. 2024).
Despite these advances, it remains unclear how trade off structured encodings versus natural language use dynamic environments where the state evolves continuously and the agent must maintain consistent representation across long history of interactions.
4

---

Published in Transactions on Machine Learning Research (04/2026)
2.4 Spatial grounding
Beyond purely textual representations, recent work explores whether grounding state information in percep- tual form can improve spatial reasoning, Vision- language models (VLMs), which accept images alongside text, enable agents to condition decisions on visual input, providing direct mechanism for spatial grounding.
On static vision benchmarkk, adding real images can be beneficial. For example, on MathVista, GPT-4V substantially outperforms its text-only counterpart, indicating that access to raw visual input can improve mathematical reasoning in visual contexts (Lu al.) 2023). However, results are more mixed in dynamic settings: in sequential environments that require repeated state updates and long-horizon planning, con- ditioning on raw images can be ineffective or even harmful, with VLM agents underperforming text-only variants (Paglieri et al. 2025).
Several methods introduce spatial grounding without relying on raw images. Visualization-of Thoughh prompts models to generate structured textual diagrams, such as ASCII grids, that explicitly encode spa- tial layouts (Wu et al., 2024a), while Visual Chain-of-Thoughu extracts salient visual regions and integrates them into textual reasoning trace (Shao et. al. 2024h. Rather than adding sensory input, these approaches externalise the environment's spatial structure in form that remains directly accessible to the language model throughout inference.
Most methods are evaluated in static, single-turn settings, where the model receives a complete instance (e.g.. a full map and instructions) and produces a solution in one pass. While this probes multi-step reasoning, it sidesteps the central challenge of interactive control, where the agent must update its state represcntation after every action as the environment changes. We build on this line of work by comparing image-conditioned and text-conditioned grounding in dynamic environments to understand when spatial grounding facilitates effective sequential decision-making,
3 Methodology
3.1 Markov decision process
We model each environment as a Markov decision process over discrete timesteps E (l...., An episode starts in an initial state 81 and ends when a terminal condition is reached. At each timestep t, the agent receives observation 05; in fully observed settings o 8t, while in partially observed settings o, provides only partial information about St. The agent then selects action a₂ according to stochastic policy T <a; wx), where in our setting æ is instantiated by fixed, pre-trained LLM conditioned on the inference- time prompt zt. After executing a,, the environment returns a scalar reward and transitions to the next state 8t+1 according to the dynamics P(s st, a:). The goal within an episode is to maximise the cunulative reward R R= E Tt, where T denotes the episode horizon.
3.2
State representation
We use the inference-time prompt as the state interface. Because the model maintains no persistent internal state across timesteps, all information required for decision-making must be provided explicitly at each step. At timestep t, we construct a prompt
23 = MM (or,, hy-1); A].
where M is a static task manual, d(or, 1 encodes the current observation together with a history repre- sentation, and A lists the available actions. We keep M and A fixed within an environment and vary only the encoding produced by d.
The current observation o describes the instantaneous environment configuration under a chosen encoding format. Depending on the environment, o, may be fully observed or partial.
We organise the prompt-level state representation design space along three axes: granularity (full history vs. summary), structure (free-form natural language vs. symbolic encodings), and spatial grounding (text-only vs. explicit grounding via images structured text layouts).
5

---

Published in Transactions on Machine Learning Research (04/2026)
Granularity controls how the history is provided to the LLM via he-1. We compare two settings:
LONG FORM: h = h4-1
SUMMARY: P1e-1 = m, m, = (m Ot, (t:
where h = (0); 01, ..,...., 1t-1, (t-1,Tt-1) is the full interaction history and mr is a rolling natural- language summary updated each step by a fixed summarisation procedure & implemented with the same LLM, where the summarisation instruction explicitly requests a summary at most 25 tokens. The summary replaces the full history in the prompt to reduce context length while preserving task-relevant information.
For instance, when have the full past trajectory in the prompt, it might look like below:
Past trajectory:
Step 1: You took action stay. You (agent) don't have the message.
You see:
bird steps away
sword 1 step away
fish stepa away
Step 2: You took action east. You (agent) already have the message. You see:
bird 2 steps away
fish steps away
When summary is enabled, this history is compressed into summary that the LLM updates at each timestep, replacing the full trajectory in the prompt. For the same sequence above, the summary might read:
Summary past actions: Moved east, collected message from sword; bird 2 steps away, fish 4 steps away.
We note that the Summary condition introduces an additional LLM call per timestep to produce m. While the input context to the agent shorter (since m,-1 replaces h--1), the total tokens consumed by the system increase once summarisation tokens included, since the SUMMARY condition adds extra L.I.M call that takes the full history 1 (and the previous summary m,-1) input to produce the compressed state my
Structure specifies the state is encoded in the prompt. On the one hand, we consider free-form natural language descriptions. On the other hand, we consider structured encodings, including key-value dictionaries, matrices, grid-based layouts, and hybrid formats. Structured representations constrain how information is organised, which can reduce ambiguity and make parsing easier, but may limit expressive flexibility. Natural language representations retain semantic detail and contextual nuance, but leave structure implicit and roquire the model to infer it. For instance, Hanoi can be represented via natural language like:
Peg has disks 2, 1, and 0 (botton to top). Pegs and C are empty.
or in matrix form like:
[[2,1,0], [-1,-1,-1],
[-1,-1,-1]]
Spatial grounding determines whether the state is provided purely text augmented with explicit spatial grounding signals. In addition to text-only representations, we consider inputs that include ren- dered images of the environment as well as textual visualisations that encode spatial layouts directly, via Tisualiattion-of-Thouug. where ASCII sketches are generated as part of the agent's reasoning process (Wu et al. 2024ah. These variants differ in whether spatial structure is conveyed through an image or through structured text, and may interact differently with model protraining and capacity.
For instance, the state in Hanoi might be represented purely via text::

---

Published in Transactions on Machine Learning Research (04/2026)
Table 1: Environment specifications for all experiments. We report task difficulty, history length (number of past steps maintained in context), rollout length (episodes per run), action-space size, and maximum timesteps per episode. Runs are reported as S/M(L), where S and M denote the number of runs for small and medium models, respectively, and the value in parentheses denotes the number of runs for large models. In our setup, ALAMA3.3-70% and ----3-----3BB--SSTTEII models are classified as small or mid-sized.
are classified large models; the remaining
Environment
Difficulty
History
Rollout
Actions
Timesteps
Runs
Hanoi
Medium
30
10
30
10 (5)
Messonger
Hard
10
20
10
10 (5)
A--yAI-GoTo
Easy
128
10
128
25 (10)
AT-Ope
Medium
128
10
128
25 (10)
Baby Al-byAl-iccke Al-PickUp
Medium
128
10
128
25 (10)
Hard
128
10
128
25 (10)
Baby Ne-PutNext
Hard
128
10
128
25 (10)
Peg has disks 1, and Pegs and C are empty.
or via textual "visualisation of thought" that makes the spatial layout explicit, where the numbers refer to the size of the disks.
Maр (Top-Down View):
Rod A: [2, 1, 0] (top is 0, bottom is 2)
Rod B: 0
Rod C: 0
Spatial grounding can additionally be provided through rendered images of the agent's partial observation. In this vision-language setting, the prompt includes both the textual state description and an image depicting the current view, allowing models to recover spatial relations directly from images rather than from symbolic text alone.
! provides illustrative examples for each axis. Full prompts and examples are provided in the Ap- pendix.
Figure
3.3 Test environments
We evaluate three dynamic environments: Tower of Hanoi and Messenger from SmartPlay (Wu et al, 2024b), and AI from BALROG (Paglicri et al. 2025). All environments require multi-step planning and spatial reasoning. Tower of Hanoi requires moving three disks across three pegs while respecting the constraint that no larger disk may be placed on a smaller one. Messenger is a grid-world task where the agent must pick up message and reach a goal while avoiding an enemy, Entities are described using synonyms to test semantic robustness. BabyAl consists of five navigation and interaction tasks of increasing difficulty (Chevalier- -Chevelier-Boisvertt et al. 2019 Carta et. al.) 2023). The agent operates in 2D grid-world and follows natural-language missions, ranging from simple navigation (GoTo) and door interaction (Open) to object manipulation (PickUp, PutNext) and short compositional instructions (PickUpSeqGoTO).
BALROG provides both text and image observations, whereas SmartPlay text-only. To support our spatial grounding experiments, we implemented an image renderer for SmartPlay that produces visualisation of each state.
3.4 Experimental setup
We evaluate set of open-source LLMs to study how model scale and architecture affect performance. The models were selected to cover range of parameter scales and architectural families, including both

---

Published in Transactions on Machine Learning Research (04/2026)
text-only and vasion-language models. This allows to examine whether the observed representation effects are consistent across heterogeneous models rather than being specific to single architecture. For text-only representations, we use Pm14-14B, LLAMA3. 1-8B, DEEPSEEK-RI-14B, and 1LLMA3.3-7000 For multimodal representations (vision + text), we use QWEN2.5VL-77 LLAVA-PHI'S 8B, LLAVA-7B, and QwEN3-VI-33R-- -1----1----------CCC We run the largest models, LLAMA3.3-70B and QWEN3- (1-332-IINTRUUCT on two machines equipped with NVIDIA A100 GPUs; the remaining models run on NVIDIA RTX 4090, RTX 3060, or A40 GPUs. All experiments use temperature 0.2 with nucleus sampling (top p 0.95) to reduce stochastic variability while avoiding fully deterministic decoding.
3.5 Scoring
Performance is evaluated using the task-specific episode score at the end of evaluation episode e. In Tower of Hanoi, & is the number disks placed on the goal peg, taking values [0,3]. In Messenger, & is the episode reward, which lies in [-1,1]. In Baby 8e binary, with indicating successful task completion and 0 otherwise.
For each run, average scores over E evaluation episodes,
E
8run
8e1
E
where is the number of evaluation episodes per run. We then report the mean over R runs,
Srun,r:
where indexes runs and Brun,r denotes the mean score of rum
To enable comparison across environments, we linearly rescale the mean score ă to normalized score E [0. 1]. where O denotes the worst outcome and 1 denotes suecessful task complction. For Tower of Hanoi, we use = 8/3, and for Messenger, = (3 + 1)/2. Baby scores already lie in [0, 1], so no additional normalization is required. Standard deviations are transformed using the same linear scaling.
For all comparisons against the baseline, we assess significance using a run-level bootstrap test of the mean difference. For each run, first average the benchmarksspecffcc episode scores obtain a single run-level score. We then generate 10,000 bootstrap replicates by resampling these per-run scores with replacement and compute percentile 95% confidence intervals for the mean difference, and significance markers are based on correccr two-sided bootstrap p-values.
4 Results
We present results for each of the three state-representation axes introduced in Scction 3.2 gramularity (full trajectory vs. summary), structure (natural language vs. symbolic encodings), and spatial grounding (text-only vs. images textual map encodings). For each axis, report performance, highlight recurring patterns and failure modes, and conclude with an overall summary of the main findings.
4.1 Granularity
We study how the granularity of the historical trajectory in the agent prompt affects performance. For these experiments, we use SmartPlay's default state representations: ToggedList for Tower of Hanoi and NaturalLanguage for Messenger (sce Appendix for the format). In Long Form, the prompt includes the full interaction history h--1, whereas in SUMMARY the history replaced by a rolling summary m:-1 updated at each time step.
8

---

Published in Transactions on Machine Learning Research (04/2026)
Table 2: Performance under Long Form versus Summary prompting on Tower of Hanoi and Messenger across open-souree LLMs. Entries report mean normalised score + SD over runs (n=10 seeds per model- environment setting, except LLAMA3.3-70F with n=5), with scores scaled to [0. 1] where 1 denotes task completion. A superscript marks a significant difference from the baseline (Long Form) for the same model and task, using run-level bootstrap lest of the mean difference over per-run scores (10,000 resamples; percentile 95% CT). Two-sided bootstrap p-values are adjusted with the Holm procedure across all pairwise comparisons reported within this table, controlling the family-wise error rate at a 0.05.
Overall, summarisation tends to help Hanoi for mid-to- lo-rge models (notably SWEN2.5VL-77 LLAMA3.1-83 DEEPSEEK-RAL R1-14B, and QWEN3-VL-32B- ----322-IISRRUTT but is less reliable in Messenger, where it produces large gains for some models (e.g.. DREFSEEK-RI-11B and LLAMA3. 1-8B) while degrading others (e.g., QWEN2.5VI-7B and S----VVL------ISTRRCCE
Hanoi
Messenger
Model
Long Form
Summary
Long Form
Summary
LLaVA-PhiB-3.SB
0.16 (0.12)
0.08 (0.10)
0.24 (0.11)
0.14 (0.06)*
LLaVA-7B
0.00 (0.10)
0.14 (0.16)
0.26 (0.10)
0.31 (0.05)
Qwen2.5VL-77
0.08 (0.09)
0.39 (0.00)
0.11 (0.06)
0.05 (0.13)
Llama3.1-50
0.03 (0.09)
0.25 (0.10)
0.20 (0.13)
0.35 (0.07)
Phi4-148 14B
0.37 (0.17)
0.44 (0.11)
0.34 (0.25)
0.44 (0.36)
DeepSeek-RR-11I
0.46 (0.12)
0.70 (0.21)*
0.53 (0.28)
1.00 (0.17)"
Qwen3-VL-32BB -nn3uu--stt
0.81 (0.07)
0.98 (0.03)
0.09 (0.09)
0.00 (0.13)
Llama3.3-70B
0.66 (0.47)
1.00 (0.00)
0.28 (0.19)
0.22 (0.05)
Table 3: Performance under Long Form vs. SUMMARY prompting on five Baby tasks (multimodal models only). Entries report mean normalised score + SD over runs (n=25 seeds per model task setting, except QWEN3-VL-32B-B VL-32B-INSTRUCT with z=10), with scores scaled to [0, 1] where 1 denotes task completion. A superscript marks a significant difference from the baseline (Long Form) for the same model and task, using a run-level bootstrap test of the mean difference over per-run scores (10,000 resamples; percentile 95% CI). Two-sided bootstrap p-values are adjusted with the Holm procedure across all pairwise comparisons reported within this table, controlling the family-wise error rate at 0.05.
Overall, summarisation most reliably helps in simpler navigation tasks (e.g., Goloo and can improve performance in compositional tasks (e.g., PickUp and PSekpPSe but offers limited benefit in harder tasks (Open and PutNext).
GoTo
PickUp
PickUpSeqGoTo
Open
PutNext
Model
Long Form
Summary
Long Form
Summary Long Form Summary Long Form
Summary
Long Form
Summary
LLaVA-PhiS-3.8B
0.28 (0.46) 0.52 (0.51) 0.01 (0.20) 0.12 (0.33) 0.08 (0.28) 0.08 (0.28) 0.00 (0.00)
0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
MLaVA-7B
(0.46)
0.44 (0.51)
(0.33) 0.20 (0.41)
(0.33)
0.16 (0.37)
(0.00)
0.00 (0.00)
(0.00)
0.04 (0.20)
Owe-2.5VL--BB
0.88 (0.33)
0.96 (0:20)
0.24 (0.44) 0.24 (0.44) 0.16 (0.37)
0.32 (0.48) 0.00 (0.00) 0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
Qwen3-VVL.327 -nntect
0.60 (0.52) 0.60 (0.52) 0.00 (0.00) 0.10 (0.32)
0.10 (0.32)
0.30 (0.48)
0.80 (0.42) 0.40 (0.52) 0.00 (0.00)
0.00 (0.00)
4.1.1 Hanoi
Summarisation is beneficial for most models in Tower of Hanoi (Table 2 With the exception of the smallest model (LLAVA-PA) P.BB-3.8B, 3.8B, which degrades under compression from 0.16 to 0.08), all larger models improve under Summary, with gains increasing with model scale. For example, QwEN2.5VL-7B improves from 0.08 from 0.46 to 0.70*, and SWEN3-VL-332--INee
from 0.81 to 0.98*. The largest model, LLAMA3.3 3-B., reaches perfect task completion under Summary, improving from 0.66 to 1.00. This pattern is consistent with Hanoi admitting a compact abstraction: once the current disk configuration is preserved, earlier trajectory details are largely redundant, so summarisation primarily removes noise while retaining the task-critical state.
to 0.39*, PPePSEEK-RI-IA
9

---

Published in Transactions on Machine Learning Research (04/2026)
4.1.2 Messenger
Table 2 reports the Messenger results. Unlike Hanoi, the effect of summarisation is mixed: some models benefit substantially, while others are unchanged or degrade. In particular, DEEPSEEK-R1-111 achieves a perfect score, improving from 0.53 to 1.00", LLAMA3. 1-8B from 0.20 to 0.35", and Pm14144B from 0.34 to 0.44. In contrast. several models regress under SUMMARY, including QWEN3 -T---V-T--2R--STTRRUU (from 0.09 to 0.00) and LLAMA3.3- 70B (from 0.28 to 0.22).
The summarisation failures largely depend what information is retained. Models often produce generic summaries. For example, 1LAMA3.1-81 writes "Agent has not obtoined the message yet, and there are entities nearby (scientist, ship, robot)", while omitting critical details such as relative distances, outcomes of recent actions, or progress toward subgoals. Failures also arise from reasoning limitations, such as choosing to "do nothing" even when the message holder is one step away. Qualitative traces further indicate that the benefit of summarisation is model-spocific and depends on whether the model can maintain a coherent internal state. For instance, with DEEPSERK-RI-14D on Messenger, adding summaries stabilises the trajectory: without summaries the agent oscillates ("Move North - Move West Move East Move West") and finishes with score 0, whereas with summaries (e.g., "Agent moves north toward the dog to get the message") ") the same model reaches score 1 in short rollout.
Overall, summarisation shows more mixed results in Messenger. While a summary can help by focusing on the important parts, it causes the agent to fail leaves out any of the critical details, and it cannot compensate for the model's reasoning limitations.
4.1.3 BabyAl
Table 3 shows the Baby results for multinodal models. Summarisation is most beneficial in navigation- dominated settings, while gains diminish as tasks require more precise instruction following and relational reasoning. In GoTo, weaker models can improve under Summary (e.g.. LLaVA-PhiB-3.BE 0.28 to 0.52), but this effect is not uniform: LLaVA-7B collapses under Summary (0.72 to 0.44). By contrast, larger models achieve good performance under Long Form and reccive little benefit from Summary (c.g., Qwen2.5VL-7B: 0.88 to 0.96;
0.60 to 0.60).
In PickUp, where the agent must navigate to a specified object and pick it up, summarisation provides small gains for weaker models (c.g., LLaVA-Phis-3.. 0.04 to 0.12; LLaVA-7B: 0.12 to 0.20), but does not help stronger models (Qwen2.5VI--7B 0.24 to 0.24). This consistent with summaries helping mainly by retaining the target identity and approximate location during exploration, rather than improving the interaction itself.
In the Gooo task, summarisation yields modest but consistent improvements (e.g., LLaVA-7: 0.12 to 0.16; Qwen2.5VL-7B: 0.16 to 0.32), while LLaVA-Phi3-3.SB remains unchanged (0.08 to 0.08).
In contrast, in harder settings, summarisation offers limited benefit and can even harm performance. In degrades under
Open, most models remain at regardless of representation, and VWenn--V--32B--nstree Summary (0.80 to 0.40), suggesting that compression can be harmful when success depends on precise instruction execution. Similarly, PutNext remains near zero across all models and both representations, indicating that summarisation does not resolve the underlying demands of the task. Overall, summarisation helps most when the task state can be captured as simple spatial progress, but provides limited value instruction complexity and relational constraints increase.
Qualitative inspection suggests that task success depends on the summary correctness. High-quality sum- maries act as efficient memory by preserving goal locations and progress, whereas weaker models sometimes hallucinate state updates (e.g., reporting that an object was picked up when it was not), which can trigger premature termination because the agent incorrectly believes (sub)goals have already been completed.
10

---

Published in Transactions on Machine Learning Research (04/2026)
Table 4: Performance under Long Form, SUMMARY and ORACLE SUMMARY on Tower of Hanoi. Entries report mean normalised score SD over 5 runs, with scores scaled to [0, 1] where denotes task comple- tion. ORACLE SUMMARY provides the summary extracted programmatically rather than produced by an LLM. A superscript marks a significant difference from Long Form for the same model and environment, using bootstrap lest of the mean difference over per-run scores (10,000 resamples; 95% CI); to address multiple comparisons, p-values are adjusted with the Holm- Mol--Boffrrron method (a = 0.05) across all pairwise comparisons, controlling the family-wise error rate.
Overall, LLAMA3.1-81 and HEEPSEEK-RR-L44 perform best under ORACLE SUMMARY, indicating per- formance loss from imperfect LLM-generated summaries, whereas LLAVA-PHI collapses under ORACLE SUMMARY, suggesting limited ability to act compressed state interface even when the summary is correct.
Environment
LLaVA-Phi8
LlaMA311-80
DeepSeek-R11 RI-144
Long Form
Sumonary
Oraele Summary
Long Form
Summary
Oruele Summary
Long Form
Summary
Oracle Summary
Hanoi (3-disk)
(0.08)
0.16 (0.02)
0.00 (0.00)" 0.15 (0.03) 0.23 (0.04)
(0.01)
0.35 (0.08)
(0.10)
0.57 (0.08)
4.1.4 Ablation: summary oracle
The Summary condition requires the model to summarise its own interaction history, which conflates two factors: whether a compressed state an effective interface for control, and whether the model can reliably produce such summary.
To disentangle these effects, introduce an Oracle Summary ablation on Tower of Hanoi, where the enviromment admits an exact Markov state description: a summary that specifies the current configuration is sufficient for action selection as the optimal next move depends only on the present dig-disk arrangement, not on the trajectory history. In the standard Summary condition, the rolling summariser converts this structured state into a natural-language description (e.g.: "Peg has disks 3, 2, and from boltom to lop; pegs 2 and are empty"). In Oracle Summary, we bypass LLM summarisation and provide the acting agent with the ground-truth current configuration and goal directly from the simulator. This experiment isolates whether performance gains from rolling sunmaries arise from presenting state information in more usable form, or whether ALM-generater summaries introduce noise that degrades control.
Table shows that the benefit of compression is model-dependent. For LLAMA3. 1-8B, performance improves from 0.15 Long Form to 0.23 Summary and further to 0.27 Orucle Summary. DEEPSEEK-RI-133 shows the same pattern, improving from 0.35 to 0.49 and to 0.57. These gains indicate that both models can exploit perfectly compressed state, and that the gap between Summary and Oracle Summary is attributable to information loss or distortions introduced during summary generation.
In contrast, L.LAVA-PHI does not benefit from summarisation: it drops from 0.20 Long Form to 0.16 Summary and collapses under Oracle Summary (0.00). This suggests a different limitation: even when the summary is accurate, some models struggle to interpret and act on it.
4.1.5 Cross-tase comparison
Across tasks, Fig. 2 shows that Summary typically improves score per inpul loken by keeping the acting prompt short, which reduces long- horizon prompt noise and focuses the agent on task-relevant state.
Performance effects, however, are task- and model-dependent. Summarisation tends to help when the task- relevant state can be captured by a compact abstraction, the summary is faithful and accurate, and contains all information needed for action selection, which is more likely in simple navigation tasks than in settings that require tracking relations and completing multi-step subgoals. Finally, in the most challenging environments, differences between Long Form and Summary become difficult interpret: poor performance reflects task- level capability bottleneck rather than representational trade-off.
11

---

Published in Transactions on Machine Learning Research (04/2026)
Normalized Score per Input Token
Hanoi
Messenger
Long form
Long form Summary
0.030
Summary
0.025
0.20
0.020
0.15
0.015
0.10
0.006
0.05
0.00
LLAVA-78
Liamad 1-88
Phi6-148
DeepSaoke 148
Chwwrd AL -228
Aamm833-708
LLAV--7
Quen WVL-250
Liamad, 1-88
PNA--148
DeepSeek 148
Dwwend-Aiss -29
Llamred 3-708
Qwenn.5VL
(a) Tower of Hanoi and Messenger.
Normalized Score per Input Token
GoTb
Open
PickUp
PickUpGoTO
PutNect
0.05
0312
0.933
0100020
0332
0.005
0300115
0.04
0334
0300
0300
0.00
(b) Baby AT tasks.
Figure 2: Normalized score per input token for Long Form vs. Summary prompting. Bars report normalised score per input token (x1000), computed as the episode score (normalised to 0-1) divided by the average input prompt tokens. Higher values indicate better performance per token spent. The Long Form condition provides the acting agent with the full interaction history at each step, whereas Summary replaces this history with a rolling summary intended preserve task-relevant state while reducing context length. Empty bars indicate settings where the agent achieves zero score. Token counts include only the input prompt shown to the acting agent. The Summary condition additionally requires separate summarisation call, those tokens are excluded to isolate the effect of input context length on decision quality (i.e., are not measuring end-to-end efficiency here.)
4.2 Structure
We next study how the structure of the prompt-level state representation aflects agent performance. We treat NaturalLanguage as the reference format and compare it to symbolic encodings, ranging from semi- structured encoding (c.g., tagged lists or natural-language augmented with coordinates) fully symbolic formats (e.g., dictionaries or symbolic grid layout). All results are reported in Table examples of each format provided in the Appendix.
12

---

Published in Transactions on Machine Learning Research (04/2026)
Table 5: Performance under alternative state structures on Tower Hanoi and Messenger across open- source LLMs. Entries report mean normalised score + SD over runs (n=10 seeds per rode-treprtsennttionn setting, except LLAMA3.3- 70B with n=5), with scores scaled to [0, 1] where 1 denotes task completion. A superscript marks significant difference from the NaturalLanguage bascline for the same model and environment, using a bootstrap test of the mean difference over per-run scores (10,000 resamples; 95% CI); to address multiple comparisons, p-values are adjusted with the Holm Bonferroni method (a = 0.05) across all pairwise comparisons, controlling the family-wise error rate. In Hanoi, dictionary-style structure (DictList) can help models that reliably interpret the schema, while grid-like Matriz encodings are often brittle and Trequently degrade performance. In Messenger, structure offers limited upside and symbolic encodings lend to hurt, suggesting that preserving linguistic cues is critical for maintaining entity state and resolving synonym variation. A notable exception is PREPSEEK-RI-IAR which benefits from explicitly providing the agent's own position (NaturllanguuggePoss) that uses this information to guide its actions, whereas other models cannot.
Hanoi
Model
NaturalLanguage
TaggodList
Matrix
DictList
Structure
Natural Language
Hybrid
Symbolic
Symbolic
LLaVA-Phi3-3.8B
0.25 (0.05)
0.21 (0.06)
0.24 (0.08)
0.31 (0.09)
LLaVA-7B
0.12 (0.04)
0.12 (0.05)
0.00 (0.00)
0.01 (0.02)"
Qwen2.5VL-7B
0.33 (0.00)
0.16 (0.05)
0.33 (0.00)
0.33 (0.00)
Llamma3118B
0.13 (0.07)
0.13 (0.05)
0.03 (0.03)
0.08 (0.07)
Phi4-14B
0.33 (0.10)
0.32 (0.09)
0.36 (0.09)
0.33 (0.08)
DeepSeek-R1-14B
0.33 (0.12)
0.37 (0.07)
0.31 (0.10)
0.45 (0.18)
Llama3.3-70B
0.67 (0.00)
0.48 (0.26)
0.22 (0.08)
0.27 (0.04)
Messenger
Model
NaturalLanguage
NaturalLanguuaeePPr
Coordinates
Symbolic
Structure
Natural Language
Hybrid
Hybrid
Symbolic
LLVVA-Phi-...B
0.43 (0.06)
0.23 (0.06)
0.42 (0.04)
0.36 (0.05)
LLaVA-7B
0.44 (0.05)
0.40 (0.04)
0.42 (0.06)
0.29 (0.08)
Qwen2.5VT.-7B
0.36 (0.03)
0.39 (0.08)
0.38 (0.05)
0.31 (0.10)
Llama3. 1-8B
0.41 (0.07)
0.39 (0.06)
0.42 (0.05)
0.41 (0.06)
Phi4-144 14B
0.48 (0.13)
0.39 (0.03)
0.43 (0.09)
0.40 (0.09)
DeepSeek- Ree-eee--1114BB R1-14B
0.58 (0.15)
0.75 (0.10)*
0.44 (0.09)
0.39 (0.09)*
0.45 (0.10)
0.37 (0.07)
0.39 (0.05)
0.40 (0.05)
4.2.1 Hanoi
In Tower of Hanoi, state structure can matter, but its value is highly model-ddeendeett. Relative to Nat- uralLanguage, symbolic encodings make peg disk assignments explicit and can reduce relational inference, benefiting models that reliably map syntax to task constraints.
The semi-structured TaggedTast format yields mixed effects: performance drops for QWEN2.5VI.-77 (0.33 to 0.16*), leaves LLAMA3.1-88 unchangod (0.13 to 0.13), and produces only small changes for LLAVA- and DEEPS This suggests that hybrid list synlax with natural-languaee cues does not consistently align with the patterns models exploit.
Dictionary encodings, cquivalent to a JSON representation, can help when the model reliably grounds symbols to task constraints. Under DislLisl, ALAVA-PH13-3.80 improves (0.25 to 0.31) and DEEPSEEK- R1-14B improves from (0.33 to 0.45). Qualitative analysis indicates that successful agents exploit explicit structural cucs-sude as empty lists (D) to rule out illegal moves before planning. However, other models drop in performance: LLAVA-7B collapses (0.12 0.01*) by repeatodly selecting moves from empty pegs, and LLAMA3.3- 70B degrades markedly (0.67 to 0.27*), suggesting failures to interpret the JSON-like schema.
13

---

Published in Transactions on Machine Learning Research (04/2026)
Matriz encodings are the least reliable and often harmful (LLAVA-775 0.12 to 0.00"; LLAMA3.1-8BB 0.13 to 0.03"; I..AMA3.--70F: 0.67 to 0.22*), consistent with an added decoding burden and recurring misreads (e.g.- treating -1 as a valid disk instead of a padding value). For example, in Hanoi under Matrix, LLAMA3.1-8P repeatedly proposes moves from the empty rod B ("Action: 4... move top of rod B to rod C*).
Overall, Naturallanuuage remains the most robust choice across models, and the best-performing large model (LLAMA3.3-701) achieves its top Hanoi score under NaturalLanguage (0.67). Structured formats can yield gains for specific model families (notably DictList for DEEPSEEK-R1-14B) but benefits not universal, implying that representation design should be matched to model-specific strengths.
4.2.2 Messenger
In Messenger, the impact of representational structure is weaker than in Hanoi, and symbolic encodings are largely ineffective. Unlike Hanoi, which is strictly rule-governed, Messenger requires maintaining entity references, synonyms, and relationships over time. As a result, representations that abstract away linguistic cucs can discard the semantic information models rely on to infer roles and intent.
Naturallanguagee adds the agent's own coordinates to the natural language prompt. The effect model- dependent: DEEPSEEK-RL- most others change modestly, suggesting that added spatial detail can reduce ambiguity for higher-capacity models but distract smaller ones.
improves (0.58 to 0.75*), while LLA VA-Pm13-3.8) drops (0.43 to 0.23*);
The fully Symbolic encoding typically degrades performance (LLAVA-PH3-33.8)) 0.43 to 0.36; LLA VA-7B: 0.44 to 0.29"; BEGGSSEE-RR-1444 0.58 to 0.39*), consistent with qualitative failures to map abstract state variables back onto the manual-defined entity roles. Agents fail recognize which object corresponds to the enemy, the message, and the goal, lcading to incorrect role assignments and downstream action choices.
Coordinates is generally closer to the NaturalLaguugge baseline (e.g., LAAVA-PP1--... LLAMA3. 1-8B: 0.41 to 0.42) but can still hurt (DEEPSEEK- R.1.-4.B111BBB 0.58 to 0.44). Qualitative analysis reveals that deriving directionality from raw coordinates remains difficult for LLMs. We find that agents often fail to recognize the correct trajectory and instead select actions that increase the distance to the goal.
0.43 to 0.42;
In the Messenger environment, models that rely on explicit reasoning traces often perform best when obser- vations are expressed Natuvallanguage. For example, given the description "You see bird two steps west, a fish two sleps north, and a sword Iwo sleps south", ", DEEPSREK-RI-14D correctly identifies the sword as the message and reasons that moving south will reach it. The textual description directly exposes entity names and relative spatial relations, allowing the model to reason over the environment in way that resembles natural language instructions.
4.2.3 Cross-task comparison
Across both tasks, NaturalLanguage remains the most robust representation. It is consistently competitive across model families. That said, when structure helps, it is most reliably provided by DictList, In Hanoi, DictList can reduce relational inference by making constraints explicit, which some models exploit to avoid illegal moves. These interactions appear correlated with a model's ability to treat the prompt as a structured data interface. Models with stronger evidence of code or structured-output training (e.g., DEEPSEEK-RI- 14B (Guo et al., 2025 and QWEN2.5VL-7B (Yang et al.) 2024)) are generally more robust to JSON-like schemas, while others degrade under the same format, suggesting schema. interpretation failures rather than task difficulty. Technical reports for the Qwen2.5 family emphasize coding, structured data extraction, JSON-style outpuls, and lool use (Yang et al.) 2024), which may make Qwen-based models better able to interpret prompts such as DictList, Consistent with this view, DEEPŠEEK-RAA -1-1E----11BBB... distilled from Qwen2.5, may inherit similar structured-output priors. By contrast, documentation for Llama 3.1 (Grattafiori et al. 2024), and LLaVA (Luu al., 2023) emphasizes dialogue, reasoning, or multimodal instruction following rather than explicit schema-oriented outputs.
14

---

Published in Transactions on Machine Learning Research (04/2026)
Table 6: Effect of spatial grounding variants on Tower Hanoi, Messenger, and BabyAl across multimodal models. Entries report mean normalised score + SD over runs, with n=10 seeds for Hanoi and Messenger (except QWEN3-VL-32BB-INSTHE
with n=5) and n=25 runs per BabyAl configuration (except QWEN3- VL--22BINSTTRUCT with n=10), and scores scaled to [0, 1] where 1 denotes task completion. A superscript marks a significant difference from the Summary (text-only) baseline for the same model and environment, using a bootstrap test of the mean difference over per-run scores (10,000 resamples; 95% CI); to address multiple comparisons, p-values are adjusted with the comparisons, controlling the family-wise error rate.
method (a = 0.05) across all pairwise
We compare this bascline to augmentations that add either an image (Vision) or structured basod spatial map (VoT). Shading is normalised within each game family. Vision is generally inconsistent and often degrades performance, suggesting that images rarely provide useful additional grounding beyond the text state. In contrast, Vo more frequently improve results, especially in navigation-focused Baby AI tasks, though gains are not universal and depend on the model's ability to construct and use the map reliably.
Environment
LLaVA-Phi--3.a0 Qwen2.5VL-7B
LLaVA-7B
Qwen3 -nstruuttttuctt
Hanoi Summary
0.16 (0.05)
0.33 (0.00)
0.19 (0.09)
0.65 (0.02)
Hanoi Summary + Vision
0.11 (0.05)
0.33 (0.00)
0.25 (0.10)
0.47 (0.22)
Hanoi Summary + VoT
0.18 (0.04)
0.38 (0.05)
0.39 (0.05)
0.60 (0.18)
Messenger Summary
0.38 (0.03)
0.33 (0.07)
0.47 (0.03)
0.30 (0.07)
Messenger Summary + Vision
0.37 (0.04)
0.38 (0.05)
0.48 (0.04)
0.33 (0.09)
Messenger Summary + VoT
0.38 (0.05)
0.37 (0.06)
0.38 (0.04)
0.33 (0.03)
AlGoTo Summary BabyAAGooT Summary + Vision
0.52 (0.51)
0.96 (0.20)
0.44 (0.51)
0.60 (0.52)
0.36 (0.50)
0.96 (0.20)
0.72 (0.46)
0.40 (0.52)
AIGTT Summary VoT
0.12 (0.33)
0.60 (0.50)
0.56 (0.51)
1.00 (0.00)
Baby:AOOpen Summary
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.40 (0.52)
AlOpenOpen Summary Vision Baby Summary + VoT
0.08 (0.19)
0.00 (0.00)
0.00 (0.00)
0.30 (0.48)
0.04 (0.20)
0.32 (0.48)
0.04 (0.20)
1.00 (0.00)
Baby A/PickUpSeeGGool Summary
0.08 (0.28)
0.32 (0.48)
0.16 (0.37)
0.30 (0.48)
Baby Alahy-APPicklle UpSeik Summary + Vision
0.08 (0.28)
0.48 (0.51)
0.24 (0.44)
0.60 (0.52)
ABbyyAAPPccUUpSecCooTT Summary + VoT
0.00 (0.00)
0.32 (0.48)
0.20 (0.41)
0.70 (0.48)
BabyAIPickUp Summary
0.12 (0.33)
0.24 (0.44)
0.20 (0.41)
0.10 (0.32)
Baby A AlPickUp Summary + Vision
0.12 (0.23)
0.28 (0.46)
0.12 (0.33)
0.30 (0.48)
Baby AIPickUp Summary + Voll
0.00 (0.00)
0.68 (0.48)
0.24 (0.44)
1.00 (0.00)
Baby Next Summary
0.00 (0.00)
0.00 (0.00)
0.04 (0:20)
0.00 (0.00)
Baby Summary + Vision
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
BabyAIPutNNe Summary VoT
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.10 (0.32)
4.3 Spatial grounding
We finally examine whether augmenting the state with additional spatial information improves in-context decision-making For these experiments, we use SmartPlay's default state interfaces: TaggedList for Tower of Hanoi and NoturalLanguage for Messenger (see Appendix for examples), and the summarised historical trajectory. We treat this text-only prompt as the baseline and compare to two spatially grounded variants, the first is adding an image of the state, and the second is adding text-based spatial map generated via VoT. Table reports the normalized performance and Figure 3 visualises the performance differences relative to the text-only baseline.
4.3.1 Hanoi
Table 6 shows that adding raw pixel imput Vision is often unhelpful and can be detrimental: LLA -A-PH3- 3.8B significantly degrades under Summary+ Vision (0.16 to 0.11), and QWEE3--TL-32B-INSTRRE also drops markedly (0.65 to 0.47). Effects are otherwise weak or inconsistent (QwEN2.5VL-7B: 0.33 to 0.33; LLA 0.19 to 0.25), suggesting that images are mostly unhelpful.
15

---

Published in Transactions on Machine Learning Research (04/2026)
Vo is more beneficial. Summary+ VoT improves mid-scale models (QWEX2.5VL-77B 0.33 to 0.38; LLAVA- 7B: 0.19 to 0.39*) and slightly improves LLAVA-
(0.16 to 0.18). For example, comparing LA- 7B runs with and without the VoT protocol under the same summary setting illustrates this effect. With VoT, the model maintains an explicit intermediate spatial state (c.g. "Map (Top-Doum View)" and "Map Updale Noles") "), producing a coherent sequence of valid moves and reaching a higher final score (2). With- out VoT, responses frequently exhibit action-rationall mismatches. For instance, the reasoning proposes moving a disk from A-B while the selected action corresponds to B-*A, and the episode ends with score O. This comparison suggests that explicitly externalising the state can improve alignment between the model's reasoning and the executed action.
However, VoT is not uniformly helpful (QWEN3-VL-332B-NNG VL-32B- 0.65 to 0.60), and several models fail to consistently construct a good map, as illustrated by the following example from LLA VA-PH13-3.8B.
Map (Top-Down View):
A
B
c
A
1
1
1
2
B
1
4
5
C
1
7
1
8
Qualitative analysis shows that VoT only helps when the model can reliably generate map that preserves the lask-relevant structure accurately, which is especially difficult for smaller models.
4.3.2 Messenger
In Messonger, spatial grounding yields consistent gains across models. Vision produces small, mixed results relative to the baseline (LLAVA-PHI3-3.8)) 0.38 to 0.37; QWEN2.5VL-7B. 0.33 to 0.38; QWEN3- VI-32BB-INSTRUCT: 0.30 to 0.33), with only a marginal but significant improvement for LLAVA-7B (0.47 to 0.48). Vol' is similarly model-dependent and can reduce performance: while it improves QWEN2.5VL-77 (0.33 to 0.37) and 0.38*) and leaves
(0.30 to 0.33), it significantly degrades LLAVA-7B (0.47 to unchanged (0.38 to 0.38).
Qualitative results show that VoT' often produces unreliable maps that are fabricated or internally inconsis- tent (e.g, VA-BB draws # grids while describing them as both "walls" and the "dangerous airplane," " and uses labels such as as both "east" and an object identifier). In addition, onym errors (c.g., labeling the goal ship as "neutral" while treating other entities as enemies), and even when models state the correct rule ("need the message"), they may still navigate to the goal without it and terminate.
observe frequent role/syn-
4.3.3 BabyAl
In Aa,yAI, the usefulness of spatial grounding depends strongly on task demands. In simpler navigation tasks such as GoTo, VoT significantly degrades QWEN2.5VTL-7B, reducing performance from 0.96 to 0.60*. Yet the same representation can strongly help larger models, with SWEN3-VLL-22B-INSTREE from 0.60 to 1.00 under VoT.
improving
In contrast, tasks with explicit interaction and manipulation demands show clearer gains from structured grounding. In OPEN, Vision provides little benefit and can degrade performance (c.g., QWEN3-VLL-32B INSTRUCT drops from 0.40 to 0.30), while VoT yields large, significant inprovements (e.g., QWEN3-VL-32B- INSTRUCT improves from 0.40 to 1.00"; QWEN2.5V LV-N2.5VL-7B improves from 0.00 to 0.32*). The effect is even more pronounced in PICKUP: Vision is modest (c.g., QWEN2.5VL--78 improves from 0.24 to 0.28), whereas VoT can unlock large gains, improving QWEN2.5VL-775 from 0.24 to 0.68 and from 0.10 to 1.00"
However, this advantage does not carry to all Baby tasks. In PICKUPSEQGoTO,, Vision often the more reliable augmentation (e.g., QWEN3-VL-32B- WLL-2B-INSTRRUE improves from 0.30 0.60), while VoT can be neutral harmful for weaker models (e.g., I.LAVA-PHi3-3668 degrades from 0.08 to 0.00). Finally,
16

---

Published in Transactions on Machine Learning Research (04/2026)
Normalized performance difference with spatial grounding compared to the text only baseline indicates significant difference (bootstrap 95% CI excludes 0)
Hanoi
Messenger
BabyAl Open
0.15
Nomallzer diferrene against thethee tatellre
10P
0.10
969
075
-092
15.00
0.05
10.00
-0011
10.04
10.44
360
-2.1
-0.00
4.31
-025
4.10
-0.50
0.18
OHoid
LLA06-78
Ow 77
LLOW
Deero- -28
Sit-- 78
LLOW--7
BabyAl Pickup
BabyAI PickUpSeqGoTo
BabyAI Putnext
0.5
9.15
Normalozd score aiterence againes the tatzlino
1.0
0.10
0.5
365
12.00
10.24
10.04
-3.5
-0.08
-0.6
Ow
Owen
LLANN-8
Qwerr-V 323
11906-7703011
Dewnl 78
Vision
VoT
Figure 3: Effect of spatial grounding on performance relative to the text-only baseline. We report normalised performance differences from adding spatial grounding via images ( (Vision or text-based spatial maps ( (VoT); positive values indicate improvement and negative values indicate degradation, while near-zero values indicate no measurable change (neutral). A superscript marks a significant difference from the baseline for the same model and task, using a run-level bootstrap test the mcan difference over per-run scores (10,000 resamples; percentile 95% CI). Two-sided bootstrap p-values are adjusted with the Holm procedure across all pairwise comparisons, controlling the family-wise error rate at 0.05. Overall, VoT improves 15/24 instances, degrades in 5/24, and is neutral in 4/24; Vision improves in 10/24, degrades in 6/24, and is neutral in 8/24. Significant improvements are from VoT in Hanoi for LLA VA- in BabyAI OPEN for ML-N3-VT-332-INSSTT
baseline in
and in Baby AT PICKUP for QWEN2.5VT. B and WI-N3--VT-32--INSTWAA VoT also shows a significant degradation mostly degradations in Hanoi for LLAVA-PHI--S. PH13-3.80 and QWEN3- significant improvement in Messenger for LLAVA-7B.
Messenger for LLA LL-7B, For Vision, significant effects are
with only a marginal
PUTNEXT remains near floor across representations, indicating that explicit spatial grounding alone does not resolve the longer-horizon relational reasoning required.
Larger models benefit most from VoT: for GoTo, OPEN, and PICKUP lifts performance a perfect score (e.g,, QWEN3- VL-32B-INSTRUCT, 0.60 1.00; 0.40 to 1.00"; 0.10 to 1.00*), and in PICKUPSBOGa it approaches optimal performance (0.30 to 0.70), suggesting that constructing and exploiting useful spatial map requires sufficient model capacity.
Qualitatively, failures in harder BabyAl lasks follow a small set of recurring patterns: agents atlempt interactions from too far away, struggle to decompose instructions into stable substeps (find + pick up + navigate + place), lose track of held objects or completed subgoals, and fail to maintain procise relative positioning (c.g., "next to"). Overall, spatial grounding helps in Baby AI when it directly supports interaction- level control, but it does not overcome limitations in long-horizon relational planning.
17

---

Published in Transactions on Machine Learning Research (04/2026)
Table 7: Oracle VoT ablation on BabyAlPickUUp. We compare Summary. Summary + VoT, and Summary + VoT Oracle (programmatic ground-truth ASCII map), reporting mean normalised score SD over 25 runs. +VTT improves performance only for models with sufficient reasoning capacity, whereas providing ground-truth maps alone yields consistent gains. This suggests that the benefit comes less from access to spatial information and more from the act of constructing spatial representation A superseript marks conditions that differ significantly from Baseline, using bootstrap test of the mean difference over per-run scores (10,000 resamples; 95% CT); to address multiple comparisons, p-values are adjusted with the Holm-Boo Bonferronrr method (a = 0.05) across all pairwisc comparisons, controlling the lasely-wise error rale.
reasoning scaffold.
Environment
LLaVA Phi3
LLaVA-7B
Qven2.5VI
Baseline
VoT
VoT Oracle
Baseline
+oT
VoT Oracle
Baseline
VoT
VoT Oracle
BabyAI-PickUp 0.12 (0.32) 0.00 (0.00) 0.08 (0.27)
0.20 (0.40)
0.24 (0.43)
0.24 (0.43)
0.24 (0.43)
0.68 (0.47)
0.16 (0.37)
4.4 Ablation: Visualization-on of- Thought oracle
The Visualization-of- offooighh (VoT) condition relies on the LLM to generate structured spatial sketch from observations. This conflates two factors: whether an explicit spatial abstraction is useful for decision- making, and whether the model can reliably construct such an abstraction. To disentangle these effects, we introduce an Oracle VoT ablation in the BABYAI-PICKUP task.
We compare three conditions: (1) I.M--eeerated VoT, where the agent produces its own ASCII map; (2) pracle-generated ASCII maps, constructed programmatically from the true environment state; and (3) the text-only bascline. This comparison isolates whether failures under VoT stem from the usefulness of spatial grounding itself or from errors in map construction.
Across models, providing ground-truth spatial maps yields no consistent benefit over the text-only baseline, whereas LLM-generated VoT can substantially improve performance, but only for models with sufficient reasoning capacity.
In particular, QWEN2.5VL-7B improves markedly from 0.24 (Bascline) to 0.68* under LLM-generated VoT, yet drops to 0.16 when given oracle maps, performing worse than baseline despile access to perfect spa- tial information. LLAVAS VA-BB shows no reliable differences across conditions (0.20 + 0.24 + 0.24), while LLA fails to benefit from cither intervention, degrading from 0.12 (Bascline) to 0.00 under VoT and 0.08 under Oracle VoT.
These results suggest that performance gains do not come from spatial information alone. The benefit of VoT appcars to arise from the process of constructing a spatial representation: generating an explicit map forces the model to reason sequentially about object locations. This effect depends strongly on model capability. Weaker models struggle to generate reliable spatial maps, producing outputs that can be uninformative or wrong. Stronger models use the act of generation to make spatial relationships explicit and improve control. However, perfect spatial maps do not improve performance. This indicates that the main challenge is not having access to the correct map, but performing the step-by-step reasoning understand the state. Therefore, VoT functions as a mochanism to facilitate spatial reasoning rather than a simple description of the state.
In addition, providing oracle map can lead to representation model mismatch. Although oracle maps contain correct spatial information, their specific ASCII layout and symbol conventions may be out-of- distribution for the model as an inpul formal, making them harder to parse and exploit. In contrast, when the model constructs the map itself, it can choose token patterns and conventions that align with its learned priors, and condition on more reliably. Under this view, VoT helps not only by externalizing spatial relations, but by allowing the model to express those relations in a format closer the structured-texx patterns it has seen during training.
18

---

Published in Transactions on Machine Learning Research (04/2026)
4.4.1 Cross-task comparison
As shown in Fig. 3 adding spatial grounding generally improves over the text-only baseline, with VoT yielding more frequent gains than adding images ( (Vision). However, VoT is only reliable when the model is sufficiently capable to construct and use a consistent spatial map; smaller models often produce incomplete or incorrect maps, limiting benefits or causing regressions.
Importantly, the gains from VoT are not simply due to exposing additional spatial information. Providing perfect spatial maps does not improve performance, indicating that the key benefit comes from the act of constructing the map itself. This suggests that VoT functions as a mechanism to elicit step-by-sten spatial reasoning, rather than a static description of the state.
5 Conclusion and discussion
In this study, we examined how the inference- time representation of the state, expressed along the axes of granularity (long-form trajectories versus summaries), structure (natural language versus concise symbolic encodings), and spatial grounding (text-only representations versus explicit spatial grounding via images or textual map encodings) influences the reasoning capabilities of LLMs in dynamic environments.
Trajectory summarisation is the most broadly useful intervention. Replacing the full interaction history with rolling summary often improves long-horizon docision-making by reducing redundancy and removing noise. This also highlights the difficulty current models face in processing long temporal sequences. Maintaining the interaction history presents a trade-off. Long-form trajectories preserve complete information but increase the number of tokens processed at each timestep, leading to higher inference costs and latency. Conversely, summarised trajectories reduce context length and computational overhead, but require an additional sum- marisation steps and risk omitting task-relevant information. The benefit depends on whether compression preserves task-critical details, and summarisation alone cannot lift performance when the underlying task demands exceed the model's capabilitics.
Across tasks, Natural Language remains the most robust representation and is consistently competitive across model families. When structured encodings help, models with stronger exposure to code or structured outputs are generally more tolerant of code-like schemas (e.g., JSON-style formats) and can exploit them reduce relational inference, whereas other models become brittle under the same representations and often fail to interpret the schema reliably.
Adding spat ial information can further improve performance, and among the spatial grounding variants, VoT is the most consistently useful. However, these gains require sufficient model capability to reliably construct a faithful map; weaker models often generate noisy or incorrect skotches. Notably, providing perfect spatial maps does not improve performance over the text-only baseline, indicating that the main challenge is not access to the correct map but carrying out the step-by-step reasoning needed to use the state effectively. Tn this sense, VoT functions primarily as a mechanism that elicits spatial reasoning, rather than as a richer description of the state.
Furthermore, spatial grounding remains a bottleneck that is not automatically resolved by multimodal architectures. Contrary to the common intuition in embodied Al that direct visual access should enable better physical reasoning (Driess et al. 2023 Reed et al., 2022), we find that providing an image often reduces performance relative to text-only baselines. One possible explanation is that. VLMs under-utilise image tokens during generation ("visual neglect") (Chen et al. 2025). making pixel inputs a noisy and weakly-informative state interface in long-horizon control.
Together, these results highlight that how the state is represented matters as much what information it contains. State representations do not merely transmit information to the model, but actively shape the form of reasoning that the model performs. Compact representations are beneficial when they preserve task-critical structure while removing irrelevant information, but can be harmful when they omit details required for decision-making. Similarly, structured or spatial representations are only effective when the model is sufficiently capable of reliably interpreting or constructing them; otherwise, they introduce noise or mismatches that degrade performance.
19

---

Published in Transactions on Machine Learning Research (04/2026)
Reflecting on the criteria for effective state representations described in 2.1 our findings suggest that the main challenge is to provide state that is compact while still capturing the full context needed to choose actions and predict their oon-term success. Summaries and structured encodings can reduce context length, but might omit details necessary to evaluate the quality of the current situation. Conversely, richer natural- language stales better preserve semantic information and support generalisation to new scenarios, al the cost of longer context. As such, the encoding format is not a guarantee of quality; effectiveness must be proven by measuring the actual performance of the downstream controller (Lesort et al. 2018).
Future research should consider the following. First, we observed that failures often arise from drift in the LLM-maintained state reprosentation of the summary trajectory or VoT, where small omissions, halluci- nated facts, or internal inconsistencies accunulate over long horizons and eventually influence downstream decisions. A promising direction is therefore to pair the agents with verifiers during the process. This can be done through self-verification (Madaan et al., 2023 Shinn et al. 2023 Wang et al., 2022) or external verification using tools for tsub) that involve math or formal rules (Chen et al. 2024b Gao et al. 2023)
Second, we observe that current. VLMs remain weak at temporal spatial state (racking: even when object recognition is reliable, models often fail to maintain and update a coherent spatial belief over long sequences. Current pretraining regimes rely mostly on image- text pairs that rarely require precise spatial discrimination or persistent geometry across time (Chen et al. 2024a]. Progress may therefore require training signals that directly encourage maintaining a consistent spatial state over many steps. For example, combining explicit geometric supervision with interactive trajectories that force the agent to repeatedly update its beliefs under partial observability |Chen et al.] 2024a| Song et al., 2025). Third, in this study, we have compared two trajectory settings: either maintaining the full interaction history or using a fully summarised history, and we have not considered hybrid memory schemes. For example, summarising carlier steps while retaining window of more recent context. Evalusting such hybrid trajectory designs is an importani direction for future work. Fourth, our findings are based on a limited set of open-source models and benchmarks, and we have not evaluated how these representational trade-offs manifest in proprietary models. Extending the analysis to a broader range of task families and environments remains future work, because real-world environments are dynamic (Plaat et al.] 2025),, static or one-shot benchmarks can overestimate competence by avoiding a key challenge of interactive decision-maine making: maintaining and updating state under (partial) observability as new evidence arrives. We therefore argue for shift toward dynamic evaluation, true agentic competence is not defined by one-shot accuracy, but by the ability to dynamically reconcile new evidence with internal world model to navigate the unpredictability of the physical world.
References
Wendelin Böhmer, Jost Tobias Springenberg, Joschka Boedecker, Martin Riedmiller, and Klaus Obermayer. Autonomous learning of state representations for control: An emerging field aims to autonomously learn state representations for reinforcement learning agents from their real-world sensor observations. KI- Künstlüche Intelligenz, 29())333 362, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcemnnt learning. In International Conference on Machine Learning, pp. 3676-3713. PMLR, 2023.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognation, pp. 14455-14465, 2024a.
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Nicbles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why spatial reasoning hard for vims? an attention mechanism perspective focus areas. Xiv preprint ar Xim:25003 01779, 2025.
20

---

Published in Transactions on Machine Learning Research (04/2026)
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disen- tangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2024b.
Maxime Boiisertr-Boissertt,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: First steps towards grounded language learning with a human in the loop. In International Conference Learning Representalions, 2019. URL https://openreviesr
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gchrmann, et al. PaLM: Scaling languago modeling with pathways. Journal of Machine Learning Research (JMLR), 24(244):1-113, Oct 2023.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 2023.
Ayoub Echchahed and Pablo Samuel Castro. A survey of state representation learning for deep reinforcement learning. Transactions on Machine Learning Research, 2025. ISSN 1835-8856. URL https:///pperrevies het/forrm2?drrgQ03AWWHHt Survey Certification.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference Machine Learning, pp. 10764-10799. PMLR, 2023.
Lyle Goodyear, Rachel Guo, and Ramesh Johari. The effect state representation on Ilm agent behavior dynamic routing games. arXiv preprint arXiv:2006. 15624, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprinl ar 2x7::2407.217888 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement (atu22011.22445 2025.
learning. arXiv preprint
Simeng Han, Tianyu Liu, Chuhan Li, Xuyuan Xiong, and Arman Cohan. Hybridmind: Meta selection of natural language and symbolic language for enhanced 1lm reasoning. arXie preprinl SSG::2409.19389 2024.
Hanxu Hu, Hongyuan Lu, Huajian Zhang, Yun-Ze Ze Song, Wai Lam, and Yue Zhang. Chain-of-symbbl prompt- ing for spatial reasoning in large language models. First Conference on Language Modeling, 2024. URL httpa://openreviier
Timothée Lesort, Natalia Diaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108:379-392 2018.
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 6342 6353, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction luning, Advances in neural information processing systems, 36:34892-3499 2023.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: 10.1162/tacl a 00638. URL httpp://aclaathhollgy org/2024. tacl-1.9/
21

---

Published in Transactions on Machine Learning Research (04/2026)
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation 0.t:2300002255. 2023.
models in visual contexts. arXiv preprint
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with belf-feed Advances in Newral Information Processing Systems, 36:46534-4665 -45344,, 2023.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pp. 2263-2279, 2022.
Melanic Mitchell. Artificial intelligence learns to reason, 2025.
Davide Paglieri, Bartlomiej Cupial, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Ed- uardo Pignatelli, Lukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder and Tim Rocktäschel. BALROG: Benchmarking agentic LLM and VLM reasoning on games. In The Thir- teenth International Conference on Learving Representations, 2025. forum?id=fp6tc36697
URL httpp::/opeerrevies net/
Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 3806 3824, 2023.
Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg, Agentic large language models, a survey. Journal of Artificial Intelligence Research, 84:29:1-29:74 2025.
Aske Plaat, Annie Wong, Suzan Verberne, Joost Brockens, and Niki Van Stein. Multi-step reasoning with large language models, a survey. ACM Computing Surveys, April 2026.
Md Masudur Rahman and Yexiang Xue. Natural language-based state representation in deep reinforcement learning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1310-1319, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v 10/////11//2222 findiigs naacl.83. URL https://accanthoolog// org/2024: www..iffiiinnnnaeee naacl. 83/
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, el al. A generalist agent. arXiv preprint vrXic:2205.06117 2022.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neurul Information Processing Systems, 37:8612 8642, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Lan- guage agents with verbal reinforcement learning. Advances in Newral Information Processing Systems, 36: 8634-8652, 2023.
Fatemeh Shiri, Xiao- Yu Guo, Mona Golestan Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.
2024.
Xinshuai Song, Weixing Chen, Yang Liu, Weikai Chen, Guanbin Li, and Liang Lin. Towards horiznorizon vision-language navigation: Platform, benchmark and method. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2025.
Rob Sullivan and Nelly Elsayed. Can large language models act as symbolic reasoners? In 2025 IEEE 4th International Conference on Computing and Machine Intelligence (ICMT), pp. 1-6. IEFF, 2025.
22

---

Published in Transactions on Machine Learning Research (04/2026)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Sell-consistency improves chain of thought reasoning in language models. ar arXiv preprinl DrXiiv2203.11174 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions Mu- chine Learning Research, 2022a. ISSN 2835-8856. URL Survey Cortification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-offthought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24822 2022b.
Annie Wong, Thomas Bäck, Aske Plaat, Niki van Stein, and Anna V Kononova. Reasoning capabilities of large language models dynamic tasks. In 3rd International Conference Foundation and Large Language Models (FLLM2025), arXiv preprint arXiv:2505.1055 10543, 2025.
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Dong, Lei Cui, and Furu Wei. Mind's eye of Ilms: risualizationn----honghh elicits spatial reasoning in large language models. Advances in Neural Information Processing Systems, 37:90277-903111 2024a.
Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. Smartplay : A benchmark for LLMs as intelligent agents. In The Twelfth International Conference on Learning Representations, 2024b. URL https:
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayihong Liu, Fci Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1077 10671, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9556 9566699577-70 2024.
23

---

Published in Transactions on Machine Learning Research (04/2026)
A Appendix
Example 1: Agent Prompt
You are an intelligent agent with the goal of succeeding in the game by maximizing the cumulative score. Your decisions should be based on the game manual, the current observation, and your past trajectory.
# Task:
1. Analyze the current observation and past trajectory to select the most suitable action that aligns with the game's objectives and maximizes cumulative rewards.
2. Choose action from the provided list of actions, starting from index and provide concise reason for your choice.
# Response Constraints:
1. Select only one action from the provided list.
Provide reasoning that directly links the chosen action to the game's objectives and observed patterns.
Respond strictly with the action and the reason in the specified format.
# Response Format:
Action: |action number]. Reason: [explanation]
# Input data
Game Description: fmanual)
Current observation: {obs}
Past trajectory: (Full trajectory) OR (Summarized trajectory]
Question: {question}
Example 2: Summarization Prompt
Your role is to compress the recent trajectory into a concise summary that an agent can reuse next step.
Game Description: {manual}
Recent history (most recent last): Previous rolling summary: {previous summary)
Task:
Produce a new summary <25 tokens.
Mention agent location, key Deens/inventor, heals/hazards, and momenturn toward objec- tives.
Do not restate every step; only the most salient facts.
If the history is empty, return "Start of game".
Do not only repeat an action, but summarize what actions led to what outcomes.
Respond strictly this exact format. The summary should be one sentence only, but you can use a comma.
Summary: <concise summary here>
24

---

Published in Transactions on Machine Learning Research (04/2026)
Example Visualization-of- Thought Prompt
You are an intelligent agent whose objective is to maximize cumulative score in the game. Ground every decision in the game manual, the current observation, and your remembered trajectory. # Visualization-of-Thouggg Protocol
You will:
Draw a compact top-down ASCII map of the current situation before choosing an action.
Update that map you reason about candidate moves, annotating notable changes (e.g., planned path, hazards, inventory).
Decide the single best next action from the provided list.
Guidelines for the ASCII map:
The ASCII map must represent only what is explicitly visible in the observation.
Keep it <6x6 unless the observation explicitly demands more detail.
Use consistent symbols (e.g., A for agent, G for goal, for wall, for empty) and include legend when helpful.
If multiple hypothetical moves are considered, show intermediate marks such as or arrow glyphs to illustrate your updates.
the environment non-grid-based (c.g., Hanoi), draw the most compact symbolic state instead of a grid.
Required Output Sections
Map (Top-Down View): Latest ASCII grid after incorporating reasoning updates.
2. Map Update Notes: Bullet list (<3 bullets) summarizing how the map changed during reasoning.
3. Reasoning:
Brief chain-of-thouuht tying observation/mamuul to the chosen action.
Action:
Action: [number] ([action name]). Reason: [concise
justification].
5. Summary: Finish with Summary: <<25-token recap of agent location,
inventory, hazards, and next intent> so the rolling summary slays current.
Live Input Game Description: {manual}
Current observation: {obs}
Past trajectory: [Full trajectory] OR (Summarized trajectory
Question: {question}
Example 4: Hanoi:Naturallanggungg
Peg A has disk 2 at the bottom, disk in the middle, and disk 0 on top. Peg is empty. Peg C is empty.
25

---

Published in Transactions on Machine Learning Research (04/2026)
Example Hanoi:DictLii
'A': [2, 1, 0J, B:: D, CC::
Example 6: Maroi-Matrix
[2, 0], [-1, -1,-1]] -1]]
Example 7: Hanoi: Lisg
A: |bottom, [2, 1, 0], topl
B: |bottom, [. top
C. bottom, [ top
Example 8: Messenger:Vatturallaanugg
You took action Move North.
You (agent) already have the message.
You see:
bird steps away
ship 9 steps away
sword 5 steps away
Example NessengeeNaauurallanggggggoss
You are an agent with the message. You are currently in position 5, 6. You can see mage 3 steps to the west, dog 1 step to the east, ball steps to the northwest.
Example 10: Messenger: Coordinates
COORDINATE SYSTEM:
Agent: (5,
Entities:
airplane_ 0: (5, 3)
ball_0: (7, 5)
queen_O: 0: (3, 5)
Original View:
You (agent) don't have the message.
You see:
airplane 2 steps to your west.
-ball 2 steps to your south
queen steps to your north
26

---

Published in Transactions on Machine Learning Research (04/2026)
Example 11: Messenger:Symbolic
G.A.M...
...E....
Legend:
A A=agent (no msg)
P P-agent with msg)
eempl
Entities:
E-fish
M M=scientist
G G-robot
27


Reviews:
[{"review_id": "n1k7w3SPEw", "reviewer": "Reviewer_Q1qS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper systemically studies the design principles of state representations in LLMs, from three aspects: state granularity, state structure, and spatial grounding. The authors draw conclusions from experiments in multi-step, interactive environments. The conclusions indicate the importance of history summarization, the robustness of natural language representation, and the effectiveness of Visualization of Thought.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors draw conclusions from multiple multi-step, interactive environments where the state evolves in response to the agent’s actions and success requires long-horizon planning, memory, and spatial reasoning.", "audience_yn": "Yes", "audience_explanation": "The paper systemically studies the designs of state representations, which can inspire future LLM agents methods and frameworks.", "requested_changes": "1. Realistic agentic LLM setups can be different (and more complicated) from the setups considered in this paper. For example, it is very popular to use the combination of the trajectory summarization and the long-form trajectories. In other words, the agents may compact the earlier histories and append all of the recent context. Such differences may weaken the conclusion of the paper.\n2. For the same state representation, different state designs may also impact the conclusions. For example, the effectiveness of summarisation strongly depends on the quality of the generated summary. If important details are omitted or distorted, performance can degrade, making the approach unreliable in some tasks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YyPSveQN89", "reviewer": "Reviewer_2SXt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies how the state representation in prompts has impact on the performance of LLMs on multi-step reasoning. The paper systematically studies three aspects of state representation, which are history granularity (full trajectory or summary), structure (natural language or symbolic encodings), and spatial grounding (text-only or images or text-based maps). The paper conducts experiments on environments including Tower of Hanoi, Messenger, and BabyAI. The results show that representation choices significantly change performance of LLMs when the model is unchanged. In summary, the experiment results indicates that full trajectories, natural language representations and text-based spatial maps can improve long reasoning, encouraging step-by-step spatial reasoning instead of simply providing more information.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are generally supported by empirical evidence. The experiments directly compare the three proposed representation methods against baselines. It would be beneficial to further evaluate these methods on additional non–grid-based tasks and larger models (e.g., GPT-5.2 or Claude-4.6).", "audience_yn": "Yes", "audience_explanation": "The paper is relevant to researchers working on LLM agents, prompting methods, and sequential decision-making, so the results on improved prompt representations could help make the usage of LLM-based systems more effective.", "requested_changes": "1. Evaluate the proposed representation methods on a broader set of tasks beyond grid-based environments.\n2. Include experiments with additional large frontier models to test whether the observed trends hold at larger model scales.\n3. Add a short analysis or qualitative examples to better explain why certain representations work better for some models.\n4. Since LLMs have memory limits, discuss the trade-offs between performance gains and inference cost.", "broader_impact_concerns": "There are no impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zcqXCre2jj", "reviewer": "Reviewer_8XVq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how inference-time state representations shape LLM/VLM performance in dynamic, multi-step environments, holding model parameters fixed and varying only the prompt-level state interface. The authors organize \"state design\" along three axes: (i) granularity (full interaction history vs summarized trajectory), (ii) structure (natural language vs symbolic encodings such as dictionaries/matrices), and (iii) spatial grounding (text-only vs image inputs vs textual map encodings). Empirically, the paper evaluates open-source LLMs/VLMs on Hanoi & Messenger (SmartPlay) and BabyAI tasks (BALROG) using multiple representation variants, and reports normalized score improvements and bootstrap-based significance testing. In this paper, a representation refers to the inference-time encoding used to translate the environment state into the model's input context. Different representations contain similar information but structure it differently, thereby inducing different reasoning behaviors in the same underlying model. The paper presents a systematic study that shows that representation choice can substantially influence performance. In other words, changing the interface language to a fixed intelligent system changes its apparent reasoning ability.\n\nOther more specific findings include:\n* Trajectory summarization often improves long-horizon performance, especially when the task admits a compact abstraction (e.g., Hanoi), but can harm when critical decision details are omitted. \n* Natural language appears the most robust representation across model families, while structured encodings can help mainly for models with strong code/structured-output priors, and can be brittle for others. \n* Spatial grounding via images is generally inconsistent and often degrades performance, suggesting that images rarely provide useful additional grounding beyond the text state. VoT tends to be more consistently beneficial than alternative spatial grounding variants.\n\n\nStrengths\n* Clear research question and controlled/rigorous methodology: varying state representations while keeping model weights fixed and comparing across multiple tasks/models. \n* Strong use of oracle-style ablations to separate interface utility from model ability. \n* Inclusion of qualitative failure modes. \n* Supplementary material provides concrete prompt templates (agent prompt, summarization prompt, VoT protocol) and examples of the representations used, supporting reproducibility.\n* The authors provide code at https://anonymous.4open.science/r/state-representations-llms-dynamic-tasks-87BA/README.md\n\nWeaknesses\n* The authors perform many comparisons (tasks $\\times$ models $\\times$ representations), but I think there is no control of the Family-Wise Error Rate (FWER) and/or explicit discussion about multiple comparisons (potentially inflating false positives). \n* The paper does not include an introductory explanatory figure that would help readers quickly understand the scientific framing of the work. In fact, the first figure only appears on page 11.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims of the paper are supported by empirical evidence obtained through controlled experimental comparisons in which model parameters are held fixed while only the state representation interface is varied. This design directly targets the central research question and reduces confounding factors related to model scale or training differences.\n\nAcross multiple environments (SmartPlay and BabyAI) and model families, the authors report performance differences associated with representation choices, supported by repeated runs and bootstrap-based confidence interval estimation. Importantly, the paper complements performance results with oracle-style ablations (e.g., Oracle Summary and Oracle VoT), which help disentangle improvements arising from information availability versus representation construction.\n\nThe claims made in the manuscript are generally descriptive and empirically grounded, rather than speculative or beyond the presented evidence. While some interpretative conclusions could benefit from additional clarification, the experimental results presented are sufficient to support the core empirical findings reported in the paper.", "audience_yn": "Yes", "audience_explanation": "The findings are directly relevant to researchers working on LLM agents, prompt and interface design, sequential decision-making with foundation models, context compression and memory mechanisms, and multimodal reasoning evaluation. Importantly, the paper provides actionable insights about representation choices without proposing new architectures, aligning well with TMLR's emphasis on technically sound empirical understanding rather than novelty.", "requested_changes": "Below, I propose several changes or potential improvements to the paper, none of which are critical. That is, in my view, the paper as it currently stands is reasonably robust.\n\n1) The paper reports statistical significance using bootstrap confidence intervals computed independently for each comparison. However, given the large number of model $\\times$ task $\\times$ representation comparisons, no explicit control of the family-wise error rate or false discovery rate is provided. Clarifying that would strengthen statistical interpretability.\n\n2) I believe the paper would benefit from a figure in the Introduction to visually illustrate its main methodological proposal and/or scientific goals. In particular, I refer to a figure that would allow readers, at a glance, to understand what the authors aim to do at the methodological level.\n\n3) Regarding the experimental setup and the choice of LLMs, if possible, it would be beneficial to provide some justification or rationale explaining why those specific LLMs were selected, as well as the reasoning behind the chosen hyperparameter values (e.g., \"temperature 0.2 and nucleus sampling with top_p=0.95\"). Are the results sensitive to these values?\n\n4) I think it would be positive to include a clarification regarding the statistical methodology used for bootstrap confidence interval estimation. The paper reports bootstrap-based confidence intervals to assess statistical significance of performance differences across representations. However, it is not entirely clear what constitutes the resampling unit in the bootstrap procedure. In particular, are confidence intervals computed by resampling runs or individual episodes within runs? In my opinion, this distinction is important because, I assume, episodes executed within the same run share the same model configuration, prompting setup, and experimental conditions, and may therefore exhibit non-negligible correlations. If resampling is performed at the episode level rather than at the run level, the independence assumption underlying bootstrap estimation may be violated. Correlated samples effectively reduce the number of independent observations, causing bootstrap variance estimates to be underestimated and confidence intervals to appear overly narrow (i.e., optimistic). Could the authors clarify the level at which bootstrap resampling is applied, and whether independence between resampled units is assumed or empirically justified?\n\n5) I believe it would be helpful to introduce certain concepts before mentioning them as if any reader were already perfectly familiar with them. For instance, VoT (Visualization-of-Thought). On the other hand, it is not entirely clear how performance is measured. What performance metrics are computed? Where exactly do the reported scores come from? While the paper frames evaluation in terms of episodic cumulative reward in an MDP, the manuscript does not specify how this reward (or any benchmark-specific success signal) is converted into the reported normalised score in each environment. Providing an explicit mathematical definition of the evaluation metric would substantially improve reproducibility and interpretability.\n\n6) If I'm not mistaken, representations are enforced only through prompt construction. In that case, would it be possible that performance differences arise from prompt-induced behavioural changes unrelated to representation fidelity? Have the authors considered verification analyses (e.g., trajectory ablations or information masking) to confirm reliance on the provided representation?\n\n7) The results suggest relevant model-representation interactions. Why do certain models interact better with specific representations? For instance, the paper suggests that models trained on code handle JSON formats (DictList) better, but a more detailed discussion of these pre-training strategies would add significant value.", "broader_impact_concerns": "From my point of view, the work does not present direct ethical risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)