BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Authors: Davide Paglieri
Bartłomiej Cupiał
Samuel Coward
Ulyana Piterbarg
Maciej Wolczyk
Akbir Khan
Eduardo Pignatelli
Łukasz Kuciński
Lerrel Pinto
Rob Fergus
Jakob Nicolaus Foerster
Jack Parker-Holder
Tim Rocktäschel
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
VLM
Agents
Benchmark
RL
Reasoning
Games
Primary area: datasets and benchmarks

Abstract: Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies—areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). 
We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as several models perform worse when visual representations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community. Code and Leaderboard at balrogai.com

Published as a conference paper at ICLR 2025
BALROG: BENCHMARKING AGENTIC LLM AND VLM REASONING ON GAMES
Davide Paglieri's ; Bartlomiej Cupial Samuel Coward Ulyana Piterbarg'. Maciej Wolczyk², Akbir Khan 15, Eduardo Pignatelli', Lukasz Kucinski Lerrel Pinto* Rob Fergus', Jakob Nicolaus Foerster Jack Parker-Holder', Tim Rocktäschel' AI Centre, University College London, IDEAS NCBR, University of Oxford, "New York University. AAthhooiic University of Warsaw
Institute of Mathematics, Polish Academy of Sciences d. maglieriecss ucl.ac.. acl.ac.uk
ABSTRACT
Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well in complex, dynamic environments. Real-world tasks re- quire handling intricate interactions, advanced spatial reasoning, long-term plan- ning, and continuous exploration of new strategies- areas in which we lack effec- tive methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, novel benchmark designed to assess the agen- tic capabilities of LLMs and VLMs through diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environ- ments with varying levels of difficulty, including tasks that are solvable by non- expert humans in seconds to extremely challenging ones that may take years to master (c.g., the NetHack Learning Environment). We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle signif- icantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based paccision-making several models perform worse when visual rep- resentations of the environments are provided. We release BALROG as an open and user-frient weer-fiennlyy benchmark facilitate future research and development in the agentic community. Code and Leaderboard at balrogai.com
INTRODUCTION
Recent successes of Large Language Models (LLMs) have renewed interest in building general- purpose agents capable of autonomously achieving complex goals Yang et al. (002)). LLMS possess vast knowledge across domains (Brown 2020 Hendrycks et al. 2020), can reason in specific scenar- ios (Wei et al., 2022a Shinn et al. 2023 Rein et al. (023), and can reliably follow human instruc- tions in simple settings (Ouyang et al.) 2022). These abilities suggest that LLMs have the potential to become efficient agents, capable
autonomously performing a wide range of human tasks that require sequential decision making. In the present day, however, oftt--of-the-rt models continue to exhibit persistent failure modes on many of the skills that are crucial for autonomous real-world in- teraction. For example, L.MSs fail to act robustly in dynamic environments, and they cannot reliably leamn from mistakes, reason about space and time, or plan over long time horizons (Xing et al. [024] Yamada et al. 2023) Kambhampati et 2024). Improving our understanding of LLM capabilities through rigorous, safe evaluations is key for assessing the risks and limitations of deploying agentic LLMs the real world.
Current agentic benchmarks evaluate LLM performance in settings that involve no more than few dozen rounds of interaction between a model and an environment, e.g., solving simple office
"Equal contribution.

---

Published as a conference paper at ICLR 2025
python eval.py
env wrapper.py
client.py
class Envirapper wampperrlym..aappppe nrapper):
def oreate, Lln, (ln.ccieett(lleeet))
def step(seif. action): return salf (action)
client zs openat.
raturn Opan[Bragaar(connt
Laurch BAUROS
clsent gerini:
(eaet(selr):
self.env. reset()
elif client zz claude:
return claudotrapper (Lenf))
elaf client =z vllm return vLL VLLmrrapper (cantt))
Gemini
Claude
evaluator.py
agent.py
run. run_opissoee env, agent): retal.reward
class (Bste(Bse:)
def act(self, obs, prev_act
obs, reward, done, info revere)
self.histurg builder upcate obs, prey act)
history self.history www.htsttryy... cuttooryyuuider butlder woullder.ga._j.e get_prompe()
for
steps):
action (abtt
Agentic Strategy
R
ebs, revard, done, anfo e.w. (ww.step(action) potal_reware +- reward
action self .In_clien((hiatorr))
return action
return total_roword
An overview the BALROG Benchmark for evaluating L.Ms on long-context in- teractive tasks. Submissions of new inference-time methods for improving the capabilities of an existing model via an "agentic strategy" need only modify the agent py file. Similarly, bench- marking new model zero-shot can be done by adjusting configuration file in client. py..t.py. The agent class includes a prompt builder to manage observation history, and a client that abstracts the complexities of various APIs and model-serving frameworks. The env www.waapppe.pp.. py file standard- izes interaction across settings, and the evaluator executes agents and collects performance metrics.
Figure 1:
tasks (Wang et al. 2024), navigating the Internet (Zhou et al. 2023), and resolving GitHub is- sues (Jimenez al. 2023). New agentic prompting frameworks and improvements to short-horizon reasoning via LLMs like OpenAl ol have led to dramatic and fast-pacer aist-paced gains in state-of-the--eee art per- formance on these benchmarks (OpenAI (024b) [2024]] Wang et al. [2033] Fernando et al.) [2023] Hu al.]
2024). However, many realistic tasks require orders of magnitude more interactions (Pignatiello)
et al. 2020 Wansink Sobal 2007).
In this paper, we argue that the next frontier for language and vision-language model capabilities lies in long-horizon reasoning and decision-making To that end, we propose BALROG: Bench- marking Agentic LLM and VLM Reasoning On Games. BALROG is a benchmark and framework that aggregates a diverse set complex reinforcement learning game environments into a unified testbed for research long-context LLMs. Games have historically served highly effective metrics for evaluating progress in deep reinforcement learning research (Bellemare et al. 2013) Sil- et al.) 2018 Schrittwieser et al. 2020 Vinyals et al., 2019). By aggregating many different game environments into a single evaluation, we look to spur progress on developing truly generalist agents that can meaningfully address embodied, real world tasks. Specifically BALROG enables seamless running of LLM and VLM agents on BabyAI, Crafter, TextWorld, Baba Is AI, MiniHack.
and NetHack (Chevalier- Boisvert et al. 2019 Hatner 202 Côté et al., 2019; Cloos et al. 2024] Samvelyan et al.) 2021 Küttler et al.) 2020). These environments have lightweight simulators, ensuring that the benchmark is affordable for the research community. Furthermore, while all of these games are long-horizon, they span broad range of difficulty levels, from tasks where we see fair zero-shot performance by state-of-the-arr long-context models (Baby to those where even specialized neural models trained on billions of in-domain datapoints make very limited progress (NetHack) (Piterbarg et al. 2024, Klissarov et al.) 20033, Wotezyk et al. (024).. BALROG dif- ficult solve through simple memorization all of the environments used in the benchmark procedurally generated, and encountering the same instance of an environment twice is unlikely.
Using the six proposed environments, vLMs. We employ a fine- fine-graing geainerined metric that captures how close each model is to completing a task, which gives us thorough understanding of the resulting trajectories. In qualitative analysis,
evaluate the capabilities of various popular LLMs and
2

---

Published as a conference paper at ICLR 2025
study the agents' capabiliites for spatial reasoning, systematic exploration, long-term planning, and discovering environment dynamics. We find that the current top LLMs show promise on the simplest tasks but completely fail to make meaningful progress on the more difficult tasks, such as MiniHack and NetHack. Some of the models exhibit knowledge about the game from pre-training but fail to use it in practice. For example, in NetHack, GPT-4o often dies from the consumption of rotten food, even though, when prompted, it correctly identifies it as very dangerous. Furthermore, we study the impact of the input representation. Although the majority of the environments were created with vision in mind, we find that multimodal LLMs perform much worse when also presented with an image of the environment rather than a textual-only description of the observation. This suggests that reliable vision-based decision-making is currently far outside our reach.
Our results show that BALROG is a very difficult benchmark that still allows us to observe fine- grained progress in crucial areas such as long-term planning, spatial reasoning and navigation. We share the codebase and open the benchmark for external submissions. We summarize our contribu- tions as follows:
BALROG, suite of six reinforcement learning environments for testing the agentic capa- bilities of long-contex, LLMs. We provide finegrained metric for model evaluation, and we develop a novel data-informed progression system for NetHack.
Baseline evaluations of state-of-the LLMs on BALROG using zero-shot prompting, in both Language- Vision and Language-only modalities. We show that while models exhibit decent performance on easier games, all are very far from solving the hardest game in the benchmark, NetHack. We observe that the performance drops further when images of the environment are presented, suggesting severe problems with VLM recision-making.
We perform a qualitative analysis the results across capabilities such as spatial reasoning, systematic exploration, and long-tent planning. We identify an intriguing knowing- -oing gap where the models cannot employ the knowledge they possess.
An open-source toolkit for benchmarking long-context models on BALROG. This toolkit enables researchers and practitioners quickly evaluate model performance. While the baseline evaluations performed in this paper are zero-shot, the BALROG toolkit supports inference-time prompting strategies like chain-of- (Wei et al. 2022b), few-shot learning, and more.
2 BALROG
BALROG is a benchmark and framework that aims to improve understanding of whether exist- ing long-contex LLMs are agentic, i.e., whether they can be used to automate complex activities that require sequential decision-making. It supports model evaluation on challenging reinforcement learning environments that test skills such as long-term planning, spatial reasoning, and the ability deduce the mechanics of the enviromment.
By design, the BALROG framework explicitly decouples inference-time prompting strategies from underlying models. The goal of this design choice is two-fold: (1) to facilitate rapid prototyping of inference- -tieeneee--meeeeeeeee methods for improving model performance on long- context eecision-making beyond zero-shot prompting and (2) to ensure that model evaluations are consistent and rigorous.
In the remainder of this section, we introduce the game environments evaluated in the benchmark and we discuss our protocols for model submission to the BALROG Benchmark Leaderboard
2.1 ENVIRONMENTS
BALROG evaluates long-context models as agents on the games described below.
AI.yAI. Coiiseete-Boisvertt et al. 2019 Carta et al. 2023) A simple, two-dimensional grid-world in which the agent has to solve tasks of varying complexity described in natural language (c.g., "go the blue ball, then pick up the grey key"). Agents are tested across five different types of navigation tasks, see Appendix
This I Leaderboard will open the public the time of publication.
3

---

Published as a conference paper at ICLR 2025
Table 1: The tested skills, time horizons, and complexities of interactive decision-making tasks evaluated in BALROG. Compared to existing benchmarks, BALROG provides infrastructure for evaluating model reasoning and decision-mal making on harder, longer time-horizon interactive settings. The evaluated tasks span range of difficulties.
Skills
BabyAI
TextWorld
Crafter
Baba Is AI
MiniHack
NLE
Navigation
Exploration
Resource Management
X
X
Complex Credit Assignment
X
Deducing Env. Dynamics
X
Long-term Planning
X
X
Turns to Complete
10'
10a Minutes
10a Hours
10a Hours
10a
10--10²
Time Master for Humans
Seconds
Hours
Years
Minecraft-inspires grid environment where the player has explore, gather resources and craft items to ensure their survival. Agents are evaluated based the number of achieved milestones, such as discovering new resources and crafting tools, see Appendix B
Crafter. (Hafner 2021)
TextWorld. (Côté et al. 2019) An entirely text-based game with no visual component, where the agent has explore mazes and interact with everyday objects through natural language (e.g., "cook potato with oven"). Unlike the other environments in BALROG, TextWorld is not a grid-world. Models are evaluated on three different tasks, see Appendix
Baba Is AL (Cloos et al. 2024) An environment based on the popular puzzle video game Baba Is You. The player manipu the rules of the game world by pushing word blocks, altering how objects interact. Agents tested 40 puzzles, see Appendix(D) [D]
MiniHack. (Samvelyan et al. 22021) MiniHack is multi-tase framework built on top of the NetHack Learning Environment KKüttler et al. 2020). We select five different tasks, Maze, Corridor, Corridor- Battle, Boxoban, and Quest. Collectively, they assess a wide range of skills, including exploration, navigation, long-term planning, and resource management, see Appendix
NetHack Learning Environment (NLE) (Küttler et al. 2020) is based on the classic roguelike game NetHack, known for its extreme difficulty and complexity. Success in NetHack demands both long-term strategic planning, since a winning game can involve hundreds of thousands of steps, as well short-term tactics to fight hordes of monsters. Accurate credit assignment is also crucial to understanding which actions contributed to success or failure. It takes human players years to master NetHack without accessing external guides. Notably, we find that research shows that LLMS can answer questions about the game mechanics and optimal strategies (see Appendix F55 but they fail to apply this knowledge in practice. See Appendix F for more details.
Table 1 provides overview of the environments used in the benchmark, detailing the reasoning and agentic capabilities required to succeed in each. This diverse set of environments positions BALROG comprehensive benchmark for assessing the capabilities of LLM agents, making it valuable tool for evaluating their performance for years to come.
2.2 SUBMITTING TO THE BENCHMARK LEADERBOARD
The BALROG benchmark accepts two types of submissions.
Submissions may include any type of new model, such as large language models (LLMs), vision-language models (VLMs), large-action models (LAMs), fine-tuned versions of existing models. The key requirement is that these models must be capable of generating actions in natural language. By default, these models will be evaluated zero-shot.
New Models.
Agentic Strategies. Submissions may propose novel inference-time time prompting strategies for im- proving the reasoning, plamning, or in-context learning capability of an existing model. These strate- gies should extend beyond simple zero-shot prompting for direct action prediction, demonstrating more sophisticated techniques for inference-time decision-making.
4

---

Published as a conference paper at ICLR 2025
3 LERO-SHOT EVALUATION PROTOCOL
In this section, provide description of our protocols for evaluating state-of-the-art, long-context LLMs and VLMs on BALROG. These evaluations are intended to serve as baselines for the bench- mark. As a result, they probe zero-shot performance only.
3.1 EVALUATION SETTING
We aim to keep the evaluation setting simple. During each timestep of interaction, agents are prompted to output the next action natural language string. conditioned their past interaction history in the environment. To perform successfully in BALROG, models must demonstrate robust instruction-followee capabilities, including reading and interpreting game rules, understanding the action space, and producing valid actions to complete tasks effectively.
To address cases where the LLMs/VLMs output hallucinated invalid actions, AgentQuest pro- vides feedback to the agent indicating the action's 's invalidity, it then executes a default fallback action (such "do-nothing" action standard move like "north"), and logs the occurrence for trajectory statistics. This ensures that the interaction remains continuous and robust while enabling users to analyze the context and frequency of such errors post-evaluation analysis.
A diagrammatic visualization of BALROG is shown in Figure I We conceptualize the agent as combination of the underlying LLM/VLM model and a particular prompting strategy. We provide unified client wrapper that seamlessly integrates APIs for closed LLMs and VLMs such as OpenAI, Gemini, and Claude and allows users effortlessly switch and evaluate models. For the evaluation of locally-served models, we include native support for the vLLM library (Kwon et al.) 2023 which optimizes throughput by efficiently batching generation requests. We use multiple seeds for each environment to ensure the statistical significance of the results.
Metries To ensure a fair and interpretable evaluation, we introduce a standardized metric, scoring performance each task within range of 0 to 100. For environments like MiniHack, BabyAl, and Baba AI, each episode is scored as either or 100 based on task completion. For Tex(World, Crafter, and NetHack portion of achievements toward the maximum score. For NetHack, as the game scoring system does not adequately reflect actual progression (Wotezyk et al. 2024),, propose novel, data-informed progression metric, described Appendix F22 to better capture agent performance.
the score real number between O and 100, representing the pro-
Performance BALROG supports highly parallelized evaluations, leveraging the lightweight simu- lators of each of the environments in the suite. These evaluations allow multiple agents and environ- ment instances to run concurrently with minimal computational overhead. Environment instances run asynchronously from one another, accommodating varying observation lengths and ensuring that agents with faster generation speeds (per action) are not affected by slower agent bottlenecks.
3.2 OBSERVATIONS
In the initial prompt. the agent is introduced to the game rules and provided with list of avail- able actions, each accompanied by brief description. To prevent model overspecialization, we design general prompt that is not fine-tuned to any specific LLM. Subsequent prompts present the observation-actee action history from the perspective of the "user", while prior actions are attributed to the "assistant" or "model" role, depending on the type of model used. This structure mirrors the standard format used for fine- tuning instruction-followe appendices.
chat-based format. The game rules and observations are conveyed
LLMs. Detailed examples of game observations are included in the
Except for TextWorld, which lacks visual component, evaluate all environments using two observation modalities:
Language Only Format Observations are expressed as natural language descriptions of the envi- ronment's s state (c.g.. "a wall 5 steps ahead, a wall 2 steps to the left... "). For environments without native textual representations,
either generate descriptions using open-source language wrappers (BabyAI (Carta et al. 2023), Crafter (Wu et al., (023). NetHack, and MiniHack (Goodger et al. 2023 develop custom wrapper ourselves (Baba is AI, see Appendix
5

---

Published as a conference paper at ICLR 2025
Evaluating Existing State-of-the-Art Models on AgentQuest
Zero-Shot Prompting Long LLMs/VLMs Output Textual Conditionad on Interaction History
BabyAl
100
30
gt 4p vas
genuus d-dd
Burr kann farma 1-bbu
Spo
2.
ciaumi states cimsdin disprreet
1D
git Ap-reat gerac amoiie fare
gomsnt ppo
lann 8.
denua -700
Ioen 4.8. 11b
8-gorneet
ane Aereeeee
comeedd firs
gomstn pm Bante furm
4.O Tibe 1-7000
diacie B-annwe
Banta
chaste 3.5
LLM
Bebelaki
MiniHack
NesHack
80
Propuss R3
Proguess (S)
40
Bee reon
peroed (--ead geemi mo lamn Ammed Shut
p0u4d
Arre 2.1 F-ibed 1-70h-
dame gunn: 2.ttil disuie S-aau disdee Sxarme
pussos qentie yoo gero 1.2 thu
got stald
dare Aeme 1-12000 22.0d
--bid
Aure taae E-goreel
for permss dead gentis aed thu
leme Eiiie 2: Thhed dumm aaaoo diutte of
femer
Figure 2: Baselines for BALROG. We evaluate the zero-shot performance of seven kate-of-the-an and long-context LLMs and VLMs on BALROG. During each timestep of interaction, models are prompted to output the next in-game action conditioned past interaction history. Standard error is obtained by running multiple replicate seeds, as detailed in the Appendix.
Vision-Language Format For VLMs, the observation consists of an image representing the envi- ronment's current state, alongside natural language description (mentioned above). this format, the image corresponds only to the current observation, although we support including multiple im- ages in the observation history.
For the most complex environments, i.e., MiniHack and NetHack, augment the anguaae-bbsedd observations with two-dimensional map rendered using ASCII characters. For all experiments, we use history length of 16 observations to maintain consistency across tasks. However, participants submitting to this benchmark are allowed to modify the observation history length as needed for their respective models and experiments.
3.3 MODELS
We evaluate range of popular closed-source and open-source models, including Gemini-1.5-Flan
and Gemini- 1.5-Pro (Reid al. 2024). GPT-4o-mini (2024-07-18 release) and GPT-4o (2024-05-13
release) (Achiam et al.) 2023 OpenAl 2024a). Claude 3.5 Sonnet and Claude 3.5 Haiku (2024-10- 22 releases) (Anthropic (024). ) as well Llama 3.1 instruct (8B and 70B) (Dubey et al. 2024) and
Llama 3.2 instruct (IB, 3B, 1IB and 90B) (MetaAl 2024). Additionally, we test ol -preview (2024-
09- 12 release) (OpenAl 2024b) exclusively on the NetHack environment due to budget constraints.
RESULTS
In Figure we present the results of our experiments using the BALROG evaluation script for both language-only and vision-language formats. Most leading models demonstrate fair average progression on Al,yAl, Crafter, and Baba Is AI, with GPT-4o and Claude 3.5 Sonnet performing best. Interestingly, the open-source Llama 3.1 70B and Llama 3.2 90B models achieve the highest results on the Baba AI language-only format, narrowly surpassing GPT-4o and Claude 3.5 Sonnet. In TextWorld, GPT-4o and Claude 3.5 Sonnet lead, while Gemini models fail to complete any tasks,

---

Published as a conference paper at ICLR 2025
being flagged as 'unsafe' by the Google Gemini API, despite the prompts containing no actual safety concerns. The MiniHack suite proves very challenging for all models, especially the quest and boxoban tasks, which were never solved by any model. Finally, all models flat line with NetHack, with the best-performing model, ol opprevi, achieving a meager 1.5% average game progression.
Table summarizes the aggregated results across all environments in the language-only format. Overall, Claude 3.5 Sonnet is the best-perforning model, with an average progression 32.64%, followed closely by GPT-4o and Llama 3.1 70B few points behind. Gemini-1.5-Pro lags behind the other large models, partly due to its 0% performance on TextWorld. However, results differ for the vision-language format, as shown in Table 3 Here, we observe that both GPT-4o and Llama 3.2 exhibit decline in performance when image observations are included, likely due to confusion arising from the added visual input. In contrast, Demini-11.5-Pro and Claude 3.5 Sonnet especially, maintain consistent performance across both formats. This suggests that current multimodal Trans- former architectures are still better equipped handling textual information than visual input, topic explore further in Section 6 We show more detailed results for each environment in their appendices.
Table 2: Language-Oni Performance
Table 3: Vision-Language Performance
Model
Average Progress (%)
Model
Average Progress (%)
claude-3.5-sonnet
32.64 + 1.93
claude-3.5-sonnet
35.48 + 2.02
gpt-4o
32.34 + 1.49
gemini-1.5-pro
25.76 + 1.36
Hlama-3.1-70b--
27.88 1.43
gpt-4o
22.56 + 1.44
Ilama-3.2-90B-it
27.29 + 1.44
(lama-3.2-90B-
20.99 + 1.58
gemini-1.5-pro
21.00 1.18
gpt-4o-mint
15.36 + 1.29
claude-3.5-haiku
19.32 + 1.83
gemini- 1o5
14.94 + 1.40
gpt-4o-mini
17.36 + 1.35
1.2a--3.2-11Birt
8.43 = 1.26
11ama-3.2-11B::
llama-3.22 3.2-11
16.82 + 1.47
Hlama-3.1-b--4
15.14 1.55
gemini-1.5-flas
14.63 + 1.37
-lama-3.2-3B-i-
10.13 1.28
-lama-3.2-1B
6.65 + 1.04
4.1 QUALITATIVE ANALYSIS
We conducted analysis of the model trajectories across the environments identify common behaviors and challenges specific to each setting.
Spatial Reasoning While language models demonstrate some proficiency in basic navigation, they exhibit significant limitations in more complex spatial reasoning tasks. the BabyAl suite, we observed significant shortcomings in the agents' ability place objects adjacent other objects, which is required in some scenarios. In NetHack and MiniHack CorridorBattle, good spatial reason- ing crucial during combat, as players need maneuver within confined corridors to avoid being surrounded by monsters. However, the agents frequently ended up cornered.
Systematic Exploration Our experiments revealed a significant weakness in the models' ability to explore. In TextWorld's Coin Collector, where agents must explore a house to locate a coin, agents often wander aimlessly, revisiting rooms they' already explored while missing important areas entirely. An efficient agent would behave in DFS-like manner, methodically searching each room, keeping track of visited areas and prioritizing unexplored spaces. The more complex quests in MiniHack expose similar issues, with models failing to efficiently navigate maze-like structures.
Long-term planning The agents exhibit substantial deficiencies in devising and executing long- term plans. We observe near-zero performance on MiniHack, and NLE, which both require careful planning. In particular, we do not observe a single successful trajectory in the Boxoban logical puzzles in MiniHack, which requires careful planning at every step in order to avoid irreversible failures. LLMs, with the finite amount of compute available to them in a single forward pass, are necessarily confined solving some subset of reasoning problems. We observe that with the current models' depth, number of flops, and reasoning solution templates embedded in the weights, these models cannot solve the reasoning tasks in BALROG. We see notable improvement with OpenAl
7

---

Published as a conference paper at ICLR 2025
ol's chain of thought capabilities NetHack, performing close to three times better than its closest competitor in language-only mode Claude-3.5-Sonnee However, its average progression of 1.57% is still far from satisfactory.
Discovering and Leveraging Environment Dynamics Some games require inferring non-trivial causal structure through experimentation to come up with new strategies. For example, player might identify a potion of paralysis by drinking it, and then realize they can use this strate- gically by throwing such potions at enemies incapacitate them. This kind of experimentation and strategic thinking is crucial for success in NetHack. However, current models struggle to for- mulate and execute such context ceneex--deeeddet strategies. In MiniHack Quests environments, models fail to devise and implement multi-step strategies, such as utilizing wand of cold or r ring of levi tat on to cross lava rivers. In Crafter, where agents can handle basic tasks such as collect- ing wood, crafting items, drinking water, and even engaging in combat, they fail to learn long-term survival skills such as building shelters for protection against nocturnal threats.
Srowing-Doing Gap We observe pronounced "knowing-doing" gap, where models execute unde- sirable actions during gameplay despite knowledge of their negative consequences. For instance, in NetHack, models often exit the dungeon shortly after starting the game, resulting in an instant game termination. When queried in a separate thread about the consequences of exiting the first level in NetHack, they correctly identify that it results in instant death, making it is a highly undesirable action. Similarly, although the models correctly identify that eating rotten food in NetHack can re- sult in death, this remains common cause of failure, underscoring a disconnect between knowledge and decision-making Additionally, models tend to ignore even the hints directly present in the input prompt and die from overeating even when advised against it. To study this problem in more detail, we prepared a questionnaire probing basic NetHack knowledge (see Appendix F.5).
RELATED WORK
The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks. Benchmarks such SuperGLUE (Wang al. (2019), which tests general- purpose language understanding and MMLU (Hendrycks et al. 2020). which measures massive multitask language understanding, have been instrumental in advancing LLM research. BigBench (Sivastaaa et al. 2022) further expands the scope by including a diverse set of linguistic and cog- nitive challenges. Mathematical reasoning datasets like GSM8K and MATH (Cobbe et al.] [021]] Hendrycks et al. 2021) assess models' abilities to solve grade-school and competition-level math problems, while Shi et al. (2022) explore multilingual chain-of-thouuhh reasoning. In the domain of code understanding and generation, benchmarks such as HumanEval (Chen et al. 2021) and CodeXGLUE (Lu et al. 2021 evaluate models capabilities in programming tasks.
These benchmarks, however, are limited to single-turn short-context scenarios, do not require se- quential decision-making or adaptation to changing environments and have been saturating rapidly (Kiela et al., (021I. Static benchmarks may not fully capture the progress we are seeking, since the research community aims to push the frontier of agentic foundation models capable of acting in dynamic environments, using tools, planning ahead, and reasoning about their surroundings. Re- searchers have recently investigated how L.MSs use these skills to solve practical tasks, including using computer interfaces to perform office-related chores (Wang al. 2024 Qqn al. (2024), navi- gating web pages (Yao et al. [022 Zhou et al. 2023). and solve GitHub issues (Jimenez et | 2023) Several works studied the multi-agent capabilities of LLMs to see if they can co-operate (Gong al.,
2023 Piatti et al. 2024) or effectively play against other agents (Jin et al. 2024 Wu et al. 2024).
In this work, we study agentic skills in the context of video games, as they offer challenges well- tailored for human players and test skills that are useful for embodied agents. Previously, some related works employed games to benchmark L.MS (Liu et al. 2023b Todd et al., 2024, Wu al. 2023 Ruoss et al.) 2024), highlighting their emphasis on problem-solving spatial reasoning, and well-defined rules and objectives. Some of these benchmarks, however, are already reaching satu- ration, with environments like Crafter being the most challenging in their suite. In contrast, BAL- ROG fills an important gap by providing a wide range of games at varying difficulties including the NetHack Learning Environment (Küttler et al.] 2020), which takes humans years to master, and where zero-shot LLMs struggle greatly, as also seen in prior work (Jeurissen et al., 2024). These tasks represent rich and granular testhed for evaluating agentic foundation models, push-
8

---

Published as a conference paper at ICLR 2025
ing decision-making evaluations of LLMs/VLMMs to the very limit of their context lengths. Other environments such as MineDojo (Fan et al. 2022) and MineRL (Guss et al. 2019) also present open-ended challenges for agentic capabilities, their steep computational requirements and reliance on multimodal inputs make them less practical for accessible, large-scale benchmarks.
While BALROG currently focuses evaluating single-agent foundational capabillitis, future tensions could explore multi-agent collaboration environments that provide unique opportunities to Test teamwork and coordination skills in LLMs. For example, Overcooked (Carroll al., 2019
Liu et al. 2023a) simulates a cooperative cooking environment where agents must collaborate effi- ciently under time constraints and task dependencies, testing planning and communication abilities. Another compelling environment is Hanabi (Bard et al. 20200) a cooperative card game where play- ers must rely on indirect communication and inferential reasoning to achieve a shared objective under partial observability. These environments present rich opportunities to benchmark advanced collaboration and multi-agent decision makinn skills, which are essential for broader deployment of agentic LLMs.
OPEN RESEARCH PROBLEMS
Aside from its utility for model evaluations, BALROG also offers a test-bed for rapidly prototyping new inference-time methods for improving the agentic capabilities of LLMs and VLMs. There are many open research problems in this space. As of the writing of this paper, some of the most per- formant methods for improving model reasoning capabilities problems are infeasible to apply naively to BALROG due to the extremely long-context nature of tasks. Addressing these challenges could further enhance the development of stronger autonomous agents. We highlight several key areas for future work below.
short-form and/or shorter-context
In-Context Learning and Few-Shot Prompting BALROG enables evaluation of In-Context Learning (ICL) agents, which can use few-shot examples adapt out-of-distribution tasks. We provide a small dataset of human demonstrations for each environment and an implementation of few-shot conditioning in the BALROG codebase. The benchmark codebase also supports the study of In-Context Reinforcement Learning (Lee et al., 2024 Laskin et al. 2022 Lin et al. (023). where agents learn to improve from mistakes during inference. On the large models benchmarked Sec- tion 4 naive few-shot learning (i.e., prompting L.MM and VLM agents with examples of full human games in-context) extremely computationally expensive to run on BALROG. For example, a sin- gle demonstration of NetHack game-play can require upwards of 700, 000 input tokens to represent in prompt. Despite advancements in fast inference technologies like caching and falling API costs for long-context prompting, we found these experiments to be infeasible to conduct at this time. few-shot prompting
Sub-selecting only the relevant parts of demonstrations via strategies (Lewis al., 2020) might offer way to circumvent these challenges. We leave explo- ration of such methods for future work.
Advanced Reasoning Strategies Beyond simply prompting LLMs and VLMs to directly predict the next action of game-play, BALROG also supports the study of more advanced reasoning tech- niques like chain-of-thought (Wei et al. 2022b). self-refinement (Madaan et al. 2024), and basic planning. These methods have been demonstrated to improve model performance on shorter-context problems. We believe them to be an exciting direction for future work on long- contect reasoning and recision-making For example, model performance on the tasks in BALROG might be improved by
integrating multi-agen collaboration (Chang) 2023; Khan et al. 2024 Yao et al., (024) and tool usage (Shen et al. [2244] Ruan et al. [223) Schick et al. [2024] Qin et al. 2023) in decision- decision-making Additionally, incorporating memory mechanisms or reinforcement learning techniques could help bridge the "knowing-doing" gap, enabling models to apply their knowledge effectively in practical,
long-horizon tasks. Finally, experimenting with open-ended self celf-improvement loops (Wang al.)
2023
Hu et al.] 2024) could lead to more adaptive and general agents (Team et al. 2023 Hughes al. 2024), offering pathway toward truly autonomous systems.
Limitations of Current Vision-Language Models
Despite their potential, our benchmark shows
significant variability in VLM performance. While some models, like Llama 3.2, struggle to inte- grate visual information into coherent decision-making. others-moo notably Sonnet 1e5-denon-

---

Published as a conference paper at ICLR 2025
strate stronger performance in VLM mode. This disparity highlights significant variability in VLM capabilities, which may stem from differences in training objectives and datasets. For example, be attributed in part to its training tasks involving com-
Sonnet 3.5's superior performance puter usage (Anthropic) 2024), which inherently require integrating visual and textual inputs for action-based reasoning.
Recent studies have identified key limitations of VLMs that align with our findings, including bi- ases toward natural image-text pairs, optimization for image description rather than action-oriented reasoning, and challenges with out-of-distribution inputs (Tan et al. 2024 Tong et al., 2024 Rah- mmanzadehgervi et al., 2024 Zang et al. 2024 Guan et al., (023). These limitations are further exem- plified in our benchmark, where grid-baud base-baeed image observations differ significantly from the natural image-text pairs on which many VI.Ms are trained (Yu et al. 2023 Rahmanzadehgervi et al. 2024). Moreover, the computational cost of image processing constrained our evaluation to single image per observation, with the remainder of the history provided in text. While this constraint may hinder performance for some models, our results show that certain VLMs like Claude 3.5 Sonnet can still perform robustly under these conditions.
To address these challenges, our codebase already supports multi-image observation histories, and future iterations will incorporate video observations, which are likely better suited for the long- horizon sequential decision-maken better evaluate and leverage the potential of VLMs in complex reasoning scenarios. We plan to introduce support for video observations capabilities become available, ensuring that our benchmark remains aligned with the latest advance- ments in VLM technology.
tasks central to benchmark. These enhancements aim to
prominent models with efficient video-processing
Computational Limitations of Large Language Models Mechanistic interpretability could pro- vide valuable insights for understanding the computational limitations of agentic LLMs. The compu- tational expressiveness of LLMs is fundamentally linked with the ability to solve complex reasoning problems (Wei et al.] (022a).. While current models perform well on simple tasks such as navigation and object manipulation, they struggle with more complex tasks that could require non-trivial and general-purposs computation, for example, building a shelter or developing combat strategies. This could be due to the models' inability retrieve relevant computational circuits (Olah et al.. 2020), limitations to inference-tiee important questions about the scope of effectively solvable tasks for LLMs and VLMs, which is de- pendent on factors such as model depth, context size, and the distribution shift between pre-training and downstream tasks. Further research is needed to understand the underlying causes of these limi- tations and to develop strategies for overcoming them, such as adaptive simulation of computational circuits during runtime.
budget (Snell et al. 2024). or representational expressivity. This raises
CONCLUSION
We introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMS and VLMs across a diverse set of challenging, long-horizon tasks. Through easily reproducible evaluation protocols, BALROG reveals critical shortcomings in current models, particularly in areas such as vision-based decision-making and long-term planning, identifying clear gaps between model performance and human-level capabilities. These deficiencies, uncovered through our qualitative analysis, reflect the challenges faced in real-world scenarios, underscoring the practical relevance of our benchmark for agentic applications. Our evaluation framework leverages fast, procedurally generated environments, ensuring rigorous and fair comparisons by preventing test-set leakage, a common issue in other benchmarks. We believe that BALROG will serve critical tool for supporting and advancing research towards autonomous LLM agents.
ETHICS STATEMENT
This work provides a benchmark for the agentic capabilities of LLMs. We believe that experimen- tation in simulated environments, where the behavior of the agents is easy interpret, is crucial for building safe agentic systems. It is important to address questions on how to ensure that the agent's behavior is well aligned with human intentions.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We strive to make all experiments in this paper fully reproducible. We share the codebase for evaluation, which is available in the supplementary materials. We describe the full descriptions of the evaluation schemes of the specific environments in Appendices to F.
ACKNOWLEDGMENTS
We thank the Gemini Academic Program and Divy Thakkar for their support. We further thank Roberta Raileanu, Pierluca d'Oro, Mikayel Samvelyan, Sam Devlin, Eric Hambro, and Heinrich Küttler for the insightful discussions.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint drXiv:2303.0877,,222 2023.
Anthropic, Developing a computer use model, 2024. URL
com/
news/developingg computer use Accessed: 2024-11-17
Anthropic. Claude 3.5 sonnet: Enhanced intelligence and versatility, 2024. URL wttp::ttt////./ //www.
sonnet
Accessed: 2024-11-18.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 103216, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279,25 2013.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14166. 2020.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. International Conference on Machine Learning, pp. 3676-3713. PMLR, 2023.
Edward Y Chang. Prompting large language models with the socratic method. In 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0351-0360.
IEEE, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Maxime Chevalier-Boisvent, Boisvert, Dzmitry Babdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations, 2019. URL net Eorum wtt.rrmmmidr..eeeooeeo.
htttss://epenevviie htw.:/rrwwwwee.eeeeoouunnaaaaaaa
Nathan Cloos, Meagan Jens, Michelangelo Naim, Yen-Ling Kuo, Ignacio Cases, Andrei Barbu, and Christopher J Cueva. Baba is ai: Break the rules to beat the benchmark. In ICML 2024 Workshop LLMS and Cognition, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.11668 2021.
11

---

Published as a conference paper at ICLR 2025
Mare-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asni, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Con- junction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pp. 41-75. Springer, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21785 2024.
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309. 2023.
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song- Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint (FXX::2309.0977,, 2023.
Nikolaj Goodger, Peter Vamplew, Cameron Foale, and Richard Dazeley. A nethack learning envi- ronment language wrapper for autonomous agents. Journal of Open Research Software, 11, 06 2023. doi: 10.5334/jors.444.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diagnostic suite for en- tangled language hallucination and visual illusion in large vision-language models. arXiv preprint arXiv:2310. 14566, 2023.
William Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv: 190.. 13400, 2019.
Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arx11:2092:2000007770 2021.
Eric Hambro, Sharada Mohanty, Dmitrii Babaev, Minwoo Byeon, Dipam Chakraborty, Edward Grefenstette, Minqi Jiang, Jo Daejin, Anssi Kanervisto, Jongmin Kim, al. Insights from the neurips 2021 nethack challenge. In NeurlPS 2021 Competitions and Demonstrations Track, pp- 41-52. PMLR, 2022a.
Eric Hambro, Roberta Raileanu, Danielle Rothermel, Vegard Mella, Tim Rocktäschel, Heinrich Küttler, and Naila Murray. Dungeons and data: A large-scale nethack dataset. Advances in Neural Information Processing Systems, 35:24864-2488 2022b.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint MoXiv:2408.084333 2024.
Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktaschel. Open-endedness is essential for artificial superhuman intelligence. arXiv preprint arXiv:2406.044: 04266, 2024.
12

---

Published as a conference paper at ICLR 2025
Dominik Jeurissen, Diego Perez-Liebana, Jeremy Gow, Duygu Cakmak, and James Kwan. Playing nethack with llms: Potential & limitations as zero-shot agents. arXiv preprint (XXx::4403300690 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Xuanfa Jin, Ziyan Wang, Yali Du, Meng Fang, Haifeng Zhang, and Jun Wang. Learning to discuss strategically: A case study on one night ultimate werewolf. arXiv preprint arXiv:2405. 19946, 2024.
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy, Llms can't plan, but can help planning in Ilm-modulo frameworks. arXiv preprint arXiv:2402.01817, 2024.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Ed- ward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive Ilms leads to more truthful answers. arXiv preprint 2xXiv:2402.06782 2024.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in nlp. arXiv preprint arXiv:2104.143: 2021.
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint 1r01v:2310.00166, 2023.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:22 14215, 2022.
Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-aie -augeeteed genera- tion for knowledge-intensst intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 2020.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision- making. Advances in Neural Information Processing Systems, 35:31 199-31212, 2022.
Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforce- ment learning via supervised pretraining. arXiv preprint arXiv:2310.08566 2023.
Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. Lim Lim-powered hi- crarchical language agent for real-time human-ai coordination. arXiv preprint arXiv:2312.15224 2023a.
Xiao Liu, Hao Yu. Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating 1lms as agents. arXiv preprint ar0iv:2308. 03688, 2023b.
13

---

Published as a conference paper at ICLR 2025
Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing the shoulders of giant foundation models. arXiv preprint arXiv:2405.15143; 2024.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang. Duyu Tang, et al. Codexglue: machine learning benchmark dataset for code understanding and generation. arXiv preprini arXiv: 10w.:2200204664,, 2021.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
MetaAI. Llama 3.2:
Revolutionizing edge ai and vision with open, URL https: http://aaaaaaa..aeeeeeeeeeeeaeet meta. com/blog htt-::220.0o.--00-1////-0------- devieeeeee Accessed: 2024- 09-28.
customizable models, 2024.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in; An introduction to circuits. Distill, 5(3):0000240001,, 2020.
OpenAI. Hello gpt-4o, 2024a. htl.://pps//pppell...coo///////s cessed: 2024-09-28.
Ac-
OpenAl. Introducing openai ol preview, September 2024b. URL httphs://ppeeaaa... com/
Index/introducinn -ot.ttooooooddiiooooeoeii.ii -ol ol-preview/ Accessed: 2024-09-27.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744 2022.
Giorgio Piatti, Zhijing Jin, Max Kleiman- Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mihalcea. Cooperate collapse: Emergence of sustainability behaviors in a society of Ilm agents. arXiv preprint 240iv:2404. 16698, 2024.
Grant A Pignatiello, Richard Martin, and Ronald I, Hickman Jr. Decision fatigue: A conceptual analysis. Journal of health psychology, 25(1):123-135, 2020.
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. diff history for neural language agents. Forty-first International Conference Machine Learning, 2024.
Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen. Zenan Zhou, Wentao Zhang, et al. Sysbench: Can large language models follow system messages? arXiv preprint
0oo:2448.11944 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprint arXiv:2307. 20l0:2337.6611 16789, 2023.
Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. arXiv preprint arXiv:2407.0658/ 2024.
Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.061775 2022.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-prod q&a bench- mark. arXiv preprint arXiv:2311 /2022, 2023.
14

---

Published as a conference paper at ICLR 2025
Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, et al. Tptu: Task planning and tool usage of large language model-based ai agents. In NeurlPS 2023 Foundation Models for Decision Making Workshop. 2023.
Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Genewein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations. arXiv preprint arXiv:2412.01 2024.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. arXiv preprint vrXiv:2109.33200 2021.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoer. Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609 2020.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multi- lingual chain-of-thouggh reasoners. arXiv preprint arXiv:2210.03003 2022.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 2(5):9, 2023.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1144) 1140- 1144, 2018
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314. 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonss et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04612 2022.
Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, et al. Towards general computer control: A mul- timodal agent for red dead redemption ii as a case study. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmice Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timessale adaptation in an open-ended task space. arXiv preprini arXiv:2301.07608 2023.
Graham Todd, Tim Merino, Sam Earle, and Julian Togelius. Missed connections: Lateral thinking puzzles for large language models. arXiv preprint arXiv:2404. 11730, 2024.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai. Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9968-95777 2024.
15

---

Published as a conference paper at ICLR 2025
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, nature, 575(7782)3350-352) 2019.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh,
Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 2023.
Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang. Officebench: Benchmarking language agents across multiple applications for office automation. arXiv preprint arXiv:2407.19056, 2024.
Brian Wansink and Jeffery Sobal. Mindless eating: The 200 daily food decisions we overlook. Enviroument and Behavior, 10()):06--133 2007.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0700 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022b.
Maciej Wolczyk, Bartiomiej Cupial, Mateusz Ostaszewski, Michat Bortkiewicz, Michat Zajac, Raz- van Pascanu, Lukasz Kuciáski, and Piotr Miloš. Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem. arXiv preprint arXiv:2402.02888 2024.
Shuang Wu, Liwen Zhu, Tao Yang. Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu. Enhance reasoning for large language models in the game werewolf. arXiv preprint arXiv:2402.02330, 2024.
Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. Smartplay: A benchmark for lims as intelligent agents, arXiv preprint arXiv:2310.01557, 2023.
Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. Understanding the weakness of large language model agents within a complex android environment. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6061 2024.
Yutaro Yamada, Yihan Bao, Andrew K Lampinen, Jungo Kasai, and Ilker Yildirim. Evaluating spatial understanding of large language models. arXiv preprint 1rXiv:2310.14555 2023.
Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Wehshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-20757, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-moda models: Pretraining and instruction tuing. arXiv preprint arXiv:2309.0259/ 2(3), 2023.
Yuhang Zang, Hanlin Goh, Josh Susskind, and Chen Huang. Overcoming the pitfalls of vision- language model finetuning for ood generalization. arXiv preprint 1rXiv:2401.15114 2024.
16

---

Published as a conference paper at ICLR 2025
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307.13854, 2023.
17

---

Published as a conference paper at ICLR 2025
A
BABY AI
Ba Srevaller-Boisverr et al. 2019) is research platform designed to study grounded language learning and instruction following in artificial agents. consists of a suite of 2D grid world envi- ronments with increasing levels of complexity. In these environments, an agent navigates through rooms and interacts with various objects like doors, keys, balls, and boxes of different colors. The agent receives natural language instructions, called "missions" which describe tasks it needs to com- plete, such as picking specific objects or navigating to certain locations. Many existing works on decision-making have studied model performance this environment (Reed et al. 2022 Li al.) 2022). We use it as historically relevant environment that we expect to be relatively easy to solve.
A.1 BABYAI-TEXT
We evaluate the agents on tasks introduced BabyAI-Text (Carta et al. 2023), which provides a description of each observation instead of a symbolic representation. textual description consists of a list of template descriptions with the following structure:
"You a <object> <location>" if the object key, ball, a box wall.
"You see a(n) open/closed door <location>", if the agent sees door.
"You carry a <object>", the agent carries an object.
A.2 BABYAI RESULTS
We provide Ba results for LLM and VLM mode in Tables and Errors are computed with 25 seeds for each of the 5 tasks of Baby AI. GPT-4o leads, closely followed by Llama 3.1 70B. When vision is added to the observation, GPT4o all models performance decrease, except for Gemini-1.5. Pro, whose performance remains stable.
Table 4: LLM Performance on babyai
Table 5: VLM Performance on babyai
Model
Average Progress (%)
Model
Average Progress (%)
gpt-4o
77.60 + 3.73
claude-3.5
82.00 = 5,43
11-0-it
73.20 3.96
1lama-3.2-90b-it
66.00 6.70
llama-3.2-90b-i
72.00 + 6.35
gpt-4o
62.00 4.34
claude-3.5-sonner
68.00 6.60
gemini-1.5-pp 1.5-pro
38.40
gemini-1.5-pro
58.40 + 4.41
fl5-
43.20 4.43
claude-3.5-haiku
52.00 7.07
gpt-4o-mini
38.00 + 4.34
gpt-4o-mini
50.40 + 4.47
1lama-3.2-11b--.
18.00 5.43
gemini-1.5-fie
50.00 7.07
1lama-3.2-1 1b-it
50.00 7.07
llama-3.1-88 1-8b-it
36.00 + 6.79
1lama-3.2-3b-it
20.00 5.66
llama-3.2-1b-it
8.00 + 3.84
A.3 OBSERVATIONS
Example of instruction prompt and observation for Baby AI
18

---

Published as a conference paper at ICLR 2025
You are un agent playing simple navigation gumne. Your goal is to open the yellow dour. The following are the possible actions you take in the game, followed short description each action:
lurn right: turn to the right. go forward:
step forward, pick up: pick up the object below you,
drog: drop the object that you are holding,
toggle:
manipulate the object front you.
moment present you observation.
Tips:
Once the desired objeect you
pickup l'rmnt of you, "loggle" action to interact
with it.
decsn't make repeat action and the observation doesn't change.
PLAY!
a wall 5 steps forward
wall 2 steps left
a rod key stop right and step forward blue key step right
a yellow key 2 steps right and steps forward
step forward red box 2 steps right
green key steps right
a blue ball 3 steps right steps forward blue bux, 3 steps right and forward
a blue box 3 steps right
Image observation provided.
Go forward
B CRAFTER
Crafter (Hafner 2021) is an open-source 2D survival game designed specifically for research on strong generalization, deep exploration, and long-term reasoning in reinforcement learning. It is Minecraft-inspired procedurally generated environment that combines resource gathering, craft- ing, and combat elements. Additionally, the game includes a comprehensive set of tasks and achievements, enabling researchers to evaluate agent performance across multiple objectives and time scales. To enable interaction with language models we use the same language wrapper as proposed in (W et al. (2023).
Figure 3: Crafter's examples of unique procedurally generated maps.
19

---

Published as a conference paper at ICLR 2025
B.1 CRAFTER RESULTS
We provide Crafter results for LLM and VLM format in Tables and 7 standard errors are computed using 10 seeds. GPT4o leads in language-only mode, and Gemini-1.5-Pre leads in vision-language mode. Surprisingly, Llama 3.2 90B performance decreases very sharply when images are added, getting worse average progress than its smaller IB model.
Table 6: LLM Performance on crafter
Table 7: VLM Performance on crafter
Model
Average Progress (%)
Model
Average Progress (%)
gpt-4o
33.10 2.32
claude-3.5-somnc
37.27 3.14
claude-3.5-somnet
32.73 + 3,20
gemini-1.5-pro
33.50 + 2.07
1lama-3.2-90b-it
31.68 1.36
gpt-4o
26.81 + 3.74
1lamb-3.1-70b-it
31.21 + 2.68
gemini-1.5-flash
20.70 + 4.42
gemini-1.5-pro
30.21 2.86
gpt-4o-mini
19.91 3.13
claude-3.5-haikt
26.36 + 2.79
1lama-3.2-1 Ib-it
15.91 + 1.16
1lama-3.2-1 3b-it
26.19 3.29
llama-3.2-90b-it
14.54 + 1.80
Ilama-3.1-888 3.1--b-it
25.45 3.23
gemini-1.5-faar
20.00 0.74
llama-3.2-3b-it
17.27 + 2.79
gpt-4o-mini
15.90 2.05
1lama-3.2-Ib-it
12.73 + 1.91
20

---

Published as a conference paper at ICLR 2025
B.2 OBSERVATIONS
You are agent playing Crafter. The following the only valid actions you take in the followed short description of each action:
Noop: do nothing.
Move West:
ground,
Move East: move cast on flat ground,
Move North: move north on flat ground.
Move South:
south flat ground.
Do: Multiuse action to collect material, drink from lake and hit creature front,
Sleep: sleep energy
maximum.
Place Stone: place stone front.
place
Place Purnace: place furnace.
Place Plant: place plant.
Make Wood Pickaxe: crull wood pickuxe with nearby table and wood inventory.
Make Stone Pickaxe: craft stone pickaxe with nearby table, wood, and stome in inventory.
Make Iron
nearby table l'rrace, Make Wood Sword: craft wood sward with nearby table and wand in inventory.
iron pickaxe
and inventory.
Make Sword: stone
nearby table, wood, stone in inventory.
Make Iron Sword: craft iron sword with nearby table and furnace, wood, coal, and iron inventory.
These are the game achievements you can get:
1. Collect Wood Place Table
3. Bat Cow
Collect Sampling
5. Collect Drink
6. Make Wood Pickaxe
Wood
8. Place Plant
Defeat Zombie
10. Collect Stone
11. Place Stone
12. Plant
13. Defcat Skeleton
15. Make Stone Sword
16. Wake Up
17. Place Furnace
18. Collect Coul
19. Collect Tron
20. Make Iron Pickaxe
22. Collect Diamood
will history actions and observations from game. Your goal is get far as possible by completing the achievements. PLAY!
Current Observation:
Your status:
health: 9/9
Food: ч/У
drink: 9/9
energy:
nothing your inventory.
You seo:
grass sleps your
free steps to your north west steps your
grass at your
Image observation provided.
Move West
21

---

Published as a conference paper at ICLR 2025
B.3 ANALYSIS VLM PERFORMANCE IN THE CRAFTER ENVIRONMENT
To investigate the agentic capabilities of VLMs in visually complex environments, we focused on the Crafter environment, simplified 2D version of Minecraft. The environment uses tiled 2D tex- tures to render scenes, which may be limitation, we introduced an augmented version of the environment where scenes are rendered in 3D using Minecraft's s 3D models and textures. This approach leverages data types more likely to be present in the training datasets of VLMs, given the abundance of interleaved text and images about Minecraft the web.
for many VLMs. To address this potential
Results Surprisingly, the switch to 3D rendering did not improve VLM performance. In some cases, performance on the 3D-rendered environment was slightly worse than on the original 2D textures. This result challenges the hypothesis that familiarity with Minecraft-like 3D data in training datasets would lead to better performance. Instead, it suggests that VLMs may struggle with agentic tasks for reasons unrelated to the type of visual texture or rendering style.
Discussion Our findings indicate that current VLM models may not be well-suited for complex vision-based reasoning tasks, even when provided with familiar visual contexts. The lack im- provement with 3D rendering suggests that factors other than texture familiarity, such limitations in spatial reasoning or task-specific adaptation, may play a more significant role in their underper- formance. While these preliminary results insightful, further experimentation is necessary to draw stronger conclusions.
6
2
1
Figure 4: Crafter's example of 3d scene visualization with Minecraft 3d models and textures.
Table 8: Average Progress of VLMs in Crafter Across Rendering Types
Model
3d textures %
2d textures(
claude-3.5-somee
25,45 1 4.18
37.27 + 3.14
gemini-1.5-pro
30.45 + 3.08
33.50 2.07
gpt-4o
24.55 - 3.65
26.81 3.74
gemini-l.5-flah
19.09 2.78
20.70 4.43
gpt-4o-mini
17.27 + 2.38
19.91 3.13
1lama-3.2-11B-it
15.90 + 1.15
23.63 + 1.48
-lama-3.2-90B-it
12.27 + 1.44
10.00 1.13
22

---

Published as a conference paper at ICLR 2025
C TEXTWORLD
TextWorld (Côté al. 2019) is text-based game environment developed by Microsoft Research that allows for the creation and customization of interactive fiction games. In our experiments, we utilize three specific games from the TextWorld domain: "Treasure Hunter", "The Cooking Game", and "Coin Collector". Each task can be generated with different levels of difficulty by changing number of rooms, enabling obstacles and including distractor rooms. We use the generation rules introduced in Lu et al. (2024).
C.I TREASURE HUNTER
In Treasure Hunter, we create challenging maze-like environment with 20 rooms. The game is set to the maximum difficulty level of 30, introducing locked doors and containers that must be manipulated locate the target object. To increase complexity, we remove the solution description and filter out tasks that can be solved optimally in 20 steps or fewer. This setup requires the agent to navigate complex space, interact with various objects, and devise strategies to overcome obstacles in its quest to find the treasure.
C.2 THE COOKING GAME
The Cooking Game presents culinary challenge set across 13 rooms. We maximize the complexity by including up to ingredients and enabling all additional challenging options. The agent must navigate through doors, process food items using tools like knives, and cook ingredients using var- ious methods such as grilling, fryyngg and roasting. This game tests the agent's ability to plan and execute multi-step processes in dynamic environment, simulating the complexities of real-world cooking tasks.
C.3 COIN COLLECTOR
Coin Collector features an expansive environment with 40 rooms, including potential distractor rooms to increase navigation difficulty. Similar to Treasure Hunter, we remove the solution descrip- tion to enhance the challenge. The optimal path from the agent's starting point to the target is set to 20 steps, requiring efficient exploration and decision decision-making. This game tests the agent's ability to navigate large spaces, avoid distractions, and efficiently reach its goal in a complex, maze-like structure.
You are hungry! Iet's cook meal. Check the coothock in the kitchon for the recipe. Once done, enjog your meal!
Recipe
Rellhe fol loving dollow Ehe directions to peosase tasty
arrive typical kitcben. begin to
Crest
Ingredients:
You closed fridge in the
You anything
carzot
night
handy,
lalliuse
it now. You aue gleam over cosner, where you can counter. Make a note of this, you night have to put stuff on kmife and cookbook ca it nev.
Doane
Directions:
It later. You do
siice the carrot rry the Tectioe nix ingcedients
dooz Loading south.
Dcelbook
crmtth inventory
take knite from counter Taken
What do you want to slice the cATzot with??
the kmifs
[Your score incressed by see point.]
pappor
Figure 5: TextWorld interface along with visualization.
23

---

Published as a conference paper at ICLR 2025
C.4 TEXTWORLD ReSULTS
In Table ??, we provide results for Tex(World. Standard errors are computed using 20 seeds for each of the 3 tasks. GPT-4o once again leads, obtaining more than twice the average progression of its closest competitor Llama 3.1 70B. The coin collector task was by far the most challenging, with GPT-4o managing to solve only once out of 20 attempts. Gemini models' APIs often failed to return completions on textworld, flagging the inputs as "unsafe". despite there being absolutely no real safety concerns the textworld gameplays. This made it impossible to complete full round of evaluation the Gemini models, thus marked them 0% progression.
Table 9: LLM Performance on textworld
Model
Average Progress (%)
glaude-3.5-sonee
42.06 + 5,41
gpt-4o
39.31 + 5.24
claude-3.5-haiku
18.04 5.81
1lama-3.1-70biit
15.00 4.61
gpt-4o-mini
12.25 3.55
1lama-3.2-90b-it
11.18
llama-3.2-11b-it
4.16
1lama-3.1-88-it
6.08 2.41
3.53 1.06
1lama-3.2-1b-it
3.33
semini-1.5-flash
0.00
gemini-1.55 1.5-pro
0.00 - 0.00
24

---

Published as a conference paper at ICLR 2025
C.5 OBSERVATIONS
You agent playing TextWorld, text-based adventure game where you navigate through different rooms, interact with objects, and solve puzzles.
Your goal to first find the recipe, find and prepare food according the secipe, and finally prepere and eut the meall
Here are the available commands:
look: describe the current room goal: print the goal
game
inventory:
player's inventory
go dir: move the player north, east, south or west. You can only go to directions indicated with an exit or door.
examine ...: examine something car ... cat edible food
closely
open open a door container. You to open a closed betore you can go through
drop ..: drop am object onto the floor
.... take an object
put place object supporter
take from ...: take an object from container or supporter
insert into place object into container
lock with ...: lock a door or container with key
...
unlock
or a container
key
cook with ... cook cookable food with something providing heat
...
cuttable
something sharp
chop with chop cuttable food with something sharp
dice with ... dice cuttable fond with something sharp prepare combine ingrodients from inventory into meal.
only prepure meals in
You can examine the cookhook see the recipe when it is visible.
BBQ grilling things, stove for frying things, the oven roasting things. Cooking ingredients the wrong way will lead to a failure of the game. Once you have got processed ingredients and the appropriate cooking tool ready, cook of them according to
the recipe.
'here are two conditions to correctly cook something (gril/ffryrous)):
ingredient you want cook in your inventory
h) there is suitable cooking tool in the rom, and then use 'cook. with command.
When you need to chop'slico/dice ingredients, you need to take knife and ingredient your inventory and then 'stico/chop/dice with knife'
menu). you can 'prepare meal' Make sure to first process the food (chopálico/dice) before you try to cook them. ingredients (that got processed cooked according
you
in the kitchen and then "eal meal' to win the game.
ingredients should EXACTLY would be fine. When 'take.. with ..., use the EXACT name you see.
the
recipe, the recipe 't specify color, any color
You don't need to examine the container/suppotte (c.g. toolbox) when it says something like "there isn't a thing bas nothing it"
steps complete task. Resturting forbidden.
PLAY!
Street
You yourself stroet. An usual kind of place.
There is a closed sliding door leading north. There an exit to the south.
Street =-0/1
open sliding doue
D BABA IS AI
Baba Is AI is a benchmark environment based on the puzzle game "Baba You". In this gridworld game, players interact with various objects and textual rule blocks to achieve specific goals. The unique aspect of Baba Is Al that the rules the game can be manipulated and rearranged by the player, creating dynamic environment where agents must identify relevant objects and rules and then manipulate them to change or create new rules to succeed. This benchmark allows researchers explore a broader notion of generalization compared current benchmarks, as it requires agents
25

---

Published as a conference paper at ICLR 2025
to not only learn and follow the rules but also to combine previously rules in novel ways. Agents are tested on 40 different puzzle levels.
wall
stop
g
door
baba
you
Figure 6: One of the Baba Is Al puzzles, where the agent has to break the "wall is stop" rule, create new rule "door win" and go green door solve the task.
D.1 BABA IS AI LANGUAGE WRAPPER
To enable interaction with language models, we made a custom language wrapper for Baba Is AL It constructs language observation from active rules and creates description by formatting object positions relative to the player. We don't provide the solution for the agent and don't specify grid boundaries in the text-only experiments.
D.2 BABA IS AI RESULTS
We provide the Baba Is AI results for LLM and VLM mode in Tables ?? and ??. Standard errors are computed using 5 seeds for each of the 40 Baba Is AI tasks. Surprisingly, the Llama models lead, with Llama 3.2 90B surpassing GPT-4o by 10% margin in language-only mode. Once again, when vision is added, model performance suffers, with only Gemini-1.5-Pro remaining stable.
Table 10: LLM Performance on babaisai
Table 11: VLM Performance on babaisai
Model
Average Progress (%)
Model
Average Progress (%)
1lama-3.2-90b-4
43.90 3.47
claude-3.5-somme
34.45 + 4.36
11ao--.1--70-iit
40.00 + 3,42
gemini-1.5-pro
31.40 + 3.24
claude-3.5-sonnet
37.50 4.42
llama3.2--00-it
21.90 2.89
gpt-4o
33.66 + 3.30
gpt-4o
18.62 + 2.72
1.5-pn
32.02 = 3.26
gpt-4o-mini
16.41 + 2.59
llama-3. 1-8bitt
18.33 + 3.53
gemini-1.5-flash
8.30 + 1.92
1lama-3.2-3b-it
17.50 + 3.47
1lama-3.2-11b-it
5.76 + 1.63
gpt-4o-mini
15.60 2.53
llama-3,22 1imma-3,.-11bb lb-it
15.60 2.50
cemini-1.5-ffash
12.80 +
1lama-3.2-1b-it
10.83 2.84
claude-3.5-haiku
8.33 + 2.52
26

---

Published as a conference paper at ICLR 2025
D.3 OBSERVATIONS
Baba You is puzzle game where you can manipulate rules of each level. The following are the possible actions you take the gume, followed short description each action:
for step. up: lake one step
right: take one step the right,
down: take step down.
left: take one step to the left.
Tips:
Examine
carefully. noting objects Identify the current rales, which are formed by text blocks in the format "[Suhjecc]] IS [Propert]] (e.g. "BARA
present.
YOU").
-Consider how you change create rules by moving text blocks around. Remember that you can only move objects or text that are not delined "STOP" or similar immovable
properties.
Your goal is usually to reach an object defined as "WIN", this can he changed.
Think creatively
propentes behaviors of abjects unexpected ways. If stuck, try breaking apart existing rales or forming completely new ones.
changing rules
Sometimes solution involves making yourself a different object or changing what counts win condition.
PLAY!
Current Observation:
Active roles:
baba. is you
Objects on the
rule "is" step the left and up rale "win' step up
rule 'eyy' stop the left key I step to the right and 2 steps down ball 2 stops to the right and stops down rule "baba" rule 'is" step to the left and steps down rale "you" steps down rale "ball" steps to the right and steps down
the left and steps down
Image observation provided.
left
E MINIHACK
MiniHack (Samvelyan et al. 2021) is a powerful sandbox framework built on top of the NLE (Küttler et al., 2020) that enables researchers to easily design rich and diverse environments for RL. It provides a flexible platform for creating custom RL tasks ranging from simple grid-world navigation to complex, procedurally generated worlds with intricate game mechanies. The frame- work allows users to define environments using human- -eaaanreedable description language or simple Python interface, giving fine-gra mme-grained control over environment elements such terrain, objects, monsters, and traps. MiniHack offers a diverse array of tasks, which can be broadly categorized into three main groups: Navigation Tasks, Skill Acquisition Tasks, and Ported Tasks. To enable interaction with language models, we use NetHack Language Wrapper described in the NetHack Appendix F
From the MiniHack Navigation Tasks, picked Maze 9x9, Maze 15x15, Corridor and Corridor- Battle, which challenge the agent to reach the goal position by overcoming various difficulties on their way, such fighting monsters in corridors and navigating through complex procedurally generated mazes. These tasks feature a relatively small action space, i.c., movement towards compass directions, and based the environment, search, kick, open, and eat actions.
27

---

Published as a conference paper at ICLR 2025
Figure 7: Examples of MiniHack Corridor task.
Figure 8: Example of MiniHack CorridorBatte task.
From the MiniHack Skill Acquisition Tasks, we picked Quest (with two different difficulty levels, Easy and Medium), which challenges the agent to use objects found in the environment to cross a lava river (these objects provide levitation or freezing abilities), fight monsters, navigate through rooms or mazes and towards the end of the quest use wand of death to defeat a powerful monster guarding the goal location.
Figure 9: Example of MiniHack Quest Easy task.
We additionally test the agents on MiniHack Boxoban. This family of environments is an adaptation of the Boxoban puzzle game, which itself is inspired by the classic Sokoban. These environments present challenging puzzle-solving task within the MiniHack framework, leveraging the NetHack game mechanics. The primary goal in MiniHack Boxoban to push four boulders (MiniHack's equivalent of boxes) onto four designated goal locations, which are represented by fountains. This task requires strategic thinking and planning, as the agent must carefully maneuver the boulders through the environment without getting them stuck in corners against walls.
We provide MiniHack results for LLM and VLM mode in Tables 12 and [3] standard errors were computed using 5 seeds for each task. Here, GPT-4o and Gemini- 1.5-- equal each other both in language-only and vision- laniuage mode, with both models only managing to complete some of the corridor and corridor battle tasks. None of the other models solved any task.
28

---

Published as a conference paper at ICLR 2025
Figure 10: Example of MiniHack Boxoban Hard task.
Table 12: LLM Performance minihack
Table 13: VLM Performance minihack
Model
Average Progress (%)
Model
Average Progress (%)
claude-3.5-sonne
15.00 5.65
claude-3.5-sommet
22.50 + 6.60
claude-3.5-haiku
10.00 4.74
gpt-4o
5.00 1 3.44
gpt-4o-mini
10.00 + 4.74
gemini-1.5-pro
5.00 + 3.44
gpt-4o
10.00 + 4.74
gpt-4o-mini
2.50 + 2.47
1lama-3.1-70b-ir
7.50 4.16
gemini-1.5-flash
2.50 + 2.47
gemini-I.5-flash
5.00
1lama-3.2-1Ib-i
2.50 2.46
gemini-1.5-pro
5.00 3:45
llama-3.2-90b-it
2.50 + 2.47
1lama-3.2-1b-it
+3.45
llama-3.1-8b-1
5.00
3.2-90-
5.00 4.44
1lama-3.2-3b-it
2.50
1.21--t2-1lb-itt
2.50 2.47
29

---

Published as a conference paper at ICLR 2025
E.1 OBSERVATIONS
You agent playing MiniHack. The following the possible actions take game, Jollowed by a short description each action: north: move north, cast: move cast,
south,
west: move west, northeast: meve northeast,
southeast: southeast,
southwest: move southwest, northwest: northwest, far north: move far north. move east.
far south: move far south, far west: move far west. far noriheast:
far northeast. far southeast: move far southenst.
far southwest:
southwest,
far northwest: move far northwest. down: go down stairs,
wait: rest one move while doing nothing.
more: display more tool.
message.
apply: apply
close: close an adjacent door,
something.
force: force a lock.
kick; an enemy
deor or chest,
loot: loot a box on the floor.
open: open un adjacent door.
pickup: pick things the current location there any.
pray: paa to the gods for help.
puton: put
accessory.
quaff: qualf (drink) something.
scarch: search hidden doors and passages,
zap: wand.
moment will present history of actions and observations from the game.
goal explore level, fight monsters, navigate rooms ultimately stairs down.
PLAY!
30

---

Published as a conference paper at ICLR 2025
Current Observation:
statistics:
Strength: 18/18
Dexterity: 15
Constitution:
Intelligence: 8
Wisdom:
Charisma: 8
Depth: I
Gold:
IIP: 15/15
Energy:
AC: 4
XP: 1/0
Time:
Position: 27-9
Hunger: Hungry
Monster Level:
Encumbrance: Unencumbered
Dungeon Number:
Level Number:
Score:
Aligament: Lawful
Condition:
inventory:
katana (weapon in hand)
b: wakizashi (altemaae weapon; not wielded)
yumi
d: 42 +0 (in guiver)
rusipooof splint (being wor)
message:
language observation:
castnortheast. east, urea of lava near eastsoutheast horizontal wall
vertical wall oear southwest and west horizontal wall very near north, northeast, northwest gublin adjacent south
hom adjaceut west adjacent northwest
cursor: Yourself samurai
Image observation proided..
F NETHACK LEARNING ENVIRONMENT
The NetHack Learning Environment (NLE) KKittler al.) 2020) is scalable, procedurally gen- erated, stochastic, rich, and challenging environment designed to drive long-term research in RI. on problems such as exploration, planning, skill acquisition, and language-conditione RI. Built around the classic and highly complex terminal roguelike game NetHack, NLE provides complex and dynamic environment where agents must navigate through procedurally generated dungeons, interact with hundreds of entity types, and lear to overcome various challenges.
The goal the player is to descend through procedurally generated dungeon levels while killing monsters, solving puzzles, and gathering better equipment in order to retrieve the Amulet of Yen- dor and finally ascend back to the surface to win the game. NetHack notoriously challenging, even for human players. Mastering the game take years even with online resources like the
31

---

Published as a conference paper at ICLR 2025
NetHack Wiki. Success in NetHack demands long-term strategic planning, winning game involve hundreds of thousands of steps, as well as short-term tactics to fight hordes monsters. Accurate credit assignment is also crucial to understanding which actions contributed to success or failure. NetHack has already been used extensively as a testbed for RL agents (Wotezyk et al.) 2024 Piterbarg al., 2024] HHmmbo et al. 2022b); tabula-rasa RI. agents particularly struggle due to sparse reward, complex credit assignment, extremely oig-time-horizon, and high stochasticity of the game. The current state-of-the-an agent still remains hand-coded symbolic policy (Hambro et al. 2022a).
F.1 NETHACK LANGUAGE WRAPPER
The NetHack Language Wrapper (Goodger et al., 2023) a tool designed to interface with the NLE and MiniHack by translating non-language observations into text-based representations. This wrapper, converts various NLE observations such as glyphs, blstats, ttt__chharrtecct inv letters, s__sss,, and tty_eursor into readable text equivalents. For example, transforms the visual display of the game environment into a textual description, including details about the surroundings, inventory, and player statistics. The wrapper also supports text-based tions, allowing users to interact with the environment using commands like wait, apply, and north, which then converted into the discrete actions required by the NLE. This functionality enables easier interaction with the NetHack environment, particularly for language models.
Winrata by dungaon level
Winrate by exoerience leval
DM.
level 1
0M12
DMC
0M5
level
008
level
DM
kaxd
DM-1
88-19
BM-15
lovi
Hone
leve 13
BM-21
leve
BM-31
Expellene Lend
BM225
Home 2
DM
level
Home
Invel 17
BM 23
leve
BM:31
BM 33
[M 1
leve 22
BM
lnvel 20
leve
BM
BM:45
BM-49
BM647
leve
leve 25
Oetal
0.5
oftrate
(a) NetHack progression by dungcon level reached
(b) NetHack progression by experience level
Figure 11
F.2 NEW NETHACK PROGRESSION SYSTEM
NetHack features an in-game scoring system that rewards players for actions such as killing mon- sters, identifying objects, eating food, collecting gold and items, and ultimately ascending in the
32

---

Published as a conference paper at ICLR 2025
game. However, argue that this scoring system does not effectively capture true game progres- sion, as players can win the game with scores ranging from a few hundred thousand to several million points. To address this limitation, using a dataset of human-played NetHack games (Hambro et al. 2022b). Specifically, we recorded the dungeon levels and experience levels achieved in each game, well as whether the game sulted in an ascension. Utilizing these statistics, we constructed data-centric progression system where each data point represents the probability of human player winning the game after reach- ing specific dungeon level or experience level. The resulting progression curves are presented in Figure [I] For practical purposes, we define Dungeon Level I (DIvI:1) and Experience Level 1 as representing 0% progression, corresponding to the game's starting point, and ascension as 100% progression. The agent's overall progress is thus determined by the highest progression achieved between the dungeon level and experience level attained.
developed novel, data-informed progression metric
F.3 NETHACK RESULTS
We provide NetHack results for LLM and VLM mode in Tables 14 and 15 computed using 5 seeds. ol-preview achieves the highest progression out of all the tested models. However, it is still very far from making any significant progression in the game. The best indi- vidual run was achieved by Gemini-L.5-Pro vision-language mode, reaching dungeon level 3 and experience level 4.
Standard errors are
Table 14: LLM Performance on nle
Table 15: VLM Performance on nle
Model
Average Progress (%)
Model
Average Progress (%)
op-preveew
I.57 0.40
claude-3.5-sonnet
1.16 + 0.42
claude-3.5-haide
1.16 + 0,42
gemini-1.5-pro
0.48 + 0.48
claude-3.5-sommet
0,58 = 0.52
gpt-4o
0.37 + 0.37
gpt-4o
0.37 = 0.37
gpt-4o-mini
0.00 + 0.00
gemini-1.5-pro
0.37 + 0.37
gemini-1.5-flash
0.00 + 0.00
llama-3.1-70b-it
0.35 + 0.35
1lama-3.2-11br
0.00 + 0.00
gpt-4o-mini
0.00 0.00
llama-3.2-90b-it
0.00 + 0.00
gemini-1.5-flash
0.00 +.00
1lama-3.2-1b-it
0.00 +.00
1lama-3.2-3b-it
0.00 0.00
llama-3.1-8b-i.
0.00 0.00
1lama-3.2-1 3.2--1
0.00 0.00
1lama-3.2-90b-it
0.00 + 0.00
F.4 OBSERVATION
Despite having language wrapper that describes its observations (Goodger et al., 2023). NetHack is not meant to be played with language only, thus provided the ASCII map language mode and the RGB tiles map in vision-language mode, In the LLM context, we only keep information that important be kept in the long term, i.e., the game message and language observation. Agent stats and inventory are only needed in the current step, so we do not keep them in the context. This done also prevent the context length of NetHack to explode out of control.
33

---

Published as a conference paper at ICLR 2025
You are an agent playing Neillack.
The following the possible actions you take in the game, followed short description each action:
north: move north, cast: move cast,
move south,
mortheast: move northeast, southeast: southeast, southwest: move southwest, northwest: move northwest, far north: move far north, move east. lar south: move south, far west: move far west, northeast, far southcast: move far southcast. southwest:
far northeast:
wuihwest.
far nortwwes:: move far nontwest, up: go up a staircase,
down: go down staircase (tip: you only go down you standing the stairs),
wait: rest one move while doing nothing,
more: display more
message (tip: ONLY when current message cads with -Moro-).
annotate: leave note about the level,
apply: apply (use) a tool,
call: name a monster or object, or add an annotation,
cast: cast a spell.
close: close adjacent donr,
open: open an adjacent door,
dip: dip object semething.
drog: drop an item.
droptype: drop specific types (specify next prompt),
ear: eat something (tip: replenish food when hungry),
esc: exit menu or message.
engrave: engrave writing the floor (tip: Elbereth),
cahancc: advance or check weapons skills,
fire: fire ammunition from quiver, fight: fight a monster (even if you only guess one is there),
force: lock, [..]
read: read scroll or spellbook.
accessory.
rub: rub a lamp or a stone.
and passages, swap: wielded and secondary weapons,
hidden
takcoff: take one piece armor,
takeoffaall: take off all armor,
teleport: teleport to another level (if you have the ability).
throw: throw something (e.g. dagger dart),
travel: travel to specific location on the map (tip: in the next action, specity or for stairs, for fountain, and
for allur).
twowespon: toggle two- weapon combat, untrap: untrap something.
wear: wear piece of armor.
wield: wield a weapon.
wipe: wipe your face,
zag: wand.
Tips:
When message asks completion, such as: "What de you want to cat? [d ?*1". you should respond catluse.
with single character corresponding the item you For example. "What do you want to eat? (dgh or 2*]" -g Possible answers are "d", "g". or "h" to eat the associated food.
When the message asks for direction, such as: "In what direction?" you should respond with direction. the end, your next
the message
"more" to see the rest of message.
Explore the environment find the stairs down to the next level.
Always carefully read the last message to understand the current state the game and decide your next action
accurdingly.
If you keep moving in the same direction, you will eventually hit wall and stop moving. Your message might be: "It's solid stone", "It's wall", Change your action to in another direction continue exploring the cavironment.
Read language observation carefully look ascii map or image observation provided to next action to take and where to move next.
You can attack monsters by moving into them.
moment I will present history of actions and observations from the game. Your goal get possible the game.
PLAY!
34

---

Published as a conference paper at ICLR 2025
Current Observation:
statistics:
Strength: 14/14
Dexterity: 9
Constittionn 11
Intelligence: 8
Wisdom:
Charisma: 18
Depth:
Gold: 0
16/16
Energy:: 6/6
AC:
XP: 1/0
Time: 1
Hunger: Not Hungry
Monster
Encumbrange: Unencumbered
Dungeon Number:
Level Number:
Score:
Aligmment:
Condition: None
inventory:
a: +1 long sword (weapon in hand)
lance (alternate weapen; wielded)
am uncursed +1 ring mail (being wern) an uncursed helmeet (heing wor) an uncursed +0 small shield (being worn)
f: uncursed +0 pair of leather gloves (being worn]
11 uncursed apples
h: 10 uncursed carrots
iessage:
Salutations Agent,
are lawful Knight.
language observation: vertical closed door far dark area far cast
scroll
eastsoutheust
horizontal wall near southeast and south area west
vertical wall very
southwest
vertical open door very near west
horizontal wall adjacent north, northeast, and northwest
tame pany adjacent southeast
cursor:
Yourself knight
Image observation provided.
WESE
F.5 LLMS POSSESS EXTENSIVE NETHACK KNOWLEDGE
Agents need extensive knowledge to play NetHack effectively. Skilled players leverage their under- standing of the game, acquired through hours of gameplay, while new players often rely written
35

---

Published as a conference paper at ICLR 2025
resources online, such the NetHack Wiki. We use questionnaire test if LLMs possess some of this knowledge and assess agents can apply The questions are as follows:
1. Are there any consequences of eating rotten food in NetHack? (Answer: It might lead to blindness, hallucination or even instant death.)
2. Are there any consequences of eating too much food in NetHack? (Answer: The player character might choke on the food and die.)
3. What happens when you ascend the stairs level of NetHack? (Answer: leads to instant game over.)
4. Is it good idea to use an invisibility wand on a monster in NetHack? (Answer: No, as will be harder to fight them.)
5. Is it good idea to pray to an angered god NetHack? (Answer: No, the god will punish the player.)
These questions test a fundamental understanding of the game mechanics, particularly focusing on behaviors that new players may mistakenly attempt and should be avoided.
We summarize each LLM's responses in the tables below. For each question, we assess whether the response is accurate, whether the conclusion is correct (i.e., the LLM recognizes the need to avoid such behavior), and whether the agent successively avoids these mistakes.
LLM
Q1
Q2
Q3
Q4
Q5
Correct
V
GPT 4o
Conclusion
V
J
Behaviour
X
X
N/A
Correct
x
GPT 4o-mini
Conclusion
Behaviour
X
N/A
N/A
Correct
X
X
X
Gemini 1.5-flash
Conclusion
J
X
X
Behaviour
N/A
N/A
Correct
X
Gemini 1.5-pro
Conclusion
Behaviour
X
N/A
N/A
Correct
x
X
J
Llama 3.1 70B Instruct
Conclusion
x
Behaviour
X
x
X
X
X
Correct
X
x
Llama 3.2 11B Instruct
Conclusion
V
X
Behaviour
X
X
X
N/A
N/A
Correct
V
2
X
Llama 3.2 90B Instruct
Conclusion
V
Behaviour
X
N/A
N/A
Table 16: Comparison of each LLMs (ability to apply) knowledge in Nethack. We manually grade the responses to each question based on the correctness of the response given (i.e. does the response match information from the Nethack wiki), the correctness of their conclusion (i.e. does the LLM correctly identify that such behaviour should be avoided), and whether LLM agent's behaviour during evaluation consistent with the ground truth (i.e. does the agent successfully avoid the behaviours indicated in the questions). For answers that are partially correct, we award ~v, We record behaviour as N/A when the agent does not encounter scenarios where knowledge of the corresponding question should be applied.
36

---

Published as a conference paper at ICLR 2025
We observe that while generally the LLMs understand to avoid common mistakes, regardless of whether their reasoning is completely correct, they still struggle to consistently exploit that knowl- edge. Agents will often consume rotten food and prematurely exit the game by ascending the steps on the first level. This illustrates a gap between LLM agents ability to exploit knowledge in practice.
37


Reviews:
[{"review_id": "y4nZN9NTz8", "reviewer": "Reviewer_LP97", "summary": "The paper presents AgentQuest, a benchmark to evaluate LLMs and VLMs on complex, long-horizon tasks. By consolidating multiple reinforcement learning environments (e.g., NetHack, TextWorld, BabyAI), AgentQuest provides a unified testbed for assessing skills like spatial reasoning, long-term planning, and exploration. Findings show that while LLMs perform adequately on simpler tasks, they struggle with complex ones, especially when handling visual inputs. AgentQuest highlights gaps between current model capabilities and human-level performance, encouraging further research to enhance agentic qualities in AI.", "questions": "1. How might the benchmark adapt to new multimodal architectures that integrate more advanced vision processing, such as video-based reasoning?\n\n2. Could the “knowing-doing” gap (where models have knowledge but fail in practical application) be addressed by incorporating memory or reinforcement learning methods in future iterations of the benchmark?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. AgentQuest integrates diverse environments, testing a wide range of agentic skills from navigation to complex reasoning, making it a well-rounded benchmark.\n\n2. The benchmark’s emphasis on agentic capabilities addresses key limitations in current LLMs and VLMs, which are crucial for autonomous AI applications.\n\n3. The benchmark’s metrics allow for detailed analysis of model performance, identifying specific areas like spatial reasoning and long-term planning where models struggle.", "weaknesses": "1. It is weird that VLMs underperform when visual information is included. It indicates that current architectures may not be well-suited for complex vision-based reasoning. Or the current prompt is not suitable for the model. I think this point deserves a careful study.\n\n2. More long-context models should be included for evaluation, for example Qwen2, Llama-3-8B-Instruct-80K, LongAlpaca-7B, LongChat-v1.5-7B-32k.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRUStBrlb7", "reviewer": "Reviewer_pjfN", "summary": "The paper introduces AgentQuest, a suite of six reinforcement learning environments for testing the agentic capabilities of long-context LLMs and VLMs. The benchmark includes diverse environments, ranging from simple tasks such as BabyAI, to complex tasks such as NetHack. The authors provide baseline evaluations of state-of-the-art LLMs/VLMs on AgentQuest using zero-shot prompting. The authors perform a qualitative analysis of the results across capabilities and identify an intriguing knowing-doing gap where the models cannot employ the knowledge they possess. Finally, the authors develop an open-source toolkit for AgentQuest.", "questions": "- Does the AgentQuest support multi-agent LLM decision-making?\n- How does AgentQuest deal with LLM hallucination? For instance, the LLM/VLM returns an invalid action.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive experiments. This paper is well-supported by comprehensive experiments and a diverve selection of environments that cover a wide range of agentic challenges. \n- The proposed benchmark incorporates several typical environments and could facilitate rapid verification of LLM/VLM-based decision-making. Considering that existing LLM/VLM-based decision-making methods are evaluated by their own tasks or environments, such a unified benchmark will be beneficial and valuable to the whole community.\n- The authors provide analysis into the experimental results on AgentQuest, including Spatial Reasoning, Long-term planning, Discovering and Leveraging Environment Dynamics, and Knowing-Doing Gap. The analysis provide insight into this problem, and could facilitate relevant research in this area.\n- The writing is clear and easy to follow. The motivation is clearly presented.\n- The code is provided to make the benchmark reproducible.", "weaknesses": "- More related decision-making environments should be discussed and compared. For instance, the authors include the Crafter in the benchmark, however there are also Minecraft environments such as Minedojo[1] and MineRL[2]. There are also LLM-based multi-agent cooperation benchmarks such as Overcook[3]. \n- More visualizations of the environment, agent behavior, and agent trajectories can be provided so that the analysis can be more intuitive.\n- Does the AgentQuest support multi-agent LLM decision-making?\n- How does AgentQuest deal with LLM hallucination? For instance, the LLM/VLM returns an invalid action.\n- More cases should be provided for the limitations of VLMs.\n\n\n[1] Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., ... & Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35, 18343-18362.\n\n[2] Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., & Salakhutdinov, R. (2019). Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440. \n\n[3] Liu, J., Yu, C., Gao, J., Xie, Y., Liao, Q., Wu, Y., & Wang, Y. (2023). Llm-powered hierarchical language agent for real-time human-ai coordination. arXiv preprint arXiv:2312.15224.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ltlL9knovj", "reviewer": "Reviewer_HZcx", "summary": "This paper introduces a new benchmark designed to assess the agent ability of both LLMs and VLMs, with a particular focus on their performance in complex environments. By incorporating diverse tasks and refined metrics, this benchmark highlights the strengths and limitations of current models, especially in visual decision-making tasks where shortcomings are evident. Evaluation results indicate that while current models perform well on simple tasks, they continue to face significant challenges with complex ones. AgentQuest provides a platform for advancing research in intelligent agents.", "questions": "Creating a comprehensive benchmark to test model agent capabilities is very meaningful; however, the authors' work has certain limitations. It would be helpful if the authors could provide further explanation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. AgentQuest integrates various reinforcement learning environments, covering tasks from simple to highly complex, to comprehensively evaluate the agency capabilities of LLMs and VLMs.\n\n2. Different metrics have been designed to provide quantitative analysis of model performance across multiple dimensions, including interactivity, spatial reasoning, and long-term planning.\n\n3. According to the authors, AgentQuest is convenient for researchers to expand and apply, helping to advance the field of intelligent agents and improve model usability in complex, dynamic environments.", "weaknesses": "1. The main issue is that this paper, as a benchmark for evaluating the agent capabilities of models, only assesses base models. Many recent studies focus on enhancing models’ agent capabilities, but this paper does not evaluate these works. While base models are important, they weren’t specifically trained to improve agent abilities. This may be one reason why they perform poorly on many tasks. Evaluating the latest agent-focused methods would better reflect the field's progress.\n\n2. As far as I know, this paper simply combines several existing benchmarks without further integration. While bringing existing benchmarks together to evaluate model performance may be important, it doesn’t qualify as innovative work. The authors need to provide further explanation on this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2wEXQkI8Q", "reviewer": "Reviewer_SuiJ", "summary": "This paper presents a benchmark to evaluate LLMs' capabilities in video games. Six video games are included. One metric that captures how close the LLMs are to completing a task is proposed for evaluation. State-of-the-art LLMs are used for evaluation. The paper presents findings from quantitative and qualitative experiments.", "questions": "1. What does the \"reasoning\" refer to in L72 \"action = self.llm_cilent(reasoning)\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation of this benchmark is timely.\n2. The paper presents problems about the knowing-doing gap, which might spark more research.", "weaknesses": "My major concern is the comprehensiveness of the proposed benchmark. Current VLMs or MLLMs are usually trained on image-text pairs in the real world. The appearance of the involved video games might differ largely from the training data. It is not very surprising, at least to the reviewer, that providing game images leads to worse performance. More scenarios, especially general ones, should be involved to support the conclusion. This can also further reflect the overall capability of MLLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)