BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

Authors: Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, Tim Rocktäschel
Primary area: datasets and benchmarks
Keywords: LLM, VLM, Agents, Benchmark, RL, Reasoning, Games

Abstract:
Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies—areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). 
We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as several models perform worse when visual representations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community. Code and Leaderboard at balrogai.com

Decision: accept (poster)
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729515194786

thread_type:
official_review

<thread_object>
creator: reviewer_suij

<content>
summary:
This paper presents a benchmark to evaluate LLMs' capabilities in video games. Six video games are included. One metric that captures how close the LLMs are to completing a task is proposed for evaluation. State-of-the-art LLMs are used for evaluation. The paper presents findings from quantitative and qualitative experiments.

strengths:
1. The presentation of this benchmark is timely.
2. The paper presents problems about the knowing-doing gap, which might spark more research.

weaknesses:
My major concern is the comprehensiveness of the proposed benchmark. Current VLMs or MLLMs are usually trained on image-text pairs in the real world. The appearance of the involved video games might differ largely from the training data. It is not very surprising, at least to the reviewer, that providing game images leads to worse performance. More scenarios, especially general ones, should be involved to support the conclusion. This can also further reflect the overall capability of MLLMs.

questions:
1. What does the "reasoning" refer to in L72 "action = self.llm_cilent(reasoning)"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640435020

thread_type:
official_review

<thread_object>
creator: reviewer_hzcx

<content>
summary:
This paper introduces a new benchmark designed to assess the agent ability of both LLMs and VLMs, with a particular focus on their performance in complex environments. By incorporating diverse tasks and refined metrics, this benchmark highlights the strengths and limitations of current models, especially in visual decision-making tasks where shortcomings are evident. Evaluation results indicate that while current models perform well on simple tasks, they continue to face significant challenges with complex ones. AgentQuest provides a platform for advancing research in intelligent agents.

strengths:
1. AgentQuest integrates various reinforcement learning environments, covering tasks from simple to highly complex, to comprehensively evaluate the agency capabilities of LLMs and VLMs.

2. Different metrics have been designed to provide quantitative analysis of model performance across multiple dimensions, including interactivity, spatial reasoning, and long-term planning.

3. According to the authors, AgentQuest is convenient for researchers to expand and apply, helping to advance the field of intelligent agents and improve model usability in complex, dynamic environments.

weaknesses:
1. The main issue is that this paper, as a benchmark for evaluating the agent capabilities of models, only assesses base models. Many recent studies focus on enhancing models’ agent capabilities, but this paper does not evaluate these works. While base models are important, they weren’t specifically trained to improve agent abilities. This may be one reason why they perform poorly on many tasks. Evaluating the latest agent-focused methods would better reflect the field's progress.

2. As far as I know, this paper simply combines several existing benchmarks without further integration. While bringing existing benchmarks together to evaluate model performance may be important, it doesn’t qualify as innovative work. The authors need to provide further explanation on this point.

questions:
Creating a comprehensive benchmark to test model agent capabilities is very meaningful; however, the authors' work has certain limitations. It would be helpful if the authors could provide further explanation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649751998

thread_type:
official_review

<thread_object>
creator: reviewer_pjfn

<content>
summary:
The paper introduces AgentQuest, a suite of six reinforcement learning environments for testing the agentic capabilities of long-context LLMs and VLMs. The benchmark includes diverse environments, ranging from simple tasks such as BabyAI, to complex tasks such as NetHack. The authors provide baseline evaluations of state-of-the-art LLMs/VLMs on AgentQuest using zero-shot prompting. The authors perform a qualitative analysis of the results across capabilities and identify an intriguing knowing-doing gap where the models cannot employ the knowledge they possess. Finally, the authors develop an open-source toolkit for AgentQuest.

strengths:
- Comprehensive experiments. This paper is well-supported by comprehensive experiments and a diverve selection of environments that cover a wide range of agentic challenges. 
- The proposed benchmark incorporates several typical environments and could facilitate rapid verification of LLM/VLM-based decision-making. Considering that existing LLM/VLM-based decision-making methods are evaluated by their own tasks or environments, such a unified benchmark will be beneficial and valuable to the whole community.
- The authors provide analysis into the experimental results on AgentQuest, including Spatial Reasoning, Long-term planning, Discovering and Leveraging Environment Dynamics, and Knowing-Doing Gap. The analysis provide insight into this problem, and could facilitate relevant research in this area.
- The writing is clear and easy to follow. The motivation is clearly presented.
- The code is provided to make the benchmark reproducible.

weaknesses:
- More related decision-making environments should be discussed and compared. For instance, the authors include the Crafter in the benchmark, however there are also Minecraft environments such as Minedojo[1] and MineRL[2]. There are also LLM-based multi-agent cooperation benchmarks such as Overcook[3]. 
- More visualizations of the environment, agent behavior, and agent trajectories can be provided so that the analysis can be more intuitive.
- Does the AgentQuest support multi-agent LLM decision-making?
- How does AgentQuest deal with LLM hallucination? For instance, the LLM/VLM returns an invalid action.
- More cases should be provided for the limitations of VLMs.


[1] Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., ... & Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35, 18343-18362.

[2] Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., & Salakhutdinov, R. (2019). Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440. 

[3] Liu, J., Yu, C., Gao, J., Xie, Y., Liao, Q., Wu, Y., & Wang, Y. (2023). Llm-powered hierarchical language agent for real-time human-ai coordination. arXiv preprint arXiv:2312.15224.

questions:
- Does the AgentQuest support multi-agent LLM decision-making?
- How does AgentQuest deal with LLM hallucination? For instance, the LLM/VLM returns an invalid action.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730729159493

thread_type:
official_review

<thread_object>
creator: reviewer_lp97

<content>
summary:
The paper presents AgentQuest, a benchmark to evaluate LLMs and VLMs on complex, long-horizon tasks. By consolidating multiple reinforcement learning environments (e.g., NetHack, TextWorld, BabyAI), AgentQuest provides a unified testbed for assessing skills like spatial reasoning, long-term planning, and exploration. Findings show that while LLMs perform adequately on simpler tasks, they struggle with complex ones, especially when handling visual inputs. AgentQuest highlights gaps between current model capabilities and human-level performance, encouraging further research to enhance agentic qualities in AI.

strengths:
1. AgentQuest integrates diverse environments, testing a wide range of agentic skills from navigation to complex reasoning, making it a well-rounded benchmark.

2. The benchmark’s emphasis on agentic capabilities addresses key limitations in current LLMs and VLMs, which are crucial for autonomous AI applications.

3. The benchmark’s metrics allow for detailed analysis of model performance, identifying specific areas like spatial reasoning and long-term planning where models struggle.

weaknesses:
1. It is weird that VLMs underperform when visual information is included. It indicates that current architectures may not be well-suited for complex vision-based reasoning. Or the current prompt is not suitable for the model. I think this point deserves a careful study.

2. More long-context models should be included for evaluation, for example Qwen2, Llama-3-8B-Instruct-80K, LongAlpaca-7B, LongChat-v1.5-7B-32k.

questions:
1. How might the benchmark adapt to new multimodal architectures that integrate more advanced vision processing, such as video-based reasoning?

2. Could the “knowing-doing” gap (where models have knowledge but fail in practical application) be addressed by incorporating memory or reinforcement learning methods in future iterations of the benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731953541238

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their thoughtful feedback, which has been invaluable in enhancing the quality of our work. We are glad that reviewers recognized that AgentQuest is a timely (Reviewer **SuiJ**) well-rounded (Reviewer **LP97**), and comprehensive benchmark (Reviewers **pjFN**, **HZcx**), encompassing a diverse selection of environments testing a wide range of skills (Reviewers **LP97**, **pjFN**), with metrics allowing for detailed analysis of model performance (Reviewers **LP97**, **HZcx**). Reviewers also agreed that AgentQuest could facilitate rapid verification and spark more research (Reviewers **pjFN**, **SuiJ**) that would be beneficial to the whole research community (Reviewer **pjFN**).

To further strengthen the comprehensiveness of our evaluation, we have now added the following models, as also requested by Reviewers **LP97** and **HZcx**
- Claude 3.5 Sonnet, 
- Llama-3.1/3.2 variants (1B/3B/8B)-it. 
- Qwen2.5 (7B/72B)-instruct, 
- Qwen2 VL (7B/72B)-instruct, 

You can view all the results in our updated paper, as well as in our fully anonymous leaderboard https://agentquestanon.github.io/ which will be released to the public upon open-sourcing.

One common question raised by reviewers relates to the underperformance of some Vision-Language Models (VLMs) in AgentQuest. Reviewer **LP97** noted that this deserves careful study, Reviewer **pjFN** suggested providing additional cases, and Reviewer **SuiJ** highlighted the potential impact of out-of-distribution (OOD) visual inputs. We fully agree that understanding VLM performance is crucial and appreciate the opportunity to clarify.

Not all VLMs underperform in AgentQuest. While some models, such as Llama 3.2, struggle to effectively leverage visual inputs in decision-making tasks, other models, such as Gemini 1.5 Pro and the newly evaluated Claude 3.5 Sonnet, demonstrate improved performance in VLM mode, effectively utilizing added visual inputs. This diversity of performance highlights the importance of AgentQuest in identifying specific strengths and weaknesses in current architectures and guiding future research.

AgentQuest, as a benchmark, is designed to discover precisely these kinds of gaps in model capabilities. Its role is not only to evaluate current models but also to identify areas where improvement is needed. These findings align with recent work [1][2][3][4][5], which has documented similar challenges with VLMs. Thus, we view this as a strength of AgentQuest, highlighting its value as a tool for advancing research in this domain.

We have more thoroughly addressed each reviewer’s individual concerns in their respective rebuttals.

Once again, we thank the reviewers again for their constructive feedback, which has enabled us to strengthen our paper. We hope that the enhancements and clarifications we have made have addressed your questions and that the reviewers will consider increasing their scores to further support the acceptance of our paper.

Kind regards,

The Authors


### **References**
[1] Tan, Weihao, et al. "Towards general computer control: A multimodal agent for red dead redemption ii as a case study." ICLR 2024 Workshop on Large Language Model (LLM) Agents. (2024).

[2] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Rahmanzadehgervi, Pooyan, et al. "Vision language models are blind." arXiv preprint arXiv:2407.06581 (2024).

[4] Zang, Yuhang, et al. "Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization." arXiv preprint arXiv:2401.15914 (2024).

[5] Guan, Tianrui, et al. "HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models." arXiv preprint arXiv:2310.14566 (2023).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a benchmark to evaluate LLMs and VLMs on complex, long-horizon tasks and evaluates it in multiple reinforcement learning environments. While the finding the somewhat expected that LLMs perform well on simple tasks and struggle on tasks with visual inputs, the paper is a good step toward benchmarking the capabilities of the existing models. All of the reviewers recognize the contribution of this work. The main weaknesses are requesting evaluation on more models and request for more analysis on the result. The authors will also make the benchmark publicly available with a leaderboard. Overall, the AC thinks there is enough contribution from the paper, however, the paper's finding is somewhat expected and there isn't a strong technical contribution component.