SmartPlay : A Benchmark for LLMs as Intelligent Agents
Authors: Yue Wu
Xuan Tang
Tom Mitchell
Yuanzhi Li
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Primary area: datasets and benchmarks
TL;DR: We introduce SmartPlay, a benchmark from 6 distinct games augmented with language descriptors for visual observation, offering up to 20 different settings.

Abstract: Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks a systematic benchmark for evaluating LLMs' abilities as agents. We introduce SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs as agents. SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges a subset of 9 important capabilities of an intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness. The distinction between the set of capabilities each game test allows us to analyze each capability separately.
SmartPlay serves not only as a rigorous testing ground for evaluating the overall performance of LLM agents but also as a road-map for identifying gaps in current methodologies. 
We release our benchmark at https://github.com/microsoft/SmartPlay

Published as a conference paper at ICLR 2024
SMARTPLAY: A BENCHMARK FOR LLMs AS INTELLI- GENT AGENTS
Yue Wu 22 Xuan Tang', Tom Mitchell', Yuanzhi Li Carnegie Mellon University, *Microsoft Research
ABSTRACT
Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks systematic benchmark for evaluating LLMs' abilities as agents. We introduce SmartPlay: both challenging benchmark and a methodology for evaluating LLMs agents. SmartPlay consists of different games, including Kock-Paper-Sciisone of Hanoi, Minecraft. Each game features unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges subset of important capabilities of intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness. The distinction between the set of capabilities each game test allows to analyze each capability separately. SmartPlay serves not only as a rigorous testing ground for evaluating the overall performance of LLM agents but also as a road-map for identifying gaps in current methodologies. We release our benchmark at algithub.com// lgg.ibub.com/mccrroooooooooo /icroosott/Smmmalla
Tower
INTRODUCTION
SmartPlay Unified Text API
AI
Block
Auto gpt
AgentGPT.
Figure 1: SmartPlay provides a unified and expandable API with text observations and guidance to perform turn by turn LLM inference Two-armed Bandits, Rock Paper Scissors, Messenger (Hanjie
et al. 2021). Crafter (Hafner 2021), and Minecraft (Fan et al. 2022) creative navigation tasks.
Creating intelligent agents (Wooldridge & Jennings 1995). that perceives its environment and per- form autonomous actions, has been one of the core objectives A.I. (Laird et al.) 1987 Russell
2010) Recently, large language models (LLMss) (Smith et al. 2022 Chowdhery et al. 202 Ope-
2023; Manyikal Driess et al. 2023; Touvron et al. 2023) have made remarkakb progress in various tasks (Bubeck et al. 2023). Some language models demonstrate exceptional planning (Ahn
ett al. 2022 Wu et al. (023b) reasoning (Wu et al., 2023a Shinn et al. 2023). and problem-
solving (Madaan al. 2023 Kim et al. 2023) abilities, enabling the potential generalist agents
for virtual-reality PPark et al.
2023 or real-world problem-solving.
Such potential has attracted strong interest on applications where LLM systems actively invoke tools and APIs to complete a wide range of tasks goals Eegnificant-Graiine Yoheinakajima Reworkd
Work done during internship Microsoft Research. Correspondence to wwwus@nndewwcmmu.el

---

Published as a conference paper at ICLR 2024
Wang et al. 2023a; Qin al. 2023). and actively interact and make changes in environment to achieve specific results Wang et al. 2223b a Wu et al. 2023b c). LLMs as agents could be seen as an important step toward next-gen automation.
Despite great public attention, the capabilities of LLMs as agents have not been systematically studied, partly due to the lack of standardized LLM benchmark for agent-environment interaction. Current LLM benchmarks have been designed for static knowledge and reasoning (Hendrycks et al.)
2020: Liang et al. 2022 Srivastava et al.] 2022a
Zhong et al.) (023), or helpful and harmless conversations (Bai et al. 2022 Zheng et al. 2023a Dubois el al. 2023). overlooking applications to intelligent agents.
We note 4 key challenges for intelligent LLM agents not captured in previous benchmarks. First, lots of real-world tasks require an agent to do long-horizon planning. Second, many events are probabilistic and an intelligent agent is expected to understand the odds. Third, an intelligent agent needs spatial reasoning to understand our 3D world. Fourth, when encountered with unseen situations, an intelligent agent should be able to learn from interactions or mistakes.
On the other hand, games have long been identified as go-to benchmarks for intelligent generalist agents (Pell 2011 Genesereth et al. 2005
Whiteson et al. 2010, Schaul et al. 2011 Bellemare
jet al. 2013] Côté et al. 2019: Hafner 2021] Guss et al. [002 Fan et al. 2022). At the core of game design (Koster 2013), successful games often involve "problem-solving
"calculation of odds", "spatial reasoning", "changing difficulties", and "well-defined and quantifiable outcome", therefore offering perfect complement to existing LLM benchmarks. Finally, some game environments procedurally generated and game states grow exponentially, making games more robust against evaluation dataset contamination as observed in recent works (Touvron al. 2023). Experimentally, we observe LLMs struggle to memoize intermediate states of a simple 3-disk Tower of Hanoi game.
Taking a unique agent perspective in benchmarking L.LMs, introduce SmartPlay, benchmark from 6 distinct games augmented with language descriptors for visual observation (Figure[I. offering up to 20 different settings and infinite environment variations. Each game presents unique challenges that span multiple dimensions of intelligent agents, as detailed in Table 3 The games range in complexity. from requiring simple one-step reasoning and follwwoln in Bandits, to intricate long-term planning, multi-hop dependencies, and learning from interactions in Crafter (Hafner (021) and Hanoi. SmartPlay engages LLM agents in both deterministic and stochastic settings, demanding skills from basic text understanding to 3D spatial reasoning.
Games in SmartPlay have been built with well-defined objectives and evaluation metrics: completion rate, reward, score. Therefore, SmartPlay provides fully automated pipeline to conduct standardized evaluation for LLMs. We use SmartPlay to compare the agent performance of recent LLMs, and identify several research gaps for applying LLMs as agents. We believe that SmartPlay sets goal that is reachable in a short tine-fram yet formidable to require new breakthroughs.
2 CAPABILITIES NECESSARY FOR INTELLIGENT AGENTS
Borrowing concepts from game design Koster 2013), we identify 9 key abilities important for intelligent L.M agents, and identify multiple degrees for each capability:
a) Long text understanding: general LLM capability.
We define degrees based on document length and syntactic variations: 1) few fixed lines,
2) few fixed paragraphs, 3) with syntactic variations, 4) and longer than page (500 words).
b) Reasoning: multi-hop logical reasoning and deduction, often required for analyzing the interac- tions of in-game objects or action conditionssdeeenndencis..
We define degrees based on reasoning hops: 1) (0 1). 2) (2 ~ 3), 3) (> 3).
Instruction/Rule following: follow rules and instructions set by environment or users.
We define 3 degrees based on number of game rules: 1) single rule, 2) (< 5), 3) (5+)
d) Planning: long-horizon in-context planning achieve complex goal.
We define degrees based on planning steps, and concurrent objectives which requires goal prioritization: 1) < 5 planning steps, 2) 5+ planning steps, 3) concurrent objectives
e) Generalization: Excels wide range of tasks.
2

---

Published as a conference paper at ICLR 2024
We define 3 degrees based the variability the game provides: 1) fixed environment, 2) fixed game word with random objectives, 3) procedurally generated game world f) Understanding the odds: analyze and estimate the probability of random events.
We define 3 degrees based on the importance randomness in the environment: 1) no randomness, 2) randomness present in game, 3) randomness as core game mechanism
g) Learning from interactions: acquire environment knowledge from live interactions.
We define 4 degrees based on the number of unique interactions to learn from: 1) no learning required, 2) single interaction, 3) interactions, 4) 5+ interactions
h) Error/Mistake handling: recover from mistakes (e.g., correcting from erroneous trajectory).
We define degrees based on if mistake handling may be necessary and if additional reasoning and re-planning is necessary: not required, 2) simple rollback corrects error, 3) reasoning and re-planning required to correct error.
i) Spatial reasoning: understand our world in 2D/3D. Spatial reasoning is typically required to understand directions and navigate through the game world (e.g., navigating the 2D/3D world).
(a) We define degrees based on dimensionality: 1) 0 ~ 1D, 2) 2D, 3) 3D
GAMES IN SMARTPLAY
3.1 RESEARCH CHALLENGES
The SmartPlay benchmark encapsulates diverse set of challenges that evaluate various AI capabil- ities, as itemized in Figure 2 For instance, Bandits primarily focuses on understanding the odds, requiring minimum text understanding and rule-following. On the other hand, Rock Paper Scissors uniquely puts an emphasis on understanding the odds and multiple game rules. Hanoi presents an advanced setting for object dependency reasoning, strategic planning, and handling mistakes. Messenger puts challenge on 2D spatial reasoning, reading syntactic variations and conducting multi-hop reasoning. Meanwhile, Minecraft offers a unique challenge in 3D spatial reasoning and generalization within a procedurally generated world. We hope the SmartPlay benchmark would serve as a tool for identifying these nuanced gaps and directing future research.
While each game poses its unique challenges, the SmartPlay benchmark also evaluates an agent's capability to integrate these skills. For example, Crafter stands as the most comprehensive testbed, combining long texts, multiple interactions, concurrent objectives, and error handling into a single environment. Crafter highlight the need for future research to focus not just on isolated skills, but also on combining these skills into unified, adaptive agent.
3.2
Two ARMED BANDITS
The two armed bandit benchmark is inspired by popular implementation off bandit problems.
The LLM agent is provided two slot machines with hidden pre-defined reward probabilities p1: P2- For slot machine i, the reward for the two possible out-comes are: for pay-off event and -r; for no-pay-off event. The goal of the game is find the arm with better return and maximize the reward over the course of 50 rounds. The human written manual informs the LLM of the number of slot machines (two) and the objective.
An agent must keep track of win/losses from its past roll-out and balance exploration across the two slot machines vs. exploitation of the more rewarding one. Overall, the challenges include: 1) long context understanding. 2) understanding randomness, 3) learning from interactions.
To prevent game exploitation caused by biased actions, we randomize the score and probabilities for each action by shuffling the order of the paired list: [(pi,Ti), (p2, r2)].
3.3 ROCk PAPER SCISSORS
Same rules as the famous zero-sum game Rock Paper Scissor:
ggghhab.omm/KKCooorz/y/mmmaaa
wikipeddia.org//ii//RRkkk paper scissors
3

---

Published as a conference paper at ICLR 2024
Long Text Understunding
Machine
Machine
few pre defined lines
few paragraphs
syntactt variations
longer than page
Multi-hop Reasoning
0.16
0.21
1-hop
3-hop
taxbo
multi hop
Instruction/Rule Following
single game rule
2. 5 gime rules
fot ded
Dey
3. G++ game rales
Planning
<: 5 planning steps
2. 5++ planning sleps
Lerv TOW troug
concurrest objectives
Generalizatin
fixed environment
weeld, rundom objective
procedurally generated world
Understanding the Odds
randomness
minivmness present game
randomness mecha-
si-eent puerren cruch
Learning from Interactions
Bu ciaper rading may appatiic pollig sorical
no learning requinal
Ac neny is dewly
9-8 11
5
2. single interaction
Pedla podttoosteeeee
1. < 5 inserattios
Oulio
5-+ interactions
Rrooe/Mistake Handling
not roquieal
rollback only
reason and replan
Spotial Reasoning
ressoning
2D jeasoning required
2
3D reasoning required
Figure 2: We identify a set important capabilities (section 2 for an intelligent agent. We identify different degrees of challenge for each capability as shown on the left. Each game in SmartPlay challenges unique set of capabilities at different degrees, as shown in the spider charts. We include numerical values of the spider plots in Table 3
The LLM agent plays against hand-coded opponent that follows a hidden pre-defined strategy with probabilities P1.P2. Ps for rock, paper, and scissors respectively. The scores for winning under each action is pre-defined and revealed to the LLM as 81, 82, 83. The human written manual provides instruction on the possible actions and how the win/draw/lose of each round is calculated.
An agent must keep track of win/losses from its past roll-outs to analyze the opponent behavior, and then exploit the opponent to maximize payoff. Overall, the challenges include: 1) long text understanding. 2) understanding the odds, 3) learning from interactions, 4) instruction following.
To prevent game exploitation caused by biased actions, we randomize the score and probabilities for each action by shuffling the order of the paired list: [(P1, (p2, 82), (a),
3.4 TOWER OF HANOL
The Tower of Hano from one rod to another, using third rod as an auxiliary. The game has two rules: only one disk can be moved at time, and larger disk cannot be placed on top of a smaller one.
B is a classic puzzle game that challenges the player to move a stack of disks
The goal of the game is to move all the disks from the first rod to the last one in the minimum number of moves, and the game can be solved using recursive algorithm that follows these steps:
4

---

Published as a conference paper at ICLR 2024
Move u
source rod muxiliary using destination rod an internediate.
2. Move the largest disk from the source rod to the destinution rod.
3. Move n I disks from the auailiary nod to the desination rod. using the source rud an intermediate.
The human written manual contains a description of the game set-up and allowed actions. addition, we also include an example illustration of the starting and goal configuration, alongside an example of allowed/disallowed moves.
The Tower of Hanoi requires the agent to think strategically and plan ahead, and put strict requirements on the L.M agents' ability to understand and follow the rules of the game. The game can become more challenging if the agent makes mistake. Sometimes, an agent may have to undo several moves to correct an error. Overall, the challenges include: I) planing, 2) reasoning, 3) error handling.
3.5 MESSENGER
MESSENGER (Hanjie et al. 2021) which features multiple game variants with procedurally generated game dynamics and accompanying text manuals. The overall game mechanics of MESSENGER involve obtaining a message and delivering difficulties (referred as stages in Hanjie et al. (2021)) )
a goal. The benchmark shipped with 3 levels of
To succeed in MESSENGER, an agent must first relate entities and dynamies of the environment to their reference synonyms in the manual, identify message and goal objects, and navigate to bring the message to the goal while avoiding the enemy. The manual, by design, is challenging to understand even for human readers. Level primarily challenges the agent's 's 1) long text understanding and 2) generalization. Level 2 includes additional challenge on the agent's 3) reasoning, and 4) 2D spatial reasoning. Level increases difficulty by adding distraction objects.
The original manuals provided by Hanjie et al. (2021) contain descriptions of the entities and world dynamics obtained through crowd-sourced human writers. We augment the manual with specification on the game objective, and an "advice" for LLM agent to first identify goal objects and then approach its objective. The "advice" reduces the difficulty of the hard-to-parse manual.
3.6 CRAFTER
The Crafter environment (aafnr 2021) is procedurally generated, open-world survival game designed to test RL algorithms. Inspired by Minecraft, features a grid-world with top-down observation and a discrete action space of 17. The game includes 22 achievements in tech-tree of depth 7 and provides information on the player's health, food, water, rest levels, and inventory. Crafter captures many of Minecraft's key research challenges, offering more streamlined and faster environment for conducting experiments and gathering results.
We provide the "context" string from Wu al. (2023c) as the manual, generated by parsing the и8т source-code of (Hafner 2021). The "context" string has been shown to greatly improve performance of GPT-4 and text-davinci-003 on Crafter (Wu et al. 2023c).
To succeed in Crafter an LLM agent has to first understand and master a variety of reusable skills composed of 17 actions. The agent needs to learn navigate through up 2D terrains (biomes), avoiding obstacles and dangerous creatures. The agent also needs to collect different resources and craft more advanced weapons/tools to unlock more skills and achievements, while the same time balance crafting goals with survival goals like maintaining health, thirst, food, and rest (Hafner 202T). Overall, the challenges include: 1) 2D spatial reasoning. 2) mistake handling, 3) long text understanding. 4) planning. 5) generalization, 6) correcting from mistakes. Interestingly, the "context" string does not capture all information necessary to succeed the game, i.c., it requires 2 woods to craft the crafting table, and 8 stones craft the furnace. The agent has to 7) learn from interaction.
3.7 MINECRAFT
Minecraft is one of the most popular games in history The game world is virtually infinite and procedurally generated. The game observation is composed of rough 3D objects representing various materials, such as dirt, stone, ores, tree trunks, water, and lava. Minecraft has been widely studied as
wwkippdii..rgg/iiii/innneaii
5

---

Published as a conference paper at ICLR 2024
benchmark for intelligent 2023; Yuan et al. 2023; Wang et al., 2023b a). However, due to the fact that most current LLMs do not have vision capabilities,
agents (Guss al. 202 Fan al. 2022 Hafner et al.
simplity the Minecraft benchmark (Fan et al. 2022) and only
consider a small set of creative tasks where the primary objective is to find specific biomes, so an LLM could control hand-coded agent to perform navigation in the 3D world.
For the human written instruction manual, we inform the agent that its goal is to find a specific biome g in Minecraft, and offer advice how interpret the visual descriptor output for Minecraft.
To succeed in the creative "find" tasks, LLM agent has to have enough domain knowledge about different biomes in Minecraft, and be able to correlate visual observation (text description of visual world) with domain knowledge, and navigate in 3D environment. Overall, the challenges include: 1) planning, 2) domain knowledge, 3) 3D spatial reasoning, 4) generalization.
4 USING SMARTPLAY
4.1
ENVIRONMENT INTERFACE AND EVALUATION PROTOCOL
Env
Input
Manual
History
Rollout
Action Space
Trials
Bandits
Text
Background
50
50
20
RockPaperSeissors
Text
Background,Rules
50
50
20
Hanoi
Text
Background,RulesEExamppp
30
30
6
10
Messenger
Visual description
Background. Background.Rules.Addie Advice
2
4~~128
5
100
Crafter
Visual description
Background,
5
10k
17
10
Minecraft
Visual description
Objective
200
Table 1:
Specifications for each game in SmartPlay. In addition to the table, the manual input contains list available actions for all games. Input, manual, action space, and rollout length should not be modified. History length and trial numbers could be increased to suite future needs.
For ease of use and wide compatibility, SmartPlay follows unified OpenAI Gym interface (Brockman cet al.] 2016) for all games, with text-based observations, text-based manuals with content as described in Table text describing historical actions and observations covering past steps of length "history length", and flat categorical actions. Due to the randomness in some games, we recommend running each game multiple times and reporting the average metrics.
Input, manual, action space, rollout length (the maximum environment steps allowed by each game), and trial numbers for each game are specified in Table [ These settings are fixed and should not be modified. However, future research may require longer history length or more trials for some games. These parameters can be adjusted to suit specific needs, but the changes should be explicitly stated. We provide recommended values (also used in our experiments) for the parameters in Table I
For completeness, we provide example inputs for each game in Appendix C Note that all directions in SmartPlay are described in "east, south, west, north, above, below" In the actual gameplay, SmartPlay API also includes list of actions for the LLM agent to pick from.
4.2 EVALUATION METRICS
We define three metrics: reward, completion rate, score. To ensure compatibility with prior works, reward aligns with the score/reward definition in games originally designed for RL (i.e., Bandits, Rock Paper Scissors, Messenger, Crafter (Hanjie et al.) [021]] Hafner 202I)). Completion rate measures the rate of successful completion for games with quantifiable objectives (i.e., Hanoi, Messenger, Minecraft). Finally, we introduce score for every game in the benchmark to provide summary of performance. For Bandits and Rock Paper Scissors, the score is defined the number of times the LLM action matches the environment optimal action; for Hanoi, the score is defined as the number of disks successfully moved to the goal peg; for Messenger, the score is the same as the reward (Hanjie al. 2 of each round of game; for Crafter, the score is defined the number of unlocked achievements at every step, summed across the whole game; for Minecraft, the score is defined as the indicator of whether the "find" objective for the game has been completed.

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTAL RESULTS
Using the SmartPlay API, we follow Wu et al. (2023c) and directly prompt an LLM: "What the next action take, let's think step by step.", with manual, history, and current observation as context. We then query the LLM: "Choose the best executable action from the list of all actions. Write the exact chosen action." " for an answer directly mapped to one of the environment actions.
5.1 QUANTITATIVE ANALYSIS
LLM
Bandit
RPS
Hanoi
MessengerL.
MessengerL.2
Crafter
Minecraft
Human Baseline
L.00
1.00
1.00
1.00
1.00
1.00
1.00
GPT-4-0613
1.00
0.91
0.83
0.90
0.93
0.26
0.61
GPT-4-0314
0.97
0.98
0.90
0.87
0.97
0.32
0.59
lext-davinci-003
1.04
0.40
0.50
0.62
0.46
0.07
0.45
Claude
0.72
0.47
0.67
0.44
0.60
0.05
0.50
Bard
0.86
0.30
0.67
0.61
0.40
0.04
0.54
llama-2-13b
0.50
0.35
0.37
0.12
0.13
0.04
0.61
Ilama-
0.68
0.50
0.33
0.16
0.06
0.04
0.50
vicuna-I
0.64
0.17
0.07
0.00
0.12
0.02
0.43
Table 2:
Comparison of performance of different LLMs in terms of average score on Bandir/wooAmeeHiihh........... Low.ieidoo...xxd.... RockkPaperScissorBasiee.0 Hanoi3Disk-v0, Messengerl.1- vO, Messengerl.22 .2-00, Crafter-vO, Minedojo All scores are normalized relative to human performance (unnormalized version Table 4 GPT-4 variants out-perform other LLMs by signifi- cant margins, but still greatly under-perform human baselines. We observe significant performance gaps between SOTA LLMs and human baseline on Hanoi, Crafter, and Minecraft. Hanoi, Crafter challenges planning and reasoning, and Minecraft challenges 3D spatial reasoning.
To reduce the cost of queries, we pick 7 settings that requires minimal experimentation but provides comprehensive coverage over important agent capabilities. We experiment with 9 recent popular open-source and proprietary LLMs and report the average score in Table 2
Overall, GPT-4 variants significantly out performs other proprietary models, which outperform open-source models by significant margins.
There is still significant room for improvement for LLM as agents: Despite the impressive performance of GPT-4 variants, there is still significant gap between GPT-4 and human baseline performance on more challenging benchmarks, with a 10% gap on 3DiskHanoi, 40% on Minecraft creative tasks, and 70% on Crafter.
Other proprietary LLMs struggle to keep up with GPT-4: We observe a more than 20% gap between GPT-4 and other proprietary models like Claude, Bard, and text-davinci-003 across all games except Minecraft. Furthermore, on comprehensive benchmarks like Crafter, GPT-4 variants achieves 3 times higher scores than other proprietary models.
Open-source LLMs have a long way to go: Open-source LLMs achieves less than half the perfor- mance of GPT-4 variants on simple Bandit and Rock Rock-Paper- Pape--aappsssseeeee Scissors tasks, and 1/8 the performance on more challenging tasks. The fine-tuned Vicuna- 3b model performs much worse than the base ILAMA- 13b.
The Minecraft benchmark appears equally challenging to all LLMSs due its unique requirement for 3D spatial reasoning. All LLMs behave similarly in Minecraft creative tasks, with the best model at 60% of human baseline performance.
3D Spatial reasoning remains challenge for LLMs:
To offer additional insights into the individual agent capabilities of L.MSs as identified in Figure 2 we compute, for each capability the capability score pf PLLM of an L.I.M as the average of human
normalized score s, over each game g. weighted by the degree dg at game g presents challenge c:
Σ, d2 We plot the capability scores into groups Figure proprietary models, and open-source models.
PLLM Σ, de
GPT-4 variants, other

---

Published as a conference paper at ICLR 2024
Pentar Bodds
lot NET.ORRS 003
Teme 2.32
1833
FALNOW ToRlwing
Este relceing
EvAnlard
Log LnAnieery
Lor Toi JeArladdy
Leas sayrd kansoding
Loariig
Leaning
Figure 3: Left: comparing the two GPT-4 variants with Human Baseline performance reference. Middle: comparing text-davinci-003, Claude, and Bard. Right: comparing open-source llama-2-130 Ilama-138 13b, vicuna-11 13b models.
The two GPT-4 variants perform similarly overall, with GPT-0614 doing slightly worse on planning and reasoning. We also identify that GPT-4 variants score lower on learning from interactions, error/mistake handling, and spatial reasoning.
Claude demonstrates overall better performance than Bard, especially in planning, reasoning, in- struction following. Compared to the other two proprietary models, text-davinci-003 appears biased toward learning from interaction and randomness, and is particularly weaker at instruction following. planning and reasoning.
LLAMA-2-13b and LLAMA-1-13b performs similar on the high level, with LLAMA-2-133 perform- ing better at plamning, reasoning, and error handling, but worse in learning from randomness and interactions. Vicuna- 13b loses a lot of reasoning, planning, long text understanding, and error/mistake handling capabilities after fine-tuning.
5.2 QUALITATIVE ANALYSIS
Learning from interactions: In Bandits and Rock Paper Scissors, proprietary LLMs demonstrate promising potential for learning from history and interactions. We observe the agents first following a exploratory strategy and then exploiting the biased opponent based on the past observations. In Crafter, GPT-4 variants consistently attempts build crafting table with wood and recovers from the failure to build crafting table with 2 woods.
For the Tower of Hanoi, it is expected that the LLMs have been trained on the exact same problem. Surprisingly, although all LLMs are able to provide the solution at the starting configuration where all disks are on the first rod (some may even write out the recurrence for the solution). most LLMs could not solve the problem and gets confused quickly after a few moves, where the disks are distributed over all three rods. We suspect that this due to the intermediate states do not appear often in the LLM's training sets. Such observation verifies our belief that games could be more robust to dataset contamination.
Data/environment contamination:
Spatial Reasoning: We observe that LLMs often have bad sense of spatial locations and struggle with navigating to new locations. For example, Minecraft, we often observe LLMs often take moves that are contradictory over time, i.e., bunch of "move north" followed by bunch of "move south", undoing lot of its own efforts at exploration.
RELATED WORKS
6.1 LLM EVALUATION
The task of evaluating LLM performance has become increasingly challenging given the rapid progression of LLMs. Generalist benchmarks usually employ a wide range of tasks and languages to test general knowledge and reasoning (Hendrycks et al. 2020: Liang al. 2022 Srivastava et al. 2022a Zhong et al. 2023). where small language models are getting close performance compared to the state-of-the-ee the-te--rtttttt large language models Li al. (2023); Gunasekar et al. (002)) Eldan & Li (2023). However, those benchmarks struggle to cover interaction styles like instruction following Ziegler et al. (009)) conversations Bai al. (2022). The go-to approach for evaluating L.M for conversation
8

---

Published as a conference paper at ICLR 2024
is pairwise model comparison, which performs pair- wise comparison of output of the LLM and reference LLMs to produce a ranking (Zheng et al. 2023b). The ranking was originally performed by human, but could be automated with significantly more powerful LLM (Chiang & Lee 2023 Zheng et al., 2023a Dubois et al. 2023). However, such evaluation techniques depend on an expert model or human who can reliably compare the performance of different L.Ms,, which limits the application to SOTA LLMs like Claude-2 or GPT-4. Moreover, existing benchmarks fail to capture key characteristics of intelligent agents like understanding of randomness, spatial reasoning, and error handling.
6.2 USING GAMES TO EVALUATE GENERALIST AGENTS
The idea of using games to evaluate the performance of agents has a long history in A.I. Pell (2011) Schaul et al. 2011): : Whiteson al. (2011) presented early ideas and motivation for using games to measure the general capabilities of an agent, and discussed challenges in measuring A.I. agent performance. A series of popular benchmarks (Brockman et al., 2016 Vinyals et al. 2017
Tunyasuuunakool et al. 2020) were created including Atari (Bellemare et al. 2013) and MeepMind lab (Beattie et al., 2016). As the capabilities of A.I. agents improve, researchers developed open- ended generalist games (Savva et al. 2019; Abramson et al., [0200] Hafner 2021 Srivastava et al. 2022b) like NetHack (Küttler et al. 2020) or Minecraft (Guss et al.
2021 Fan al. (022).
SmartPlay takes a suite of benchmarks (Brockman et al. 2016 Hafner 2021
Fan et al. 2022
developed over different times best represent a broad range of difficulties and skills.
6.3 CREATING/CONVERTING TO TEXT GAMES
Text games (Côté al. 2018: Küttler et al. 2020: Zhong et al. 2019 Hanjie et al., 2021
interactive simulations where the game state and action space are in natural language, often used to benchmark skills like planning, exploration, and memory. SmartPlay features text game (Messenger) with procedural game rule generation (Hanjie et al., 202T) to test the generalization of the LLM agents language understanding and planning.
To capture real-worid challenges like spatial-reasoning, we study converting 2D/3D games into text-games, [Shridhar et al. (2020b) demonstrated the possibility of converting 3D embodied indoor environment (Shridhar et al., 2020a) into TextWorld (Côté et al. 2018) game by "listing" all the objects in text. However, such conversion relies on low-level controllers and teleportation, trivializing the environments for current LLMs (Micheli & Fleuret 2021; Wu et al. 2023b). Therefore, we follow Wu et al. (2023c) to offer a list of objects/observations with directional relationship to the agent: "to your south-east." Such description allows LLMs to make meaningful progress without low-level controllers (Wu et al. 2023c).
CONCLUSION
In this work, we introduce SmartPlay, both challenging benchmark and methodology for evaluating LLMs' performance as agents. Our initial release of SmartPlay consists of Two-armed Bandits, Rock Paper Scissors, Messenger (Hanjie et al. 202T). Crafter (Hainer 2021). and Minecraft (Fan et al. 2022) creative navigation tasks. SmartPlay benchmarks not only basic abilities like instruction following and in-context reasoning, but also evaluates capabilities like planning, understanding of randomness, 2D/3D spatial reasoning, and error handling, which are often underrepresented in existing LLM benchmarks. To achieve next-gen automation, we believe that language models should go beyond speaking fluent language (Eldan & Li (2023), and become more intelligent agents that could interact with the world and human users. We hope that SmartPlay would catalyze research on building more capable and reliable LLM agents.
Finally, SmartPlay offers guidelines for easily adding games to the benchmarking suite. SmartPlay will be continuously improved to provide up-to-date challenges for next-gen I.LMs.

---

Published as a conference paper at ICLR 2024
REFERENCES
Josh Abramson, Arun Ahuja, lain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoe, Andrew Dudzik, al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05677 2020.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint urXiv:2204.0169) 2022.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 2022.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefraneq, Simon Green, Víctor Valdés, Amir Sadik, al. Deepmind lab. arXiv preprint arXiv:1612.03801. 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01/54 2016.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2rXiv:2303.12712 2023.
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXhv:2305.01937, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas. Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla Asri, Mahmoud Adada, et al. Textworld: learning environment for text-based games. In Workshop on Computer Games, pp. 41-75. Springer, 2018.
Mare-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asti, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. Computer Games: 7th Workshop, CGW 2018, Held Con- junction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pp. 41-75. Springer, 2019.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid. Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0337 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback, 2023.
Ronen Fldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07753 2023.
Linxi Fan, Guanzhi Wang. Yunfan Jiang. Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang. De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-enu open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.0888 08853, 2022.
Michael Genesereth, Nathaniel Love, and Barney Pell. General game playing: Overview of the aaai competition. Al magazine, 26(2):62-6) 62, 2005.
10

---

Published as a conference paper at ICLR 2024
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint 2.Kv:23006.1644. 2023.
William H Guss, Mario Ynocente Castro, Sam Devlin, Brandon Houghton, Noboru Sean Kuno, Crissman Loomis, Stephanie Milani, Sharada Mohanty, Keisuke Nakata, Ruslan Salakhutdinov, et al. The minerl 2020 competition sample efficient reinforcement learning using human priors. arXiv preprini 1rXiv:2101.1107) 2021.
Danijar Hafner. Benchmarking the spectrum agent capabilities. arXiv preprint arXiv:2109.06780 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0411 2023.
Austin W Hanjie, Victor Y Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In International Conference Machine Learning, pp. 4051-0544062 PMLR, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint urXiv:2009.03300 2020.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arXiv:2303. 17491, 2023.
Raph Koster. Theory of fun for game design. O'RReilly Media, Inc.", 2013.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.
John E Laird, Allen Newell, and Paul S Rosenbloom. Soar: An architecture for general intelligence. Artificial intelligence, 33(1):1-64 1987.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Texthooks all you need ii: phi-1.5 technical report. arXiv preprint wxXiv:230.055633 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprini arXiv:221 11.221100000 1.09/ 10, 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprini arXiv: 17651, 2023.
James Manyika. An overview of bard: early experiment with generative ai. https: / Accessed: May 27, 2023.
google/ http:///sttttcccccccttteet wttp:t/sssssoooeeebtoob--eeeee
Vincent Micheli and François Fleuret. Language models are few-shot butlers. arXiv preprint arXiv:2/04.07972, 2021.
OpenAl. Gpt-4 technical report, 2023.
Joon Sung Park, Joseph C O'Brien, Carric J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra human behavior. arXiv preprint 0344:2230..33442,, 2023.
Barney Pell. Strategy generation and evaluation for meta-game playing. KI-Kinstlichhe Intelligenz. 25(1):71-72, 2011.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
11

---

Published as a conference paper at ICLR 2024
Reworkd. reworkd/agenigpt: Assemble, configure, and deploy autonomous ai agents in your browser.
URL http:///hhii..oooooooo//////////
Stuart Russell. Artificial intelligence a modern approach. Pearson Education, Inc., 2010.
Manolis
Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans. Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9339-9347, 2019.
Tom Schaul, Julian Togelius, and Jürgen Schmidhuber. Mcasuring intelligence through games. arXiv preprint arXiv: 109.1/99.1/34 2011.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 2023.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zeutlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020a.
Mohit Shridhar, Xingdi Yuan, Hausknecht, Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03776 2020b.
Côté, Yonatan Bisk, Adam Trischler, and Matthew
Significant- Gravitas. Grrnvicant-Gravitas...
suaffiaant-gaavitssatt---ops An experimental open-source attempt to make gpt-4 fully autonomous. URL /tLlhtts:///gtthaa wwt.///igiiilicaatttttttataaa
GPT
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train meegttron-trring NLG 530b, A large-scale generative language model. CoRR, abs/2201.1 11990, 2022. URL https: / arxiv...... org/ abs 22012221200 11990
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.046155 2022a.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on Robot Learning, pp. 477-490. PMLR, 2022b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020. ISSN 2665-9638. doi: https://doi. rect com/ cemmm/ccieneee
org/10.1016//jsimppt 2020. Axticee///1/22265533000000
2020. 100022. https:/ts: http://wwww...ic.eeee.cm /
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yea, Alireza Makhzani, Heinrich Kütttler, John Agapiou, Julian Schrittwieser, al. Starcraft ii: new challenge for reinforcement learning. arXiv preprint arXiv: 7008.0000000 04788, 2017.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint ar (0l.::3335.1299 2023a.
12

---

Published as a conference paper at ICLR 2024
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables ppen-world multi-tash agents. arXiv preprint arXiv:2302.01560, 2023b.
Shimon Whiteson, Brian Tanner, and Adam White. Report on the 2008 reinforcement learning competition. Al Magazine, 31(2):81-8) 1-81, 2010.
Shimon Whiteson, Brian Tanner, Matthew E Taylor, and Peter Stone. Protecting against evaluation overfitting empirical reinforcement learning. In 2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 120-127. IEEE, 2011.
Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice. The knowledge engineering review, 10(2):115-552 1995.
Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. arXiv preprint arXiv:2302.0444 04449, 2023a.
Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, and Shrimai Prabhumoye. Plan, eliminate, and language models are good teachers for embodied agents. arXiv preprint arXivv2305.02412. 2023b.
Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li. Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning. arXiv preprint arXiv:2305.15886 2023c.
Yoheinakajima. babyagi
yoheinakajima/babyagi
URL betpesr/ggtthubb.co//ooeeeeeeee
Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint ar arXiv:2303. 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprini arXiv:2306. 05685, 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-jue auddae---udde with mt-bench and 2023b.
chatbot arena. arXiv preprint arXiv:2306.0566
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dynamics via reading. arXiv preprint arXiv: 191iv:1910.082111 2019.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai La, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centri benchmark for evaluating foundation models. arXiv preprint ar0iv:2304. X00:22304.06664. 2023.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint
2019.
13

---

Published as a conference paper at ICLR 2024
A RESEARCH CHALLENGES
Games
Bandits
Rock Paper Scissors
Hanoi
Messenger1.2+
Crafter
Minecraft
Long text Understanding
2
3
fow pre-defined lines
few paragraphs
syntactic variations
longer than page
Reasoning
3
2
0 1-hop
2. 2 3-hop
3. multi-hop
Instruction/Rule Following
1. single game rule
2. < 5 game rules
3. 5+ game rules
Planning
3
2
< 5 planning steps
5+ planning steps concurrent objectives
Generalization
2
1
2
3
fixed environment
2. fixed world, random objective
3. procedurally generated world
Understanding the Odds randomness
2
2
2. randomness present in game
3. randomness as core mechanism
Learning from Interactions learning
2. single interaction
3. 5 interactions
4. 5+ interactions
Error/Mistake Handling not required rollback only
2
2
reason and re-plan
Spatial Reasoning
2
3
ID spatial reasoning
2. 2D reasoning required
3. 3D reasoning required
Table 3: Research challenges associated with each of the 6 games. Since MessengerL does not cover multi-hop reasoning, only 1.2+ is included.
B MINECRAFT VISUAL DESCRIPTOR
The ground truth MineDojo FFnn al. 2022) is block level matrix (2D matrix for lidar rays and 3D matrix for surrounding blocks), which very hard for human LLMs comprehend. Inspired by Wu et al. (2023c), we adopt a directional visual description scheme to encode the scene observation in text. Speciticall, we first run connected component algorithm to group the same blocks that are connect into groups, and then describe the group's is relative position to the agent based on the closest block from the group. For example, observation in Figure will be descrihed
14

---

Published as a conference paper at ICLR 2024
Block Names
20
40
40
50
200
10
340
22.5
17.5
Figure Left: Raw Minecraft environment observation Right: Segmentation map of the environment observation. detected by lidar rays in MineDojo (Fan et al. 2022
grasa, 3.87 blocks away to your bottom, taking 388 of screen
plant, blocks away your bottom-right right, taking
screen
water, 33.68 blocks away
your bottom-left, taking 18 of screen wood, 18.76 blocks away to your bottem left, taking 08 of screen
fflowe, 32.98 blecks away to your sight, taking 0t of screen
-wood, 24.50 blocks away to your left, taking 04 of screen taking leaves, 56.55 blocks away to your zight, taking 1$ of sczeen
leaves, blooks away
C EXAMPLE INPUTS
C.I BANDITS
Example input
Instruct Manual:
You are in the casino with sloz machines in front of you. Your goal is to try to earn the most from these slot machines.
observation example:
pulled slot machine sis, recaived reward $r_is.
C.2 ROCK PAPER SCISSORS
Example input
Instruction Menual:
the game Reck Paper scissers, you and opponent choose one three options: rock, paper, or scissors. After both p ayera have chosen, the winner is determined as follows
Rock crushes scissors (Reck wins, scoze $a_i$)
Scissors paper (Scissors win,
$s_25)
saper covers rock (Paper wins, score $s_39)
you lose, your
is the negatiive of the w'rner's
II both players choose the same option, itfs a draw (acore 0).
Your goal to maximize your soore.
Observation Example:
chose $Rack$, and the apponent chose $8cissor8, You Swons and received $s_15. Naw round bagins.
C.3 HANOI
Example input
Instruction Menual:
The game consists of three rods (A,8,C) and number of disks of various sizes, which can onte any rod.
order decreasing size, amallest at tha top (righthand side) The objective ls to move the entire stack rod c, obeying the following rules:
game begins
stacked
only disk be moved time.
Each move consists of taking the cop disk from one of the stacks and placing it on top of another stack
an empty zod.
You cannet place bigger dick en top of smaller disk.
15

---

Published as a conference paper at ICLR 2024
example, considering movements
follewing setting:
-A: Ibottom, [0], topl
Ibotton, [[], topl
C: Ibottom, [2], topl
You are only allowed to meve from B to C but not A, since the top of (1) is smaller than the tep C (2)
bigger than the of (0).
rinally, starting configuration
A: Ibottom, [2,1,91] topl
[botLom, [] topl
[bottom, [],
the geal configuration 1s: A: lbottom, Ur topl
Ibottom, [], topl
C: |bottom, 12,1,9], topl
with top on the zight and botton on the left
Observation Example:
triec
top diak rod top Current configuration:
A: IbotLom, [2, 1, C], topl
Ibottom, [] topl
C: Ibottom, Ur
C.4 MESSENGER
Example input
Instruction Menual:
in the game, MESSENCER, each entity can take one of three roles: an enemy, message, goal. The agent's objective is bring the message to the geal while avoiding the enemies. if the agent encounters enemy any point in the game, the goal without first obtaining the message, it losas the gama and obtains a reward of -...
the dangerous enemy can be found next to the plane, which can not be moved, you are being approached by the whole is the mzin objactive.
document
solve game, you may find it helpful to list the objects that you Then for each object, match it with an entity desoription, and Idencify whather it la good or bad to interact whth the onject specifications in-game objects may
matches, Please try identifying synonyms.
Observation example:
You took oction Move South.
(agent) don't have the measage.
airplane 7 steps to your south steps your south-east robot 5 steps to your south-east
C.5 CRAFTER
Example input
instruction Menual:
writa all information helpful for the gama in numbered list.
Collect reeources such as wood, stone, and iron to craft toola and weapona,
2. Build shelters to protect yourself from monsters at night.
3. Use tools and weapons to defend yourself against monsters.
Build bridges cross
5. pig tuanels to surprise monsters and outsmart them. Plant saplings
against monsters to ensure steady supply. 7. Fat cow to restore health,
defend
8. Collect Drink to zestoze thirst.
Place Flant
for heslth.
10. Make Nood Pickaxe to collect Stone.
Nood
Zombies.
12. Make Stone Pickaxe to collect iron.
13.
Stone Sword to 14. Place Furnace to amelt Tron.
skeletens.
15. Collect Coel to smelt Iron.
make Iron rickaxe and Sword. 17. Make on Iron Pickaxe to collect Dismond,
16. collect
an iren
19. collect piamond to progress further.
ook achievement. to receive rewards.
16

---

Published as a conference paper at ICLR 2024
up to start episcde.
In plain text. blat all objects need Interact/avoid to survive in the game.
Use *I would like to object Y in each step. Replace the actual object, X by the actual interaction. 1 would like to avoid zombies, skeletons, and spiders. would like collect aaplings.
I would like to oraft a wood pickaxe.
would
wood.
I would 1ike to craft stone pickaxe.
I would Like collect stose.
wouldd like creft furasce.
1 would like to collect coal.
would 1ike collect iron.
I would like to oratt an Iron pickaxe.
collect diamonds.
I would like to craft Iron sword.
I would like chase cows.
fruits. 1 would like to drink from lake.
would like
11ke
place,
I would like to oraft a table. would
eat
I would like to drink water.
I would like to rest.
would like build
tcols defend myself against
1 would like to build bridges to groas lakea.
like dig tunnela hide
I would like to block aιroиs with stones.
would
dig through
surprise
I would like seek shelter in caves.
1 would like to build plantations of saplings and defend then agains: mensters.
groving fruits ensure steady food supply. 1 would 1ike to place a table.
would 1ike
would like
I would like to place a plant.
would
defeat zombie.
would like place stene.
1 would like to eat a plant.
would 1ike defeat skeleton.
I would like to wake up.
like place Eurnace.
Writa game cbjectives
list. For each objactiva, reguirements.
1. Collect Wood: requirements
2. Place Table: Requires Collecs Wcod
No requiramenta
4. Collect Sampling: No requirements
Collect Drink: requirements
6. Make wood Pickaxe: sequires Place Table
Make Wood sword: Raquirea
Table
place Plant: Requires Collecs Sampling
9. Defeat Zambie: No remencements
Stones Requires
Mood Pickaxe
11. Place Stoce: Bequires Collect Stone
Flant: sequires
Plant
13. Defeat Skelcton: No requirements
14. Make Stone Pickaxes Requires Cellect Stone
15. Make Stone Sword: Requires Collect Stone
16. Make Up: No "equirement"
Place
Requires Collect
15. Collect Coal: Requires Make Roed Pickaxe
iron: Requires
Stena Pickaxe
20. Make Tron Pickaxe: Requires Place Furnace, Collect Coal, and Col lect Tvon
21. Make Iron Sword: Requires Place Furnace, collect Coal, and Collect Iron
22. Collect Diamond: Requires Make Iron Pickaxe
W:ite
numbered list. 1. Move West: Flat ground west of the agent.
action,
requirements.
Mova East: Flat ground east
agent.
3. Move North: Flat ground north of the agent.
4. Move South: Flat ground south of the agent.
Do: Fecing
have necessary tool.
6. Sleep: Energy level is below maximun.
Stone: Stone inventory.
8. Place Table: sood in inventory.
Plane Purnsee: Stone inventory.
10. Place Plant: Sapling in inventery.
11. Make Wood Pickaxe: Nearby table; wood inventory.
12. Make Szone lickexs: Nearby table; wood, stone in inventery.
13. Make Iron Pickaxe: Nearby Lable, furnacer wood, coal, iron an inventory. Nearby table;
inveatery.
17

---

Published as a conference paper at ICLR 2024
Szone sword; Rearby table; wood, stone inventory.
16. Make Iron Sword: Nearby toble, furnaca; wood, coal, iron in inventory.
17, Noop: Always applicable.
observation example;
took oction meve_
sec:
water 5 steps to your south-west
grass steps to your weat
sand 4 steps your south-west
face graas your front.
Your status:
health: 9/9
food: 9/9
drink: 9/9
energy: 9/9
You have nothing in your inventory.
C.6 MINECRAFT
Example input
Instruction Manual:
You are in Minecraft and youz goal is to Lind Torest bione. You aze not allowed to craft anything.
in your observation, you are provided the anount of space an object takes in your field of view. Note that jejtcts of the same size takes more space when they are closer to you.
Observation Example: action Move
Cocrdinate (-253.12, 71.
Facing east.
You're not siming at any block. Around you:
leaves, 3696 blocka away, above you to morth-west
wood, blocks away,
grass block, 1,34 blocks sway, below you to north east
3.33 blocks away, below you nosth-east
stone, 4.67 blocks away, below to north-east
grasa block, 1.36 blocks away, below you to north-weat, taking 518 of acreen west, Laking -leaves, 4.47 blocks away, above you to aorth-west, taking 178 of screen you north-west, taking wood, 11.09 bi ooks away, above to north-west, taking 0$ o* screen
sand, 8.23 blocks away, below you
-grass,
away,
D ADDITIONAL EXPERIMENTAL RESULTS
D.I HUMAN BASELINE
players (including the authors) who are very familiar with the environments and API played the games through the SmartPlay interface. Each human player performed rounds of Bandit, RPS; 1 round of Hanoi, Crafter, Minecraft; 5 rounds of MessengerLI, MessengerL2. We report the final average score over all trials and all players.
D.2 NORMALIZED HUMAN SCORE
Given the game score of an LLM on game g. (raw), we compute normalized human score s, from the human baseline on sy (human), and the minimum possible game score $ (min).
(human)
(raw)
8g
hhmann
(min)
89
89
D.3 RAW SCORES
18

---

Published as a conference paper at ICLR 2024
LLM
Bandit
RPS
Hanoi
MessengerL.1
MessengerL.2
Crafter
Minecraft
Human Baseline
45
43
2680
1
GPT-4-0613
45.09
39.25
2.5
0.8
0.85
700
0.61
GPT-4-0314
43.86
42.05
2.7
0.74
0.93
845.6
0.592
text-davinci-O00
46.92
17.0
1.5
0.24
-0.07
186.25
0.449
Claude
32.43
20.3
-0.12
0.2
143.3
0.5
Bard
38.85
12.9
0.22
-0.21
112.3
0.54
Hlama-2-13b
22.33
15.05
-0.76
-0.745
115.3
0.606
Ilama- 13b
30.5
21.4
-0.68
-0.885
100.2
0.5
vicuna-133
28.81
7.1
0.2
-0.76
56.7
0.43
Table 4:
Comparison of performance of different LLMs in terms of average score on SandiiTWwAmmmcHHggooooooiicc RockPapeSScisooBassievv0 Hanoi3Disk-v0,, MessengerLI- vO, Messengerl.2-v0. Crafter-vO, MinedojoCreetive00-0..
19


Reviews:
[{"review_id": "ttjPf56Pgv", "reviewer": "Reviewer_Jri2", "summary": "The paper introduces SmartPlay, a benchmark with 6 games, designed to evaluate the capabilities of recent large language models (LLMs) when applied as agents in intelligent automation. The authors identify 4 key challenges important for general intelligent LLM agents but not captured in previous benchmarks: 1) long-horizon planning and execution, 2) understand the odds, 3) spatial reasoning, 4) learn from interactions or mistakes.  The authors claim that each of the games of the new benchmark offers a unique challenge mentioned previously.  Overall, SmartPlay is positioned to push the boundaries of current LLM evaluation methodologies. By systematically assessing the performance of LLMs as agents across a range of games and challenges, SmartPlay aims to provide insights into the current state of LLMs and identify gaps that need to be addressed for the advancement of intelligent agents.", "questions": "See weaknesses section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, SmartPlay is a good benchmark for evaluating the performance of large language models (LLMs) as agents. It introduces a diverse range of games carefully chosen to assess different critical capabilities required for intelligent agents, making SmartPlay a well-structured and challenging platform. \n\n2, The paper is well-written. The authors clearly articulate the need for such a benchmark and provide enough background and related works to well-position the benchmark. \n\n3, In addition to introducing the benchmark, the authors also conduct a comparative analysis of current state-of-the-art LLMs using SmartPlay. This comparison is crucial as it validates the effectiveness and rigor of the benchmark, and it provides a snapshot of the current landscape of LLMs' abilities as agents. The findings from this analysis enhance the understanding of LLMs' strengths and weaknesses, identifying areas that require further development and offering clear directions for future research.", "weaknesses": "1, Evaluation metrics proposed in the paper are commonly used in the domain of reinforcement learning (RL). While these metrics are established and provide a common ground for comparison, they may not be entirely suited to capture the unique nuances of planning and reasoning that are specific to LLMs functioning as agents. The paper could be strengthened by proposing or developing novel metrics that are tailored to the particular dynamics of LLMs' operational framework, offering a more precise measurement of their planning, reasoning, and adaptability skills in agent-based scenarios.\n\n2, Is it possible to craft the prompt to show the flow of reasoning and planning when LLMs as agents play the games?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tVj9LRCovT", "reviewer": "Reviewer_E24b", "summary": "This paper aims to address the gap in systematic benchmarks for evaluating Large Language Models (LLMs) in the context of intelligent agents. The authors propose SmartPlay, a benchmark consisting of six diverse games, each designed to test different capabilities vital for intelligent agents, such as reasoning with object dependencies, planning, and spatial reasoning. The games include Rock-Paper-Scissors, Tower of Hanoi, and Minecraft, among others. They claimed both benchmark and methodology contribution of testing LLM performance beyond language-solely-based tasks. Also, they tested some well-known LLM on the proposed game benchmarks, including GPT variants, llama variants, and etc.\n\n---\n# Post Rebuttal\n\nI appreciate the efforts made by the authors. Their rebuttal clarify lots of my concerns, and thus, I raised my scores. However, I believe I am not an expert in the field of LLM-Agent --> I am just okay with its acceptance.", "questions": "It seems the github repo is empty? https://github.com/LLMsmartplay/SmartPlay\n\nPlease also address the point mentioned above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper takes a step towards a crucial need in the field of LLM for a standardized benchmark to evaluate the agent-like abilities of LLMs. SmartPlay is presented as both a benchmark and a methodological tool, which may be a good contribution to the research community. By offering a variety of games that test a comprehensive set of agent capabilities, the benchmark allows for a detailed assessment of LLMs beyond language-solely-based tasks. The commitment to providing an open-source benchmark (referenced GitHub repository) is commendable and encourages community engagement and continuous improvement. The paper is well-structured and provides good explanations of the games and the intended capabilities they aim to evaluate.\n\nExtensive evaluation of existing LLM are provided in Table2. Some analysis are also included to emphaize the need of the proposed benchmark. Also, the results show the gap between open-sourced LLM and commercial LLM.", "weaknesses": "I believe the starting point, i.e. evaluating the LLM in the context of intelligent agent, is crucial to our community. However, I am unsure if the proposed games can well evaluate this aspects: (1) the games are still too simple to solve real-world challenges; (2) It is unclear which games correspond to which nine abilities to which levels; (3) why human has all 1 for all the games? why davinci-model has 1.04 on bandit?\n\nBesides, there may be concerns regarding the scalability of the benchmark and whether it can keep pace with the rapid advancements in LLMs. \n\nAdditionally, the paper might benefit from a more detailed discussion on the implications of these benchmarks in real-world scenarios and how they reflect the complexities of actual agent tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KgHfeqdFsV", "reviewer": "Reviewer_j3e9", "summary": "Summary: \n\nThe paper introduces \"SmartPlay,\" a benchmark designed to evaluate Large Language Models (LLMs) in the role of intelligent agents. The authors emphasize the growing potential of LLMs in intelligent agents and next-generation automation, highlighting the need for a systematic benchmark to assess their capabilities.\n\nMain Contributions are as follows:\n1. **Introduction of SmartPlay**: The paper presents SmartPlay as both a challenging benchmark and a methodology for evaluating LLMs as intelligent agents.\n2. **Diverse Game Set**: SmartPlay includes a set of 6 different games, such as Rock-Paper-Scissors, Tower of Hanoi, and Minecraft, each with its unique setting. These games collectively offer up to 20 evaluation settings and infinite environment variations.\n3. **Capability Assessment**: Each game in SmartPlay is designed to challenge a subset of 9 crucial capabilities of an intelligent LLM agent. These capabilities include reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness.\n4. **Comprehensive Evaluation**: Through the diverse set of games and challenges, SmartPlay aims to provide a comprehensive evaluation of LLMs’ abilities as agents.", "questions": "1. The author can use **one** simulator to assess the various abilities of the LLM agent, rather than relying on multiple simulators for each ability. For instance, Minecraft offers an open world with numerous engaging tasks that require long-term planning and learning from interactions. However, the author only utilized Minecraft to evaluate the LLM agent's 3D spatial reasoning skills. Another convenient and popular option is a readily installable simulator package that is user-friendly.\n2. **Comparison with Existing Benchmarks**: The paper could be strengthened by comparing SmartPlay’s performance and effectiveness in evaluating LLMs against existing benchmarks or evaluation methodologies, if any are available.\n3. There are some typos and table type errors.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Definition and Problem Formulation**: The paper introduces a new benchmark, SmartPlay, specifically designed to evaluate Large Language Models (LLMs) as intelligent agents. This represents a novel approach to addressing the current gap in systematic evaluation methods for LLMs in agent-based roles.\n2. **Well-Defined Benchmark**: SmartPlay is presented in a structured manner, with explanations of the different games and the specific capabilities they assess. This clarity aids in understanding the paper's objectives and the proposed methodology.", "weaknesses": "1. **concerns on fairness for different LLM**: The author solely relied on a pre-trained LLM for evaluation in these environments. However, since various language models are trained on different corpora and some remain undisclosed (such as close-source OpenAI language models), it becomes challenging to ascertain whether the LLM has been exposed to these specific environments. For instance, GPT possesses extensive knowledge of Minecraft, whereas LLAMA has relatively limited knowledge in this area. Consequently, ensuring fairness in direct comparisons of evaluations becomes difficult. To address this issue, I recommend that the author gather sufficient data for each environment and evaluate the capabilities of different language models separately using zero-shot learning, few-shot in-context learning, and instruction tuning approaches.\n2. **concerns on 3d spatial reasoning performance of LLM**: I noticed that current Large Language Models (LLMs) struggle with 3D spatial reasoning in environments like Minecraft. This could be due to the lack of visual information in their training data, making it difficult to directly apply LLMs to tasks in this dataset. The author attempted to describe visual images using text, but previous experiments have shown this approach to be impractical. Additionally, the source of these visual descriptions was not explained by the author. Can models like LLaVA, GPT-4V, and Flamingo overcome these challenges by being fine-tuned with visual images and other relevant information?\n3. **concerns on difficulty computation**: The setting of difficulty for each game seems quite arbitrary. How are these difficulties determined? Are they based on human evaluation? I suggest the author provide clear explanations regarding this.\n4. **concerns on prompt design for different llms**: Despite instruction tuning, the performance of LLM still relies on prompt design. It is best to conduct explicit experiments and explanations on prompt design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rbTFFv2FPJ", "reviewer": "Reviewer_itPd", "summary": "The paper presents SmartPlay, a benchmark that turns a number of control-agent environments applicable for LLMs by providing observations and actions in text format. The benchmark involves 20 unique evaluation settings, together spanning 9 different important capabilities highlighted by the paper. Results with different LLMs indicate that GPT-4 and its variants outperform other LLMs across the different tasks.", "questions": "1) How were numbers for the spider plots in Figure 2 came up with? On left it shows three degrees for every capability, but it is not obvious if it directly maps to different levels in the spider plot. If numbers on spider plot should reflect these three levels, I'd recommend adding the labels to the spider plots so reader knows to connect the two. Additionally Table 3 in the Appendix has four degrees for some of the capabilities (e.g., long text understanding), but this is not shown in Figure 2. Is this to save space or an oversight? If anyway possible, I'd recommend keeping the two consistent to avoid confusion.\n2) How were the capabilities chosen, and how would you exactly define their different degrees and their importance to the LLM research? By reading the paper, it seems they stem from the environments you chose, but they still come out as somewhat vague (e.g., what is \"Randomness\", exactly? Randomness in what? What is \"Reasoning with object dependencies\"? What is \"rollback\" error handling?). A valuable contribution would be to add as exact definitions as you can come up with, and detail them in a new section of the paper. The choice of your games helps you keep grounded to what can be measured, and by defining the metrics accuratelly, other researchers can bring new environments / dimensions into the mix.\n3) Likewise, can you provide exact details on the environments used in your benchmark package? While the code will allow exact reproduction, it might get updated as it lives on Github, so having a one solid description of the originally proposed setup would be beneficial.\n4) Have you experiment with ways to improve model performance by, e.g., chain-of-thought [1] prompting? These seem to improve models' performance across different tasks, and I feel this should be supported by this benchmark. E.g., you could add a flag or mutator or wrapper to your environment that adds/modifies the instructions/observations. Seeing initial results with these would strengthen the paper, but not necessary; my core request is to think these possibilities and also support them.\n5) How was the \"human baseline\" obtained, exactly? Did they play the games through the same exact API as the LLMs play (i.e., via text)? Were they familiar with the environments beforehand? Were there multiple human players? Likewise, how was the zero value defined in Table 2? Was it a random agent or literal 0 value for the score?\n\nReferences:\n- [1] Jason Wei, *et al*. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.\n\n--------------------------------------------------------\n## 16th Nov\n\nI read and acknowledged authors' rebuttal, and have increased my score from 6 to 8.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This type of work, presenting benchmarks and evaluation settings, is direly required in the LLM-as-agents space. Recent publications have each come up with their own ways of interfacing with environments and defining new tasks, and it is impossible to tell which agents are \"more capable\" than others. This work is a clear stepping-stone in defining standardized sets across different domains.\n\n### Originality\n\nVery original, no other exhaustive benchmarks in this domain exist.\n\n### Quality\n\nModerate to high quality. The benchmark is exhaustive in terms of different environments and challanges included, as well as conducting basic baselines. However some details and potentially expected benchmarks are missing (see weaknesses).\n\n### Clarity\n\nPaper is well structured and easy to read for the most part.\n\n### Significance\n\nThis paper, and the benchmark, can be important motivators and stepping stones for people to create comparable results between different LLM agent solutions. While I do not expect this to become de-facto way of measuring agent performance, I believe it will serve as a fixed benchmark, and in future people can improve upon it when needed. I believe this is a very significant contribution for ICLR readers.", "weaknesses": "The paper omits bunch of important details and is vague at parts, which is why I am setting my recommendation to \"borderline accept\" in the first rating.\n\n### Missing details\n\nSome important details to function as a benchmark are missing, such as  _how_ the evaluation of new LLMs/agents should be done (e.g., number of episodes, at minimum. Are users allowed to change the environment? Can users change the prompt manually before feeding to the agent?)\n\nDetails/motivation of the different capabilities is missing (see questions). I feel this need to be clarified and carefully motivated for a solid case why these capabilities matter and how they are properly measured.\n\nOther important details (e.g., human baseline collection) are missing. See questions.\n\n### (Minor) Missing baselines/experiments\n\nWhile having a fixed, single way of defining prompts makes the benchmark fixed and results comparable across models, I think there should be an option to try different techniques of prompting the models. E.g., chain-of-thought [1] can, with simple modifications to the prompt, improve results in LLM space (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)