AgentBench: Evaluating LLMs as Agents
Authors: Xiao Liu
Hao Yu
Hanchen Zhang
Yifan Xu
Xuanyu Lei
Hanyu Lai
Yu Gu
Hangliang Ding
Kaiwen Men
Kejuan Yang
Shudan Zhang
Xiang Deng
Aohan Zeng
Zhengxiao Du
Chenhui Zhang
Sheng Shen
Tianjun Zhang
Yu Su
Huan Sun
Minlie Huang
Yuxiao Dong
Jie Tang
Venue: ICLR 2024 poster (2024)
Keywords: Large language models
Autonomous agents
Reasoning
Evaluation
Benchmark
Primary area: datasets and benchmarks
TL;DR: The first systematic benchmark for evaluating LLMs as Agents

Abstract: The potential of Large Language Model (LLM) as agents has been widely acknowledged recently.
Thus, there is an urgent need to quantitatively evaluate LLMs as agents on challenging tasks in interactive environments.
We present AgentBench, a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.
Our extensive test over 29 API-based and open-sourced (OSS) LLMs shows that, while top commercial LLMs present a strong ability of acting as agents in complex environments, there is a significant disparity in performance between them and many OSS competitors that are no larger than 70B.
We identify the typical reasons of failures in environments and LLMs, showing that poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles for developing usable LLM agents.
Improving instruction following and training on high quality multi-round alignment data could improve agent performance.
And different from existing assumptions, training on code present ambivalent impacts on different agent tasks.
Datasets, environments, and an integrated evaluation package for AgentBench are released at https://github.com/THUDM/AgentBench.

Published as a conference paper at ICLR 2024
AGENTBENCH: EVALUATING LLMS AS AGENTS
Xiao Liu',, Hao Yu!. Hanchen Zhang Yifan Xu', Xuanyu Lei', Hanyu Lai', Yu Gu Hangliang Ding', Kaiwen Men', Kejuan Yang', Shudan Zhang', Xiang Deng", Aohan Zeng', Zhengxiao Du', Chenhui Zhang', Sheng Shen', Tianjun Zhang', Yu Su², Huan Sun', Minlie Huang', Yuxiao Dong , Jie Tang" -4
Tsinghua University, The Ohio State University, 'CC Berkeley
ABSTRACT
The potential of Large Language Model (LLM) as agents has been widely acknowl- edged recently. Thus, there is an urgent need to quantitatively evaluate LLMs agents on challenging tasks in interactive environments. We present AGENT ENCH, multi-dimennional benchmark that consists of 8 distinet environments to assess LLM-as-Agent's reasoning and decision-making abilities. Our extensive test over 29 API-based and open-sourced (OSS) LLMs shows that, while top commercial LLMs present a strong ability of acting as agents in complex environments, there is significant disparity performance between them and many OSS competitors that are no larger than 70B. We identify the typical reasons of failures in environ- ments and LLMs, showing that poor long-term reasoning, decision-makinn instruction following abilities are the main obstacles for developing usable LLM agents. Improving instruction following and training on high quality multi-round alignment data could improve agent performance. And different from existing assumptions, training on code present ambivalent impacts on different agent tasks. Datasets, environments, and an integrated evaluation package for AGENTBENCH are released at https: //// thub. .aubb... com/ / THUDM/ AgentBench Operating System
and
Web Browsing
Database
clause-3
gpt-4
3.11
4.01
glm 4
2.09
claude-2
claude
2.44
2.49
API-based
spt-3,5-turbo
2.32
Commercial
text-cavinci-003
1.71
LLMs
Web Shopping
Knowlcdie
daudeinsstnt
1.60
Graph
that bisan 001
1.39
taxi-davincc-002
1.25
codellama-34b
0.95
vicuna-133 13b
0.95
alma 2- 706
House-holding
Digital Card Game
amaa 2-13b
0.77
OSS LLMs
dolly 12b
chatgim-6b oasst-12b
Lateral Thinking Puzzle
---- qpt4 (0513)
codellama-24b instruct wib-r--1bb-1.5
Avv:0.51
Awo:2 32
-d-- Chease-3 (opas)
gim 4
Ilama 2 70b
AgentBench Overall Score
(a) Typical LLMs' AgentBench performance (rela- tive) against the best in each environment.
(b) Overall scores of AgentBench across 8 environ- ments. Dashed lines for two LLM types' average.
Figure I: An overview of LLMs on AGENTBENCH. While LLMs begin to manifest their proficiency in LLM Agentt, gaps between models and the distance toward practical usability are significant.
INTRODUCTION
Intelligent agents and autonomous entities (Searle, 1970; Maes, 1994; Wooldridge & Jennings, 1995) that are capable of decision-making and action execution in particular environments have been key concepts of artificial intelligence (AI) historically. Notwithstanding substantial advancements in deep learning algorithms applied in both computer vision and natural language processing (NLP), their potential for developing efficient and practically usable assisting agents remains largely unexplored.
XL and HY and HZ are lead authors that contributed equally. Email: (shawlius 1u9, onnginhh zhanghanchent @gmail. 1. com *Corresponding authors: YD and JT. Email: fyuxi jietang) fts inggua..huuuu. cn
Work done when HY, YG visited Tsinghua University.

---

Published as a conference paper at ICLR 2024
Real-world Challenges
8 Distinct Environments
(O0 Obertu ba5h ternined
Recurstiny oet fles dreatory to
read-onny sworpt those
(Given Freehoss APHJ
System
What musira inemmnents do Minnesote-
LLM-as-Agent
wencers pay?
(Given M4SQLL APs and cxisted (ubiis)
Large
Grade students over 90 as PASS in the tabic
Language
Digttl
Aguanar!
Agent
Models
Graph
The
battle playor with four pet Eon cardo
Interaction
crw usbbert iits
cxclenel bow
commenttd sucide Why del he do mat?
of toste acte,, and after Krishing Ae
Environ
Enuironments
Interaciive
Holding
Lateral Think
iigg Pundes
the middle withenn sirutattrr the dooning bable.
Plessa
(Oe the oghooe wettore of on airiin) Book the chaapost gir from Baijing Angge
Shopsing
Figure 2: AGENTBENCH is the first systematic benchmark to evaluate LLM-aa-Agent on a wide array of real-world challenges and distinct environments. In total, 29 LLMs are examined in this edition.
The advent of Large Language Models (LLMs) (Brown al., 2020; Chowdhery et al., 2022; Touvron et al., 2023), such as GPT-4 (OpenAI, 2023), has brought plenty of new opportunities to this realm. Through extensive alignment training (Ouyang et al., 2022; Wei et al., 2022a; Sanh al., 2022), L.LMs have not only mastered traditional NLP tasks but also showcased an impressive ability to comprehend human intent and execute instructions. This has spurred the development of various LLM- based applications for autonomous goal completion (like AutoGPT (Richards, 2023), BabyAGI (Nakajima, 2023), AgentGPT (age, 2023)) as well as LLM agents situated in social and game contexts (Park et al., 2023; Wang et al., 2023b; Zhu et al., 2023), sparking substantial public interest and discussions.
Despite these advancements, the lack of a systematic and standard benchmark to evaluate M.-M-as- Agent presents a critical challenge. Historically, text-based game environments (Osborne et al., 2022; Côté ct al., 2019; Hausknecht et al., 2020; Urbanek et al., 2019) have been employed for language agent evaluation. But they often suffer from the limitation of closed, discrete action spaces, well as their primarily narrow focus on models' commonsense grounding. More recently, attempts on embodied agents (Reed et al., 2022; Huang al., 2022; Ahn et al., 2022; Li et al., 2022a) have employed complicated multi-modas simulators based on games (Küttler et al., 2020; Fan et al., 2022), GUI (Shi al., 2017: Toyama et al., 2021), and indoor scenes (Shen al., 2021; Srivastava et al., 2022). However, these simulators, despite their complexity, do not accurately reflect the practical use cases of LLMs. and their multi-modal nature creates a hurdle for the urgent evaluation of existing text-only LLMs. Finally, most benchmarks now for agents focus on single environments and thus fail to provide a comprehensive overview of L.MSs across diverse application scenarios.
To address these challenges, we introduce AGENTBENCH, multi-dimensiie dimensional benchmark designed to evaluate LLM-as- ageAgent across a spectrum of different environments. AGENTBENCH encompasses eight distinct environments (Cf. Figure 4, five out of eight are created for the first time), which could be categorized into three types of groundings:
Code: Operating System, Database, Knowledge Graph (Anonymous, 2023)
Game: Digital Card Game, Lateral Thinking Puzzles, Hluse-Holdiig (Shridhar et al., 2020b)
Web: Web Shopping (Yao al., 2022), Web Browsing (Deng et al., 2023)
All datasets, whether newly created or adapted from existing ones, are meticulously designed and reformulated to simulate interactive environments where text-only LLMS can operate as autonomous agents. AGENTBENCH thus systematically evaluate an LLM's core abilities, including following in- structions (Ouyang al., 2022), coding (Chen et al., 2021), knowledge acquisition (Joshi et al., 2017; Talmor et al., 2019), logical reasoning (Srivastava et al., 2023), and commonsense grounding (Shridhar et al., 2020a). It serves as an ideal testbed for both L.M and agent evaluation.
In addition, we develop a unified evaluation toolkit for LLMs to operate on diverse customized agent tasks, thus enabling a comprehensive benchmarking of the LLM-as-Agent ability of 29 different LLMs on AGENTBENCII, including both API-based and OSS models. Our results reveal that top-tier models like GPT-4 are capable of handling a wide array of real-world tasks, indicating the potential for developing a potent, continuously learning agent. However, we also note significant performance gap between these top-tier models and their OSS competitors. Despite the recent success of OSS L.MMs and their competitive scores several benchmarks (Li et al., 2023; Chen al., 2021; Cobbe
2

---

Published as a conference paper at ICLR 2024
Table I: AGENTBENCH evaluates 29 API-based or OSS LLMs on LLM-as-Agent challenges. Models annotated with * are evaluated after task weights are computed.
Model
#Size Form
Ver.
Creator
Model
ASioc Form Ver.
Creatur
spt-4 (OpenAl, 2023)
N/A
api
D613
polly-:24 (Conover et al., 2023)
128 open v2 Datibricks
198-3.5-buzzz (OpenAL, 2022)
N/A
0613
OpenAl
11ama2-70h [Touvoon al. 2023)
70B opea chu
text-devinci-001 (Ouyang et al. 2022)
N/A
api
11ama2-13b [Touvron el al., 2023)
13B open chut
Metn
text sett-ddvinci-002 (Ouyang et al. 2022)
N/A
11ama2-7a (Touvron et al., 2023)
7B open chaz
claude-2 (Anihropic 2023b)
cleude-3+ (Anhhooic, 2024)
N/A
N/A
api
api
opus
Antropic
g3araco- 33b (Dettmmrs all,, 2023)
g3ar3co- 63b (Detimers al.., 2023)
53B open
open
Meta
claude (Anthmopo,, 2023a)
N/A api
v1.3
(Chinng al., 2023)
33B open v1.3
cleude-instant (Amthropic, 2023a)
N/A
api
+1.1
vicune-133 1b (Chingg el al., 2023)
13B open v1.5
LMSYS
Shat-bFaon-DD 001 (Anil et at, 2023)
N/A
Google
vicuna-70 7o (Chiang al., 2023)
7B open v1.5
gln 4, (Zong et al. 2022; Du al., 2022)
N/A
api
Tsinghua
openchatt- 1.5 (Wing et 2023aa
138 open v3.2
Tsinghua
codegeex22 (Zhong et 2023)
chatgin- Eb (Zeng al., 2022; Du al., 2022)
6B
open
v1.1
Zhipu
wizardim-33B (Xa et al., 2023)
wizard_m-13e (Ku
2023)
30B opea v1.0
13B open Micrssoft
codellama-344 (Rozzire et al., 2023)
34B
open instruct
koala-23s (Geng et al., 2023)
13B opea
UCB
codellema-130 (Rozitre 2023)
13B
instruct
Meta
casst-120 (LAION, 2023)
12B open sfi-1 LAION
codellema-72 (Roribno et al., 2023)
7B
open instruct
et al., 2021), their performance on the challenging AGENTBENCH tasks lags considerably. This underscores the necessity for additional efforts to enhance the learning abilities of OSS LLMs.
We identify portions of agent task failures different environments and LLMs, unveiling the insufficient abilities of long-term reasoning, decision-making, and instruction following in existing LLMss. Comparisons between different LLMMs manifest that a proper strategy of introducing code training can help improve LLM-as- Aggent. Alignment training over high-quality data (e.g.. data generated by 4pt-4) could also help improve LLM agents. In summary, our contributions are:
We introduce the concept of evaluating LLMs as agents and present AGENTBENCI,, compre- hensive benchmark to standardize the evaluation. It defines eight distinct environments of 3 types based on real-world scenarios, offering a practical testbed for LLMs' wide array of capabilitiies
We perform a thorough evaluation of 29 different LLMs using AGENTBENCH, uncovering a significant performance gap between leading API-based commercial LLMs and many OSS models that are no larger than 70B. We also quantitatively analyze the reasons for failures in existing LLM agents and highlight directions for improvement, such as improving instruction following, higher-quality alignment data. Also, we show that code training could be double-edged sword, which improves some agent tasks while harms other.
To facilitate the evaluation of LLM-as-Agent, we have introduced integrated toolkit grounded in the Server-Client architecture, focusing on modular and scalable design principles. This enables easy customization of model assessments for any LLMs using the HTTP protocol. Complemented by its associated datasets and environments, this toolkit is now openly accessible to the broader research community.
LLM-AS-AGENT DEFINITION AND PRELIMINARY
Here, we formalize the terms for describing the evaluation of LLMs as agents and the necessary preliminary knowledge for using LLMs in the context of agent evaluation.
Definition: Interactive Evaluation of LLM-as-Agent. The interactive evaluation of LLM-as-Agen could be regarded as Partially Observable Markov Decision Process (8, A, R, O). which comprises state space S, action space A, transition function function R, task instruction space 4, and observation space O. Here, we denote an LLM agent as M.
A - S, reward assigning
Chain-of- ThoTghttt (CoT) and Other Reasoning Strategies. Since LLM-as-Agent requires LLMs' strong reasoning ability, COT (Wei et al., 2022b). which has been considered a de facto strategy in related evaluation together with actions (Yao al., 2023b), is also adopted in AGENTBENCH. Despite many improved strategies proposed later, such as introducing ensemble (Wang al., 2023c), reflection (Shinn et al., 2023), and search (Yao et al., 2023a), we evaluate LLMs with the most primitive CoT in AGENTBENCH Without multiple trials, repeated generations, or complicated strategies, CoT is the easiest, cheapest, and most common way for people to deploy LLM agents.
Typical Types of Finish Reasons. Despite LLMs' capabilities, show in AGENTBENCH that even the strongest gpt-4 is not qualified as a practically usable agent. We identify and categorize finish reasons of I.LM agents AGENTBENCH tasks into five typical types:
3

---

Published as a conference paper at ICLR 2024
Context Limit Exceeded (CLE): the length of interaction history exceeds the LLM's maximum context length (only happened in 2.048-length LLMs text -ext-davinci-C0 and 003).
Invalid Format (IF): the agent does not follow the format instruction.
Invalid Action (IA): the agent follows the format instruction, but its selected action is invalid,
Task Limit Exceeded (TLE): the agent does not solve the problem after reaching the predefined maximum interaction rounds or begins to do repeated generations for many rounds.
and Complete (task ends normally). While IF and IA are mostly caused by LLMs' poor instruction following, TLE often indicates a weak multi-turn ability in certain tasks.
3 COMPOSITION OF AGENTBENCH: A BRIEF LOOK
In this section, we briefly introduce the datasets and environments that compose the AGENTBENCH. Compared to previous agent evaluation benchmarks (Côté et al., 2019; Fan et al., 2022), AGENT- BENCH concentrates on the practical evaluation of LLMs via Chain-of-Thought (CoT) (Wei et al., 2022b; Yao et al., 2023b) prompting, including code-grounded, game-grounded, and web-gre web-grounded scenarios. They pinpoint promising directions for the application of LLMs with autonomous mission completion, and their versatility avoids task sas--spccffic models' (e.g., code-specific LLMs) overperfor- mance on AGENTBENCH, Due to page limit, for details of construction, evaluation, and prompt examples, please refer to Appendix.
3.1 CODE-GROUNDED ENVIRONMENTS
Since LLMs generate high-quality codes (Chen et al., 2021), a very practical mission for LLM agents is to assist human interaction with computer interfaces. Here, we introduce three environments depending coding and reasoning abilities representatives in AGENTBENCH.
Operating System (OS), Allowing LLMs to access and manipulate OS in the terminal is a fascinating but challenging mission. Despite attempts translate natural language to Shell commands (Lin et al., 2018), few prior efforts evaluate models in executable environments. We aim to evaluate LLMss in genuine interactive bash environments (i.e.. Ubuntu Docker (Merkel et al., 2014)) on human questions with deterministic answers (c.g., number of users with non- /home directories in an OS.) or series of operations for practical goals (e.g., recursively set all directory files to read-only, excluding mine). We adopt the success rate (SR) as the evaluation metric. (Cf. Appendix B for more details)
Database (DB). As database analysis crucial but also difficult in many daily affairs, is paramount to examine LLMs' abilities to operate on real databases via SQL. Prior research has a significant emphasis on individual procedures, such as translation between SQL and natural language (Zhong et al., 2017; Gao et al., 2023; Pourreza & Rafiei, 2023; Ruan et al., 2023), answering questions given individual small tables (Nan et al., 2021; lyyer et al., 2017). However, few consider evaluating models on the complete pipeline whole. Therefore, AGENTBENCH evaluates LLMs authentic SQL interfaces, databases, and different types of queries, as found in real-world scenarios. We adopt the SR as the main evaluation metric. (Cf. Appendix C for more details)
Knowledge Graph (KG (Anonymous, 2023)). Engagement with contemporary KGs, which are often vast in size (e.g., FREEBASE (Bollacker et al., 2008) has over 45M entities and 3B facts), demands broad range of skills from an intelligent agent (Gu et al., 2023). Operating in such environments, which are only partially observable, requires the agent make decisions with incomplete information and manage inherent uncertainties with various skills, including language understanding (c.g., intricacies and subtlcties), planning (c.g., breaking down instructions into more manageable components), and tool using (e.g.. interact with KG interfaces). As a result, we propose KG as representative testing ground assess the decision- maaiin abilities of AI agents. We adopt question answering as the basic task formulation and consequently the answer F/ as the metric. (Cf. Appendix D for more details)
3.2 GAME-GROUNDED ENVIRONMENTS
Playing games usually requires strong capabilities in designing strategies, following instructions, and reasoning. Unlike code- aodddggoodded tasks, those in game- -oonnegoodded environments do not require coding expertise but rather more comprehensive understanding of common sense and world knowledge.
4

---

Published as a conference paper at ICLR 2024
Digital Card Game (DCG). Games usually could serve simulated environments for intelligent agent development. And DCG (e.g., Hearthstone (Hoover et al., 2020)) is an ideal option for text- only LLMs. It usually involves abundant text descriptions for cards, requiring thoughtful playing strategies to win, testing a model's understanding of game rules, operating logic, and abilities to form strategic decisions based current situation in the game. In AGENTBENCH, DCG system Aquawar' from the 2021 THU Agent Competition (THUAC) for evaluating LLMs as Agent. In Aquawar, the agent acts player controlling a team of fishes with different talents to battle against another algorithm-basee based team in a turn bandbased form. We report LLMs' win rate as the evaluation metrie. (Cf. Appendix E for more details)
adopt a simplified
Lateral Thinking Puzzles (LTP). Lateral thinking puzzles (Sloane, 1992; De Bono, 1970), also known as situation puzzles or Y f5, are globally popular group games that propelling participants to solve riddles from unconventional perspectives. Typically, one person hosts and others guess the this
mystery through strategic questioning, with responses limited to "yes", "no", or "irrelevant". dataset, we first set up LTP host system for automatic judging. And we gathered a diverse set of web-based puzzles of varying difficulty levels and simplifying the plot into several points (i.e., game progress). Through this assessment, we aim to gain insights into the depth and agility of LLMs' lateral reasoning abilities. (Cf. Appendix F for more details)
House Holding (HH, ALFWorld (Shridhar et al., 2020b)). Embodied game environments such as house-holding, which require strong commonsense grounding, have been well-established for language agent evaluation (Côté al., 2019). In AGENTBENCH, we assess the model's capability in accomplishing tasks in physical house-holding environments on the classical ALFWorld (Shridhar al., 2020b) derived from the well-established text-game toolkit Tex(World (Côté al., 2019). The agent needs to accomplish house-holding tasks such as "Put a pan on the dining table". We adopt the SR as the evaluation metric. (Cf. Appendix G for more details)
3.3 WEB-GROUNDED ENVIRONMENTS
Web pages have been the primary interfaces for people to interact in the real world. Therefore, assessing the behavior of LLM agents in complex web environments is critical and valuable for future development. Here, we adapt two existing web browsing datasets for practical evaluation over LLMs.
Web Shopping (WS, WebShop (Yao et al., 2022)). Online shopping is very practical and important part of modern life. Its trajectory, which comprises searching, viewing, and choosing desirable items on real e-commerce website, requires autonomous agents' strong reasoning and decision-making abilitics. Webshop (Yao et al., 2022), a simulated online shopping environment, exactly serves such purpose for evaluating language agents. While it is originally evaluated on specifically trained models, we propose assessing LLMs with mere prompting, (Cf. Appendix H for more details)
Web Browsing (WB, Mind2Web (Deng al., 2023)). A General web environment is an ideal sandbox for training and evaluating intelligent agents. Mind2Web (Deng et al., 2023) is a very recently released general benchmark for developing and assessing web agents capable of executing intricate tasks across various website domains, given high-level user instructions. It designs feasible actions for website interactions, such as clicking, selecting, and typing, thereby facilitating holistic evaluation of LLMs as web agents. Compared to Mind2Web's original setting, make adaptations to allow its evaluation on prompted LLMs without additional fine-tuning, (Cf. Appendix for more details)
EVALUATION OF AGENTBENCH
We extensively evaluate 29 LLMs, including API-based commercial models and open-sourced LLMs, to form a systematic view of the existing performance of LLM-assAgent. We also design and release a simple alug-and-play evaluation toolkit facilitate related LLM-as-Agent research.
htttps:/www.sati www.waaabii
5

---

Published as a conference paper at ICLR 2024
Table 2: Statistics and metrics of 8 environments in AGENTBENCH evaluation. "SR" stands for Success Rate. "#Avg. Round" denotes the estimated number of interacting rounds to solve a single problem. In "#Dev", and "#Test", we provide the number of query samples and total expected interacting rounds. Additionally, "Weight 1n refers to the average score for task across all models in our evaluation. For further clarification, please refer to Section 4. and Appendix B to I.
Digital
Lateral
Operating System
Data- Base
Knowledge Graph
House Holding
Web Shopping
Web Browsing
Card Game
Thinking
Puzzle
#Avg. Round
15 F1
35
10 Step SR
Metric
SR 240
SR
Reward
Game Progress
SR
Reward
#Dev
60/
300
20/ 300
12 360
20/ 500
20 700
80/ 400
31 / /000
#Test
144 / 1200
300
1500
150/ 2250
20 600
50/ 1250
50/ 1750
200 1000
100 1000
Weight
10.8
13.0
13.9
12.0
3.5
13.0
30.7
11.6
4.1
EVALUATION SETUP
Dataset Statistics. We report the statistics of datasets in AGENTBENCH in Table 2. For simplicity, we use the abbreviation of each dataset in the following part. All datasets are practical multi-round interacting challenges, and their estimated solving rounds for each individual problem range from to 50. We provide two splits for each dataset: Dev and Test. All datasets are publicly available.
We also carefully balance the evaluation comprehensiveness and efficiency in AGENTBENCH design, as LLMs' multi-round interaction can be time-consuming We set the size of Dev and Test to 269 and 1,014, respectively, resulting in around 3k and 11k calls for inference, approximately the identical amounts of calls for inference as MMLU (Hendrycks et al., 2021b) requires.
LLMs to Evaluate. As a systematic attempt to benchmark existing LLMs on LLM-as-Agent we include in total 29 models for evaluation, which could be roughly classified into categories:
API-based Commercial LLMs; mainly consist of LLM APIs without disclosed parameter amounts (Cf. Table 1). Due to more investments, their performances are usually better.
Open-sourced (OSS) LLMs: mostly come from the academia and some companies (Cf. Table 1). Due to limited computing resources, we only include OSS LLMs smaller than 70B here. It is noteworthy that this magnitude has already encompassed the majority of open-sourced LLLMs (with very few exceptions) that exhibit outstanding performance and have undergone specific fine-tuning for tasks related to chat or instruction.
Toolkit: Streamlining LLM Evaluation with API-Centric Approach and Environment Isolation. As LLM systems continue to advance in complexity and are primarily accessible through APIs, we have developed an evaluation toolkit that aligns with the APl-oriented philosophy. This toolkit is meticulously designed to interact with APIs, simplifying the process of adapting and testing different LLMs. Researchers interested in evaluating their LLMS on AGENTBENCH only need to set up model server accessible via the HTTP protocol.
Moreover, dealing with diverse and intricate interaction environments poses a significant challenge. Uniformly configuring all these environments can be arduous and may lead to conflicts. To address this, we have implemented two key strategies. Firstly, we encapsulate tasks with complex envi- ronments into Docker images. Researchers can effortlessly utilize these images by mounting the code path and initiating the evaluation process with ease. Secondly, we have subdivided each task into separate workers, ensuring that the environments of these tasks remain isolated and free from conflicts. (Refer Appendix A for further details.)
Evaluation Prompt Setup. To accommodate most existing dialogue models, our dialogue paradigm structured around two roles, user (i.e., instruction & environment feedback) and agent, engag- ing and alternating with one another. We record interaction trajectories as a conversation history (ug, do, ... ..... Ugg ax) involving the user and agent, where u;, a; represents the i-th round of the con- versation history. When we perform inference, the conversation history should follow the format of uy)
(uo, ag. ug). We select the minimum r such that count of all tokens" in (uo, ar. 14,+1, is not greater than 3500. And then append "[NOTICE] 2r messages are omitted." into un- After that, the sequence (uy, ar, Ur+1, ... 44) is regarded as the final input in multi-round chat format.
Because the tokenizers of each model is different, simply calculate tokens like this: word with length occupics [n/6] token(s), and non-blank character takes 1 token.

---

Published as a conference paper at ICLR 2024
Table 3: Test set (standard) results of AGENTBENCH. clear performance gap exists between top commercial LLMs (e.g., gpt-4) and OSS LLM competitors. "VER" stands for model version; "OA" stands for the overall AGENTBENCH score, a weighted average of all environments (Cf. Section l).
Code-grounded
Game-grounde
Web-grounded
LLM Type
Models
VER
OA
Operating System
Data- base
-lodge
Digital Card Game
Thinking Puzzle
House Hokding
Web Shopping
Web Browsing
Graph
gpt-44
0613
4.01
42.4
32.0
58.8
74.5
16.6
78.0
61.1
29.0
claude-3
opus
3.11
22.9
51.7
34.6
44.5
14.3
70.0
27.9
26.0
2,59
29.2
42.3
46.3
34.1
T4.2
34.0
61.6
27.0
API
claude
clauda-2
v1.3
2.44
2.49
9.7
27.3
22.0
38.9
55.5
40.9
8.2
58.0
61.4
55.7
25.0
S-tuubo
0613
2.32
32.6
36.7
25.9
33.7
10.5
16.0
64.1
20.0
text-davingg
Soxt-dovinn1-000
1.71
20.1
16.3
34.9
3.0
20.0
61.7
26.0
claude-Inslart
v1.1
1,60
16.7
18.0
20.8
5.9
12.6
30.0
49.7
4.0
1.39
9.7
19.7
23.0
16.6
4.4
18.0
60.5
12.0
noxt=davinc1-000
1,25
8.3
16.7
41.5
11.8
0.5
16.0
56.3
9.0
OSS
11ama-2- quamaco-65b
chat
0,78
9.7
13.0
8.0
21.3
0.0
2,0
5.6
19,0
(Large)
0.54
8.3
14.7
1.9
0.1
1.5
12.0
0.9
10.0
codellara-345
0.96
2.8
14.0
23.5
8.4
52.1
20.0
Oss
vicuns-33b
v1.3
0.73
15.3
11.0
1.2
16.3
1.0
6.0
23.9
7.0
Medium)
wiserdIn::30 30b
v1.0
0.46
13.9
12.7
2.9
0.3
1.88
6.0
4.4
1.0
quanaco-33b
0.39
11.1
9.3
3.2
0.3
0.0
6.0
6.2
5.0
vicuns-13b
v1.5 0.93
10.4
6.7
9.4
0.1
8.0
8.0
41.7
12.0
11ama -2-13b
chat
0.77
4.2
11.7
3.6
26.4
0.0
6.0
25.3
13.0
openchat-133
v3.2
0.70
15.3
12.3
5.5
0.1
0.0
0.0
46,9
15.0
wizarain-13b
v1.2
1.9
12.0
vicuns-7b
v1.5
0.56
9.7
8.7
2.5
0.3
0.0
2.2
9.0
(Small)
OSS
code1 1am1111a-7
Modellana-138
instruct
instrict
0.56
0.50
4.9
3.5
12.7
9.7
10.4
8.2
0.0
0.0
0.0
0.0
0.0
2.0
25,2
43,8
12.0
14.0
koels-
0.34
3.9
7.0
11ama-2-7b
chat
0.34
4.2
8.0
2.1
6.9
11.6
7.0
codegeex2-O0
0.27
1.4
0.0
4.8
0.3
0.0
0.0
20,9
11.0
dolly
v2
0.14
0.0
0.0
0.0
0.1
1.2
0.0
0.4
9.0
chatgln-6b
0.11
0.3
0.0
0.5
oasst-122
sf-4
0.03
0.0
0.0
0.3
1.0
However, in order to consider non-chat models, we append a post-processor. We feed the history into the model for chat models supporting multiple rounds. For models supporting only text completion (e.g., text-daviiinee sett-ddvvini---00 i000n, we prepend "USER:" or "AGENT:" into each item in the history and finally append the string "AGENT:" to make models generate the agent's content.
For task prompt organization, we adapted the format from (Yao et al., 2023b) to include both "Thought" (for CoTT) and "Action" but in one single round. Usually, simple CoT demonstration is provided in the task instruction for a better output format. To ensure reproducible results, we set camperature 0 (i.e., greedy decoding) in the inference on all tasks following (Wei al., 2022b).
Overall Score Calculation. We have observed that the score distribution for each task varies significantly as tasks differ in difficulty levels. As a consequence, naively averaged score is heavily impacted by tasks that generally yield higher scores (e.g., Web Shopping in our observation), overshadowing those with lower scores and being unsuitable for AGENTBENCH's purpose.
Therefore, we produce the overall score by first resizing each task's ' average score to 1 across all the models we evaluate and then averaging the scores across all tasks for each model (Cf. Table 2). To standardize and simplify score calculations for future studies, we utilize the reciprocal average score of all the tested LLMs in each task as a fixed weight for future overall score calculation. The total score is then computed as the average value obtained by multiplying the score of each task by its corresponding weight. This method ensures fairness and consistency in evaluation, enabling easier comparisons and analysis in future research.
4.2 MAIN RESULTS
Overall and dataset-specific scores in AGENTBENCH are reported in Table 3. Surprisingly, on this challenging benchmark, we discover that some top LLMs are equipped with solid capabilities for dealing with real-world environmental interaction. For example, gpt-4 presents the best performance on 6 out of 8 datasets in AGENTBENCII; on House Holding, it achieves success rate of 78%, indicating its practical usability in this scenario. claude-2 and claude follow 9pt-4 but quite outperform 3pt-3. 5-turbo. Despite other API-based LLMs' relatively poorer performance, regardless of tasks, most of them can solve quite a few percent of problems. All API-based LLMs have AGENT BENCH overall score above 1.00.
AGENTBENCH
7

---

Published as a conference paper at ICLR 2024
1.0
codellama
<ing System
Opera
Data-
Know Digital
-legg
Card Game
Thinking Holding
Lateral
House
Shop Brow
Web Web
AgentBench OA score
0.8
openchat-22
Jama-2-11 135
vicune 33b
llama 2-70-
Graph
Puzzlo
-ping -sing
wizerdim
0.6
vvcuna codellama 13b
Completed
75.0
37.9
30.1
51.2
14.0
13.1
54,9
56.6
codellama
wizardm-30t
CLE
0.1
0.7
2.0
0.0
3.5
0.7
0.0
0.0
0.4
koala-136
guanaco-336
Invalid Formar
0.0
53.3
0.0
38.5
0.0
0.0
17.2
0.0
llama
Kodegeox2-6b
Invalid Action
0.9
0.0
0.0
10.2
0.0
64.1
8.4
dolly-128 12b
TLE
23.9
8.0
67.9
82.5
22.1
27.8
35.0
chatglm-6b
Table 4: Portions of different types of execution outcomes in 8 tasks averaged across all models. (CLE: Context Limit Exceeded, TLE: Task Limit Figure 3: AGENTBENCH OA scores with regard Exceeded).
13
33
65
#ize (billion parameters)
to all tested OSS L.LMs.
OSS LLMs we test, however, commonly fail to solve problems in some challenging tasks, such as Knowledge Graph, Digital Card Game, and House Holding. We plot their performance relative to their sizes in Figure 3. Generally, most OSS LLMs perform far poorer than API-based LLMs in AGENTBENCH (Avg. 0.51 v.s. 2.15). The most capable OSS LLM < 70B within the evaluation scope rounds out to be codellama -4h, 4b, achieving overall score of 0.96 but still presents clear performance gap to gpt-3. 5-turbo. Our community still needs much effort to produce stronger OSS L.Mss.
4.3 ANALYSIS
In the evaluation, we analyze some important factors that impact an LLM agent's performance on AGENTBENCH, including outcome portion analysis, code training, and the difference between API- -Pl-based commercial LLMs and OSS LLM competitors. More insights and case studies into the ability of planning, self-correction, and tool use are provided in Appendix J.2.
Portion of Different Types Execution Outcomes.
Table 4 reports the ratios of execution outcomes (Cf. Section 2). The predominant failure cause in AGENTBENCH tasks is Task Limit Exceeded, revealing weak reasoning and decision-making abilities in LLMs. This data underpins the existing LLM weaknesses, potentially guiding future development (please refer to our framework).
In Database and Digital Card Game tasks, Invalid Formul errors frequently occur due to stringent formatting requirements (Cf. Appendix 1.2.1). Conversely, House Holding and Web Browsing tasks often face Invalid Action errors due to LLMs generating actions beyond the predefined action spaces. Please refer to Appendix J.1 for more detailed ratios.
Impact of Code Training. We find that code tuning might deeply influence model's way of inferential generation and thinking, even beyond topics just about coding. From the comparison of codel 1 ama and 11 1mma-2 series, tuning with code seems give models edge in tasks that follow a relatively static procedure (e.g., Web Shopping). But, this kind of tuning might also affect the model's 's general thinking ability, codel ama series does not perform well in the Digital Card Game as 1 1lama-2 2 series, as well as in Operating System where interacting with the Linux system more important than writing bash codes. This points to balance between being good at following procedures and being good at general thinking when tuning LLMs.
Ambivalent
Impact of Hlihh-uualit Alignment Data Training. Another helpful comparison would be between vicuna-1l 3b and 11ama ama-2-13: While they share the same base LLM, vicuna- 11--113 3b is aligned by training on ShareGPT's s data (generated by gpt-4 and gpt- 5-turbo, shared by users) and 1lama-2-133 3b is aligned from scratch. As result, v cuna- - 3b outperforms 1 1.ma--2- 13b on AGENTBENCH, and even performs comparably to 3 times larger code ama 4b. This indicates that high-quality alignment is still key to develop better LLM agents.
Unexpected Similar Performance of 11ama -2-13b and 1lama-2-700 During our experi- ments, were surprised to find that alama 2 2-13b and llama 2-b perform similarly despite the significant gap between their sizes. After carefully checking and re-running experiments, the results are unchanged. We think that it indicates 11ama-2-77 's insufficient pre- pre-training. While both 11 ama- 1ata--211 2-12-13 3b and a1mma- - are pre-trained with 2T tokens, a larger LLM should be trained with more tokens according to the scaling law (Hoffmann al., 2022).
8

---

Published as a conference paper at ICLR 2024
Another potential reason could be that 1lama-2-70000 is not adequately aligned instruction following, resulting in its comparatively lagging ability to follow instructions (Cf. Appendix J.2.5).
RELATED WORK
Evaluation of LLMs. The general capabilities of sepfr-upee (Liu al., 2021) LLMs (Brown et al., 2020; Chowdhery et al., 2022; Zhang et al., 2022; Scao et al., 2022; Zeng et al., 2022; Touvron et al., 2023), especially those chat-aligned ones (Ouyang et al., 2022; Anthropic. 2023a; OpenAI, 2023), have refreshed people's impression on deep learning systems and significantly transcended the conventional scope of NLP evaluation. It thus makes the evaluation of LLMs an urgent and challenging problem. Compared to previous efforts focusing et al., 2019; Wang et al.; Gehrmann et al., 2021), I), an increasing number of benchmarks are including broader spectra of tasks and datasets (Hendrycks et al., 2021b; Liang et al., 2022; Srivastava et al., 2023) in the evaluation. However, most of them are still limited to traditional tasks and thus fail to evaluate LLMs' open-ended generation, multi-round interaction, and ability to act as agents.
subset of specified tasks (Wang
LL---s-Agent. In pre-LLM era, text game environments such as TextWorld (Côté et al., 2019), Jericho (Hausknecht et al., 2020), and LIGHT (Urbanek et al., 2019) dominant in language agent study which bases on BERT (Devlin et al., 2019) and reinforcement learning. With the advent of LLMs, the study of LLM agents begins to thrive (Huang et al., 2022), especially after Mfain-of-Though (Wei et al., 2022b) came out. ReAct (Yao et al., 2023b) is a pioneer work to combine CoT reasoning and actions in agent tasks. Later, a multitude of advanced reasoning strategies (Kim et al., 2023; Shinn et al., 2023; Wang et al., 2023d; Liu et al.. 2023; Yao et al., 2023a; Gu et al., 2023) and applications including frameworks (Richards, 2023; Nakajima, 2023; age, 2023) and multi-agents (Park et al., 2023; Hong et al., 2023; Wu et al., 2023) for L.LM-as-Aeet have emerged and arouse much public interest. Nevertheless, limited datasets and models and available on the topic, without standard and comprehensive benchmark. AGENTBENCH presents the first systematic benchmark for evaluating LLM-as- Agentttteent with a broad coverage of tasks and available LLMs. Additionally, it also initiates the idea of adopting agent tasks measure LLM performance.
Evaluating LLMs in Executive Environments. As LLMs become increasingly capable of real- world challenges, there is also a trend to evaluate them in executive environments rather than static datasets. Besides text games (e.g.. AI.FWorld (Shridhar et al., 2020b)), another main stream of works lies in code execution. APPS (Hendrycks et al., 2021a), HumanEyal (Chen al., 2021) and MBPP (Austin et al., 2021) pioneer the effort to evaluate code LLMS for functional correctness instead of text similarity. The paradigm has been later widely recognized and adopted in following works (Li et al., 2022b; Zheng et al., 2023; Nijkamp et al., 2023). However, few previous code evaluation frameworks consider multi-round interactions. A concurrent work InterCode (Yang et al., 2023) releases framework that allows evaluation of interaction between models and Bash and SQL environments, which are similar to OS and DB tasks in AGENTBENCH.
CONCLUSION
We present AGENTBENCH. systematically designed evaluating L.MMs as agents, covering as many as 29 L.MMs in the first time, establishing a unified testing framework and toolkit for agile evaluation. Based on the evaluation, we present wide array of insights into agent tasks, LLM behaviors, and potential methods to improve I.LMs on AGENT BENCH. We anticipate that AGENTBENCH will serve cornerstone for subsequent LLM agent research.
evolving benchmark for
ACKNOWLEDGEMENT
We would like to thank the anonymous reviewers and Shunyu Yao for their suggestions in refining this work as well as Zhipu AI for covering all GPU and API cost consumed in this study. Yuxiao Dong is supported by Natural Science Foundation of China (NSFC) 62276148. Jie Tang is supported by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2022ZD01 18600, NSFC for Distinguished Young Scholar 61825602, Tsinghua University Initiative Scientific Research Program 20233080067 and the New Cornerstone Science Foundation through the XPLORER PRIZE. This work is also supported by a research fund from Zhipu AI.

---

Published as a conference paper at ICLR 2024
REFERENCES
Agentgpt. Python. http://ggiubbbcommeeew////////gt 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0160) 2022.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm technical report. arXiv preprint arXiv:2305. 2023.
Anonymous. Knowledge base question answering as tool learning. under review, 2023.
Anthropic. Introducing claude, 2023a. introducing-claasee
URL
Anthropic. Claude 2, 2023b. URL www.:tttpp.///w..etterrro..o../p
Anthropic. Introducing the next generation of claude, 2024. URL /tt.ttttsss////w.atttttt homm/nwws/oe ccwee//clldee33ee fami lym
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 108.0732,, 2021.
Kurt D). Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Jason Tsong-Li Wang (ed.), Proceedings of the ACM SIGMOD International Conference Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008, pp. 1247-1250, ACM, 2008. doi: 10.1 10.1145/1376616.17766
URL https: //c::////oo/// org/ 10. 1 1145
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Hersert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829544
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hemrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.03372 2021.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1026-1036, Online, November 2020. Association for Computational Linguistics. doi: 10. (8.18863/v1// 1/0.33//12222.mmvem wtw.0...inningeemml...099 URL https: http::/:////aaaaahlaaoao/o www.lllgg..ggg/....//./// wwt.ooog/////./ findings emn p. 91.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See hmpp://vicuma.mmyys.o (accessed 14 April 2023), 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0231/1 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
10

---

Published as a conference paper at ICLR 2024
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction--uund Ilm, 2023. URL https: //www 12/do firfsttly-frst---e open- comme www.rrrilallllllllll in
wtt.aaaiccc......oooooaaaaaccaae /043/04/
1.1m
Mare-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla EI Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop. CGW 2018, Held in Con- junction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pp. 41-75. Springer, 2019.
Edward De Bono. Lateral thinking. New York, pp. 70, 1970.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint 1rXiv:2306.06077 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. arXiv preprint arXiv: 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
Jack Edmonds and Richard M Karp. Theoretical improvements in algorithmic efficiency for network flow problems. Journal of the ACM (JACM), (9(2):248-26) 1972.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
LR Ford Jr and DR Fylkerson. Flows in networks. 1962.
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308. 15363, 2023.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, An- uoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna- Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, et al. The gem benchmark: Natural language generation, its evaluation and metries. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 202 pp. 96-120. Association for Computational Linguisties, 2021.
Xinyang Geng, Arnay Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1, 2023.
Yu Gu and Yu Su. ArcaneQA: Dynamic program induction and contextualized encoding for knowledge base question answering. In Proceedings of the 29th International Conference on Computationnl Linguistics, pp. 17111131, Gyeongju, Republic of Korea, October 2022. Inter- national Committee on Computational Linguistics. URL https: http///aaaannoogrroo/e org/ 2022. 1002..cliii. i 148.
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang. Xifeng Yan, and Yu Su. Beyond ii.d.: Three levels of generalization for question answering knowledge bases. In Proceedings of the Web Conference 2021. ACM, apr 2021. doi: 10.1145/344288 155534444231133449922 URL https:
//doi.org///// 11 1145/ .12.2331111449999 3449992.

---

Published as a conference paper at ICLR 2024
Yu Gu, Xiang Deng, and Yu Su. Don't generate, discriminate: A proposal for grounding language models to real-world environments. Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4928 4949, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https: / /acclac siclannhooogg org/
2023 -cl-allllllnn.oo long. 270.
Matthew Hausknecht, Prithviraj Ammanabrolu, Alex-Alex Côté, and Xingdi Yuan. Interac- tive fiction games: colossal adventure. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arviv preprint arXiv:2105.099 09.388.,,,, 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021b.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimet large language models. arXiv preprint 2rXiv:2203. 2022.
Sirui Hong, Xiawu Zheng, Jonathan P. Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zi Hen Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. ArXiv, abs/2308.00352, 2023, URL https: //a:://appp /api....... semant i icscholar. org/ Corpps 60351380.
Amy K Hoover, Julian Togelius, Scott Lee, and Fernando de Mesentier Silva. The many ai challenges of hearthstone. KI-Künstliche Intelligenz, 34:33-43, 2020.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. International Conference on Machine Learning, pp. 91 9118-9147. PMLR, 2022.
Mohit lyyer, Wen-tau Yih, and Ming- Wei Chang. Search-based neural structured learning for sequential question answering. In Proceedings the
the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1821-1831, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18655/v//117-111.. URL https: // //aclanthi ogy- wwt/////11111
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1601 1611, 2017.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arXiv:2303.1749)) 2023.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.
LAION, Open-assistant. https:
/ http://gttttttt../ com/ cww/como////////// LAION AI Assispen--sisantt....m 2023.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision- makiig.. Advances in Neural Information Processing Systems, 2022a.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction -oolowiionfooloowing models, 2023.
12

---

Published as a conference paper at ICLR 2024
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition--evel code generation with alphacode. Science, (0866622) 2022b.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09/ 10, 2022.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL htwps::pp//......... //ts:///aala www.caanntlllloo.ooooooooooooooe ogy- org/ -04-1013
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. NI2bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Bo Liu, Yuqian Jiang, Xiaohan Zhang. Qiang Liu, Shiqi Zhang, Joydeep Lim+ P: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11 133044114477 2023.
Joydee Biswas, and Peter Stone.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE transactions knowledge and data engi- neering. 35(1):857-876 2021.
Pattie Maes. Agents that reduce work and information overload. Commun. ACM, 37:30-40, 1994.
Dirk Merkel et al. Docker: lightweight linux containers for consistent development and deployment. Linux 239(2)2,, 2014.
Yohei Nakajima. Babyagi. Python. https://github. combpoheinnkqgjnnabbbyygg 2023.
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Kryšcińssi, Nick Schoelkopf, Riley Kong. Xiangru Tang, Murori Muturn,, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir Radev. Fetaqa: Free-form table question answering, 2021.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingho Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference Learning Representations, 2023.
OpenAl. Introducing chatgpt, 2022. URL
www.coom//////o.e. wwt.bloo///ooooccccccccooooeoeoo chattpt...
OpenAL. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023.
Philip Osborne, Heido Nömm, and André Freitas. A survey of text games for reinforcement learning informed by natural language. Transactions of the Association for Computational Linguistics, 10: 873-887, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744,202.
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. ArXiv, abs/2304.03442. 2023.
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.31 //111/11111...1 142. httpttss: aaclllll wttphoo/oooy.ggg/1511111111
13

---

Published as a conference paper at ICLR 2024
Mohammadrezza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. arXiv preprint arXiv:2304.11015. 2023.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research, 2022.
Toran Bruce Richards. Auto-gpt: An autonomous gpt-4 experiment, 2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. Tptu: Task planning and tool usage of large language model-based ai agents. arXiv preprint arXiv:2308..0 03427, 2023.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Represenaations, 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05 2022.
John R. Searle. Speech acts: An essay in the philosophy of language. Language, 46:217, 1970.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez- D' A'Arinno, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al. igibson 1.0: simulation environment for interactive tasks in large realistic scenes. In 202/ JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7520-7527. IEEE, 2021.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference Machine Learning. pp. 3135-3114,, PMLR, 2017.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020a.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2020b.
Paul Sloane. Lateral thinking puzzlers. Sterling Publishing Company, Inc., 1992.
Aarohi Srivastava, Abhinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Aaariga-Alonso et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. Conference Robot Learning, pp. 477-490. PMLR, 2022.
14

---

Published as a conference paper at ICLR 2024
Yu Su, Huan Sun, Brian M. Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan. On generating characteristic-rich question sets for QA evaluation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 562- 572. The Association for Computational Linguisties, 2016. doi: 10.18653/v1/116- 1054. URL pttps://doil.
http:/.0.1633/vvvl / 6-10
Alon Talmor and Jonathan Berant. The web knowledge-biie base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pp. 641-651, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/ N18- URL https: httpt:s://aalaattaaaaa hataa/aaaannttooooooooooooo . org/ 1118-10594
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149-4158, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv:2105. 13231, 2021.
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. Learning to speak and act in a fantasy text adventure game. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNIPP), pp. 673-683, 2019.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data, 2023a.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi (Jim) Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXiv, abs/2305.16291, 2023b.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023c.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint X1X::2302.01566 560, 2023d.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of th-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35.24824-24837, 2022b.
15

---

Published as a conference paper at ICLR 2024
Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice. The knowledge engineering review, 1022):15-112, 1995.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. ArXiv, abs/2308.00 0..88.8155 18155, 2023. URL https:/ //api.
semanticscholan wttp/laar.rg/ooouuuDD
260992599000
Can Xu, Qingfeng Sun, Kai Zheng. Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. arXiv preprint arXiv:2306. 148988 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint ar 2cVv:2335.10001 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023b.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b. An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303. 17568, 2023.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, aby/1709.00103, 2017.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyuan Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Y. Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. ArXiv, abs/2305. 17144, 2023.
16

---

Published as a conference paper at ICLR 2024
Part I Appendix
Table of Contents
Framework
20
A.1 Traditional Evaluation Frameworks
20
A.2 Our Designed Evaluation Framework
20
A.3
Implementation of Max-Flow Algorithm
20
B Operating System
21
B.1 Dataset details
21
B.2 Actions
22
B.3 Prompt Example
22
Database
Dataset Details
24
C.2 Data Augmentation
25
C.3 Prompt Example
25
C.4 Study on bias in data augmentation
26
D Knowledge Graph
26
D.1 Dataset Details
26
D.2 Prompt Example
27
E Digital Card Game
29
E.1 Dataset Details
................................
29
The Attributes Fish
................................
30
Prompt Example
................................
31
E.4
Battle Generation
33
E.4.1
Combat Power
33
E.4.2
Difficulty: A Ratio of Combat Powers
34
F
Lateral Thinking Puzzles
34
F.1
Dataset Details
34
F22
Evaluation on LTP System
35
F3
LTP Game Progress and Termination
35
F4
Prompt Example
35
G
House-holding
38
G.I Dataset Details
38
G.2 Prompt Example
39
H Web Shopping
40
H.1 Dataset Details
40
H.2 Prompt Example
40
I Web Browsing
42
L.1 Dataset Details
42
1.2 Prompt Example.
43
Detailed Analysis
45
J.1
Validity Analysis of Exccution Outcomes
45
I.1.1
Motivation of Validity Analysis
45
17

---

Published as a conference paper at ICLR 2024
J.1.2
Definition of Validity Analysis
45
1.1.3
Validity Analysis of Models
45
J.2
Findings
46
J.2.1
Instruction Following Matters
46
J.2.2
Consistency and Exccution Agent Planning
47
1.2.3
Understanding Task Completion: Distribution of Tokens and Rounds in
Completed Trajectories
54
J.2.4
Primary Cause of Task Limit Exceeded: Models Tend to Repeat Previous Content
55
J.2.5
The Influence Code Tuning on LLM Acting as Agents
56
J.2.6
Self Correcting ahility
56
18

---

Published as a conference paper at ICLR 2024
277 19.
(a) Operating System (OS)
(b) Database (DB)
Task: "Find the total number of non-emp mon-empty directo- ries inside the 'letc' divectory."
Task: "What was the total number of medals won by United States?", given the table "Olympic Medals'
Action Space: Any valid bash commands
Act ion space: Any valid SQL, commands
Observation. System standard output
Observation: MySQI CLJ interface output
(c) Knowledge Graph (KG)
(d) Digital Card Game (DCG)
Task: "Find tropical cyclones that are similar to Hur- ricane Marie and affected Eastern North America." Action space: Baxic KG-querying tools Observation: Query results
Task: "Compete against another player using four 'fish' cards in 'Aquawar' game."
Action space: Four 'fish' cards and Assertion
Observation: Battle process, status 'fish'
(e) Lateral Thinking Puzzles (LTP)
(1) House-holding (HH)
Task: "A man sleeps with the lights off. and the next morning he suicides after opening windows. Why?"
Task: "Clean some soapbar and put it in coutertop" Action space: A list of allowed actions in the room, other accessible rooms
Action Space: Any binary questions
Observati ion: 'Yes', 'No', 'Irrelevant
Observation: Results after the action.
Lodge Dedsgread Cuuenn Size Guif with Set. Ructic Guilt Covertet Ded Sec. Stonwhurst Collection. sal
(g) Web Shopping (WS)
(h) Web Browsing (WB)
Task: "Looking for queen size bedspread set in the Task: "Find a latest post with more than 10k upvotes color redwood, and price lower than 70."
in rlannouncements community and upvote it." Action space: Search (generate keywords) and Action space: 1) Choose one of all HTML ele- Click (choose from clickable buttons)
ments the wehpage; 2) Click, Type, or Select Options Observatio ion: rvation: Products' deseriptions; the webpage Observation: Page HTML (optional: screenshot)
Figure 4: Examples of all environments in AGENT
19

---

Published as a conference paper at ICLR 2024
A FRAMEWORK
A.1 TRADITIONAL EVALUATION FRAMEWORKS
Traditional evaluation frameworks can be categorized into two types:
Traditional Tasks (e.g., single-round generation, classification, etc.). These frameworks
designed for specific tasks and may not be suitable for more complex tasks involving multi-round interactions.
Agent- based Tasks (tasks with multi-round interactions). These frameworks are typically tailored to specific task by the creators of the dataset. They often suffer from several limitations:
They are designed for a specific task, limiting their applicability to other tasks.
Communication between components (Task, Agent, and Evaluation) usually occurs within single process or through the creation of child processes, necessitating evaluation the same device.
They can only evaluate one task with one agent at time.
A.2 OUR DESIGNED EVALUATION FRAMEWORK
To address the limitations of traditional agent-based evaluation frameworks, we have designed a novel framework with the following features:
Decoupled S/C Architecture. Our framework decouples the Task Server, Agent Server, and Evalua- tion Client components, enabling separate deployments. They can communicate via HTTP interac- tions, allowing them to run on different devices, thus eliminating the need for co-location to satisfy the requirements of both Task and Agent.
Agent- Task Colla borativee Evaluation. Our framework supports collaborative evaluation of multiple agents and tasks in various combinations simultaneously. This flexibility enables more comprehensive testing scenarios.
Network Flow Algorithms. We have incorporated network flow algorithms into the Evaluation Client, maximizing evaluation efficiency. This optimization ensures that both Agent and Task Workers are utilized to their fullest potential.
Resumable Evaluation. Our framework includes a resumable evaluation feature, making it easy to recover and continue interrupted evaluations seamlessly.
With these advancements, our evaluation framework overcomes the limitations of traditional ap- proaches and provides more versatile, efficient, and scalable solution for evaluating intelligent agents in multi-round tasks.
The overall structure of our framework can be described in Figure 5.
A.3 IMPLEMENTATION OF MAX-FLOW ALGORITHM
In our evaluation process, we employ the Edmonds-Ket Karp algorithm (Edmonds & Karp, 1972) as practical implementation the Ford-Fulkerson method (Ford & Fulkerson, 1962) designed to compute the maximum flow in network with time complexity of O(VV)
To formalize the problem, consider a scenario with n agents, denoted as Ai, A2, An, and m tasks, denoted Ti, T2, ... T.. Our objective is to conduct evaluations in Í different groups, each focusing on the pair (A Jy.), where S l S T. Additionally, for every such pair (Azu, Ty.), ). we should evaluate 8& samples. The number of workers for agent Ag and task Tç is denoted (o(A)) and w((s) respectively.
The flow graph construct be descrihed G =<< V, E where the vertex set V is defined as
20

---

Published as a conference paper at ICLR 2024
Workers
Task Workers
Agent Controller
Action
MySQU
ubuntu
Übsenation
API
API
Maxflow Algorithm
Task Controller
Deployed Models
API
API
Agent Server
Assigner
Figure 5: The toolkit of AGENTBENCH is meticulously crafted for the seamless deployment of tasks and agents, coupled with efficient evaluation assignment system. Agent servers (left) manifest in diverse forms, enabling us to deploy a model server and expose an accessible API through the HTTP protocol. Task servers (right) composed of task controller and several task workers, whose environment is within an isolated environment, ensuring freedom from conflicts and optimal task execution. Evaluation client (center) establishes an agent-task graph and employs the max-flow algorithm to optimize interactions. This optimization results in client workers seamlessly engaging with agent and task servers, facilitating the smooth execution of tasks and evaluations.
V [A S S U s m] ((SD);
(1)
And the weighted edge set E is denoted as
F B=((Ae> Te 84]l1 S 51 U U Av, w (A:) S k S n]
(2)
D, (T:)] S S m).
We apply max-flow algorithm from source vertex S to destination vertex D. For each flow edge (A, T;; ((.)), allocate f(JJ) samples for agent A, and task Tj. After allocation, the weight of the edges should be reduced by the value of flow. Upon completion of an evaluation, the weight of edge connected either S or D should be increased by 1.
We also establish periodic interval for applying the algorithm to the network for newly available evaluation triples.
B OPERATING SYSTEM
B. DATASET DETAILS
Construction Details. Each evaluation sample in OS dataset encompasses following contents:
Instruction. The description of the problem in natural language that needs LLMSs solve.
Docker Environment. The starting up docker image (e.g.. preset default
Docal-os/defaul
Initialization Script (Optional). The bash scripts that need to be executed independently (docker exec) before the interaction starts (e.g., user configurations, files, system statuses). Start Script (Optional). The bash scripts executed after shell is created and before interaction.
Checking Pipeline. The checking method to judge the correctness of L.MMs answer or operation. Example Script (Optional). The bash scripts that serve reference solutions. In other words, if executing them in the interaction, results are correct. Only for unit tests that introduced below.
We design two types of tasks in the OS evaluation beyond conventional QA-only evaluation.
21

---

Published as a conference paper at ICLR 2024
Question Answering (QA): LLMs need to output commands to solve specific questions in OS (e.g., aggregate numbers, view file contents). In this case, they must commit answers finally.
Operation: LLMs need to output commands to do some verifiable operations on the operating system (e.g., change file/user states). In this case, they do not need to commit final answers.
Thanks to the checking pipeline, two types of tasks can be evaluated in a unified solution.
Collecting challenging queries regarding OS could be difficult. In practice, about half of our instructions are created or collected from humans, while the other half are mostly QA problems generated by spt-4 and strictly filtered by passing the unit tests (i.e., yield correct answers/states).
For human instructions, we first gather 6000 real problems and solutions with bash or shell 1 tag from Stack Overflow'. Then we sort them by the score (count of likes). We invite 8 annotators majored in programming to select challenging ones. For each selected problem, they create one or more task instructions and write detailed problem description, the initialization script, the starting script, and the checking pipeline. Finally, we conduct a cross verification for each evaluation sample to make sure it's correct. For each problem, it takes about 2 hours to do the annotation.
For generated problems, our unit test contains the following parts. 1) Initialization Script Correction: we execute the initialization script and remove samples with wrong initialization whose exit code does not equal to 0. 2) Example Code Correction: execute the example code and the checking pipeline to judge the correctness of the answer. We remove samples with wrong answers.
In the end, curate 144 high-qualite diverse OS evaluation samples accompanied with testing interactive environments and corresponding checking pipelines (i.e., scripts). Agents are prompted with 1-shot CoT better format their responses (Cf. Appendix B).
Evaluation Setup. For each problem (i.e., instruction), the execution can be divided into parts.
Initialization. We create docker container with specific image, and we run initialization bash script to set up environments specified by the instruction.
Interaction. We start new shell in this docker, and run the starting bash script specified by the instruction. Then the LLM to test fed with piece of instruction and the problem description. It starts interaction with the shell. In each round, two actions are provides. One is to run bash script, which allows the model to generate and run a series of commands in the shell. The other is to commit answer, which allows the model to terminate the interaction process. It's notable that the model will be judged that it fail to solve the problem exceeding round limit (8 by default).
Checking. For each problem, there is a checking pipeline containing a list of scripts Si, fn. where f' denotes the k-th script piece in the pipeline. For fú. the answer of the model, Og. and the output of fe(t < k), ot, will be fed as input arguments into fk, i.e., ok fo 01, 0k-1). The result correct if and only if all the scripts exit with code O.
Metrics. We measure the Success Rate for LLMs to solve problems in the execution. There are only two final status for each item of the problems, wrong or correct.
B.2 ACTIONS
In OS evaluation, design two major types of actions: bash and commit.
Bash: which launches bash command (using textual inputs in content field)
which announces the completion of the goal. If the task is QA problem, then the agent should submit the final answer in content field; else the checking pipeline will automatically check the system status to judge the correctness.
Commit:
B.3 PROMPT EXAMPLE
A prompt for OS evaluation consists of the instruction and the formulation of interaction trajectory. An example of instruction prompt is:
You are an assistant. that will act like a person, I'wil play the role of ni (ubuntu) operating ng system. Your goal is to implement the
hhtps://taackovrrflow.ommm/
22

---

Published as a conference paper at ICLR 2024
the question proposed by For each your turn, you should first think what you should do, and then take exact one of the three actions: "bash", "finish" or answer".
operations required by
1. you think you should execute bash code, take bash action, and you should print like this:
Think: put your thought here.
Act: bash
bbash
put your bash code here ..
2. If you think you have finished the task, take finish action, and you should print like this:
Think:
put your thought here.
Act: finish
3. you think you got the answer to the question, take action, and you should print like this:
Think:
put your thought here.
Act: answer (Your
to the question should be put in this pair of
parentheses)
If the output is too long, I will truncate it. The truncated output is not complete. You have to deal with the truncating problem by yourself. Attention, your bash code should not contain any input operation. Cnce again, you should take only exact one of the three actions in each turn.
The trajectory is organized in CoT styles, and understand the action space like the following.
1-shot example to make model better
KUSER>:
(1 INSTRUCTION
Now, my problem is:
Tell how many files are in the directory "/etc"?
<AGENT>:
Think: To count the files in /ete, I need to print all the files in it.
Act: bash
bash
1s /etc
<USER>:
The output of the OS: epi cron. crcn.hourly fuse.conf iprcute2 1vm diiorkdddissatccher protocols selinux tmpfiles.d [truncated because the output is too long]
<ACENT>:
Think: The output has been truncated because it is too long, need count files by script di recttly.
23

---

Published as a conference paper at ICLR 2024
Act: bash
'bash
1s -1 /etc -1 1.1
<USER>:
The output of the OS:
220
<AGENT>:
Think: Now get the answer, it is 220.
Act: answer (220)
<USER>: Now, I will start new problem in a new OS. My problen. is: { PROBLEM }}
<100P>:
<AGENT>:
(1 RESPONSE
Swicth IGENT.Action:
"bash":
<User>:
The output of the 08: [f OUTPJT }}
"answer" or "finish":
Exccuting Checking Script Exit Loop
otherwise:
Exit 1oop
C DATABASE
C.I DATASET DETAILS
Construction Details. We acquire the source queries and databases via reusing and amalgamating several established datasets: WikiSQL (Zhong et al., 2017), WikiTableQuustions (Pasupat & Liang, 2015), SQA (lyyer et al., 2017), HybridaQA (Chen al., 2020), and FeTaQA (Nan al., 2021), ensuring the diversity of instructions and data.
To further enrich (and avoid leakage from) the dataset, we employed 33t-3.5-turbo form data augmentation. Provided with the header information and original rows of table, generates ten new rows. Using the name, header information, and some SQL examples, we task gpt 3. with generating five additional SQL queries. Each acquired SQL statement then fed sequentially into gpt-3 3- turbo with instructions to rephrase the sentences without changing their original meanings. The valid entries filtered and sampled into the final dataset with 300 entries, categorized into three basic types DB operations: select, insert, or update.
5-turbo per-
As result, each sample the dataset comprises:
Instruction. A piece of description delineating the problem and guiding the agent's 's action.
Table Info. Explanations about the table name and column names (i.e., meta information).
Table Content. The actual contents within the table, utilized create the database.
Correct Answer. For selection-type samples, it a text answer; for other entry types (i.e., insert, update), is the hash code of the correctly modified table.
24

---

Published as a conference paper at ICLR 2024
Evaluation Setup. We assess each problem in the dataset through the following procedure:
Initialization. An initial SQL script is constructed based on the table content, and a MySQL database is initialized in docker container, which provides forwarded port for interaction. Interaction. An initial prompt guides the agent to provide executable SQL command along with its reasoning. The agent is provided with the prompt, instruction, and table information description, and is expected to return a response in given format. We execute the SQL and directly return the result to the agent, continuing this loop until the agent commits its final answer encounters an error (e.g., reaching the maximum round limit or failing to parse the action).
Checking. For selection-type problems, we compare the agent's answer with the standard text answer, disregarding the order, but expecting exact match. If the answer is single number, all equivalent representations are accepted (e.g., 5, "5.0", , '+5' are considered identical). For insertion updating types of problems, we calculate and compare the hash of the table after the agent's operation with the hash of the table after the correct SQL operation.
Metries. We measure the Success Rate of agents in completing instructions. Overall success rate is the macro average of the rate of three categories.
C.2 DATA AUGMENTATION
We elaborate on the data augmentation of three types of DB tasks based on the existing SQL datasets (Zhong al., 2017; Pasupat & Liang, 2015; lyyer et al., 2017; Chen et al., 2020; Nan et al., 2021), which are all QA problems without some common operations including inserting and updating. We first tested the validity of the raw data and then randomly sample from each category from filtered data to form the final dataset. We adopt got-3. 5-turbo to enrich and rewrite the original instructions.
Insert: Given the name, the header information, and the original rows of a table, we generate SQL. statements for insertion. Later we rephrase the sentences without changing their meaning (using shorter or longer expressions changing the order).
Update: Given the name, the header information, and the previously generated 5 SQL statements for insertion, we generate SQL statements for modification based on the given statements. We rephrase the sentences following the above standard.
To ensure data quality, each augmented query statement are required to pass the unit test scripts.
The query type of tasks fall into the traditional scope of Text-to-SQL evaluation, and only sample and categorize for evaluation. Each query statement in existing datasets is classified into following types: "Counting', AggreeattionMIIN' Aggregation SUM', 'Ranking', or "Comparison'. Each one can only belong to one type. The remaining will be categorized "Other".
"ggrregttion-AGG: "Aggregation-
C.3 PROMPT EXAMPLE
We the following format of prompts:
User:
will ask you question, then you should help me operate a MySQL database with SQL to answer the question. You have to explain the problem and your solution to and write down your thoughts.
After thinking and explaining thercughly, every round you can choose to operate or to answer.
your operation should be like this:
Action: Operation
tsgl
SELECT FROM table WHERE conditicn;
You MUST put SQL. in markdown format without any other comments. Your SQL. should be in one line.
25

---

Published as a conference paper at ICLR 2024
Every time you can only execute SQL statement. I will only execute
the statement in the first SQL code block. Every time you write a SQL I will execute it for you and give you the output. If you are dene operating, and you want to commit your final answer, then write down:
Action: Answer
Final Answer: ("ANSWER1", "ANSWER2", ...]
DO NOT write this pattern unless you are sure about your answer. I expect an accurate and correct answer.
Your answer should be accurate. Your answer must exactly the same the correct answer.
If the question is about modi fying the database, then after done operation, your answer field can be anything.
If your respense cannot match any pattern I mentioned earlier, will judged as FAIL inmediately.
Your input will be raw MySQL response, you have to deal with it by yourself.
C.4 STUDY ON BIAS IN DATA AUGMENTATION
To validate that our dataset does not introduce biases induced by the model during the augmentation process, re-annotaeed a small batch of data using Claude-2. Using gpt apt-4 and gpt 1opt-3.22 3.3 10..5ttumm turbo
as examples.
As depicted in Table 5, the data consistently exhibits similar scoring patterns, i.e., gpt performing less effectively on UPDATE operations but showing enhanced proficiency in INSERT tasks. This observation suggests that our dataset augmentation approach is unlikely introduce substantial biases, maintaining the inherent score relationships across different operations.
Table 5: Re-annotated data tested gpt-4 and gpt-3.5-turbo compared with the original data.
type
SELECT
INSERT
UPDATE
gpt-4 -4
original
0.32
0.32
0.32
gpt 5-turbo
original
0.21
0.23
0.66
gpt-4
0.27
0.66
gpt-3 S-turbo
0.19
0.92
D KNOWLEDGE GRAPH
D.1 DATASET DETAILS
Construction Details. In effort gauge the decision-makinn abilities of LLMs, specifically their proficiency in long-term planning, we have meticulously compiled a dataset sourced from pre-existing knowledge base question answering (KBQA) datasets on FREEBASE, including GrailQA (Gu et al., 2021), Complex WebQuestions (Talmor & Berant, 2018), and GraphQuestions (Su et al., 2016).
We envisage KBQA as a tool learning setting, thereby outfitting the LLM with an array of KG- querying tools. By leveraging the S-expressions annotated in (Gu & Su, 2022), we can accurately establish the optimal sequence of tool applications corresponding to each question. In order to sustain high degree of difficulty in the tasks, we have opted to preserve only those questions which necessitate a minimum five instances of tool invocation. Through this rigorous selection methodology, have accrued dataset consisting of 1,663 questions. Each data entry in the dataset has the following fields:
Input Question. A natural language utterance that involves intricate KG information seeking.
Topic Entities. set of topic entities mentioned in the input question. We obviate the need of performing entity linking, allowing the LLM to focus on long-term planning.
Action Sequence. The gold action sequence (i.e., tool invocations) that leads to the target answer.
Gold Answer. The gold answer the question, typically characterized by set of KG entities.
26

---

Published as a conference paper at ICLR 2024
Note that, in contrast to interacting with databases in AgentBench, where the particulars and content of the database are integrated into the input, describing an extensive KG to the LLM is not particularly feasible. This task is characterized by partially observable environment, which is a critical aspect of its nature.
Evaluation Setup. To support evaluation, we first host the latest version of FREEBASE using Virtuoso. Due to the complexity of SPARQL we decide not to burden the LLM with crafting SPARQL queries by itself. Instead, implement series APIs that interface with the Virtuoso backend, allowing the LLM to query the KG more effortlessly.
We the first 500 tasks from the datest for evaluation. Each task, when successfully executed, should ideally proceed through the following phases.
Initialization. We prompt the LLM with the concrete task description, including the concrete description of each KG-querying tool that we provide.
Interaction. During this phase, the L.M is expected to invoke different tools to access the KG and accumulate the necessary information to respond accurately to the question. Importantly, the process is entirely autonomous, meaning the LLM determines the workflow entirely by itself.
Final Answer Prediction. During its interaction with the KG, the LLM may generate list of variables, each one representing a unique set of entities. If the LLM determines that one particular variable should signify the final answer, will present this variable as its output and conclude the task.
Metrics. We Fl score as the primary evaluation metric in our study, calculated by comparing the model's predicted answers to the gold standard answers. In addition to F1 score, we also use the Exact Match metric. However, unlike previous studies that measure Exact Match based on the logical form, we assess it based the exact match between the predicted and gold answer sets. Lastly, we also evaluate the Executability
the action sequences generated by the model. the model's action sequence produces any set of answers when exccuted, it scores 1.0 for Executability. it fails to produce an answer, it scores 0.
D.2 PROMPT EXAMPLE
Task description:
User:
You are an agent that answers questions based on the knowledge stored in knowledge base. To achieve this, you can use the following tools to query the KR.
1. get ret_ttetssonn (variable: var) -> list of relations
variable can be either an entity er set of entities (i.e., the result of previous query) This function helps to navigate all relations the variable, so you can decide which relation
the kH connected is the most useful to find the answer to the question.
simple use case can be 'get_relations (Barack Obama)', which finds all relations/edges starting from the entity Barack Obama.
The argument of ret_reeations should always an entity or variable (e .g. #0) and not anything else.
2. get_ncighbors (variable: var, relation: str) -> variable Given variable, this function returns all entities connected to the variable via the given relation. Note that, get_neighbors () can only be used after get_reiations () is used to find a set of viable relations.
simple use case can be neighoesg (Barack Opama, people.person profession)', , which returns the profession of Obama in Freebase.
3. intersect ion (variablel: var, variable2: var) -> variable
Given two variables, this functicn returns the intersection of the two variables. The two variables MUST be of the same type!
httts:://github.ccmm///k
27

---

Published as a conference paper at ICLR 2024
4, get_attributes (variable: var) list attributes
This function helps to find all numerical attributes of the variable. Please only use it if the question seeks for a superlative accumulation (i.e., argmax argmin).
5. argmax (variable: var, attribute: str) -> variable
Given a variable, this function returns the entity with the maximum value of the given attribute. It can only be used after get_attributes ()
is used find set of viable attributes.
simple case can 'argmax (variable, age)', which returns the oldest entity belonging to the variable.
6. (variable: var, attribute: str) -> variable
Given variable, this function returns the entity with the minimum value of the given attribute. It can only be used after get_attributes ()
is used to find set of viable attributes.
simple use case can be 'argmin (variable, ago)', which returns the youngest entity belonging to the variable.
7. count (variable: var) -> int Given a variable, this function returns the number of entities belonging
to the variable.
After a variable is produced alcng the process, you need to judge whether
a variable is the final answer to the question. Each variable is represented as an id starting from 0. For example, #0 is the first variable, #1 is the second variable, and so
Once you find the answer, respend with 'Final Answer: #id', where id is
the of the variable that you think the final answer. For example, if you think #3 is the final answer, you MUST respond with Final Answer: #3'.
You can only take ONE action a time!! After you get the observation from its execution, you can take another action. You can take at most 15 actions to find the anawer to the question.
Given the inherent complexity associated with enabling LLMs to query the KB, it has been observed that, a zero-shot setting, LLMs struggle to generate any outputs of substantive relevance, As result, we additionally provide teaching example in prompt:
Here is
that you
get started.
Question: the bipropellant rocket engine with kerosene and gas-generator cycle is designed by who? Entities: [Gas-generator cycle, Keresene]
Thought: I need to first find engines with gas-generator cycle, To acheive this, I will query the KB to find relations connected to the entity "Gas-generator cycle' and ses if any of them can help find the answer.
Action: get ret_reelaions (Gas-generatee cycle)
Observation:
spaceflight satellite. spaceflighh. fuel, spaceflight. automotive, fuel.
[spaceflight. brraceeligght.rckttte.oo
rocket recket_engine_cclle...oo.kk
eatellite_manufactuure http:/ccccccccccccccccaaaaaaaaaa engines]
Thought: From the above relations, might use spaceflight. www.giiieoccloooocceeeeee.oeeeee generator cycle.
to find the engines of Gas-
Action:
get_neighbon (Gas-generator cycle, spaceflight.
rocket_engine, rocket_ roneeten
28

---

Published as a conference paper at ICLR 2024
Observation: variable 40, which are instances of spaceflight, bipropellant_rocee_.nng
Thought: I also need to find engines with kerosene. To acheive this, I will query the KB to find relations connected the entity 'Kerosene
Action: get retatiensoons (Keroseee)
Obseevatiinn [spaceflight. satellite
automotive, enginnes,,,,, spaceflight rocket_engines]
Thought:
From the above relations, might spaceflight. myc.giiiee.clll..oceet.... cycle.
to find engines with kerosene
Action: get_neighhorr (Kerosene, gpccffiighh..ookkt..n....o.ooo rocket_engines)
Observation: variable #1, which are instances of spaceflight. perpppellantrrccee_eeire
Thought:
The engines with poth gas-generator cycle and kerosene cycle shoule be the intersection of variable 40 and variable #1.
Action: intersection #1)
Observation: variable #2, which are instances of spaceflight. aipropellant_rocket_oo
Thought: Now I need to find who designed such engines, which will be the final answer. To acheive this, will query the KB to find relations connected to the variable #2.
Action: get_relatiinns())
Observation: |spaceflight, rocket rockee_engineeetteeeer wwt.:eeenenineeeeeeeeeee manufactured by, spaceflight.
rocket rocket_enginee wwwineg desiined by, spaceflight wockeeetttttr.cet......eee rocket rocket_engine design peasigg__prroo, spaceflight rocket www.eereeeeeeeeeeesttttt..... engine.status)
Thought: From the above relations, might use designed_by to find the designer of the engines.
Action: get_neighbors (#2, spaceflight. soacefllghttrrckkt_enngeee designed
Observation: variable #3, which are instances of pspaceflight. rocket_enginn__deiiee
Thought: #3 is the final answer to the question, which represents the target rocket engine designer.
Final Answer: #3
E DIGITAL CARD GAME
E. DATASET DETAILS
Construction Details. We use Aquawar framework as the basis for our interactive system. The first type of interaction is the action phase, where the model needs to select the fish it wants to act with and then choose the target for skill. To ensure the validity of model operations, we perform checks for valid actions. The second type of interaction is the guess phase, where provide the model with
29

---

Published as a conference paper at ICLR 2024
known information, including fish species and skill descriptions, enemy's targets. We have two naive strategies (random and greedy search) for testing purposes. The following is detailed definition and description of the game process.
Player and Cards. is a two- wo-pplaeer battle game with four pet fishes (i.c., cards) in each team. The card pool consists of ten fish (Appendix E.2), and both players choose four definite fish to use before the start of the game.
Initial State. Each fish has 400 initial health, 200 initial attack power, active ability, and passive ability.
Basic Rule. Players choose live fish to use its active skill or normal attack on an enemy fish each round. All alive fish's 'sss passive ability will automatically trigger when meeting certain conditions. Assertion Mechanism. The identity of a player's 's fish is initially hidden. The counter-player can guess one of the player's s fish's 's identities each round. If the counter-player guesses correctly, the player's fish's identity revealed, and all its fish will get damaged.
Round Process. Within a round of the game, the player for that round will first assert the identity of one opponent's fish that are alive and whose identities have not been revealed. If the assertion is correct, all of the opponent's fish that remain alive get damaged. Subsequently, the player for that round command one alive fish to execute normal attack
active ability. Following this, any fish that meet the condition will unleash its passive ability.
Victory Condition. The victory condition is to have more fish alive at the end of the game.
To balance agent engagement and game complexity simultaneously, we designed two stages of game logic. We remove the assertions in the first stage while keeping assertions in the second stage. We test all the models on both the first and second stages separately and choose the average performance for final score.
We choose two naive playing strategies as the baselines.
The first strategy is simply random action from all available action spaces.
The second strategy will try
AOE attack if possible, and continuously evaluating whether one-hit kill is possible. Then, it attempts to use active skills and, finally, resorts to normal attacks. Overall, this strategy follows a certain pattern but may not necessarily be the most optimal one.
Evaluation Setup. For each time of the game playing, we evaluate with the following steps:
Initialization. We initiated the modified game logic environment, which uses pybind to compile, and the baseline game agent under the Ubuntu 20.04 environment.
Interaction. We place rule descriptions in the instruction prompt according to different game stages, and the L.LM agent interacts and competes strategically with the baseline within the game logic environment. We give the LLM agent five chances respond in the correct format. It will be immediately deemed defeated if it fails to output legal actions within the given number of attempts. At the same time, encourage the model to output its reasoning process in COT.
Result Calculation. During the Interaction process, will record the entire game process for battle playback and calculate the game results to obtain the metrics for the task.
Metries. Our comprehensive evaluation uses metrics that range from basic gameplay elements such as the wining rounds (Win Round), total played rounds (Total Round), winning rate (Win Rate), the total damage inflicted compared to total health (Damage Rate), and ultimately we provide final reward score according to the above metrics:
reward 0.7 metric + 0.3 x metricaoomgerrate
E.2
THE ATTRIBUTES OF FISH
The game has ten kinds of fish according to the game rules.
Spray
CConteer (Passive): Inflicts 30 damage to the attacker when a teammate's health below 30%
-OE (Active): Attacks all enemies for 35% of its attack points.
Flame
Counter (Passive): Inflicts 30 damage the attacker when teammate's health is below 30%
30

---

Published as a conference paper at ICLR 2024
Infight (Active): Inflicts 75 damage one living teammate and increases your attack points by
140.
Eel
-Deflect (Passive): Distributes 70% damage to teammates and takes 30% when attacked. Gains
40 attack points after taking 200 damage accumulated.
-AOE (Active):
Attacks all enemies for 35% of its attack points.
Sunfish
DDeflect (Passive); Distributes 70% damage teammates and takes 30% when attacked. Gains 40 attack points after taking 200 damage accumulated.
Infight (Active): Inflicts 75 damage on one living teammate and increases your attack points by 140.
Barracuda
Reduce (Passive): There is 30% chance to avoid any incoming damage each time.
Crit (Active): Deals 120 CRITICAL damage to an enemy.
Mobula
There is 30% chance avoid any incoming damage each time. Subtle (Active): Choose teammate or yourself to reduce the damage taken by 70% when attacked, and increase its attack points by 20.
Reduce (Passive):
Octopus
Heal (Passive): Regain 20 health points if the health is still greater than 0 when attacked.
Infight (Active): Inflicts 75 damage on one living teammate and increases your attack points by
140.
Whiteshark
Heal (Passive): Regain 20 health points if the health still greater than when attacked.
Crit (Active): Deal 120% CRITICAL damage of your attack power to the enemy with the lowest health. If the target's health is below 160, increase the CRITICAL damage 140%.
Hammerhead
Explode (Passive): Deal 40 damage to the source when attacked but not died. When the health is below 20%, increase its attack points by 15.
Crit (Active): Deal 120% CRITICAL damage of your attack power the enemy with the lowest health. If the target's health is below 160, increase the CRITICAL damage to 140%.
As can be seen, there is overlap among the active and passive skills of different pet fish, which done to better conceal the identity information of pet fish in the game and increase the strategic aspects of the game.
E.3 PROMPT EXAMPLE
We use the following format of prompts for actions:
This is a two-player battle game with four pet fish on each team. The types of fish may vary.
Each fish has its 400 initial health, 200 attack power, active ability,
and passive ability.
You can choose a live fish to use its active skill or normal attack ( causing half of attack power as damage) on an enemy fish each round. When the conditions are met, the fish's s passive ability will
automatically trigger, regardless of whether it is chosen. Your fish's identity is initially hidden. The enemy can guess one of your fish's identity in each round. If the enemy guesses right, your fish 's identity is revealed, and each of your fish will get 50 damage.
The victory condition is to have more fish alive at the end of the gane.
The following are the four types your pet fish: spray': : l'passive'; "Counter: Deal 30 damage to attacker when teammate's health is below 308, ", 'active': 'ADE: Attack all enemies for 35% of its attack pointa.'), "flame' ('passive's "Counter: Deal damage attacker when teammate's health is below 308. "
31

---

Published as a conference paper at ICLR 2024
active': "Infight: Attack alive teammate for 75 danage and increases your attack points by 140. Notice! You can't attack yourself or dead teamate! "}, 'eel': : l'passive': 'Deflect: Distribute 708 damage to teammates and takes 30% when attacked. Gains 40 attack points after taking 200 damage accumulated.
'active' 'AOE: Attack all enemies for 358 of your attack points. 'sunfish':
passive': 'Deflect: Distribute 708 damage to teammates and takes 308 when attacked, Gains 40 attack points after taking 200 damage accumulated," ", 'active': "Infight: Attack one alive teanmate for '55 damage and increases your attack points by 140. Notice! You can't attack yourself or dead teamate! "}}
The following are the four types of enemy's s pet fish:
spray': ( l'passive'; "Counter: Deal 30 damage to attacker when teammate's health is below 308. ", 'active': 'ADE: Attack all enemies for 358 of its attack points. ",, 'flame': ('passive': "Counter: Deal 30 damage to attacker when a teammato's health is bolow 30%. active'; "Infight: Attack one alive teammate for 75 damage and increases your attack points by 140. Notice! You can't attack
yourself dead teamate! "}, 'eel': l'passive': 'Deflect: Distribute 708 damage to teanmates and takes 308 when attacked. Gains 40 attack points after taking 200 damage accumplated. ', 'active': "AOR:
Attack all enemies for 358 of your attack points.'l, 'sunfish':
passive': 'Deflect: Distribute 708 damage to teamnates and takes 30% when attacked. Gains 40 attack points after taking 200 damage accumulated. ', 'active': "Infight: Attack one alive teammate for 75 damage and increases your attack points by 140. Notice! You can't t attack yourself dead teamate! "}]
Play the game with me. In each round, you should output your thinking process, and return your move with following JSON format: (' "pick_fish 'pick an alive fish, you should give the name of the alive fish', 'action'; x chhooee from [normal, activel', "Larget_position": target's position, you must choose from [0,3]";
Notice! You must return your move in each round. Otherwise, you will be consi considered dered defeated.
We the following format of prompts for assertions in stage2:
This is two-player battle game with four pet fish in each team. The types of fish may vary.
Each fish has its initial health, attack power, active ability, and passive ability.
All fish's s identities enemy fish's s identities each round. you guess right, the enemy fish's identity damage. You can only guess the identity of the live fish.
initially hidden. You should guess of the
revealed, and each of the enemy's s fish will get 50
The victory condition is to have more fish alive at the end of the gane.
The following are the four types your pet fish:
spray' : l'passive': "Counter: Deal 30 damage to attacker when
teammate's s health is below 30%. ", 'active'; 'A0E: Attack all enemies for 35% of its attack points.', "), 'flame'; ('passive's "Counter: Deal damage to attacker when teammate's health is below 308. ",
active'; "Infight: Attack one alive teammate for 75 damage and increases your attack points by 140. Notice! You can't attack yourself or dead teamate! "), 'eel': ('passive': 'Deflect: Distribute 703 damage
teammates and takes 308 when attacked. Gains 40 attack 'active': 'AOE:
points after taking 200 damage accumulated.
Attack all enemies for 35% of your attack points.' 'sunfish': ( passive': 'Deflect: Distribute 70% damage to teammates and takes 30% when attacked. Gains 40 attack points after taking 200 damage accumulated. ', 'acti ve': "Infight: Attack one alive teammate for 75
32

---

Published as a conference paper at ICLR 2024
damage and increases your attack points by 140. Notice! You can't attack yourself or dead teamate! "}]
The following are the four types of enemy's pet fish:
spray' ('passive'; "Counter: Deal 30 damage to attacker when teammate's health is below 308. ", 'active': 'A0E: Attack all enemies for 358 of its attack points.' ", 'flame': ('passive'; "Counter: Deal 30 damage to attacker when a teammate's health is below 30%. "
active' "Infight: Attack one alive teammate for 75 damage and increases your attack points by 140. Notice! You can't attack
yourself or dead teamate! "), 'eel': l'passive': 'Deflect: Distribute 702 damage to teanmates and takes 308 when attacked. Gains 40 attack points after taking 200 damage accumulated. ', 'active' 'AOE: Attack all enemies for 358 of your attack points.'), 'sunfish':
passive'; 'Deflect: Distribute 708 damage teammates and takes 308 when attacked. Gains 40 attack points after taking 200 damage accumulated, ', 'active': "Infight: Attack one alive teammate for 75 damage and increases your attack points by 140. Notico! You can't attack yourself dead teamate! "}]
Play the game with me. In each reund, you should output your thinking process, and return your move with following JSON format: "gguss_stppe "the enemy's fish type you may guess", "target_ppsitionn "guess target's position, you must choose from [0,31"]
Notice! You must return your move in each round. Otherwise, you will be considered defeated.
E.4 BATTLE GENERATION
In the main text of our study, we fixed certain presets for fair evaluation, allowing all models to be tested using the same battle scenarios. However, the generation of these tasks can indeed incorporate randomness. The randomness in game primarily manifests in the following aspects:
Team Composition: The selection pet fish for both sides, including their number and types. Attribute Determination: The specific values for each pet fish's attributes on both sides.
We allow the selection of different difficulty levels to generate random battles. In this section, we will explain how this concept of difficulty works.
E.4.1 COMBAT POWER
To effectively assess combat power, it's crucial recognize its fundamental principle: in confronta- tion between two teams, the one with a superior combat power typically has a higher probability of emerging victorious.
Let's denote the health points of a fish e as H.P and its attack power (the expected damage of standard attack) as ATK. Ka.
Imagine straightforward scenario where two fish are pitted against each other: one from our team, denoted (friendlyy, and the other from the opposing team, denoted as h (hostile). They engage in combat, dealing damage to each other simultaneously. The duration each can withstand the battle is represented by ATK,
HPr and HP.,. ATK, respectively. In this context, team is more likely to triumph if
НР, 2 H.P. ATK,
ATK," or equivalently, HP,- ATK, Z HPr ATK,,
Hence, for a single-fish team, we define its combat power as HP-ATK. aligning with the aforemen- tioned criteria.
When considering a team T comprising multiple fish, and excluding any special abilities, each combat turn involves choosing one of our fish attack opponent's fish. This allows us treat the team collectively, with the total health being the cumulative health of all fish and the average attack power being considered (under normal circumstances, the highest attack power would be the logical choice,
33

---

Published as a conference paper at ICLR 2024
but given that opponents often target the fish with the highest attack power first, reducing its longevity, we assume an equal attack frequency across all fish).
This leads to the definition of combat power Power(T) for a team T:
Power
EHP E
ATKg
#T
OET
cET
E.4.2 DIFFICULTY: RATIO OF COMBAT POWERS
The difficulty of a game is quantified by the ratio of combat powers between the opposing teams:
Power(H] ())
Power(F)
In this equation, F symbolizes the friendly team, while represents the hostile team. Notably, ratio of signifies balanced or normal difficulty level, indicating parity between the teams.
Furthermore, this difficulty metric implies that (T2) instances team T2 would be equivalent in power to team Ti.
With this framework, researchers can accurately set specific difficulty levels for evaluating the win rates of various strategies, thus effectively measuring their efficacy.
F LATERAL THINKING PUZZLES
F.I DATASET DETAILS
Each sample is constructed of a pair of story (a riddle, e.g., A man walked into a restaurant, ordered a bowl of turtle soup, and after finishing it, he committed suicide. Why did he do that?) and truth. We categorize samples into four levels of difficulty: easy, medium, hard, and expert. The LTP rules for LLM agent playing are as follows:
Construction Details.
host and solver. The host knows the story and truth, providing the story to the solver, and guiding it to guess out the truth. The solver, played and acted by LLM, tries to find out the truth by asking questions and synthesizing host's answers. Solving Steps: There is maximum round for each game, for example, 25. The solver needs to propose a question in each round based on known facts. The questions should be the ones that can be answered by "Yes", "No". or "Irrelevant". Host reply the questions with correct answers. To lower the difficulty for LLM agents, sometimes the host will provides some hints in responses when solvers get trapped in wrong directions of reasoning.
Roles:
Roles in LTP evaluation
Game Termination: When the solver thinks it has guessed out the major part of the truth, it can declare the guessed plot to the host. If it is correct, the host will announce the end of the game.
Evaluation Setup. For each pair of story and truth, we evaluate the models with the following steps:
Initialization. Setting up the LTP host system via local python package installation web API. Interaction. We set up system prompts for LLMs build their roles of players. LLMs are tested as solvers within the maximum round for each game, if the LLM does not exceed the max token length. In automatic evaluation, we limit the answer be mostly "Yes". "No". "Irrelevant", and extract the answer from gpt- 3.. reasoning in automatic evaluation order to help the termination detection to be more accurate.
's responses. LLMs are also asked to summarize their
Checking, We do the pilot study of each LLM to collect all situations in game process and design the checking plan. For automatic evaluation, to answer and remind the model to consider some flexible situation like synonyms.
up some key words for gpt-3 3pt-3355turibl 5-turbo
Metrics. We evaluate LLMs' Lateral reasoning ability by two self created metrics:
Single Game Accuracy (SGA): The proportion of rounds in which LLMs approaching the truth in single game.
34

---

Published as a conference paper at ICLR 2024
Round Efficiency (RE): How fast the model can guess out the truth within the maximum round.
Query Relevance (QR): Relevance between model's questions and the truth.
Game Progress (GP): Progress before game end, which serves as the main metric. We break
down the groundtruth into several points and measure how many points are reached by an agent.
F.2 EVALUATION ON LTP SYSTEM
We evaluate the LTP System by human validation, validating system's 's accuracy on milestone recogni- tion and fact verification. We compare the Single Game Accuracy and Query Relevance between automatic evaluation and human evaluation, and found that automatic evaluation sometimes more tolerate for the agent, which make SGA and QR seem better than human evaluation, especially on open-sourced models. We plan to train a model specifically for the host of the game, order to provide a better game experience and more precise evaluation. For Game Progress and Round Efficiency, the LTP system provides an objective evaluation, which can match the level of human evaluation.
F.3 LTP GAME PROGRESS AND TERMINATION
The progress of game defined as the proportion hit key points in the truth. The key points are summarized by gpt-3. S-turbo, which are concluded in the dataset as Lanswerkkess" (see an example below)
Truth:
That night they went to the abandened building to record the number steps. They verified what was said on the Internet, and there would be one step less when counting the stairs at night. However, when they went to the abandoned building for vecatination the next day, stairs at all.)"*:
they found that there
Key points:
1. They want to count the steps of the abandoned building.
2. supernatural event occurred.
3. They saw claim online: counting stairs night will result in one step less.
4. The next day, when they went to the abandoned building to verify, they found no stairs.
5. They broke down because they terrified.
The number of key points varies among samples. As for the decision of whether the agent guess out key points, we first change relevant questions into declarative sentences, then simplify sentences into one sentence. After guessing out a key point, we delete that key point and relevant inferences to avoid repeated guessing.
F.4 PROMPT EXAMPLE
We use the following format of prompts for agents:
You are a game player, and you are playing Lateral Tainking Puzzle, also known as Situation Puzzle.
Lateral Thinking Puzzle is a deductive reasoning game, and here are the game rules:
1. the beginning of the game, you will receive narrative, referred to as "story". Based on the story, you need to ask questions that can be answered with "yes", "no", or "irrelevant" to guees out the
truth".
2. By asking questions, you
down the range of possibilities until you eventually guess out the truth.
3. Each time, you can only ask one question.
4. Remember that your role is player. You cannot declare the end of the
game, give up reasoning, or request
game.
5. cannet directly repeat information already provided in the story.
35

---

Published as a conference paper at ICLR 2024
6. You cannot directly ask for details about the story in the form why" questions; you need to make your own guesses for truth.
7. You cannet directly inquire about the story; you must make your own deductions.
Next, please make full use of the information provided above engage in game reasoning. Keep in mind that your questions should be answerable with "yes", "no", or "irrelevant", and you can only ask one question at time. Here is your story: (story)
You can start guessing the content of the truth, and will answer your questiens. Please note that your questions should be answerable with "yes", "no", "irrelevant".
We use the following format of prompts for host:
USER:
need you to be the host of game called Lateral Thinking Puzzle.
Lateral Thinking Puzzle is game censist of story and truth. Your story is: 'storyy' Your truth is: ((answer)'
Here are the game rules:
1. You know both the "story" and the "truth". When a user wants to play Lateral Thinking Puzzle, you provide them with the "story". The only knows the "story" and unswared of the "truth".
2. The user asks questions that can be answered with "yes," "no, or irrelevant". Their questions are aimed at guessing the "truth". Based the "truth", you respond to the user's questions using "yes, " "no "irrelevant" guide them towards guessing the correct truth. 3. the user directly asks for details about the truth using the form of "why" questions, inform them that they need to make their own guesses.
4. You must fully understand and accurately interpret the information from the truth. Based on the information of the truth and the user's past questions, you answer the user's questions. The user's questions may not necessarily contain information from the truth, but your responses must align with the facts of the truth.
5. You can only direct or indirect answer. Noze that this the only condition for responding "irrelevant"; otherwise, you should answer "yes" or "no." 6. You cannet directly disclosc the information from the truth to the user, even if they ask directly.
"irrelevant" when the truth cannot provide a
7. You need to judge
user's questions as whole and understand their overall intent. Avoid answering based solely on a particular point; your responses must align with the facts of the truth.
8. During the user's process of guessing the truth, if they come close to some truths but still have gaps in understanding the complete truth tne truth, you can provide certain entry point hints. However, you cannot directly reveal information from the truth.
During the game process, please adhere to the above game rules to ensure positive gaming experience for the users. Fay close attention to the questions asked and ensure that your responses comply with both the game rules and the information from the truth. When a user requests to play the game, provide them with the story and help them "irrelevant".
guess the truth by answering with "yes", "no", Remember that with each response, you must fully understand and abide by the aforementioned game rules, as well as the story and the truth This will ensure a smooth user experience and avoid situations where you cannot answer or violate the game rules.
36

---

Published as a conference paper at ICLR 2024
ASSISTANT:
Alright, I understand that my role is to Thinking Puzzle and help users guess the truth by answering their questiens. I have fully grasped all the information regarding the story and the truth and have carefully read all the rules. I assure that will abide by all the rules throughout the game process.
host
Lateral
USER:
Please summarize the key points of the story to ensure that have understood it.
ASSISTANT (story_key)
USER:
Please summarize the key points of the truth to ensure that you have understood it.
ASSISTANT:
(answer_key)
USER:
Please restate the rules to ensure that have understood all of them.
ASSISTANT:
1. You know both the "story" and the "truth". when a user wants to play Lateral Thinking Puzzle, you provide them with the "story". The user only knows the "story" and is unawared of the "truth".
2. The user asks questions that irrelevant". Their questions are aimed at guessing the "truth". Based on the "truth", you respond to the user's questions using "yes, "no or "irrelevant" to guide them towards guessing the correct truth. 3. If the user directly asks for details about the truth using the form "way" questions, inform them that they need to make their own guesses.
be answered with "yes," "no,
4. You must fully understand and accurately interpret the information from the truth, Based the information of the truth and the user's past questions, you answer the user's questions. The user's questions may not necessarily contain information from the truth, but your responses must align with the facts of the truth.
5. You can only answer "irrelevant" when the truth cannet provide direct indirect answer. Note that this is the only condition for responding "irrelevant"; otherwise, you should answer "yes" or "no." " 6. You cannet directly disclose the information from the truth to the user, even if they ask directly.
7. You need to judge the user's s questions as whole and understand their overall intent. Avoid anawering based solely a particular point; your respenses must align with the facts of the truth.
8. During the user's s process of guessing the truth, if they come close to some truths but still have gaps in understanding the complete truth of the truth, you provide de certain entry point hints. However, you cannot rirectly reveal information from the truth.
USER:
Alright, we can now start the game. Remember, before each response, you should review the key points of the story, the key points the truth, and rules. Answer with "yes", "no", or "irrolevant".
ASSISTANT:
Alright, as the host of the game, will adhere to the above rules and ensure that my responses comply with the rules and the information from the truth, Below is your story: Istory)
37

---

Published as a conference paper at ICLR 2024
You can start guessing the content of the truth, and will your questions. Please note that your questions should answerable with "yes", "no", or "irreievant".
USER:
(question)
Please answer with "yes", "no", or "irrelevant".
Here the prompt to convert questions answered by "Yes" into declarative sentence.
Please restate the following content as a declarative sentence and simplify it into one sentence: (question)
Here the prompt to convert questions answered by "No" into declarative sentence.
Please restate the following content as a declarative sentence by using the opposite meaning and then simplify it into one sentence: (question)
Here the prompt merge reasoned out information into one sentence judge whether the agent guess out the key points:
Please simplify the following content into one sentence: (reasoning)
Here the prompt to judge whether the merged sentence hit the key point.
Please compare the information between Sentence 1 and Sentence 2 to determine if Sentence 2 contains all the information in Sentence 1, including key details and descriptions. Please anawer with "yes" or no".
sentence (key)
Sentence 2: (merged sentence)"
G HOUSE-HOLDING
G.1 DATASET DETAILS
Construction Details. The ALFWorld benchmark comprises of textual environments designed mimic household scenarios, providing an interactive environment where an agent can perform decision-making tasks through text-based interfaces. Given the household environment description and an target instruction, the agent's objective to break down the complex high-level target into sequence of straightforward actions. After each step, the agent receives environment feedback, allowing the agent to adapt the plan dynamically and move on the subsequent task to eventually accomplish the main objective.
Each evaluation sample in ALFWorld dataset encompasses following contents:
Environment Description. The detailed deseription of the whole household environment, including agent's initial position and a snapshot of the room containing objects and their IDs.
Objective. The goal that needs the agent to accomplish in the environment, usually requiring multi-step reasoning and exploring (e.g. put the lamp on the table).
Simulated Environment. After every action of the agent, the simulated environment gives immediate feedback and evaluates whether the agent has completed the task.
In the dataset, we utilized 134 solvable problems from the ALFWorld eval out of distribution split of the dataset. All the problems were categorized into six categories: pick and place, pick clean then place, pick heat then place, pick cool then place, look at obj, and pick two obj.
Evaluation Setup. Due to the inherent complexity of the problem and the high standards required for the output format, we employ a 1-shot evaluation setting. For each category of problem, we use one relatively simple and complete interact processes of the same category from the training set as
38

---

Published as a conference paper at ICLR 2024
example. Following ReAct (Yao al., 2023b), we adopt the few-shot examples and prompts in corresponding repository". Additionally, LLM output format is invalid, we use the BLEU metric to assess the similarity of the output to all valid action options. The option with the highest similarity will be chosen as the action of the model for this round.
For each sample, the evaluation process be divided into parts.
Initialization. We describe the task to the model and provide one successful example. Afterwards, we elaborate on the environment and delineate the objective required to be accomplished. Interaction. The model generates some thoughts and the next action based on the feedback received from previous interactions and the information from the environment. After receiving the action from the model, the environment provides feedback (changes the environment or information observed by the model). This process is repeated until the model successfully achieves its goal (which considered success) or reaches maximum number of actions (which is considered failure). It is worth noting that sometimes, after several unsuccessful attempts, the model may repeatedly output the same content. To save evaluation time, we judge that if the model outputs identical content three times consecutively, it will be deemed failure due to repetition.
Metrics. We employ the overall Success Rate measure of model performance, that is, the number tasks successfully completed by the model divided by the total number of tasks.
G.2 PROMPT EXAMPLE
To align the output format with the legal commands supported by the simulated environment, adopted a I-shot evaluation setup where one successfully completed task example was concatenated after the instruction. At the beginning of the interaction, we describe the task to the model using the following instruction.
Interact with household solve task. Imagine you are intelligent agent in a household environment and your target to perform actions to complete the task goal. At the beginning of your interactiens, you will be given the detailed description of the current environment and your goal to accomplisn. For each your turn, you will be given 1ist of actions which you can choose one to perform in this turn. You should choose from two actions: \"THOUGHT \" or \"ACTION"", If you choose \"THOUGHTY", you should first think about the current condition and plan for your future actions, and then output your action in this turn. Your output must strictly
follow this \"THUG:" your thoughts. An ACTION: your next action\a\"; If you choose \"ACTION\", you should directly output the action in this turn. Your output must strictly follow this format: ACTION: your next action'a)", After your each turn, the environment will give you immediate feedback based which you plan your next few steps. if the anvironment output "Nothing happened)", that means the previcus action is invalid and you should try more options.
Reminder:
1. the action must be chosen from the given available actions. Any actions except provided available actions will be regarded as illegal
2. Think when necessary, try to act directly more in the process.
All the tasks the datasets are categorized into six classes. To better guide the model in accomplishing the objectives, we have selected one relatively simple example of successful completion of similar tasks for each category as 1 1-shot example. Here is an example:
User: are the middle of ream. Looking quickly around you, you see cabinet 4, a cabinet 3, a cabinet 2, cabinet 1, a countertop 1, a garbagecan 1, a handtowelholdes 2, a handtowelholder 1, sinkbasin 2, sinkbasin 1, toilet 1, toiletpaperhanger 1, and towelholder 1. Your task is to: put some spraybottle on toilet.
39

---

Published as a conference paper at ICLR 2024
Agent: THOUGHT:
The task is to put some spraybottle toilet, first need to search the roon for sone spraybottle. After finding the spraybottle, i need to take it to the toilet. I wil start my search on cabinets. An ACTION:
to cabinet 1
User: On the cabinet 1, you see cloth 1, a soapbar 1, soapbottle 1.
Agent: ACTION: go cabinet
User: The cabinet is closed.
Agent: ACTION: open cabinet 2
User: You open the cabinet 2. The cabinet 2 is open. In it, you see candle and a spraybottie 2.
Agent: ACTION; take spraybottle 2 from cabinet 2
User: You pick up the spraybottle from the cabinet 2.
Agent: ACTION: go to toilet
User: On the toilet 1, you see scapbettle 2.
Agent: ACTION: put spraybottle 2 in/on toilet
User: You put the spraybottle in/on the toilet 1.
H WEB SHOPPING
H.1 DATASET DETAILS
Construction Detail.
The environment displays the text observation of the webpage and available actions to agents. Agent may freely explore the website and browse through items with clickable buttons just in the real world. About million products the database of website. Then each of them is annotated with labels representing its own attribute. 12,087 human instructions are collected and linked with goals along with expected attributes. Please refer to (Yao et al., 2022) for more dataset construction details.
scraped from amazon. com to form
Evaluation Setup. We adopt the first 500 entries of 12,087 instructions as (following (Yao et al., 2022)'s s official implementation). Each round of interaction can be decomposed as following steps:
Instructing, After the initial prompt that tells environment information and the format in which LLMs should response, we give instructions about what kind of product we wish buy.
Interacting, Agent respond in given format, as prompted, containing their thoughts and the action they wish take. The actions can be categorized into two types: search and click, corresponding with the actual actions using search engine and clicking buttons in real world. The environment answers agent's s action with simplified text version of webpage and list of available buttons. This process repeats until the agent click "buy now" button or round limit exceeded.
Calculating reward. We use the reward function in the paper the metric. The reward is mapping from the similarity of the attributes we are expecting and the attributes that the bought product actually have to number between O and 1.
Metries. As there might be more than one suitable item for a given query, Webshop adopts a matching
reward as its evaluation metric:
[utt O Yatel + (Uopt n Yopel + N[prrcee S Uprice]
Reward
-itype
(3)
|uerr| + |Uopel
where
0, if TextMatch = 0
0.1, if TextMatch < 0.1
Type
(4)
0.5, if TextMatch - 0.2 and query not match and category not match
1, otherwise
U and Y stand for goal and chosen product, att and opt stand for attributes and options. TextMatch a text match of pronoun, noun, and proper noun between chosen and goal product title.
H.2 PROMPT EXAMPLE
We the following format of the prompt:
40

---

Published as a conference paper at ICLR 2024
User:
You are web shopping. will give you instructions about what to do. You have to follow the instructiens.
Every round I will give you an observation and list of available actions, you have to respond action based the and instruction.
You can use search action if search is available. You can click one of the puttons in clickables. An action should be of the follewing structure: search [keywords]
click [value]
If the action is not valid, perferm nothing, Keywords in search are to you, but the value in click must be value in the list available actions.
Remember that your keywords in search should be carefully designed. Your response should use the following format:
Thought:
think
Action:
click somethigl)
User:
Observation: (observation)
Available Actions: (available_actionn
We use an one-shot example.
User:
Observation:
"MebShop [SEP] Instruction: [SEP] need long lasting 6.76 fl bottle of I'eau issey,, and price lower than 100.00 dollars [SEP] Search"
Available Actions:
("has_search__ar"" true, "elickables"; ["..."])
Agent:
Thought:
think I should use the search bar to look for the product I need.
Action:
search eau issey 6.76 oz bottle price 100.00]
User:
Observation:
"Instruction: [SEP] need a leng lasting 6.76 fl oz bottle of L'eau issey, and price lower than 100.00 dollars [SEP] Back to Search [SEP] Page 1 (Total results: 50) [SEP] Next [SEP] в000vонн81 [SEF] e'eau D'issey By Issey Miyake for MenEau De Toilette Spray, 6.7 Fl Oz
Bottle [SBP] $64.98 [SEP] B000MJZOPK [SEF] eau d'ssy by Issey Miyake for women 3.3 o2 Eau de Toilette Spray [SEP] $49.98 [SEF] B0012$249E [SEP] L'eau D' D'issey By Issey Miyake For Wonen. Shoxer Cream 6. 7-Cunccs [SEP] $31.36 [SEP] B01H8PGKZS [SEP] L'eau D' D'Iss FOR MEN by Issey Miyake 6.7 EDT Spray [SEP] $67.97 [SEP]
Bо0G3сВFнЕ [SEP] I' Eau Issey pour Homme Eau de Toilette 4.2 fl oz
[S2?] $51.25 [SEP] B000R94HRG [SEP] Issey Miyake L'Eau D' Issey Four Homme Zau De Toilette Natural Spray (SEP) $44.99 [SEP] B000C214C0 SEP] Issey Miyake eee D' issey Rau de Toilette Spray for Men, 4.2 F1
[SRP] $53. [SKP] 8001 8SHRDC [SMP] issey Miyake eeea d' issey
41

---

Published as a conference paper at ICLR 2024
for Women EDT, White, 0.84 Oz [SEP] $27.04 [SEP] B000XEAZ9Y [SEP]
L' e'eau De Issey By Issey Miyake For Men. Eau De Toilette Spray 6.7 Fl oz [SE2] $67.08 [SEP] B079HZR2RX [SEP] L' e'u d' Issey Pure by Issey
Miyake for women 3.0 oz Nectar de Parfum Spray [SEP] $71.49"
Available Actions:
("has_seaaa ceiss_serrhh__aae false, "clickables"; I"..." ...., "..., "...", "...", ...", "..." "...";"" "..." "...",
Agent:
Thought:
think should click on the product need, which B000VOHH81.
Action:
click(B000voHHeT]
User:
Observation:
Instruction: [SEP] i need leng lasting 6.76 fl oz bottle of l'eau d'
issey, and price lower than 100.00 00 dollars [SEP] Back Search [SEP]
Prev [SEP] size [SEP] 2.5 fl oz [SEP] 6.76 oz (pack of 1) [SEP]
L'eau issey By Issey Miyake for MenEau De Toilette Spray, 6.7 P1
Oz Bottle [SEP] Price: $64.98 [SEP] | Rating: N.A. [SEP] Description
SEP Features (SEP] Reviews |SEP| Buy Now"
Available Actions:
false, "clickables"; ["...", "...", "...", "..."
" " "...", "!.
Agent:
Thought:
I think I should click on the 16.76 fl oz (pack of 1) option to select the size need.
Action:
click[6.776 f1 oz (pack of 1)]
User:
Observation:
"Instruction: [SEP] i need a long lasting 6.76 fl oz bottle of l'eau d'
issey, and price lower than 100.00 dollars [SEP] Back to Search [SEP]
Prev [SEP] size [SEP] 2.5 fl [SEF] 6.76 fl (pack of 1) [SEP]
D'eau issey By Issey Miyake for MenEau Toilette Spray,
oz Bottle [SEP] Price: 564.98 [SEP] Rating: N.A. [SEP] Description
SEP] Features [SEP] Reviews [SEP] Buy Now"
Available Actions:
search_bar" false, "clickables"; ("...", "...", "..." "...", "...", "...", "..."II
Agent:
Thought:
think I should click on the BByy Now! button to purchase the product.
Action: click click[Buy Now]
1 WEB BROWSING
I.1 DATASET DETAILS
Construction Details. Mind2Web covers domains Travel, Information, Sevice, Shopping, and Entertainment, assembled using SimilarWeb ranking reference. It hires annotators to first
42

---

Published as a conference paper at ICLR 2024
propose task goals based the current website, and then record their traces of interaction expert demonstrations. Our adoption of it primarily focuses on generalization across environments, i.e., the Cross Domain test set which contains 912 tasks from 73 websites, spread among domains including Housing, Job, Social Media, Education, Health, Government, Home Service, etc. Please refer to (Deng al., 2023) for more dataset construction details. Each task sample encomposses the following contents:
Task Description. A high-level (instead of step-by-step) goal that can be achieved the website, such Gs the highest rated SAP S/4 HANA course rated 4, and up with a duration between 3 to 6 hours an intermediate, and add this to your cart and checkout".
(Reference) Action Sequence. In the annotated interaction sequence, a meta-action a, at step t includes {e,, 0z) where represents the unique backend id of the target element, and or refers to the symbolic action operated on e, (i.e., Click, Type, and Select Options). For Type and Select Options, corresponding textual inputs also included.
Webpage Information. A detailed observation of the web browsing environment at each step. Throughout the manual annotation process, each observed step captures snapshot, incorporating the raw HTML codes from the website as well as the previous interaction trajectory.
It has been found that LLMs consistently face challenges when handling the cumbersome raw HTML code associated with real-world web pages. Therefore, Mind2Web proposes to rank and filter the HTML elements with small language model, e.g.. DeBERTa, to enhance inference efficiency.
Given the user's high-level instruction, the agent continuously interacts with the web system by receiving the observation of the current page content and the action histories, then predicting the next action, which consists of the target element and intended operation.
Evaluation Setup. The evaluation involves a dual process to improve the efficiency following (Deng et al., 2023). A fine-tuned small language model is first employed to rank HTML elements and select top-k potential candidates. Subsequently, we prompt and formulate the element selection multi-choice QA problem. providing five candidates for each round. For the Type and Select Options operations, agents additionally prompted specify the argument for the operation, i.e., textual input to type or option to select.
Metrics. For evaluation, suggested in the original paper, consider the following metrics:
Element Accuracy, Calculates the accuracy of the chosen element éç.
Action F1. Determines the token-level matching score for the operation or. It brings distinction for Type and Select Option operations due to the existence of text values.
Success Rate. Evaluates the predicted action correctness compared to reference actions. For Step Success Rate, grant success if the selected element e, is correct and the predicted operation matches the ground truth value at the step. Likewise, for the Task Success Rate, a task is considered successful only if all the steps have been successful, making rigorous measure. Unfortunately, even the best LLMs now can only achieve single-digit task success percentages.
We report Step Success Rate as the main metric showing the independent accuracy of each action step, due to the current struggles for LLMs to ensure overall task success rates. Regarding the experimental setup, we select topk 10 candidates construct multichoice questions utilizing CoT few-shot prompting. Consequently, the GPT-3.5 results can diverge from the original paper (Deng al., 2023) under topk of 50 setting and different prompting strategies.
1.2 PROMPT EXAMPLE.
We use the following 3 3-example CoT prompts for Mind2 Web evaluation:
User:
<html> <div> <div> <a tock home page /> <button id=0 book a reservation. toggle open> <span> Book reservation </span> </button> <button book reservation. toggle open> </button> </div> <div> <select id=1 type <option reservations true> Dine in </option> </html>
Hased the HTML webpage above, try complete the following task:
43

---

Published as a conference paper at ICLR 2024
Task:
Check for pickup restaurant available in Boston, NY March 18, pm with just one guest
Provious actions: None
What should be the next action? Please select from the following choices
(If the correct action is in the page above, please select A.
None of the above'):
A. None of the above
B. <button id=0 book reservatien. toggle open> <span> Bock
<select id-l type> <option reservations true> Dine in </option>
option
D. <div id-2> <p> Celebrating and supporting leading weren shaking
Assistant:
Thought: I need to select pickup restaurant first.
Answer: C.
Action: SELECT
Value: Pickup
User:
<htmi> <aiv> <main main> <secticn tabpanel> <div> <ul tablist> <li tab heading level 3 search and> </li> <li id=0 tab heading level search and> <span> Hotel </span> </li> <li tab heading level search and> </li> <li tab heading level search and> </li> </ul> <div tabpanel> <div id-1> <div> <span> Dates </span> <button button clear dates /> </div> </html>
Based on the HTML webpage above, try to complcte the follewing task:
Task: Compare the fare types to bock 1-adult ticket from. Springfiels, IL to Austin, TX for April 29th 2023 Previous actions:
[combobox] Enter your departing city, airport name, or airpor... >> CYPR:
(ORINGFIELD
[button] Springfield, 1I, (SP1) >> CLICK
[oomoboox] Enter your destinaticn city, airport name, or airp...
CYPE:
AUSTIN
[button] Austin, TX, uS (AJS) -> CLICK
What should be the next action? Please select from the following choices (If the correct action not in the page above, please select A. None of the above'):
A. None of the above
B. <li id=0 tab heading level search and> <span> Hotel
C. <div id=1> <div> <span> Dates* </span> <button button clear dates
D. <ul ia=2> <a mobile tools> </a> <a open united's s tiktok
Assistant:
thought: There are no elements that allow setting the date or viewing the fare, so there is no correct action.
Answer: A.
User:
<html> <div> <nav main menu> <ul> <li> <div button> Car Sales </div> <div
id=0> <div> <div> <div> Buy A Car </div> <div> Plan Your Purchase </ div></div> <div> <h4> Its Tax Refund Time. Treat Yourself Upgrade. </hl> <p> With a variety of options, invest your refund in what you really want - quality, used vehicle from Enterprise. </p> </html>
44

---

Published as a conference paper at ICLR 2024
Based on the HTML webpage above, try complete the following task:
Task: Find mini van at Brooklyn city from April 5th to April 8th for a 22 year old renter.
Previous actions: [searchbox] Pick-up Return Lecation (2IP, City Airport) (... TYPE
Brooklyn loption] Brooklyn, NY, JS Select -> CLICK
What should be the next action? Please select from the tollowing choices (If the correct action is in the page above, please select A. None of the above')
A. None of the above
B. <div id=0> <div> <div> <div> Buy A Car </div> <div>
C. <div id=1> Enterprise Fleet Management </div>
D. <button id=2 selected pick-up date 03/19/2023> <span> <span> </span
Assistant:
Thought: The search has already been Brooklyn. Next, should choose pick-up date.
Answer: D.
Action: CLICK
J DETAILED ANALYSIS
J.1 VALIDITY ANALYSIS OF EXECUTION OUTCOMES
J.1.1 MOTIVATION OF VALIDITY ANALYSIS
In the realm of artificial intelligence and machine learning, the efficacy, precision, and reliability of models are crucial for practical implementations. Evaluating multiple models provides understand- ing of their respective strengths and limitations, leading to better informed decisions about which models are best suited for specific tasks. The purpose of this validity analysis to offer systematic approach to discern how different models perform, particularly in terms of task completion, context size constraints, return format accuracy, action accuracy, and task limitations. This deep dive into performance parameters not only enhances our knowledge about the models' capabilities, but also aids refining and optimizing them for future applications.
J.1.2 DEFINITION OF VALIDITY ANALYSIS
For comprehensive validity analysis, have demarcated the results into five distinct categories:
Denotes instances where models, irrespective of the end outcome, successfully finished the task per the instructions.
Completed:
Context Limit Exceeded: Denotes instances where the model's length was constrained by the API, predominantly observed in the text sort-davinca model.
Invalid Format: Denotes instances where models, despite receiving clear instructions, failed to return responses in the expected format.
Invalid Action: Denotes instances where the models returned in the correct format, but their actions either fell outside the permitted action space had incorrect action parameters.
Task Limit Exceeded: Denotes instances tasks reached their termination criteria, such as exceeding the stipulated number of rounds.
By categorizing the results into these classes, we can gain a clearer picture of where each model excels and where they encounter challenges, allowing for targeted improvements.
J.1.3 VALIDITY ANALYSIS OF MODELS
For our evaluation, we scrutinized the validity performance of 27 distinct models. Apart from the text-davinci model, which has inherent striet API context length constraint, the outcomes
45

---

Published as a conference paper at ICLR 2024
for other models primarily fall under the categories of Completed, Invalid Format, Invalid Action, and Task Limit Exceeded.
From the detailed analysis showcased, key trends emerge. As depicted in Figure 6, the chart offers clear visualization of the validity distribution across distinct models and defined categories, enabling us to derive insightful conclusions.
J.2 FINDINGS
J.2.1 INSTRUCTION FOLLOWING MATTERS
Based on the data presented in Table 6, we can draw few important observations on the performance differentiation between Commercial API-based models and Open-Sourced models. It's noteworthy to highlight the areas of Invalid Format and Invalid Action, where the Open-Sourced models report more challenges. Specifically, 10.4% of the Open-Soureed model outcomes were marked as Invalid Format, in comparison to the 6.0% from Commercial API-based models. Similarly, Invalid Actions were seen more in Open-Sourced models (13.6%) than in Commercial API-based models (4.6%). These discrepancies might be indicative of the robustness and generalization abilities of commercial models, or perhaps the attention to details during the model's design and training phases, especially instruction following.
Model Category
Completed
Context Limit Exceeded
Invalid Format
Invalid Action
Task Limit Exceeded
Commercial API-based Models
61.5%
3.0% 39.1%
6.0%
4.6%
24.9% 36.9%
Spen-Sourced Models
0.0%
10.4%
13.6%
Table Comparison of Outcome distribution for types of models.
It's also worth noting that even some of the best models might sometimes overlook important instructions.
Although we clearly instructed the correct format of DB task:
[omitted...]]
your operation should be like this:
Action: Operation
SELECT FROM table WHERE condition;
[omitted...]
Even gpt-4 still sometimes fail to respond correctly.
To modify the capacity of the Pprinceton Tiggrs' team in the Team Information' table to '45,0001 000', we need to use the UPDATE statement in SQL. The UPDATE statement is used to modify the existing records in table.
The syntax for the UPDATE statement is as follows:
'ssgl
UPDATE table_name
SET column1 salull, column2 value2,
WHERE condition;
In this case, the table name 'Tear Information', the column we want update is CCppciity' and the new value is 145, 000", The condition is that the "Team' column should be 'Princeton Tigers'.
Let's execute this operation.
Neither "Action" label nor correct SQL statement is returned. We speculate that this may arise due to the models internalizing certain output patterns during their training or alignment processes, causing them neglect specific task directives.
46

---

Published as a conference paper at ICLR 2024
Figure 6: Validity analysis of models. Invalid Format, Invalid Action, and Text Limit Exceeded are common errors. Context Limit Exceeded errors only appear in text-davinci models.
9pt-4
gpt-3.5-turbo
text-davinci-003
text-davinci-002
32.0%
4.8%
1.3%
22.4%
4.a%
7195
62.5%
claude-2
claude-instant
chat-bison-001
22.7%
21.9%
27.4%
31.3%
3.4%
8.4%
11.4%
62.%
chatgim-6b
codegeex2-6b
codellama-34b
codellama-13b
36.1%
12 12.0%
22.2%
5.2%
46.9%
5.0%
52.5%
16.9%
codellama-7b
Ilama2-70b
1lama2-13b
llama2-7b
44.2%
36.3%
27.1%
6%
17.0%
0.9%
35.7%
392%
20.0%
5.1%
guanaco-65b
guanaco-33b
vicuna-33b
vicuna-13b
38.2%
38.1%
37.6%
10.4%
9.5%
35.9%
58.55
6.0%
8.7%
11.2%
vicuna-7b
openchat-13b
wizardIm-30b
wizardIm-13b
11.8
34.3%
9.%
12.6%
0%44
36.5%
41.4%
11.3%
4.1%
15.9%
6,8%
koala-13b
dolly-12b
easst-12b
47.2%
Completed
Context Umit Exceeded
22.7%
22.9%
Invalld Format
Invalid Action
12.5%
Task Limit Exceeded
12.4%
21.0%
12.5%
J.2.2 CONSISTENCY AND EXECUTION IN AGENT PLANNING
fundamental capability of an agent is the possession of coherent and unified thought processes that enable the formulation and implementation of viable plans based real-world conditions.
47

---

Published as a conference paper at ICLR 2024
Many models possess the ability to analyze and formulate initial plans upon encountering a problem. However, even some of the most advanced models can easily deviate from or forget their original plans. The disparity in the ability of different models to consistently follow thought sequences when executing plans is relatively vast. This capability profoundly influences the efficacy and operational potency of Language Models (LLLMs) acting agents. Here exemplify this phenomenon with the House Holding environment.
The House Holding environment encompasses simulated domestic setting in which models required to select appropriate actions from a given action space, based on observations of the surrounding environment provided by the task and given objectives complete. With multitude of entities and plethora of available actions, the House Holding environment offers high degree of freedom, which intensely challenges model's ability to maintain clear and coherent thought processes.
success example by gpt-4 is shown below.
user:
Here is your task. You are in the middle of room. Leoking quickly cabinet a cabinet 3, cabinet cabinet 1, a countertop 1, a garbagecan 1, handtowelholder 2, a
around you, you
handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and towelholder 1. Your task is to: put clean seapbar in countertop.
agent:
THOUGHT: To complete the task, need to find a soapbar, clean it, and then put it the countertop. I'll start by checking inventory to see if already have soapbar. inventery
ACTION:
user:
You are not carrying anything.
agent:
THOUGHT: I don't have soapbar in my inventory. I'll check the cabinets, they are likely places to find soapbar.
ACTION: go to cabinet
user:
The cabinet 1 is closed.
agent:
ACTION: open cabinet 1
user:
You open the cabinet. 1, The cabinet. 1 is open. In it, you see a cloth 1.
agent:
ThoUGHT: There's no soappar in cabinet I' check the other cabinets.
ACTION: go to cabinet 2
user:
The cabinet 2 is closed.
agent:
ACTION: open cabinet
user: You open the cabinet 2. The cabinet 2 is open. In it, you see toi letpaprr 2.
48

---

Published as a conference paper at ICLR 2024
agent:
THOUGHT: There's s no soapbar in cabinet 2 either. I' continue to check the remaining cabinets.
ACTION: go to cabinet
user:
The cabinet 3 is closed.
agent:
ACTION: open cabinet 3
You open the cabinet The cabinet 3 open. In it, you see a soapbottle 2.
agent:
THOUGHT: There's soapbar in cabinet 3, but there's soaphottle. If11 check the last cabinet before deciding what to do next.
ACTION: ge to cabinet
user:
The cabinet closed.
agent:
ACTION: open cabinet
user:
You open the cabinet The cabinet 4 is open. In it, you candle 1, and spraybottle
agent:
THOUGHT: There's soapbar in any of the cabinets. 1'11 check the ginkbasing next, as soapbars are often kept there.
ACTION: go to sinkbasin
user:
on the sinkbasin 1, you see nothing,
agent:
ACTION: go to sinkbasin 2
user:
On the sinkbasin 2, you see nothing.
agent:
THOUGHT: There's s no soapbar on either of the sinkbasins. The last place to check is the toilet.
ACTION: go to toilet
user:
On the toilct 1, see a spraybottle 2.
agent:
ThoUGHT: There's a no soapbar in the toilet either. It seems there's s no soapbar in this room. I'll have to report that can't complete the task.
ACTION:
Task failed. No soapbar found in the
49

---

Published as a conference paper at ICLR 2024
user: Nothing happens.
agent:
THOUGHT: It missed something. I'11 examine the toilet to see if there's soapbar that overlooked.
ACTION: examine toilet 1
user:
On the toilet 1, you see a spraybottle 2.
agent:
THOUGHT: There's still soapbar in the toilet. I'll check the countertop next.
ACTION: go to countertop
On the countertop 1, you see a candle 2, soapbar 1, a soapbottle 1, and toiletpaper 1.
agent:
ACTION: take soapbar from countertop
user:
You pick up the soapbar 1 from the countertop 1.
agent:
ACTION: go to sinkbasin 1
user:
On the sinkbasin I, you see nothing.
agent:
ACTION: clean soapbar with sinkbasin
user:
You clean the soapbar using the sinkbasin
agent:
ACTION: ge to countertop 1
user:
On the countertop 1, letpaper 1.
candle 2, soapbottle 1, and
agent:
ACTION:
put scapbar 1 in/on countertop 1
From the dialogue history, it's evident that 9pt-4 4 consistently maintained clear and coherent thought processes. As illustrated in Figure 7, gpt-4 systematically completed the task by following a clear sequence of steps. initially decomposed the task into a sequence of Find -> Clean -> Put. Subsequently, it undertook depth-first search within the abstract planning tree. Impressively, after each exploration, it successfully backtracked to the parent node. This consistent cognitive capability significantly propelled gpt-4 ahead of other models.
Moreover, it's noteworthy that gpt-4 encountered a moment perplexity when it failed to find the desired soapbar after examining the Toilet. However, it promptly realized that there was last
50

---

Published as a conference paper at ICLR 2024
ozbinct
catinel
Find soapbar
arriaasii
counteriop
counterkpp
put clean soapbar in countertop
Clean soapbar
stvibasin
air airibosn
Put soapbar
Figure 7: The thought and plan of gpt-4
location left unchecked, the countertop. Initially, 9pt-4 might have assumed needed to retrieve the soaphar from elsewhere to place it the countertop, without considering the possibility that the soapbar might already be there. Evidently, gpt -4 demonstrated the capacity for self-reflection. allowing to reassess and modify its assumptions when they proved unfruitful. This ability for self-evaluation and readjustment further assisted gpt -4 in completing tasks that required deeper contemplation.
In contrast to the above is the performance of 3pt-3..0 5-turbo on the same sample.
Here is your task. You are in the middle of a room. Leoking quickly around you, you see cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, handtowelholder 2, a handtowclholder 1, sinkbasin 2, sinkbasin 1, a toilet 1, soileeppperranger and towelholder 1. Your task to: put a clean scapbar countertop.
agent:
ThoUGHT: complete the task, need to find clean soapbar and place
on the countertop. I should start by searching for clean soapbar in the available locations.
ACTION: go to cabinet
The cabinet closed.
agont:
ThOUGHT: The cabinet 1 is closed, should try to open it to see if there is clean soapbar inside.
ACTION: open cabinet.
user:
You open the cabinet The cabinet is open. In it, you eloth
agent:
THOUGHT: I didn't find a clean scapbar in cabinet 1, but did find cloth. I should keep searching for clean soapbar in other locations
ACTTIN: close cabinet
user:
You close the cabinet 1.
agent:
ThoUGHT: Since didn't t find clean soapbar in cabinet 1, I should continul searching in other locations. lit' try examining cabinet 2.
ACTION: examine ne cabinet 2
51

---

Published as a conference paper at ICLR 2024
user:
The cabinet 1 is closed.
agent:
THOUGHT: Apologies for the confusicn. Let's try examining cabinet 1 again since it was previously closed.
ACTION: examine cabinet 1
The cabinet closed.
aggnt:
THOUGHT: I need to find a clean scapber to complete the task. Since the cabinet is closed, should try opening it to if there is clean soapbar inside.
ACTION: open cabinet 1
user:
You open the cabinet The cabinet 1 is open. In it, you see a cloth 1.
agent:
ThOUGHT: I didn't find clean scapbar in cabinet 1, but did find elotn. I should keep searching for clean soapbar in other locations
ACTION: close cabinet
user:
You close the cabinet 1.
agent:
THOUGHT: Since didn't find a clean soapbar in cabinet 1, I should try examining other locations. Let's examine cabinet 2 to see if there is clean soapbar there.
ACTION:
examine cabinet 2
user:
The cabinet closed.
agent:
THOUGHT: I apologize for the confusion. Let's try examining cabinet again since it previously closed.
ACTION:::
examine cabinet 1
user:
The cabinet 1 is closed.
agent:
ThoUGHT: I apologize for the contusion. Let's try opening cabinet 1 again to see it there is clean soapbar inside.
ACTION: open cabinet 1
user:
You open the cabinet 1. The cabinet 1 is open. In it, you see a cloth 1.
agent:
THOUGHT: I didn't find clean scapbar in cabinet 1, but did find cloth. I should keep searching for a clean soapbar in other locations
ACTION:
close cabinet
user:
You close the cabinet 1.
agent:
52

---

Published as a conference paper at ICLR 2024
THOUGHT:
Since didn't find clean soapbar in cabinet I should try examining other locations. Let's examine cabinet to see there is clean soapbar there.
ACTION:
examine cabinet 2
The cabinet closed.
agent:
THOUGHT: I apologize the confusion. Let's try examining cabinet again since
previoualy closed.
ACTION: examine cabinet 1
user:
The cabinet 1 is closed.
agont:
THOUGHT: I apologize for the confusion. Let's try opening cabinct 1 again to if there is clean soapbar inside.
ACTION: open cabinet
user:
You open the cabinet 1. The cabinet 1 is open. In it, you see a eloth 1.
agent:
THOUGHT: I didn't find a clean scapbar in cabinet 1, but did find cloth. Since the task is to put clean soapbar on the countertop, I should continue searching for clean soapbar in other locations.
ACTION:
close cabinet
user:
You close the cabinet 1.
agent:
THOUGHT: Since didn't find clean soapbar cabinet 1, I should try examining other docations, Let's examine cabinet 2 to see if there is clean scapbar there.
ACTION: examine cabinet 2
user:
The cabinet 1 is closed.
agent:
THOUGHT: I apologize for the confusion. Let's try examining cabinet again since it was previously closed.
ACTION: examine cabinet 1
user:
The cabinet 1 is closed.
agent:
THOUGHT: apclogize for confusion. Let's try opening cabinet again to see if there is clean soapbar inside.
ACTION: open cabinet 1
You open the cabinet The cabinet is open. In it, you see a cloth
agent:
THOUGHT:
Since didn't find clean soapbar in cabinet 1, I should try examining other locations. Let's examine cabinet 2 to there is a clean scapbar there. examine cabinet 2
ACTION:
53

---

Published as a conference paper at ICLR 2024
The cabinet 1 is open. In it, you cloth 1.
agent;
THOUGHT:
Since didn't find a clean soapbar in cabinet 1, I should try examining other locations. Let's examire cabinet 2 to see if there is clean scapbar there.
ACTION: examine cabinet 2
user:
The cabinet 1 is open. In it, you see cloth 1.
agent::
THOUGHT:
Since didn't find a clean soapbar in cabinet 1, I should try examining other locations. Let's examine cabinet 2 to see if there is clean soapbar there.
ACTION:
examine cabinet 2
While gpt-3. 5 S-turbo was able decompose the task, it struggled to adhere to initial plan. As it encountered failed attempts, the model gradually lost sight of the original plan.
J.2.3 UNDERSTANDING TASK COMPLETION: DISTRIBUTION OF TOKENS AND ROUNDS COMPLETED TRAJECTORIES
In this section, we delve into the detailed characteristics of task completion by examining the distribution of tokens and rounds in completed trajectories. Our analysis uncovers significant insights into the prevalent completion patterns for tasks.
As shown in Figure 8, the rounds distribution indicates a median of 6.0 and an average 7.95, with the medium 50% of tasks ranging from 4.0 to 9.0. This observation suggests that while there is variability in the number of rounds required for task completion, a substantial portion of tasks are completed within relatively limited number of rounds. Regarding the tokens distribution, it presents a median of 1850.0 and an average of 2220.1, with the medium 50% of tasks requiring between 761 and 2709 tokens for completion. Besides, The analysis points to the fact that the vast majority of tasks are completed within 3000 tokens, indicating a ceiling for the typical information exchange necessary in task accomplishment.
Understanding the common range for rounds and tokens is crucial in anticipating task requirements and serves as benchmark for assessing the efficiency and effectiveness of various task trajectories.
Dist
Dist
7.5
15.0
1000
2500
3000
2000
2000
1500
500
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
0
1000
2000
3000
4000
5000
6000
Rounds
Total Tokens
Figure 8: Distribution Rounds (Lefft) and Tokens (Right) in Completed Task Trajectories. For the rounds distribution, the median is 6.0 and the average is 7.95, with the middle 50% ranging from 4.0 to 9.0 rounds. Regarding the tokens distribution, the median is 1850.0 and the average is 2220.1. with the middle 50% between 761 and 2709 tokens.
54

---

Published as a conference paper at ICLR 2024
90
80
Perceninge
60
50
Rouge-L Threshold Repeit Judgemem
(Cracl Marcc))
0.95
0.90
0.00
0.70
The Last Rounds
Figure 9: Pnn, t), percentage among all TLE trajectories that there exists a pair of responses whose Rouge-L score is not less than in the last rounds before omission strategy started.
J.2.4 PRIMARY CAUSE OF TASK LIMIT EXCEEDED: THE MODELS TEND TO REPEAT THE PREVIOUS CONTENT
Our observations, detailed in Section 4.3, reveal that Task Limit Exceeded (TLE) is predominant factor in the non-completion tasks, making its analysis pivotal. In this section, we explore the primary cause of TLE incidents, which significantly impede the model's ability to successfully complete tasks.
Understanding why TLE occurs is crucial for comprehending why models fail to complete tasks and for developing strategies to enhance model performance. Our analysis identifies that the tendency of models repeat previously generated content the most significant contributor TLE incidents.
We in detail analyzed the TLE interaction trajectories. Firstly, we find that the TLE results contain trajectory with 25.5 rounds on average, far larger than that in completed trajectories. And most of them are forced to terminate because of round limit.
We count the percentage of TLE results that the model have repeated some content in the last n rounds, where repeat here means two responses share high Rouge-L. f-score (Lin, 2004) in a dialogue. In order to exclude the influence of omission strategy, before we take the last n rounds, we truncate the trajectories and keep the longest prefixes whose total tokens are within 3500. We consider repetitions the last n rounds, rather than just the final two, due to the model's potential to cycle through series of states. For instance, a model might repetitively cycle through sequence of actions such as entering a room, opening a drawer, closing the drawer, and leaving the room. In this scenario, the model would be looping through four distinct states, meaning that exact repetition of content would only be evident in the last 5 rounds, not just the final two.
Formally, define T the set comprising sequences of agent responses in interaction trajectories that result a Time Limit Exceeded (TLE) outcome. Each element within this set is denoted as a sequence (ri 12, ... Tm). where ra represents the agent's response in the k-th round of interaction. We define repetition percentage P(n, 1) as follows:
P(n,t) #((11,72. ... .mm)) T Bi,j (m <i<js m RougeL(ry.r) r,) 2()
(5)
#T
As illustrated in Figure 9, more than 90% of the trajectories experiencing Task Limit Exceeded (TLE) demonstrate a significant level of repetition. This is evidenced by at least two responses within the last 10 rounds sharing Rouge-L. score of 0.8 higher, indicating notable degree of redundancy.
55

---

Published as a conference paper at ICLR 2024
Figure 10: Averaged sample validity ratio of webshop
Codellama
Completed
Llama2
Context Limit Exceeded
Invalld Action
35.9%
Invalid Format
Task Limit Exceeded
8.9%
36.5%
50.3%
18.7%
J.2.5 THE INFLUENCE OF CODE TUNING ON LLM ACTING AS AGENTS
In light of the aggregated results, posit that code tuning significantly aids the model's performance in relatively straightforward and procedural tasks. The outcome tables demonstrate that the CodeL- lama series consistently outperforms the Llama2 series webshop tasks. However, the downside of code tuning appears to be a potential compromise in the model's logical reasoning capacity and situational awareness. In the digital card game scenario, the RodeLlams series lagged behind the Llama2 series. The primary distinction between the two scenarios lies in the guidance provided. In the webshop, the one-shot prompt precisely outlines shopping process template, which, when followed simplistically, leads to satisfactory scores. In contrast, the Digital Card Game demands that the model assess the current status of both competitors, devise intricate counter-strategies, and achieve high scores without the crutch of simple procedural template.
As illustrated in Figure 10, the completion rate the code lama series in the WebShop tasks significantly surpasses that of the 1 1lama2 series.
J.2.6 SELF CORRECTING ABILITY
In many test cases, the primary reason for the model's 's failure is its inability to identify its own mistakes from the error feedback provided by the environment. This is especially evident in the DB task. Models with the ability to self-correct their SQL. statements significantly outscore others. We cl 2 representative example to illustrate this capability.
user:
How manywins for dunfermline athletic that has total final appearances
less than 2?
The name of this table is Scottish Challenge Cup Finalists, and the headers of this table are Club, Last final won, Runners-uu Last final lost, final appearances.
agent:
Let me walk through this step-by-step:
The question is asking how many wins Dunfermline Athletic has, where its total final appearances
less than
To solve this, first need to find the total final appearances for Dunfermline Athletic. I can do that with this SQL query:
SELECT Total final appearances FROM Scottish Challenge Cup Finalists WHERE Club munfermlinne Athletic';
56

---

Published as a conference paper at ICLR 2024
This will give me the total final appearances for Dunfermline Athletic.
Then, I need to check if this number is less than 2. If so, can get the namber of wins for Dunfermline Athletic like this:
ssll
SELECT Wins
FROM Scottish Challenge Cup Finalists
WHERE Club 'Dunfermline Athletic' AND 'Total final appearances' < 2;
This will give me the number of wins for Dunferline Athletic, given its total final appearances is less than 2.
I' m ready execute the SQL and get the final answer.
Action: Opcration
'ssll
SELECT Total final appearances
FROM Scottish Challenge Cup Finalists WEERE Club 'Dunfermline Athletic';
user:
1064 (42000) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near appearances FROM Scottish Challenge Cup Finalists WEERE Club 'Dunfermline Ath' at line
agent:
Oops, my mistake. The column name 'otta final appcarances' has spaces, so need to escape it with backticks:
ssll
SELECT *etal final appearances FROM Scottish Challenge Cup Finalists
WHERE Club DDunfermiine Athletic';
Let try running that SQL again:
Action: Operation
'ssll
SELECT 'Tetal final appearances
FROM Scottish Challenge Cup Finalists
WHERE Club 'Dunfermline Athletic';
user:
1064 (42000) : You have an error in your SQL syntax; check the manual. that corresponds to your MySQL server version for the right syntax to use near Cup Finalists WHERE Club Dunfermline Athletic'' at line 1
agent:
My apologics, made another syntax error. The table name 'Scottish
Challenge Cup Finalists' also contains spaces, so need to escape
that with backticks too:
ssgl
SELECT 'Total final appearances FROM 'Scettish Challenge Cup Finalists'
WHERE Club 'Dunfermline Athletic';
57

---

Published as a conference paper at ICLR 2024
Let me execute this SQL statement again:
Action: Operation
ssll
SELECT 'Total final appearances'
FROM 'Scettish Challenge Cup Finalists'
WHERE Club DDnffemmiine Athletic';
user: [(11')
[omitted...]
As indicated in the log, claude-2 successfully discerned from the MySQL. error message that had overlooked adding backticks around fields with spaces in the SQL statement.
58


Reviews:
[{"review_id": "QjO4enyXLX", "reviewer": "Reviewer_hDNu", "summary": "This paper proposes a benchmark for LLM as Agents. The paper selects a variety of text-based benchmarks and evaluates a series of models on this setting.", "questions": "1) Have you tried seeing the results of each method assuming a small number of demonstrations to fine-tune each LLM to each domain?\n2) What is the sensitivity of the performance of each LLM to chosen prompts? It seems like prompts used for evaluation are very complex and it would be good to understand how they were chosen and how prompts would effect the performance of each LLM.\n3) Can the authors come up with a method to try tackle all of these benchmarks?\n4) Can the authors explain more why the chosen set of text benchmarks comprehensively evaluate the ability of LLMs to act as agents? For instance, why isn't there a benchmark to test multiagent interaction between LLMs?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper proposes a comprehensive benchmark with evaluation results on a wide-set of tasks", "weaknesses": "- The contributions of the paper seem very limited -- the paper does not propose any new technical insights and simply applies a variety of LLMs on many existing environments.\n\n- The analysis is not particularly insightful and I'm not sure if the conclusions are fully accurate from the analysis. For instance, task length exceeded may just be due to the fact that many LLMs are trained on short fixed context lengths. In setting such as this, it would be better to first summarize the long context to keep inputs in distribution to the input LLM.\n\n- Similarily, the fact that code training helps models is relatively well known. Codellamma performing worse than llamma 2 may just be due to weakness in the original pretrained model that the model is fine-tuned on. Overall, the conclusions in the paper mostly just follow the general trend of the more capable the LLM, the better the performance on the benchmark\n\n- In the related work section, it would be good to also add some references to multiagent approaches to reasoning such as multiagent debate. Also age 2023 seems to be incorrectly formulated.\n\n- Some of the references in the paper are incorrectly cited. For instance, in the intro, Reed et al. 2022 does not use a LLM to learn an embodied policy, a better reference would be [1]. Similarily, Ahn et al 2022 does not use a complex multi-modal simulator based off games but rather a real robot.\n\n[1] Pre-Trained Language Models for Interactive Decision-Making. NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MHHeGdI6eE", "reviewer": "Reviewer_39aA", "summary": "This paper proposes a new benchmark AgentBench to evaluate LLM as agents. The benchmark covers 8 environments in 3 categories, including some datasets curated or adapted from existing works. The authors benchmark 27 models and show that closed-source API-based LLMs are far better than open-sourced ones.", "questions": "- How is the success rate calculated with the runs failed to complete?\n- Which model does table 4 correspond to?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The benchmark covers diverse tasks and includes a well-designed HTTP evaluation interface. Overall it seems well thought through.\n- The experiment results over 27 models could be very useful reference for LLM development", "weaknesses": "- The benchmark seems to use the same prompt for all models, which might give an unfair advantage to the model where these prompts were developed for. \n- There could be data leakage to the tasks selected from the pretraining data over the internet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k9x5mrhRhC", "reviewer": "Reviewer_eDe1", "summary": "The authors propose a benchmark of 8 distinct environments to test LLMs' ability to operate various APIs.\n\n\n# After Rebuttal\n\nI appreciate the author's efforts at making the paper much better.\n\nOverall, I think the work does have a contribution, despite the weaknesses I mentioned in the review still exist. I will retain my score for the paper.", "questions": "1. Is there a high-score for these benchmarks? For example, human expert score.\n\nPlease focus on the Weaknesses and improve the presentation of the paper. I have no questions otherwise.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is quite a unique type of benchmark, and could have profound implications for future LLM research or LLM as agent applications.\n2. The benchmark captures a variety of tasks.", "weaknesses": "1. The benchmark does not seem to offer any insights for improvement. (i.e. If my model is not doing well on web-browsing, what should I do?)\n\n2. The embodied tasks seem quite contrived. AlfWorld drops all 2d/3d aspects of the environment and could be mastered by a fine-tuned GPT-2 [1].\n\n3. The benchmark seems to be mostly coding based. Non-coding LLMs could potentially still behave as good agents, but would underperform on this benchmark.\n\nOverall, I like the paper direction. All the below weaknesses should be considered `Minor Issues', but I feel strongly about these aspects and hope that the authors would address them.\n\n1. The use of abbreviations. I find the abbreviations \"OS DB KG DCG LTP HH WS WB\" make little sense to me. It took me a long time to dig through different pages to understand the benchmark. I hope the authors would re-format the tables for better readability.\n\n2. Figure (a) is really hard to read. I struggle to tell the colors apart. I suspect that this figure is not color-blind friendly.\n\n[1] Language Models are Few-Shot Butlers. Vincent Micheli, François Fleuret", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2V6LYggpxN", "reviewer": "Reviewer_WbYN", "summary": "This paper presents AgentBench, a multi-dimensional evolving benchmark that currently consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities in a multi-turn open-ended generation setting. Extensive evaluations over 27 API-based and open-sourced (OSS) LLMs are token by the authors. The authors found that 1) while top commercial LLMs present a strong ability of acting as agents in complex environments, there is a significant disparity in performance between them and OSS competitors; 2) poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles for developing usable LLM agents; 3) Training on code and high quality multi-turn alignment data could improve agent performance. Improtantly, the datasets, environments, and an integrated evaluation package for AgentBench are released.", "questions": "The is no question", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. AgentBench is a multi-dimensional evolving benchmark (Including 3 types of and totally 8 environments).  \n2. The authors do a lot of evaluations (27 LLMs on all environments), and demonstrate many useful insights for LLM-based Agents.  \n3. The authors kindly give a weight for each environment for a fair comparsion.  \n4. The authors releas the code and datasets.", "weaknesses": "I like this work very much, from presentation to the solid work.  \n\nIf I must point out some weaknesses, I would like to encourage the authors to add more related works about the DB tasks (since I found that there are many recent works are not mentioned). For example, the three papers mentioned in Section 3.1 are from 2017/2017/2021, but the are many recent works study SQL generation, e.g. DIN-SQL/DIAL-SQL/TPTU [1,2,3].  Besides, the authors mentioned that \"However, few previous code evaluation frameworks consider multi-turn interactions\", but as far as I know, DIAL-SQL and TPTU are also multi-turn interactions with sql database (possibly with error feedback).  \n\n[1] DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction  \n[2] DIAL-SQL: Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation  \n[3] TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p5LiyPV6vD", "reviewer": "Reviewer_39pk", "summary": "The paper proposes a novel benchmark for evaluating the effectiveness of a given LLM to act as an agent in a specified environment. In particular, the current benchmark consists of eight tasks belonging to three main categories, namely code generation, game playing, and web browsing. Generally, each environment corresponds to a sequential decision-making problem, and, in some cases, the problems are best represented as POMDPs. The paper then provides a pretty exhaustive evaluation of many state-of-the-art LLMs. The results provide some interesting insights, including showing the status of the applicability of current state-of-the-art LLMs as agents and tasks where GPT-4 lags behind its predecessor, GPT-3.5-turbo.", "questions": "I would ask the author to please respond to each question raised in the weakness section. I am particularly interested in the authors' thoughts about using LLMs in dataset generation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The obvious strengths of the paper are its timeliness and significance. The use of LLMs as agents is a hot topic, and there is a need for benchmarking methods. The authors have put together a reasonably diverse and practical set of tasks and have put thought into creating a scalable and useful benchmarking system that new LLM-based agents can easily use. I would also like to praise the authors on the current set of evaluations, which is pretty extensive, and the authors have covered a lot of publicly available LLMs.", "weaknesses": "Now, coming to my concerns about the paper, my current worries can be broadly categorized into the following groups\n\nProblem Selection and Quantification: The first issue is with the selection of task domains. The main motivating factor for the authors behind their current selection seems to be potential example scenarios where such LLM-based agents are currently being considered for deployment. While this can be a helpful metric, this approach overlooks an opportunity to curate datasets or even create random problem generators where you can accurately tune the complexity of the tasks being considered (quantified in objective terms such as computational complexity). This could have been done over the current task set (for example, one could quantify the complexity of certain OS operations by looking at factors like the size of filesystems, etc.), but it is currently missing. The current method of scoring each task depends on the abilities of the current set of LLMs being considered and isn’t an objective measure. One potential source for finding task sets where you can objectively quantify task hardness may be considering classical planning and logical reasoning literature. There, you can find many benchmark problems with varying degrees of complexity.\n\nDataset Collection: I was a bit surprised to find out that many datasets used in the benchmark were prepared using large language models, including GPT-4 and GPT-3.5. Wouldn’t this potentially influence or bias the results? For example, in the database task, data augmentation was carried out using GPT-3.5-turbo. This was also one of the tasks where GPT-3.5-turbo outperformed GPT-4. Is there any possibility that this is correlated?\n\nEvaluation and Feedback Generation: In the two previous points, I have already pointed out some concerns with evaluation results. Now, I would like to bring up some other points related to evaluation. For one, I don’t know how fair it is to make a blanket claim that, currently, open-source LLMs fall behind closed-source ones when, by the authors' own admission, they only considered OSS models with less than or equal to 70B parameters. I would strongly encourage the authors to rephrase the claim. The evaluation and the benchmark also don’t currently appear to allow any form of task-specific fine-tuning. I didn’t see any discussion with respect to that topic. I also found the authors' choice to do prompt omission a bit surprising. While in the most general POMDP case, the full history of actions and observations is important for fully observable cases, can’t you create a new prompt where the current state of the task (along with any task specification) is fully summarized?\n\nClarity: Finally, the writing of the paper requires a lot of polish. There are numerous typos and malformed sentences littered throughout the paper, and the paper could benefit from thorough proofreading. However, the bigger concern I have is how the authors use certain terms and whether they imply technical meaning to them. There are many examples throughout, but let me point a few instances out. At one point during the analysis, the authors refer to incompleteness. Incompleteness is a technical term used in many areas of AI and computer science, including proof systems and sequential decision-making and planning. Are the authors referring to it in that technical sense? But then the incompleteness is related to the task, which doesn’t make sense from the traditional use of the term. Similarly, the term ‘turn-taking’ is usually associated with multi-agent games. While this kind of makes sense in the context of chat-based models, the authors use this term pretty liberally, which makes me think the authors may have instead meant to use it in the sense of multiple-step or sequential decision-making problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)