AgentBench: Evaluating LLMs as Agents
Authors: name: Xiao Liu; name: Hao Yu; name: Hanchen Zhang; name: Yifan Xu; name: Xuanyu Lei; name: Hanyu Lai; name: Yu Gu; name: Hangliang Ding; name: Kaiwen Men; name: Kejuan Yang; name: Shudan Zhang; name: Xiang Deng; name: Aohan Zeng; name: Zhengxiao Du; name: Chenhui Zhang; name: Sheng Shen; name: Tianjun Zhang; name: Yu Su; name: Huan Sun; name: Minlie Huang; name: Yuxiao Dong; name: Jie Tang
Published: 2023-08-07 16:08:11

Abstract: Large Language Models (LLMs) are becoming increasingly smart and autonomous,
targeting real-world pragmatic missions beyond traditional NLP tasks. As a
result, there has been an urgent need to evaluate LLMs as agents on challenging
tasks in interactive environments. We present AgentBench, a multi-dimensional
evolving benchmark that currently consists of 8 distinct environments to assess
LLM-as-Agent's reasoning and decision-making abilities in a multi-turn
open-ended generation setting. Our extensive test over 25 LLMs (including APIs
and open-sourced models) shows that, while top commercial LLMs present a strong
ability of acting as agents in complex environments, there is a significant
disparity in performance between them and open-sourced competitors. It also
serves as a component of an ongoing project with wider coverage and deeper
consideration towards systematic LLM evaluation. Datasets, environments, and an
integrated evaluation package for AgentBench are released at
https://github.com/THUDM/AgentBench