AgentBench: Evaluating LLMs as Agents
Authors: name: Xiao Liu, name: Hao Yu, name: Hanchen Zhang, name: Yifan Xu, name: Xuanyu Lei, name: Hanyu Lai, name: Yu Gu, name: Hangliang Ding, name: Kaiwen Men, name: Kejuan Yang, name: Shudan Zhang, name: Xiang Deng, name: Aohan Zeng, name: Zhengxiao Du, name: Chenhui Zhang, name: Sheng Shen, name: Tianjun Zhang, name: Yu Su, name: Huan Sun, name: Minlie Huang, name: Yuxiao Dong, name: Jie Tang

Large Language Models (LLMs) are becoming increasingly smart and autonomous,
targeting real-world pragmatic missions beyond traditional NLP tasks. As a
result, there has been an urgent need to evaluate LLMs as agents on challenging
tasks in interactive environments. We present AgentBench, a multi-dimensional
evolving benchmark that currently consists of 8 distinct environments to assess
LLM-as-Agent's reasoning and decision-making abilities in a multi-turn
open-ended generation setting. Our extensive test over 25 LLMs (including APIs
and open-sourced models) shows that, while top commercial LLMs present a strong
ability of acting as agents in complex environments, there is a significant
disparity in performance between them and open-sourced competitors. It also
serves as a component of an ongoing project with wider coverage and deeper
consideration towards systematic LLM evaluation. Datasets, environments, and an
integrated evaluation package for AgentBench are released at
https://github.com/THUDM/AgentBench