TheAgentCompany: Benchmarking LLM Agents on Consequential Real World
  Tasks
Authors: name: Frank F. Xu; name: Yufan Song; name: Boxuan Li; name: Yuxuan Tang; name: Kritanjali Jain; name: Mengxue Bao; name: Zora Z. Wang; name: Xuhui Zhou; name: Zhitong Guo; name: Murong Cao; name: Mingyang Yang; name: Hao Yang Lu; name: Amaad Martin; name: Zhe Su; name: Leander Maben; name: Raj Mehta; name: Wayne Chi; name: Lawrence Jang; name: Yiqing Xie; name: Shuyan Zhou; name: Graham Neubig
Published: 2024-12-18 18:55:40

Abstract: We interact with computers on an everyday basis, be it in everyday life or
work, and many aspects of work can be done entirely with access to a computer
and the Internet. At the same time, thanks to improvements in large language
models (LLMs), there has also been a rapid development in AI agents that
interact with and affect change in their surrounding environments. But how
performant are AI agents at helping to accelerate or even autonomously perform
work-related tasks? The answer to this question has important implications for
both industry looking to adopt AI into their workflows, and for economic policy
to understand the effects that adoption of AI may have on the labor market. To
measure the progress of these LLM agents' performance on performing real-world
professional tasks, in this paper, we introduce TheAgentCompany, an extensible
benchmark for evaluating AI agents that interact with the world in similar ways
to those of a digital worker: by browsing the Web, writing code, running
programs, and communicating with other coworkers. We build a self-contained
environment with internal web sites and data that mimics a small software
company environment, and create a variety of tasks that may be performed by
workers in such a company. We test baseline agents powered by both closed
API-based and open-weights language models (LMs), and find that with the most
competitive agent, 24% of the tasks can be completed autonomously. This paints
a nuanced picture on task automation with LM agents -- in a setting simulating
a real workplace, a good portion of simpler tasks could be solved autonomously,
but more difficult long-horizon tasks are still beyond the reach of current
systems.