The BrowserGym Ecosystem for Web Agent Research
Authors: name: Thibault Le Sellier De Chezelles; name: Maxime Gasse; name: Alexandre Drouin; name: Massimo Caccia; name: Léo Boisvert; name: Megh Thakkar; name: Tom Marty; name: Rim Assouel; name: Sahar Omidi Shayegan; name: Lawrence Keunho Jang; name: Xing Han Lù; name: Ori Yoran; name: Dehan Kong; name: Frank F. Xu; name: Siva Reddy; name: Quentin Cappart; name: Graham Neubig; name: Ruslan Salakhutdinov; name: Nicolas Chapados; name: Alexandre Lacoste
Published: 2024-12-06 23:43:59

Abstract: The BrowserGym ecosystem addresses the growing need for efficient evaluation
and benchmarking of web agents, particularly those leveraging automation and
Large Language Models (LLMs) for web interaction tasks. Many existing
benchmarks suffer from fragmentation and inconsistent evaluation methodologies,
making it challenging to achieve reliable comparisons and reproducible results.
BrowserGym aims to solve this by providing a unified, gym-like environment with
well-defined observation and action spaces, facilitating standardized
evaluation across diverse benchmarks. Combined with AgentLab, a complementary
framework that aids in agent creation, testing, and analysis, BrowserGym offers
flexibility for integrating new benchmarks while ensuring consistent evaluation
and comprehensive experiment management. This standardized approach seeks to
reduce the time and complexity of developing web agents, supporting more
reliable comparisons and facilitating in-depth analysis of agent behaviors, and
could result in more adaptable, capable agents, ultimately accelerating
innovation in LLM-driven automation. As a supporting evidence, we conduct the
first large-scale, multi-benchmark web agent experiment and compare the
performance of 6 state-of-the-art LLMs across all benchmarks currently
available in BrowserGym. Among other findings, our results highlight a large
discrepancy between OpenAI and Anthropic's latests models, with
Claude-3.5-Sonnet leading the way on almost all benchmarks, except on
vision-related tasks where GPT-4o is superior. Despite these advancements, our
results emphasize that building robust and efficient web agents remains a
significant challenge, due to the inherent complexity of real-world web
environments and the limitations of current models.