WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web
  Tasks
Authors: name: Atsuyuki Miyai; name: Zaiying Zhao; name: Kazuki Egashira; name: Atsuki Sato; name: Tatsumi Sunada; name: Shota Onohara; name: Hiromasa Yamanishi; name: Mashiro Toyooka; name: Kunato Nishina; name: Ryoma Maeda; name: Kiyoharu Aizawa; name: Toshihiko Yamasaki
Published: 2025-06-02 17:59:45

Abstract: Powered by a large language model (LLM), a web browsing agent operates web
browsers in a human-like manner and offers a highly transparent path toward
automating a wide range of everyday tasks. As web agents become increasingly
capable and demonstrate proficiency in general browsing tasks, a critical
question emerges: Can they go beyond general browsing to robustly handle tasks
that are tedious and complex, or chores that humans often avoid doing
themselves? In this paper, we introduce WebChoreArena, a new fully reproducible
benchmark comprising 532 carefully curated tasks designed to extend the scope
of WebArena beyond general browsing to more labor-intensive and tedious tasks.
WebChoreArena systematically integrates three key challenges: (i) Massive
Memory tasks requiring accurate retrieval of large amounts of information in
the observations, (ii) Calculation tasks demanding precise mathematical
reasoning, and (iii) Long-Term Memory tasks necessitating long-term memory
across multiple webpages. Built on top of the fully reproducible and widely
adopted four WebArena simulation environments, WebChoreArena ensures strict
reproducibility and enables fair, direct comparisons with the established
WebArena benchmark, offering key insights into agent progress. Our experimental
results demonstrate that as LLMs evolve, represented by GPT-4o, Claude 3.7
Sonnet, and Gemini 2.5 Pro, significant improvements in performance are
observed on WebChoreArena. These findings suggest that WebChoreArena is
well-suited to measure the advancement of state-of-the-art LLMs with greater
clarity. Nevertheless, the results also indicate that even with Gemini 2.5 Pro,
there remains substantial room for improvement compared to WebArena,
highlighting the increased challenges posed by WebChoreArena.