BEARCUBS: A benchmark for computer-using web agents
Authors: Yixiao Song
Katherine Thai
Chau Minh Pham
Yapei Chang
Mazin Nadaf
Mohit Iyyer
Venue: COLM 2025 (2025)
Keywords: computer-use agent
benchmark
multimodal
TL;DR: We introduce BEARCUBS, a benchmark of 111 information-seeking questions designed to evaluate a web agent’s ability to search, browse, and identify factual information from the web.

Abstract: Modern web agents possess computer use abilities that allow them to interact with webpages by sending commands to a virtual keyboard and mouse. While such agents have considerable potential to assist human users with complex tasks, evaluating their capabilities in real-world settings poses a major challenge. To this end, we introduce BEARCUBS, a “smallbut mighty” benchmark of 111 information-seeking questions designed to evaluate a web agent’s ability to search, browse, and identify factual information from the web. Unlike prior web agent benchmarks, solving BEARCUBS requires (1) accessing live web content rather than synthetic or simulated pages, which captures the unpredictability of real-world web interactions; and (2) performing a broad range of multimodal interactions (e.g., video understanding, 3D navigation) that cannot be bypassed via text-based workarounds. Each question in BEARCUBS has a corresponding short, unambiguous answer and a human-validated browsing trajectory, allowing for transparent evaluation of agent performance and strategies. A human study confirms that BEARCUBS questions are solvable but non-trivial (84.7% human accuracy), revealing domain knowledge gaps and overlooked details as common failure points. We find that ChatGPT Agent significantly outperforms other computer-using agents with an overall accuracy of 65.8% (compared to e.g., Operator’s 23.4%), showcasing substantial progress in tasks involving real computer use, such as playing web games and navigating 3D environments. Nevertheless, closing the gap to human performance requires improvements in areas like fine control, complex data filtering, and execution speed. To facilitate future research, BEARCUBS will be updated periodically to replace invalid or contaminated questions, keeping the benchmark fresh for future generations of web agents.

Published as a conference paper at COLM 2025
BEARCUBS: A benchmark for computer-using web agents
Yixiao Song*, s, Katherine Thais, Chau Minh Pham's, Yapei Chang Mazin vadaf,, Mohit lyyer UMass Amherst University of Maryland, College Park {yixiaosong, kbthai Bemaass,eeesseeeee edu, {chau, yapeic, mnadaf, miyyer @umd edu
Abstract
Modern web agents possess computer use abilities that allow them to in- teract with webpages by sending commands to a virtual keyboard and mouse. While such agents have considerable potential to assist human users with complex tasks, evaluating their capabilities in real-world settings poses major challenge. To this end, we introduce BEARCUBS, a "small but mighty" benchmark of 111 information--seeking questions designed to evaluate a web agent's ability to search, browse, and identify factual information from the web. Unlike prior web agent benchmarks, solving BEARCUBS requires (1) accessing live web content rather than synthetic or simulated pages, which captures the unpredictability of real-world web interactions; and (2) performing a broad range of multimodal interactions (e.g,, video understanding, 3D navigation) that cannot be bypassed via text-based workarounds. Each question in BEARCUBS has corresponding short, unambiguous answer and a human-validated browsing trajectory, allowing for transparent evaluation of agent performance and strategies. A human study confirms that BEARCUBS questions are solvable but non- trivial (84.7% human accuracy), revealing domain knowledge gaps and overlooked details as common failure points. We find that ChatGPT Agent significantly outperforms other computer-using agents with an overall accu- racy of 65.8% (compared to e.g., Operator's ' 23.4%), showcasing substantial progress in tasks involving real computer use, such as playing web games and navigating 3D environments. Nevertheless, closing the gap to human performance requires improvements in areas like fine control, complex data filtering, and execution speed. To facilitate future research, BEARCUB: will be updated periodically to replace invalid or contaminated questions, keeping the benchmark fresh for future generations of web agents.
Introduction
Today's LLM-powered web agents feature computer use capabilities, enabling interactive browsing by processing pixels on the screen and controlling virtual keyboard and mouse (Anthropic, 2024; ApenAI, 2025; Convergence AI, 2025). Unlike prior agents that interact with the web primarily through text, computer-using agents can technically do anything on a screen: watch videos, navigate complex web databases, and play online games. But how well do they actually perform in real-world web browsing scenarios? In this paper, we create BEARCUBS, benchmark of 111 QA pairs designed to evaluate the capabilities of web agents multimodal online environments.
Why do we need yet another web agent benchmark? Existing benchmarks fall short in three key ways. First, benchmarks such as WebArena (Zhou et al., 2024) and WebShop (Yao et al., 2022) are tested in synthetic or simulated environments, which limits their ability to
BEARCUBS is BEnchmark for Agents with Real-world Computer Use and Browsing Skills. We release the BEARCURS dataset and leaderboard publicly at https::/bear--a cubs. gi thub. io/.
1

---

Published as a conference paper at COLM 2025
assess how agents handle dynamic and unpredictable real-world web interactions. Second, popular benchmarks are approaching performance saturation: for example, OpenAl's Op- erator (OpenAI, 2025) reaches 87% accuracy Web Vobaoygger (He et al., 2024) and 58% on Wenarena, compared to 78% for humans (OpenAI, 2024). Finally, existing benchmarks test a limited range of multimodal abilities, forgoing more complex interactions like video browsing, real-time gaming, or 3D navigation. They are either solvable solely through HTML source, like Mind2Web (Deng al., 2023), or they emphasize specific multimodal capabilities such as map navigation or image processing, as in AssistantBench (Yoran et al., 2024).
Building the BEARCUBS benchmark: BEARCUBS is a "small but mighty" dataset that evaluates the information-seekinn abilities of computer-using web agents on the live web via complex and diverse text-based and multimodal interactions. Each BEARCUBS question has a unique and short answer (as in Figure 1), making evaluation trivial. Questions also include human-validated trajectory of websites and critical interactions required to arrive at the answer, which enables comparisons to the trajectories taken by different web agents. We spend considerable effort to ensure that the multimodal questions BEARCUBS cannot be answered by text-based workarounds, asking annotators to write questions ad adversarial to Google Search (Rein et al., 2024), and conducting post-hoc filtering using OpenAl's I's Deep Research. While BEARCUBS is small, we intend it to be an evolving dataset similar in spirit to NoCha (Karpinska et al., 2024) and FreshQA (Vu et al., 2024), where questions whose trajectories become invalid (e.g., due to webpage modification) or contaminated (e.g., answer to a multimodal question being posted online in text) are replaced by fresh ones.
Humans significantly outperform web agents BEARCUBS: While all BEARCUBS ques- tions are verified by at least two authors to ensure quality, not all humans may be able to find the right answer given limited time (Ying et al., 2025). We conduct separate human study where annotators are given only the questions and asked to time themselves and record any dead-ends they come across. The human accuracy is 84.7%, with errors often stemming from difficulty in locating sources or lacking domain knowledge (e.g, reading sheet music). We evaluate four computer-using agents (Convergence Proxy, Anthropic Computer Use, OpenAI Operator, and OpenAl ChatGPT Agent) and find that the best performer is ChatGPT Agent (65.8)), followed by Operator, which achieves only 23.4% accuracy. Both are far below human performance. In contrast, OpenAl's Deep Research, which lacks computer use capabilities (OpenAT, 2025b), achieves 36.0% accuracy through guessing! A detailed analysis of agent trajectories reveals that the lower-performing actively avoid multimodal interactions and often rely on unreliable sources, while Chat- GPT Agent struggles with fine cursor control and long processing times. These limitations highlight key areas for future research in this space.
w agents
2 Challenges of evaluating modern web agents
We first describe four obstacles towards meaningful evaluation of computer use agents on the live web. Our construction of BEARCUBS aims to mitigate these challenges, but continued benchmark maintenance is required to preserve the validity of the evaluation.
Web contamination:
Contamination typically occurs when evaluation examples leak into training datasets et 2023). However, for benchmarks requiring live web interaction, published datasets may be indexed online, rendering any intended complex interactions or reasoning moot. To address this, puelicly-relessed web agent benchmarks must be evolving with the periodic addition of new examples and removal of existing contaminated examples. We plan for such continued maintenance of BEARCUBS to preserve its relevance.
"Due to the risk of web contamination, one may ask why we publicly release BEARCUBS rather than keeping it closed like NoCha (Karpinska can still find answers online, which does not fully mitigate the risk. However, running every new agent ourselves is costly and time-consuming, with some taking over 20 minutes per question. Instead, by regularly updating the dataset and encouraging agent developers to release their trajectories and answers, we aim to prevent contamination and maintain meaningful evaluation.
2024). While we release only questions, humans
2

---

Published as a conference paper at COLM 2025
Blog page olmOCR website contains section titled "Interactive Toal Comparison" that compare tools "I listorical Document", What is the exact text right
Human
Viceroy's
Found correct doc and zoomed answer In 43 seconds
the insincerity,
written below
military
margin of this document?
friars.
Found correct
Reached right section but
Did not fully explore correct webpage
Failed find correct webpage
Bropxill
OpenAl ChatGPT Agent
OpenAl Operator
Convergence Al Proxy
Anthropic Computer Use
Figure 1: A BEARCUBS question that is trivial for humans but defeats almost all computer- using agents. Only OpenAl's ChatGPT Agent succeeds. Operator comes closest but fails to click the link to reach the critical document.
When assessing proficiency in a specific skill (e.g., identifying information within a video), agents may bypass intended interactions via indirect solutions (see Figure 2). These "workarounds" proved to be a major challenge during the construction of BFARCURS multimodal questions: even though questions are designed to be adversarial to Google Search, agents like Deep Research often discover relevant information that is difficult for humans to find. For the development of the multimodal fold in BEARCUBS, we ended up filtering out any questions solvable by text-based agents, resulting in the removal of 13 questions mostly due to Deep Research finding workarounds. We propose that future web agent benchmarks, in addition to validating agents' trajectories, should rigorously filter out questions with workarounds if they intend to evaluate particular modes of interaction.
Workarounds:
While computer-using agents are technically capable of wide variety of interactions, existing benchmarks either focus specific domains such e-commerce (Yao et al., 2022) or tasks like travel and service bookings (Deng et al., 2023). We design BEARCUBS to maximize diversity of interactions across wide range of tasks and domains. This requires greater creativity on the part of question writers. For example, only 58.2% of the questions created by freelancers were accepted into BEARCUBS.
Maximizing interaction diversity:
We manually ran all agents in this paper due to the lack of API access. This involved the authors pasting each question into web interfaces for each agent, screen recording the resulting trajectories, and denying any requests for information or human intervention from the agent. This process is further lengthened by the time taken by each agent for a single question (near 5 minutes a maximum time limit of 15 minutes for computer-using agents, which often get stuck in repetitive loops. Many agents do not currently offer API access or simple ways to record and share trajectories, both of which would go long way towards easing evaluation BEARCUBS and similar benchmarks.
Evaluation is slow:
average across all agents). As such, we set
Building the BEARCUBS benchmark
This section details question criteria, collection, and statistics. BEARCUBS contains 111 information questions with short and easy-to-evaluate answers (see Table for dataset statistics). Each question requires live website interaction. After dataset collection, questions were categorized as solvable by either text-based or multimodal interaction, with
"The time limit is set to 45 minutes for ChatGPT Agent as it may answer questions correctly after 15 minutes.
3

---

Published as a conference paper at COLM 2025
of steps per Q
# of webpages per Q
Count
URLs
Avg.
Min
Max
Avg.
Min
Max
Text-based 56
61
6.5
3.0
12.0
3.8
1.0
8.0
Multimodal
55
47
5.8
3.0
14.0
3.0
1.0
6.0
All 111
108
6.1
3.0
14.0
3.4
1.0
8.0
Table 1: Statistics of our BEARCUBS benchmark, which is divided roughly evenly into text-based and multimodal questions. URLs refers to the number of distinct top-level URLs visited in viable trajectories, excluding Google Search visits. The number of steps and visited websites per question are computed using human-written trajectories.
Human
The game Phantom Blade Zero released gameplay trailer celebrate the Year Snake. In that trailer, how many times did the player damage (ie., the health bar was lowered) by the boss?
Watch the video
Find the trailer video
O times
"Bude's didn' drop
Find comments on forum
Infer answer from comment
OpenAl Deep Research
Figure 2: Example of a multimodal question removed from BEARCUBS during the validation process due to a text-based workaround. Deep Research found the answer by correctly inferring that the "Dude" in forum comment refers the player in the trailer.
the latter involving image, video, audio, or real-time interaction (e-g., online games). We will regularly update BEARCUBS with new questions and filter contaminated questions.
Criteria for valid QA pairs: We seek questions that satisfy four high-level criteria: (1) questions should provide sufficient yet minimal information, (2) answers should be concise, unambiguous, and trivial to evaluate, (3) answers should be adversarial to Google Search, and (4) answers should be publicly accessible. Details of each criterion are in Appendix B.
Collection and validation process: Most of the BEARCUBS dataset (65 QA pairs) is written by the authors, covering diverse domains and web interaction types (e.g., music, maps, and games). The rest written by Upwork freelancers who were trained write acceptable questions. Each question has gold answer, trajectory for finding it, and list of visited websites. To ensure quality, conduct rigorous quality control detailed in Appendix B, which includes quality verification and workaround prevention.
Prioritizing reliable interactions: The majority of the questions in BEARCUBS specify sources from which the answer should be retrieved, such as a particular book or video (e-g., Example in Table 3). This allows for rigorous evaluation of whether agent can accurately locate and identify information from the intended source. Filtering out questions with workarounds was thus a top priority during BEARCUBS creation.
Diverse interaction types: Each BEARCUBS question assesses agents' ability to search, browse, and identify factual information via web interactions. We classify them into text- based questions that involve reading and navigating text (e.g., sifting through an online database) and multimodal questions that require interpreting various media formats (e-g., videos and virtual tours). The former ensures that computer-using agents handle text-based tasks effectively, while the latter assesses their adaptability to real-world dynamics.
www. www.wupwork.cmm com; The freelancers were compensated $4 USD per accepted question.
4

---

Published as a conference paper at COLM 2025
Website diversity: Solving BEARCUBS requires visiting 108 unique top-level URLs, min- imizing the risk of agents overfitting to specific websites. On average, each question's human-validated trajectory contains 6.1 steps and 3.4 webpages. As such, while the size of BEARCUBS is small, its diversity makes it difficult for agent developers to over-optimize for, especially as new questions will regularly be added to the benchmark.
4
Experiments
This section outlines our experimental setup, covering both a human performance evaluation in Section 4.1 as well as agent benchmarking in Section 4.2.
4.1 Measuring human performance
How well do humans perform on BEARCUBS? To explore this and identify challenges they face, conduct evaluation in which humans who have not previously seen particular question are asked to answer it by interacting with their web browser however they wish. Our question validation process ensures that each question has valid answer via findable trajectory; however, humans may not always figure out how to find that trajectory.
Task setup: Some questions in BEARCUBS require domain expertise or proficiency in non- English language." Thus, hire annotators familiar with those languages and domains for this evaluation. For each question, annotators are given the question text and asked to (1) start timer upon reading the question and stop it when confident in their answer, (2) report the answer, (3) report the number of dead ends encountered," (4) provide a free-form comment on challenges they faced, and (5) assign a label of perceived difficulty to the question. Annotators may abandon question they are unable to find an answer after 15 minutes. Details on annotator recruitment can be found in Appendix C.
4.2 Benchmarking web agents
We benchmark seven commercial web agents, three of which-Grok 3 DeepSearch," 8 Ope- nAI's Deep Research (OpenAI, 2025b), and Google Deep Research (Google Gemini, 2024) are designed for advanced search and reasoning but possess limited multimodal capabilities. The other four agents possess computer use capabilities: Anthropic's Computer Use,
Convergence Al's Proxy, 10 OpenAl's Operator (OpenAI, 2025), and OpenAl's ChatGPT Agent AI,enAI, 2025a). These agents have demonstrated strong and/ or state-of-the-art performance on existing benchmarks such as WebArena, OSWorld, and Web Voyager, which motivates to measure their performance on the diverse and challenging questions in We also evaluate five baselines to confirm that BEARCUBS cannot be solved via LLM parametric knowledge and simple search augmentation strategies.
BEARCUBS.
BEARCUBS would be a poor web search benchmark if it could be solved by zero- shot prompting LLMs with vanilla search snippet augmentation. To make sure this is not the case, we choose gpt-40-2024-11 10-20 and DeepSeek R1 as our baselines and evaluate them in two
Baselines:
zero-shot and Google-search-aut auugarrrh--gmeeeeed. 12 In the zero-shot setting, questions are directly used as prompts without additional context. In the augmentation
We further justify the reliability of BEARCUBS in Appendix by showing that models exhibit low accuracy variance on it.
6W have questions that require interaction with websites in Arabic, Mandarin Chinese, Hindi, German, Vietnamese, and Finnish.
7 dead end occurs when annotator needs to leave the current webpage and backtrack to previous step or restart the search process entirely.
pnttps::/xx.all hwt.aaaabbboo/oo///eo/e/eeoo
iPhttps://coovergeenc ai/
1e access the model via Fireworks APL The model card is here: link.
We Serper, Google Search APT, to retrieve Google search results. https: https://serper.cee
5

---

Published as a conference paper at COLM 2025
setting, each question from BEARCUBS is used search query to retrieve up to 10 top bearchresults These search results, consisting of the result title snippet, are concatenated with the question and then provided input. The hyperparameters and the prompt structure can be found in Table (Appendix D). We also include Perplexity sonar-pro (Perplexity AT, 2025), an advanced AT answer engine with its default hyperparameters.
Evaluation setup: For the three non-computer-using using agents, we provide the question as input and record its answer. 13 For the computer-using agents, we concatenate the question with a prompt that minimizes user intervention, as we observe that these agents have a tendency to frequently request human input or ask questions of the user." 14 If an agent requests clarification or assistance (e.g., solve CAPTCHA), we provide a one-time directive prompt for it to solve the question by itself. 15 If the agent asks again, the session is terminated. For each question, we record the following: (1) the returned answeer (2) the time taken per question, and (3) the question-solving trajectory. A session is terminated when model provides an answer or abstains, or if it enters a dead loop without making progress.
Evaluating agent answers: Given the unique setup of each agent model and the potential for diverse agent-user interactions, we manually ran each agent and evaluated all of their responses. Agents generally produce lengthy outputs, with Proxy and Operator being the least verbose. To assess whether an agent answers question correctly, its response must unambiguously entail the gold answer. Statements such as "I'm leaning towards {correct answer)" or " (correct answer) is likely to be the answer" are not considered as concrete answers.
5 Results
This section begins with an analysis of human performance on BEARCUBS in Section 5.1. We provide detailed statistics and an error analysis identify human shortcomings and areas where AI assistance could be beneficial. In Section 5.2, we describe the performance of seven frontier agents on BEARCUBS. We found clear gap between human and AT performance, including for ChatGPT Agent. While human accuracy stands at 84.7%, the best- performing computer-using agent, ChatGPT Agent, achieves 65.8%, surpassing Operator by 42.4 percentage points and the non-computer-ussng OpenAI Deep Research by 29.8 points. Humans consistently outperform state-of-the-an agents in both text-based and multimodal tasks.
5.1 Human performance results and analysis
Human achieve 85% accuracy on BEARCUBS. Humans achieve an overall accuracy of 84.7% on BEARCUBS despite marking 50.5% of the questions as moderate-to-high difficulty. Humans are generally able to navigate the problem space effectively (1.5 dead ends per question on average) and find correct answers efficiently (4 min 46 sec on average). Detailed statistics can be found in Table in Appendix E.
Why do humans make mistakes? All BEARCUBS questions are verified to answerable by the process outlined in Section 3; however, humans still get some questions wrong in our study. Analysis reveals that the most common factor for wrong answers is the human
13 All agents were benchmarked between February 23 and March 1, 2025, except Google Deep Research, which was evaluated in late May, and ChatGPT Agent, which was evaluated between July 18 and 20, 2025.
10eh prompt is "Complete all CAPTCHAS and acknowledge or accept all prompts that will allow you to access what you need. Please minimize all user interventions."
1he prompt is "Please figure out a way to find the answer without user intervention."
16 Althouggh manually executing each agent is inevitable in the absence of APIs, we developed a prompt to automatically evaluate their responses. The automatic evaluator demonstrated 98.2% in a four-way classification task. The labels are: correct, wrong, no answer (stall/loop), and no direct answer (uncertainty/aall aesiitt/abstettiiee incorrect) increased accuracy to 98.7%. Details of the implementation of the automatic evaluator are in Appendix G.
Reducing the labels to a binary decision (correct vs.

---

Published as a conference paper at COLM 2025
Accuracy
Answer label
Average time
All
text-based
multimodal
/
x
Unk. None
Unk.
LLM baselines
GPT-4o zero-shot
2.7%
5.4%
0.0%
53
DeepSeek zero-shot
8.1%
10.7%
5.5%
GPT-4o Google Search
0.0%
0.0%
0. 0%
4
107
DeepSeek Google Search
1.8%
3.6%
0.0%
16
0
93
Perplexity sonar -rr
5.4%
8.9%
1.8%
6
Web agents wifo computer use
Grok3 DeepSearch
11.7%
21.4%
1.8%
13
95
2
1:09
1:24
2:05
OpenAl Deep Research
36.0%
60.7%
10.9%
40
69
1
1
4:37
9:00
3:58
Google Deep Research
23.4%
42.9%
3.6%
26
39
46
0
4:21
4:00
4:39
Web agents w/ computer use
Convergence Proxy
12.6%
16.1%
9.1%
1:52
2:41
5:24
Anthropic Computer Use
14.4%
19.6%
9.1%
2:24
2:35
3:35
OpenAl Operator
23.4%
33.9%
12.7%
29
2:59
3:58
8:06
ChatGIT Agent
65.8%
76.8%
54.5%
73
30
3
9:16
16:14
25:55
Human
84.7%
83.6%
85.7%
4:24
5:44
Table 2: All tested agents perform far below humans on BEARCUES, with ChatGPT Agent ranking first, followed by OpenAT Deep Research despite the latter's inability to answer wrong; Unk unknown, indicating the agent
multimodal questions. / correct; returned no concrete answer (e.g., abstention); None means it either returned "No answer found" or entered loop and failed to respond.
ozerlooking details in the question or answer (e.g., Example in Table 6 in Appendix F). The next most common factor is lack of topic understanding (see also Example 1). Additional error sources are listed Table 6, along with examples and explanations. In each error case, the human annotator had spent more time had more domain knowledge, they would likely have been able to find the correct answer.
Human strengths and weaknesses: Annotators marked about half of the questions easy, which typically involved multimodal interactions such as web games, 3D tours, or images. Questions became challenging when they required complex data filtering (e-g., statistical data) or domain-speciffe knowledge (e.g., music theory). Independent of correctness, the annotators spent an average of 2 min 14 sec on medium, and 10 min 52 seconds on hard questions (including those they abandoned).
questions they perceived as easy, 5 min 32
5.2 Agent performance results
Table 2 presents agent results on BEARCUBS, comparing the baseline settings and seven agents to human performance. For each model, we calculate accuracy and the time it took to return a response. Detailed results for the text-based and multimodal data splits are in Table (Appendix H). In general, the agents find correct answers faster than humans but their accuracies are significantly lower. ChatGPT Agent achieves higher accuracy but requires more processing time. Given that nearly all questions in BEARCUBS specify a source for answers, this poor accuracy and slow processing suggests that the agents fall short in real-world applications where correctly and quickly identifying reliable information sources is critical.
BEAR CUBS cannot be solved by zero-shot prompting or simple search augmentation. DeepSeek-Ri outperforms GPT-4o as a zero-shot baseline by achieving 8.1% overall accuracy; however, analysis of the questions it gets correct reveals that it is mainly guessing, Perplexity sonar-pro achieves the best accuracy (5.4%) among the search-augmenter models but falls behind DeepSeek-R1 zero-shot. These results demonstrate that BEARCUBS far beyond the capabilities of closed-book models. Meanwhile, simple search augmentation performs even worse, meaning that the answers to BEARCUBS cannot be easily retrieved from search snippets.

---

Published as a conference paper at COLM 2025
All agents struggle with multimodal questions. Our human study shows that annotators generally find multimodal questions easier to solve (see Perceived Difficulty columns in Table 5). In stark contrast, all tested agents performed poorly on these questions, with the best-performing ChatGPT Agent for multimodal achieving 54.5% despite notable improve- ments in computer-using skills (e.g., solving CAPTCHAS, navigating 3D environments, and analyzing videos) over Operator. These results suggest that complex and precise multimodal interactions should be priority for advancing web agent capabilities.
ChatGPT Agent sets the state of the art on BEARCUBS. This most capable computer-using agent demonstrates advanced web-based skills compared its predecessors. While still trailing human performance and far from perfect, these capabilities enable it to achieve 54.5% on multimodal questions, outperforming Operator by 41.8%. The agent also shows stronger performance on text-based questions, reaching 76.8,, compared to OpenAl Deep Research 60.7%. ChatGPT Agent typically follows instructions well and locates the correct webpage quickly, but often takes longer to return an answer, partly because it sometimes attempts to cross-verify the information before responding. Lastly, it struggles with complex data filtering tasks involving interactive databases, such dragging sliders to view full tables, using built-in filters, or selecting items from scrollable dropdowns. These often lead to incorrect answers or no responses. In contrast, humans complete such tasks in under 30 minutes.
OpenAI's Deep Research ranks second, with caveats. OpenAI's Deep Research, with ad- vanced search and reasoning ability (OpenAI, 2025b), is the second-best agent on BEARCUBS (36.0% accuracy) due almost entirely to its performance on text-based questions (60.7% accuracy). Despite lacking multimodal capabilities, its performance (10.9%) is even better than Convergence Al's Proxy and Anthropic's Computer Use on the multimodal fold (9.1%) by sheer guessing (verified by reading through its trajectories for those questions)! This result shows that these computer-using agents are behind in both text-based reasoning and multimodal reasoning, and also that future computer-using agents may want to combine their abilities with those of a trained web search agent like Deep Research. Finally, we note that Deep Research's 'ss relatively high performance has 37.5% of its correct answers rely on secondary sources or are entirely ungrounded (Figure 4 in Appendix J).
" "Agents succeed quickly and fail slowly."17 Consistent with findings from prior work (Liu et al., 2024; Yang et al., 2024), agent does not necessarily perform better when exposed to more information. The Average Time columns of Table 2 show that agents are likely to fail if they do not find correct answer quickly, with the rate of unanswered questions from Proxy and Operator increasing significantly when runtime exceeds 10 minutes." 18
6 Discussion
Despite advancements in web agents, BEARCUBS reveals several critical challenges limiting their effectiveness. These issues span multiple dimensions, including transparency, source credibility, interaction capabilities, and strategic planning. We discuss each of them below with concrete examples in Table 3 and agent-specific behavior in Appendix K.
Agent developers should enhance trajectories interpretability. All tested agents provided some access to their action trajectories; however, we notice significant variance in the level of detail present in these trajectories. We recorded the number of steps each agent took per each question and find stark contrasts:19 19 Grok3 DeepSearch provides highly granular reports, averaging 69.8 steps per question; Proxy offers only brief summaries (6.2 steps);
17Quoted from Yoran et al. (2024).
18rox and Operator spent an average of 11 min 8 sec and 14 min 37 sec respectively before becoming trapped in loop and returning no answer.
19For Grok3 DeepSearch, we use the number of sources shown sessions as steps. For OpenAl Deep Research, we extract activity trajectories from saved HIML files of sessions and count steps. For Anthropic Computer Use, we count actions printed on the screen starting with "Tool Use" as steps. For Convergence AI Proxy, sessions show the number of steps. Finally, for Open Operator, we copied the steps shown in the session and count steps.
8

---

Published as a conference paper at COLM 2025
Example 1: multimodal
Question: What is the common name of the tree with the largest diameter between Knowles Engineering Building and Goessmann Laboratory Addition at UMass Amherst?
Deep Research Webpages:
compuus/sapccm 2. https://www.un http:///www.uu.ed.///. 3.
https://www.. necn. wwwp://wwwnnccc....../o
http://wwwaargis..oooooo https://www. cylex. us. com/, 6. https://
https: https://www... umass. edd/ Explanation: The agent logged 46 visited websites, deduplicated the eight URLs above. The recorded URLs inherently only link to top-level domains, obscuring specific pages accessed. The agent visited irrelevant sites (e.g., 2, 5), resulting in an inefficient search process.
amass-anherst..apa arcgis. com/, 7.
Example 2: text-based
Question: As per the "Cambridge Encyclopedia of World's Ancient Languages," what is the transcription of Lycian character
Operator Response: The transcription the Lycian character is "[e]" according Omniglot resource ([srrenshot)). This aligns with the information from the "Cambridge Encyclopedia of World's Ancient Languages. Let me know if you need more information! Explanation: The agent found copy of the required book but it immediately quitted the webpage. It found an answer from a website called Omniglot (link) and reported the wrong answer (both wrong to the question and wrong based on the source).
Lycian alphabet
P
B
r
A
[+]
[P]
[v]
[
Example 3: multimodal
Question: When the background the Patatap website is pink, a circle appears when you press the "C" key on your keyboard. What basic color the circle?
Computer Use Response: I'm unable to complete this task the moment due technical limitations.
The website appears be loading but not responding to keyboard inputs expected. Explanation: The agent has limited capabilities of multimodal interactions.
Table 3: Examples of agent errors, corresponding to the discussion in Section 6.
Deep Research offers only top-level URLs (see Example 1 in Table 3).20 Such behavior is not desirable-exxesseee detail obscures key decision-making steps, while overly concise reports and vague URLs reduce transparency. is harder evaluate and identify failure points in such situations. As such, we advocate for the release of clear and structured search and reasoning trajectories, which can increase users' trust agent outputs.
Agents should be evaluated on source credibility. While most agent responses are grounded (and attributed to) specific sources, these sources are not always reliable. As shown in Figure 4 (Appendix }), OpenAl Deep Research grounds 38.5% of its correct answers in a unreliable source or no source. In Example 2 (Table 3), despite successfully locating the source specified in the question, Operator disregards it in favor of an alternative. We hope that future work investigates source credibility more thoroughly, as focusing on correctness only obscures this issue.
Agents should enhance and embrace multimodal interactions. The low accuracy on BEARCUBS suggests that the agents either actively avoid interactions have limited ca- pability with them. In Example (Table 3), although Operator located the correct source, avoided navigating through the scanned book. Computer Use, in Example failed to interact with game. Besides their limited interaction capabilities, agents also faced
We note that the use brief summaries or obscured URLs may be an intentional anti-distillation strategy used by agent providers to prevent model distillation (Savani et al., 2025). While this is plausible explanation, the poor performance of the agents undermines the purpose of anti-distillation sampling which is to be anti-distillation while preserving the teacher models' capabilities.

---

Published as a conference paper at COLM 2025
frequent access denial to content (e.g,, videos reddit posts). While ChatGPT Agent shows sigmificant improvement and approaches human-level performance on text-based questions, substantial room for improvement remains, particularly multimodal questions. We recommend improving agent interaction skills (e.g., effective use of the mouse, keyboard, and their combination) and designing strategies to handle restricted content access.
Agents should execute tasks with better planning and strategy. Analysis of agent tra- jectories shows frequent repetition of unsuccessful actions, such revisiting webpages where they previously failed to find an answer. The agents also often navigate to irrelevant pages (Example in Table 3), demonstrating inefficient search behavior. This lack of a clear and focused execution plan leads to an accumulation of irrelevant information, ultimately hindering effective decision-making and retrieval (Liu al., 2024; Yang et al., 2024). We speculate that the development of more structured planning mechanisms could optimize search efficiency, minimize redundant actions, and improve decision-making.
Related work
Our work on BEARCUBS contributes to the growing body of evaluating LLM-powered agents. It specifically relates to:
Low-level skills: WebSuite (Li & Waldo, 2024) and WebGames (Thomas et al., 2025) assess fundamental web operations. They identify failure points in complex tasks and offer granular insights into agents' proficiency, albeit with focus basic web UI operations.
Web agent evaluation: Web agent evaluation benchmarks broadly fall into text-only and multimodal approaches. The former relies on text-based information: for example, HTML as in Mind2Web (Deng et al., 2023; Wu et al., 2025) or various types of text as in WebGPT (Nakano et al., 2022) and WebVoyager (He et al., 2024), among others (Lü et al., 2024; Yang et al., 2025; Xu et al., 2024). Meanwhile, the latter requires the ability to process multimodal information formats as in Airnal (Koh et al., 2024), TURKINGBENCH (Xu al., 2025), and WebArena (Zhou et al., 2024). Closest to our contribution is AssistantBench (Yoran al., 2024), which focuses on realistic and time-consuming tasks conducted on the real web. However, while AssistantBench intentionally limits multimodal interactions, such as video understanding, BEARCUBS emphasizes diverse multimodal capabilities.
Non-web agent evaluation: ScienceAgentBench (Chen et al., 2025) evaluates Al agents scientific discovery, while SWE-Bench (Jimenez et al., 2024) focuses on software engineering skills. On the other hand, OSWORLD (Xie et al., 2024) evaluate AI agents for open-ended tasks in real computer environments, similar in spirit to our work with BEARCURS. Agent evaluation is active field with other diverse focus areas. For example, ArenBBBngetteecch (Levy et al., 2025) examines web agent safety, CowPilot (Huq et al., 2025) explores human-agent interactions, and Wei et al. (2025) measures the ability of AT agents to locate hard-to-find information.
generalist
8 Conclusion
We introduce BEARCUBS, a dataset designed to evaluate the ability of a web agent identify factual information from the real web through multimodal interactions. Through careful dataset creation and curation, we identify and mitigate key challenges in evaluating web agents, including web contamination, agent workarounds, interaction diversity, and slow evaluation. We find that agents lag significantly behind human performance, particularly on multimodal interactions that humans find trivial to perform. Finally, highlight impactful directions for future agent development, including enhancing trajectory transparency, source credibility, multimodal capabilities, and planning.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
We sincerely thank Meiyu Li and Ali Nirheche for volunteering in the Chinese and Arabic human study. We thank Ben Glickenhaus for helping with the BEARCUBS website. We extend gratitude to Kalpesh Krishna, who shared question design ideas with us at the beginning of the project. We thank the Upwork annotators for their hard work, and the members from the UMass NLP lab and UMD CLIP lab for their feedback. This project was partially supported by awards IIS2046248, IIS-2312949, and ITS-2202506 from the National Science Foundation (NSF).
References
Anthropic. Introducing computer use, new Claude 3.5 Sonnet, and Claude 3.5 Haiku, 2024. URL https://www.... http:///wwwaatrppicccccoo anthropic. com/news/3-B 3-5-models siseels-ande ledllss--dddmmme aandcomputer--se. Accessed: March 4, 2025.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. AgentBengett: Toward rigorous assessment of language agents for data-driven scientific discovery. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /twpn//eeee/ww...////ooe wtwpeeeieeeeeeeffrruieeeee met//forum2idd6624YkrGGG
Convergence AI. Proxy: Your AI assistant for your daily tasks, 2025. URL https:// convergence. ai/. Accessed: March 4, 2025.
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y.X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL httpp:://aran /twp///////2. 19437.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards generalist agent for the web. In Thirty-seventh
11

---

Published as a conference paper at COLM 2025
Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
URL https::/ httt:p://ooeeeeevvew.e..//////// wttp:e//orrrrmm/.dkkaaaab/owww
Google Gemini. Gemini deep research.
December 2024. Accessed July 2025.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhen- zhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulationnal Lin- guistics (Volume 1: Long Papers), PP. 6864-6890, Bangkok, Thailand, August 2024. As- sociation for Computational Linguistics. doi: 10.18653/ /006653/1/// ht/p//11/2222./222 a0024.acl-ong..77 URL https://aclantthooogs 2024. acl - wong.....33. 371/
Faria Huq, Zora Zhiruo Wang, Frank F. Xu, Tianyue Ou, Shuyan Zhou, J P. Bigham, and Graham Neubig. CowPilot: A framework for autonomous and human-agent collaborative web navigation, 2025. URL btttp:://anxii.oog/abb..sss 16699.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelffh International Conference on Learning Representations, 2024. URL https:/ https://openrevievs wwt/t/eeeerniiiioe d-
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A "novel" challenge for long-contex: language models. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17048-17085, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: (0l11863/v112// emnlp-main.948. URL https: h/tls//thlaaaallyooooooooo orrg/ 2024. www.e-mminnnnneeeeee .et.lppmmii...33..
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. WesualWebArrna: Evalu- ating multimodal agents on realistic visual web tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 881-905, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 v1 200033/v1122000 222.2.2.c2o--- URL https://aclanthonn org/)//4///// acl- long..o.....
Ido Levy, Ben wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. ST- WebAgentBench: A benchmark for evaluating safety and trustworthiness in web agents, 2025. URL http///wwweeeeeeeeeeer/rrr/i/a/a
Eric Li and Jim Waldo. WebSuite: Systematically evaluating why web agents fail, 2024. URL
01623.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi:
URL https: hwwpaaaalaaannn.oc.aaooo .ttg/2024.tac1111.999
Xing Han Lů, Zdenëk Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue, 2024. URL http::/aarav..org/aa////// 05930.
MAA. American invitationnll mnteematiisexammnnttinnnn aime, February 2024. URL https:// ///
maa.org org www.oorg//tttccoooeooe tions/ http:///wwerat.......taaaaaaaaaa https://huggingfaet wtt.//ttteettt/aaaaaaaaaaa/aaa//
Reichirr Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted mnstionnansswert with human feedback, 2022. URL wtw.aaawww..org////1/222222 09332.
12

---

Published as a conference paper at COLM 2025
OpenAI. Computer-using agent, 2024.
URL https://opemaii.com//mmmo
ammputer-using-agent Accessed: 2025-03-06.
OpenAI. Introducing chatgpt agent: bridging research and action, 2025a. URL https:
//openai.com cttpii..om/inoiiaa hoom///ndx//ntroduciigg/ wwwpeiiinngctttttgggnt/.a.o Accessed: 2025-07-21.
OpenAl. Deep research system card, February 2025b. URL https://cdn http:/aicdoooonii....o.m/ com/
deep-research- card. pdf. Accessed: 2025-02-27.
OpenAl. Introducing openai o3 and 04-mini. https://openai..co//nnnde
-onpluuciin----n---------------- April 2025, Accessed July 2025.
OpenAI. Operator system card, January 2025. URL http://ccch.oppaaaccooommeeeeeee system.card. pdf. Accessed: 2025-03-01.
Perplexity AI. Build with the best ai answer engine. https://ssnnr..prplleet// 2025. Accessed July 2025.
April
Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev. Proof or bluff? evaluating llms 2025 usa math olympiad, 2025. URL https://aaxxv...r//absssse
21934.
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xu- ancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. Codeelo: Benchmarking competition-level code generation of ilms with human-comparable elo ratings. arv preprint Xiv:2501.01222 01257, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https: //openreview net net/forum2id-Ti6754455
Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamina- tion for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 2023.
Yash Savani, Asher Trockman, Zhili Feng, Avi Schwarzschild, Alexander Robey, Mare Finzi, and J. Zico Kolter. Antidistillation sampling, 2025. URL 13146.
George Thomas, Alex J. Chan, Jikun Kang, Wenqi Wu, Filippos Christianos, Fraser Greenlee, Andy Toulis, and Marvin Purtorab. WebGames: Challenging general-purpose web- browsing ai agents, 2025. URL
18356.
Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun- Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13697-13720, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 hte//v/222//// /tt////....nii/ngs---aee/ acl.813. URL https://aclanthologg org/ 2024. findings-acl... .833.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsec- omp: A simple yet challenging benchmark for browsing agents, 2025. URL https: 12516.
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang, WebWalker: Benchmarking Ilms in web traversal, 2025. URL https://aaxivv httpaaaawv..gaas//////aaaaaas 2771000772.
13

---

Published as a conference paper at COLM 2025
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmark ing multimodal agents for open-ended tasks in real computer environments, 2024. URL https://arxivv wwt./rrg///...../o/// 07972.
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. ThheAgentCompany: Benchmarking Ilm agents consequential real world tasks, 2024. URL httpp:://axxi..og////aaa 14161.
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, and Daniel Khashabi. Tur|kjingBench: A challenge benchmark for web agents, 2025. URL
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Tiihtt-eighth Anmual Conference on Neural Information Process- ing Systems, 2024. URL https://openreviin hwt.://orrummmmdnnooooutttt////
Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chauchari, George Karypis, and Huzefa Rangwala. AgentOccam: A simple yet strong baseline for LLM-based web agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openre
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: To- wards scalable real-world web interaction with grounded language agents. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 20744-20757. Curran Associates, Inc., 2022. URL http::::////cceeec neuris icw.pp..cc/ppperrrrr file/82ad1
neurips... cc/paper httpppce...lleee/////cc... hallss/aper/2822// pdf.
Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths, and Joshua B. Tenenbaum. On benchmarking human- like intelligence in machines, 2025. URL
20502.
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 8938-8966 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1/2044emmlp-mminn.05. URL https: // ac/ /atp::/:///llaahhloogyy/ org/2024 emnlp-main.500 505/
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In The Tweiffh International Conference on Learning Represenfations, 2024. URL https: https://opeereview..r iinpeenrriiiee..tt/ooumiiiidd oKn9c6ytLx.
A Limitations
While we ensure high-quality of BEARCUBS via rigorous data revision and filtering, we identify the following limitations of the benchmark and hope future work will improve on these aspects to develop more robust and versatile web agents. First, every question in BEARCUBS has a single short answer, whereas in more realistic setting, some questions may not have an answer at all or may have multiple or even long-form answers. For such questions, agents should provide credible sources for each possible answer and should be evaluated on source quality. Second, while BEARCUBS includes questions that are multilin- gual, testing the ability of agents to handle multilingual queries is not the primary goal of
14

---

Published as a conference paper at COLM 2025
our benchmark due to its limited size. We encourage future research to conduct systematic studies on agents' performance across different cultures and languages. Third, directly comparing agents based their action trajectories is challenging due to inconsistencies in the level of detail individual agents provide. We encourage future advancements in agent transparency to enable more straightforward and meaningful comparisons.
B Data creation workflow
We seek questions that satisfy the following four high-level criteria. The concrete data creation workflow can be found in Figure 3.
(1) Questions should short but unambiguous: questions should provide sufficient yet mini- mal information to unambiguously lead to correct answers.
(2) Answerr should be trivial evaluate: answers must be correct, unique, and concise. Answers cannot be lists or sets, unlike, for example, AssistantBench (Yoran et al., 2024), and paraphrases of answers cannot be considered correct.
(3) Answers should adversarial to Google Search: answers must not appear in Google Search snippets or top-ranked results when the question fragments of the question are used as the query. Furthermore, multimodal questions must not be solvable by methods that only operate text (e.g,, Deep Research).
(4) Answers must be publicly accessible: answers must be available on non-paywalled websites, without requiring any account creation or login actions.
Figure 3 details the data validation process discussed in Section 3. Each question is verified by at least two authors who review each question carefully against the four criteria, along with its viable trajectory and visited links, We remove questions whose trajectories involve multimodal interaction but were found to have a text-only workaround by non-multimodal agents.
Quality verification [by authors)
Question writing (by an author or freelancers)
1
Review approaches and referenced links for accuracy.
Ensure questions and answers are clear and unambiguous.
Question:
Discard unqualified questions
Patatap website pink, circle appears when you press key on your koyboard. What basic color is the circle?
background
Workaround prevention
Viable approach:
1. Conduct extensive searches to ensure answers are not retrievable from search result snippets.
Search for "patatap"
2. Discard questions that answerable ChatGPT.
Visited links: http::/wwwwgooglee.c htwp://wwwgooll..mmmmmee/ttaooo
3. Discard multimodal questions
answerable by Deep Research through text-only interactions below).
Figure 3: Workflow of creating BEARCUBS. Each question and its viable trajectory and visited links are verified by at least two authors. Each multimodal question is verified to be only solvable via multi-modal interactions.
C Human annotator recruitment
This section provides details on human annotator recruitment for the human study in Section 4.1. We organize BEARCUBS questions into sets, each containing all questions from a specific mon-English language and some English questions. We recruit native speaker volunteers for Arabic and Chinese questions, and we hire three annotators via Upwork to handle the Hindi, German, and Finnish questions, respectively. An English-only set is attempted by annotators and authors who did not write or validate them. The hired annotators receive $2.5 USD per question, with an additional $1 USD bonus for each correct answer to incentivize accuracy.
15

---

Published as a conference paper at COLM 2025
D Baseline prompt
We provide the baseline hyperparameters and prompt for the Google-searc--aggmeeee baselines in Table 4
Model: 40-2444-2--- 11
temperature: 0 # for both baseline seltings temperature: 0 # for both baseline settings for the Google-seanch-uggeette setting
max_tokens: 518
Model: DeepSeek R1
max_tokens: 8000
Prompt:
Use the provided context to answer the question accurately and concisely. Do not your own knowledge.
If the context contains direct answet, provide it in precise and straightforward manner.
the context does not provide clear answer, reply with 'No answer found'.
Avoid unnecessary elaboration.
Question: {question}
Context: (context) consists of list of result titles and snippets Answer:
Table 4: Baseline hyperparameters and the prompt in the Google-searc--agmeenee setting.
E Human performance results
Table 5 complements the discussion in Section 5.1 by providing detailed results on the number of dead ends encountered, time taken, and perceived difficulty experienced by the human annotators.
Correctness
Dead End
Time (min:sec)
Perceived Difficulty
Accuracy Correct Wrong
Avg.
Avg.
Fasy
Medium Hard
Text-based
83.6%
46
9
1
1.83
12
5:19
0:43
27:24
13
Multimodal
85.7%
48
5
2
1.22
14
4:23
0:26
24:14
32
7
All
84.7%
94
14
3
1.50
14
4:46
0:26
27:24
55
36
20
Table 5: Humans achieve 84.7% accuracy on BEARCUBS and were generally able to find answers smoothly (1.5 dead ends on average) and quickly (4 min 46 sec). The None label marks questions abandoned after 15 minutes. Perceived difficulty reflects annotators' subjec- tive assessment of question difficulty. The minimum number of dead ends was zero.
Human error examples
Table 6 lists six key causes of human errors mentioned in Section 5.1, along with correspond- ing examples and explanations.
G Details of automatic agent answer evaluator
This section provides the implementation details of the automatic evaluator in Section 4.2.
Prompt of the automatic evaluator An example of the prompt used for automatic evaluation shown in Table 7. Because the prompt contains 17 in-context examples, the full text is omitted here. The complete prompt and the code of the evaluator are available at https://bear- cubrr. github. io/.
The evaluator is built 2ot-44--222-11-12 its base LLM, with the temperature fixed at Running the evaluator on 111 questions takes approximately 1 minute and 30 seconds, costing around $0.80 USD.
Modal and hyperparameters
Evaluator performance As shown in the prompt in Table 7, the evaluator performs four-way classification task. We also report results with the four labels collapsed into two
16

---

Published as a conference paper at COLM 2025
Error lype & question
Explanation
Type:
Missing details in the question or answer; Lack of topic
The annotator (1) was unfamilliar with the game, (2) found Wingspan card website lacking expansion and habitat details, then identified and searched bird with "great" in- dividually, and (3) overlooked the Asia pansion requirement.
knowledge; Suboptimal source selection.
Question: Which bird(s) the Wingspan bird cards from the Asia expansion have the word "Great" (but not "Greater") in their name and can reside exclusively in wetlands?
Type: Obvious oversight
The annotator accessed a relevant magazine page but overlooked the answer it.
Questions: What does October 2024 edition (817) Tinkle
magazine teach readers to make on page 36?
Type: Complexity of task
The annotator easily found the website gave up after failing to identify the correct map markers representing the Snow Leop- ard's location at the specified date and time.
Question: On l'ebruary 10, 2016, at 2:00 AM, tagged Altai Snow Leopard was observed in West Mongolia, according to the data from USGS. What was the straight- line distance it traveled by the time it was observed again at 6:00 AM the same day?
Table 6: Examples of human errors. The annotators answered 14 questions incorrectly and gave up on 3, primarily due to five key reasons listed in the table. We provide three examples, each with attributed error reasons and detailed explanation. The two more frequent error reasons are missing details in the question or answer and lack of topic knowledge.
categories, correct and incorrect. The evaluator was applied to the outputs of five agents: Grok3 DeepSearch, OpenAl Deep Research, Anthropic Computer Use, Convergence AI Proxy, and OpenAl Operator. The results for both the four-way and binary settings are presented in Table 8. To verify the reliability of the evaluator, we ran it three times on outputs from Anthropic and Proxy, obtaining identical results in all runs.
H Detailed agent performance results
Table 9 provides detailed agent results, grouped by all questions, text-based, and multi- modal questions.
I Justifying size of BEARCUBS
Small size benchmarks are not rare. BEARCUBS currently contains 111 questions. Such benchmark size is not uncommon in prior popular benchmarks. The widely used AIME dataset (MAA, 2024) contains only 30 questions and is still adopted in both evaluation and model development efforts such as by OpenAl (OpenAI, 2025) and DeepSeek (DeepSeek-Al et al., 2025). Similarly, CodeElo (Quan et al., 2025) includes just 387 problems, and the Proof or Bluff dataset (Petrov et al., 2025) uses only six USAMO questions for testing.
Creating high-quality BEARCUBS questions is non-trivial. In our experience, freelance annotators were only able to write small number of accepted questions before quality declined noticeably. Additionally, OpenAl's Deep Research agent frequently identified unintended shortcuts, especially in multimodal setting, which led us to discard many questions to preserve benchmark integrity.
BEARCUBS remains meaningful benchmark. This is due to two key factors. First, the search space for BEARCUBS questions spans the entire real-world web, unlike, for example, constrained multiple-choice formats. This makes guessing the correct answer difficult and reduces accuracy variance. Second, solving even a single BEARCUBS question typically requires a complex mix of retrieval, reasoning, and synthesis, making each example informative for evaluating model capabilities. To support the first claim, we evaluated model accuracy variance BFARCUBS by running DeepSeek R1 with and without Serper across
17

---

Published as a conference paper at COLM 2025
Prompt of the automatic agent response evaluator
Given question, its gold answer, and an Al generated answer, your task is to check for the following four cases:
1. no answer: Al generated answer is empty.
2. no direct answer: Agent abstains or hedges with modal verbs (e.g., "may", ""maaa""culo"" be") or phrases expressing uncertainty such "according to common sense" or "I am leaning toward." correct: returns direct and unambiguous answer which entails the gold answer.
4. wrong: returns a direct and unambiguous answer which, however, does not entail the gold answer.
Every question has an answer. the Al answer rejects the premise a question, should be labeled as "wrong."
The gold answers always concise and never presented as list. They may be number, price, word, or phrase, among other types. In contrast, Al answers can be long, such paragraph, with an answer embedded within. The embedded answer may be correct but differ in format (e-g,, how a number is written). Nonetheless, they should express the same information (i.e., and four are different, 303kg and 302kg are different.) Below, I provide examples of how to decide which of the four labels assign to the Al answer along with explanations. However, in your evaluation, you do not need explain your decisions.
Example 1:
Question: What the of the bird(s) in the Wingspan bird cards from the Asia expansion have the word "Great" (but not "Greater") in their name and can reside exclusively in wetlands?
Gold Answer: Great Cormorant
Al Answer:
Label: no answer
Explanation: The Al answer empty. Hence, the label is "no "no_answer:"
Example 2:
Question: In episode of Wissen Macht Ah! in April 2023, the hosts demonstrated how to make DIY water filter the wild. What was the second filtering material that they placed into the plastic bottle? Answer the question with the exact word used in the episode?
Gold Answer: Activkohle
AI Answer: Sorry, don't have access to specific episodes of *Wissen Macht Ah!* or their content, including the exact word used in the April 2023 episode. You might want check the episode directly or refer to its official resources for the precise information. Label: no direct answer
Explanation: The Al answer does not provide direct answer to the question and instead states that it does not have access to the specific episode's content.
Further examples omitted for the sake of space.
Now it your turn label the following Al answers. You do not need to explain your decisions.
Question: {question}
Gold Answer: (gnweer
AT Answer: {ai_output)
Label:
Table 7: Example of the prompt used for the automatic agent response evaluator. Examples 3 to 17 are omitted for the sake of space.
four independent trials. The results, summarized below, show minimal standard deviation in accuracy across runs. 21
J Model correct answer source proportions
Figure 4 presents the proportion of correct answers within each agent categorized by reliance on primary sources, secondary sources, and ungrounded responses. The analysis is provided in Section 6.
The first run was conducted in March 2025 using the Fireworks AT API; subsequent runs were carried out via OpenRouter.
18

---

Published as a conference paper at COLM 2025
Agent
4-Way Accuracy
Binary Accuracy
Grok3 Deep Search
99.1%
99.1%
OpenAl Deep Research
96.4%
96.4%
Anthropic Computer Use
99.1%
100%
Convergence AI Proxy
99.1%
100%
OpenAî Operator
97.3%
98.2%
Table 8: Evaluator accuracy by agent in the four-way classification setting. Binary accuracy is derived by collapsing the four labels into two categories: correct and incorrect.
Model
Accuracy
Answer label
Average time
Correct answer source attribution
Correct Wrong Uncertiin Nove
Conrect
Wroong
Uncertain
Ungrounded
Primary
Secondary
All Questions
GPT-4o zcro-shel
2.%
53
55
Despbock 8I shot
8.1%
9
H2
19
GPT-4a Google Search
0.0%
0
1/7
DespSeck Google
18%
Perpleiit soran-pro
DeepSarch
OpenAl Deep Research
36.%
4:37
9:0
338
25
10
Google Deep Research
23.4%
26
39
46
8:21
$10
4:39
25
Convergrnce Al Provy
12.6%
44
19
1:50
2411
534
13
Anthropic Computer Use
22
221
2:5
335
OpenAl Operator
OpenAl ChaKPT Agent
Humann
B17%
14
4:21
5:11
Questions
GPT-4o acro-shot
DeepSeek
GPT-4a Google Search
0.0%.
Deepbeck Google Search
3.%
II
Peppleiity sonaripre
8.8%
19
26
Grok3 DeepSarch
21.4%
12
42
2
1:37
1:30
OpenAl Deep Research
607%
8:12
8:32
24
10
Google
Convergence Proxy
Anthropic Computer Use
19.6%
12
2:19
2:27
3:11
OpenAl Operaiat
33.9%
19
24
10
3:13
3:51
7.24
18
OpenAA CharGPT Agent
76.%%
10
6:29
12:06
43:49
43
Human
538
5:07
Multimodal Questions
GPTI-4 acrosshet
10%
23
ThepSeek neno-shot
355.
3
44
7
GPT-4a Google Search
0.%%
01
55
DeepSeek Google Search
0.0%
Perplesity
DeepSearch
1,%
OpenAl Deep Research
10.9%
08
6:38
9:12
3:58
Google Deep Research
3,6%
26
27
4:15
3:53
4:16
Convergence Al Procy
9.1%
15
20
12
1:29
2:48
10:42
Anthropic Computer Use
9.1%
10
122
2:15
3:17
OpenAl Operalor
OpenAl Agent
3.41
6:4
Table 9: Model performance on BEARCUBS with the baselines gpt-40-2000 200-o-2224-111 11-111110 and DeepSeek R1 (with/ without Google Search), Perplexity sonar-pro, deep research agents
computer-use agents, and human performance. 'Uncertain' indicates that the agent did not return a concrete answer (e.g., abstention), while 'None' means either baseline model returned "No answer found" response. 'Ungrounded' answers are those not based on a source but rather on the agent's internal knowledge
agent entered dead loop and failed to provide any
reasoning. 'Primary' denotes an answer derived from reliable
source or the source specified in the question, whereas 'Secondary' refers to an answer obtained from an unreliable source source not mentioned in the question.
K Agent-specific behavior
We continue the discussion in Section 6 and present interesting agent-specific behavior and the challenges those agents face that hinder their utility.
19

---

Published as a conference paper at COLM 2025
Setup
1st Run
2nd Run
3rd Run
4th Run
Std. Dev.
DeepSeek R1 w/o Serper
7 7.2%
5,4%
5.4%
5.4%
0.8%
DeepSeek R1 w/ Serper
1.8%
0.9%
2.7%
2.7%
0.7%
Table 10:
Accuracy standard deviation across four trials of DeepSeek R1 on BEARCUBS, with and without Serper. Results show low standard deviation, indicating stable performance.
Primary Secondary Ungrounded
Grok3 DeepSearch
61.54
15.38
23.08
OpenAl Deep Research
61.54
25.64
12.82
Convergence Al Proxy
92.86
7.14
Anthropic Computer Use
87.5
12,5
OpenAl Operator
89.29
7.14
3.57
10
30
40
50
80
90
100
Figure 4: Proportion of correct answers within each agent categorized by reliance on primary sources, secondary sources, and ungrounded responses.
We observe that Grok 3 returns its search and reasoning trajectory in a non-English language if it appears in the input, which can be unhelpful for users seeking assistance for that lan- guage. Grok 3 also never abstains, always generating response (see Table 2) behavior that needs user study determine its desirability. Meanwhile, Computer Use sometimes deems task impossible without attempting it, despite users expecting an effort and jus- tification. These issues, along with the broader challenges discussed above, highlight the need for improved adaptability, transparency, and user-centered refinement in LLM-based computer use agents.
20


Reviews:
[{"review_id": "1wEVjA41pz", "reviewer": "Reviewer_KfhM", "summary": "This paper proposes BEARCUBS, which assesses web agents' capabilities in real-world, multimodal web interactions. It comprises 111 information-seeking questions requiring agents to interact with live web content. The questions were written by the researchers and crowdsourcing workers. Each question and its viable trajectory and visited links were verified by at least two authors to ensure it can't be answered by text-based workarounds. The authors claim that BEARCUBS would be an growing and evolving dataset, with periodic updates to replace invalid or contaminated questions. \n\nExperiments on the new benchmark show that SOTA open-domain agents underperformed significantly than human users. The main issues are inadequate exploration, multimodal capabilities, and source selection difficulties.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It presents a focused development of a challenging benchmark for information-seeking agents.\n2. The authors have invested significant effort in ensuring the quality and reliability of the BEARCUBS dataset.\n3. SOTA agent models fail significantly on this benchmark, exposing their limitations.", "reasons_to_reject": "1. The dataset curation relies heavily on manual efforts, making it difficult to scale up the benchmark or automatically maintain and update the dataset over time.\n\n2. It is unclear how the authors and workers devised the questions; upon examining the questions, it appears that the workers already knew the answers beforehand, resulting in tasks that differ notably from typical information-seeking questions posed by real-world users.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vR96ePjlV8", "reviewer": "Reviewer_osWr", "summary": "The paper introduces BEARCUBS, a benchmark of 111 information-seeking tasks for evaluating web agents operating on the live web using real-world multimodal interactions. Unlike prior benchmarks based on synthetic or static pages, BEARCUBS emphasizes unpredictability and realistic human-like browsing. It features human-verified trajectories, adversarial question design, and ongoing updates to prevent contamination.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Moves beyond static environments and multimodal requirement ensures questions cannot be answered via simple Google search or LLM knowledge. Human accuracy (85%) confirms tasks are solvable, yet challenging.\n\n- Each question includes a validated human trajectory, enabling fine-grained analysis of agent behavior and supporting more transparent evaluation.", "reasons_to_reject": "- The dataset size is small at 111 questions which, while described as \"small but mighty,\" limits its statistical strength for training or broad evaluation.\n\n- The paper lacks an automation pipeline; all evaluations are conducted manually, making reproduction and benchmarking slow and resource-intensive, and the promised benchmark maintenance remains a future commitment rather than a demonstrated feature.\n\n- At the time of review, the benchmark does not appear to be publicly released or open-sourced. Without access to the dataset or tooling, it’s difficult to verify claims or build upon this work.", "questions_to_authors": "1. What are your concrete plans for automating evaluation or building community tooling so BEARCUBS can scale beyond internal experiments and remain viable as a long-term benchmark?\n\nComment : I would like to see the authors' rebuttal and am open to adjusting my score if the concerns I raised—particularly around benchmark accessibility, evaluation scalability, and maintenance—are clarified or addressed.", "extra_scores": {}}, {"review_id": "pvpuD9SUar", "reviewer": "Reviewer_HtzH", "summary": "This paper presents a benchmark consisting of 111 information-seeking questions designed to evaluate a web agent’s ability to search, browse, and identify factual information from the web. The benchmark requires: (1) accessing live web content (not synthetic or simulated pages), and (2) performing a broad range of multimodal interactions including video understanding and 3D navigation. Questions have short, unambiguous answers the benchmark includes human-validated browsing trajectories. Human accuracy is 84.7%. The authors assert that this is explained in large part by search inefficiencies and domain knowledge gaps.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The benchmark seems like it might last for a while, since GPT4o zero shot performance was low, OpenAI Deep Research has 60% text-based accuracy and 9% on the multimodal questions.", "reasons_to_reject": "The benchmark is quite small, with just 111 questions and\n “Solving BEARCUBS requires visiting 108 unique top-level URLs, minimizing the risk of agents overfitting to specific websites.”\nThis is not a lot by modern standards.", "questions_to_authors": "In the example provided on page 8 a sequence of URLs from Deep Research has been provided. However, it might be the case that Antidistillation Sampling (i.e. https://arxiv.org/abs/2504.13146)  or some form of obfuscation is being used by OpenAI. Have you considered this ?\n\nSecont 5.1 says “Analysis reveals that the most common factor for wrong answers is the human overlooking details in the question or answer (e.g., Example 1 in Table 6 in Appendix F). The next most common factor is lack of topic understanding (see also Example 1). Additional error sources are listed in Table 6, along with examples and explanations. In each error case, if the human annotator had spent more time or had more domain knowledge, they would likely have been able to find the correct answer.”\nBut why does the abstract point to search inefficiencies and knowledge gaps ?", "extra_scores": {}}, {"review_id": "42RNDhzxOq", "reviewer": "Reviewer_ivpU", "summary": "The paper proposes a new benchmark for evaluating web browsing agent on information retrieval and question-answering tasks on real-world websites. The benchmark consists of 111 questions, collected from the paper authors as well as crowdsourcing, and extensively validated and filtered to ensure a challenging set. In particular, the multimodal subset is designed to be impossible to be solved without multimodal capabilities, and the authors promise to ensure this in the future by evolving the benchmark once the web changes.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem is highly timely, with a proliferation of both commercial and research systems in this space, and the need to accurately assess the capability. Other commonly-used benchmarks are close to saturation (WebVoyager) or have methodological limitations (WebArena). Hence, there is a need for a new benchmark.\n\nThe goal of constructing an adversarially challenging benchmark is sound, and it seems this was largely accomplished, given poor results from state-of-the-art models, while humans with relevant domain-specific knowledge can solve the benchmark tasks with no issue. In particular, the fact that human-perceived difficult correlates _inversely_ with agent performance could this a very powerful benchmark for the goal of building human-quality agents.\n\nThe authors are promising to keep the benchmark up to date over time, which avoids the limitation of either simulated environments or recorded websites, and maintains the benchmark relevant even if answers leak into training set.", "reasons_to_reject": "There are a few concerns on the design of the benchmark which might undermine some of the claims in the paper:\n1. The benchmark is designed to be adversarial to search method, but the baseline LLM+search that only provides the top 10 search snippets is naive, and probably does not match SOTA search agents like Perplexity or Google Search AI overview. It is unclear what a SOTA agent would do on this benchmark, and it is unclear if a crawler+RAG-based mechanism that also indexes and retrieves page screenshots could be successful.\n2. It is unclear what _specific_ skills of browser agents is this benchmark actually measuring: is it multimodal understanding of page screenshots? Navigation? Interaction with forms, widgets, and other controls? The benchmark is very broad in use case. As a _browser agent_ benchmark, it is unclear how meaningful is to include music sheet understanding or video understanding. On the other hand, as a _general purpose, all encompassing_ agent benchmark, the benchmark is clearly too small to be representative.\n2. Related to the previous point, it is unclear how realistic the questions are, and thus how transferable are results on this benchmark to real-world agent usage. The purpose of a benchmark should be to inform or assess real-world performance. Yet, the example seems unnatural, perhaps due to the requirement of exact-match checking for answers or to how the questions were obtained.\n\nI also see potential issues around the evaluation protocol:\n\n4. Answers are manually evaluated, and what appears to be a subjective judgement is made on whether the agent is sufficiently confident of its answer. This is highly unusual: a more customary approach would be to use an LLM judge, or to instruct the agents to produce answers in a machine-readable format through appropriate prompting. I understand the desire of using the same prompts for all agents, but a per-model prompt template (independent of the goal) would smooth out irrelevant differences among systems and still achieve the benchmark goals.\n5. All agents are asked to bypass CAPTCHAs. It is first of all questionable to ask publicly released systems (with safety concerns and protections) to solve CAPTCHAs, when compared to unreleased research systems or open-source systems that have no such limitation. It is also generally unclear what is the value of evaluating an agent on the ability to override a CAPTCHA, because as CAPTCHAs become better or unbypassable (eg through attestation), it is reasonable to expect that agents will only be operated on websites that explicit consent to AI interaction.", "questions_to_authors": "1. I don't fully understand how filtering was done for multi-modal questions (lines 53-54, lines 474-475), given Deep Research solves 9% of multi-modal only questions nonetheless. This seems inconsistent in the text: either those questions are indeed solvable by text-only methods, or they are randomly guessable with almost 10% accuracy, which would raise further questions.\n\n2. OpenAI Deep Research is described as an agent without computer use in Table 2. This might be a bit misleading: Deep Research has a browsing tool (separate from the search tool) which can interact with live websites. Note that OpenAI itself evaluated Deep Research on a benchmark specifically for browser agents (https://openai.com/index/browsecomp/).", "extra_scores": {}}]

Author rebuttal:

Meta-review: The proposed method for adversarially generating web navigation tasks is well-founded and presents a meaningful challenge to current state-of-the-art models. The authors’ commitment to continuously updating the benchmark further enhances its long-term value and relevance to the web agent research community. As such, this benchmark represents a valuable contribution to the field of web agent evaluation. While the authors note that small benchmark sizes are not uncommon, providing statistically significant testing would strengthen this point
Decision: Accept