VideoWebArena:  Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks
Authors: Lawrence Keunho Jang
Yinheng Li
Dan Zhao
Charles Ding
Justin Lin
Paul Pu Liang
Rogerio Bonatti
Kazuhito Koishida
Venue: ICLR 2025 Poster (2025)
Keywords: agents
benchmark
video understanding
multimodal agents
Primary area: datasets and benchmarks

Abstract: Videos are often used to learn or extract the necessary information to complete
tasks in ways different than what text or static imagery can provide. However, many
existing agent benchmarks neglect long-context video understanding, instead focus-
ing on text or static image inputs. To bridge this gap, we introduce VideoWebArena
(VideoWA), a benchmark for evaluating the capabilities of long-context multimodal
agents for video understanding. VideoWA consists of 2,021 web agent tasks based
on manually crafted video tutorials, which total almost four hours of content. For
our benchmark, we define a taxonomy of long-context video-based agent tasks with
two main areas of focus: skill retention and factual retention. While skill retention
tasks evaluate whether an agent can use a given human demonstration to complete
a task efficiently, the factual retention task evaluates whether an agent can retrieve
instruction-relevant information from a video to complete a task. We find that the
best model achieves a 13.3% success rate on factual retention tasks and 45.8% on
factual retention QA pairs—far below human success rates of 73.9% and 79.3%,
respectively. On skill retention tasks, long-context models perform worse with
tutorials than without, exhibiting a 5% performance decrease in WebArena tasks
and a 10.3% decrease in VisualWebArena tasks. Our work highlights performance
gaps in the agentic abilities of long-context multimodal models and provides as a
testbed for the future development of long-context video agents.

information Published as a conference paper at ICLR 2025
VIDEOWEBARENA:
EVALUATING LONG CONTEXT MULTIMODAL AGENTS WITH VIDEO UNDERSTANDING WEB TASKS
Lawrence Jang Yinheng Li Dan Zhao
Charles Ding', Justin Lin', Paul Pu Liang?, Rogerio Bonatti Kazuhito Koishida !Camegie Mellon University, *Massachusetts Institute of Technology, "Microsoft
ABSTRACT
Videos are often used as a unique source of information in learning how to perform tasks in ways different than what text static imagery can provide. However, many existing agent benchmarks neglect long-context video understanding, instead focus- ing on text or static image inputs. To bridge this gap, we introduce RideoWebArena (VideoWA), benchmark for evaluating the capabilities of long-context multimodal agents for video understanding. Video WA consists 2,021 web agent tasks based manually crafted video tutorials, which total almost four hours of content. For our benchmark, we define a taxonomy of long-conme cong-context rideo-based agent tasks with two main areas of focus: skill retention and factual retention. While skill retention tasks evaluate whether an agent can use a given human demonstration to complete task efficiently, factual retention tasks evaluate whether agent retrieve instruction-relewwe information from a video to complete a task. We find that the best model achieves 13.3% success rate on factual retention tasks and 45, 5.8% on factual retention QA pairs far below human success rates of 73.9% and 79.3%, respectively. On skill retention tasks, long-context models perform worse with tutorials than without, exhibiting a 5% performance decrease in WebArena tasks and a 10.3% decrease VisualWehArena tasks. Our work highlights performance gaps in the agentic abilities of long-context multimodal models when it comes to video understanding and provides as a testbed for the future development long-context video agents
INTRODUCTION
Humans often use videos to complete tasks fookowwil tutorials to retrieving information from within one, or across several, videos. As large foundation models are adapted into agents Zheng et al. (2024); Wang et al. (2023); Putta et al. (2024), these multimodal agents may also require similar capabilities to understand and process videos in completing various tasks- especially if they need to retrieve and process said multimodal information (c.g.. from the web) to complete tasks that require new knowledge. In these scenarios, the ability of models to maintain long-term memory Zhang et al. 2024b), retrieve and use said information Lewis al. 2020), and adapt continuously Wu et al. (202cc) is critical for complex multi-step tasks.
While videos contain spatial and temporal dynamics that images or text alone may not convey, integrating video input into multimodal models introduces unique challenges (Tang et al. 2024) such temporal coherence, context retention, efficient retrieval over lengthy sequences. However, recent vanncements in long-contex understanding of large vidco-capable vision language models, such as LLa (Liu al. 2023). LongVILA (Xue et al. 2024), and Gemini (Google) 2023) have enabled these agentic models to not only process and understand more information than before, including long video understanding, but also generate and act more freely than before. While many
Link to code httpps::s::gtthptp http::///gsthtuoooomiiiigeeeeeee
Link to video http:://w/wwwwwwbbbteeeete http:///////wwoyyoyoooo////////r
https://drive.. google..gg.oooo.o http:://...t..oooooo.o//ccm///ee
link

---

Published as a conference paper at ICLR 2025
existing agentic benchmarks test, they focus primarily on text and image modalities for tasks like interactive QA Mialon et al. (202)), computer tasks Bonatti et al. (2024): Deng et al. (2023) etc.
As result, from evaluative perspective, there remains significant gap in existing benchmarks that can comprchensively evaluate the agentic capabilities these models across diverse multi- modal scenarios, particularly those involving video inputs. The requirement for agents operate across varying modalities and time frames makes developing and properly evaluating long-context multimodal models essential. Existing benchmarks (Wu et al., 2024b; Mangalam et al., 2023; Li et al. 2024. Päträucean et al. 2023) often fall short in testing for long-term memory retention and multimodal integration within an agentic workflow, limiting our understanding of how long context multimodal models can generalize and perform in real-world settings as agents.
VideoWebArena Domains
Tasks are Video-Based
Shopping Video
Video-
"Day that the pirson wrobe 4 ytar revien
Osclass
reddit
Westlydly
Task-Based Video
Map
GitLab
CMS
are video
Astemutte Drckattion
Figure Overview of VideaWehArena. VideoWehArena visually grounded benchmark that tests the video understanding of agentic models across various realistic domains and environments, mirmring real-life tasks, All tasks require video inpur and consist of Q/A to test agentic ahilities in video
retrieval, understanding. mure.
Contributions. To address this gap, we present novel, open-source video-based benchmark that evaluates multimodal models' agentic ability to process, understand, and utilize long-context video to accomplish various tasks. In sum, our contributions are:
We present Video SeeeWoebrre--- benchmark focusing specifically on evaluating a model's ability process long video sequences alongside text and images to complete tasks that require memory retention, information retrieval, multimodal reasoning, and skill retention. consists of 2.021 tasks and details approximately 4 hours of video content.
Of these tasks, Video WebArena consists of 400 factual retention tasks, which agents' abilities to retrieve information from video to perform tasks, and 1,621 skill retention tasks, which test agents' abilitics use tutorials in-context to perform tasks more efficiently.
We evaluate popular and recent video//iage-capabll state-of-ther the---r LLMs (e.g., GPT-4o and Gemini 1.5 Pro) to better understand their current long-context video understanding capabilities. Our results show that video/image-capabl agents are still limited and far from human levels of performance, highlighting a considerable gap in the information retrieval and agentic abilities of current ofate-of the-art long-contest models.
The of our paper is organized follows. We begin with discussion of prior works in Section before detailing the benchmark environment Section Section 4 describes our experimental setup and Section 5 laays out our benchmarking results. We conclude and discuss our findings in Section
BACKGROUND
Large Vision Language Models. Large vision language models (VLMs) have been popular candidates for incorporating video input understanding. Popular state-of-the-art (SOTA) models like the GPT-4 (OpenAl 2024) family models, Claude (AI Anthropic (024),, and Gemini (Goggle) 2023 are now able to handle not just text but also visual and even audio input. Similar to LLMs, VLM architectures typically revolve around two types models with either a joint encoder-decoder
architecture such LLaVA and its variations (Liu al. 2023)
decoder-on. architecture. Encoder-decoder VLMs tend to project different modalities through a shallow neural network or fully connected layer link modalities while decoder-style models typically rely decoder-only
2

---

Published as a conference paper at ICLR 2025
Benchmark
Duration (Avg)
Multi-domain?
Multi-skill?
Audio?
Agentic?
MSVD-QA QA ex all 2017
0.33
X
MSRVTT-QA Xure
0.25
ActivityNet. Qa Toee 4019
1,85
X
NExT-QA Nian
MoVQA Zhang of
16.5
X
X
MovieCllat-AA SSng afal 2023
9.4
X
X X X X
MVBench eL
0.27
Penception (Parzzneein et
2023
0.38
CinePile Rawal
EigoSchema Mangalam ei 2024
2.7
X
X
LongVideo
20744
X
Vidzo-MME Fu et AU2431)
16.9
X
Vide (this work)
3.1
Table I: Comparison of relevant bonchmarks/dataser focusing on vidco modality understanding. Duration denotes the average vidco duration minutes, multi-domain denotes videos cover diverse range across multiple domains, multi-skill denotes whether benchmark multiple types/aspects of video understanding. audio demotes if the video's audio information is included or used, and agentic denotes whether the henchmark uses QA part agent's workflow as rutorialaaid, note that, unlike many beachmarks here, MederWa comes with fully interactive covironment, uses open-answer questions relating video understunding (rather than strictly providig multiple choice options to the model), evalnates task completion on both the final state but also intermediary states,
input required completing multi-step task, etc.
five typeslaspects of vidco understanding grouped into categorics (factual vs, skill retention) and
LLM that processes raw inputs (e.g., text tokens, image patches, etc.) such as VILA (Lin al. 2023) and its variants like VILA (Fang et al. 2024) and X-VILA Ye et al. 2024). As multimodal understanding with long-contexs capability becomes more important in processing increasingly large amounts of input information, like video data, models like Long VILA (Xue et al. 2024) process a much larger number of video frames in its input for longer videos.
Agents. Recent work has turned to adapting large multimodal models like VLMs into autonomous agents, given their capabilities in visual question answering, instruction-follovinn and high-level reasoning. Many approaches have been developed to improve supplement the agentic capabilities of these large models, ranging from data collection and synthesis for specific kinds of training (Lai
al. 2024 Patel et al. 2024) or for at inference time (Pan al. [024 Fu et al., 2024bb Koh et al. 2024bb Sarch et al. 2024; Wang et al., 2024). Similar works have shown how certain fine-tuning prompting (e.g., Set-of-Marke (Yang al. (023). (Chi et al., 2024 can
(Furuta et al.) 2024)
improve performance in navigating web pages to accomplish tasks. Other works have attempted to
improve how agents dynamically compose/search for policies themselves (Sodhi al. 2024
Agent Benchmarks. As more works adapt LLMs and VLMs into agents, there is need to evaluate their performance. These benchmarks can range across variety of settings: from general web browsing YYo et al., 20222] Deng et al. 2023 Zhou et al. [2224] Koh et al. (0024)), where agents are evaluated on their abilities navigate the web and accomplish specific tasks, to mobile environments, where agents are expected to perform tasks within a mobile OS simulation like Android (Zhang et al., 2024a| Rawles et al. 2024). Other general environments attempt to emulate
OS or computing environment like MMIn (Zhang et al.] 2024c), OSWorld (Xic et al. 2024), and Windows Agent navigate
multiple computer applications online and offfine. Custom environments like Work Arenarena (Drouin et al., 2024) instead target more specific platforms like ServiceNow in constructing tasks for agent evaluation.
Arena (Bonatti et al.
2024 where agents
Long-Contexs Video Benchmarks. For large multimodal models, long-context capabilities are essential for detailed planning, action, and understanding, especially for the video modality. There have been multiple benchmarks dedicated towards video understanding, with shorter video inputs (Li et al. 2024), around few minutes, and longer video inputs (Wu et al.] 2024b Păträucean et al.) 2023), up to over an hour. Video understanding benchmarks dedicated to temporal reasoning and thematic reasoning (Xiao et al., 2021b: Tapaswi et al. 2016 Lei et al. 2019) also exist. Many of these benchmarks cover subsets and define categories of video understanding tasks related to spatial reasoning, causal reasoning, and temporal reasoning.
3

---

Published as a conference paper at ICLR 2025
Domain
Factual Tasks
# Skill Tasks
Total Tasks
Reddit
(22%)
205 (13%)
293 (14%)
Variable
Value
Classifieds
(15%)
320 (20%)
(19%)
Videos
74
Shopping
121 (30%)
654 (40%))
(38%)
Total Duration
3:48:19
Min Duration
01:16
Shapping
47 (12%)
182 (11%)
229 (11%)
Max Duration
10:41
(Admin)
Avg. Duration
3:05
Avg. Factual Retention Tasks Video
5.4
Gitlaa
(18%)
191 (125)
(13%)
Avg. Skill Retcation Tisks per Video
19.6
Avg. Videos per Domain
12.3
Map
15 (4%)
(4%)
83 (4%)
Table 2:
Video statistics for VideoWebArenn.
Total
(100%))
2021 (1009)
Table 3: Distribation of tasks for VideoWehAren WehArena broken down between tasks test retention and factual retention.
3 VIDEOWEBARENA ENVIRONMENT
3.1 SUMMARY & OVERVIRW
VideoWA centers around six key thematic environments created by VisualWebArena (Koh al. 2024a) and WebArena (Zhou et al. 2024): Reddit, Classifieds, Shopping, Shopping Admin, Map, and Gitlab. See Tables and for finer characterization of the tasks and videos within the benchmark.
These domains' websites are locally hosted since the docker images for each website are publicly available online. There is an Amazon Machine Image and instructions dedicated to hosting these websites EC2 instance; we refer readers to our codebase for further information. By doing this, we can make our benchmark realistic and reproducible, leveraging data and code from real and popular websites on the internet. We refer readers to WebArena (Zhou et al., 2024) and VisualWebArena (Koh et al. 2024a) for more information on each site and its setup.
dee bifeuity Tosk Der button
Agent 3ificuly Dettrtrtutto
Marl
25.2%
22.1%
4%
Tany
(138)
Neduu
Figure 2: VideoWehArena Difficulty Task Distribution. Right: VideoWehArena Agent Difficulty Distribution.
Videoitebrrenns Task Category Counts
354
176
152
302
85
35-
Misal Percaptin
Audio Percaction
Vioe
Turrporsl Reasoring
Figure VidroWehArena Factual Retention Task Counts Each Category. The categorics non-exclusive. One task under multiple video perception categories.
4

---

Published as a conference paper at ICLR 2025
Action Type
Description
click [elen]
Click element elon.
hover [elen]
Hover on element elem.
type [elen] [text]
Type text on
press [key.conb]
Press key combination.
new.tab
Open a new
tab, [index]
Focus the i-th tab.
tabuclase
Close current tab
goto [url]
Open url.
go.back
Click the back button.
go forward
Click forward button.
scroll [up]down]
Scroll up down the page.
clear [elen]
Clear a text element.
upload [file puth] [elen]
Upload local file
upload bution.
stop [answer]
Fin the task with an output.
Table 4: List of Action Types and Descriptions
3.2 ENVIRONMENT DETAILS
3.2.1 FRAMEWORK
We define agent's trajectory (POMDP) (S, A, T R) with state space S, observation space O, action space A containing actions a, transition function T': A + and reward function R S. x A R. Given current observation E O, an agent generates executable action a E A. resulting in a new state 8141 E S and new partial observation Ot+1 E O. The reward function value at the final step if the agent state achieves the task objective and zero otherwise. We list available rewards and evaluators in Table
tasks partially observable Markov decision process
A [0, 1] returns non-zero
3.3 OBSERVATION SPACE
The observation space for VideoWA environment is predicated the Set-of-Marks observation space in VebualWebArena. The environment uses executable JavaScript code at each step to extract the interactable HTML elements from the webpage and present them top-down order. Similarly, the JavaScript code extracts the bounding boxes of each interactive element and a screenshot of the webpage with bounding boxes over said elements generated for input into the agent along with the text state representation. At each step, the agent is presented with the overlaid Set-of-Marks screenshot and text observation space, along with the chosen video information be put into context.
3.3.1 ACTION SPACE
We describe the action space of agents within the VideoWA environment in Table 4 Agents are prompted generate single action from the action space at each time step. Each action is associated with Playwright Python code that automatically performs the action within the browser. The '[elem]' represents the unique Set-of-Marks element that can be interacted with from the observation space provided through the environment's JavaScript code.
3.4 TASK DESIGN
The taxonomy covers two overall task types-skii skill retention and factual retention-inn iine--iidddddddee by real-world use cases. We illustrate the taxonomy breakdown in Figure We define skill retention as the ability to learn from and use a given human demonstration to efficiently complete task For example, using YouTube tutorials screen recordings of expert demonstrations to learn how to perform a task a form of skill retention. On the other hand, factual retention the ability to retrieve information relevant to user's specific question/task present in a video that may not be the video's main focus (e.g., an incidental detail). For example, one might want to buy the shoes a particular NBA player is wearing but only shown within short duration of much longer basketball highlights video. To complete the task, the model must extract not only information about the specific player but also their shoes, even if this information is secondary the main content of the video.
5

---

Published as a conference paper at ICLR 2025
Domain Video Tutorial
Task Category
Intent
Intermediate Intent
Buy Cheapest Ieem
Skill Retention
Buy the cheapest
NIA
red blanket from Blankets & Throws.
lxSkySky
Leave Comment
Audio Perception
Search for the company the person said they work the video and find the first post"s corments.
What company did the person in the video say they
reddit
How to Star a Repo
Full Video Unserstandding
Can you assign the with critical priority label that showed up earliest in the video to @earlev4?
What was the the name the issue with critical priority label showed up earliest in the video?
GitLab
Find Optimal Route
Temporal Reasoning
Find the that shows the zipcode of the 2nd destination in the video.
What the of the 2nd destination used in the vide??
Listing Ratingy
Reasoning
the red vehicle listing that appears video.
was nane the first red vehicle listing that appears the video?
Table Examples Each Wehroneh Taxonomy: intermediate intent tests the multimodal agent's ability to extract the necessary information to perform the task from the video. Skill retention will require.
agent perform
intermediate as they do not require recalling specific information factual
We present example tasks in Table and example of an agent stylized task in Figure 7 Each task has an 'intent', which is the objective of the task. For all of the newly created factual retention tasks, there is also 'intermediate intent', a video-based question that must be answered correctly to have the information necessary to complete the task. Each task also has an automatic evaluator function for both 'intent' and "intermediate_inter iintent' that returns score of environment and response given by the LLM agent. Each task also has an agentic difficulty, distributed between easy, medium, and hard. The agentic difficulty for each task signifies the complexity of the action sequence needed to complete an intent successfully. For agentic difficulty, we classify task as easy it can be completed in 1-3 steps, medium if can be completed in 4-9 steps, and hard if can be completed in more than steps. This classification is adopted from VebualWebArenaa (Koh ctt al. 2024a). Figure provides a detailed breakdown of task difficulty.
based on the
3.5
VIDEO CREATION AND SKILL RETENTION TASKS
Our benchmark contains 74 unique videos, totaling almost 4 hours of video content (see Table for details)-all of our video tutorials are based on tasks in WebArena and RisuaeWeeArena. We provide videos online through YouTube channel and Google Drive link containing the zip file of all the videos. We formulated these videos by accumulating all the feasible intent templates WebArena and VisualWebAAret We take 297 unique templates from VesuelWebArena and 220 unique templates from WebArena, totaling 1,621 total intents. Further details can found Appendix A.1
We map each of video tutorials the respective tasks in the WebArena and Visual WebArena task set to create skill retention tasks. We then create 400 original factual retention tasks based on these same tutorials. We had three of the paper's authors create videos and corresponding tasks for each

---

Published as a conference paper at ICLR 2025
video they created. We then conducted cross- cross-validation quality assurance with an author who did not make the video/tasks to ensure the task was understandable and able to be completed. We conduct human performance tests similarly, having author who didn't 't create the video or tasks attempt the task and have it evaluated by a third author. Further details on task creation and human evaluation can be found in A. 2 and A.3
3.6 FACTUAL RETENTION TASKS
For factual retention, we further divide this category into four finer sub-categories: (1) Visual Perception (OCR. Spatial Reasoning), (2) Audio Perception, (3) Full Video Understanding (i.e., tasks that require information across several parts of the video), and (4) Temporal Reasoning (i.e., tasks that require understanding the video with respect to time). One key difference between the factual and skill retention tasks the intermediate intent and evaluation we create for each factual retention task. The intermediate intent is the video-base question that must be answered correctly to have the information necessary to complete the task. This is intended to decouple the evaluation of agentic abilities in long context video models for video information retrieval tasks; by checking if the model can extract information necessary to complete the task from the video and evaluating that separately from the agent's 's success, this process pinpoint the failure modes of the model, whether they come from generating agent actions or video processing.
Additionally, provide video difficulty ratings for all intermediate intents: easy, medium, and hard. The difficulty ratings signify the complexity of returning the correct answer for a given task's intermediate intent. Easy tasks require returning one piece of information and can be solved with less than 3 frames, medium tasks require returning/retrieving 2 to 3 pieces of information and can be solved with less than half the video, and hard tasks require returning more than 3 pieces that require watching more than half the video. We provide a breakdown Table VideoWA contains 1 111 unique intent templates across the 400 intents the factual retention task set.
3.7
TASK EVALUATION
Each task has final and intermediate "eval' function. We import the automatic functionality from Visual WehArena (Koh et al. (2224)) and WebArena (Zhou et al. 2024) to evaluate our agent tasks. For the intermediate intent evaluation, we use the string-based existing functionality evaluators to assess the agent's 's response to the video-based question.
All tasks have a final evaluation function (i.e., evaluator) that determines an agent's reward on each task. The reward is typically binary, returning zero unity depending whether the agent performs the task unsuccessfully or successfully. Reward values are determined by evaluating the state of the environment at the end of the agent's 's trajectory determine if said state matches the correct state corresponding to the correct task exccution.
4 BASELINE AGENTS
We evaluate our benchmark using three different types of baseline agents with multimodal models as a backbone. Each type is distinguished by the type of video input provided to the model/agent. At each step, the agent given the task objective, in-context examples, current state 8, and the input video to the objective as context to generate one action. We describe each type in detail below.
4.1 VIDRO IN-CONTEXT AGENT
We define video input agent that takes the video in at every time step to generate actions. We provide the whole video in-context to the model with the Gemini model. The Gemini model automatically processes the audio, eliminating the need to process audio separately. The specific prompts use are in Appendix(E [ We use Set-of-Marks on the website HTML page, the Set-of-Marre element tree string, and the prompt along with the video input to the model.
7

---

Published as a conference paper at ICLR 2025
Vidao Surmany
Agart
Action
"click [40]"
Toe you OnT the de pr
Ge pertree iau RD teus hood pitti
Weippuee hel
Figure 4: VideoWebArena Baseline Agent Framework: We use 3 baseline agents: 1.) Vidco Summary Agent, where the vidco summary is in-context. Video Agent, where the
fed in-context. 2.) Video Frame Agent, where a number frames and audio transcription video is fed in an mov file in-context. The vidco information is put in-content along with the Set-of-Marks state representation to generate singular action, following the multimodal SoM agent in VisuaWWebArena (Kob et al. (024a).
4.2 VIDEO FRAMES IN-CONTEXT AGENT
We define video input agent that takes set amount of video frames every time step along with the video audio to generate actions. To obtain the information from the video, we follow the practice from (Wu et al., 2024b). We sample I frame per second (max 60 frames) for the video and include them into the context for the LLM. In addition, we use OpenAl's Whisper (Radford et al. 2022) to transcribe the audio and append it to the context. In this way, we can still pass the information from the video to our LLM. This may not be perfect method as the video information remains missing during framing sampling. However, since most LLMs on the market only support image and text input, it is essential to experiment with this setting. We use GPT-4o, and the prompt can be seen in Appendix E We again use Set-of-Marks on the website HTML page, the Set-of Marks- element tree string, and the prompt along with 60 video frames and audio transcriptions input to the model.
4.3 VIDEO SUMMARY IN-CONTEXT AGENT
We define a summary in-context agent that takes a video summary related to the objective at hand in-context every time step generate actions. To obtain this summary, call GPT-4o and feed the video using 60 frames and the Whisper transcription into the model and prompt it to summarize the video concerning the task hand. Again, our prompt can be seen in Appendix E Similarly, the summary agent also uses Set-of-Marks for the observation space and generates actions in the aforementioned action space.
5 RESULTS
5.1 MODEL PERFORMANCE
From Tables 6 7 and g see varying degrees of agentic performance across the video-capable Gemini and GPT family of models; however, we note several consistent trends across LLM agent results. We outline failure modes in Appendix B There is no best performing baseline agent or model across skill and factual retention tasks. For factual retention tasks, the summary agent performs the best in task success at 13.3% while the 30 and 100 Frame GPT-4o Agent perform the best in intermediate intent success at 45.8%. For skill retention, we see that long-context models with tutorials actually perform significantly worse than models without tutorials, suggesting that tutorials introduce negative noise that hurt action selection. Although intermediate scores tend to be higher than final scores, this did not translate to task success. This is a constant failure mode of the long- context agents, they can perform the necessary VQA to extract the necessary information for the task at hand but fall short due to hallucinations, action grounding, and high-level planning errors. For example, in Figure 5 the L.M agent successfully identifies the item to buy from the video. Still, it.
8

---

Published as a conference paper at ICLR 2025
Model
Task Domain
Final Score
Intermediate Score
Steps (Ang)
Classifids
6.7%
41.7%
17.1
Gitlab
5.7%
35.7%
18.5
Gemini Pro Video Agent
Reddit
Map
6.7%
3,4%
39.0%
18.2
9.9
Shopping (admin)
8,5%
48.9%
23.7
Shopping
10.0%
24.7%
21.6
Total
7.0%
37.0%
19.4
Classilieds
10.0%
40.0%
9.7
GPT4-u Summury Agent
Map
26.7%
66.7%
3.8
Reddit
11.5%
39.0%
13.8
Shopping (admin)
8.5%
29.1%
13.7
Shopping
15.7%
33.8%
14,3
Total
13.3%
Classifieds
18.3%
9.3
Gitlab
5.7%
50.0%
11.8
GPT4-o Frame Agent (30 Frames)
Map
26.7%
73.3%
4.7
Reddit
6.9%
42.5%
Shopping (admin)
8,5%
57.4%
16.8
Shopping
37.2%
Total
11.0%
45.8 %
14.0
Classifids
10.0%
30.0%
9.5
Gitlab
5.7%
55.7%
13.4
Map
26.7%
60.0%
3.5
GPT4-o Frame Agent (60 Frames)
Reddit
2.3%
44.8%
11.2
Shopping (admin)
4.3%
48.9%
13.6
Shopping
5.0%
38.0%
16.9
Total
6.0%
43.5%
13.0
Classifieds
13.3%
41.6%
7.64
7.1%
Map
20.0%
$3.39
3.8
GPT4-o Frame (100 Frames)
Reddit
5.7%
Shopping (admin)
8,55
51%
14.4
Shopping
10.7%
38.8%
16.4
Total
9.5%
45.8%
Classifieds
0.0%
Gitlab
2.9%
10.0%
6.9
Phi-3.5V (30 Frames)
Map
Reddit
0
0.0%%
26.7%
14.1%
16.9
7.5
Shopping (admin)
2.1%
12.8%
12.2
Shopping
0.08%
5.8%
8.6
Total
1.2%
13.0%
10.1
Classifidss
61.5%
69.2%
7.9
Gitlab
81.3%
81.3%
7.1
Human Performance
Map
69.2%
76.9%
4.8
Reddit
81.8%
86.4%
9.0
Shupping (admin)
Shopping
75.0%%
82.1
5.0
Total
73.99
6.4
Table 6: Results VideoWebArema Factual Retention Tasks. Performance of GPT4-o, Gemini Pro, Phi.3.5V, and human performance scores the overall performance successfully in its entirety), while intermediate scores measure the performance on the intermediate intents.
400 retention
by
completed
Model
WebArena Final Score
Steps
VisualWebAAena Final Score
Steps
GPT4-o (No Tutorial)
14.9%
19.8%
GPT4-o Summary Agent (Tutorial)
13.8%
13.9
11.6%
GPT4-o Frame Ageni (Tutorial)
9.9%
11.4
9.5%
12.5
Tutorial)
12.0
Human Perforance (Tutorial)
93.1
6.1
88.6%
8.2
Table 7: Results on WebAeea Skill Retention Tasks. Overall performance comparison GPT4-o human performance on skill retention tasks. Human performance shows tutorials should help task performance success and efficiency, However, adding tutorials context to the model does not necessarily help. but in fact hurts performance by a signiticant margin. See the failure modes in Appendix, Bor more analysis. Dashes (-) that unavailable
purticular metric.
does not successfully plan and complete the intent. We tested on a smaller subset of tasks with the GPT4-0 agent and tested the full set of tasks with the Gemini agent due compute constraints.

---

Published as a conference paper at ICLR 2025
Task Category
GP-T-4o Summary
GPT-4o (30 F Franes)
GPT-40 (60) Frames)
GPT-4o (100 Frames)
Gemini 1.5 Pro
Phi-3.5V
(30) Frames)
Perception Tosk Success
11.1%
0.8%
Audio Pereeption Task Success Rate
14.8%
18.1%
7.7%
12.5%
11.1%
1.1%
Full Video Undkrstnndnn Task Success Rate
15.5%
10.0%
7.2%
10.5%
6.5%
1.1%
Temporal Reasoning Task Success Rase
13.7%
12.4%
6.2%
10.4%
8.8%
0.5%
Agentic Fasy Success
9.0%%
0.0%
Agentic Medium Task Success Rate
14.2%
13.4%
5.7%
9.4%
7.7%
1.6%
Agentic Hard Task Success Rate
10.8%
8.1%
6.2%
9.15%
6.9%
0.7%
Perception Intemndiatee Success Rate
Audio Perception Intermediate Success Rate
50.0%
60.2%
62.8%
62.5%%
67.9%
23.5%
Full Video Unddrstanding Intermediate Success Rate
34.2%
40.0%
40.9%
41.2%
26.2%
7.3%
Temporal Reasoning Intermediate Success Rate
35.9%
50.5%
50.9%
50.05%
38.9%
8.0%
Video Easy Intermediate Suceess Rate
39.558
$2.9%
52.2%
53.2%
47.1%
17.355
Video Medium Intermediate Suceess Rate
89.458
46.2%
50.4%
48. 356
46.6%
9.0%
Video Hard Intenmediatie Success Rate
32.2%
42,4%
40.7%
41.0%
26.1%
5.3%
Table 8: Factual Retention Results Breakdown: Performance breakdown baseline agents across task categorics and difficultics the factual retention set The summary agent has the best task performance, even without having any visual aspect of the video in context. However, lags behind in intermediate VQA intents, as the frame and video agents all perform very similarly on intermediate tasks.
5.2 HUMAN PERFORMANCE
To understand the level of human performance expected on the tasks within VideoWA, three authors attempted the tasks and provide intermediate answers for a random sample of each unique task template in the factual retention set. Details on the human evaluation set can be found in Appendix A.3
They also tested 74 unique skill retention tasks, with each task having 2 separate humans attempt the task: one with tutorial and one without. They performed actions and recorded steps using the VideoWA action space, achieving a success rate of 73.9% on factual retention tasks while only taking an average of 6.4 steps per task (Table 6). Additionally, the intermediate intent and intent performance are linearly correlated while LLMS are not, citing a deficiency in the agentic abilities of these models. For skill retention tasks, human performance registers 93.1% on WebArena tasks and 88.6% in Visual WebArena with tutorials, and 82.6% and 72.7% without tutorials. Naturally, see drop in human performance and efficiency without tutorials. In terms of both task success rates and average number of steps taken, video-capable LLM agents lag behind significantly, further emphasizing the need improve agentic reasoning with video capacity today's 's models.
DISCUSSION
We present VideoWebArena. video-based agent benchmark that tests the video capability of long- context multimodal models through an agentic lens, consisting of 2,021 tasks that are all baded- along with 74 manually created videos. These tasks follow custom defined taxonomy of video-based tasks, providing a wide coverage of task types including skill retention and factual retention to create comprehensive test bed.
From our experiments, we see that introducing video can result in "noise" on skill-retention tasks, causing the agent fail at following action generating instructions. For instance, the agent would return try to click using the name of the element rather than the ID number. Another issue was where the agent would generate multi-action responses when the prompt explicitly says generate one action. From analyzing agent trajectories, we see that agents show inconsistencies in both their reasoming and planning, such as getting stuck in loop and repeating the same action/step even when the observed state or information does not change. We leave details to Appendix B We hypothesize that further improvements can result from better distillation of multimodal chain thought reasoning in the presence of an increasing number of modalities or having the agent learn which modalities are the most relevant for certain tasks over others.
Overall, baseline agents do not perform well on most tasks, falling far behind human performance. It clear that there is still a long ways ahead in developing intelligent agents capable of effectively utilizing and understanding video in completing regular tasks. As such, further analysis of failure modes from different video agent architectures on our benchmark is key. We hope our environment/ennchmar can help facilitate improvement and additional work on improving long-context multimodal agents.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
The authors are committed to making this work reproducible. Our code is will be open- sourced and available on Github. An anonymized version of our code base can be found at https//anonyymos..Apeer hlappp...ieenee/vvieeeeerre--eee
Our videos will also be made available through Google Drive and YouTube. Our data details are provided in Section 3 and our models and prompts are specified in Section 4.
ETHICS STATEMENT
Our benchmark is intended for safe and responsible innovations of video-based LLM agents. With the rising popularity of LLM agents and the excitement around their deployment, measures to ensure their safe practical deployment and use cases must be present. The authors are committed to the ethical development of LLM agents. For our paper, we did not use human subjects, find any potentially harmful insights, or any ethical concerns. Our benchmark is self-contained environment test agents on synthetic tasks.
We would also like to be cognizant of and make aware the possible biases that are introduced in the video tutorials that we have created. Our main focus is studying the ability of agents to make use of video information accomplish tasks in dynamic settings. As the issues of multi-modal video-capable agent performance and in-context video understanding pose huge obstacles, we made this the main focus of our video and benchmark creation. all the videos are English and created by graduate students in an American university, our videos may unintentionally contain possible linguistic and cultural biases. To help mitigate this, we plan on adding Jarean/Chinese/Portuge versions of the video dataset well. Given the significant time and labor required to implement new websites and the main focus of our paper, adding non-English websites unfortunately is beyond the scope of this paper.
We have, however, made it so that our benchmark and its videos can be easily accessed and spun-up by the open-source community to test/study potential biases. We have intentionally left room for updates and extensions to the dataset to incorporate additional underrepresented content or address gaps identified by the community. Our work and its documentation make it very easy for users and the open-source community to create their own tasks customized with their own videos, websites, etc. that reflect specific variations of interest that may be underrepresented here. Existing websites can be used to create new tasks. We plan on monitoring our benchmark's kss release and usage in the open-source community address mitigate possible biases as they arise or are made aware us through changes in our benchmark or otherwise.
REFERENCES
AI Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci. Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024. URL http:://axxxvv.o./abbbe 2489.08264
Wayne Chi, Ameet Talwalkar, and Chris Donahue. The impact of element ordering Im agent performance, 2024. URL http:/ttss::/aax..orrg/////
12089
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint 0rXiv:2403.077/8. 2024.
Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. Vila2: Vila augmented vila. arXiv preprint arXiv:2407.1745 2024.
11

---

Published as a conference paper at ICLR 2025
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Iims in video analysis. arXiv preprint arXhn:2405.21077 2024a.
Yao Fu, Dong-Ki Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of state-aware guidelines for large language model agents. arXiv preprint arXiv:2403.08978 2024b.
Hiroki Furuta, Kuang- Kueng-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finete In International Conference on Learning Representations (ICLR), 2024.
foundation models.
Google. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. 2023.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 2024 Annual Meeting of the Association for Computational Linguistics (ACL), 2024a
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, 2024b. /tt.hhttss///ariiv... wrw.oogg/////..ooo abs//4///7 01476
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, al. Autowebglm: Bootstrap and reinforce large language mosel-based web navigating agent. arXiv preprint arXiv:2404.03648 2024.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. Tvga: Localized, compositional video question answering. 2019. URL httppp://aaxi..or//aaaaaaa/
01696
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. -eumeetel- generation for www.wldge-inttnssiee nlp tasks, 2020.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311. 17005, 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modas video understanding benchmark, 2024. URL htttt:://axxxi http:///www....r//////31111/.... 17005
Ji Lin, Hongxu Yin, Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, arXiv preprint arXiv:2023b, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. 2023. /tt/htts:://aaxiv.ooo
abs/2308.00122
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In Advances Neural Information Processing Systems, volume 36, 2024.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023.
OpenAl. Hello
2024.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents, 2024. 00
06474
12

---

Published as a conference paper at ICLR 2025
Ajay Patel, Markus Hofinarcher, Claudiu Gilovennu-Condre Burch, and Sepp Hochreiter. Large language models can self-improve at web agent tasks. arXiv preprint arXiv:2405. 2024.
Dinu, Chris Callison-
Viorica Pätrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimezak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multi-modal video models. In Advances in Neural Information Processing Systems, 2023.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024.
Viorica Pätrăucean. Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimezak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models, 2023. URL httphhttts///xii http:/wwww.rgg////////// 13786
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. BRtphtss:://axxi org/ abs/2212.
Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Gold- stein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813 2024.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents, 2024.
Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, and Katerina Fragkiadaki. Ical: Continual learning of multimodal agents by transforming trajectories into actionable insights, 2024. URL httptttt/////iii wwwp/rr////aaeeee/ooo/eee 14596
Paloma Sodhi, SRK Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked Ilm policies for web actions. arXiv preprint arXiv:2310.03720v2 2024.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307. 16449, 2023.
Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song. Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang. Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, and Chenliang Xu. Video understanding with large language models: A survey, 2024. URL
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through reestion-answerigg 2016. URL
httpss///arii..or/bbb///S/ 02902
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004. doi: 10.1110/T1122200.8889999 819861.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang. Xiaojian Ma, and Yitao Liang. Jarvis-I: Open-world multi-task agents with memory-augmennted multimodal language models, 2023.
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory, 2024.
URL htt.:httss///axivoorrraaaaa////s
13

---

Published as a conference paper at ICLR 2025
Haoning Wu, Dongxu Li, Bei Chen, and Jumnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754. 2024a.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interlcaved video-language understanding, 2024b. URL
Tongtong Wu, Linhao Luo, Yuan Yuan-Fang Li, Shirui Pan, Stay-Trang Vu, and Gholamreza Haffari.
Continual learning for large language models: A survey, 2024c.
Junbin Xiao, Xindi Shang. Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9777-9786, 2021a.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase 08276
phase of question-answerrigg httptttpss//aaaiii.o///////////s
to explaining temporal actions, 2021
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972 2024.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan Hc, and Yueting Zhuang, Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia, pp. 1645-1653, 2017.
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-contex. visual language models for long videos. 2024.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Sct-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, and Hongxu Yin. X-vila: codaaity alignment for large language model. CoRR, abs/2405. 193335 2024.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qar
A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understanding. arXiv preprint arXiv:2312.0485 02.1177, 2023.
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: www.oo-----iooouuuht-heee for gui agents. arXiv preprint arXiv:2403.027133 2024a.
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. survey on the memory mechanism of large language model based agents, 2024b.
Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal internet agents. arXiv preprint arXiv:2404.0999 09992, 2024c.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded, 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference Learning Representations (ICLR), 2024.
14

---

Published as a conference paper at ICLR 2025
A VIDEOWEBARENA DATA DETAILS
A.I VIDEO CREATION DETAILS
400 New Agent Tasks and Corresponding Video- Based QA
1641 Tasks WebArena
74 Video Tutorials All Tasks Types
WebArena
Agent Tasks with Tutorials
Figure Dataset Creation Process walkthrough of the VedooWehArems dataset creation. From 1641 existing tasks in WehArena and VisunlWebArena, the authors grouped these tasks by their intent templates. For each inteat template, the authors created new video tutorial showing to perform lasks. each video, the authors made minimum retention 400 factual retention tasks.
skill
Three authors of the paper created 74 original video tutorials with audio narrating the actions taken. The three authors evenly divided the vidcos based off the site the video was based on. We based these 74 tutorials off of 1641 intents in WebArena and VisualWebArena. Each of these tasks mapped to a video, creating a skill retention task. Each video creator manually checked all of the tasks that they
to create tutorial for before, then made the functionality of the task was shown in the tutorial. We post these videos on YouTube at hhtpps://ww. youtuube com/ @webarenawarrior We also provide them online Google Drive link: https://drive google httpe///////w//////////yzyz/ htwp://wwwwwwit.t/aaaaaaaatto/tn
A.2 TASK CREATION DETAILS
Every video is mapped to minimum one AeeAAra WebArena task, creating skill retention tasks. The authors of each video were also tasked with creating a minimum of four factual retention tasks per video, with one task type each from the factual retention taxonomy. The taxonomy can be seen in Figure 6 The authors of each task also are tasked with creating intermediate questions for the factual retention tasks that test the model can extract the information necessary to complete the task. The authors also create evaluation functions for the intermediate intent and task. Once created, a second author verified and completed each task for quality assurance purposes.
Video-Based Agent Tasks
Factual Retention
Skill Retention
Visual Perception
Audio Perception
Full Video Understanding
Temporal Reasoning
Figure WehAere Task Taxonomy define taxonomy for tasks beachmark. namely splitting them into a factual and skill retention groups. Under the factual retention group. there are types tasks: Visual Perception. Audio Perception. Full Video Understanding, and Temporal Reasoming.
15

---

Published as a conference paper at ICLR 2025
Evaluator Functions
Reward Condition
paactnatch(a, A)
1 if exactly A.
must. inclldee, 6)
if in the set
fuzzy match( a. a)
are if is not in the set
semantically equal by an I.I.M.
must.en exclude(e (a. à)
waa. (img, question, à)
the output MoA.Modek,ng, question) contains
fuzzy inage nanch img)
SSIM [Wang et 2004 between img, img higher given threshold.
Table 9: List of VideeWebAAree evaluntor functions and descriptions: All rewards are binary. We adopt our evaluntors from WebArena
Zhnna 2024 and Visu//WehArema (Kah 2024
A.3 HUMAN EVALUATION DETAILS
We conducted two sets of human evaluation, one for the skill retention and one for the factual retention tasks. For the factual retention tasks, author who did not make the task was given the video, along with the intermediate intent and task intent. Each author was given the agent action space and recorded their number of actions defined by the agent action space. The answers to the intermediate and task intent were then verified by another author. We did human evaluation on a subset of the factual retention tasks, simply taking random sample of each unique intent template. We had 11 111 factual retention tasks for human evaluation. For the skill retention tasks, two authors who did make the original tutorial were tasked with completing skill retention task. One author was given the video tutorial before and the other author was not. They then recorded their action steps and completed the task, which was then evaluated and verified by a third author. We did skill retention human evaluation on singular task per tutorial, totaling to 74 human evaluation skill retention tasks. Given the extremely high success rate for both types of tasks, many of the failures came from human carelessness or interpretation mistakes.
A.4 ENVIRONMENT DETAILS
We provide table of the VebeoWebArena reward functions in Table These are adapted from VisualWebArena (Koh et al. 2024a).
B FAILURE MODES
B. COMMON LLM AGENT FAILURE MODES
Many of the basic failures captured in the baseline agents were common repeats of agent errors seen in other agent academic benchmarks. These include ballucinations. where the agent produces a nonsensical action unrelated to its context or task at hand. We attribute this to the lack of instruction tuning and model alignment on agentic tasks. Another common failure mode displayed in the baseline agents was failure to do visual grounding. The agents will recognize the correct plan of action, but choose the wrong element with respect to the Set-of-Marks image input and take the wrong action.
Action grounding and planning was also a common failure mode of the baseline agents. An agent can simply generate the wrong plan action that will yield unsuccessful trajectories, and not change this plan even with negative feedback from the environment. This suggests using inference time search or memory based methods inference can also help the agents recover from failures in action grounding and planning. The lack of self-refiection is especially
be effective to combat these failures. Incorporating self-reflection during
when the agent generates the same action repetitively, leading the task to terminate. Even though an action is shown to be unsuccessful towards completing a task, an agent will continue repeatedly attempt the same action to try and complete the task.
B.2 LONG-CONTEXT SPECIFIC FAILURE MODES
Within our skill and factual retention tasks, there were many failure modes that presented issues relevant long-conter modeling. One constant issue noticed failure adhere to the prompt instructions for generating actions. With the extra noise provided with the video information in-context, the agent did not always adhere the action generation guidelines provided in the prompt.
16

---

Published as a conference paper at ICLR 2025
For example, under the Set-of-Ne Markks elements, click action must be generated using click [elem] where elem is the numeric ID the SoM element. However, the agent would return click [elem] where elem was the name of the element. This formatting issue persisted for other actions with the longer prompt.
A common issue for skill retention tasks the agent began generating multi-action responses when the prompt explicitly says to generate one action. Given the tutorial or summary to complete a similar task, the agent would get distracted by the comprehensive plan and generate multiple actions from the video information, straying away from the prompt guidelines. This led to failure to complete tasks.
could pinpoint that the ardeo-frame and summary agents would simply miss visual information due to the nature of their video processing. Additionally, the video agent also showcased many of these video grounding errors. For example, a common task was to Take me to the page in the video when event happened. However, the frames or summary did include this page, there was no way for the agent to get to this page know about its existence. This issue was exacerbated in tasks that required full video understanding or temporal reasoning across video. This setup we proposed. Many of the audio tasks were completed at a much higher rate than the video perception tasks, citing that video grounding is a larger issue than audio grounding when processing these modalities within videos. We encourage better video understanding agent systems with our benchmark.
common issue for factual retention tasks
video grounding. Specifically,
flaw in baseline agent
C TASK CREATION DOCUMENTATION
C. EXAMPLE TASK JSON
We provide an example task below in their JSON file format.
"sites": ; "classifieds"
1. "task_id": 4,
"require_login". true,
estorage_state": www.ut/hlassiiiss.s.s... json", "start_url": "__CCASSIFIEDS__." "geolocation": null, "intent_template"" "Can you take me to [description]?? "intent": "Can you take me the cheapest primary red boat for sale in the video?",
"intermediate_intent" "What was the name of the post for the cheapest primary red boat for sale in the video?", "video":
" "description": "the cheapest primary red boat for sale in the video"
}.
"require_reser resett: false, "intermediate_eval" " "eval_types": [ "string_match"
J. "eeereecceansserss "must_ must_include"" "1986 nova x1 23ft"
"reference_url". "" " "program_htme html". [],
17

---

Published as a conference paper at ICLR 2025
"string_note".
""
evaal: {
"eval_types" [ "url_match"
].
"reference_anseers". null,
" "reference_url", "__CCAASSFFEEDS__ /index, happppae=ittmmii-
9894",
"program_hi html": [], "url_note": "EXACT"
iintermediateddiffiicllll "hard",
"overall_difficcuttty.. "medium",
"visual__errception" true,
"audio__errceptionn: false,
Fuul_viddo_understandnnn true, "temporrl__eesooning false, "comments":
C.2 TASK CREATION DOCUMENTATION
We used the following instructions task creators. These can be used to create new tasks. Addition- ally, you can use any video as input to your LLM agent our evaluation suite simply chooses from own 74 manually made tutorials.
In order to create a task, choose or make video of your own to reference to order to create task. Each task should be hideo-grounded such that the task can only be completed after watching the video. For each new task, you need to fill the following values in json task file:
1. Sites: The site(s) used in the task.
2. Start URL: The url the task starts on.
3. Video: The video name associated with the task.
4. Task ID: A unique ID for the task.
5. Intent: The prompt for the video-based task.
6. Intent Template: The task template specific the intent with default parameters.
7. Intermediate Eval: Evaluates if your Vision- Rision-Language Model (VLM) extract the necessary information from the vidco through a task-based VQA. More documentation on existing evaluators can be found in the evaluators.py file in repository.
8. Eval: The evaluation suite to test whether the agentic task was completed successfully. More documentation on existing evaluators can found in the evaluators.pp py file in our repository.
9. Intermediate Difficulty: The difficulty level of the intermediate VQA intent, distributed between easy, medium, and hard. The video difficulty ratings signify the complexity of returning the correct answer for a given task's s intermediate intent. Easy tasks require returning one piece of information and can be solved with less than 3 frames, medium tasks require returning 2 to things and can be solved with less than half the video, and hard tasks require returning more than 3 things and require watching more than half the video.
10. Overall Difficulty: The overall difficulty level of the task, distributed between easy, medium, and hard. The agentic difficulty for each task signifies the complexity of the action sequence needed to complete intent successfully. For agentic difficulty, classify task easy if can be completed 1-3 steps, medium it can be completed in 4-9 steps, and hard if can be completed in more than 9 steps
18

---

Published as a conference paper at ICLR 2025
11. Visual Perception: Whether visual perception of the video input is required. True False.
12. Audio Perception: Whether audio reasoning of the video input is required. True False.
13. Full Video Understanding: Whether full video comprehension of the video input is necessary. True or False
14. Temporal Reasoning: Whether reasoning across different time points in the video input is required. True False.
D RESULTS
D.1 ADDITIONAL RESULTS
We provide another result breakdown plot at Table 10
This shows the average steps per task type.
Table 10: Model Comparison Average Steps per Task Type
GPT-4o Summary
GPT-4o (30 Frames)
GPT-4o (60 Frames)
GPT-4o (100 Frames)
Category
Gemini 1.5 Pro
Visual Perception
12.9
14.5
13.7
13.2
19.7
Audio Perception
10.5
10.5
10.2
10.5
17.0
Full Video Understanding
14.4
13.6
13.0
20.3
Temporal Reasoning
14.9
14.6
13.9
14.5
20.1
Agentic Easy
12.6
10.1
10.1
10.2
20.6
Agentic Medium
11.6
14.7
13.5
12.5
19.2
Agentic Hard
14.3
15.6
14.9
15.4
19.4
Video Easy
13.4
11.9
13.4
19.5
Video Medium
13.1
13.2
12.9
11.8
19.5
Video Hard
13.2
15.5
14.6
13.8
19.6
(a)
(b)
(c)
(d)
Figure 7: Abridged Example VideeWebArenn Task. A stylized example of task VedeoWehArde starting from (a) to (b), the task defined, an agent interacis along its trajectory for the task hefore concluding (incorrecly) in (d), where it receives final reward of zemo for failing to complete the task correctly.
inpul create a plan perform actions.
continues actions and planning
19

---

Published as a conference paper at ICLR 2025
E AGENT PROMPTS
E VIDEO AGENT TASK PROMPT
You sutonomous intelligent agent tasked with navigating browser. will be given web-based tasks that be done based on information in video. These tasks will be accomplished through the use of specifiic actions you can issue.
Here's 's the information you'll 'll have:
The user's 's objective: This is the lask you re trying to complete.
A video tutorial ahout this task similar task will he provided to assist you.
3. The current web page's 's accessibility tree: This is simplified representation of the webpage, providing key information.
The
web page's URL: This
page you'se ro currently navigating.
5. The open tabs: These are the tabs you have open.
The previous action:
action you just performed. may be helpful track your progress.
actions you can perform fall several categories:
Page Operation Actions
cllik [id] "; This action clicks on an element with a specific id on the webpage.
1* "type [id] [content] ""; Use this type the content into the field with By default, the "Enter" key pressed after typing unless pross.entrr_fferr is set to O, i.e.. "" 'ypp [id] [content] [0]
** 'bover lid]" ": Hover over element with
""presss [eoo]] ":; Simulates the pressing of key combination on the keyboard (e.g. Cirl+w).
*** "scroll [down]" 1* or 1* serolll [up] "; Scroll the page up or down.
Tab Management Actions
new tab ""; Open a new, empty browser tab
"" "tab_focus [tab_index] " Switch the browser's focus specific using its index.
"closeiab"" *": Close the currently active tab
URL Navigation Actions
goto [url] "*; Navigate to a specific URT...
** "go.back" " Navigate to the proviously viewed page.
** 20 forward" *1: Navigate the next page (if previous 'go back' action performed).
Completion Action
stop [answer] ": Issue this action when you beliove task complete. If objective to find text-based answer, provide the
amswer in the bracket.
Homepuge:
you want visit other websites, check out the homepage at http://homepage.ce com. has list of websites you can visit. http://hmmppgg.co commpassword..bbiee hinl lists all the account name and password for the websites. You can use them to log in to the websites.
To be successful, very important follow the following nales:
1. You should only issuc an action that is valid given the current observation.
You should only issue one action time.
3. You should follow the examples to reason slep by step and then issue the next action.
Generate the action
correct format. Start with "In summary, the action will perform is" phrase, followed by action inside
For example, "In summary, the next action I will perform is "eliik [1234]* "*""
5. Issuc stop action when you think you have achioved the objective. Don't 't gencrate anything after stop.
Example 1:
VIDEO: Shopping Tutorial MOV File
OBSERVATION: [28) [A] [Pre-baked Gingorbread House Kit Value Pack, 17 oz., Pack of 2, Total 34 oz.] [] [Static [19 99] [30] [34] [A] [V8 +Energy. Healthy Energy Drink, Steady Energy from Black and Green Tea, Pomegranate Blueberry, 8 Ounce Can, Pack of 24] [1 [StaticTex]] [14.47] [36] [BUTTON] [Add to Cari] [40] [A] [Elmwood Inn Fine Teas, Orange Vanilla Caffeine-fres Fruit Infusion, 16-Ounce Pouch] [] [StaticTxx]] [19.36] [42] [BUTTON] [Add to Cart] [43] [A] [Add Wish Liss]
[BUTTON| [Add
URL: http://onsstommarrt..cc
OBJECTIVE: Can you take me to the product page for the first item added to the cart in the video?
PREVIOUS ACTION: None
OUTPUT: Let's think stop-by-step. This page the home page for the One Stop Market and includes the first item added to the cart in video, which Elmwood Ian Hine Teas, Orange Vanilla Caffeine- free Fruit Infusion. 16- 16-Ounce Pouch. video tutorial, shows that clicking the title of the listing will take me the item page. Therefore, I will issua the click action with the item listing. In summary, next action will perform is ** "click [40]"
Examplo
VIDEO: Reddit Tutorial MOV File
OBSERVATION: [StaticText]] [Forums)] [1] [A] [Forums] [9] [A] [Alphabetical] [I (StaticText) [allentown] [10] [A] [allentown] II
[StaticText] [baltimore] [16] [A] [baltimore] (] [Static Text] [books] [17) [A] [books] (1 [StaticText] [boston] [18] [A] [boston]
[StaticText] [MachineLoarring (52) [A] [MachineL.co Larming) [I [Static [pittsburgh] (78) [A] [pittsbrrh]] II [StaticText] [sechnology]
[90] |A] [echhology I| |StaticTest Text [television] 191| |A] [television] II [Static [Running Postmill| [105] ||A] [Postmill]
URL: http://rediiiccom
ORJECTIVE: Can you take me post video the user commented on?
PREVIOUS ACTION: None
OUTPUT: Let's 's' think step-by-step. This page lists all the forums on the website, In the video, the post that the user left comment under the sttchnology forum, so should navigate that. navigate that forum by first clicking on the otechnology link. Therefore, I will issue the click action. In summary, the next action I will perform is click (90]
20

---

Published as a conference paper at ICLR 2025
E.2 VIDEO FRAME AGENT TASK PROMPT
You autonomous intelligent agent tasked with navigating browser. will be given web-based tasks that be done based on information in vidco. These tasks will be accomplished through the use of spocific actions you can issue.
Here's 's the information you'll 'l have:
The user's 's objective: This is the task you re trying to complete.
A video tutorial about this task similar task will be provided to assist you.
3. The current web page's 's accessibility tree: This is simplified representation of the wehpage, prowiding key information.
The
web page's URL: This
page you'se 'e currently navigating.
5. The open tabs: These are the tabs you have open.
The provious action: This the action you just performed. may be helpful to track your progress.
The actions you can perform fall into several catogories:
Page Operation Actions
click [id] " This action clicks on an element with a specific id on the webpuge.
1* "typpe [id] [content] ""; Use this type the content into the field with By default, the "Enter" key pressed after typing unless press www.enttrrrro after is set to O, I.e.." " "ttpp [id] [content] [0]
** "bover lid]" ": Hover over element with
pprrsss (key_comby _comb] ""; Simulates the pressing of key combination on the keyboard (e.g.. Ctrl+v).
"scroll Idown]" or ""scroll lup]" ": Scroll the page up or down.
Tab Manugemeent Actions
"Jb "": Open a new, empty hrowser tab
"" "tabufocus [tab [tab_index] ""; Switch the browser's 's focus specific using its index.
"closc tab" "*: Close the curreatly active tab
URL Navigation Actions
goto [url]" " Navigate to a specific URL.
"go.buck" ": Navigute to the previously viewed puge.
go. forwaor " Navigate the next page (if previous 'go back' action performed).
Completion Action
stop [answer] **; Issuc this action when you believe the task complete. If objective to find text-based answer, provide the
answer in the bracket.
Homeppuge:
you want to visit other websites, check out the homepage at http://homppage.oo com. It has list of websites you can visit. https/homeppgg.coom wwt.ge..o//ooeeoeeoe wwwmp//wwrrdd.biii himl lists all the account name and password for the websites. You can them log in the websites.
To be successful, very important follow the following nales:
1. You should only issue an action that is valid given the current observation.
You should only issuc one action time.
3. You should follow the examples to reason step by step and then issue the next action.
Generate the action
correct format. Start with "In summary, the next action will perform is" phrase, followed by action inside
For example. "In summary, the next action I will perforn is "click [1234]"*
"." "*"
5. Issue slop action when you think you have achieved the objective. Don't generate unything after stop.
Example
VIDEO FRAMES:: 5 Frumes from Shopping Tutorial
AUDIO: Hi everyone, welcome to a tutorial on the One Stop Market. Today this is just a general tutorial video and how to get around on things. So one thing you need get to item simply click
title image. And you see here going to the title. From here I can edit the quantity. add the cart, add to my wish list, add to my comparisons, and I can access through views here. can see views,
also leave my own review at the bottom here. And then I want add items to my cart and just click add to cart, I want to add my wish list I can click the red beart button. Similarly I add a cart, sometimes you are going to gel prompted with an option to add items details before add the cart. Other times is going to he option like here, add your comparison page here well. And so If want go different sections can go here, let's go to Xbox One, let's go and have a look. There are also subsections within categorios, so these are also categorics, so accessories is arrow, says my cart, let's get it in black, and then similarry you can go my carl here. I'm going to go to view it un cant and we can see that our cart is here and want to go
Similarly I go this will very similar,
one has 12 views
category. Let's the most expensive item. And you can do by sorting my price and then fipping
helps OBSERVATION: [28] | [A] [Pre buked Gingerbread House Kit Value Pack. 17 oz., Pack of 2, Total 34 oz.] [StaticTexi] [19 [19,99] [30] [BUTTON] [Add to Cari] [34] [A] [V8 +Fnergy, Healthy Energy Drink, Stendy Energy from Black and Green Tea, Pomegranate Blueherry, Ounce Can, Pack of 24] [StaticTexr]] [14.47] [36] [BUTTON] [Add to Carl] [40] [A] [Elnwood Inn Fine Teas, Orange Vanilla Caffeinc-free Fruit Infusion. 16-Ounee Pouch] [I [Static Text] [19.36] [42) [BUTTON] [Add to Cart] [43] [A] [Add to Wish List]
to the One Stop Market
how things go. So hope
thanks for watching.
URL: http://onestopmarrt....
OBJECTIVE: Can you take me to the product page for the first item added to the cart in the video?
PREVIOUS ACTION: None
OUTPUT: Let's think step step..p. This page the home page for the One Stop Market and includes the first item adided the cart in the video, which was Elmwood Inn Fine Teas, Orange Vanilla Caffeine-free free Fruit Infusion, 16- Ounce Pouch. In the video tutorial, title the listing will take the item pago. Therefore, will issue click action with the item listing. In
shows that clicking summary, the next action I will perform is "click [40]
Example 2:
VIDEO FRAMES: 5 Frames from a Reddit Tutorial AUDIO: I wanted to make a quick tutorial on how to use the reddit site. So let's say I wanted to make response to comment under one the top posts under the rtechnology forum. So can click under this forums link here. Scroll down technology, And let's i's say I wanted view its comments so can click here. And then look all those comments. And I can that this is the top comment here. And let's say I wanted to reply great comment. So can get a quick preview. can post it, And then now it shows me that I have successfully made comment under this single comment. Right.
21

---

Published as a conference paper at ICLR 2025
OBSERVATION; [StaticTest] [Forumss] [1][A]] [A] [Forumss] [9] [A] [Alphubetical] [StaticTesl [allentown| [10] [A] [allentown] II
[StaticTex]] [baltimore] [16] [A] [baltimore]] [] [StaricTexi] [books] [17) [A] [books] [] [StaticTxx]] [baston] [18] [A] [baston] []
[StaticText] [Machinel_eamiin [52] [A] [Machinel corning] [] [StaricTent [pittsburgh] [78] [A] [pittsburg]] [StaticText] [technology]
[90] [A] [technology] [] [StaticTexd [television] [91] [A] [television] [StaticTex] [Running Postmill) [105] [A] [Postmill]
URL: http//teedit.com
OBJECTIVE: Can you take
post the video the commented on?
PREVIOUS ACTION: None
video, the post that the user comment was under the rtechnology forum, so I should navigute to that. I can navigate to that forum by first clicking on the otechnology link. Therefore, I will issue click action. In summary, the next action I will perform is 1 "click [90]
OUTPUT: Let's 's think step sles.ep. This page all forums on the wehsite,
E.3 VIDEO SUMMARY AGENT TASK PROMPT
You are an autonomous intelligent agent tasked with navigating weh hrowser. You will he given web- hased tasks that can he done based information vidco. These tasks will be accomplished through the of specific actions you can issue.
Here's 's information you'll 'Illll have:
1. The user's objective: This is the task you'te 'e trying to complete.
A summary from a tutorial for a similar task: This provides useful information for solving this task.
The current web page's accessibility tree: This simplified representation of the webpage, providing key information.
The current weh page's URL: This is the page you's 'e currently navigating.
The open rabs: These the tabs you have open.
6. The provious action: This is the action you just porformed. may be helpful to track your progress.
The actions you can perform fall into several categories:
Page Operation Actions
** elick [id] "h This action clicks
element with a specific id on the webpage. ""ype [id] [content] ":: Use this to type the content into the field with id. By defiultt, the "Einter key is presseed after lyping unless press_enter_affee is
ie.." "" "tyee [id] [content]
""ovver [id]] " Hover over an element with id.
1* "pooss (key ccom]" Simulates pressing key combination on the keyboard (e.g.. Ctrl+v),
".presss
"scroll [down]"" " or "scroll [up] " Scroll the page up or down.
Tab Management Actions
"Dew Jaa '": Open new, empty browser tab
"tab_focus [tab_index]] ": Switch the browser's locus a specific tab using its index.
"close_tah" >": Close the currenlly active tah.
URL Navigation Actions
"goto [url] Navigate to a specific URL.
** go back * Navigate to the previously viewed page.
"""goback""
"go_forward" ": Navigate the next page (if a previous "go.back" action was performed).
Completion Action
stop [answer]] "t Issue this action when you believe the task is complete. If the objective is to find a text hased answer, provide the
answer the bracket.
Homepage:
If you want to visit other websites, check out the homepage at http://hmmppagecco com. It has list of websites you can visit. http http://hmmppgg..o com/password httil lists all the account name and password for the websites. You can use them to log in to the websites.
To be successful, is very important to follow the following rules:
You should only issue action that is valid given the current observation.
You should only issue one action at a time.
You should follow the examples reason step by step and then issue the action.
4. Generate the action in the correct format. Start with a "In summary, the next action I will perform is" phrase, followed by action inside
..., For example, summary, the next action
perform " ""click [1234]" -
5. Issuc stop action when you think you have achieved the objective. Don't 't' genorate anything after stop.
Example 1:
VIDEO SUMMARY: The tutorial explains how to navigate the One Stop Market website and manage items. To view an item, simply click on its tidle image, which takes you page where you can adjust the quantity, add it your cart, wish list, compurison list, and leave a review. Some items may prompt you to provide details hefore adding them to your cart. You can also browse different sections, like Xbox One Accessories,
soct items price. After selecting item and adding your cart, you
your
and return to the main marketplace. The vidco concludes with a note of thanks.
Pack. oz.. Pack 2, Total 34 oz. || [StaticText]] [19.99] [30] [BUTTON] [Add to Cart] [34] [A] [V8 +Enorgy. Healthy Energy Drink, Steady Energy from Black and Green Tea, Pomegranate Blueberry, Ounce Can, Pack of 24] II [exttic/ext]] [14.47| [36] [BUTTON] Iddd to Carl [40] [A] [Ellwood Inn Fine Teas. Orange Vanilla Caffeine free Fruit Infusion, 16 Ounce Pouch] [StaticTes]] [19.36] [42] [BUTTON] [Add to Cart] [43] [A] [Add Wish List]
OBSERVATION: [28] |A| |Pre- baked Gingerbread House Kit
URI.: http://onestopmrrkee
OBJECTIVE: Can you take
product page for the first item added cart the video?
PREVIOUS ACTION: None
OUTPLT: Let's think step step..p. This puge the home page the One Stop Market includes first stem adided cart in the video, which was Elnwood lan Fine Teas, Orange Vanilla Caffeine-free Fruit Infusion. 16-Ounce Pouch. In the video tutorial, shows that clicking on the title of the listing will take me the item page. Therefore, I will issue the click action with the item listing. In summary, the next action will perform is * cciik [40]
Example
22

---

Published as a conference paper at ICLR 2025
VIDEO SUMMARY: The tutorial explains how leave commentt on a Reddit post, start by logging into your account. Navigute to subreddit of your choice, such as thtchnology, either by searching or selecting it from your subserihed subreddits. Once there, select post you want comment on, and scroll down view existing comments. you wish to comment on the post itself, scroll to the bottom where you'll Il find an Add a comment box. To roply specific comment, click the Reply button under that comment. After typing your comment, you can preview it by clicking the Preview button il you d like to see how it will look. When you'se re ready, click Post to submit the comment. Your comment should uppear immediately beneath the post the specific comment you replied to.
OBSERVATION: [StaticText] [Forums] [1] [A] [Forums] [9] [A] [Alphabetical]] [] (StaticText] [allentown] [10] [A] [allentown] D [StaticText] [baltinnore] [16] [A] [baltimoree] II [StaticTexi] [books] [17] [A] [books] [I [SuticTexx]] [baston] [18] [A] [buston] [] [StaticTexi] [Muchinel caminn] [52] [A] [Machinel earning] [] [Staric [pittsburgh] [78] [A] [pinsburgh]] [StaticText] [technology]
[90] [A] [technology] [] [StaticText [television] [91] [A] [telovision] O1 [StaticTex]] [Running Postmill] [105] [A] [Postmill]
URL: hup./teddit..oom com
OBJECTIVE; Can you lake me to the post in the video the user commentted on?
PREVIOUS ACTION: None
OUTPUT: Let's s think step-by-step. This page lists all the forums on the website. In the video, the post that the user left comment was the r/tchhology forum, so I should navigate that. navigate forum by clicking on the ntechnology link. Therefore, I will issue the click action. In summary, the next action I will perform is "" "click [90]"*)
E.4 VIDEO AGENT INTERMEDIATE TASK PROMPT
You autonomous intelligent agent that extracts information from videos. You will be given this video and question. You need to answer the question based on the video provided.
Example I:
VIDEO: Shopping Tutorial MOV File
QUESTION: What is the first item that gets added to the cart on the One Stop Market in the video??
ANSWER: Elmwood Inn Fine Teas, Orange Vanilla Caffeine-free Fruit Infusion, 16-Ounce Pouch
Example 2:
VIDEO: Reddit Tutorial MOV File
QUESTION: What the name of the author of the post that the person in the video commented on?
ANSWER: Sorin61
E.5
VIDEO FRAME AGENT INTERMEDIATE TASK PROMPT
You autonomous intelligent agent that extracts information from videos. You will given a list frames sampled from a video and its audio transcription. You need to answer the question based on the video provided.
Example I:
VIDEO FRAMES: 5 Frames from Shopping Tutorial
AUDIO: Hi everyone, welcome to a tutorial on the One Stop Market, Today this is just a general tutorial video and how to get around on things. So one thing you need to get to item simply click
to title image. And you can see here going to the litle. From here I can edit the quantity. add the cari, add to my wish list, add to my compurisons, and
Uhrough views here. Similarly I go to this will he very similar, this one has 12 views so can see 12 views, I can also leave my own review at the bottom here. And then I want add items to my and just click add to cart, I want add my wish list click the red beart button. Similarly I add a cart, sometimes you are going to get prompted with an option to add items details before add the cart. Other times it is so want go different sections can gu here,
your comparison page here as let's go to Xbox One, let's go and have a look. There are also subsections within categories, these are also categories, accessories is also entegory. Let's the most expensive item. And you can do this by sorting my price and then flipping the arrow, says to my that our cart here and I want to go back to the One Stop Market this how things go. So 1 hope this helps and thanks for watching. QUESTION: What the first item gets added ANSWER: Elmwood Ian Fine Teus, Orange Vanilla Caffeine-free Fruit Infusion. 16-Ounce Pouch
not going to an option
here,
cart, let's get black, and then similarly you go my carl here. I'm going go view it cart and
cart the One Stop Market the video?
Example 2:
VIDEO FRAMES: 5 Frames from Reddit Tutorial
AUDIO: I wanted to make a quick tutorial on how to use the reddit site. So let's say I wanted to make response to comment under one of the top posts under the riechnology forum. So can click under this forums link here. Scroll down to technology, And let's is's say I wanted view its comments so can click here. And then look all those comments. And can that this is the top comment here. And let's say I wanted to reply great comment. So can get a quick preview. can post it, And then now it shows me that I have succesfully made comment under this single comment. Right.
QUESTION: What is the name of the author of the post that the person in the video commented on? ANSWER: S
E.6 VIDEO SUMMARY AGENT INTERMEDIATE TASK PROMPT
You are an autonomous intelligent agent that extracts information from summaries. You will be given summary of a video and question about the video. You need to answer the question based on the summary provided.
Example
VIDBO SUMMARY: Let's think step-by-step. To add an item to your cart on the One-Start Market, first navigate to the website and browse search for the desired item. You can add an item directly by clicking the blue 'Aid Cart' button next which updates the cart icon to reflect the addition. Altematively, click on the item listing to access its detailed page, where you can select options like size or color and adjust the quantity before adding the cart. For example, you can select the size and add cight flannel shirts for your family.
23

---

Published as a conference paper at ICLR 2025
View your cart by clicking cart emblemn/con, which provides access all added items and options to proceed to checkout. QUESTION: What is the first item that gets added to the cart on the One Stop Market in the video? ANSWER: Elmwooo Ion Fine Teas, Orange Vanilla Caffeine-free Fruit Infusion, 16-Ounce Pouch
Example 2:
VIDEO SUMMARY: Let's think step-by-stop. To leave comment Reddit post, start by logeing into your account. Navigate to the subreddit of your choice, such as stechoology, either by searching or selecting it from your subscribed subreddits. Once there, select a pod you want comment on, and scroll down view existing comments. If you wish to comment the post isself, scroll to the botton where you'll 'Il find an Add a comment hox. To reply a specific comment, click the Reply button under that comment. After typing your cumment. you can preview by clicking the Preview button il you like the comment. Your comment should uppear immediately beneath the post or the specific comment you replied to. QUESTION: What the name the author the post that the person in the video commented on?
how will look. When you'se re ready, click
submit
ANSWER: Sorin61
E.7
VIDEO SUMMARIZATION PROMPT
You are an autonomous intelligent agent tasked with learning from vidco to accomplish a task. You will be given video. You will be given a task to complete. You will need to extract useful information to accomplish the task.
Example
VIDEO: Shopping Tutorial MOV File
ORJECTIVE: Add item cart One Stop Market. SUMMARY: Let's think step-by-step. To add an item to your cart on the One Stop Market, first navigate to the website and browse or search for the desired item. You can add an item directly by clicking the blue 'Add to Cart' button next to it, which updates the cart loon reflect the addition. Altematively, click the item listing access its detailed page, where you can select options like size color and adjust the quantity before adding it to the cart. For example, you can solect the size and add eight flannel shirts for your family. View edit your cart by clicking the cart emblem/icon, which provides access all added items and options to proceed to checkout.
Example
VIDEO: Reddit Tutorial MOV File
ORJECTIVE: Leave comment Presmiil post.
SUMMARY: Let's think stop-by-step. To leave comment Reddit post, start by logging into your account. Navigate to the subreddit of your choice, such as thtcchology. either by scarching or selecting it from your subscribed subreddits. Once there, select a post you comment on, and scroll down to view existing comments. If you wish to comment the post itself, scroll to the bottom where you'll 'Il find an Add a comment box. To roply to a specific comment, click the Roply button under that comment. After typing your cumment. you can preview by clicking the Preview button if you the comment. Your comment should appear immediately beneath the post or the specific comment you replied to.
bow will look. When you're 'e ready, click
submit
E.8 VIDEO FRAME SUMMARIZATION PROMPT
an aulonomous intelligent agent tasked with leurn froming a video accomplish task, You will be given list frames sampled from video and its audio transcription. You will he given task to complete. You will need to extract useful information to accomplish the task,
Example I:
VIDEO FRAMES: S PNG Frumes from Shopping Tutocial
AUDIO: Hi everyone, welcome to a tutorial on the One Stop Market. Tiday this is just a general tutorial video and how to get around on things. So one thing you need get
title the imago. And you see here going take the title. From here I can edit the quantity. add the cart, add to my wish list, add to my comparisons, and I can access through views here. can see views,
item simply click
one has 12 views here. And then I want to add items to my cart and just click add to cart, if want to add to my wish list I can click the red heart button. Similarly I add a cart. sometimes you are going to get prompted with an option to add items details before add the cart. Other times not going to he option like here, add your comparison page here well. And so if want go different sections can go here, let's go to Xbox One, let's go and have a look. There are also subsections within categories, so these are also categories, so accessories is arrow, says my cart, let's get it in black, and then similarly you can go to my carl here. I'm going to go to view it on cart and we can see that our cart is how things go. So hope helps and thanks for watching. ORIECTIVE: Add an item to the cart on the One Stop Market.
Similarly I go
will very similar,
also leave my own review at the botton
also category. Let's find the most expensive item. And you can do by sorting my price and then flipping
here and I want to go back to the One Stop Market
SUMMARY: Let's think step step. To add an item to your carl on the One Stop Market, first navigate to the website and browse or search for the desired item. You add item directly hy clicking the hlue 'dd tn Cart' hutton to it, which updates the cart icon to reflect the addition. A Altematively, click on the item listing to access its detailed page, where you can select options like size or color adjust the quantity before adding to cart. For example, you can select the size and add eight flannel shirts for your family. View and edit your cari by clicking the cart emblem/icon, which provides access to all addeed items and options to proceed to checkout.
Example 2:
VIDEO FRAMES: 5 Frames from Reddit Tutorial
AUDIO: I wanted to make a quick tutorial on how to use the reddit site, So let's say I wanted to make response to comment under one of the lop posts under the ritchmology forum. So I can click under this forums link here. Scroll down to technology. And let's say I wanted view its comments so can click here. And then look all those comments. And here. And let's say I wanted to reply great comment. So can get a quick preview. I can post it, And then now it shows me that I have sucossfully made comment under single comment. Right. OBJECTIVE: Leave comment a Postmill post.
that this is the top comment
SUMMARY: Let's think step bt-s. To leave comment un Reddit posi, start by logging into your account. Navigate to the subreddit of your choice, such ritehnology, either by searching selecting from your subserihed subreddits. Once there, select a post you want comment on, and scroll down to view existing comments. If you wish to comment the post itself, scroll to the bottom where
24

---

Published as a conference paper at ICLR 2025
you'll Add comment reply specific comment, click Reply button comment. Alter lyping your comment, you preview it by clicking the Preview button if you'd 'd like to see how it will look. When you'l 're ready, click Post to suhmit the comment. Your comment should appear immediately beneath post or spocific comment you replied to.
25


Reviews:
[{"review_id": "Y7sLOjGa77", "reviewer": "Reviewer_pi6N", "summary": "This paper introduces VideoWebArena, a benchmark designed to evaluate multimodal AI models’ abilities to process and understand long video sequences alongside text and images for completing tasks. As AI assistants increasingly need to understand video inputs to perform workflows, learn skills, and make decisions autonomously, challenges arise with maintaining temporal coherence, long-term memory, and information retrieval over extended sequences. VideoWebArena addresses a gap in existing benchmarks by focusing on these long-context multimodal capabilities, comprising 2,021 tasks with approximately 4 hours of video content. The benchmark includes 400 factual retention tasks that test information retrieval and 1,621 skill retention tasks that assess the use of in-context tutorials. Evaluations of advanced video-capable models, like GPT-4o and Gemini 1.5 Pro, show that while they exhibit basic capabilities with video content, significant gaps remain compared to human-level understanding, particularly in long-term memory and task execution. This benchmark provides a critical tool for advancing and assessing long-context video comprehension in multimodal AI.", "questions": "refer to above concerns", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It introduces VideoWebArena, a comprehensive benchmark designed specifically to evaluate the long-context understanding and multimodal reasoning of models, addressing a gap in existing evaluation tools. The benchmark includes a wide range of tasks (2,021 in total), split between factual retention and skill retention tasks, which test both memory retrieval and the application of learned information.\n\nThe study provides valuable insights by evaluating prominent video-capable LLMs like GPT-4o and Gemini 1.5 Pro, showcasing their current performance levels and identifying specific challenges they face. It effectively points out the significant gap between human capabilities and the current state-of-the-art in long-context video understanding, thus paving the way for targeted improvements in AI development.", "weaknesses": "The paper’s focus on evaluating models using video records, rather than interactive environments, introduces several limitations. This approach confines the assessment to passive information retrieval and task completion without testing an agent’s adaptive capabilities in real-time. Consequently, it fails to simulate dynamic, interactive challenges where agents need to respond to changing conditions and feedback, limiting the benchmark’s applicability for real-world scenarios. Additionally, video records do not fully capture the complexity of decision-making in live environments where agents must process incomplete or misleading information. \n\nThe paper does not specifically address potential biases related to the content of the videos used in the VideoWebArena benchmark, which raises concerns about the inclusivity and fairness of the evaluation. If the content records primarily reflect cultural, linguistic, or social contexts from a limited demographic, this could introduce biases that affect the model’s performance and generalization capabilities. Such biases may skew the results, favoring models trained on similar datasets while disadvantaging those that have been exposed to more diverse inputs. Additionally, there is the risk that the benchmark may not adequately represent ethnic and cultural variations in how information is presented or interpreted, limiting its applicability for global use cases. Addressing potential ethnic or cultural content biases is crucial for ensuring that the models evaluated can fairly and effectively serve users from diverse backgrounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "73652Xr65H", "reviewer": "Reviewer_4THA", "summary": "This paper presents VideoWebArena, a novel, open-sourced video-based benchmark, designed to evaluate the capabilities of long-context multimodal agents in video understanding tasks. The dataset consists of 2,021 tasks based on four hours of video tutorials across six domains. Moreover, the paper conducts experiments to validate that the current intelligent agents do not perform well on most the task, and is important to the relevant research field.", "questions": "1.\tCan new tasks or new domain be easily added into the benchmark?\n2.\tThrough analysis, what is the possible aspects that the existing agent can be improved?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper provides a novel video benchmark that is very welcome to the research community.\n2.\tThe benchmark offers a well-defined taxonomy that focuses on two main areas: factual retention and skill retention, which test different facets of a model's abilities to retrieve information from videos and efficiently apply learned skills.\n3.\tThe paper conducts experiments to validate that the current intelligent agents do not perform well on most the task, leading to future improvement.", "weaknesses": "1.\tThe used LLMs are all closed-source, thus may cause obstacles to reproduction. Experiments on open-sourced intelligent agents may be included.\n2.\tThis paper does not provide a comparison or discussing over itself and other similar video benchmarks, like MVBench[1], with respect to some tasks like temporal reasoning, etc.\n\n[1] Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., ... & Qiao, Y. (2024). Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22195-22206).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O76aSQjapq", "reviewer": "Reviewer_a1Hy", "summary": "This paper present VideoWebArena (benchmark) to evaluate a model's ability to process long video sequences alongside text and images to complete tasks that require memory retention, information retrieval, multimodal reasoning, and skill retention. Moreover, this papaer show that these models are still a far reach from human levels of performance, highlighting a wide gap in the information retrieval and agentic abilities of current state-of-the-art long-context models.\n\nStrengths:\n+ This paper construct a benchmark called VideoWebArena.\n+ The idea of this paper is novel and interesting.\n+ This paper test many retention tasks, I think it is a hard task and the author complete this task.\n \nWeakness:\n+ Can author release the source code for this paper, I would like to try this agent.", "questions": "See Summary", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "See Summary", "weaknesses": "See Summary", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LLSkGBoc82", "reviewer": "Reviewer_QoA2", "summary": "This paper presents the VideoWebArena benchmark to evaluate the capabilities of long context multimodal agents. The tasks are divided into factual retention and skill retention. The authors evaluate GPT-4o and Gemini-1.5-pro on their benchmarks and provide an in-depth analysis.", "questions": "Overall, this paper offers a useful benchmark with valuable insights for the community.  However, the current presentation is not clear and many clarifications are needed. Please refer to weaknesses for my questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The proposed benchmark is valuable to the community.\n+ The task designs make sense to me (divide into skill retention and factual retention), which challenge current modes’ capabilities to extract information from a demonstration video to complete tasks.\n+ The experimental findings are insightful, especially Table 7, which suggests that current models struggle with “learning” from tutorial videos, in contrast to humans.", "weaknesses": "+ The introduction lacks references.\n\n+ While the related works section reviews a few agent benchmarks and long-context benchmarks, it is suggested to include a dataset comparison table of VideoWA with existing benchmarks. This would clarify VideoWA’s distinctiveness, highlighting differences in task diversity, domain, video length, etc., against existing ones. \n\n+ To improve clarity, it would be helpful for the authors to explicitly discuss how VideoWebArena differs from VisualWebArena and WebArena in terms of environment and task design. Are there methodological innovations in this paper beyond extending these benchmarks to long videos?\n\n+ Could the authors further clarify the difference among video agent, frames agent and summary agent in terms of their input? Does the difference lie in audio input (gemini directly takes audio while gpt-4o takes audio transcriptions)? What about the summary agent?\n\n+ In Table 4, the example provided for temporal reasoning appears more like a counting task; substituting it with a clearer example focused on temporal reasoning might improve clarity.\n\n+ The benchmark only evaluates Gemini and GPT-4o. I wonder why the authors do not include open-source VLMs such as LongVILA, and LLAVANextVideo (as mentioned in the intro).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sYGNgWizba", "reviewer": "Reviewer_pezb", "summary": "The authors present VideoWA, a benchmark designed to assess the capabilities of long-context multimodal agents, particularly their ability to understand and utilize video information to accomplish tasks on web-based platforms. The paper emphasizes the gap in existing benchmarks, which predominantly focus on static image or text inputs, and seeks to bridge this with comprehensive video-based task evaluation.", "questions": "Please refer to the weakness 1-3.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written, with well-structured sections that guide the reader through the motivation, methodology, and results.\n2. The experimental setup and evaluation in the paper are robust, employing a range of state-of-the-art models such as GPT-4o and Gemini 1.5 Pro. The results are well-documented and provide clear evidence of the limitations of current models in handling long-context multimodal inputs. \n3. The significance of this work lies in its potential to drive advancements in the development of long-context multimodal models. Although some tasks may not fully justify the necessity of video tutorials, identifying more suitable tasks that would effectively leverage the richness of video input remains a challenge. Nevertheless, the benchmark's foundation provides a valuable starting point for refining task design and exploring the benefits of video-based learning in AI agents.", "weaknesses": "Major problems I have found are as follows:\n1. One significant concern is that the contributions of VideoWA appear to be relatively incremental compared to prior work such as WebArena and VisualWebArena. While the inclusion of video content does enhance task complexity and data richness, the paper does not convincingly demonstrate how these video tutorials meaningfully improve agent learning or task performance. Specifically, some tasks, like “buying the cheapest item” as presented in Table 4, do not seem to justify the necessity of video input. Agents could reasonably complete these tasks with textual intent or static images. This raises doubts about whether the chosen tasks fully exploit the potential advantages of video tutorials or merely add unnecessary complexity.\n2. Another perplexing issue is the low success rate of human performance on seemingly straightforward tasks. The authors report that these metrics were gathered using the authors themselves, who should be well-acquainted with both the data and the tasks. The review questions how the human success rate could be so limited (e.g., 73.9% on factual retention), suggesting a potential shortcoming in task design or evaluation methodology. If even familiar humans struggle, it may reflect an underlying flaw in the task construction or evaluation criteria, which could compromise the benchmark's real-world applicability and reliability.\n3. The paper presents an unexpected and concerning trend where multimodal models, even when provided with video tutorials, perform worse than models without them. This result contradicts the assumption that video tutorials should aid skill retention, as evidenced by the drop in performance on WebArena and VisualWebArena tasks (5% and 10.3% decreases, respectively). The authors should address whether the noise introduced by video information or ineffective agentic reasoning mechanisms is the primary cause of these shortcomings. Without further investigation or hypothesis testing, the paper leaves this critical aspect unresolved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)