Learning from Online Videos at Inference Time for Computer-Use Agents
Authors: Yujian Liu
Ze Wang
Hao Chen
Ximeng Sun
Xiaodong Yu
Jialian Wu
Jiang Liu
Emad Barsoum
Zicheng Liu
Shiyu Chang
Venue: Accepted by TMLR ()

Abstract: Computer-use agents can operate computers and automate laborious tasks, but despite recent rapid progress, they still lag behind human users, especially when tasks require domain-specific procedural knowledge about particular applications, platforms, and multi-step workflows. Humans can bridge this gap by watching video tutorials: we search, skim, and selectively imitate short segments that match our current subgoal. In this paper, we study how to enable computer-use agents to learn from online videos at inference time effectively. We propose a framework that retrieves and filters tutorial videos, converts them into structured demonstration trajectories, and dynamically selects trajectories as in-context guidance during execution. Particularly, using a VLM, we infer UI actions, segment videos into short subsequences of actions, and assign each subsequence a textual objective. At inference time, a two-stage selection mechanism dynamically chooses a single trajectory to add in context at each step, focusing the agent on the most helpful local guidance for its next decision. Experiments on two widely used benchmarks show that our framework consistently outperforms strong base agents and variants that use only textual tutorials or transcripts. Analyses highlight the importance of trajectory segmentation and selection, action filtering, and visual information, suggesting that abundant online videos can be systematically distilled into actionable guidance that improves computer-use agents at inference time.

Published in Transactions on Machine Learning Research (January 22026)
Learning from Online Videos at Inference Time for Computer- Use Agents
Yujian Liu' Jialian Wu?
Ze Wang? Jiang Liu?
Hao Chen Emad Barsoum
Ximeng Sun Zicheng Liu
Xiaodong Yu²
Shiyu Chang
'UC Santa Barbara
"Avancedd Micro Devices, Inc.
{yujianliu, change7) edu
Reviewed on OpenReview: http:://ppaaarriie....eeooooroer
Abstract
Computer-use agents operale computers and automate laborious tasks, but despite recent rapid progress, they still lag behind human users, especially when tasks require domain-specific procedural knowledge about particular applications, platforms, and multi- step workflows. Humans can bridge this gap by watching video tutorials: we search, skim, and selectively imitate short. segments that match our current subgoal. In this paper, we study how to enable computer-use agents to learn from online videos at inference time effectively. We propose framework that retrieves and filters tutorial vidcos, converts them into structured demonstration trajectories, and dynamically selects trajectories in-context guidance during execution. Particularly, using VLM, we infer UI actions, segment videos into short subsequences of actions, and assign each subsequence a textual objective. At inference time, a two-stage selection mechanism dynamically chooses a single trajectory to add context each step, focusing the agent on the most helpful local guidance for its next docision. Experiments on two widely used benchmarks show that our framework consistently oulperforms strong base agents and variants that use only textual tutorials transcripts. Analyses highlight the importance of trajectory segmentation and selection, action filtering, and visual information, suggesting that abundant online videos can be systematically distilled into actionable guidance that improves computer-use agents at inference time. Our code is available at bottps://githb.... com /CBB/UCSBBBBA NLP Cha.---Caaa//veeeo demo
1 Introduction
Computer-use agents promise to automate labor-inten sabor-intensive digital workflows, reshape how people interact with computers, and boost productivity across everyday and expert tasks (Deng et al. 2023
Xie et al. 2024 Zheng et al. 2024 PBC 2024 OpenAl 2025). Recent works have made rapid progress on a wide range of desktop and web tasks, yet a substantia gap remains between their performance and that of human users (Wang et al. 2025aa Agashe et al. 2025b Wang et al., 2025b). We argue that a key reason for this gap is the agent's limited access domain-spocific procodural knowledge: how operate particular applications, cope with idiosyneratic UI conventions, and execute multi-step workflows that depend software version and platform. While agents continue to improve, progress these capabilities is constrained by the scarcity of such data in large language models' (LLMs) training corpus.
On the contrary, humans are usually not limited by the lack of such domain knowledge. When we encounter unfamiliar applications or platforms, we rarely rely on our general knowledge alone. Instead, we consult online video tutorials and quickly learn useful information from the demonstrations. More specifically, we are able to search, skim, and selectively imitate video segments that match our immediate subgoals. These videos are abundant and richly visual, showcasing concrete UI manipulations that text-only resources frequently

---

Published in Transactions on Machine Learning Research (annuary/2000)
omit. This observation motivates our central research question: How can we enable computer-use agents to learn effectively from online videos al inference time?
Answering this question is challenging for two reasons. First, there is a gap between watching videos and completing tasks as computer- use agents. Videos are contimuous streams of frames, whereas agents issue actions as individual events, and they only observe sparse screenshots after completion of each action. Moreover, agents need to generate explicit UI actions, but actions performed in the videos are implicit and need to be inferred from screen transitions. Bridging from pixel changes in a video to actionable steps is therefore nontrivial. Second, raw videos are long and often contain digressions. During execution, an agent usually needs only short, locally relevant snippet that matches its current observation and subgoal, so injecting entire videos can distract more than help.
To address these challenges, we introduce an inference- time friv-v framework that turns raw video tutorials into compact, actionable demonstrations the agent can consult stop by stop. Our approach has three components. First, given the lask description, the agent generates search queries, searches online videos, and filters them to small set of relevant tutorials. Second, processing pipeline converts each filtered video into structured demonstration trajectories. Specifically, an off-the-shelf vision language model (VLM) infers underlying UI actions and synthesizes a concise objective for each subsequence of actions. With additional filtering of low-quality subsequences, this step yields a set of demonst ration trajectories, each with a textual objective and a sequence of actions and observations. Third, a dynamic selection method ensures the agent consumes only what matters at the moment of decision. Particularly, before issuing the next action, the agent performs a coarse ranking of demonstrations followed by detailed inspection of action lists to select a single trajectory that is most helpful for the next action. The selected trajectory is then provided in context. to guide the agent's next decision. This design operationalizes how humans learn from videos: we search, skim, and imitate only the few seconds that matter. By converting videos into short, validated, visually grounded trajectories, the agent receives exactly the local guidance it needs, in the format it can act on.
We evaluate our method on OSWORLD- VERIFIED (Xie et al. 2024) and WEBARENA (Zhou et al. 2024)) two widely used benchmarks for desktop and web tasks, respectively. Across both settings, our approach consistently outperforms strong agent baselines that lack video access, as well as variants that use only textual tutorials or vidco transcripts. For example,
our method outperforms the state-of-the-art Jedi framework (Xie et al.] 2025b) by 2.1. On WEBARENA, our method outperforms AgentOccam (Yang et al.) 2025b) by 4.2. Additional analyses show that our method benefits from more relevant videos, indicating the potential for further performance scale-up. Ablation studies also demonstrate the effectiveness of our designs, such as segmenting videos into short trajectorics and dynamically selecting them, as well as filtering out irrelevant actions during video processing. Together, these results suggest. that abundant online video tutorials can be systematically and automatically harvested and distilled into compact, actionable guidance that improves the ability of current computer-uss agents.
2 Related Works
Computer-use agents. Recent advances in computer-uss agents aim to endow LLM- VLM-based
systems with the ability to interact with GUTs on desktops and the web to complete tasks. Early work has focused on constructing realistic benchmarks and environments to support systematic evaluation (Xie et al.
2024
Deng et al.,
2023 Zhou et al. 2024 Lin et al. 2024 Rawles et al. 2025 Bonatti et al. 2024 Zhao et al,
2025
Chen et al. 2025). Building on these benchmarks, subsequent systems explore different ways to couple
multimodal perception with planning and control, including vision-onabled generalist agents, compositional gennerlist--seecialis architectures, and reinforcement learning pipelines that improve agents' ability through
training (Agashe et al.
2025b Wang et al. 2025a Qin et al. 2025 Wu et al.) 2024 Yang et al. 2025c Sun
ct al. 2025
Tan et al.
2025 Xu et al. 2025b Xie et al. 2025a Jia et al. 2025 Gao et al. 2024).
Similar to our work, some recent papers have investigated collecting demonstration trajectories and applying them in context or leveraging them for training. For example, /Su et al. (2025) collects and filters trajectories from the agent's sclf-exploration in the environment. Ou et al. (2024) extracts trajectorics from online tutorials, and Xu et al. (2025a) further simulates execution traces by following retrieved tutorials. However, self-exploration is limited by the agent's own capability, making it difficult to synthesize complex trajectories
2

---

Published in Transactions on Machine Learning Research (January /2026
and tasks. While online tutorials provide abundant knowledge, most existing work focuses on textual tutorials, leaving the rich visual information in videos under-exploited.
Imitation learning from observations. Our approach is conceptually related to imitation learning from observations in reinforcement lcarning, where agents learn from state-only demonstrations without access to expert actions @Liu et al. 2018: Aytar et al., 2018 Yu et al. 2018 Torabi et al. 2019 Guo et al. 2019 Schmeckpeper et al. 2021). In particular, [Torabi et al. (2018) has shown that it is possible to train an inverse dynamics model that infers latent actions from state transitions. The trained model is then used label
trajectorics in expert demonstrations, which are in turn used for supervised training.
Several recent works have adopted similar perspective for computer-use agents: tutorial videos provide rich observational demonstrations of how to operate software, but must be converted into actionable steps that agents can execute (Zhang et al., 2025 Jang et al., 2025b; Song et al. 2025 Lu et al., 2025). Specifically, Zhang et al. (2025) crawls online multimodal GUT futorials (both videos and articles) and converts them into agent trajectories via VLM-based tutorial processing, trajectory generation, and data filtering. The collected trajectorics are then used to improve agents by continuous pro-training. Similarly, two concurrent works (Song et al. 2025 Lu al., 2025) also conduct continuous pre-training from unlabeled videos. To more accurately extract the underlying actions in the video, they further train inverse dynamics model specialized for the GUI operations.
These works are complementary to ours. They focus on how to accurately label actions and scale up video-derived datasets for offline training, whereas we focus on an inference-time method that does not require additional parameter updates. Given a set of labeled trajectories, our framework studies how to search, filter, and contextualize them so that computer-use agent can effectively exploit this procedural knowledge on the fly. In addition, our method can potentially be further enhanced if equipped with the specialized inverse dynamics model trained from their works.
3 Methodology
3.1 Pipeline Overview
In this paper, we aim to develop computer-use agents that automatically complete tasks on the desktop. Given an input task, e.g., "Fill all the blank cells in B1:E30 with the value in the cell above it", the initial state, e.g., an Ubuntu desktop with specific files opened, the agent operates on the computer by issuing mouse and keyboard actions. At each step, the agent observes the screenshot of the current desktop and optionally the accessibility tree, and issues the next action by specilying the action lype (e-g., click type) and content (e.g., where to click or what to type). Our goal is to improve the ability of the agent at inference time by retrieving relovant online demonstration vidcos and processing them in a way that allows the agent to learn useful information in context.
To achiove such a goal, we design a pipeline that consists of the following three steps, as illustrated in Figure 1
1. Video retrieval: The agent retrieves and filters relevant online videos that contain useful demon- strations applicable to the testing task.
Video processing: A VLM extracts underlying actions the video and summarizes important video segments into demonstration trajectories that contain a description and sequence of actions and observations.
3. Video application: During inference, we design a two-stage selection method to select the most relevant demonstration trajectory at each step. The selected trajectory is provided in context. to help the agent better decide the next action.
In the following section, we will describe the details of each step. All prompts used in our pipeline are presented in Appendix A
3

---

Published in Transactions on Machine Learning Research
(January Haauary/2026)
Video Retrieval
Video Processing
Video Application
Action labeling
Trajectory selection
Task Fill blank B1:E30 with the value in the cell above
Action: click B1 cell
Stage based objectives Stage based actions and initial screen
Start frame:
End frame: 6
Filter irrelevant actions
Generate search queries
Add in context
Context preserving
YouTube search
Trajectory labeling
Filtering Computer-use demo?
Objective: select B1:E30
Agent issues action
Actions: click B1, drag
Relevant apps?
Observations: frame 2,9
Figure 1: Overview of our inference pipeline. Step 1 outputs set of videos relevant to the query task. Step 2 yields set of demonstration trajectories, each with a textual objective and sequence of actions and observations. Step 3 dynamically selects single trajectory to add in context before issuing each action, focusing the agent on the most helpful local guidance for its decision.
3.2 Video Retrieval
Given a computer-use task described in natural language, our first step is to retrieve online video tutorials that plausibly demonstrate procedures relevant to the task. To achieve this, the agent generates queries for each task, searches for videos using those queries, and filters retrieved videos.
Retrieving online videos. We first prompt LLMs to convert the input task into concrete search queries that reflect the application, objects, and intended operation. For the above example, we issue the query: 'LfrreOffice Cale, fill blank cells with value from above'. the task involves multiple applications, we allow the model to generate collect the top-50 results per query. Among the retrieved videos, only keep those that shorter than 10 minutes and in English.
query for each application. We then submit each query to the You A and
Filtering retrieved videos. The retrieved videos might be noisy, e.g., recording of person speaking without computer operations, or lutorial on Windows while the task is for Ubuntu, so filter the retrieved videos to a compact, high-precision set based on LLM assessments.
Specifically, the filtering is done in two steps. Coarse selection. Given the task description, as well as the title and description of all videos, the LLM selects the top-10 videos that are relevant. the task. Content verification. For each kept video, we use VLM to examine its transcript and uniform sample of 10 frames spanning the video. The VLM decides whether the vidco actually shows the application UI and computer operations pertinent to the lask. We only keep videos that are deemed helpful.
This stage returns a small set of videos that the VLM has judged as both topically and visually relevant to the task. Multiple videos can be kept for a single task to account for UI and version diversity. A represcntative result from this stage is a LibreOffice Calc tutorial
3.3 Video Processing
Although the filtered videos contain relevant demonstrations for the task, they not presented in way that is ready for use by the agent. Particularly, the video contains continuous frames, but the agent only has
https://www.youtuse http::/wwwttt......ttata/aaatooa
4

---

Published in Transactions on Machine Learning Research (January
sparse observations (i.e,, screenshots before and after each action). Morcover, the actions performed in the video are not explicitly labeled and must be inferred from screen transitions. To bridge this gap between videos and the agent, we convert each video into structured demonstration trajectories, where each trajectory includes an objective and a sequence of observations and actions.
Action labeling with a VLM. The first step of video processing is to label the underlying actions in the video. To make the labeling efficient, we first identify the frames with screen changes and only annotate actions across these frames. Specifically, we downsample each video with 2 frames per second and locate changing frames whose pixel value difference from the previous frame is above a certain threshold. We then prompt a VLM to label the actions in the video. To balance the performance and efficiency, we provide the VLM with context containing 20 frames with frames of overlap between consecutive context clips. The VLM then extracts ① the UI action lype (click, lype, right-click, drag, elc.), @ the larget element (e.g., 'cell B2', 'Find All' button), and @ start and end frames of the action. Finally, filter actions based action types and prompt an LLM to merge the same underlying actions in adjacent clips.
Filtering important and relevant actions. The above step is very likely to output actions that are irrelevant to the task, e.g., cursor movement hovering without any purpose. To improve the quality of labeled actions, we further prompt an LLM to analyze each action and only keep actions that are important and relevant to the main task in the video based on the title, description, and transcript of the video.
Demonstration trajectory construction. Even after filtering, video could contain many actions, which is inefficient for the agent process. In addition, to determine the next action, the agent usually only needs to focus on a local segment of the video instead of looking at the whole video. To allow the agent to focus on local segment, we further split the video into subsequences of actions and construct a trajectory for cach valid subsequence.
Specifically, we take inspiration from the hindsight relabeling method in RL (Andrychowicz et al. 20177] Su et al. 2025) and re-generate new objective for trajectory that does not complete the original task. In particular, given a subsequence of observations and actions, we prompt VLM to generate natural language objective that this particular subsequence can accomplish. For example, in our above tutorial on LibreOffice Cale, an objective generated for seg- ment is 'Sarcch for all occurrences in the spreadshect using the 'Find All' function in LibreOffice Calc.' We repeat such process for all subsequences of actions in the video 3 and we only keep the results where the VLM deems the subsequence as complete and there exists a reasonable objective. The outcome of this step is a set of demonstration trajectories, represented as { (objective,, {lor, (x) 4)))) K
Algorithm 1 Trajectory labeling and filtering
Input: action-observatiion sequence {(or a)) Output: set of labeled trajectories T с 0
for i to 1 do
for j = i to min(N, 15) do Use VLM to generate an objective gi.j for the action sequence azy
if the sequence is complete and coherent then
T (9hs. f(o,, a.))+=)))
end if end for end for return
where or is the screenshot at step t, aj is the corresponding action, 1, is the length of i-th trajectory, and is the total number of trajectories in the video.
The VLM might generate sub-optimal objectives for the trajectory, where the action sequence does not fully complete the objective. To ensure high quality of constructed trajectories, we further prompt a VLM filter out the invalid trajectories. Particularly, the VLM examines the coherence and correctness of each constructed trajectory, and trajectories failing the check are discarded. Algorithm I details the labeling and filtering process for trajectory construction.
Forr efficiency, only consider subsequences with length less than equal to 15.
5

---

Published in Transactions on Machine Learning Research (January /2026
Appendix C includes an example walking through the video processing pipeline. It illustrates how our pipeline ifintifies changing frames, labels actions, generates an objective that matches the task requirement, and filters invalid trajectories.
3.4 Video Application
With the processed videos, at inference time, the agent dynamically selects zero or one trajectory at each step to better decide the next action. We develop the following two-stage selection framework, which allows the agent to more accurately select a trajectory to focus on.
Stage 1: Coarse selection. Given the current observation and task, the agent selects a candidate pool of trajectories from all videos of the task. Specifically, the VLM reads each trajectory's objective and selects the top-3 relevant. trajectories for each video. Since the input only contains the textual objectives without. the observation and action sequence, the selection is efficient. then inspects the candidate pool more closely by additionally checking the initial observation and the action sequence of each trajectory. It selects single trajectory that is most helpful for determining the next action, or no trajectory if all candidates are irrelevant. Finally, the selected trajectory, including the objective, observation, and action sequence, is provided in context when the agent generates the next action.
Stage 2: Detailed selection. The agent
Maintaining coherent context across steps. Selecting trajectories at each step could lead to frequent context changes for the agent across consecutive steps, which might affect performance (e.g., when the agent selects trajectory and starts to follow, it could select different trajectory at the next step, which deviates from the previous context and plan). To minimize unnecessary reselection, at each step, the agent first decides if the previously selected trajectory is still relevant and applicable for the current observation. If yes, we simply continue with the same trajectory. Otherwise, we repeat the two-stage selection to obtain a new trajectory. This keeps the in-context guidance stable when progress is steady, while allowing quick adaptation when the UI diverges.
4 Experiments
In this section, we conduct experiments to verify the effectiveness our proposed method. Particularly, we aim answer two questions: Does providing video tutorials improve agents' performance on computer-use tasks? @ What factors affect the agents' ability learn from videos?
4.1 Experimental Setting
Evaluation benchmarks. We evaluate on two widely used benchmarks for computer-use tasks. OSWORLD-VERIFED (Xie et al. 2024 2025c) (hereafter as OSWORLD) contains 369 Ubuntu desktop tasks spanning real applications such as GIMP and VSCode, as well as tasks that involve multiple applications. Each task is associated with a reproducible initial state and manually dosigned evaluators. Following official guidance, we exclude 8 Google Drive lasks for ease of deployment. @ WEBARENA (Zhou el. al. 2024) contains 812 tasks focusing on the web environment across different domains, such as GitLab and shopping. The agent nceds to navigate the browser to complete a given task. As the official server for the map domain is not accessible, we evaluate on the remaining domains. For both benchmarks, we use the official evaluation implementation and report the final success rate (SR) over all tasks.
Baselines. We compare with three strong baselines. Base agent: We adopt the state-of-the-art agent frameworks both benchmarks. These frameworks do not have access to external tutorials during inference. Specifically, OSWORLD, we Jedi (Xie et al.) 2025b) as the base agent; on WEBARENA, use AgentOccam (Yang et al. 2025b). @ Text-only tutorials: We retrieve online textual tutorials relevant to the task and prepend the retrieved tutorials to the context at each step. For retrieval, we follow Agent S AAashe et al. 2025a to first generate queries for each task and then use Perplexica Search Enging to retrieve and summarize online tutorials into a comprehensive document. Video transcript only: We
https://gthuu..c//tttaaaaaaaiett

---

Published in Transactions on Machine Learning Research (annuayy/20000
Table Success rate on WEBARENA.
Shopping
Shopping Admin
Reddit
GitLab
All
AgentOccam
Yang al. 2025b
44.8
56.1
60.2
33.9
47.3
Transcript
44.6
54.2
53.6
42.0
47.9
Durs
47.1 (+2.3)
58.5
(-2.4)
60.6 (+0.4)
44.1 10.2)
51.5 (+-4.2)
retrieve and filter online videos as described Section 3.2 but we only use their transcripts without the visual information. To obtain the transcript, we use the turbo model from Whisper (Radford et al.) 2023). Note that baselines @ and @ are built upon the base agent frameworks in by providing external knowledge.
Agent settings. For all baselines and method, we 1gt-5-mmii-2025-085- VLM backbone at inference. On OSWORLD, all methods (ours and baselines) receive only the current screenshot as the observation, following prior work Xie et al. (0025b). We also set the maximum number of steps to 50. On WEDARENA, we follow Yang et al. (2025b) to use the accessibility tree of the page, as well as the additional current screenshot as the observation. The maximum steps is set to 20. Additional parameters of the agents are detailed in Appendix A
as the underlying
Video sources. On OSWORLD, we directly relrieve videos from YouTube, as described Section 3.2 After filtering, we collect videos for 211 out of 361 tasks [ and on average, each task contains 3.6 videos On WEBARENA, we use the videos from VideoWebArenn (Jang al., 2025a), which contains pre-recorded videos for 672 tasks in WEBARENA, excluding the tasks that require the map server. Note that although these videos are specifically designed to cover the skills required in WERARENA, they do not align with the evaluation tasks exactly, and one video could contain multiple skills corresponding to many tasks. Thus, the agent still needs to select relevant trajectories from each video and adapt its content to tasks at hand, instead of directly copying the video.
Implementation details. We build our method basod on Jedi and AgentOccam on the two benchmarks, respectively. For video retrieval and processing, use Qwen2.5-VL-322 Instruutt (Bai al., 20255) and --B---3---3-----tte-e implementation details are in Appendix
(Yang et al. 2025a) as the VLM and LLM, respectively. Additional
4.2 Main Results
Overall success rate. Table D shows the results on OSWORLD. We separately report the performance on the subset of 211 tasks for which we retrieve at least one relevant video. On the remaining 150 tasks, our method and the transcript base- line behave exactly the same as the Jedi framework. As the table indicates, our method outperforms all three baselines, especially the subset that has video tutorials. This result confirms that our proposed pipeline improves the agent's 's ability by learning from videos in context. Particularly, providing transcripts only does not improve the performance, which suggests that our structured, visually grounded demonstration trajectories are more useful than unstructured textual guidance during inference. Additionally, on average, our method takes 28.8 steps to complete a task, which is fewer than that of the Jedi baseline (30.3). This suggests that having access external knowledge allows the agent to finish the tasks more directly without unnecessary exploration.
Table Success rate on OSWORLD. "Has Videos" denotes the subset of 211 tasks for which we find least one relevant video.
Has Videos (211)
All (361)
Jedi (Xie et al. 2025b
46.8
41.0
+ Text tutorial
44.0
39.0
Transcript
46.7
41.0
Video demo
50.3 (+3.5)
43.1 (+2.1)
Thh remaining 150 tasks do have relevant video. On those tasks, our method behaves the same as the baseline agent. 6The number is calculated on the subset of 21 tasks, and these are the videos after our filtering process.
7

---

Published in Transactions on Machine Learning Research (annuary/2000))
Table 2 shows the results WEBARENA across different domains. Since the websites are synthetically built and do not exist in the real world, we do not include the text tutorial baseline. As can be observed, the results show similar trends as in Table[] where our method consistently achieves better performance than the baseline framework and the variant that adds transcript information. Particularly, in more challenging domains like GitLab, our method achieves a more significant improvement (+10.2), indicating a good utilization of the videos. By contrast, in casier domains like Reddit, the agent already has enough knowledge of how complete the task (e.g., how to create a post or make comment), therefore adding additional information in the video does not make a big impact.
4.3 Additional Analyses
Computational costs. Section 4.2 shows that incorporating video demonstrations improves task success, but it also introduces extra computation from processing videos and selecting relevant trajectories at each step. We now quantify this overhead and discuss potential optimization. Specifically, we mcasure the additional cost for the three stages of our method on a random subset of 100 tasks on OSWORLD. Table S summarizes the latency for each step. Overall, video retrieval contributes minimal overhead, as it involves relatively lightweight operations. The dominant cost in video processing comes from trajectory construction, since it needs to iterate over subsequences of actions to generate objectives. In the final video application stage, trajectory selection increases per-task runtime by 48.9% compared to the baseline that uses no video information. We view this overhead as acceptable given the corresponding performance gains and the fact that our approach is inference-time (f.e., it does not require paramcter updates or additional training data). In practice, our method is most attractive when users can afford roughly a ~1.5 inferencetimm cost, or when training is infeasible due to data, compute, or access constraints.
Additionally, several implementation choices can be introduced to further reduce the overhead. First, the first two stages can be done offline, so that the actual user inference only pays the trajectory selection cost [ Second, the videos can be processed at lower frame rate (currently 2 frames per second) when identifying changing frames and labeling actions, so that both the number of actions and constructed trajectories will be reduced. Third, during execution, the agent can self-assess whether it already has sufficient knowledge to decide the next action; if so, it can skip trajectory selection and proceed using its internal knowledge.
Failure cases. To inspect the failure mode of our method, we mamually examine 20 tasks in OSWORLD where the baseline agent succeeds, but our method fails. Appendix B lists the categories of failures and shows concrete examples. Overall, we observe that a majority of the failure cases are not caused by adding video information, and among video-related failures, the main causes are: @ video retrieval, where the system cannot find useful vidco tutorials; and @ video application, where the agent cannot adapt when the UI version in the video tutorial is different from its environment. This suggests that future improvements can be made by improving the agent's 's ability to adapt the selected trajectory to the testing environment.
Robustness to noisy videos. We next study the robustness of our method to noisy videos. Specifically, instead of using videos after careful filtering in Section 3.2 we take the videos that do not pass the first-step coarse selection of our filtering process. These videos are usually irrelevant to the task (e.g., for the task "Could you help me stretch this image fill the entire page, keeping its proportion and centering the image?" on LibreOffice Impress, one video title is "How to insert picture into table in word"). We test our method's performance when using those noisy videos while koeping everything else the same. On subset of 50 tasks in OSWORLD, we observe that using noisy videos actually solves one more task than using filtered videos. Together with the error analysis in Figure 2 where the agent is misled by a video showing a similar task but in a mismatching UI version, the results show that our system is more robust to clearly irrelevant videos, but it is more likely to be affected by the adversarial videos that look similar but contain misleading guidance.
Following Zhang et al. (025), can synthesize set of seed task queries and them to retrieve, filter, and process vidcos before receiving user queries. When user query arrives, we can retrieve similar synthetic queries and use the corresponding proccesse videos for inference.
8

---

Published in Transactions on Machine Learning Research (January /2026)
4.4 Ablation Study
We now investigate how different factors in our pipeline alfect the final performance. Specifically, we analyze the impacts from four aspects, including the number of videos per task, the proposed two-stage trajectory selection method, the filtering stop in vidco processing, and the importance visual information in the video. We conduct all the following analyses on the subset of 211 tasks in OSWORLD that have at least one retrieved video.
Number of videos per task. We first study how the number of videos per task affects the performance. Specifi- cally, during inference, we restrict the agent's s access to only subset of retrieved and filtered videos. Table 3 shows the performance of our method as we vary the amount of available vidcos. As can be seen, having access to more videos improves the overall performance, since the agent can select the demonstration trajectory from larger pool. This suggests that our method has the potential to further improve performance more relevant videos become available, assuming retrieval quality and selection remain effective.
Performance on OSWORLD with dif- Table 3:
ferent. numbers of videos per task.
Success Rate
1 video per task 3.6 videos per task
46.9
50.3
The two-stage trajectory selection. We then explore the effect of our dynamic two-stage trajec- tory selection method, which allows the agent to focus on a local segment of the video at cach step. Specifically, we compare with a variant of our method that disables this mechanism: instead of constructing trajectories for all subsequences of actions and dy- namically selecting a trajectory at each step during inference, we always prepend the longest trajectory that corresponds to the entire video. a task has multiple videos, we first prompt the agent select the most relevant one and always provide that video during inference. The second row of Table 4 (no trajectory selection) shows that the performance of this variant drops significantly compared to our original method. This indicates the importance of splitting the video into short trajectories and dynamically selecting suitable trajectories at each step.
Table 4: Ablation study on OSWORLD.
Success Rate
Ours
50.3
No trajectory selection
45.8
No action filtering
16.6
No visual information
45.6
Action filtering in video processing. Next, we investigate the impact of filtering actions during vidco processing in Section 3.3 Recall that after labeling actions in the video, we filter actions and only keep those that are relevant to the main task of the video. To validate the impact of this filtering step, we compare with a variant of our method without this step, while keeping everything else the same, including the trajectory selection method. Results in Table 4 show that this variant (no action filtering) is worse than our original method, which illustrates the benefits of filtering labeled actions.
Visual information in trajectories. Finally, we study whether the visual information in trajectories cibtrisutes to the final performance. Specifically, we modify our method so that when providing the selected trajectory in context, we only provide the textual information (objective and action sequence), without the screenshot at each step. Table 4 shows that this variant (no visual information) is worse than our original method, which provides the sequence of screenshots in context. This indicates that text-only summaries cannot cover all useful information in the video, and adding visual information can further boost the performance of the agent.
5 Conclusion
We studied how computer-use agents can learn from online vidoos inference time, inspired by how humans watch and imitate short segments of tutorials to acquire domain-specific procedural knowledge. We proposed
9

---

Published in Transactions on Machine Learning Research (Januar/
framework that retrieves and filters tutorial videos, converts them into structured, visually grounded demonstration trajectories using a VLM, and dynamically selects single trajectory as in-context guidance at each step. Experiments show consistent gains over strong base agents and variants using only textual tutorials or transcripts, and analyses highlight the importance of trajectory segmentation and selection, action filtering, and visual ion..
6 Limitations
Despite the performance improvements, our method has two limitations that may affect its practicality.
Inference-time overhead. Our approach introduces additional inference-time cost from video processing and per-step trajectory selection. As shown in Section 4.3 the final application stage increases per-task runtime by 48.9% compared to a bascline agent without vidco guidance. While this overhead is moderate relative to the observed gains, it may be undesirable in latency-sensitive deployment seltings. As a result, our method is best suited for scenarios where higher task reliability is prioritized over minimal response time, or where training is impractical due to limited data or compute access. More generally, this overhead reflects trade-off between inference efficiency and leveraging external procedural knowledge at test time.
Sensitivity to UI mismatch and near-miss demonstrations. Although system is generally robust to irrelevant videos, it remains sensitive to videos that appear highly relevant but differ subtly from the execution enviromment due to software version changes or alternative UI layouts. In these cases, the agent may over-rely on the selected trajectory and fail to adapt it to the current observation, leading to execution errors. This limitation suggests that stronger mechanisms for grounding demonstrated actions in the current environment are important directions for future work.
7
Broader Impact
This work studies how computer agents can acquire procedural knowledge inference time by learning from publicly available tutorial vidcos. While this dircction has the potential to substantially improve agent adaplability and usability, it also raises imporlant considerations, which discuss below.
Copyright and Terms of Service. Our system retrieves and processes publicly available tutorial videos from YouTube. We do not rodistribute raw videos. Instead, the system operates on limited derived information, including transcripts and sparsely sampled frames. All processed artifacts are used for research purposes only and are not intended replicate the original content. We design the pipeline comply with the platform's 's terms of service and do not enable downloading redistribution of copyrighted videos.
Security and Safety Risks. Tutorial videos may demonstrate operations that are unsafe, irreversible, or undesirable if blindly executed by the agent. In our experiments, agents operate in sandboxed virtual enviromments, which limits real-world risk. Nevertheless, this risk highlights the importance of incorporating additional safety mechanisms, such as restricting action spaces, filtering or flagging potentially dangerous operations, and requiring agent self-verification before exocuting irreversible actions. We therefore recommend that users adopt this system with appropriate caution and safety safeguards.
Bias and Representational Concerns. In this work, restrict our experiments to Ubuntu-based environments on OSWORLD and therefore retrieve and process only Ubuntu-related tutorials. On WEBARENA, we use the officially built browser environment. Generalization to other operating systems or substantially different interfaces is not evaluated here and remains an important. direction for future work.
Privacy Considerations. Our approach relies exclusively on publicly accessible content and does not access private user data. The agent does not store infer personal information beyond what is explicitly contained in public tutorials.
10

---

Published in Transactions on Machine Learning Research (January
8 Acknowledgements
The work of Yujian Liu and Shiyu Chang was partially supported by National Science Foundation (NSF) Grant IIS-233825,, NSF Grant IIS-2207052, NSF Grant IIS-2302730, and the Open Philanthropy Research Award.
References
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like human. In The Thirteenth International Conference Learning Represenlations, 2025a.
Saaket Agashe, Kyle Wong, Vincent. Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist--peciaalst framework for computer use agents, 2025b.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachol Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2017.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal agents scale, 2024.
Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Huichi Zhou, Qihui Zhang, Zhigang He, Yilin Bai, Chujie Gao, Liuyi Chen, Yiqiang Li, Chenlong Wang, Yue Yu, Tianshuo Zhou, Zhen Li, Yi Gui, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. GUI-world: A vidco benchmark and dataset for multimodal GUL-oriented understanding. In The Thirteenth International Conference on Learning Representations, 2025.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijic Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. In Thirty-seventl Conference on Neural Information Processing Systems Dalasels and Benchmarks Track, 2023.
Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, and Mike Zheng Shou. Assistgui: Task-oriented desktop graphical user interface automation, 2024.
Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, and Murray Campbell. Hybrid reinforcement learning with expert state sequences, 2019.
Lawrence Keunho Jang, Yinheng Li, Dan Zhao, Charles Ding, Justin Lin, Paul Pu Liang, Rogerio Bonatti, and Kazuhito Koishida. Videowebarena: Evaluating long context multimodal agents with vidco understanding web tasks. The Thirleenth International Conference on Learning Representations, 2025a.
Yunscok Jang, Yeda Song, Sungryull Sohn, Lajanugen Logeswaran, Tiange Luo, Dong Ki Kim, Kyunghoon Bae, and Honglak Lee. Scalable video-to-dataset generation for cross-platform mobile agents, 2025b.
11

---

Published in Transactions on Machine Learning Research (annuary/2200
Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, and Zhiyong Wu. StoreStore: Scalable integration of heterogeneous agents as specialized generalist computer assistant. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria, July 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ton Stoica. Efficient memory management for large language model serving with pagedattention, 2023.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Qinchen WU, Mingyi Yan, Zhengyuan Yang, Lijuan Wang, and Mike Zheng Shou. VideoGUI: A benchmark for GUI automation from instructional videos. In The Thirty-eight Conference on Newral Information. Processing Systems Datasets and Benchmarks Track, 2024.
YuXuan Liu, Abhishek Gupta, Picter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation, 2018.
Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang, Zekun Wang, Junlin Yang, Hongjin Su, Jixuan Chen, Junda Chen, Yuchen Mao, Jingren Zhou, Junyang Lin, Binyuan Hui, and Tao Yu. Vidcoagenttrek: Computer use pretraining from unlabeled vidcos, 2025.
OpenAL. Computer-using agent, 2025. URL http:://opemal.ccm/mdd//// htt/////ddd///ooppttttsssiigggee agentt// Ac- cessed: 2025-10-20.
Tianyue Ou, Frank F. Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Anthropic PBC. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. https //www. anthropic. www/neww//n/neeeeeeeeeeeeeeeee 5- models- and- computer-use Oct 2024. Accessed: 2025-10-20.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering aulomated gui interaction with native agents, 2025.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Tlya Sutskever. Robust. speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning. PMLR, 2023.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Ti, William F. Bishop, Wei Li, Folawiyo Campbell-Ajala Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. In The Thirleenth International Conference on Learning Representations, 2025.
Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Reinforcement learning with videos: Combining offline observations with interaction. In Jens Kober, Fabio Ramos, and Claire Tomlin (eds.), Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pp. 339 354. PMLR, 16 18 Nov 2021.
Chan Hee Song, Yiwen Song, Palash Goyal, Yu Su, Oriana Riva, Hamid Palangi, and Tomas Pfister. Watch and learn: Learning to use computers from online videos, 2025.
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan O Arik. Learn-by-interace A data-centric framework for self-adaptive agents in realistic environments. In The Thirteenth International Conference on Learning Representations, 2025.
12

---

Published in Transactions on Machine Learning Research (annuary/2200
Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, and Jiaqi Wang. Seagent: Self-evolving computer use agent with autonomous learning from experience, 2025.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jicchuan Jiang, Yewen Li, Ruyi An, Molci Qin, Chuqiao Zong, Longtao Zheng, YuJic Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng YAN, and Zongqing Lu. Cradle: Empowering foundation agents towards general computer control. In Forty-second International Conference on Machine Learning, 2025.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation, 2018
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation, 2019.
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, Wanjun Zhong, Yining Ye, Yujia Qin, Yuwen Xiong, Yuxin Song, Zhiyong Wu, Aoyan Li, Bo Li, Chen Dun, Chong Liu, Daoguang Zan, Fuxing Leng, Hanbin Wang, Hao Yu, Haobin Chen, Hongyi Guo, Jing Su, Jingjia Huang, Kai Shen, Kaiyu Shi, Lin Yan, Peiyao Zhao, Pengfei Liu, Qinghao Ye, Renjie Zheng, Shulin Xin, Wayne Xin Zhao, Wen Heng, Wenhao Huang, Wenqian Wang, Xiaobo Qin, Yi Lin, Youbin Wu, Zehui Chen, Zihao Wang, Baoquan Zhong, Xinchun Zhang, Xujing Li, Yuanfan Li, Zhongkai Zhao, Chengquan Jiang, Faming Wu, Haotian Zhou, Jinlin Pang, Li Han, Qi Liu, Qianli Ma, Siyao Liu, Songhua Cai, Wenqi Fu, Xin Liu, Yaohui Wang, Zhi Zhang, Bo Zhou, Guoliang Li, Jiajun Shi, Jiale Yang, Jie Tang, Li Li, Qihua Han, Taoran Lu, Woyu Lin, Xiaokang Tong, Xinyao Li, Yichi Zhang, Yu Miao, Zhengxuan Jiang, Zili Li, Ziyuan Zhao, Chenxin Li, Dehua Ma, Feng Lin, Ge Zhang, Haihua Yang, Hangyu Guo, Hongda Zhu, Jiaheng Liu, Junda Du, Kai Cai, Kuanye Li, Lichon Yuan, Mcilan Han, Minehao Wang, Shuyue Guo, Tianhao Cheng, Xiaobo Ma, Xiaojun Xiao, Xiaolong Huang, Xinjie Chen, Yidi Du, Yilin Chen, Yiwen Wang. Zhaojian Li, Zhenzhu Yang, Zhiyuan Zeng, Chaolin Jin, Chen Li, Hao Chen, Haoli Chen, Jian Chen, Qinghao Zhao, and Guang Shi. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning, 2025a.
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foundations for computer-use agents, 2025b.
Zhiyong Wu, Chengeheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with selfimproveement, 2024.
Jingxu Xie, Dylan Xu, Xuandong Zhao, and Dawn Song. Agentsyuth: Scalable task generation for generalist computer-use agents, 2025a.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2024.
Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yulni Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025b.
Tianbao Xie, Mengqi Yuan, Danyang Zhang, Xinzhuang Xiong, Zhennan Shen, Zilong Zhou, Xinyuan Wang, Yanxu Chen, Jiaqi Deng, Junda Chen, Bowen Wang, Haoyuan Wu, Jixuan Chen, Junli Wang, Dunjic Lu, Hao Hu, and Tao Yu. Introducing verifid. zlang.at ai, July 2025c. URL https //xlang. ai/bblo/osvorldd-errfff.
13

---

Published in Transactions on Machine Learning Research (January /202)
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tho Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In The Thirleenth International Conference on Learning Representations, 2025a.
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous GUI interaction. In Forty-sccond International Conference on Machine Learning, 2025b.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a.
Ke Yang, Yao Liu, Sapana Chandhary, Rasool Fakoor, Pratik Chandhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for L.MM-basdd web agents. In The Thirleenth International Conference on Learning Representations, 2025b.
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Silvio Savarese, Caiming Xiong, and Junnan Li. Gtal: Gui test-time scaling agent, 2025c.
Tianhe Yu, Chelsea Finn, Annic Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning, 2018.
Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song- Chun Zhu, and Qing Li. Tongui: Building generalized gui agents by learning from multimodal web tutorials, 2025.
Henry Hengyuan Zhao, Kaiming Yang, Wendi Yu, Difei Gao, and Mike Zheng Shou. Worldgui: An interactive benchmark for desktop gui automation from any starting point, 2025.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Cpt-4v(ision) is a generalist web agent, if grounded, 2024.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024.
14

---

Published in Transactions on Machine Learning Research (Jamary/2026)
Table 5: List prompts in our pipeline.
Prompt
Query generation
Figure 11
Video Retrieval
Video filtering step 1
Figure 12
Video filtering step
Figure
13
Action labeling
Figure
14
Action merging
Figure
15
Video Processing
Action filtering
Figure
16
Trajectory objective generation
Figure
Trajectory filtering
Figure
18
OSWORLD Trajectory selection stage
F Figure
19
OSWORLD Trajectory selection stage 2
Figure
20
Video Application
WEBARENA Trajectory selection stage 1
Figure
21
WEBARENA Trajectory selection stage 2
Figure
22
Trajectory continuation
Figure 23
Table 6: Agent setting on OSWORLD.
Table 7: Agent setting on WEBARENA.
Value Screenshot 50 medium Default
Value
Observation space
Screenshot accessibility tree
Observation space
Maximum steps
20
Maximun steps Reasoning effort for VLM Temperature
Reasoning effort for VLM Temperature
low
Default
A Implementation Details
A.1 Video Retrieval and Processing
We follow the procedure described in Sections 3.2 and 3.3 retrieve and process online videos. We use (Yang al.] 2025a)
5wen2.5-VL-3255 --L---2-----sttuuee (Bai al. 2025) and Qwen3-305 30B -0----------B-Intttee -
the VLM and LLM, respectively. Table 5 summarizes the detailed prompts we use in pipeline. Note that the video retrieval step is only done for OSWORLD. On WEBARENA,
use the pre-recorded videos from Jang ett al. (2025a). Additionally, since each video in Jang et al. (2025a) covers knowledge for multiple tasks, we do not filter labeled actions on WEBARENA. Instead, we directly construct demonstration trajectories for all subsequences of actions.
In particular, to construct demonstration trajectories the video processing stage, we iterate over all subsequences of actions that have length shorter than or equal to 15. For each subsequence of actions, provide the VLM with all actions (in text), well as the beginning and end frames of the subsequence (we omit the intermediate frames for efficiency). The VLM is instructed with the prompt in Figure 17 to generate an objective for this subsoquence, or None if the actions do not achieve any reasonable task. We then filter the labeled trajectories using the prompt in Figure 18 to make sure the final trajectories are coherent. and complete. Because we limit each trajectory to at most 15 actions, the complexity of this step reduces from O(N2) to O(N). Additionally, since we only provide the VLM with start and end frames without intermediate frames, the latency for each VLM query is relatively small, making scalable to videos with hundreds of actions.
15

---

Published in Transactions on Machine Learning Research (January /2026)
A.2 Agent Framework
Our method is built on the Jedi (Xie et al. 2025b) and AgentOccam Yangg el al. 2025b) | frameworks, respectively. We use gpt-5-mini -m2ni-2025-08-0' as the underlying VLM backbone for all methods. Table 5 shows the prompts we use for the trajectory selection at cach step. Tables and summarizo the agent parameters for OSWORLD and WEBARENA, respectively.
B Analysis of Failure Cases
To analyze the behavior of the agent, we conduct an error analysis for our method. Particularly, we manually examine 20 tasks in OSWORLD where the baseline agent (Jedi) correct but our method is wrong. We categorize the error causes into two groups: one group that related our video pipeline and the other that is not.
For the errors that can be attributed to our video pipeline (6/20), we identify the following reasons:
Mismatch between the video's 's UI version and the agent's ' UI version (1/20). We identify case where the video shows how to click "Display Google Chrome in this language" in order to change the default language, but the agent is operating on a Chrome in different version that does not have the same button. This discrepancy led to confusion and repetitive clicking of the same button (Figure 2
The agent already completed the task but followed the video to do extra step (2/20). For example, for a task that requires scarching for specific flights. The agent already applied the required constraints and obtained the search results, but it clicked on one of the flights for subsequent purchase (Figure 3).
The agent did not select the useful trajectory did not find useful videos (2/20). For a task that requires showing the power percentage, although the video processing pipeline has constructed a trajectory demonstrating how to achieve that in settings, the trajectory is not selected during inference. In another task that requires exporting the address book from Thunderbird and converting to a xlax file, our framework did not find any video tutorial on exporting in Thunderbird and only retrieved a video about using LibreOffice convert file to xlsx. In both cases, the agent lacks useful guidance and fails the task.
The agent did not follow the selected trajectory for the next action (1/20). In a task that requires copying table and pasting the transposed table, the agent already selected a trajectory demonstrating the procedure to complete the task, but it did not follow the trajectory to click the buttons and instead used hotkey, which led wrong outcome (Figure
For the errors that are not related our video pipeline (14/20), we identify three main groups:
Grounding issues (6/20)- Tn these cases, the agent planned the correct next action (e.g., scroll down the list find a specific item or drag an element to a specific position). However, actually execute these actions, it needs to generate code that specifies the precise amount for scrolling and the precise (x, y) coordinates for dragging and clicking. When the coordinates are wrong, the actions will not be successfully exccuted, and the agent keeps repeating the same actions (Figure 5p. This is also noted as a major failure case in the original OSWORLD benchmark Xie et al. (2024
Other issues (6/20). These failures are caused by unrelated system errors, such as network connection errors, or the agent is distracted by pop-up windows while browsing the web.
Evaluation errors (2/20). In two tasks, the agent correctly completed the task (e.g., flip an image horizontally), but the evaluation was judged as incorrect.
In summary, we observe that a majority of the failure cases are not caused by our method that adds the video information. Among the failures that related to method, most errors are from @ video retrieval,
16

---

Published in Transactions on Machine Learning Research (January /202)
where the system could not find useful videos; and @ video application, where the agent could not adapt when the UT version in the video is different from what it sees, or it follows the videos to do extra actions that are not required in the tasks. This suggests that future improvements can be made by improving the agent's 's ability to adapt the selected trajectory to the testing task.
C Example of Video Processing
In this section, we provide concrete example showing our video processing pipeline. The Lask is Please help me set Chrome to delete my browsing data automatically every time I close the browser", and a video our system retrieved https::/www..yuuuusse com/ com///aaaccccveeoooxs// Our video processing consists of the following five steps:
1. Changing frames detection. Given a video, we sample frames at 2 frames per second and identify frames that are significantly different from the previous frames, based on the absolute pixel difference. These identifiod frames represent candidate timestamps where an action may happen. Figure f shows two examples.
2. Action labeling. We provide a sequence of 20 frames from the last step to the VLM. The VLM instructed to label actions in the sequence. For each labeled action, it also needs to specify the start and end frames the action. For example, if click action involves moving the mouse, clicking the button, and waiting for the page load, then the start frame is before the mouse movement, and the end frame is after the page loading. The intermediate frames of action will be ignored in later steps. Figure 7 shows two examples. Note that to label the entire video, split the video into chunks of 20 frames, and keep 3 overlapping frames between consecutive chunks to maintain the context.
3. Acton filtering. Given the labeled actions, merge the duplicate actions at the boundary of two chunks and remove irrelevant actions.
Trajectory objective generation. We iterate over all subsequences of actions with length shorter than or equal to 15. For each subsequence, we provide the VLM with the action list, well as the start and end frames of the subsoquence. The VLM is instructed to generate objective for the subsequence, or None if the actions do not complete reasonable task. Figure shows example.
Trajectory filtering. Given labeled trajectories, the system filters and removes trajectories that are incomplete, incoherent, or contain unnecessary steps. The resulting trajectories are used during task inference, so that the agent select the most suitable trajectory at each step. Figure shows example of the final structured trajectory that is provided to our agent. Figure 10 shows example of a filtered trajectory.
Detailed Computational Cost
We measure the latency on a random subset of 100 tasks on OSWORLD. For a fair comparison, we run our method and the baseline (Jedi) with the same configuration during the same time period (to control for OpenAl API latency variations). For video retrieval and processing, we serve the LLM/VLM with vLLM Kwon et al. (2023) 8 AMD MI250 GPUs. Table g presents the results.
17

---

Published in Transactions on Machine Learning Research (January /202)
Task: am more familiar with Korean as am from Korea. want to use chrome with my mother tongue. Could you help me change the Chrome interface language to Korean?
Paramibries
Acuder
Cafrr cil poulti Goole Derine
prfcieer Orenee
Traket
Pleck parnie
Connec sepeodeee
A frame in demonstration
2313 4713
Soy malodi
Settings
Actooditty
Reve
Agent's environment
Figure 2: An example that shows the UI mismatch between the vidco tutorial and the agent's environment. The tutorial demonstrates how to set English the default language in Chrome by clicking the "Display Google Chrome in this language" button (translated). However, in the agent's 's environment, there is no such button, which leads to the task failure.
18

---

Published in Transactions on Machine Learning Research (January (lanury/2026) 2026)
Task: Find flights from Seattle to New York on 5th next month and only show those that can be purchased with miles.
DELTA
SE6 NYC
KOHL
One Way SEA NYC Wed
Min
Doarlcod the Fly Dellts App
Soter Nosters
andare
3:40pm
Bale
12,000
TOL600
At this step, the agent has correctly scarched for flights satisfying the task, but proceeded by clicking and purchasing the first flight.
Bonker
Trip Summary
XXX
Wet
700m Jedpp
Figges
Nrsch
Classic
Classic
7,000
154,
The agent finally ended at the purchase page of a specific flight, which failed the evaluation.
Figure 3: An example that shows the agent has correctly completed the task by searching the flights, but it followed the demonstration to click on specific flight.
19

---

Published in Transactions on Machine Learning Research (January /202)
Task: Apply matrix transposition to the table in B2:F5 and paste the transposed table at B8 (i.c., the top-left cell of the transposed table should be at B8)
Bryve
Paste
A frame in demonstration. In this demonstration, the labeled actions are (1) click the [Paste Special] option, and (2) click the [Paste Transposed] option.
Eiit
Masei Polnti
Stect
Tela Welco
Agent's environment. A Although the demonstration clearly shows how to paste the transposed table, the agent did not follow and instead used hotkey for pasting, which led to a wrong outcome.
Figure 4: An example that shows the agent has selected useful demonstration trajectory, but it did not follow the demonstration and ended in failure.
20

---

Published in Transactions on Machine Learning Research (January (lanary/2026) 2026)
Task: need to set the decimal separator a comma (.) for localized data representation and clarity in visualization. Can you help me to update all the numbers in the shect? Also plcase keep the decimal numbers as-ís.
Li sePPPcc
Decieed Sepertes
FIRT
HOME
VIDE
HR
General
toaffoo
1.2
Driend kag
Jimepi
1.3
Sweth
Asdic ChannO
:1:
this step, the agent correctly planned to select German, which uses comma separator, but it failed to scroll the list to find German.
Oytiine Laryagy
Data
Beat
After scrolling, it missed German, and the agent repeated the scrolling back and forth.
Figure 5: An example that shows the agent failed at grounding the correctly planned action to the precise operation command.
21

---

Published in Transactions on Machine Learning Research (January/200
(2)
(I)
(3)
(4)
(4)
(I)
(3)
(4)
(4)
Figure 6: Two examples that show how our system identifies frames that have significant changes from the previous frames. The resulting frames are candidate timestamps where an action may happen, and they are provided as input to the next step of action labeling.
22

---

Published in Transactions on Machine Learning Research (January/20) /2026
Action: click the [Site settings] button
Start frame:
End frame:
(I)
(2)
Action: click the
[On-device site data] button
Start frame:
End frame: 3
(3)
Figure 7: Two examples that show how our system labels actions from sequence of frames. Note that in the actual implementation, we pass a sequence of 20 frames to the VLM, but we only show the relevant frames here for ease of visualization. Each action is labeled with associated start and end frames, which indicate the start and completion point of the action. The intermodiate frames will be ignored for later processing.
23

---

Published in Transactions on Machine Learning Research (January /2026
Action sequence:
1. click the [Privacy and security] button
click the [Site settings] button
click the [Additional content settings] button
4. scroll down the settings menu
5. click the [On-device site data] button
6. highlight the [Delete data sites that have been saved to your device when you close all windows] option
Start frame
End frame
Objective: Configure Chrome to delete site data when all windows are closed.
Figure 8: An example that shows how our system generates an objective for sequence of actions. For efficiency, we limit the action sequence to at most 15 actions, and we only provide the start and end frames of the sequence to the VLM.
24

---

Published in Transactions on Machine Learning Research (Jamary/2026)
Objective: Configure Chrome to delete site data when all windows are closed.
Action 1: click the [Privacy and security] button
Action 2: click the [Site settings] button
Action 3: click the [Additional content settings] button
Action 4: scroll down the settings menu
Action 5: click the [On-device site data] button
Action 6: highlight the [Delete data sites that have been saved to your device when you close all windows] option
Figure 9: An example of the final structured demonstration trajectory provided to our agent. Note that images and text are interleaved, and the text at the bottom shows the corresponding action that should be performed on the image.
25

---

Published in Transactions on Machine Learning Research (Jmmary/2026)
Objective: Configure Chrome to allow specific sites to save data on the device by adding them to the allowed list.
ago
Action 1: click the [Additional content settings] button
Action 2: scroll down the settings menu
Ago
Action 3: click the [On-device site data] button
Action 4: highlight the [Delete data sites that have been saved to your device when you close all windows] option
Conker
Abregs Aaia your doyfer shen dree Ohrone
Nit alooedd to devic
Action 5: highlight the ['Customised behaviours] section
Action 6: click the [Add] button next to 'Allowed to save data on your device'
An example of filtered demonstration trajectory. It is filtered because it contains extra steps not required in the objective (e.g., the action of "highlight the [Delete data sites that have been saved to your device when you close all windows] option"). Note that this is already reasonable trajectory, and it is filtered only because of the unnecessary actions. For completely invalid trajectories, they will not be generated in the previous step.
Figure 10:
26

---

Published in Transactions on Machine Learning Research (January /2026
Table Latency of each step our framework.
Operation
Latency (seconds)
Video Retrieval (per task)
Query generation
0.1
YouTube search
1.2
Video filtering step 1
0.6
Video filtering step 2 Total
19.9
21.8
Video Processing (per task)
Changing frame identification
11.2
Action labeling
260.3
Action deduplication and merging
26.6
Important action filtering
13.0
Trajectory construction
616.3
Trajectory filtering
198.7
Total
1126.1 1
Video application
Time per step (without videos) Time per task (without videos) Time per step (with videos) Time per task (with videos) Additional time per step Additional time per task
31.8
963.5
49.8
1434.2
18.0
470.7
27

---

Published in Transactions on Machine Learning Research (annuary/20000
You search-query genarator.
Given: (1) laptop task user wants to accomplish, and (2) the related desktop applications.
Goal: Produce concise web search query that typical user would type to find a tutorial or guide to complete the task with those agps.
Guidelines:
Return only the plain query text Ino labels, quotes, or operators like site:).
Make concise, use keywords phrases, e.q., search for specific application specific website. Include leas *-bbsaacc the general functionality goal++ behind detailsw like filenames, personal data, file paths, page numbers, timestarpa, etc. If the task involves multiple steps, focus on those that require the most application knoxledge.
10 words in each query.
task. **vvoid task-specific
If the task involves only one app, generate exactly one query.
the task involves N apps (N 1), you can generate one or more (but at most N) queries each at a new line. For example:
If there is a single primary app, generate a single query focused on that app. there are multiple prinary apps, generate separate queries
app.
Figure 11: Prompt used to generate scarch queries for video tutorials.
You are a helpful assistant.
Given: (1) laprop task a user wants to accomplish, (2) the related desktop applicationn
and (3) a list retrieved videos teach with an ID, tisle, and description Goal: Select the videos that are most likely to help the user complete the task.
Output Requirements
Output list of for the selected videos. Output JSON format with single key velected_video_____ ids' For example:
json
"selected vidctedddidoo_dd""" (1, 5, 8, 12, 13, 17]
Select videos, ordered by relevance.
of the videos relevant, return empty list.
# Guidelines:
Rxclude videos that cover unrelated appl catiens or of--topic content, even if they are tangentially related to the task.
Figure 12: Prompt used for coarse filtering of retrieved videos.
You helpful assistant.
Given: (1) task user wants to accomplish on their Ubuntu laptop, and (2) retrieved
video (with title, description, trenscript, and 10 sample frames).
Goal: Determine is the video (1) demonatrates the use of desktop applicatiens on an Ubuntu lastop, AND (2) can help the user complete the task.
Guidelines:
The video must centain directly applicable information that enables the user complete the
28

---

Published in Transactions on Machine Learning Research (annary//22000
The video terminals, IDEs, office software, media players, etc.) running Jbuntu operating system.
actual usage desktop applications (e.g., file managers, browsers,
The demonstration should clearly show the interaction with the Ubantu desktop interface ( mouse clicks, typing, application windows).
à video should **пос** be considered a demonstration is it only shows:
Slides or presentation decks
Talking-head segments (person speaking zo the camera)
Non-desktop applications (e.5., mobile apps)
Output Requirements
Generate your output in two sect
O3SERVATIONS:
List your observations about the video frames, focusing on elements that indicate whether desktop appl applications are being used on an Ubuntu laptop.
Be specific about what you see in the frames, such as application windows, desktop environment features, or any other relevant details.
JUNGRMENT:
Output your judgement
format with single key "judge". example:
jsonn
("judge": <true false>) ...
where 'true" means the video is demonstration of using Uburtu desktop applications and can help the user complete the task, and "false" means otherwise.
Figure 13: Prompt used for content verification of retrieved videos.
You are given consecutive frames from screen recording, Based on the visual change between these frames, infer the most. likely user act on (s) that caused the change.
Below the list of possible actions:
click [button]: Click element button. E. "click [Submit] button'.
type [text] [box]: Type text in box. R.g., "type [Rello World] in the [search box]", right click [butten]: Right click button or link. E.g., "right click the product link
drag [element] to [position]: Drag the element to speci fied pasition. E.g., 'daag [text box] to top of the page".
press [key]: Press one or more keys. B.g., 'press [Ctrl+F)' or press [Esc]',
Instructions
Carefully analyze the visual changes between these frames datermine most likely user action from the 1ist above.
2. For each action, also indicate the and end frame numbers (inclusive) the action was observed.
#1 Output Format
Generate your response in two sections:
ORSERVATTON AND RRASONTNG:
Describe the visual changes step-by-ateg and explain what user (c) likely caused these changes.
ACTIONS:
Conclude your response with all identified actions JSON format. example: json
{ ("action:: "click the [Subnit] button", "start_framme 1, "end [and_="ame" 2), ["action:: "sype [Eello World] in the [search box]", ", "start "raaee_"fram"" 4, "end frame": 7h, // more actions if applicable
Guidelines
29

---

Published in Transactions on Machine Learning Research (annurry/20000
vvActien Duration: Include all frames from the **strr the actionss (e.g., curser onto the button) to the weend of resulting changevx (e.g.r page finishes loading)
+-Atomic Actions:** Break down actions into the smallest clear units. For example, two consecutive button clicks should be annotated as two separate "elick" actions with different frame ranges, However, if a user types a sentence in a box, annotate it as a single typing action.
only annetate click or typing action when there is vxcleaz, observable evidencex (e.g., button is visibly pressed or text is entered). Ignore cursor movements hovers without results.
EvEidence Requirement:**
vvTerminology Consistency:ax Use the given action
instance, "click" .o Overlaps:<^ Frame ranges for different actions should not overlap.
insted of "opan" "select".
Empty Results:** If action can be determined, return an empty list '[)'.
Figure 14: Prompt used for action labeling on OSWORLD. Note that use the same prompl on WEBARENA except for the list of possible actions.
You are given list of UI actions. Your task is to detect duplicate or equivalent actions.
Two actions are considered duplicates if they have the same action type and target the same
UI element the same intent.
duplicates are found, merge them into single action that accurately represents the combined intent.
#4 Output Format
Generate your response in two sections:
REASON:
Describe the reascning process for identifying duplicates.
MERGED ACTIONS:
Conclude your response with all identified duplicates in JSON format. Por example: json
"mmergedaaction" "click the [Submit] button", toriginal_action ids"; [0, jarged_acttion: "type [lello World] in the [search bex]", *original_acti actiiottttttoot ids": [2, 3, 51)
where 'merged_acte actioon is the merged action, and original_action idd" is the list of original action TDs that were nerged.
Guidelines
actions are partial segments of the same continueus action (e.g., two typing steps that form single sentance), merge them into complete action.
**Natural Contimuationve:
v>Do Not Merge Distinet Azomic Aetionss:: IE actions have different purposes target di fferent UT el elements, they must remain separate.
**Fresevve Action Type**: Keep the merged action in the same category as the original actions.
*. Empty Results:*. If no duplicates are found, return an empty list.
Figure 15: Prompt used for merging duplicated actions.
30

---

Published in Transactions on Machine Learning Research (annuary/20000
You given (1) information of YouTube video (title, description, and transcript) and (2) 1ist of UI actions identified from the video. Your task is to select actions that are relevant to the task demonstrated in the video.
#4 Output Format Generate your response in two sections:
ANADYSES:
Jeseribe the task demonstrated the transcript.
on the title, description,
Analyze each action line by line, describe whether it is important relevant the task, and explain why.
XEPT ACTIONS: Output your results with JSON list containing the indices of relevant actions. For example:
1son
[1, 2, 4, 7, 8, 12, 13, 17, 20]
#1 Guidelines
-*RRelevance to Task++: Remove acticns that are tangential or unrelated to the primary
task. Example of irrelevant actions include (but are not limited to):
Mouse movements or hover actions that do not lead to a click or interaction.
Waiting actions unless they are crucial for understanding the sequence of events.
Scrolling actions that do not contribute to the task.
same intent or outcome, keep most Redddndancy: maltiple actions represent comprehensive and the others.
Figure 16: Prompt used to filter out irrelevant actions.
(platform) before any actions, sequence of actions performed the (platform), and screenshot the (platform) after the actions. Your task is to determine whether the sequence of actions is:
will given screenshot
1. v*Coherentvx: The actions are possible given the initial screenshot and lead logically to the final screenshot.
2. v=Completewx: All steps necessary to finish the task are included. If any essential step is missing, it is wyrot** complete.
3. C+Contains no unnecessary actions.: Every action should be relevant to achieving the task. Extra, unrelated steps make the sequence invalid.
<*Matching a speci fic, reasonable task.*: actions clearly fully accomplish ene identiflable goal.
If the sequence contains extra actiona, is missing required steps, does not fully achieve the task, output "No task". Example: If the intended task is creating new issue, but there is no acticn elicking Create issue], this is asnotan valid task.
Output Format
Generate your respense in two sections:
ORSERVATION REASONING:
Describe what is visible in before and after screenshots. through the sequence of actions step-by-step, explaining their effects.
State whether they accomplish clear, reasonable task land describe it) Confirm rm that all required steps for the task were perforned otherwise state that it is incomplete.
Confirm that all steps are necessary and that there are no irrelevant actions.
TASK:
clear, complete task actieved, output:
ison
("task": *sconcrete descripsion of task>"}
31

---

Published in Transactions on Machine Learning Research (annuary/20000
the actions incomplete, inccherent, contain actions, do achieve any reasonable task, output:
json
("task": "No task")
Example
ison
("Lask"; "Filter issues labeled as bug.")
Figure 17: Prompt used to generate objectives for sequence of actions.
You will be given task description and a trajectory that consists of screenshot of a platform) before any actiona, a sequence of actions performed on the (platform), and a screenshet of the (platforn) after the actions. Your job is to decide whether the trajectory successfully accomplishes the task according to the criteria belcw.
Rvaluation Criteria
**Aligmeent*:: The trajectory's final state clearly fulfills the task described.
2. +*Coherence.*: Each action logically follows from the previous observation, and action contradicts the task.
3. +*Maturalnessee: The actions resemble realistic human interaction with the (platform) in this context.
4. --Reasenableeesssw: The task is completed efficiently, xithout unnecessary steps, backtracking, or overly complex methods.
Output Format
Generate your respense in sections: 03SERVATION REASONING:
Jescribe what is visible in the before and after scraenshots.
Determine if the firal state of the (platform) matches the expected outcome of the task. Go through the sequence of actions step-by-step, explaining their effects.
Confirm that all necessary steps were performed and that each action was required.
JUDGMENT:
.** json
[" ("dugee": <true or false?, "reason": "creasoning for the judgment>")
Figure 18: Prompt used filter labeled trajectories.
You autonomous intelligent agent tasked with completing task Ubantu desktop. Given the progress far, the current observation, and list of tasks with human demonstrations, your task is to select up to 3 most relevant demonst.aations that can help you determine the next action.
Output the of the demonstration tasks inside triple backticks, aplit by commas. For example, "*2,
2, 15, "3
you think of the demonst relevant, output ""None""
Figure 19: Prompt used for stage 1 selection of demonstration trajectories OSWORLD.
32

---

Published in Transactions on Machine Learning Research (annuayy/20000
You autonnmouus intell igent agent tasked with completing task Ubuntu opsktop. Given the progress far, the current osservation, and 1ist of tasks with haman demonstrations (each with task description, the initial observation, and sequence of act actions), your task is to select the most relevant demonst whose detailed informat ion can help you determine the next action. Assume that for each demonstration, you may later access intermediate observations (i.e., after each action), which can further guide your Docision-making
Output the of the demonstration inside triple backticks. For example, "22"" ""''"
If think none of the demonstrations are relevant, output "'None""
Figure 20: Prompt used for stage 2 selection of demonstration trajectories on OSWORLD.
are an autonomous intell igent agent tasked navigating browser.
Given the progress
the current observation, and list of demonstration tasks, your task is to select up 3 most relevant ramonstratioon that help determine the next action.
Output your response in three steps.
1. First, analyze whetter you already have enough knowledge to determine the next action.
2. If not, inspact the demonstration ions to see if they contain the requi red information.
3. Output your final decision in the following two formats:
action, demonstration needed: If you already
enough knowledge determine the relevant, output None inside triple
of the denonstrations backticks. For example, "Nooe
Case 2 dsmonstration.ss) needed: If need the detailed information from some insonstrations output the id of the demonstration tasks inside triple backticks, split by commas. For example, "*,, 15
Figure 21: Prompt used for stage selection of demonstration trajectories on WEBARENA.
You autonomous intelligent agent tasked with navigating browser.
Given the progress far, the current observation, and list of demonstration tasks extracted from video (each with a task description, the initial observation, and sequence of actions), your task is to select the most relevant demonstration that can help determine the next action.
steps. First, analyze whether you already enough knowledge determine the actien. if they contain the required information. 3. Output your final decision in the folloxing two fornats:
Output your response in
2. If not, inspect demonstrations
demonstration needed: If already have enough knowledge to determine the next action, or none of the ienonstrations are relevant, autput None inside triple backticks. For example, ""Noon"" Case 2 demonstration needed: If you need the detailed information from specific demonstration, output the of the demonstration inside triple backticks. For example, ""2"" 119 "".....
Case
33

---

Published in Transactions on Machine Learning Research (January /202)
Figure 22: Prompt used for stage 2 selection of demonstration trajectories WEBARENA.
You are an autonomous intelligent agent tasked with completing a task on an Ubuntu desktop. You have previously found demo trajectory for related task, and now your task to determine if the demo still relevant and useful for choosing your next action.
You will be given:
The current state of your task and envi ronment (1.0., your progress and current observation)
The demo trajectory you found previously
Respond with one of the following, inaide triple backticks: ... YYe" if the demo trajectory is still relevant and can help you determine the next action.
... "N'N"" No* i-
trajectory no langer relevant helpful.
Figure 23: Prompt used to determine whether to preserve the previously selected trajectory.
34


Reviews:
[{"review_id": "9i5FAiFHEj", "reviewer": "Reviewer_JfdE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### **Summary:**\nThis paper proposes an inference-time framework that enables computer-use agents to learn from online tutorial videos to improve task performance on desktops and the web, without any additional parameter updates. The method retrieves relevant videos, converts them into structured demonstration trajectories (with inferred actions, screenshots, and textual objectives), and then dynamically selects a single most relevant trajectory at each step to provide in-context guidance to the agent.\n\n### **Key technical components:**\n1. Video retrieval: LLM-generated search queries, YouTube search, and LLM/VLM-based coarse and content filtering.\n\n2. Video processing:\n    (i) VLM-based action labeling across frame transitions.\n    (ii) Filtering of irrelevant actions (e.g., mouse hovering, tangential interactions).\n    (iii) Generation of sub-trajectories with VLM-synthesized natural language objectives.\n    (iv) Validation/filtering of trajectories for coherence and completeness.\n\n3. Video application at inference time:\n    (i) A two-stage trajectory selection mechanism (objective-based coarse ranking + detailed inspection using initial screen and action sequence).\n    (ii) A context-preservation heuristic that decides whether to continue using the previous trajectory or re-select a new one.\n\n4. Experiments on OSWorld-Verified and WebArena show that the method:\n    (i) Outperforms strong base agents Jedi and AgentOccam.\n    (ii) Outperforms variants that use only textual tutorials or only transcripts, indicating that structured, visually grounded trajectories are more useful than unstructured text.\n    (iii) Benefits from more relevant videos and from design choices such as action filtering and trajectory segmentation.\n\n### **Strengths:**\n1. Clear, compelling motivation: agents lack domain-specific procedural knowledge; humans often rely on short tutorial segments to fill this gap.\n2. Well-structured pipeline that is modular and compatible with existing agents.\n3. Evaluated on two widely used, realistic benchmarks (OSWorld, WebArena) with strong baselines.\n\n### **Ablations convincingly show the importance of:**\n1. Trajectory segmentation and dynamic selection.\n2. Filtering irrelevant actions.\n3. Incorporating visual information.\n4. Training-free at test time; relatively easy to plug into other frameworks.\n\n### **Weaknesses:**\n1. Heavy reliance on VLM quality for accurate action labeling and trajectory construction; limited analysis of compounding errors.\n2. Limited qualitative analysis of failure modes (e.g., incorrect trajectories, UI mismatches).\n3. Retrieval stage effectiveness is not quantitatively evaluated (no precision/recall of video filtering).\n4. No systematic measurement of latency and compute overhead, which are crucial for real deployments.\n5. Dependence on external video platforms (YouTube) raises questions about temporal reproducibility and data availability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "## The main empirical claims are:\n1. Inference-time use of video-derived trajectories improves agent performance on computer-use benchmarks.\n2. Structured, visually grounded trajectories are more helpful than text-only tutorials or transcripts.\n3. The proposed design choices (segmentation, selection, filtering, visual inclusion) are each important contributors.\n\n## These are supported as follows:\n1. Quantitative results:\n    (i) On OSWorld, the proposed method improves success rate over Jedi by +3.5 on tasks with available videos and +2.1 overall (Table 1).\n    (ii) On WebArena, the method improves over AgentOccam and a transcript-only variant, with particularly large gains (+10.2) on the more challenging GitLab domain (Table 2).\n\n2. Ablation studies:\n    (i) Removing trajectory selection and always providing the full video trajectory significantly hurts performance, showing the necessity of segmenting videos and dynamically selecting relevant trajectories.\n    (ii) Removing action filtering degrades performance, demonstrating the value of removing irrelevant UI actions.\n    (iii) Removing visual information from trajectories (using text-only) reduces performance, supporting the claim that visual context provides additional, non-redundant information (Table 4).\n\n3. Scale analysis:\n    (i) The paper shows that having more videos per task (e.g., 3.6 vs 1) improves success rate (Table 3), consistent with the claim that abundant online videos can be leveraged at inference time.\n    (ii) While some implementation details (e.g., error analysis of the VLM labeling, robustness under noisy or misleading videos) are underexplored, the core quantitative claims are well-supported by standard benchmarks and strong baselines.", "audience_yn": "Yes", "audience_explanation": "This work sits at the intersection of:\n1. Computer-use / GUI agents\n2. Multimodal LLMs/VLMs\n3. Imitation learning from observations (video)\n4. Test-time augmentation / retrieval-based methods\n\nThese are active and fast-moving research areas. The specific idea of using online tutorial videos at inference time, without retraining, to guide agents is:\n1. Novel in its test-time focus (complementing existing training-time video-pretraining works).\n2. Immediately relevant to practitioners building real-world agents (desktop or web).\n3. Conceptually interesting for the broader community interested in in-context learning and multimodal grounding.\n\nGiven the prominence of OSWorld/WebArena and the increasing interest in practical agent systems, the paper’s findings are clearly of interest to a non-trivial portion of TMLR’s readership.", "requested_changes": "Below are proposed adjustments. Each item is marked as Critical (necessary for a positive recommendation) or Recommended (would strengthen the paper).\n\n### **Critical:**\n1. Quantify computational cost and latency:\n    (i) Report end-to-end runtime per task and per step with and without video integration.\n    (ii) Break down additional overhead from:\n        (a) Video retrieval and processing (offline vs. online).\n        (b) Per-step VLM/LLM calls for trajectory selection.\n    (iii) Clarify whether the pipeline is practically deployable under typical inference budgets.\n\n2. Provide failure mode and qualitative error analysis:\n    (i) Show concrete examples where:\n        (a) The selected demonstration trajectory is clearly mismatched with the agent’s current state.\n        (b) The VLM mislabels actions or produces incoherent objectives.\n        (c) Visual mismatches (OS differences, app versions, window layouts) lead to incorrect guidance.\n    (ii) Summarize the dominant failure patterns and how often they occur.\n\n3. Discuss robustness to noisy or misleading videos:\n    (i) Analyze or discuss how the system behaves if:\n        (a) Tutorials are outdated (UI version changed).\n        (b) Videos contain mixed or irrelevant workflows.\n        (c) There is adversarial or low-quality content.\n    (ii) If possible, provide an experiment or at least a controlled case study illustrating robustness or failure.\n\n### **Recommended:**\n4. Quantify video retrieval effectiveness:\n    (i) Provide statistics on:\n        (a) Fraction of initially retrieved videos that pass coarse and content filtering.\n        (b) Human-judged relevance of filtered videos for a small subset.\n    (ii) This would better justify the retrieval stage and illustrate how often the pipeline can find useful tutorials.\n\n5. Add qualitative trajectory visualizations:\n    (i) Include 1–2 figures showing:\n        (a) Example video frames, labeled actions, and synthesized objective.\n        (b) A “good” trajectory vs a “filtered-out” or invalid trajectory.\n    (ii) This will improve the interpretability of the VLM-driven processing steps.\n\n6. Clarify scaling behavior and limits:\n    (i) Explore or at least discuss:\n        (a) Impact of increasing the number of videos per task beyond the current average.\n        (b) Potential issues with very large demonstration pools (e.g., context length, selection noise).\n\n7. Improve reproducibility of video-based experiments:\n    (i) Since YouTube search is non-deterministic over time, consider releasing:\n        (a) The list of video IDs/URLs used.\n        (b) Pre-processed trajectories (or a subset) as a dataset.\n    (ii) This is important for replicability and future comparison.\n\n8. Explicate assumptions on VLM capabilities:\n    (i) Discuss what level of VLM performance is required for the method to be effective (e.g., approximate accuracy on action detection, objective coherence).\n    (ii) This will help readers assess applicability under different model choices.", "broader_impact_concerns": "The paper should expand the Broader Impact discussion to address:\n\n### **Copyright and Terms of Service:**\n1. The system systematically retrieves and processes YouTube tutorial videos. Clarify:\n    (i) Compliance with YouTube’s terms of service.\n    (ii) Whether only metadata + transcripts + frame samples are used, and how they are stored.\n    (iii) Any restrictions on sharing or redistributing processed trajectories derived from copyrighted videos.\n\n### **Security and Safety Risks:**\n1. Tutorial videos may instruct sensitive or dangerous operations (e.g., editing system settings, deleting data). Agents blindly following them may perform harmful actions.\n2. Discuss mitigations such as:\n    (i) Restricting actions to safe environments (sandboxed VMs).\n    (ii) Detecting and filtering risky operations through additional safety filters.\n\n### **Bias and Representational Issues:**\n1. If the video corpus is skewed toward certain software versions, platforms, or user workflows, the agent might inherit these biases.\n2. Consider how this affects generalization and fairness across different user populations and setups.\n\n### **Privacy:**\n1. Clarify that only publicly available video content is used and that the method does not access or rely on private account-linked resources.", "additional_comments": "1. The paper is clear and well-organized, with a logical progression from motivation to method, experiments, and analysis.\n2. The human analogy (“search, skim, imitate short segments that match subgoals”) is not just rhetorical; it is operationalized in the pipeline via segmentation and dynamic selection.\n3. The two-stage selection framework is conceptually clean and appears to be a major practical contributor; further analysis of selection stability across long trajectories (e.g., how often trajectories are switched) would be interesting.\n4. Including confidence intervals or task-level variance for success rates would improve the statistical rigor of the reported results.\n5. The work is nicely positioned as complementary to training-time video pretraining efforts; a short explicit subsection comparing pros/cons (e.g., compute, flexibility, data requirements) of inference-time vs training-time usage would be a useful addition.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DJK747jIJp", "reviewer": "Reviewer_iEPs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a two-stage pipeline consisting of **video retrieval** followed by **video processing**. Third stage is application so i am skipping in the summary.\n\nIn the **video retrieval** stage, an agent is tasked with finding videos relevant to a given instruction or task. First, the task for the agent is defined. The agent then retrieves the top 50 candidate videos using standard search signals (e.g., title and basic metadata), without any sophisticated reasoning at this point. Next, a large language model (LLM) is used to filter these results based on the actual video content, narrowing them down to the top 10 candidates. For each of these 10 videos, the system samples 10 frames and feeds them into a vision–language model (VLM) to assess whether the video is likely to be relevant to the target task. **The exact sampling strategy (e.g., whether this corresponds to 10 frames total, 10 fps, or another rate)** is not fully clear and is left somewhat underspecified in the paper. After this step, the system has a set of videos that are contextually appropriate and exhibit the right kinds of actions, but they are still only *potential* matches for the full task trajectory.\n\nThe second stage is **video processing**. Here, the system resamples the selected videos at 2 frames per second and performs frame differencing. Large differences between frames are interpreted as potential boundaries between different subtasks or trajectories. Each such trajectory segment is then passed to a VLM, which assigns a label describing the underlying action or subtask. This produces a sequence of subtasks. For example, in a “cooking” video, the system might identify subtasks such as “cutting onions,” “cutting tomatoes,” and “mixing ingredients,” even though the video might only be labeled at a high level as “cooking.”\n\nHowever, identifying subtasks alone does not guarantee that the full, desired task is demonstrated correctly. To address this, the authors introduce a **relabeling and verification** step. The VLM-generated trajectory labels (e.g., “cutting onions,” “cutting tomatoes,” “mixing together”) are passed to an LLM, which checks whether these subtasks indeed form part of the intended overall task. This process is iterative, allowing the agent to gradually refine its understanding of which videos truly show how to complete the target task from start to finish.\n\nThe authors evaluate their method on two widely used benchmarks and report incremental (rather than dramatic) performance improvements. Nonetheless, the combination of video retrieval and structured video processing is presented as a novel and interesting approach. The paper also provides detailed implementation information and extensive references to prior work, which appear sound and well-chosen. Overall, it offers a clear, methodical framework for teaching an agent to learn tasks from videos.\n\n** I am aware of author mentioning uniform sampling, this raises more question such as what if the video is 2-3 min long or very short. Both can have little information to provide due sparsity or due to repetative frames.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the paper has been benchmarked and shows clear evidence. Also the paper provides implementation details in supplementary section.", "audience_yn": "Yes", "audience_explanation": "This paper is not research or innovative in terms of new design, task or new discovery rather its improvement of agent task. Agent task is new domain, so this paper definietly contributes to innovation.", "requested_changes": "Concerns around sampling rate used while video retrieval", "broader_impact_concerns": "none", "additional_comments": "I am aware of author mentioning uniform sampling, this raises more question such as what if the video is 2-3 min long or very short. Both can have little information to provide due sparsity or due to repetative frames.\nAlso can you add few more visualization of the approach. The one you have provided is good, but the flow can be more cleaned and well presented.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S8aMn7dTiP", "reviewer": "Reviewer_n2V8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a framework to enable computer-use agents to learn from online video tutorials at inference time. The core idea is to mimic how humans learn: by searching for relevant tutorials, watching short, relevant segments, and imitating the demonstrated actions. The proposed pipeline consists of three main stages: (1) **Video Retrieval**, which finds and filters relevant tutorial videos from online sources like YouTube; (2) **Video Processing**, which uses a Vision-Language Model (VLM) to convert raw videos into structured, actionable trajectories, each containing a sequence of actions, observations, and a high-level textual objective; and (3) **Video Application**, which at each step of a task dynamically selects the single most relevant trajectory to provide as in-context guidance to the agent. The method is evaluated on two standard benchmarks, OSWORLD and WEBARENA, where it shows improvements over baseline agents and variants that use only textual information.\n\n**Strengths:**\n*   The paper addresses the critical limitation of domain-specific knowledge in current computer agents with a well-motivated and intuitive solution inspired by human learning. The focus on inference-time learning from videos is a practical and timely direction for the field.\n*   The framework demonstrates good empirical results, consistently outperforming state-of-the-art base agents (Jedi, AgentOccam) and other guidance methods on two diverse and widely-used benchmarks. The performance gains in some challenging domains are notable.\n*   The ablation studies convincingly demonstrate the importance of several key components of the proposed framework, highlighting the value of dynamic trajectory selection, action filtering, and the visual information present in videos.\n\n**Weaknesses:**\n*   The proposed pipeline appears to be an engineering-heavy workflow that is computationally expensive and likely suffers from high latency. The video processing stage seems to scale quadratically with the number of actions, and the inference-time selection mechanism involves VLM calls at every step. The paper completely omits any discussion or measurement of these practical costs, which undermines its claim as an effective *inference-time* method.\n*   The paper lacks qualitative analysis and a rigorous examination of failure cases. The aggregated success rates obscure the specific reasons for failure, making it difficult to understand the bottlenecks in the pipeline (e.g., poor video retrieval, incorrect action labeling, or flawed trajectory selection).\n*   The methodology lacks sufficient detail for reproducibility. Key steps, such as the generation and filtering of trajectories from \"all subsequences of actions\" (Algorithm 1), are described too abstractly. The $O(N^2)$ complexity of this step is a significant scalability concern that is not addressed.", "claims_and_evidence_yn": "No", "claims_explanation": "While the paper presents a promising direction, the evidence provided is not yet sufficient to fully support its claims in a convincing manner. First, the claim of practicality for an \"inference-time\" method is unsubstantiated, as the paper provides no analysis of the significant computational cost and latency its pipeline likely introduces. Second, the evidence for effectiveness is incomplete; the paper fails to investigate or explain counter-intuitive results, such as why adding video transcripts lowers performance on several WEBARENA tasks, and provides no failure analysis to add depth to the aggregate scores. Finally, the evidence for the method itself is weakened by a high-level methodological description that omits key implementation details, making it difficult to assess the correctness and robustness of the experiments.", "audience_yn": "Yes", "audience_explanation": "The field of autonomous agents is a highly active area of research. The problem of endowing agents with procedural knowledge to operate diverse software is a fundamental challenge. Researchers and practitioners in machine learning, particularly those focused on agents, reinforcement learning, and multimodal models, would be very interested in this paper's approach to inference-time learning from unstructured videos. The topic is timely and the empirical findings, despite their current shortcomings in presentation, point to a valuable research direction.", "requested_changes": "1.  The paper must include a dedicated analysis of the computational costs. This should include: (a) the complexity of the video processing pipeline; (b) the measured latency added at each inference step by the trajectory selection mechanism; and (c) a discussion of the practicality of the approach and potential optimizations. This is essential to validate the \"inference-time\" claim.\n2.  The authors must supplement the quantitative results with a qualitative analysis. This should include: (a) an analysis of several failure cases, identifying the root cause of the failure (e.g., retrieval, processing, or selection error); and (b) an explanation for counter-intuitive results, specifically the performance degradation observed when using only transcripts on WEBARENA (Table 2).\n3.  The methodology section requires more detail to ensure reproducibility. Specifically, explain the process for \"Demonstration trajectory construction,\" addressing its $O(N^2)$ complexity and scalability. Also, clarify how the \"3.6 videos per task\" statistic was calculated.\n4.  The paper would be significantly improved by an appendix section with a concrete, step-by-step example of the pipeline. Show key frames from a video, the extracted actions, the generated objective, and the final structured trajectory provided to the agent.", "broader_impact_concerns": "The proposed agent could be misused if it learns from tutorials for malicious activities. Furthermore, its reliance on unvetted online content raises safety and reliability concerns, as it may imitate incorrect or unsafe procedures.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)