ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks
Authors: Philip Schroeder
Ondrej Biza
Thomas Weng
Hongyin Luo
James R. Glass
Venue: NeurIPS 2025 poster ()
Keywords: vision-language model
video reasoning
embodied intelligence
recursion
Primary area: general_machine_learning
TL;DR: We introduce ROVER, a recursive framework that improves the video reasoning accuracy and efficiency of vision-language models in embodied settings.

Abstract: Vision-language models (VLMs) have exhibited impressive capabilities across diverse image understanding tasks, but still struggle in settings that require reasoning over extended sequences of camera frames from a video. This limits their utility in embodied settings, which require reasoning over long frame sequences from a continuous stream of visual input at each moment of a task attempt. To address this limitation, we propose ROVER (Reasoning Over VidEo Recursively), a framework that enables the model to recursively decompose long-horizon video trajectories into segments corresponding to shorter subtasks within the trajectory. In doing so, ROVER facilitates more focused and accurate reasoning over temporally localized frame sequences without losing global context. We evaluate ROVER, implemented using an in-context learning approach, on diverse OpenX Embodiment videos and on a new dataset derived from RoboCasa that consists of 543 videos showing both expert and perturbed non-expert trajectories across 27 manipulation tasks. ROVER outperforms strong baselines across three video reasoning tasks: task progress estimation, frame-level natural language reasoning, and video question answering. We observe that, by reducing the number of frames the model reasons over at each timestep, ROVER mitigates model hallucinations, especially during unexpected or non-optimal moments of a trajectory. In addition, by enabling the implementation of a subtask-specific sliding context window, ROVER's time complexity scales linearly with video length, an asymptotic improvement over baselines.

ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks
Philip Schroeder! Ondrej Biza²
Thomas Weng?
Hongyin Luo!
James Glass!
MIT CSAIL
RRA Institute
pachro@mit.edu
Abstract
Vision-language models (VLMs) have exhibited impressive capabilities across di- verse image understanding tasks, but still struggle in settings that require reasoning over extended sequences of camera frames from video. This limits their utilityy in embodied settings, which require reasoning over long frame sequences from a continuous stream of visual input at each moment of task attempt. To address this limitation, we propose ROVER (Reasoning Over VidEo Recursively), framework that enables the model recursively decompose long-horizon video trajectories into segments corresponding to shorter subtasks within the trajectory. In doing so, ROVER facilitates more focused and accurate reasoning over temporally localized frame sequences without losing global context. We evaluate ROVER, implemented using an in-context learning approach, on diverse OpenX Embodiment videos and new dataset derived from RoboCasa that consists of 543 videos showing both expert and perturbed non-expert trajectories across 27 robotic manipulation tasks. ROVER outperforms strong baselines across three video reasoning tasks: task progress estimation, frame-level natural language reasoning, and video question an- swering. We observe that, by reducing the number of frames the model reasons over at each timestep. ROVER mitigates hallucinations, especially during unexpected or non-optimal moments of trajectory. In addition, by enabling the implementation of subtask-specific sliding context window, ROVER's 's time complexity scales linearly with video length, asymptotic improvement over baselines. Demos, code, and data available at: https::/roverr: vlm. gi github.rt. io
Introduction
Vision-language models (VLMs) have demonstrated remarkable generalization capabilities across a wide range of tasks, including image captioning, visual question answering, and grounding language in complex visual scenes 3 22 [2]. These models excel at extracting high-level semantic understanding
from visual inputs and can generate fluent natural language that reflects this understanding [18] (66). As result, VLMs have become a key component in efforts to build general-purpose Al systems.
Given their success, VLMS are increasingly viewed as promising foundations for embodied intel- ligence: systems that perceive, reason about, and act within the physical world [38] 9 19 15]
31
57 25 54 63). Embodied tasks require VLMs to recognize and reason about what is happening at each moment in time (given a continuous stream of visual input from camera vidco) when attempting to complete task. Existing frameworks for VLM reasoning in this setting either perform highly or attempt to reason over the entire sequence by concatenating all video frames into single context (II. Both strategies have limitations: the former sacrifices global context, while the latter is computationally expensive and overwhelms the model with irrelevant or redundant visual input. In this work, we seek advance VLM capabilities in embodied settings by enabling high-precision frame-by-frame reasoning, without compromising efficiency losing global context for the full task trajectory. With this motivation, propose Reasoning Over VidEo Recursively (ROVER).
localized reasoning over a small set frames [57 25 54 63
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Task:
"Thww
"Open
Task: "Grasp door handte"
[oo]
[oo]
[op]
The gripper front of
The oripper is in front of microveve... The robot needs to: "Grasp door handle"
gripper
0%
robot "Open
door"
[ote]
[oo]
[o:]
The robot opened microueve door.
The robot needs to: "Put steak in nicrowave"
The rebot needs to: "Pull door open"
The robot fully grasped handle.
The gripper noved closer to the door...
Task progress: 48%
[0
[014]
[oo]
The robot put the steak in the microwRve. robot "Close nicrowave
The robot opened the nicrowave door.
Task comp complete.
The robot fully grasped the door...
Task progress: 180%. Task is cosplete.
Figure 1: ROVER is recursive framework for reasoning over camera video that decomposes task into subtasks to maintain a compact temporal context, improving reasoning accuracy and efficiency.
ROVER is a framework that enables the model to recursively decompose the task corresponding the given video input, reducing the number of camera frames the VLM must reason over at any given moment during the trajectory (Figure [1. Instead of generating a long, single line of reasoning spanning all timesteps of the video input for task attempt (e.g., opening a door), ROVER decomposes the task and generates separate line of reasoning for each subtask (e.g., grasping the door handle). When a subtask complete, the corresponding line of reasoning terminates and new one is created for the video input for the next subtask (e.g., pulling the door open). We show that the decomposition not only improves accuracy by focusing reasoning on relevant temporal segments, but also enables the implementation of subtask-specifs sliding context window, which further reduces the number frames the model must reason over at each moment of a trajectory.
We evaluate ROVER, implemented using an in-context learning approach, in the setting of robotic manipulation tasks using a large-scale dataset videos collected from robot-mounted and third- person camera viewpoints during both successful and unsuccessful task attempts. We create this dataset by automatically perturbing expert demonstrations collected RoboCasa [39 produce diverse trajectories, ranging from near-optimal to fully random action sequences. In addition, we compute ground-truth task progress estimates each timestep based on geometric distance to goal states. The generated dataset includes 543 videos across 27 tasks, each collected in random kitchen scene. We leverage this dataset to evaluate ROVER across three benchmarks for embodied reasoning over camera video: 1) frame-level task progress estimation, 2) frame-level natural language reasoning, and 3) video question answering (QA). We also test ROVER on previous benchmarks based on diverse real-world OpenX Embodiment videos
41
ROVER outperforms baselines across all settings: it achieves a stronger correlation with ground truth task progress, lower frame-level reasoning етrоr rates, and higher video QA accuracy. Specifically, we observe that ROVER reduces model hallucinations, especially during moments of trajectory that deviate from expected or optimal behavior. We show these types of hallucinations during non-optimal states are most likely to occur when the model is reasoning over long sequences of camera frames. ROVER mitigates these hallucinations by reducing the number of frames the model must reason over at each timestep. In addition, by leveraging a sliding window within each subtask, ROVER's s time complexity scales linearly with video length, asymptotic improvement over baselines. Overall, the contributions this work including the following:
We propose ROVER, a recursive framework that improves video reasoning accuracy and effi- ciency in embodied settings.

---

We introduce protocol to generate diverse non-expert trajectories with ground truth progress labels from expert demonstrations in simulation.
We release large-scale video dataset featuring both expert and non-expert robotic task execu- tions, annotated with natural language descriptions and fine-grained task progress signals.
We establish video reasoning benchmark that evaluates models on frame-level progress estimation, frame-level natural language reasoning, and video QA across diverse task trajectories.
Problem setup
We model tasks goal-conditioned finite-horizon partially observed Markov decision processes:
M :
(I)
The tuple (S, A. P. R, G) forms Markov Decision Process, where S is the (unobserved) state space, A is the action space. P SxA ASS) is the state-transition function and R SSG + IR is the reward function. An agent observes the world through O S + A the observation function, where 52 is the of observations. Finally, I' the time horizon and G specifies the task semantically. Conditioned on a history he = (a<a, Oge) E H and goal G, we aim to find
policy HXG + A that maximizes the 7-discounted return Jx = E Eto 7 R(s;, az g)
The condiiontt value function V/ is defined as the expected cumulative reward when starting from an initial state 8o and following policy n
V (" ETT
y *R(s at g) 80
Given a task description g E and sequence of observations {or)?=1 from video input, our goal is to generate text, {me) [m following each observation that reasons in natural language about the task trajectory relative to the goal g. including estimating task progress, at every timestep. Section outlines the recursive framework we propose for this reasoning task. To evaluate reasoning accuracy, we generate a diverse dataset of videos, D, exhibiting task attempts with varying levels of expertise by inserting random perturbations within given dataset, Dere. of expert trajectories in simulation (described in section 4) In addition, we compute ground-truth task progress estimates at each timestep of the generated videos based on geometric distance to goal states (described in section 4.2).
3 ROVER framework for reasoning over video trajectories
Given a task description and visual input Ω from camera video, ROVER recursively generates sequences of reasoning at each timestep of the task attempt. The sequences of reasoning are comprised of camera frames for each timestep, interwoven with natural language text (Figure D. ROVER can be implemented using the recursive function shown in Algorithm 1. The function takes two inputs: the context for the given task, and the sequence of tokens, Y. that have been provided or generated so far for this task. includes text tokens generated by the VLM and image tokens representing the frames from the camera input. We treat token sequence (e.g., and Y) list of tokens.
ROVER begins with = and includes text tokens of the initial task description (e.g.. "Thaw steak in the microwave") and the first obser- vation (image tokens of the camera frame). The model generates, ((c + Y). one token at a time conditioning the given context, appended with the growing token sequence, Y.
Algorithm 1 ROVER
1:
function ROVER(C, Y)
2:
while True do
3:
Y = Y O(c+
4:
if Y progresses to next frame then
If Y includes the tokens for retrieving the next observation (details explained below), then the image tokens for the camera frame at the next
5:
Y= + Opext
6:
else if Y specifies a task then
7:
y= Y + vb( ROVER(( ())
timestep, Opents are appended to Y. If Y speci-
8:
else if Y indicates task complete then
fies new subtask, then a new line of reasoning
9:
return Y
created with ROVER(Ø(Y). (]) where Y for the child process is initialized as empty list
3

---

and e is based on the token sequence of the parent, o(Y). When the child process ends, its output tokens are appended to the parent's 's token sequence and the parent continues generating. If Y indicates the task is complete, or the end of the video is reached, then the process returns Y (its full token sequence).
Implementation of Φ and ve functions. The & function defines the context for a child process based on the full token sequence of the parent at the time the child is spawned, including tokens directly generated by the parent or returned as output from previous child processes of that parent. The ei function defines the output tokens of child based on its full token sequence at the time terminates. In our implementation of ROVER (as depicted in Figure I returns the new task description provided by the parent along with the last frame of the parent line of reasoning. We implement & as a function that returns the final frame, and corresponding final text description, of the child process.
Special text tokens for retrieving next frame and specifying new subtasks. When the VLM is done generating the natural language text tokens for reasoning about the current camera frame, it either moves on to the frame for the next timestep or specifies new subtask. The next frame of the video can be retrieved by generating the text tokens "[next-frame]].. A new subtask can be specified by generating the text tokens "The robot needs to: (new_ "uee_subtask]" !", as depicted in Figure !
Sliding context window for video. In our implementation of ROVER, we find that using a sliding window at each timestep not only improves inference efficiency, but also mitigates observed problems with hallucination. When appending a new frame, Y = Window(> (Y + Onext), the window extracts three frames: the first frame of the current line of reasoning and the most recent previous frame (each paired with their previously generated text descriptions), along with the newly added next frame. We observe a complementary effect between the sliding window and the recursive decomposition. The window forces the model to focus on the most relevant frames for the current subtask and the current timestep, while the recursive specification of new subtasks ensures the current subtask description and first frame of the subtask-specific window are continually updated as task milestones are achieved during the video. Further, by capping the maximum number of frames included in the context to three, the inference time complexity of ROVER scales linearly with the total video frame count. This provides an asymptotic improvement over baselines, which scale quadratically with total frame count.
4 Generating diverse video trajectories and ground-truth value estimates
To test vision- laiiuanl reasoning across diverse video trajectories, we seek to use a source dataset of expert demonstrations, Drre, for a task in simulation to build a dataset of trajectories, DD exhibiting a wide spectrum of expertise with that task, along with value estimates for all states. We focus on robot manipulation tasks and, following prior work [36 11. we assume tasks consist of a known sequence of object-centric subtasks. If we let E- f2,, .2.... .} be the set of entities (objects) in task M, we assume that tasks consist of sequence of object-centric subtask trajectories = (Tx(Er;) ) f==1 where the manipulation in each subtask trajectory T: (e,,,) is relative to a single object's coordinate frame (er, E E). We include details regarding how this sequence is specified for each task in Appendix M
4.1 Non-expert trajectory generation
In this section, we outline an approach which, given a set of expert demonstrations, D Dares generates a dataset of trajectories, D, exhibiting varying levels of expertise, ranging from nearly expert to fully random action sequences. Each expert demonstration, 7 E D Dare. is decomposed into object-centric segments - (T.6(e,,, (e,,)) (see Appendix M for full details). Following MimicGen [36), each
subtask trajectory Tf [(> )f corresponds to manipulation relative to single object e..
Non-expert trajectories are derived by injecting random deviations into these expert demonstrations (Figure 2) For a given subtask trajectory T.". we construct a corresponding non-expert version
T, (sNN ) by inserting random actions at selected moments Each time in Q, marks a branching point, at which T/N diverges from 75 for some number of steps, w. The index of each step in the deviation specified based on the index of the expert state, q. at which the deviation began, along with the number of steps, (ranging from 0 to w - 1), it has deviated from that expert state. For each step during the deviation (g.5) w-1 a randomly sampled action N (A) is executed, where (AA denotes the uniform distribution over the action space.
Q S (1,..., "I....,T,5. T.F.,

---

expert trajectory: "pick up the salt shaker from the counter"
S4
a
220
100
state interpolation
Task progress
75
52.
a
random actions a U(A)
3
Step
Figure 2: Generating diverse video trajectories by inserting random deviations during expert path.
The deviation leads to a sequence states and actions {$g.5 ag 1 1w-1 that diverges from the expert trajectory, where $..u is the expert state from which the deviation began. Following deviation, the trajectory can either terminate or start to recover to the expert trajectory. Recovery is implemented by interpolating between the deviated state and downstream expert state (Figure 2 Specifically, given
..V. the final deviated state $q.w and a selected future expert state 9+h (for some / > 0), we generate sequence interpolated states:
N g,w.=
(1= 5qg + a2 Sgth
02
Tiintrrp
miter
gradually transition back the expert trajectory. The index of each step during recovery is specified based on the index of the expert state, q. at which the original deviation began, the deviation length, w, and the number of interpolation steps, (from 1 to Rimeerp) that have occurred. Aiintrp determines the smoothness of the recovery. This procedure results in trajectories that deviate to varying extents from the expert trajectory. We ensure diversity across trajectories by varying deviation timesteps, number and magnitude of deviations (including nested deviations), and recovery points.
4.2
Value functions
For each task, define ground-truth value function to evaluate the accuracy of VLM reasoning throughout the video trajectories. robotics, value functions can be difficult to define due to task heterogeneity. We follow prior work [14] 29 BT
45 46 53) in modeling value using task progress, which provides a normalized temporal measure of how far an agent has advanced toward completing task. Formally, define progress-basse based value function, V S G - observation and goal specification to a scalar value between 0 and 1.
[0, 1]. mapping an
Goal-focused distance measures. To quantify progress, define subtask-specific metrics based on spatial relationships in the observable state. For each subtask, we identify key geometric distances that reflect meaningful aspects of completion and continuous and monotonic with respect to progress. These include the sum of distances between the contact points of the robot end effector, (ri, r2,- ... , rc). and the corresponding contact points of the target object, (1i, 12, ,lc), in the scene, denoted y along with the distance between the current position of the target object and its respective goal position, denoted yf e..
112
Σ
p
vi
Pt
Pf
CTi
where e,; is the -ppciffcc target object discussed above, p3 and p are positions of the jth robot and object contact point, respectively, at time f, Pt the position e,, at time t, and p, "T: the target position of e,, All distances are computed using the L2 norm (Euclidean distance). The environment includes B entities, each of which is represented by position in three dimensional Cartesian space within global coordinate frame. The total goal-focused distance, y {yr 22: is
is
5

---

(8)
(b)
ROVER
Tempora Concat LocalCancet
+I- LV
GVL
LocalCenca::
Corr with ground trohh
Can groure toun
0.7-
0.6
0.4
e72 axd lavd lend lead lovd lav? lexd evz lavd lave lest Pidlaand-pllce
lod lave leed onielf appliances
les lasz lovs leed levd Thogle
Toople Inturmlleruts
Open/cliss
applamnces
Corr 1ith ground tutn
Con with ground touns
0.4-
0.2
M03 Microwavee
lees
uet les2 lec
lnst lesz Arrange
lnc2 lex2 lev3 teez lev2 lex3 lest
Microwaave thawing
Resbeck
Arragge
Prepare
Preseaat
Restock
Prepare
Pre-scak
pantry
vegetables
thawing
partryy
veputables
pan
Figure 3: Mean and standard error of correlation between ground-truth values and progress values predicted for all videos (a) and stratified across trajectory level (b). Highest-level videos for each task show near-expert completion. Amount of non-expert behavior increases as level decreases.
the sum of these metrics, y1 (1 (1- B) B) vi= + 3 e-j where f is scalar with range [0. 1] that is manually tuned for each object-centric subtask to ensure the degree to which y and yd contribute y is proportional how meaningful each component is to the overall subtask completion.
Trajectory-focte focused distance measures. The goal- focusfocueed metrics are necessary to track what we value in the task (e.g., moving the gripper closer to the target object, the target object moving closer to the goal position). However, they may fail to capture important nuances real-world trajectories. For instance, the shortest Euclidean path between the robot gripper and target objects may intersect with obstacles that must be avoided. To address this, we refine the value function by leveraging known expert states within each trajectory. Within the expert demonstrations, we identify sparse set of keypoints, x = { V that form the local maxima in the goal-focused distance, {yr } for given object-centric subtask. Since these keypoints derive from the expert demonstrations, we know that, despite forming local maxima in y, they represent intermediate states that lie along high-value trajectories. For each timestep in a non-expert trajectory, we identify the nearest downstream expert keypoint and use this as an auxiliary signal to adjust the local value estimate.
Ue = =|$
12. kiee = min (k, E K. K; > t},
where j = 2 = when & Q
where s is the environment state at the current step of the non-expert trajectory and &f E BR3x the full environment state at time knext during the expert trajectory.
Final value estimates. We transform the raw distance measures for the full subtask trajectory into normalized progress values, V = {v between and by inverting and rescaling them:
Ut - d, + max (d)
de = y ut, d [d
min(d) + max (d)
To form the value estimates for the full trajectory,
sequence together the value estimates corresponding to each subtask, adding the value estimates of each subtask to the final value of the previous subtask and, when complete, rescaling the list of values to the range [0, 1].
(e,,
5 Experiments
We conduct large-scale experiments to evaluate ROVER across diverse tasks for embodied reasoning over camera video. The baseline methods include the state-of-the-ar value model LIV [32], multi- modal contrastive model [3 fine-tuned with value learning objective. We also include the best existing in-context learning approach, Generative Value Learning (GVL) (3I] and an ablated version of GVL, called TemporalConcat, that processes frames in temporal order. Further, we include a local reasoning baseline, LocalConcat, that processes frames temporal order, but only includes the last three frames of the video in the model's context at each timestep. Similar to GVL, we implement ROVER using an in-context learning approach, with the same prompting used for all backbone VL.Ms (details in Appendix D) We test ROVER using various backbone models. The main results

---

Task: "Put the water bottle in the cabinet."
ROVER
0%
10%
16%
7%
3%
0%
GVL
0%
12%
18%
15%
34%
23%
TemporalConcat
0%
21%
44%
62%
87%
Ground truth
0%
9%
17%
8%
5%
3%
Figure 4: ROVER exhibits more accurate reasoning and progress prediction during non-expert states.
GPT-4o, and Examples showcasing ROVER video reasoning, along with anonymous links to all code and data, are provided at: hattps://rover: vlm. github. 1o/ /
based on Gemini-1.5-Pro. Appendix ! includes results for Qwwen22.-VL-332--nnntt
We RoboCasa [39 as the simulation environment and leverage the demonstrations they provide (collected via human telcoperation) as the source dataset expert trajectories (details in Appendix E) Our generated evaluation dataset, comprising trajectories that exhibit wide range of task expertise, includes 543 videos across 27 tasks (Appendix [B), each collected a random kitchen scene. The videos separated into levels based the amount of the task completed during the video (Appendix B). The highest-level videos in each task group show full task completion with near-expert behavior.
5.1 Vision-language reasoning tasks with generated video dataset
Our sonullation-genertte video trajectories, each annotated with dense value estimates, enable the evaluation of multiple vision-language reasoning tasks that test fine- -eaiigr temporal understanding and visual grounding. We separate the evaluation into the following three tasks: 1) frame-level task progress estimation, 2) frame-level natural language reasoning, and 3) video question answering (QA). Further details regarding the task design and performance metrics are provided in Appendix
5.2 Task Frame-level task progress estimation
This benchmark involves predicting task progress at each timestep of video (see Appendix C. for details). Following prior work (31), we implement this by having the model predict integer values representing the percentage progress achieved relative to the first frame of the video, which is always assigned a progress value of zero. We implement ROVER such that the model predicts task progress values up 100 for each subtask, depicted in Figure I To form progress values for the full task horizon, we divide the values within each subtask by the total number of subtasks and add them to the value for the last timestep of the previous subtask (see Appendix
D for details).
ROVER shows higher correlation with ground-truth task progress estimates. For videos that exhibit task completion with near-expert behavior (i.e., the highest level within each task group), ROVER, GVL, TemporalConcat, and LocalConcat achieve a Pearson correlation near or above 0.5 (Figure 3 and 1.2 distance below 200 (Appendix (A.]), relative the ground-truth value estimates, across most task groups. However, for videos with incomplete task execution, GVL, TemporalConcat, and LocalConcat deviate significantly from the ground truth (as depicted in Figure 4.. These deviations become more extreme as the trajectory level decreases (i.c., as the number of non-expert states in the video increases).
ROVER achieves higher correlation with frame number on real-world expert demonstrations. In Appendix KJ we include the same value estimation analysis conducted in [3]] using 1,000 videos from 50 real- world datasets from OXE [4]. We see that ROVER achieves higher correlation with ground-truth value estimates (which are based frame number) across the OXE datasets, most of which contain human-collected expert demonstrations.
5.3 Task 2: Frame-level natural language reasoning
At each frame of the video, ROVER, along with baselines, generates a description of the frame before predicting task progress for that timestep. We cross-reference these descriptions with information about the ground-truth simulator state at each timestep to calculate the reasoning error rate and
7

---

(R)
ROVER
(b) 100
TemesraKennt
GVL
LecalGancat
40
Erar rate 1811
60-
Enoor rate
80
40-
40
20
20
0
Pick-nddllace
oneeff dndms/trawerss
Pick-
Opengeiose doors/drwwers
app lances
Turn onlelf
Toggle fituresknobss
nutures/enebs
100
Entor rate (8)
Emor mat (N4
80
80
20
lex2 lev2 lex3 lavé Ind
lev2 lex2 lev3 Rectock pantry
wv2 ex2 ev2 Arrange vegetačles
lexz lex2 lov3 Prepare
tex lax2 lova leve Pre-soak
Microwave
Restock
Arrange
Pregare
Pre-soak
Microwave
thawing
pan
Figure 5: Mean and standard error of reasoning error rate (percentage of frames model states something that is verifiably wrong) for all videos (a) and stratified across trajectory level (b).
success rate (described in Appendix prompt the model with the generated description paired with the ground-trute truth information and ask the description includes any statements that are verifiably true or false relative to the ground truth.
treat this evaluation as a language model task, where
ROVER achieves lower error rate in frame-level reasoning during video. The reasoning error rate similar for all methods for videos containing near-expert task completion (Figure 5). However, as videos deviates from expert behavior, the error rate increases significantly for GVL, femporalConcar and LocalConcat. We observe similar pattern for reasoning success rate (Appendix (A.2). These trends mirror the results of the progress prediction task, suggesting that errors in progress prediction result from errors the natural language descriptions the frame that precedes the progress prediction at each timestep. We further evaluate the nature of these errors in the video QA analysis below.
ROVER reduces errors by focusing context. When stratifying results across state type (expert, non- expert, and interpolating states) and context length Appendix we see that GVL, TemporalConcat, and LocalConcat are most likely to error when encountering non-expert states and when the number of frames included in the context extends beyond 10. We observe similar pattern when stratifying the progress prediction results across state type (Appendix HD. ROVER reduces these errors by using recursive decomposition, along with a sliding window, to narrow the context at each timestep.
5.4 Task 3: Video QA
20
GVL
The video QA task includes questions about whether specific events occur (and the time they occur) during the video (de- tails in Appendix C.3 We generate an- swers to each question for this task using the frame-level descriptions produced by each method. For each question, we use a language model to generate an answer given all frame descriptions concatenated together. In doing so, the QA performance provides secondary measure of the quality of the frame-level natural language reason- ing of each method.
Frequueccy
15
ROVER
-20
-10
10
Difference time (in frames) between VL VLM-ddeniffed time actual time of events that occurred video
Figure 6: Difference in time (in frames) between VLM- identified time and actual time of events that occurred in video. One second in real time corresponds to -3 frames. Negative frame difference indicates VLM states something occurs before it actually occurs during video.
ROVER reduces hallucinations during videos exhibiting non-expert behavior: ROVER shows significantly higher accuracy than GVL, TemporalConcat, and LocalConcat on the video QA bench- mark across all task groups (Appendix[A.33 The video QA results reveal a hallucination problem with GVL., TemporalConcat, and LocalConcat, where the VLM is likely to state that an event occurred during video, regardless of whether actually occurs. This is illustrated by the low precision (20 to 50%) and high recall values (near 100%) across all task groups for GVL, TemporalConcat, and

---

LocalConcat (Appendix A.3 The hallucination problem is also highlighted in the analysis of the distance between the time when something occurs video and the time when the VLM states that it occurred (FigureG) We see that, even when GVL. correctly states something occurs during videa, it much more likely than ROVER to state this prematurely (as shown by the negative frame difference in Figure 6p.
5.5 Error analysis
hand-reviewed 100 real-world videos and 100 videos from simulation to identify the most common types of errors for each method (Table In Appendix C.4. we outline the full list of error types and provide examples for each.
The most common enror type for ROVER is specifying incorrect subtasks, where 9% of videos and 7% of simulation videos include least one occurrence of this error. The most common error type for GVL is perception errors, where 47% of real-world videos and 32% of simulation videos include least one occurrence this error. Further, we observe that 83% of perception errors from GVL occur when there are over 10 frames of the video included in the context. This supports our findings that the VLM is prone to perception errors, such as hallucination, when attempting to reason over many frames of video. ROVER mitigates these perception errors by shortening the number frames the model must reason over at each timestep of the video (through the synergistic combination of recursive decomposition and sliding context window). Overall, we find that ROVER introduces the possibility of decomposition errors, but dramatically reduces the total error rate by mitigating the model's tendencies toward hallucination.
Table 1: Error rates (%) observed among 100 real-world videos and 100 videos from simulation
Error type
Real-world videos
Simulation videos
GVL
ROVER
GVL
ROVER
Incorrect subtasks
NA
9%
NA
7%
Redundant subtasks
NA
3%
NA
4%
Perception error
47%
7%
32%
5%
Reasoning error
14%
4%
14%
4%
Other
15%
3%
11%
3%
Total
76%
26%
57%
23%
5.6 Ablations
We ablate ROVER to demonstrate how different algorithmic design decisions impact the performance gains we observe relative to baselines. Further, we show that ROVER is robust to video length and frame rate (Appendix H), camera view (Appendix []. and backbone VLM (Figure 7p. Appendix D)
We test two ablations of ROVER (details in Appendix Lb. One version ablates only the sliding window while the other ablates only the recursive reasoning. Consistent with prior work (3I) we observe that baseline methods such as TemporalConcat often hallucinate steady, monotonically increasing task progress values, even when the video includes significant stalls or regressions. GVL mitigates this problem via random frame shuffling, which disrupts the strong temporal priors and forces the model to assess each frame more carefully (3I). We observe that the sliding window (even without recursion) similarly reduces hallucination by retaining temporal context while avoiding overly rigid sequential processing. Our ablations show that the window-only variant of ROVER significantly outperforms TemporalConcat and matches exceeds GVL's performance short-horizon tasks, such as turning on/off appliances (Figure ( We see that adding the recursive decomposition further improves performance for tasks consisting of numerous subtasks, such lock-and-plcce tasks.
6 Related work
There is significant prior work leveraging VLMs in embodied reasoning settings [3 9 19 15 32
54 63 have applied VLMs for reasoning over video sequences, either by attending to small number of keyframes processing the entire trajectory. However, these approaches struggle to balance fine- fine-grained reasoning with the preservation of global task context. Separately, decomposition- based reasoning has been explored in both language and vision-language language
Methods such as 31 57
23

---

(8)
ROVER
GM
(b)
Cant eith gruns toen
6.8
Recurssn on without ding window
Tanosralaanca
6.4
Riceann-ppac place Ooeriiinee
Tim ondeef
Toople
Nicneawe
Restock
Arrang
Pregare
Fre-soak
OP 4o
Oweet
Figure 7: Mean and standard error of correlation between ground-truth value estimates and progress values predicted across all videos for (a) design ablations and (b) changing backbone VLM.
Recent work has also explored structured prompting and 44 though primarily in static text-centric settings. ROVER builds on these ideas by introducing a recursive, subtask-oriented video-reasoning framework to address challenges of continuous visual input in embodied tasks, allowing for precise and scalable temporal reasoning without sacrificing global context. More discussion in Appendix 0
domains
16
28 37
recursive task breaking in VLMS 21
Limitations
ROVER improves reasoning over videos by recursively decomposing the given task into subtasks and, in doing so, maintaining a compact temporal context that mitigates hallucination. However, when decomposition fails (e.g., by introducing unnecessary may become fragmented or misaligned with the actual task progression. addition, we implement ROVER with in-context learning approach. We leave future work the evaluation of fine-tuning methods that can improve ROVER's performance with frame-level reasoning and value estimation.
invalid subtasks), the resulting reasoning
Broader impact
In this work, show how ROVER improves VLM reasoning in embodied settings. VLMs being increasingly deployed to autonomously interact with external environments. By improving this capacity, our work has the potential for amplifying risk associated with automated decision-making Addressing these risks requires careful consideration ethical guidelines and robustness checks.
Conclusion
We present ROVER, recursive framework that improves VLM reasoning over video sequences in embodied tasks. By decomposing complex tasks into subtasks and enabling localized, temporally bounded reasoning within each, ROVER effectively reduces cognitive load on the model and mitigates common failure modes such as hallucination during off-nominal behavior. Our experiments across three benchmarks (progress estimation, frame-level reasoning, and video QA), along with previous benchmarks based on diverse real-world OXE videos, demonstrate consistent improvements over strong baselines, particularly in handling diverse, suboptimal trajectories. In addition, by enabling use of subtask auptask- sliding window, ROVER's time complexity scales linearly with video length, an asymptotic improvement over baselines. By combining architectural insights with new large-scale dataset and evaluation protocol, our work lays a foundation for more robust and scalable embodied reasoning systems grounded in real-world video input.
10 Acknowledgements
This work was supported by the MIT-IBM Watson AI Lab and Quanta Computer, Inc.
References
[1] Ahn, M.. Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C.,, Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022). Do can, as say: Grounding language in robotic affordances. arXiv preprint 2xx422204.01661.
[2] Andreas, J., Rohrbach, M., Darrel,, T., and Klein, D. (2016). Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39 48.
10

---

[3] Bai. J.. Bai, S., Chu, Y., Cui, Z., Dang, K. Deng. X., Fan, Y., Ge. W., Han, Y., Huang, F., et al. (2023). Qwen technical report. arXiv preprint (iKi::2309.160000
[4] Baumli, K., Baveja, S., Behbahani, F. Chan, H., Comanici, G., Flennerhag. S., Gazeau, M., Holsheimer, K.. Horgan, D.. Laskin, M., al. (2023). Vision-languees models source rewards. arXiv preprint arXiv:2312.0914
[5] Bhateja, C., Guo, D., Ghosh, D., Singh, A., Tomar, M., Vuong, Q. Chebotar, Y., Levine, S., and Ku- mar, A. (2023). Robotic offline from internet videos via value-function pre-training. arXiv preprint arXiv:2309.3300
[6] Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X.. Choromanski, K., Ding. T., Driess, D., Dubey, A., Finn, C., et al. (2023). Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arv:::330 15818.
[7] Brohan, A.. Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C.,, Gopalakrishnan, K., Hausman, K., Herzog. A., Hsu, J., et al. (2022). Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:2212.0683
[8] Chen, A. S., Nair, S., and Finn, C. (2021). Learning generalizable robotic reward functions from "in-the- wild" human videos. arXiv preprint (xXiv:203.1668)
[9] Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L.,, and Xia, F. (2024). Spatialyim: Endowing vision-language models with spatial reasoning capabilities, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455-14465.
[10] Dasari, S., Ebert, F., Tian, S., Nair, S., Bucher, B., Schmeckpeper, K.. Singh, S., Levine, S., and Finn, C. (2019). Robonet: Large-scale multi-robot learning. arXiv preprint arXiv:1910.1122
[11] Di Palo, N. and Johns, E. (2022). Learning stat tasks with one demonstration via self-replay. In Conference on Robot Learning, pages 1180-1189, PMI.R.
[12] Ding, Y., Zhang, X., Paxton, C., and Zhang, S. (2023). Task and motion planning with large language models for object rearrangement. arXiv preprini arXiv:2303.062.
[13] Ebert, F., Yang. Y., Schmeckpeper, K., Bucher, B., Georgakis, G., Daniilidis, K., Finn, C., and Levine, S. (2021). Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109, 13396.
[14] Eysenbach, B., Salakhutdinov, R., and Levine, S. (2020). C-learning: Learning to achieve goals via recursive classification. arXiv preprint 1xXix20011008000
[15] Gao, J.. Sarkar, B., Xia, F., Xiao, T., Wu, J., Ichter. B., Majumdar, A., and Sadigh, D. (2024). Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462-12469. IEBE.
[16] Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. (2023). Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR.
[17] Gupta, N., Lin, K., Roth, D., Singh, S., and Gardner, M. (2019). Neural module networks for reasoning over text. In International Conference Learning Representations.
[18] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arX1v:2009.033000.
[19] Hong. Y., Zhen, H., Chen, P.. Zheng, S., Du. Y.. Chen, Z., and Gan, C. (2023). 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Syxtems, 36:20482-20494,
[20] Huang, S., Jiang. Z., Dong, H., Qiao, Y. Gao, P., and Li, H. (2023a). InstructZact: Mapping multi-modalle instructions to robotic actions with large language model. arXiv preprint arXiv:2305.11176
[21] Huang, W. Xia, F., Xiao, T., Chan, H. Liang, J., Florence. P., Zeng, A., Tompson, J., Mordatch, Chebotar, Y., et al. (2023b). Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pages 1769-1782. PMLR.
[22] Hurst, A., Lerer, A., Goucher. A. P. Perelman, A.. Ramesh, A.. Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. (2024). Gpt-4o system card. arXiv preprint arx22:2410.2127.
11

---

[23] Kalashnikov, D.. Irpan. A., Pastor, P., Ibarz, J.. Herzog, A., Jang. E., Quillen, D., Holly, E., Kalakrishnan. M.. Vanhoucke, V., al. (2018). Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR.
[24] Khot, T., Trivedi. H., Finlayson, M.. Fu, Y., Richardson, K., Clark, P., and Sabharwal. A. (2023). Decom- posed prompting: A modular approach for solving complex tasks, The Eleventh International Conference on Learning Representations.
[25] Kim, C., Seo, S.-W., and Kim, S.W. (2025). Lamour: Leveraging language models for recovery in reinforcement learning. arXiv preprint arXiv:2503. 17125.
[26] Klissarov, M., D'Oro, P., Sodhani, S., Railcanu, R., Bacon, P.-L.,, Vincent, P., Zhang, A., and Henaff, M. (2023). Motif: Intrinsic motivation from artificial intelligence foedback. arXiv preprint 1r.0002310.00166.
[27] Kwon, T., Di Palo. N., and Johns, E. (2023). Language models as zero-shot trajectory generators. arXiv preprint arXiv:2310. 1604.
[28] Lee, S. and Kim, G. (2023). Recursion of thought: ddwidd--nddcongeee approach multi-context reasoning with language models. Findings of the Association for Computational Linguisticx: ACL 2023, pages 623-658.
[29] Lec, Y., Szot, A., Sun, S.-H., and Lim, J. J. (2021). Generalizable imitation lcarning from observation via inferring goal proximity. Advances in Neurul Information Processing Systems, 34:16118-1111 18-1113..
[30] Liang, J., Huang. W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng. A. (2023). Code as policies: I Language model programs for embodied control. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pages
IEEE.
[31] Ma, J., Hejna, J., Fu, C., Shah, D., Liang, J., Xu, Z., Kirmani, S., Xu, P., Driess, D., Xiao, T., et al. (2024). Vision language models are in-context value learners. In The Thirteenth International Conference on Learning Representations.
[32] Ma, J.J. Kumar, V., Zhang, A.,, Bastani, O., and Jayaraman, D. (2023a). Liv: Language- representa- tions and rewards for robotic control. In International Conference on Machine Learning, pages 23301-23320. PMLR.
[33] Ma. Y. J., Liang. W.. Wang. G., Huang. D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. (2023b). Fureka: Human-level reward design via coding large language models, arXiv preprint arxiv:2310. 12931.
[34] Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., and Zhang, A. (2022). Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint (0XX::220000000
[35] Mahmoudich, P., Pathak, D., and Darrell, (2022). Zero-shot reward specification via grounded natural language. International Conference on Machine Learning, pages 44743-1475, PMLR
[36] Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, I.,, Zhu, Y., and Fox, D. (2023). Mimicgen: data generation system for scalable rohot learning using human demonstrations. arXiv preprint arXiv:2310. 11200.15900 17596.
[37] Murali, V., Qi, L., Chaudhuri, S., and Jermaine, C. (2018). Neural sketch learning for conditional program generation. In International Conference on Learning Representations.
[38] Nag. S., Zhu, X., Song. Y.-Z., and Xiang. T. (2022). Zero-shot temporal action detection via vision-language prompting. In European Conference on Computer Vision, pages 681-66 1619697 Springer.
[39] Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, (2024). Robocasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.0255 02222223.......
[40] Nye, M., Hewitt, L., Tenenbaum, J., and Solar-Lezama, A. (2019). Learning to infer program sketches. In International Conference on Machine Learning, pages 4861 18700 PMLR.
[41] O'Neill, A., Rehman, A., Maddukuri, A., Gupta, A.. Padalkar, A., Lee, A.. Pooley, A.. Gupta, A..
Mandlekar, A., Jain, A., et al. (2024). Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA). pages
IEEE
[42] Prasad, A., Koller, A., Hartmann, M., Clark, P., Sabharwal, A., Bansal, M., and Khot, T. (2023). Adapt: As-needed decomposition and planning with language models. arXiv preprint a.X.:2331.00577..
12

---

[43] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.. Sastry, G., Askell. A., Mishkin. P. Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR.
[44] Schroeder, P., Morgan, N., Luo, H.,, and Glass, J. (2024). Thread: Thinking deeper with recursive spawning. arXiv preprint arXiv:2405. 17402.
[45] Sermanet, P., Lynch, C., Chebotar, Y.. Hsu, J., Jang, E., Schaal, S., Levine, S., and Brain, G. (2018). Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages I 1134-1141, IEEE.
[46] Sermanet, P. Xu, K., and Levine, S. (2016). Unsupervised perceptual rewards for imitation learning. arXiv:/6/2.06690
[47] Shafiullah, N. M., Rai, A., Etukuru, H., Liu, Y., Misra, I., Chintala, S., and Pinto, L. (2023). On bringing robots home. arXiv preprint arXiv:2311.16009
[48] Singh, L., Blukis, V., Mousavian, A., Goyal, A.. Xu, D.. Tremblay. J., Fox, D., Thomason, J., and Garg, A. (2023). Progprompt: Generating situated robot task plans using large language models. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523-11583 IEFF.
[49] Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Talbott, W., Metcalf, K., Mackraz, N., Hjelm, D., and Toshev, A. (2023). Large language models as generalizable policies for embodied tasks. arXiv preprint arXiv:2310.1 17722.
[50] Talmor, A. and Berant. J. (2018). The web as a knowledge-base for answering complex questions. In Proceedings the 2018 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 641-651.
[51] Tang, Y., Yu, W., Tan, J., Zen, H., Faust, A., and Harada, T. (2023). Saytap: Language to quadrupedal locomotion. arXiv preprint ar arXiv:2306.07500
[52] Team, G., Georgiev, P., Lei. V. I. Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan. Z., Wang. S., et al. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055:
[53] Tian, S., Nair, S., Ehert, F., Dasari, S., Eysenbach, B., Finn, C.,, and Levine, S. (2020). Model-based visual planning with sulf-supervised functional distances. arXiv preprint arXiv:2012.15372
[54] Venkataraman, S., Wang, Y., Wang, Z., Erickson, Z., and Held, D. (2024). Real-world offline reinforcement learning from vision language model feedback. arXiv preprint arXiv:241.00573.
[55] Wang, L., Xu, W., Lan, Y., Hu, Z.. Lan, Y., Lee. R. K.-W.,, and Lim, E. (2023). Plan-and-solve prompting: Improving zero-shot of-in-of-thought reasoning by large language models. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2609-2634.
[56] Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. (2024a). Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
[57] Wang, Y. Sun, Z., Zhang, J., Xian, Z.. Biyik. E., Held, D., and Erickson, Z. (2024b). R1-vlm-f: Reinforce- ment learning from vision language foundation model feedback. arXiv preprint arXiv:2402.0368)
[58] Wang, Y. Sun, Z., Zhang, J., Xian, Z., Biyik, E., Held, D., and Erickson, Z. (2024c). R1-vIm-f: Reinforce- ment learning from vision language foundation model feedhack. arXiv preprint arXiv:2402.03688
[59] Wang, Y.. Wu, Z., Yao, J., and Su, J. (2024d). Tdag: multi-agent framework based dynamic task decomposition and agent generation. arXiv preprint arXiv:2402. 101778
[60] Xie, T., Zhao, S., Wu, C. H., Liu, Y., Luo, Q. Zhong, V., Yang. Y., and Yu. T. (2023). Text2reward: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309. /1489.
[61] Yao. S., Zhao, J., Yu, D.. Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations.
[62] Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Gonzalez Arenas, M., Chiang, H.-T. L., Erez, T., Hasenclever, L., Humplik, J., et al. (2023). Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08%.
13

---

[63] Zeng, Y.. Mu, Y., and Shao. L. (2024). Learning reward for robot skills using large language models via self-alignment. arXiv preprini arXiv:2405.07162.
[64] Zheng, W., Sharan, S., Jaiswal, A. K., Wang, K., Xi, Y., Xu, D., and Wang, Z. (2023). Outline, then details: Syntactically guided coarse-to-fine code generation. In International Conference Machine Learning, pages 22403-42415 PMLR.
[65] Zhu, Y., Wang, Z., Merel, J., Rusu, A., Erez, T., Cabi, S., Tunyasuvunakool, S., Kramár, J., Hadsell, R., de Freitas, N., et (2018). Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint 1rXivv:802.09566
[66] Zhu, Y., Wong. J., Mandlekar, A., Martín Martin-Marttn, R., Joshi, A., Nasiriany, S., Zhu, (2020). robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprimt arXiv:2009. 122.22900..2293.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction describe the contributions of our work along with the scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We describe the limitations of our work in the section on limitations above.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details for all models, hyperparameters, and experiments in the experiments section or appendix sufficient detail to reproduce results (and the code provided here:
gi github. io/anon/).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All code and data (along with demos and steps for reproducing all experiments and datasets) are provided here: https://rover wwm.mithmii io/anon/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: This paper does not involve any training. For testing all methods, we outline the details needed to reproduce experiments in the appendix, along with all the code/steps needed to reproduce experiments at httttts://rovee vlm, github. io/ anon/
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We plot error bars (and explain their meaning) in all figures of the Experiments section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe compute resources and outline all details needed to reproduce experiments in the Appendix, along with all the code/steps needed reproduce experiments at /tttps://rover- vlm. we..m.githuu github. lo/anon/
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/:///.aaeaeeccccc////////a
Answer: [Yes]
Justification:
Our research fully conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe the broader impact in the dedicated section above.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no such risks associated with this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all papers/authors corresponding to any data or code that we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We outline all steps, along with the data and the code, reproduce the benchmarks at allttps://rover-vlln waw.ll....ttttte github. io/anon/
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There was no crowdsourcing or human subjects involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There were no study participants involved in this work.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: There were no usage of LLMs the creation or writing of this work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neunee http://ariipaccomomeeeeeeeee/e// Cc/Connferecces/2225//LL for what should or should be described.
21

---

Appendix Contents
A Additional results for main experiments
A.
Task 1: Frame-level task progress estimation
A.2 2
Task 2: Frame-level natural language reasoning
Task 3: Video QA
B
Details generated evaluation dataset
Table 2
Tasks within each task group-
Table
Description of levels for non-expert trajectories and video dataset counts.
Details on evaluation tasks
Task 1: Frame-level task progress estimation
Task 2: Frame-level natural language reasoning
Task 3: Video QA
Error analysis: Error types and examples
D
Prompting and implementation details
GVL
D.2 ROVER
Simulation environment and dataset of expert demonstrations
Results for additional experiments
Analysis of inference cost
Extended baselines
Stratifying results by state type and context length
Robustness to video length and frame rate
Robustness camera view
Testing across different backbone models
Task progress prediction with real-world OpenX Embodiment datasets
Ablations
M Additional details on object-centric subtasks
O Full related work
22

---

A Additional results for main experiments
A.1 Task 1: Frame-level task progress estimation
ROVER becomes less correlated with frame number as the number of non-expert states during video increases. To further explore how reasoning quality changes videos deviate from the expert behavior, we evaluate how the correlation between the video frame number and the predicted task progress changes as the number of non-expert states in the video increases. This provides measure of reasoning quality that is independent of the ground-truth value estimates. The progress values generated by GVL and TemporalConcat show similar correlation with frame number regardless of the number of non-expert states in the video (Figure 8). In contrast, ROVER shows significant drop in its correlation with frame number as the number of non-expert states the video increases, as would be expected with accurate reasoning (and as is observed with the ground-truth value estimates).
GVL
Corr. with frame number
TemporaiConcat
1.0
LIV
Ground truth
0.5
1
1
0.0
40
50
60
70
80
90
Percentage of non-expert states during video
Figure 8: Mean and standard error of correlation between video frame number and the frame-level progress values predicted by each method stratified across proportion of non-expert states in video.
ROVER achieves smaller distance between predicted and ground-truth progress values with non-expert trajectories. Figure D is based on the same set of results presented Figure 3 but uses L2 distance, instead of correlation, between model-predicted and ground-truth progress estimates as the performance metric. Similar to what we observe with the correlation results Figure we see in Figure that ROVER significantly improves performance (i.e., better agreement with ground-truth value estimates reflected by smaller L2 distance) for videos exhibiting non non-expert behavior. The performance improvement of ROVER over baselines becomes significant the trajectory level decreases (i.e., as the number non-expert states the video increases).
Diaace paorerraaon rocrdaaee to gooogd both
RemprraCinnal
300-
OVL ROVER
200-
107:
1002
love Kvs
lov6
lov1
lov3
le6
lov3
lov2 lvd
1o5 ledl 1ce8 loy4 Raturesdenebs
le12
place
doonsibrewers
appliances
Dotangs edgphe to geoonn mods
400
300
100
Nex2
1e.5
1nd2 Bestock panctyy
lenz: Arranige vegetablcs
Iex2
1as2
lav3
laod
Micrtomaee thowing
Preparr
Prenatalk
pon
Figure 9: Mean and standard error of distance between model-predicted and ground-truu trrdh task progress estimates stratified across trajectory level.
23

---

A.2 Task 2: Frame-level natural language reasoning
100
Succea rate 1h
80-
TanooraiConcat
40-
Pick-and
onvolt appliances
Toggle fotunneenntte
coars)trasers
100
60-
20-
le42
lev3
1ee4
lev5
lev2
lev]
lev2
lev3
lecl
led2
lev3
levt
Arrange
thowing
coffee
Figure 10: Mean and standard error of frame-level reasoning success rate (percentage of frames model states something that is verifiably correct relative to ground-tru information about simulator state) for videos stratified across trajectory level.
24

---

A.3
Task 3: Video QA
ROVER
GVL
80
60
All Turn onfort appliances
AlI Prepare
Operidese Pick-and place.
Togale fixtues/knobs
Microwave thawing
Restock pantry
Arrange vegetables
Pro Pressoak pan
doorsserawers
Figure 11: Video QA accuracy (the percentage of questions for which the model correctly identifies whether the event occurred) across all task groups.
Tempor Concat
80
60
20
Al Openidose Pick-and place Turn onfoff doorskirawers
All
All Microweve thawing
All Restock pantry
All Arrange vegetables
All Prepare coffee
Togale fistures/kabbs
Pre soak pan
appliances
Figure 12: Video QA precision (the percentage of predicted occurrences that actually occurred the video) across all task groups
ROVER
TemporalConcat
All Open/ckss Rick-and place Turn on/of! appliances
All All Toggle fixtures/nobs
All Restock pantry
Al Arrange vegellables
All Preisoak pan
Microwave thawing
Propare coffee
doongdraawers
Figure 13: Video QA recall (the percentage of actual occurrences correctly identified by the model) across all task groups.
25

---

B Details on generated evaluation dataset
Our generated evaluation dataset comprises videos that exhibit wide range of task expertise. The dataset includes 543 videos across 27 tasks, with about 20 videos per task (Tables 2 trajectory is collected random textures) in RoboCasa [39]. The videos are separated into levels based on the amount of the task completed during the video (Table The highest-level videos in each task group show full task completion with near-expert behavior. The dataset contains both atomic and composite tasks from RoboCasa (descriptions of each task are provided the appendix of (39]. The composite tasks involve two to four times as many actions as the atomic tasks. For evaluation, we downsample videos 30 frames for the atomic tasks (following (31] and donwsample videos to 60 frames for the longer composite tasks.
and 3). Each random kitchen scene (random floor plan, random kitchen style, and
Table 2: Tasks within each task group.
Task group
Task
Task type
Pick-and-place
PickPlaeCaabToCounte
Atomic
PickPlaceCounterToCCa
PesePlaceCounteeTTMMccvvvev
PickPlaceCounterTTSSni
PickPlaceCounterr ToSove
MiccPlaccec ToConter
PickPlaceSSinkToCoumee
TickPlaceSoveeTooConii
CoffeeSStupMMg
CoffeeServeMug
Open/close doors/drawers
OpenSingleDoor
Atomic
CloseSingleDoor
OpenDrawer
CloseDrawer
Turn on/off appliances
TumOOnMicrowave
Atomic
BoftcePressButton
Toggle fixtures/knobs
TurnSinkSpout
Atomic
TurnOSSinkFaucet
FurnOfSinkFaucet
TurnOnStove
TurnOffStove
Microwave thawing
MicrowaveThawwin
Composite
Restock pantry
RestockPantry
Composite
Arrange vegetables
ArrangeVeggeables
Composite
Prepare coffee
PrepareCoffee
Composite
Pre-soak pan
PreSoakPan
Composite
26

---

Table Levels for non-expert trajectories and video dataset counts. For each level, we assume that all task components corresponding to the preceding levels have already been successfully completed.
Task group
Non-expert trajectory levels
Dataset count
Pick-and-place
1: Fail to approach {obj]
2: Approach [obj}
3: Contact (obj)
4: Pick up (obj)
5: Keep grasp of [obj]
6: Approach placing [obj) in [target[location
7: Place [obj) in [target_location]
3
Open/close doors/drawers
1: Fail to approach the door/drawer
4
2: Approach door/drawer
3: Contact door/drawer
4: Start opening/closieee door/drawer
5: Finish opening/closing door/drawer
Turn on/off appliances
1: Fail to approach the on/off button/lever
2: Approach the on/off button/lever
6
3: Successfully adjust the on/off button/lever
Toggle fixtures/knobs
1: Fail to approach the lever/knob
2: Approach lever/knob
4
3: Contact lever/knob
4: Start turning/twisting lever/knob
5: Finish turning/twisting lever/knob
Microwave thawing
1: Fail to open microwave door
4
2: Open microwave door
3: Pick up the food item and place it in the mi- crowave
4
4: Close the microwave door
5: Press the microwave start button
Restock pantry
1: Fail to pick up the first item
2: Pick up the first item and place it in the pantry
3: Pick up the second item and place in the pantry
Arrange vegetables
1: Fail to pick up the first vegetable
2: Pick up the first vegetable and place on the
cutting board
3: Pick up the second vegetable and place it the
cutting board
Prepare coffee
1: Fail to pick up the mug
2: Pick up the mug and place it in the coffee ma- chine
3: Press the start button on the coffee machine
Pre-soak pan
1: Fail to pick up the pan
2: Pick up the pan and place it in the sink
2: Pick up the sponge and place it in the pan
4: Turn the sink handle turn the sink
27

---

C Details on evaluation tasks
C.1 Frame-level task progress estimation
This task evaluates the model's ability to estimate the scalar progress of a rohot toward task completion every frame of video, conditioned is calculated based on the simulator state at each moment of the video (Section 4.2). Given only task description and the video frames, the model must provide a numerical value reflecting task progress at each timestep. The evaluation metrics for this task include the L2 distance and the Pearson correlation coefficient between the predicted and ground-truth progress values across all frames. addition, as a measure of reasoning quality that is independent of the ground-trus value calculations, we evaluate the correlation between the video frame count and the VLM --r-eedccteed task progress, which should decrease as videos deviate further from the known expert trajectory.
natural language task description. Ground-truth progress
C.2 Frame-level natural language reasoning
For this task, evaluate the model's natural language text output it reasons about what is happening at each frame of the video, such as "The robot gripper is moving closer to the mug" or "The robot is holding the carrot." " The evaluation metries include the frame-level reasoning error rate and success rate. The error rate is the percentage of frames where the generated description includes a statement that is verifiably false (i.e., directly contradicts information from the ground-truth simulator state at that moment). The success rate reflects the percentage of frames where the model states something that is verifiably true (i.e., consistent with information from the ground-trui simulator state at that moment) and does not state something verifiably false. This task measures the model's capacity to precisely localize events in time, avoid hallucinations, and generate semantically faithful descriptions at fine granularity.
To implement this for the results shown in Section 5, we treat the evaluation as language model task, where we prompt the model with the frame description (generated by ROVER, GVL, or TemporalConcat) paired with the ground-truth information and ask if the description includes any statements that are verifiably true or false relative to the ground truth. We use gpt-4. 5-0--.55220 5--55---00-2---- as the language model for this evaluation. After reviewing 200 examples of the language model's evaluation by hand, we observed 97.5% alignment with human judgement. In Figure [4] we include examples below of frame-level reasoning from GVL versus ROVER where the model states something that, when compared to the ground-truth information, is verifiably false (red), verifiably true (green), or inconclusive (gray). In Figure 15 we include an example of where the LLM evaluation does not align with the human evaluation.
At each frame of the video, ROVER, along with baselines, generates a description of the frame before predicting task progress for that timestep. We cross-reference these descriptions with information about the ground-truth simulator state each timestep to calculate the reasoning error rate and success rate (described in Appendix C.2). We treat this evaluation as a language model task, where we prompt the model with the generated description paired with the ground-truth information and ask if the description includes any statements that are verifiably true or false relative to the ground truth. We use g.t-4.5-- 2-2025-02--22222 as the language model for this evaluation (details in Appenedix
C.3 Video question answering
The Video QA task tests a model's ability to reason over entire trajectories and answer questions about actions that may may not have occurred. Each question is posed in the format: "Did the robot action? If so, when?" where action includes task-relevant operations like "contact the mug". "drop the potato",, or "place the bowl in the microwave". Questions automatically generated from a templated action grammar, grounded in the specific object and scene configuration of each video. The 27 robot manipulation tasks grouped into 9 categories (Table 2 We ask similar questions within each task group based on the shared action types (Table 4p. The evaluation metrics include:
Accuracy: the percentage of questions for which the model correctly identifies whether the event
occurred.
Recall: the percentage of actual occurrences correctly identified by the model.
Precision: the percentage of predicted occurrences that actually occurred in the video.
28

---

Temporal distance: the difference in time between when something occurs during video and when the VLM states that occurred.
For the results shown in Section 5, we generate answers to each question for this task using the frame-level descriptions produced by ROVER, GVL, and TemporalConcat. For each question, we use language model to generate an answer given all frame descriptions concatenated together. We again use gpt-4. 5-2055-2222 as the language model for this evaluation.
Ground-truth information:
The carrot is the counter sect to the stnk.
the close the carret.
The gripper
The carret is not moving.
x
GVL reasening:
The robet above sink, baving just placed carrot.
ROVER reasoning: noved stightly
which is where the carrot is Located, This represents
negetive progress towards the subtesk of moxing the gripper
Ground-rruth informati lom: is costact The robet
zink, carret. The gripper is aaving closer to the carrot, The gripper is not clase to the carret. moving.
GUL reasoning:
ROVER reasonding
The robot are noved closer to the currot, which is still
Ground-rutt informatiinn ion:
The carrst is the counter next to the sirk,
clower carrot. The gripper clese the
The
The carrot is not soving.
X
GVL reasoning:
The robot grapper holding carrot.
reasoning:
pripper positioned right touching st.
alwest
Ground informatioo:
The robut heldiing Lie carrel.
The carret 15 only in contact with the robet gripper. The novingg clozer
GVL reasondng)
grapper
ressoning:
The holding the carrot above sink. goterisa can be seee next to the sink.
Figure 14: Examples of frame-level reasoning from GVL versus ROVER where the model states something that, when compared to the ground-truth information, is verifiably false (red), verifiably true (green), inconclusive (gray).
29

---

The Towen the oi the sost the 13M4
The cripper do clese the Seners
Exslaatoo
ALIGNED
4
petnting plant
NOT ALIGNED
Figure 15: Examples of LLM evaluation compared to human evaluation frame-level reasoning from GVL and ROVER.
30

---

Table 4: Questions for video QA task.
Task group
Questions
Pick-and-place
Did the robot contact the {obj}? If so, when?
Did the robot pick up the [obj]? If so, when?
Did the robot drop the [obj]? If so, when?
Did the robot place the {obj} in
If so, when?
Open/close doors/drawers
Did the robot contact the door/drawer handle? so, when?
Did the robot grasp the door/drawer handle? If so, when?
Did the robot start opening/closing the door/drawer? If so, when?
Did the robot completely open/close the door/drawer? If so, when?
Turn on/off appliances
Did the robot press the start/stop button of the microwave/coffee machine? If so, when?
Toggle fixtures/knobs
Did the robot contact the sink spout/stove? If so, when?
Did the robot start turning the sink spout/sink handle/stove knob? If so, when?
Did the robot completely turn on/off the sink/stove? If so, when?
Microwave thawing
Did the robot open the microwave? If so, when?
Did the robot pick up the [obj]? ? If so, when?
Did the robot drop the (obj)? If so, when?
Did the robot place the [obj] in the microwave? If so, when?
Restock pantry
Did the robot pick up the [objl] }?? ! If so. when?
Did the robot drop the [objl] ! so, when?
Did the robot place the [obj1] in the cabinet? If so, when?
Did the robot pick up the [obj2] so, when?
Did the robot drop the [obj2]? If so, when?
Did the robot place the (obj2) in the cabinet? If so, when?
Arrange vegetables
Did the robot pick up the [vegetablel| !? so, when?
Did the robot drop the (vegetablel]? ? If so, when?
Did the robot place the [vegetablel] in the cabinet? If so, when?
Did the robot pick up the [vegetable2] If so, when?
Did the robot drop the (vegetable2) If so, when?
Did the robot place the (vegetable2) in the cabinet? so, when?
Prepare coffee
Did the robot pick up the mug? If so, when?
Did the robot drop the mug? If so, when?
Did the robot place the mug in the coffee machine? If so, when?
Did the robot press the coffee machine start button? If so, when?
Pre-soak pan
Did the robot pick up the pan? If so, when?
Did the robot drop the pan? so, when?
Did the robot place the pan in the sink? If so, when?
Did the robot pick up the sponge? If so, when?
Did the robot drop the sponge? If so, when?
Did the robot place the sponge in the pan? If so, when?
Did the robot contact the sink handle? If so, when?
Did the robot turn on the sink? If so, when?
31

---

C.4 Error types and examples
Incorrect subtasks: When decomposing the task "put the mug the coffee machine", the model specifies the subtask of "open the cabinet door" when the mug is on the counter.
Redundant subtasks: The model first decomposes the task "put the steak in the microwave" into "pick up the steak from the counter" and "place the steak in the microwave", but then incorrectly
decomposes the "place the steak in the microwave" subtask into "grasp the steak" and "move the steak to the microwave".
Perception error: The model states that the gripper is holding object when it is not.
Reasoning error: Despite correctly identifying the gripper has dropped an object, the model states that task progress is increasing.
32

---

D Prompting and implementation details
D.1 GVL
Generative Value Learning (GVL) [3] is the existing state-of-the-ard in-context learning approach for VLM reasoning over video input for embodied tasks. The method is motivated by the observation in [3I] that, when presented choronological sequence of video frames, VLMs often ignore the trajectory quality and instead hallucinate monotonically increasing task progress. To disrupt this temporal bias, GVL randomly shuffles the input frames. In doing so, GVL forces the model to pay attention to what happening to each frame, improving frame-level task progress prediction. We implement GVL using the following prompt from [11
** SYSTEM PROMPT
You are expert roboticist tasked to predict task completion percentages for frames of a robot for the task of (task_der descrippiinn. Note that the robot has unknown level of expertise with the task and may perform actions that lead it signicantly further from accomplishing the task. Note that these frames so please pay attention to the individual frames when reasoning about task completion percentage. If the progress the current frame is less than the progress of the initial robot scene, then the task completion percentages should negative.
in random order,
For the of (task_ deaakpdescri output the task completion percentage for the following frames that are presented in random order. For each frame, format your response follous: Frame description: { Task completion percentage: (3%
w TASK PROMPT
Initial robot scene: [IMG]
Frame description: latpest_fraameeescrpppee
Task completion percentage: 0%
Frame 1: [IMG]
Frame [IMG]
33

---

D.2 ROVER
Similar GVL, we implement ROVER using an in-context learning approach, with the same prompting, shown below, used for all backbone VLMs. As discussed in Section 3, for each frame, the VLM describes the frame and either specifies a new subtask that needs be completed or predicts progress for the existing subtask. If new subtask is specified, then new line of reasoning is created (using the same prompting shown below) with the updated {task_descripts description) for the new subtask. To specify new subtask, the VLM generates the text tokens "The robot needs to: (new "ne__subtask"" as depicted in Figure If new subtask is not specified, the VLM proceeds with the frame-by-frame progress prediction for the existing subtask. To move to the next frame, the VLM generates the text tokens "[next-frame]" as shown in the prompting below. As described Section 3, we implement sliding window when appending the next frame, Onett to the existing context, Y. When appending a new frame, Y = Window(< + Onext). the window extracts three frames: the first frame of the current line of reasoning and the most recent previous frame (each followed by their previously generated natural language descriptions), along with the newly added next frame. The Window function formats these three frames shown in the prompt below.
SYSTEM PROMPT
You are an expert roboticist tasked to predict subtask completion percentages for frames of robot for the subtask ttask_description.)
The frames are shown in temporal order. Frame 0 represents the beginning of the subtask. Note that the robot has an unknown level of expertise with the subtask may perform actions that lead it signicantly further from accomplishing the subtask. Therefore, please pay attention to the individual frames when reasoning about subtask completion percentage. If the progress at the current frame is less than the progress of the initial robot scene, then the completion percentages should be negative.
If the given subtask can be decomposed into multiple subtasks, please specify new subtask. Some examples of decomposition are provided below. Subtask: 'pick the cheese from the counter and place it in the cabinet' New Subtasks: ['grasp the cheese', 'place the cheese in the cabinet']
If you decompose the subtask further or progress to the next subtask, format your response as follows:
Frame description: O The robot needs to: 0
If you do not decompose the subtask further progress to the next subtask, format your response as follows:
Frame description: 0
Subtask completion percentage: 03%
[next-frame]
TASK PROMPT
Initial robot scene: [IMG]
Frame description: (first_frame ffistt_rrane_dsccipttiilll
Subtask completion percentage: 0%
###*Н# If VLM has progressed beyond second frame of existing subtask Most recent previous frame: [IMG]
Frame description:
Subtask completion percentage: Barev_subtakkpprorreet #*####
Current frame: [IMG]
34

---

E Simulation environment and dataset of expert demonstrations
We leverage RoboCasa 39
the simulation environment, which includes large dataset of expert demonstrations, collected via human teleoperation, for household tasks in its initial release. RoboCasa large-scale simulation framework centered around home environments for training generalist robots. RoboCasa builds upon RoboSuite [66]] framework based in MuJoCo with a focus on physical realism, high speed, and modular design. RoboCasa contains 120 kitchen scenes and over 2,500 3D assets spanning 153 unique object categories, created with the aid of generative Al tools [39 (depicted Figures T6 and [7] from [39]). RoboCasa includes dataset of demonstrations collected through human telcoperation. A team of four human operators collected 50 high-quality demonstrations for each atomic task, along with 5 composite tasks, using a 3D SpaceMouse [65] 66
Figure 16: Figure from
39
showing RoboCasa kitchen scenes.
Figure 17: Figure from
39
showing RoboCasa interactable appliances.
35

---

Analysis of inference cost
We measure two key indicators of inference cost across tasks with different video lengths: (1) the total number of tokens processed per video and (2) the number of VLM calls. The tables below show that ROVER consistently reduces token count compared to baseline methods, even though requires slightly higher number of model calls due to its recursive structure. Importantly the increase in inference cost with respect to video length is approximately linear, consistent with our design goal. For instance, doubling the video length from 30 to 60 frames results in approximately doubling of token usage for ROVER (from 47k to 95k tokens), supporting the linear scaling claim. Due to ROVER scaling linearly with video length (with baselines scaling quadratically), the improvements in inference cost we observe with ROVER increase as video length increases. For videos with 30 frames (the single-stage atomic tasks), ROVER shows about 3x reduction in the number of tokens per video. For videos with 60 frames (the multi-stage composite tasks), ROVER shows more than 5x reduction in the number of tokens per video.
Table 5: Inference cost of each method for videos with 30 frames.
Method
Avg number of tokens per video
Avg number of VLM calls per video
ROVER GVL TemporalConcat
47,650
36
140,307
30
134,855
30
Table 6: Inference cost of each method for videos with 60 frames.
Method
Avg number of tokens per video
Avg number of VLM calls per video
ROVER
95,404
68
GVL
539,102
60
TemporalConcat
527,572
60
G Extended baselines
The results below include two recent video-specific baselines: VideoLLaMA-3 and Video-LLaVA. These models designed specifically for video-language tasks and incorporate temporal modeling and memory mechanisms that target long-sequence video comprehension. We also include the baseline VideoGemini that concatenates all frames of the video provide input the model.
Table 7: Correlation with ground-truie truth values for frame-level progress prediction with extended baselines.
Method
Real-world videos
Simulation videos
ROVER
0.62
0.56
GVL
0.31
0.35
TemporalConcat
0,43
0.17
LocalConcat
0.46
0.18
LIV
0.12
0.09
VideoGemini
0.46
0.29
Vama3
0.34
0.21
VideoLLaVA
0.19
0.14
36

---

Table 8: Reasoning error rate for frame-level natural language reasoning with extended basclines.
Method
Error rate
ROVER
45%
GVL
59%
TemporalConcat
67%
LocalConcat
62%
VideoGemini
62%
Vameo
71%
VidcoLLa
76%
Table 9: Accuracy for video QA with extended baselines.
Method
Accuracy
ROVER
77%
GVL
51%
TemporalConcat
28%
LocalConcat
25%
VideoGemini
68%
VideoL
62%
VideoLLaVA
58%
H Stratifying results by state type and context length
GVL and TemporalConcat attempt to reason over the full video by concatenating all frames together. T the correlation GVL and TemporalConcat achieve with ground-truth value estimates when reasoning over (-10 frames, 10-20 frames, and 20-30 frames. In addition, the figure further stratifies results based the state type at the current timestep (expert, non-expert, interpolating between expert and non-expert) of the trajectory. These are the same results presented in Section 5.2, but now stratified across these two dimensions (instead of stratifying across trajectory level, as done in Figure 3) We see that the performance of GVL and TemporalConca degrades when encountering non-expert moments of the trajectory, especially when the number of frames included in the context extends beyond 10. We observe similar trend below when stratifying the eтror rates with the frame-level natural language reasoning across context length and state type (Figure 19). ROVER limits the maximum number of frames the model must reason over to three frames using the sliding context window (described in Section 3). In doing so, ROVER avoids these errors observed in long-context reasoning.
37

---

H.1
Frame-level task progress estimation
GVL
TemperalConcat
0.8
Con:: with ground truth
0.6
0.4
0.0
-0.2
20-30 frames Expert of video
0-10
10-20
0-10
10-20 frames
20-30 frames
0-10
10-20 frames
20-30 frames
frames
frames
frames
frames
Non-expert
Interpolating from non-expert to expert
Figure 18: Average correlation GVL and TemporalConcat achieve with ground-truth value estimates the frame-level progress prediction task when reasoning over frame sequences of different lengths.
H.2
Frame-level natural language reasoning
GVL
TemporalConcat
80
Eiror rate (%))
60
20
20.30 frames Expert moments of video
0-10
10:20 frames
20:30 frames Non Non-exper expert moments of video
0:10 frames
10-20 frames
0-10
10:20 frames
20:30 frames
frames
frames
Interpolating from non-expert to expert
Figure 19: Average error rate of GVL and TemporalConcal with the frame-level natural language reasoning task when reasoning over frame sequences of different lengths.
38

---

H
Robustness to video length and frame rate
The total frame count video is determined by two factors: 1) the video length: the total amount of time that the video represents and 2) the frame rate: the number of image frames used to represent each unit of time. These two factors can vary widely for given video at test time. Therefore, for method to achieve reliable performance in the wild, it should be robust to both factors.
ROVER scales more readily to longer videos and higher frame rates. We see that ROVER, by leveraging recursive reasoning and using maximum context of three frames, is not only more efficient, but also less sensitive video length and frame rate of the given video relative GVL and TemporalConcat, We first observe this in the results stratified across task group (Figures 3 and 5 from the main results in Section 5), where we see the improved performance of ROVER for the composite tasks (bottom row of each figure), which include much longer videos than the atomic tasks (top row of each figure). We also observe ROVER's robustness frame rate when varying the total number of frames used to represent each video between 30 and 240 (i.e., increasing the effective frame rate by factor of 8) for the pick-and-place tasks for the progress prediction task (Figure 20) and frame-level natural language reasoning (Figure 21). These figures show that the performance of GVL and TemporalConcal significantly degrades when given the same pick-and-place videos, but with effective frame rate increased.
ROVER
ROVER
GVL
GVL
0.7
100
0.6
Error rate (%6
0.4
0.3
0.0
001
120
120
Total video freme count
Total video frame count
Figure 20: Average correlation each method achieves with ground-truth value estimates in the frame-level progress prediction task when increasing the total frame count from 30 to 240 (by increasing the frame rate) for the 10 pick- and-place tasks.
Figure 21: Average error rate of each method with the frame-level natural language reasoning task when increasing the total frame count from 30 240 (by increasing the frame rate) for the 10 pick-and-place tasks.
39

---

I Robustness to camera view
The main results presented in Section 5 the wrist view (eye in hand view), as all methods were observed to perform best on the manipulation tasks using this view. The figures below show the results of the progress prediction task (Figure (2) and frame-level natural language reasoning task (Figure (3) when using different camera views. The left view and right view are fixed third-person -errsnn left and right, respectively, of the robot at a distance sufficent to show the full robot arm within the camera frame. Although all methods decline in performance with these fixed external views (likely due to occlusions the object or gripper at different moments during the trajectory), ROVER continues to outperform baselines actoss all camera views.
views with the camera positioned
TemporaiConcat
Tempora
CVL
GVL
ROVER
ROVER
0.6
Corr. with ground truth
0.5
0.4
Error rate (%)
6.0
0.3
400
0.2
20
0.1
0.0
wrist view
left view
right view
wrist view
left view
right view
Figure 22: Mean and standard eπror of correla- tion between ground-truth value estimates and progress values predicted by each method strati- fied across different camera views.
Figure 23: Mcan and standard error of error rate in frame-level natural language reasoning of each method stratified across different camera views.
40

---

Testing across different backbone models
To ensure consistency across benchmarks and with prior work, We use gemini-1.. 5-pro for Gemini- 1.5-Pro and gemini-2 1eni--2.-pppe. 2-proo------- preview We use gpt-4o from the OpenAI API for GPT-4o. We use Quen2.5-VL-333 ----------------- --ntt-rut-ttttttt for Swen2.5-VL-32h available on Huggingface. For the open souree model, all experiments were performed on A6000 GPUs. Overall, we see that ROVER consistently outperforms baselines across all backbone VLMs.
--rro-previe for 2ommiii..5-Pro-Preeeee
from Google's Gemini API.
TemporaiConcat
ROVER
GVL
Distance relative ground truth
ROVER
500
Corr. with ground truth
0.6
300
0.2
200
0.0
100
Gemini 1.5-Pro
Gemini
GPT-4o
Qwen2.5 VL-32B
Gemini 1.5-Pro
Gemini 2.5-Pro
GPT-4o
Qwen2.5
2.5-Pro
VL-32B
Figure 24: Mean and standard error of correla- tion between ground-truth value estimates and progress values predicted by each method strati- fied across different backbone VLMs.
Figure 25: Mean and standard error of dis- tance between ground-truth value estimates and progress values predicted by each method strati- fied across different backbone VLMs.
Tempora GVL
Tempora GVL
ROVER
80
ROVER
Success rate (%)
60
Error rate (%)
60
40
40
20
Gemini 1.5-Pro
Gemini 2.5-Pro
GPT-4o
Owen2.5 VL-328
Gemini 1.5-Pro
Gemini 2.5-Pro
GPT-4o
Qwen2.5 VL-328
Figure 26: Mean and standard error of error rate in frame-level natural language reasoning of each method stratified across different backbone VLMs.
Figure 27: Mean and standard error of success rate in frame-level natural language reasoning of each method stratified across different backbone VLMs.
41

---

K Task progress prediction with real-world OpenX Embodiment datasets
We perform the same task progress prediction analysis conducted in 3T using 1,000 videos from 50 real-world OpenX Embodiment datasets (20 videos randomly selected from each dataset) [41]. The OXE dataset includes an aggregation of trajectory data from 50 standalone robot datasets that consists of diverse tasks, robots, and camera viewpoints. The OXE videos include 20 robot embodiments, over 300 different task specifications, and highly diverse (often non-ideal) camera viewpoints.
The tables below show the average correlation each method achieves with the ground-truth value estimates across the OXE datasets when using Gemini-1.5-Pro (Table (O) and GPT-4o (Table TTp as the backbone VLM. Since there are no true value estimates for these real-world videos, we follow [3 and use frame number as the ground-truth value estimate. For known high-quality datasets collected from human teleoperators with fixed camera placements, such as Bridge [3]. RT-1 [7]. and Dobb-E
47 ROVER achieves significantly higher correlation with frame number than GVL (highlighted in yellow in the tables below). Both ROVER and GVL show low correlation with frame number for datasets with videos that show lower quality trajectories produced by autonomous data collection via scripted motions or motor babbling, such as QT-OPT 23 and RoboNet [10] (highlighted in green in tables below).
42

---

dobbe Table 10: Correlation of each method with ground truth progress estimates (based on frame number)
on OpenX Embodiment datasets with Semini--1.5-PP as the backbone VLM.
Dataset
GVL
ROVER
0.753
0.898
utokyo_xarm_ nyu httaaauullccoveeeeeeeeeeeeoeeeee
0.822
0.890
0.668
0.879
berkeley..uutola__..
0.761
0.874
0.619
0.870
rlds
0.740
0.866
wag.yp/pw..pppnnng......erereerr fracta120228817_datt oat.aha......pgog..aa.........rr
0.824
0.856
0.762
0.817
0.513
0.805
toto ott.aaaacccaaaaaaaaao.ooooeeorea
0,564
0.780
0.529
0.768
utaustin_mutex asu htt.bleettp_.ovverteeeetttttttt
0.576
0.767
0.477
0.762
mustin_sirius datluust_dttttt.ttm converted www.eeeelteetttteaee tords
0.539
0.755
0.487
0.754
berkeleyy.able_routiing
0,488
0.752
0.495
0.748
viola fmb oag..oatt....iccooorrroo...otttr bridge
0,415
0.747
0.555
0.739
authhinDiii..ttttr...rrrrorrrrrr usc_cloth_sim
0.350
0.727
0.457
0.722
0,457
0.717
jaco_play
0.414
0.710
0.351
0.701
bc_z
0.397
0.696
berkeley_mve
0.360
0.680
cmu_stretch
0.276
0.677
0.362
0.639
0,252
0.632
roboturk
0.314
0.631
0,281
0.615
0.119
0.598
0.115
0.564
droid
-0.002
0.546
taco_play
0,083
0.539
-0.010
0.513
inperraaccolee...aave...vvtaaaa
-0.038
0,432
-0.021
0.425
-0.072
0.371
kuka
0.304
0.316
cmu_play_fusion
-0.096
0.304
-0.110
0.295
rlds
-0.139
0.287
shanforddmask.oo vit http:aaaavvttttooeeeeeeedddeeedd uiuc_d3field robo_net
-0.131
0.270
-0.212
0.169
-0.240
0.096
columbia_cairlat catiiii__carlab_puhstttt
-0.229
0.095
dlr htw.saa..g.d.cca....ooorrrooooe.
-0.301
-0.024
-0.230
-0.078
rlds
43

---

Table 11: Correlation of each method with ground-truthh progress estimates (based on frame number)
OpenX Embodiment datasets with GPT-4o the backbone VLM.
Dataset
GVL
ROVER
utokyo_pr2
tereniinn_riidg__ooneeeeeer externalllll rldss nyu_door_opents
0.907
0.969
0.891
0,928
berkeley_autolab__
0.749
0.894
utaustin_mutex
0.837
0.893
0.835
0.889
ort.gna.......aaaaaoooooooooootr fractal20220817_daz2
0.788
0,879
0.714
0.874
0.722
0.867
rlds aut.ihassttiiaaaaar....rrrrrrrrr
0.717
0.867
toto ott:ccac...acerccrrrrrooooooooor
0.717
0.851
0.696
0.840
dobbe
0.568
0.829
rrlds ____sdlddss___ss_rr
0.554
0.824
0.726
0.822
0.629
0.814
0.603
0.807
viola aet_ffabee..g..coorrrrreetttttt
0.503
0.804
asu_table
0,505
0.803
bridge naa.ssll..faascccccorvrteettttt. roboturk
0.661
0.799
0.527
0.787
jaco_play
0.570
0.786
0.616
0.781
0.586
0.769
0,466
0.765
uiuc_d3field
0.524
0.730
0,458
0.728
0.483
0.704
0.353
0.681
berkelayy.cable..outiingg
0.278
0.676
0.394
0.674
0.245
0.663
cmu.play_fusion
0.374
0.661
0.244
0.615
Auperraacclleeg..ayyeer...sss
0.219
0.602
0.206
0.598
0.178
0.598
cmu_stretch
0.140
0.598
bc_z
0.158
0.591
wwn.laaatiyyyyy.eaeteet..oooooer robo_net
0.182
0.567
0.367
0.438
stanford_kuke multimodal htt.:iidea__aaatteccooooeeeeeeea rlds
-0.042
0.434
stanford_mask vit_
cett__onverted_eee
-0.042
0.433
calubbia_carrlaa_pusttt
-0.048
0.416
eth.agent_affordancee.
-0.088
0,416
-0.064
0.356
rlds
taco_play kuka
-0.094
0.299
0.150
0.170
dlr
-0.301
-0.047
44

---

L
Ablations
As described in Section 5.5, we test ablations of ROVER: one variant ablates only the sliding window while the other ablates only the recursive reasoning. We observe that baseline methods such as TemporalConcat often hallucinate steady, monotonically increasing task progress values, even when the video includes significant stalls regressions. GVL addresses this problem via random frame shuffling, which disrupts the strong temporal priors and forces the model to assess each frame more carefully BI]. We observe that the sliding window, even when ablating the recursive decomposition, similarly reduces hallucination by retaining temporal context while avoiding long-conted sequential processing. Our ablations show that the window-only variant of ROVER significantly outperforms TemporalConcat and matches or exceeds GVL's performance on short- mhort-horizon tasks, such as turning on/off appliances toggling fixtures/knobs. We see that adding the recursive decomposition further improves performance for more complex tasks consisting numerous subtasks.
GVL
arthout recursien Recumsin without stiding window
Distance ground trut.
novan
600
400
200
Rick-and-ppaae aaannniddccceee
Open/clsse
onfoff
Microwave
Arrange
Exturesiknobs
coffee
Figure 28: Mean and standard error of distance between ground-truth value estimates and progress values predicted by baselines and ablated versions of ROVER stratified by task group.
mppor powaaaaCnncc
Staking window without recure on
Recurston without sidding window
ROVER
Entor rasa
60
40
Turn
Arrange
doorsdrawers
fixtureskkoobs
coffee
Figure 29: Mean and standard error of error rate in frame-level natural language reasoning of baselines and ablated versions of ROVER stratified by task group.
Temporatte auporaalconn oneat
GVL
Sliding window without recurs on Rocurston without diging window RoveR
(N)
20
Pick-ard--pace
Microw thawing
Restack pantry
Pre-soak pan
doonssrrawers
applances
Rnturisiknabss
vegetab
coffee
Figure 30: Mean and standard error of success rate in frame-level natural language reasoning of baselines and ablated versions of ROVER stratified by task group.
45

---

The sliding window of ROVER includes 3 frames: the first frame of the current line of reasoning, the most recent previous frame, and the frame for the current timestep of the video. In the table below, we show the performance of ROVER on the frame-level progress prediction task when changing the sliding window size to 6, 9, and 12 frames. We see that, as the number of frames included in the sliding window increases, the performance of ROVER drops.
To evaluate the cause performance drop when increasing the sliding window size, apply the same error analysis that we describe above to the modified versions of ROVER that have increasing sliding window size. Based on the results of this analysis shown in the table below, the drop in performance with increasing window size appears to be driven by increases in perception errors, which also comprise the primary eтror type observed with GVL. Perception errors occur when the model hallucinates basic facts about what is happening in the video. We see that, in the original version of ROVER, only 5% videos include perception error, but the version ROVER with window size of 12, this rises to 19% of videos.
This supports our findings that the VLM is prone to perception errors when attempting reason over many frames of a video. ROVER (with a sliding window size of 3) mitigates these perception errors by shortening the number of frames the model must reason over each timestep of the video. When the sliding window size is increased, perception errors increase. This is also consistent with our finding from the initial error analysis that 83% of perception errors from GVL occur when there are over 10 frames of the video included in the context.
We see that these modified versions of ROVER still do not produce many perception emrors GVL (which makes a perception error on over 30% of videos, as shown Table R14). This is likely due to the fact that, even with sliding window size of 12, ROVER still shortens the model's context each moment of reasoning by decomposing the video into subtask-specific lines of reasoning (which are generally less than 12 frames in length, even without implementing sliding window).
Overall, we find that ROVER introduces possibility decomposition errors, but dramatically reduces the total error rate by mitigating the model's tendencies toward hallucination (perception errors).
Table 12: Correlation with ground-truth values for frame-level progress prediction (with varying sliding context window size).
Method
Corr. with Ground Truth
ROVER (original)
0.56
ROVER (window of 6)
0.52
ROVER (window of 9)
0.50
ROVER (window of 12)
0.44
Table 13: Error rates (%) observed with increasing sliding window size for ROVER.
Error type
window of 3 (original)
window of 6
window of 9
window of 12
Incorrect subtasks
7%
6%
5%
7%
Redundant subtasks
4%
3%
5%
4%
Perception error
5%
10%
12%
19%
Reasoning error
4%
4%
5%
6%
Other
3%
2%
4%
4%
Total
23%
25%
31%
40%
46

---

M Additional details on object-centric subtasks
When generating non-expert trajectories (described in Section 4.1) and computing ground value estimates (described in Section 4.2), we follow the work in MimicGen (36 in defining object-centric subtasks for each task trajectory. Specifically, if we let E {e1, c2 ...} be the set of objects in a task M, we assume that each expert demonstration, 7 E Derc. can be decomposed into object-centric segments o = { (FF,(e)))) )) where the manipulation in each subtask trajectory relative to a single object's coordinate frame (eT, E E). This sequence of object-centric subtasks is generally straight forward for a human to identify given a task [36]. For example, for the pick-and-place task of "pick the cup from the counter and place it in the sink", the subtasks include first picking up the cup. then placing it in the sink. This task can be broken down into two object-centric subtasks: a cup-grasping subtask (motion is relative cup) and sup-placement subtask (motion is relative to the sink basin). Similar to MimicGen [6),, we assume access to metrics that allow the end of each subtask to be detected. In the previous example, this would correspond to detecting 1) when the cup has been lifted from the counter and 2) when the cup has been placed within the sink basin.
We generate the non-expert trajectories (by inserting random perturbations within the expert demon- strations as described in Section 4.1) and compute the value estimates (based on the distance measures described in Section 4.2) for each of these object-centric subtasks separately. To create the full non- expert trajectory, N = {T: (e,,) ) we simply sequence together the subtask trajectories. To form the value estimates for the full trajectory, we sequence together the value estimates correspond- ing to each subtask, adding the value estimates of each subtask to the final value of the previous subtask and, when complete, rescaling the list of values to the range [0, 1].
O Full related work
Decomposition-Dasee reasoning. ROVER enables more precise and efficient reasoning over video for long-horizon embodied tasks by segmenting the video input into subtask-specific segments through recursive decomposition. Prior work has explored decomposition in various forms, including neural modular architectures [2] 17]. multi-hop QA (50), and multi-step reasoning for program synthesis or mathematical problems [37
16 28]. More recently, LLMs have been used to decompose problems via in-context learning or fine-tuning for complex tasks such as code generation and Adaptive decomposition methods have also emerged, including feedback- 44 and multi-agent coordination [59]). Unlike these approaches, ROVER performs recursive decomposition over video input with VLMs. Further, we show that ROVER can be used to improve reasoning about physical interactions in embodied settings.
planning [55 24
64
driven re-planning 42
Goal-conditioned value functions and reward modeling. ROVER demonstrates an emerging use of VLMs as per-frame value estimators for embodied video tasks, leveraging their semantic understanding and long-context capabilities. Prior work developed models using robot [46] or human videos with discriminators
[8]. contrastive learning [.. or offline reinforcement learning [B4] [2] 5]
With the advent of foundation models, researchers have begun incorporating language and vision models into downstream robotic applications including planning 48 12), imitation learning 6 49
and symbolic reasoning 33
[1].
27 and 35 leverage language models to assign reward values to reinforcement learning agents. 26
and again (27) utilize these models deliver preference-based feedback. Works such as 62. and [60] go further by having LLMs generate code. Notably, all these approaches rely solely on the language understanding capabilities of foundation models.
58
B3..
Prior methods leveraging VLMs for reward prediction over video input primarily relied on two extremes: reasoning over small set of frames (e.g., comparing two individual frames) (57|25 54 63
or attempting to reason over the full video by concatenating all frames into single context (31. The former sacrifices global context, while the latter is inefficient (inference cost scales quadratically with video length) and overwhelms the model with visual information that is often redundant or irrelevant. ROVER addresses these limitations by leveraging recursive decomposition enable peecisrrcision frame-by- taaee--b-framme reasoning (including reward or task progress prediction), without sacrificing efficiency or global context.
47


Reviews:
[{"review_id": "85gWCJCrIX", "reviewer": "Reviewer_AqtN", "summary": "This paper introduces ROVER, a recursive vision-language framework designed for embodied robotic tasks. It enhances reasoning by decomposing long-horizon videos into manageable subtasks. At its core, ROVER recursively segments task trajectories to maintain global context, while a sliding window within each subtask focuses the model on temporally relevant frames. This dual approach allows it to effectively balance high-level task structure with fine-grained local details. Evaluated on a new, diverse RoboCasa dataset and other large-scale benchmarks, ROVER significantly outperforms leading baselines (GVL, TemporalConcat, LIV) in task progress estimation, frame-level reasoning, and video QA. Notably, it mitigates model hallucinations and achieves linear time complexity, scaling efficiently with video length.", "questions": "See above.", "limitations": "Yes.\n\nLimitations are adequately acknowledged (Section 7 and Broader Impact), with constructive discussion on decomposition failure, reliance on in-context prompting, and the risk of increased autonomy in AI systems.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Clear and Compelling Motivation: The work is well-motivated, addressing the fundamental trade-off in VLM-based video reasoning between maintaining global context and achieving computational efficiency.\n2. Innovative Methodology: The core contribution is ROVER’s innovative methodology, which synergistically combines recursive task decomposition with a sliding context window. This elegant approach simultaneously enables fine-grained, focused reasoning while maintaining computational efficiency.\n3. Rigorous and Extensive Experiments: The claims are supported by a rigorous empirical evaluation using a novel, large-scale RoboCasa dataset alongside established benchmarks. This provides robust validation across diverse and realistic manipulation tasks, strengthening the generalizability of the findings.\n4. Valuable Community Resource: The public release of the dataset, code, and demonstrations constitutes a valuable contribution to the research community.\n\nWeaknesses:\n\n1. Lack of Decomposition Failure Analysis: The framework's performance heavily relies on successful task decomposition, yet the paper lacks a quantitative analysis of potential failure modes. The frequency and impact of incorrect subtask splits are not explored, making it difficult to assess the method's robustness.\n2. Uncertain Generalization Scope: The primary contributions and in-depth analysis are centered on simulated robotic manipulation. The framework's applicability to other challenging video domains, such as open-world videos or real-world scenarios, remains undemonstrated.\n3. Limited Set of Modern Baselines: The comparison could be strengthened by including more recent architectures designed for long-sequence video understanding, such as advanced transformer variants or memory-augmented VLMs, which would provide a more competitive performance benchmark.\n4. Potential Dependence on Task Structure: The assumption that tasks can be decomposed into object-centric subtasks with clear sequential transitions (as in RoboCasa) may not hold for more entangled or ambiguous real-world tasks. The implications or limits of this assumption warrant deeper discussion, beyond referencing previous work.", "was_revised": true, "final_justification": "After the discussion with the authors during the rebuttal, some of my concerns have been solved. However, I still believe that the task progress estimator was only trained on table-top video datasets. Like Reviewer XMz4, I have the same concern about ROVER's performance in real-world downstream task, which could be explored in future work.", "rebuttal": "&nbsp;\n### **Summary**\nWe appreciate the reviewer’s detailed feedback. We are pleased they found that our work has a clear and compelling motivation, introduces an innovative technical approach, offers rigorous experimental validation, and contributes a significant public dataset.\n\nTo address the reviewer’s concerns, we make the following additions/changes: \n\n**(1)** We perform a comprehensive error analysis for both real-world videos and videos from simulation  (**Table R1**).\n\n**(2)** We add deeper analysis and discussion for extensive real-world experiments (**Table R2**), including clear demonstrations of ROVER’s improved reasoning on real-world examples.\n\n**(3)** We add 3 more modern baselines that are designed for video understanding for our real-world and sim evaluation (**Tables R3, R4, R5**).\n\n**(4)** We add more extensive inspection to provide real-world examples of effective reasoning with tasks without standard object-centric subtasks.\n\n&nbsp;\n### **W1. Lack of Decomposition Failure Analysis**\nWe appreciate this feedback and agree it is critical to include an analysis of the types of errors made by ROVER and GVL. To address this, we hand-reviewed 100 real-world videos and 100 videos from simulation to identify the most common types of errors for each method (Table R1). Below we outline the full list of error types and provide examples for each. We will include this analysis in the main results section of the revised paper.\n\n**Findings:** The most common error type for ROVER is specifying incorrect subtasks, where 9% of real-world videos and 7% of simulation videos include at least one occurrence of this error. The most common error type for GVL is perception errors, where 47% of real-world videos and 32% of simulation videos include at least one occurrence of this error. Further, we observe that 83% of perception errors from GVL occur when there are over 10 frames of the video included in the context. This supports our findings that the VLM is prone to perception errors, such as hallucination, when attempting to reason over many frames of a video. ROVER mitigates these perception errors by shortening the number of frames the model must reason over at each timestep of the video (through the synergistic combination of recursive decomposition and a sliding context window). **Overall, we find that ROVER introduces the possibility of decomposition errors, but dramatically reduces the total error rate by mitigating the model’s tendencies toward hallucination.**\n\n**Error types with examples**\n- Incorrect subtasks\n\t- Example: When decomposing the task “put the mug in the coffee machine”, the model specifies the subtask of “open the cabinet door” when the mug is on the counter.\n- Redundant subtasks\n\t- Example: The model first decomposes the task “put the steak in the microwave” into “pick up the steak from the counter” and “place the steak in the microwave”, but then incorrectly decomposes the  “place the steak in the microwave” subtask into “grasp the steak” and “move the steak to the microwave”.\n- Perception error\n\t- Example: The model states that the gripper is holding an object when it is not.\n- Reasoning error\n\t- Example: Despite correctly identifying the gripper has dropped an object, the model states that task progress is increasing.\n\n&nbsp;\n\n**Table R1. Error rates (%) observed among 100 real-world videos and 100 videos from simulation**\n| |**Real-world**|**Videos**| \\| |**Simulation**|**Videos**|\n|:--|:--|:--|:--|:--|:--|\n|**Error type**|**GVL &darr;**|**ROVER &darr;**| \\| |**GVL &darr;**|**ROVER &darr;**|\n|Incorrect subtasks|NA|9%| \\| |NA|7%|\n|Redundant subtasks|NA|3%| \\| |NA|4%|\n|Perception error|47%|7%| \\| |32%|5%|\n|Reasoning error|14%|4%| \\| |14%|4%|\n|Other|15%| 3%| \\| |11%|3%|\n|Total|76%|26%| \\| |57%|23%|\n\n&nbsp;\n&nbsp;\n\n### **W2. Uncertain Generalization Scope**\nWe agree that it is important to evaluate our method on real-world videos. We test ROVER on the largest collection of real-world robotics datasets publicly available, the Open X-Embodiment (OXE) datasets. The OXE dataset includes an aggregation of trajectory data from 50 standalone robot datasets that consists of diverse tasks, robots, and camera viewpoints. For each of the 50 datasets, we randomly sample 20 trajectories and evaluate ROVER zero-shot on each of the sampled trajectories. The OXE videos include 20 robot embodiments, over 300 different task specifications, and highly diverse (often non-ideal) camera viewpoints. The significant improvement in performance achieved by ROVER (results described below and real-world demonstrations are included in Appendix K) on these datasets demonstrates its ability to generalize (zero-shot) to real-world conditions, including variations in lighting, clutter, occlusion, and partial observability. **To our knowledge, this is among the most comprehensive real-world VLM evaluations conducted to date across such a heterogeneous set of robot platforms and data sources.** We originally included these results in the appendix, but we will update the main results section to highlight this significant analysis, including performance breakdowns by dataset quality and embodiment diversity. Further, we include the real-world videos in the error analysis above, which will also be added to the main results section. \n\n**Findings:** We include tables (Tables 4 and 5) in Appendix K that show the average correlation each method achieves with the ground-truth value estimates across the OXE datasets when using Gemini-1.5-Pro and GPT-4o as the backbone VLM. We also include clear demonstrations of ROVER’s improved reasoning on real-world videos in Appendix K. For known high-quality datasets collected from human teleoperators with fixed camera placements, such as Bridge, RT-1, and Dobb-E, ROVER achieves a significantly higher correlation with frame number than GVL (highlighted in yellow in the Appendix K tables and shown below in Table R2). \n\n**Table R2. Examples from real-world results**\n| Real-world dataset | GVL Corr. with Ground Truth &uarr;| ROVER Corr. with Ground Truth &uarr;|\n|:--|:--|:--|\n|Bridge |0.457|**0.717**|\n|RT-1| 0.762|**0.817**|\n|Dobb-E|0.487|**0.754**|\n|Berkeley Cable Routing |0.488|**0.752**|\n\n&nbsp;\n&nbsp;\n\n### **W3. Limited Set of Modern Baselines**\nWe fully agree that a comprehensive evaluation should consider recent models tailored for video understanding. **We have added two recent video-specific baselines: VideoLLaMA-3 and Video-LLaVA.** These models are designed specifically for video-language tasks and incorporate temporal modeling and memory mechanisms that target long-sequence video comprehension. **We also add a baseline, VideoGemini,** that concatenates all frames of the video to provide as input to the model, more closely resembling the manner in which Gemini was trained for video reasoning tasks. If you have any other models or methods in mind that you would like us to include as baselines, we would be happy to add them.\n\nWe provide the updated results below for both real-world videos and videos from simulation. Our results indicate that SOTA video reasoning models underperform ROVER in task progress prediction, frame-level reasoning, and video question answering. We hypothesize that recursive task decomposition in ROVER is a highly beneficial prior for reasoning about embodied tasks. We further find that video reasoning models are relatively weaker at frame-level progress prediction and reasoning compared to video question answering, which indicates that there is room for improvement in reasoning about fine-grained robot-object interactions. We will include these results in the main results section of the revised paper. \n\n&nbsp;\n&nbsp;\n\n**Table R3.  Corr. with ground-truth values for frame-level progress prediction (with extended baselines)**\n|Method|Real-world Videos Corr. &uarr;|Simulation Videos Corr. &uarr;|\n|:--|:--|:--|\n|ROVER|**0.62**|**0.56**|\n|GVL|0.31|0.35|\n|TemporalConcat |0.43|0.17|\n|LIV|0.12|0.09|\n|**VideoGemini (new)**|0.46|0.29|\n|**VideoLlama3 (new)**|0.34|0.21|\n|**VideoLLaVA (new)**|0.19|0.14|\n\n&nbsp;\n\n**Table R4. Reasoning error rate for frame-level natural language reasoning (with extended baselines)**\n|Method|Error rate &darr;|\n|:--|:--|\n|ROVER|**45%**|\n|GVL|59%|\n|TemporalConcat|67%|\n|**VideoGemini (new)**|62%|\n|**VideoLlama3 (new)**|71%|\n|**VideoLLaVA (new)**|76%|\n\n&nbsp;\n\n**Table R5. Accuracy for video QA (with extended baselines)**\n|Method|Accuracy &uarr;|\n|:--|:--|\n|ROVER|**77%**|\n|GVL| 51%|\n|TemporalConcat|28%|\n|**VideoGemini (new)**|68%|\n|**VideoLlama3 (new)**|62%|\n|**VideoLLaVA (new)**|58%|\n\n&nbsp;\n&nbsp;\n\n### **W4. Potential Dependence on Task Structure**\n\nWe agree that object-centric subtask decomposition warrants further discussion. Firstly, we would like to distinguish between data generation assumptions and methodological assumptions. In order to generate our simulation benchmark data, we leveraged the data replication technique introduced in MimicGen, which indeed assumes that we can decompose a task into object-centric subtasks. This is used to generate ground-truth task progress values.\n\nOn the other hand, our proposed method, ROVER, does not place strict assumptions on the nature of each subtask. For example, ROVER can predict the task progress for cable routing (Berkeley Cable Routing, Table R3) with high correlation to the ground-truth, even though the task of clipping a cable into a slot cannot be easily described with rigid-body pick-and-place physics. Upon deeper inspection of reasoning traces for this task, we see that ROVER effectively reasons about the gripper “angling the cable toward the slots” observed during the video. We will add these real-world example demonstrations in the revised paper and incorporate this deeper discussion. In general, ROVER assumes that the top-level task has semantically meaningful subtasks, but the nature of these subtasks is limited only by the capabilities of the base vision-language model."}, {"review_id": "NJSxPgoVJM", "reviewer": "Reviewer_Y85s", "summary": "This paper proposes ROVER, a new vision-language model (VLM) video reasoning framework designed to enhance model understanding and reasoning capabilities for long-sequence videos in embodied intelligence tasks. The core innovation lies in recursively decomposing lengthy task videos into shorter, more semantically meaningful sub-task segments, enabling the model to perform more focused and accurate reasoning for each sub-task. The framework introduces a sub-task-level sliding window mechanism that considers only the most relevant frames for the current sub-task at each step, significantly reducing computational complexity and effectively mitigating model \"hallucination\" problems in abnormal or sub-optimal states. The authors implement ROVER using in-context learning and conduct systematic evaluation on large-scale datasets containing both expert and non-expert (perturbed) robot manipulation videos, establishing three benchmark tasks: frame-by-frame task progress estimation, frame-by-frame natural language reasoning, and video question answering.", "questions": "1. **Task decomposition failure analysis**: You acknowledge that reasoning may become fragmented when decomposition fails but lack systematic analysis. Could you provide specific cases and failure modes of task decomposition failures, and discuss potential mitigation strategies?\n\n2. **Simulation-to-real transfer**: Have you conducted any validation for transferring from simulation to real-world scenarios?\n\n3. **Video length performance analysis**: The paper doesn't seem to address performance differences between short and long videos. In task planning, different task types exist (simple, complex, common, long-horizon). Could you supplement experiments comparing ROVER's relative advantages on short vs. long videos?", "limitations": "The authors acknowledge that data generated based on RoboCasa simulation environment may have biases affecting real-world generalization. While mentioning linear scaling, the actual computational overhead of recursive calls and multiple VLM inferences may still be high.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- **Comprehensive experimental evaluation**: Experiments conducted on both self-built RoboCasa dataset and public OpenX Embodiment dataset, covering three major tasks (task progress estimation, frame-by-frame reasoning, and video QA) with convincing results\n- **Consistent performance improvements**: ROVER significantly outperforms strong baseline methods across all benchmark tasks, particularly excelling in reducing model hallucination and improving reasoning accuracy and efficiency\n- **Valuable dataset contribution**: Provides high-quality, diverse expert/non-expert robot video datasets by generating varied non-expert trajectories through expert demonstration perturbation\n- **Algorithmic efficiency**: Linear scaling with video length (compared to quadratic scaling of baselines) represents an important algorithmic improvement\n- **Novel technical approach**: Combines recursive decomposition with sub-task sliding windows, solving the trade-off between losing global information and excessive computational cost\n\n**Weaknesses:**\n- **Limited real-world validation**: Experiments are primarily based on simulation and annotated data; while appendix mentions real-world dataset results, main evaluation remains limited to RoboCasa simulation environment\n- **Task decomposition dependency**: Success relies heavily on correct task decomposition; lacks discussion of failure cases when decomposition introduces unnecessary or invalid sub-tasks\n- **Limited baseline comparisons**: While comparing with GVL and TemporalConcat, lacks comparison with more recent video reasoning methods, particularly those designed for long videos\n- **Insufficient failure analysis**: Acknowledges that reasoning may become fragmented when decomposition fails but lacks systematic analysis of failure modes", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n### **Summary**\nWe thank the reviewer for their thoughtful feedback. We are grateful for their recognition of our comprehensive experimental evaluation, novel technical advancement, significant performance improvements across tasks, and public dataset contribution. \n\nTo address the reviewer’s questions and concerns, our primary changes include the following: \n\n**(1)** We perform a comprehensive error analysis for both real-world videos and videos from simulation  (**Table R6**).\n\n**(2)** We add deeper context and discussion for our extensive real-world validation (**Table R7**), including clear demos of ROVER’s improved real-world reasoning.\n\n**(3)** We add 3 more baselines based on recent methods designed for video understanding for our real-world and sim evaluation (**Tables R8, R9, R10**).\n\n**(4)** We add an analysis of inference cost (**Table R11**).\n\n&nbsp;\n\n### **Q1. Failure analysis and mitigation strategies**\nWe hand reviewed 100 real-world videos and 100 videos from simulation to identify the most common types of errors (Table R6) made by ROVER and GVL. Below we outline the full list of error types and provide specific example cases for each.\n\nThe most common error type for ROVER is specifying incorrect subtasks, where 9% of real-world videos and 7% of simulation videos include at least one occurrence of this error. The most common error type for GVL is perception errors, where 47% of real-world videos and 32% of simulation videos include at least one occurrence of this error. Further, we observe that 83% of perception errors from GVL occur when there are over 10 frames of the video included in the context. This supports our findings that the VLM is prone to perception errors, such as hallucination, when attempting to reason over many frames of a video. ROVER mitigates these perception errors by shortening the number of frames the model must reason over at each timestep of the video (through the synergistic combination of recursive decomposition and a sliding context window). **Overall, we find that ROVER introduces the possibility of decomposition errors, but dramatically reduces the total error rate by mitigating the model’s tendencies toward hallucination.**\n\n**Error types and examples**\n- Incorrect subtasks\n\t- Example case: When decomposing the task “put the mug in the coffee machine”, the model specifies the subtask of “open the cabinet door” when the mug is on the counter.\n- Redundant subtasks\n\t- Example case: The model first decomposes the task “put the steak in the microwave” into “pick up the steak from the counter” and “place the steak in the microwave”, but then incorrectly decomposes the  “place the steak in the microwave” subtask into “grasp the steak” and “move the steak to the microwave”.\n- Perception error\n\t- Example case: The model states that the gripper is holding an object when it is not.\n- Reasoning error\n\t- Example case: Despite correctly identifying the gripper has dropped an object, the model states that task progress is increasing.\n\n&nbsp;\n\n**Table R6. Error rates (%) observed among 100 real-world videos and 100 simulation videos**\n| |**Real-world**|**Videos**| \\| |**Simulation**|**Videos**|\n|:--|:--|:--|:--|:--|:--|\n|**Error type**|**GVL &darr;**|**ROVER &darr;**| \\| |**GVL &darr;**|**ROVER &darr;**|\n|Incorrect subtasks|NA|9%| \\| |NA|7%|\n|Redundant subtasks|NA|3%| \\| |NA|4%|\n|Perception error|47%|7%| \\| |32%|5%|\n|Reasoning error|14%|4%| \\| |14%|4%|\n|Other|15%| 3%| \\| |11%|3%|\n|Total|76%|26%| \\| |57%|23%|\n\n&nbsp;\n\n**Mitigation strategies:** To avoid errors in task decomposition, correct subtasks can be ensured using constraints based on expert-derived schemas, while redundant subtasks can be reduced through deduplication modules using embedding clustering or novelty scoring. We include deeper discussion of mitigation strategies in response to Reviewer JuQn under “Recovery mechanism”.\n\n&nbsp;\n&nbsp;\n\n### **Q2. Simulation-to-real transfer**\nTo test the effectiveness of ROVER in real-world scenarios, we test ROVER on the largest collection of real-world robotics datasets publicly available, the Open X-Embodiment (OXE) datasets. As you mention, we initially include this analysis in the appendix, but we will add the extended analysis to the main results section of the revised paper to ensure it is properly highlighted. We also include clear demonstrations of ROVER’s improved reasoning on real-world videos in Appendix K and include the real-world videos in the error analysis above. \n\nOverall, compared to GVL, ROVER shows significantly higher correlation (Tables 4 and 5 in Appendix K and Table R7 below) with ground-truth values among known high-quality datasets collected from human teleoperators with fixed camera placements (e.g., Bridge, RT-1, and Dobb-E). The OXE videos include 20 robot embodiments, over 300 different task specifications, and highly diverse (often non-ideal) camera viewpoints and environmental conditions. **To our knowledge, this is among the most comprehensive real-world VLM evaluations conducted to date across such a heterogeneous set of robot platforms and data sources.** The significant improvement in performance achieved by ROVER on these datasets demonstrates its ability to generalize (zero-shot) to real-world conditions (including variations in lighting, clutter, occlusion, and partial observability). \n\n\n\n**Table R7. Examples from real-world results**\n| Real-world dataset | GVL Corr. with Ground Truth &uarr;| ROVER Corr. with Ground Truth &uarr;|\n|:--|:--|:--|\n|Bridge |0.457|**0.717**|\n|RT-1| 0.762|**0.817**|\n|Dobb-E|0.487|**0.754**|\n|Berkeley Cable Routing |0.488|**0.752**|\n\n&nbsp;\n&nbsp;\n\n### **Q3. Video length performance analysis**\nWe fully agree it is interesting and relevant to evaluate performance differences between short and long videos. We included this analysis in Appendix H, but will add discussion of these results in the main results section. **Overall, we see that ROVER is less sensitive to video length and video frame rate relative to GVL and TemporalConcat.** We first observe this in the results stratified across task group (Figures 3 and 5 from the main results section), where we see the improved performance of ROVER for the composite tasks (bottom row of each figure), which include much longer videos than the atomic tasks (top row of each figure). We also observe ROVER’s robustness to frame rate when varying the total number of frames used to represent each video between 30 and 240 for the pick-and-place tasks for the progress prediction task (Figure 18 in Appendix H) and frame-level natural language reasoning (Figure 19 in Appendix H). **These figures show that the performance of GVL and TemporalConcat significantly degrades when given the same pick-and-place videos, but with effective frame rate increased.**\n\n&nbsp;\n&nbsp;\n\n### **W1. Limited real-world validation**\nWe agree that we should better emphasize our extensive evaluation on real-world datasets in the main results section, which we detail in our response to Question 2 above. We will revise the main results section to highlight these real-world results more prominently, including performance breakdowns by dataset quality and embodiment diversity.\n\n&nbsp;\n&nbsp;\n\n### **W2. Task decomposition dependency and failure analysis**\nWe include the decomposition failure analysis as part of the comprehensive error analysis in our response to Question 2 above.\n\n&nbsp;\n&nbsp;\n\n### **W3. Limited baseline comparisons**\nWe fully agree that it is important to consider recent methods and models tailored for video understanding. **We have added two recent video-specific baselines: VideoLLaMA-3 and Video-LLaVA.** These models are designed specifically for video-language tasks and incorporate temporal modeling and memory mechanisms that target long-sequence video comprehension.  **We also add the baseline VideoGemini** that concatenates all frames of the video to provide as input to the model, more closely resembling the manner in which Gemini was trained for video reasoning tasks. We provide the updated results below for both real-world videos and videos from simulation. We will update the main results section of the paper to include these baselines. If you have any other methods or models in mind that you would like us to include as baselines, we would be happy to add them.\n\n\n**Table R8.  Corr. with ground-truth values for frame-level progress prediction (with extended baselines)**\n|Method|Real-world Videos Corr. &uarr;|Simulation Videos Corr. &uarr;|\n|:--|:--|:--|\n|ROVER|**0.62**|**0.56**|\n|GVL|0.31|0.35|\n|TemporalConcat |0.43|0.17|\n|LIV|0.12|0.09|\n|**VideoGemini (new)**|0.46|0.29|\n|**VideoLlama3 (new)**|0.34|0.21|\n|**VideoLLaVA (new)**|0.19|0.14|\n\n&nbsp;\n\n**Table R9. Reasoning error rate for frame-level natural language reasoning (with extended baselines)**\n|Method|Error rate &darr;|\n|:--|:--|\n|ROVER|**45%**|\n|GVL|59%|\n|TemporalConcat|67%|\n|**VideoGemini (new)**|62%|\n|**VideoLlama3 (new)**|71%|\n|**VideoLLaVA (new)**|76%|\n\n&nbsp;\n\n**Table R10. Accuracy for video QA (with extended baselines)**\n|Method|Accuracy &uarr;|\n|:--|:--|\n|ROVER|**77%**|\n|GVL| 51%|\n|TemporalConcat|28%|\n|**VideoGemini (new)**|68%|\n|**VideoLlama3 (new)**|62%|\n|**VideoLLaVA (new)**|58%|\n\n&nbsp;\n&nbsp;\n### **W4. Insufficient failure analysis**\nWe include a full error analysis in our response to Question 2 above.\n\n&nbsp;\n&nbsp;\n\n### **Limitation: Computational overhead**\nThe table below shows inference cost per video for ROVER versus baselines. We see that ROVER significantly decreases the average number of tokens per video, despite slightly increasing the number of calls to the model (due to the recursive decomposition). We will include a more thorough discussion of the inference cost in the revised paper.\n\n**Table R11. Inference cost of each method for videos with 30 frames**\n|Method|Avg number of tokens per video|Avg number of VLM calls per video|\n|:--|:--|:--\n|ROVER|47,650|36|\n|GVL|140,307|30|\n|TemporalConcat|134,855|30|"}, {"review_id": "IgZ5i53gl8", "reviewer": "Reviewer_XMz4", "summary": "This paper proposes a recursive visual language model (VLM) reasoning framework designed for video sequences in embodied tasks. The framework enables step-by-step reasoning and enabling localized, temporally bounded reasoning. The authors evaluate their approach on a suite of generated benchmarks covering progress estimation, frame-level reasoning, and video question answering. Across all tasks, the proposed method demonstrates consistent improvements over baselines, highlighting its effectiveness in embodied reasoning.", "questions": "1. Can you provide clearer descriptions (and ideally a diagram) of the $\\phi$ and $\\psi$ functions? They are core to the method but currently hard to follow.\n2. What do the axes in Figure 6 represent and how to analyze this figure? Also, would it be possible to present key numeric results in tables for easier comparison?\n3. Can you include any simple embodied task experiments to better validate the framework in actual planning settings?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The paper introduces a novel recursive reasoning approach that decomposes long-horizon reasoning tasks into subtasks, effectively reducing the burden on the model to retain extensive historical context. The motivation is straightforward and the solution sounds reasonable.\n2. The authors devise a method to generate non-expert trajectories with ground-truth progress annotations from expert demonstrations. They also construct a dedicated benchmark for video reasoning in embodied scenarios, where the trajectories are enriched with fine-grained task progress signals. Extensive ablation and analysis are conducted to validate the proposed method on the benchmarks.\n3. The proposed framework is evaluated on both automatically generated and real-world data. The results show that the method significantly reduces error rates and demonstrates fine-grained understanding of task completion states in embodied scenarios. Moreover, the approach exhibits a degree of generalization.\n\nWeaknesses\n1. The paper provides an unclear and somewhat abstract description of the core functions $\\phi$ and $\\psi$, particularly regarding their input-output and implementation details. A more explicit depiction—ideally in Algorithm 1 and potentially through visual illustrations—would greatly enhance clarity. Furthermore, the recursive nature of the framework raises concerns about error propagation and potential infinite loops in failure cases, which are not discussed in the paper.\n2. Although the framework is designed for embodied tasks, the experimental evaluation is limited to video reasoning benchmarks. While these benchmarks test important capabilities, they fall short of establishing the framework’s effectiveness in a complete embodied planning and execution loop. This limits the conclusions that can be drawn about its applicability in real-world embodied settings.\n3. The paper claims that the use of a sliding window results in linear time complexity with respect to video length. However, this point lacks thorough experimental validation or analysis. Additionally, many experiments rely on correlation with ground-truth as the evaluation metric. The non-absolute nature of this metric raises questions about its adequacy for evaluating task performance in a rigorous and comprehensive way.", "was_revised": true, "final_justification": "After the discussion with the authors during the rebuttal, my major concerns have been solved. I expect the further clarification (e.g., illustration figures and more detailed function clarifications) to be included in the final version of the paper.", "rebuttal": "&nbsp;\n### **Summary**\nWe appreciate the reviewer’s detailed feedback. We are pleased they found that our work introduces an innovative technical approach with significant performance improvements, performs extensive evaluation on real-world data and simulation data, and offers a large new public dataset.\n\nTo address the reviewer’s concerns, we make the following additions/changes: \n\n**(1)** We provide a clearer description of $\\phi$ and $\\psi$ functions.\n\n**(2)** We provider deeper discussion regarding the extension of this work for complete planning and execution loop.\n\n**(3)** We add an analysis showing ROVER’s linear time complexity and significant improvements in inference cost relative to baselines (**Tables R12 and R13**) and highlight expanded performance metics.\n\n&nbsp;\n\n### **Q1. Descriptions of $\\phi$ and $\\psi$ functions**\n\nWe fully agree that the methods section of our paper needs to provide clearer descriptions of the $\\phi$ and $\\psi$ functions and will update the revised paper accordingly. We will also include a more explicit depiction through visual illustrations in the method section. The **input** for the $\\phi$ function includes all of the tokens generated by the parent line of reasoning. For example, in Figure 1, this includes all of the tokens leading up to the moment the model states “The robot needs to:...”. The **output** of the $\\phi$ function is the new task description provided by the parent (based on what the model states the robot needs to do) along with the last frame of the parent line of reasoning. The **input** for the $\\psi$ function includes all of the tokens generated by the child line of reasoning. For example, in Figure 1, this includes all of the tokens leading up to the moment the model states “Task is complete”. The **output** of the $\\psi$ function is the final frame, and corresponding final text description, of the child process. \n\n&nbsp;\n&nbsp;\n### **Q2. Figure 6 axes and providing tables**\nFigure 6 is a histogram where the **x axis** represents the difference in frames between when the VLM states that something occurred (e.g., the robot picks up an object) during a video and when it actually occurred during the video. For example, if the VLM states that the robot picked the object at frame 15 of the video, but the robot actually picks up the object at frame 20 of the video, the frame difference shown in Figure 6 would be -5 on the x-axis. If the VLM instead states that the robot pick up the object at frame 25, the frame difference shown in Figure 6 would be +5 on the x-axis. Therefore, negative frame differences in Figure 6 indicate the VLM is stating things occur before they actually occur during the video. The **y-axis** is simply the number of videos for which we observe each frame difference calculation. Overall, Figure 6 supports our findings that GVL has an hallucination problem where it frequently states things are occurring before they actually occur during the video. We will make sure to update x-axis and y-axis descriptions for Figure 6 to make this much clearer in the revised paper. \n\n&nbsp;\n&nbsp;\n### **Q3 and W2. Embodied task experiments**\n\nWe are excited to experiment with on-robot planning in ongoing/future work. The focus of this paper is primarily on benchmarking fine-grained task progress prediction and natural language reasoning given videos, but we fully agree that progress prediction has downstream applications in both planning and policy learning, and we are actively pursuing these directions as follow-up work.\n\n&nbsp;\n&nbsp;\n### **W1. $\\phi$ and $\\psi$ functions and error propagation**\nWe provide more thorough descriptions of the $\\phi$ and $\\psi$ functions (particularly their inputs and outputs) in response to Question 1 above. We will add clearer descriptions, along with a visual illustration, of these functions in the methods section of the revised paper. \n\nIn our response to Reviewer AqtN, we conducted an error analysis by hand reviewing 200 videos, calculating the rate of different errors. The most common decomposition error type we observe with ROVER is specifying incorrect subtasks. For example, when decomposing the task “put the mug in the coffee machine”, the model might make an error by specifying the subtask of “open the cabinet door” when the mug is on the counter. The other decomposition error type to occur involved specifying redundant subtasks. For example, the model first decomposes the task “put the steak in the microwave” into the subtasks “pick up the steak from the counter” and “place the steak in the microwave”, but then incorrectly decomposes the  “place the steak in the microwave” subtask into “grasp the steak” and “move the steak to the microwave”. \n\nIf the model makes these errors, we do not enforce any explicit recovery mechanism. If the subtask is simply redundant and is already completed (e.g., stating the robot needs to grasp an object after it has already picked it up), the model will often recognize the redundant subtask (grasping the object) has already been completed and assign a subtask completion of 100%, causing it to move to the next subtask (placing the object) and proceed without an issue. If a subtask is incorrect, this generally causes the model to get stuck as you describe and fail in the overall video reasoning task. We discuss ideas for potential mitigation strategies below. \n\n**Error mitigation strategies:** The most direct strategy for mitigating errors associated with specifying incorrect subtasks is to incorporate subtask validity constraints during attempted decompositions. Specifically, we can leverage task schemas or symbolic priors, derived from expert demonstrations or domain knowledge, to enforce that only logically and temporally valid subtasks are generated. Errors associated with specifying redundant subtasks could be mitigated with a subtask deduplication module, implemented via either clustering of subtask embeddings or attention-based novelty scoring. This module can flag subtasks that are semantically repetitive with recent ones and merge or skip them accordingly. We could also prevent the potential unstable recursive spiraling (which is often driven by repetitive subtask specifications) by preventing the model from further decomposing a subtask after a certain recursion depth is reached.\n\n&nbsp;\n&nbsp;\n### **W3. Adding analysis of time complexity and expanded evaluation metrics**\n\n&nbsp;\n\n**Empirical validation of linear time complexity claim:** We appreciate the reviewer’s concern about the experimental support for our claim of linear time complexity with respect to video length. In the revised manuscript, we will provide a more detailed empirical analysis of inference cost as a function of video length. Specifically, we measure two key indicators of inference cost across tasks with different video lengths: (1) the total number of tokens processed per video and (2) the number of VLM calls. Tables R12 and R13 (below) show that ROVER consistently reduces token count compared to baseline methods, even though it requires a slightly higher number of model calls due to its recursive structure. Importantly, the increase in inference cost with respect to video length is approximately linear, consistent with our design goal. For instance, doubling the video length from 30 to 60 frames results in approximately a doubling of token usage for ROVER (from ~47k to ~95k tokens), supporting the linear scaling claim. Due to ROVER scaling linearly with video length (with baselines scaling quadratically), the improvements in inference cost we observe with ROVER increase as video length increases. For videos with 30 frames (the single-stage atomic tasks), ROVER shows about a 3x reduction in the number of tokens per video (Table R12). For videos with 60 frames (the multi-stage composite tasks), ROVER shows more than a 5x reduction in the number of tokens per video (Table R13). We will include these results and an expanded discussion in the revised manuscript to provide a clearer empirical basis for our time complexity claim.\n\n**Table R12. Inference cost of each method for videos with 30 frames**\n|Method|Avg number of tokens per video|Avg number of VLM calls per video|\n|:--|:--|:--\n|ROVER|47,650|36|\n|GVL|140,307|30|\n|TemporalConcat|134,855|30|\n\n&nbsp;\n\n**Table R13. Inference cost of each method for videos with 60 frames**\n|Method|Avg number of tokens per video|Avg number of VLM calls per video|\n|:--|:--|:--\n|ROVER|95,404|68|\n|GVL|539,102|60|\n|TemporalConcat|527,572|60|\n\n&nbsp;\n\n**Expanded performance metrics:** We agree that relying solely on Pearson correlation, a relative metric, for the progress prediction evaluation can be insufficient for comprehensive performance evaluation. To address this, in Appendix A (Figure 9), we include results across all task groups using an absolute error metric (L2 distance) for the progress prediction task. We will update the main results section of the revised paper to emphasize this. Further, we are happy to incorporate any additional performance metrics that you see fit."}, {"review_id": "LZQBJb40AL", "reviewer": "Reviewer_JuQn", "summary": "The paper proposes an in-context learning approach paired with a recursive algorithm to improve VLM's ability at video understanding for embodied videos. Prior works have to balance global reasoning (on the entire video) and localized reasoning (on a small set of frames). In contrast, the proposed approach, ROVER, identifies subtasks, which recursively spawn another instance of ROVER to reason about the subtask. ROVER also maintains a sliding window that includes context from the previous subtask and the first frame of the current subtask. The paper tests ROVER on a curated dataset of robot videos with various levels of expertise (where worse trajectories are achieved via random perturbation) and videos from the Open-X dataset, showing higher accuracy, especially on suboptimal videos, over other baselines, which tend to hallucinate successes.", "questions": "* What are some failure cases for incorrect/unncessary subtasks generated by ROVER?\n* In Figure 5, why does the ROVER somtimes exhibit an increasing trend in error rate even though the level of expertise in a video increases (e.g., for pick-and-place, and prepare-coffee)?\n* What is the size of the sliding window? Is it just 3 frames? Or is it the most recent previous frame, the first frame of the current line of reasoning, plus N number of frames? How does the sliding window size affect ROVER's performance?\n\nSome additional questions are sprinkled throughout the weaknesses sections.", "limitations": "Yes the authors have addressed both. \n* Another potential limitation is that: the author assumes that the next subtask can be predicted given the video frames seen so far, which is generally true for short-horizon tasks. Consider a task with ambiguous order (e.g., put multiple objects into a box). The LLM might predict the robot to begin with object A first, making that the subtask, but the video might begin with object B.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Quality\n\n## Strengths\n\n* The paper evaluates ROVER and baselines on 3 tasks, some testing an approach's ability to recognize progress and others testing the reasoning and understanding via QA. The newly introduced dataset exposes VLM's tendency to hallucinate success even when a video fail. And they also test on an existing benchmark for the progress prediction. \n* Ablation tests the contribution of its two main component in the approach: (1) recursive algorithm, and (2) sliding window. \n* The paper also tested the model's performance on various models.\n\n## Weaknesses\n* The paper is missing a baseline that performs local reasoning over a small set of frames. L217 mentions having LIV [1] as a baseline, but I cannot find any results in the main paper nor the appendix. If the author includes this baseline in rebuttal, I am willing to increase my score. \n* Compared to other baselines (GVL and TemporalCat), ROVER is making more VLM calls as it recurses. What is the token cost for ROVER compared to other baselines?\n\n[1] Ma, Y. J., Kumar, V., Zhang, A., Bastani, O., and Jayaraman, D. (2023). Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pages 23301–23320. PMLR\n\n# Clarity\n\n## Strengths\n\n*  The paper is well-written and well-organized with informative figures that clearly explain the approach.\n\n# Significance & Originality\n\n## Strengths\n\n* The paper presents a new dataset with programmatically augmented subtoptimal demonstrations. Even though there might be a risk that these demonstrations are mismatched from suboptimal demonstrations in the real world, it is still interesting experiment results that the VLM tends to hallcinate success when there's suboptimal behavior.\n* Figure 6 also presents interesting results that GVL which examines the video holistically tends to predict subtask occuring earlier compared to the ground-truth time. \n* Results with TemporalCat supports GVL's claim that VLM has a tendency to predict increasing progress regardless of the input frames. \n\n## Weaknesses\n* The recursive algorithm does not seem to have a recovery mechanism. L114 says that the algorithm returns if the VLM determines the current subtask has been completed or if it reaches the end of the video. What happens when the spawned subtask is incorrect? In that case, a successful video would get marked as unsuccessful since that incorrect subtask fails to complete until the end of the video. What is authors' thought on mitigating this?\n* Different approaches seem to require different compute and different number of LLM calls, making it difficult to compare them fairly.", "was_revised": true, "final_justification": "I appreciate the author's follow-up responses, and they have addressed most of my concerns and questions. I will increase my score accordingly. I expect the detailed analysis (e.g., failure cases, inference cost) and additional baselines to be included in the final version of the paper.", "rebuttal": "&nbsp;\n### **Summary**\nWe thank the reviewer for the comprehensive feedback. We are grateful for their recognition of our consistent performance improvements across diverse tasks, comprehensive experimental evaluation (for real-world and simulation) with ablations and different model backbones, and public dataset contribution. \n\nTo address the reviewer’s questions and concerns, we make the following additions/changes: \n\n**(1)** We perform a comprehensive error analysis of ROVER (**Table R14**).\n\n**(2)** We add an evaluation of how the sliding window size affects ROVER’s performance (**Table R15**).\n\n**(3)** We add a baseline for local reasoning (**Tables R16 and R17**).\n\n**(4)** We add an analysis showing ROVER’s significant improvements in inference cost (**Tables R18 and R19**).\n\n**(5)** We provide deeper discussion of recovery mechanisms.\n\n&nbsp;\n## **Questions**\n### **Q1. Failure cases**\nTo identify failure cases, we conducted an error analysis by hand reviewing 200 videos, calculating the rate of different errors (Table R14). Examples of each error type are shown in the response to Reviewer AqtN. The most common decomposition error type we observe with ROVER is specifying incorrect subtasks. For example, when decomposing the task “put the mug in the coffee machine”, the model might make an error by specifying the subtask of “open the cabinet door” when the mug is on the counter. The other decomposition error type to occur involved specifying redundant subtasks. For example, the model first decomposes the task “put the steak in the microwave” into the subtasks “pick up the steak from the counter” and “place the steak in the microwave”, but then incorrectly decomposes the  “place the steak in the microwave” subtask into “grasp the steak” and “move the steak to the microwave”. \n\n&nbsp;\n\n**Table R14. Error rates (%) observed among 100 real-world videos and 100 videos from simulation**\n| |**Real-world**|**Videos**| \\| |**Simulation**|**Videos**|\n|:--|:--|:--|:--|:--|:--|\n|**Error type**|**GVL &darr;**|**ROVER &darr;**| \\| |**GVL &darr;**|**ROVER &darr;**|\n|Incorrect subtasks|NA|9%| \\| |NA|7%|\n|Redundant subtasks|NA|3%| \\| |NA|4%|\n|Perception error|47%|7%| \\| |32%|5%|\n|Reasoning error|14%|4%| \\| |14%|4%|\n|Other|15%| 3%| \\| |11%|3%|\n|Total|76%|26%| \\| |57%|23%|\n\n&nbsp;\n&nbsp;\n\n### **Q2. Trend in error rate**\nWe appreciate the reviewer’s close reading of Figure 5. The observed fluctuations in error rate, where ROVER sometimes exhibits a slightly higher frame-level reasoning error even at higher levels of expertise (in pick-and-place and prepare-coffee tasks), can be attributed primarily to natural variance in the dataset and non-uniform difficulty across trajectories. The fluctuations in error rate are within the standard error margins and do not reflect a statistically significant trend in ROVER’s performance as trajectory quality improves. In addition, some of the higher-level trajectories still contain challenging edge cases. For instance, minimal deviations in timing or subtle failures in object manipulation may lead to increased reasoning difficulty even though the overall trajectory is successful. We will add deeper discussion of these trends in the revised paper.\n\n&nbsp;\n&nbsp;\n\n### **Q3. Size of the sliding window**\nThe sliding window includes 3 frames: the first frame of the current line of reasoning, the most recent previous frame, and the frame for the current timestep of the video. In Table R15 below, we show the performance of ROVER on the frame-level progress prediction task when changing the sliding window size to 6, 9, and 12 frames. We see that, as the number of frames included in the sliding window increases, the performance of ROVER drops. We will add these results to the revised paper.\n\n**Table R15.  Corr. with ground-truth values for frame-level progress prediction (with varying sliding context window size)**\n|Method|Corr. with Ground Truth &uarr;|\n|:--|:--|\n|ROVER (original)|**0.56**|\n|ROVER (window of 6)|0.52|\n|ROVER (window of 9)|0.50|\n|ROVER (window of 12)|0.44|\n\n&nbsp;\n&nbsp;\n\n## **Quality**\n### **W1. Adding baseline for local reasoning**\nWe appreciate this feedback and agree that it is important to include a baseline for local reasoning. We include **LIV as a local-reasoning baseline for the frame-level progress prediction task**, as shown in the first results figure (Figure 3, page 6). Since LIV does not provide text outputs, we could not include it as a baseline for the frame-level natural language reasoning task or the video QA task. However, to include a reference for local reasoning for these tasks, **we add LocalConcat as a baseline** method in these evaluations, which is similar to TemporalConcat, but only includes the last three frames of the video in the model's context at each timestep. We provide the results of LocalConcat for frame-level reasoning (Table R16) and video QA (Table R17) below. We will add this to the main result section of the revised paper. \n\n**Table R16. Reasoning error rate for frame-level natural language reasoning (with extended baselines)**\n|Method|Error rate &darr;|\n|:--|:--|\n|ROVER|**45%**|\n|GVL|59%|\n|TemporalConcat|67%|\n|**LocalConcat (new)**|62%|\n\n&nbsp;\n\n**Table R17. Accuracy for video QA (with extended baselines)**\n|Method|Accuracy &uarr;|\n|:--|:--|\n|ROVER|**77%**|\n|GVL| 51%|\n|TemporalConcat|28%|\n|**LocalConcat (new)**|25%|\n\n\n&nbsp;\n&nbsp;\n\n### **W2. Adding analysis of inference cost**\nThe tables below show the inference cost per video for ROVER versus baselines. ROVER significantly decreases the average number of tokens per video, while slightly increasing the number of calls to the model (due to the recursive decomposition). For videos with 30 frames (the single-stage atomic tasks), ROVER shows about a **3x reduction** in the number of tokens per video (Table R18). For videos with 60 frames (the multi-stage composite tasks), ROVER shows more than a **5x reduction** in the number of tokens per video (Table R19). In the revised paper, we will include these results and a more thorough discussion of the observed improvements in inference cost with ROVER relative to baselines.\n\n**Table R18. Inference cost of each method for videos with 30 frames**\n|Method|Avg number of tokens per video|Avg number of VLM calls per video|\n|:--|:--|:--\n|ROVER|47,650|36|\n|GVL|140,307|30|\n|TemporalConcat|134,855|30|\n\n&nbsp;\n\n**Table R19. Inference cost of each method for videos with 60 frames**\n|Method|Avg number of tokens per video|Avg number of VLM calls per video|\n|:--|:--|:--\n|ROVER|95,404|68|\n|GVL|539,102|60|\n|TemporalConcat|527,572|60|\n\n&nbsp;\n&nbsp;\n\n## **Significance & Originality**\n### **W1. Recovery mechanism**\nAs detailed in our response to Question 1 above, the most common decomposition errors we observe include errors associated with specifying incorrect subtasks or redundant subtasks. As you mention, if the model makes these errors, we do not enforce any explicit recovery mechanism. If the subtask is simply redundant and is already completed (e.g., stating the robot needs to grasp an object after it has already picked it up), the model will often recognize the redundant subtask (grasping the object) has already been completed and assign a subtask completion of 100%, causing it to move to the next subtask (placing the object) and proceed without an issue. If a subtask is incorrect, this generally causes the model to get stuck as you describe and fail in the overall video reasoning task. We discuss ideas for potential mitigation strategies below. \n\n**Mitigation strategies:** The most direct strategy for mitigating errors associated with specifying incorrect subtasks is to incorporate subtask validity constraints during attempted decompositions. Specifically, we can leverage task schemas or symbolic priors, derived from expert demonstrations or domain knowledge, to enforce that only logically and temporally valid subtasks are generated. Errors associated with specifying redundant subtasks could be mitigated with a subtask deduplication module, implemented via either clustering of subtask embeddings or attention-based novelty scoring. This module can flag subtasks that are semantically repetitive with recent ones and merge or skip them accordingly. We could also prevent the potential unstable recursive spiraling (which is often driven by repetitive subtask specifications) by preventing the model from further decomposing a subtask after a certain recursion depth is reached.\n\n&nbsp;\n&nbsp;\n### **W2. Adding analysis of inference cost**\nWe agree that it is important to consider inference cost when evaluating the performance of each method. Overall, ROVER **significantly reduces inference cost** relative to baselines. We include an analysis of the inference cost above (Tables R18 and R19). We see that, although ROVER slightly increase the number of calls to the model as it recurses, it dramatically decreases the number of tokens per video by shortening the number of frames the model must reason over at each timestep of the video (through the synergistic combination of recursive decomposition and a sliding context window). Further, since ROVER **scales linearly** with video length (while baselines **scale quadratically** with video length), the improvements in inference cost we observe with ROVER increase as video length increases.\n\n&nbsp;\n&nbsp;\n## **Limitations**\nThe issue of ambiguous subtask order is important and will be addressed in more detail in the limitations section. One solution is to explicitly specify task order (e.g., specifying the task as “place object A in box, then place object B in box” instead of “place objects A and B in box”). Alternatively, when that’s not feasible, the model can use order-agnostic phrasing, guided by appropriate prompting examples (e.g., given the task “place objects A and B in box”, the model can specify the first subtask as “place one of the objects in box” and the second subtask as “place the remaining object in box”)."}]

Author rebuttal:

Meta-review: The paper proposes an approach for video reasoning in embodied settings to primarily address the limitations of VLMs in reasoning over long sequences of frames. The paper received divergent ratings: 2 Accept and 2 Borderline Reject. The reviewers mentioned various concerns such as: (1) Lack of analysis, (2) Limited set of modern baselines, (3) Limited real-world validation, (4) Lack of validation for linear time complexity, and (5) No testing inside a complete planning and execution loop. The rebuttal addressed most of the issues. For instance, the authors addressed (2) by providing new baseline results such as VideoGemini and addressed (4) by providing a new analysis. 

After the discussions, there were some concerns such as being limited to only table-top validation. The AC checked the paper, the reviews and the rebuttal. The AC believes the positives of the paper outweighs its negatives, and the paper addresses an important problem with thorough experiments. Hence, acceptance is recommended. However, the authors are encouraged to explore applications beyond simple table-top scenarios and use the proposed method inside a complete robotic pipeline.
Decision: Accept (poster)