Title: ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision-language model

video reasoning

embodied intelligence

recursion
Abstract: Vision-language models (VLMs) have exhibited impressive capabilities across diverse image understanding tasks, but still struggle in settings that require reasoning over extended sequences of camera frames from a video. This limits their utility in embodied settings, which require reasoning over long frame sequences from a continuous stream of visual input at each moment of a task attempt. To address this limitation, we propose ROVER (Reasoning Over VidEo Recursively), a framework that enables the model to recursively decompose long-horizon video trajectories into segments corresponding to shorter subtasks within the trajectory. In doing so, ROVER facilitates more focused and accurate reasoning over temporally localized frame sequences without losing global context. We evaluate ROVER, implemented using an in-context learning approach, on diverse OpenX Embodiment videos and on a new dataset derived from RoboCasa that consists of 543 videos showing both expert and perturbed non-expert trajectories across 27 manipulation tasks. ROVER outperforms strong baselines across three video reasoning tasks: task progress estimation, frame-level natural language reasoning, and video question answering. We observe that, by reducing the number of frames the model reasons over at each timestep, ROVER mitigates model hallucinations, especially during unexpected or non-optimal moments of a trajectory. In addition, by enabling the implementation of a subtask-specific sliding context window, ROVER's time complexity scales linearly with video length, an asymptotic improvement over baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an in-context learning approach paired with a recursive algorithm to improve VLM's ability at video understanding for embodied videos. Prior works have to balance global reasoning (on the entire video) and localized reasoning (on a small set of frames). In contrast, the proposed approach, ROVER, identifies subtasks, which recursively spawn another instance of ROVER to reason about the subtask. ROVER also maintains a sliding window that includes context from the previous subtask and the first frame of the current subtask. The paper tests ROVER on a curated dataset of robot videos with various levels of expertise (where worse trajectories are achieved via random perturbation) and videos from the Open-X dataset, showing higher accuracy, especially on suboptimal videos, over other baselines, which tend to hallucinate successes.\n\nSTRENGTHS AND WEAKNESSES:\n# Quality\n\n## Strengths\n\n* The paper evaluates ROVER and baselines on 3 tasks, some testing an approach's ability to recognize progress and others testing the reasoning and understanding via QA. The newly introduced dataset exposes VLM's tendency to hallucinate success even when a video fail. And they also test on an existing benchmark for the progress prediction. \n* Ablation tests the contribution of its two main component in the approach: (1) recursive algorithm, and (2) sliding window. \n* The paper also tested the model's performance on various models.\n\n## Weaknesses\n* The paper is missing a baseline that performs local reasoning over a small set of frames. L217 mentions having LIV [1] as a baseline, but I cannot find any results in the main paper nor the appendix. If the author includes this baseline in rebuttal, I am willing to increase my score. \n* Compared to other baselines (GVL and TemporalCat), ROVER is making more VLM calls as it recurses. What is the token cost for ROVER compared to other baselines?\n\n[1] Ma, Y. J., Kumar, V., Zhang, A., Bastani, O., and Jayaraman, D. (2023). Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pages 23301–23320. PMLR\n\n# Clarity\n\n## Strengths\n\n*  The paper is well-written and well-organized with informative figures that clearly explain the approach.\n\n# Significance & Originality\n\n## Strengths\n\n* The paper presents a new dataset with programmatically augmented subtoptimal demonstrations. Even though there might be a risk that these demonstrations are mismatched from suboptimal demonstrations in the real world, it is still interesting experiment results that the VLM tends to hallcinate success when there's suboptimal behavior.\n* Figure 6 also presents interesting results that GVL which examines the video holistically tends to predict subtask occuring earlier compared to the ground-truth time. \n* Results with TemporalCat supports GVL's claim that VLM has a tendency to predict increasing progress regardless of the input frames. \n\n## Weaknesses\n* The recursive algorithm does not seem to have a recovery mechanism. L114 says that the algorithm returns if the VLM determines the current subtask has been completed or if it reaches the end of the video. What happens when the spawned subtask is incorrect? In that case, a successful video would get marked as unsuccessful since that incorrect subtask fails to complete until the end of the video. What is authors' thought on mitigating this?\n* Different approaches seem to require different compute and different number of LLM calls, making it difficult to compare them fairly.\n\nQUESTIONS:\n* What are some failure cases for incorrect/unncessary subtasks generated by ROVER?\n* In Figure 5, why does the ROVER somtimes exhibit an increasing trend in error rate even though the level of expertise in a video increases (e.g., for pick-and-place, and prepare-coffee)?\n* What is the size of the sliding window? Is it just 3 frames? Or is it the most recent previous frame, the first frame of the current line of reasoning, plus N number of frames? How does the sliding window size affect ROVER's performance?\n\nSome additional questions are sprinkled throughout the weaknesses sections.\n\nLIMITATIONS:\nYes the authors have addressed both. \n* Another potential limitation is that: the author assumes that the next subtask can be predicted given the video frames seen so far, which is generally true for short-horizon tasks. Consider a task with ambiguous order (e.g., put multiple objects into a box). The LLM might predict the robot to begin with object A first, making that the subtask, but the video might begin with object B.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a recursive visual language model (VLM) reasoning framework designed for video sequences in embodied tasks. The framework enables step-by-step reasoning and enabling localized, temporally bounded reasoning. The authors evaluate their approach on a suite of generated benchmarks covering progress estimation, frame-level reasoning, and video question answering. Across all tasks, the proposed method demonstrates consistent improvements over baselines, highlighting its effectiveness in embodied reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper introduces a novel recursive reasoning approach that decomposes long-horizon reasoning tasks into subtasks, effectively reducing the burden on the model to retain extensive historical context. The motivation is straightforward and the solution sounds reasonable.\n2. The authors devise a method to generate non-expert trajectories with ground-truth progress annotations from expert demonstrations. They also construct a dedicated benchmark for video reasoning in embodied scenarios, where the trajectories are enriched with fine-grained task progress signals. Extensive ablation and analysis are conducted to validate the proposed method on the benchmarks.\n3. The proposed framework is evaluated on both automatically generated and real-world data. The results show that the method significantly reduces error rates and demonstrates fine-grained understanding of task completion states in embodied scenarios. Moreover, the approach exhibits a degree of generalization.\n\nWeaknesses\n1. The paper provides an unclear and somewhat abstract description of the core functions $\\phi$ and $\\psi$, particularly regarding their input-output and implementation details. A more explicit depiction—ideally in Algorithm 1 and potentially through visual illustrations—would greatly enhance clarity. Furthermore, the recursive nature of the framework raises concerns about error propagation and potential infinite loops in failure cases, which are not discussed in the paper.\n2. Although the framework is designed for embodied tasks, the experimental evaluation is limited to video reasoning benchmarks. While these benchmarks test important capabilities, they fall short of establishing the framework’s effectiveness in a complete embodied planning and execution loop. This limits the conclusions that can be drawn about its applicability in real-world embodied settings.\n3. The paper claims that the use of a sliding window results in linear time complexity with respect to video length. However, this point lacks thorough experimental validation or analysis. Additionally, many experiments rely on correlation with ground-truth as the evaluation metric. The non-absolute nature of this metric raises questions about its adequacy for evaluating task performance in a rigorous and comprehensive way.\n\nQUESTIONS:\n1. Can you provide clearer descriptions (and ideally a diagram) of the $\\phi$ and $\\psi$ functions? They are core to the method but currently hard to follow.\n2. What do the axes in Figure 6 represent and how to analyze this figure? Also, would it be possible to present key numeric results in tables for easier comparison?\n3. Can you include any simple embodied task experiments to better validate the framework in actual planning settings?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ROVER, a new vision-language model (VLM) video reasoning framework designed to enhance model understanding and reasoning capabilities for long-sequence videos in embodied intelligence tasks. The core innovation lies in recursively decomposing lengthy task videos into shorter, more semantically meaningful sub-task segments, enabling the model to perform more focused and accurate reasoning for each sub-task. The framework introduces a sub-task-level sliding window mechanism that considers only the most relevant frames for the current sub-task at each step, significantly reducing computational complexity and effectively mitigating model \"hallucination\" problems in abnormal or sub-optimal states. The authors implement ROVER using in-context learning and conduct systematic evaluation on large-scale datasets containing both expert and non-expert (perturbed) robot manipulation videos, establishing three benchmark tasks: frame-by-frame task progress estimation, frame-by-frame natural language reasoning, and video question answering.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- **Comprehensive experimental evaluation**: Experiments conducted on both self-built RoboCasa dataset and public OpenX Embodiment dataset, covering three major tasks (task progress estimation, frame-by-frame reasoning, and video QA) with convincing results\n- **Consistent performance improvements**: ROVER significantly outperforms strong baseline methods across all benchmark tasks, particularly excelling in reducing model hallucination and improving reasoning accuracy and efficiency\n- **Valuable dataset contribution**: Provides high-quality, diverse expert/non-expert robot video datasets by generating varied non-expert trajectories through expert demonstration perturbation\n- **Algorithmic efficiency**: Linear scaling with video length (compared to quadratic scaling of baselines) represents an important algorithmic improvement\n- **Novel technical approach**: Combines recursive decomposition with sub-task sliding windows, solving the trade-off between losing global information and excessive computational cost\n\n**Weaknesses:**\n- **Limited real-world validation**: Experiments are primarily based on simulation and annotated data; while appendix mentions real-world dataset results, main evaluation remains limited to RoboCasa simulation environment\n- **Task decomposition dependency**: Success relies heavily on correct task decomposition; lacks discussion of failure cases when decomposition introduces unnecessary or invalid sub-tasks\n- **Limited baseline comparisons**: While comparing with GVL and TemporalConcat, lacks comparison with more recent video reasoning methods, particularly those designed for long videos\n- **Insufficient failure analysis**: Acknowledges that reasoning may become fragmented when decomposition fails but lacks systematic analysis of failure modes\n\nQUESTIONS:\n1. **Task decomposition failure analysis**: You acknowledge that reasoning may become fragmented when decomposition fails but lack systematic analysis. Could you provide specific cases and failure modes of task decomposition failures, and discuss potential mitigation strategies?\n\n2. **Simulation-to-real transfer**: Have you conducted any validation for transferring from simulation to real-world scenarios?\n\n3. **Video length performance analysis**: The paper doesn't seem to address performance differences between short and long videos. In task planning, different task types exist (simple, complex, common, long-horizon). Could you supplement experiments comparing ROVER's relative advantages on short vs. long videos?\n\nLIMITATIONS:\nThe authors acknowledge that data generated based on RoboCasa simulation environment may have biases affecting real-world generalization. While mentioning linear scaling, the actual computational overhead of recursive calls and multiple VLM inferences may still be high.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ROVER, a recursive vision-language framework designed for embodied robotic tasks. It enhances reasoning by decomposing long-horizon videos into manageable subtasks. At its core, ROVER recursively segments task trajectories to maintain global context, while a sliding window within each subtask focuses the model on temporally relevant frames. This dual approach allows it to effectively balance high-level task structure with fine-grained local details. Evaluated on a new, diverse RoboCasa dataset and other large-scale benchmarks, ROVER significantly outperforms leading baselines (GVL, TemporalConcat, LIV) in task progress estimation, frame-level reasoning, and video QA. Notably, it mitigates model hallucinations and achieves linear time complexity, scaling efficiently with video length.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Clear and Compelling Motivation: The work is well-motivated, addressing the fundamental trade-off in VLM-based video reasoning between maintaining global context and achieving computational efficiency.\n2. Innovative Methodology: The core contribution is ROVER’s innovative methodology, which synergistically combines recursive task decomposition with a sliding context window. This elegant approach simultaneously enables fine-grained, focused reasoning while maintaining computational efficiency.\n3. Rigorous and Extensive Experiments: The claims are supported by a rigorous empirical evaluation using a novel, large-scale RoboCasa dataset alongside established benchmarks. This provides robust validation across diverse and realistic manipulation tasks, strengthening the generalizability of the findings.\n4. Valuable Community Resource: The public release of the dataset, code, and demonstrations constitutes a valuable contribution to the research community.\n\nWeaknesses:\n\n1. Lack of Decomposition Failure Analysis: The framework's performance heavily relies on successful task decomposition, yet the paper lacks a quantitative analysis of potential failure modes. The frequency and impact of incorrect subtask splits are not explored, making it difficult to assess the method's robustness.\n2. Uncertain Generalization Scope: The primary contributions and in-depth analysis are centered on simulated robotic manipulation. The framework's applicability to other challenging video domains, such as open-world videos or real-world scenarios, remains undemonstrated.\n3. Limited Set of Modern Baselines: The comparison could be strengthened by including more recent architectures designed for long-sequence video understanding, such as advanced transformer variants or memory-augmented VLMs, which would provide a more competitive performance benchmark.\n4. Potential Dependence on Task Structure: The assumption that tasks can be decomposed into object-centric subtasks with clear sequential transitions (as in RoboCasa) may not hold for more entangled or ambiguous real-world tasks. The implications or limits of this assumption warrant deeper discussion, beyond referencing previous work.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.\n\nLimitations are adequately acknowledged (Section 7 and Broader Impact), with constructive discussion on decomposition failure, reliance on in-context prompting, and the risk of increased autonomy in AI systems.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}