Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations
Authors: Shivansh Patel
Shraddhaa Mohan
Hanlin Mai
Unnat Jain
Svetlana Lazebnik
Yunzhu Li
Venue: ICLR 2026 Poster (2026)
Keywords: Foundation Models based Robot Manipulation
Vision-based Robotics
Generative Video Models
6D Pose Estimation
Primary area: applications to robotics, autonomy, planning
TL;DR: Our method enables robots to execute manipulation tasks purely from generated videos, no real‑world demos needed

Abstract: This work introduces Robots Imitating Generated Videos (RIGVid), a system that enables robots to perform complex manipulation tasks—such as pouring, wiping, and mixing—purely by imitating AI-generated videos, without requiring any physical demonstrations or robot-specific training. Given a language command and an initial scene image, a video diffusion model generates potential demonstration videos, and a vision-language model (VLM) automatically filters out results that do not follow the command. A 6D pose tracker then extracts object trajectories from the video, and the trajectories are retargeted to the robot in an embodiment-agnostic fashion. Through extensive realworld evaluations, we show that filtered generated videos are as effective as real demonstrations, and that performance improves with generation quality. We also show that relying on generated videos outperforms more compact alternatives such as keypoint prediction using VLMs, and that strong 6D pose tracking outperforms other ways to extract trajectories, such as dense feature point tracking. These findings suggest that videos produced by a state-of-the-art off-the-shelf model can offer an effective source of supervision for robotic manipulation.

Published as a conference paper at ICLR 2026
ROBOTIC MANIPULATION BY IMITATING GENERATED VIDEOS WITHOUT PHYSICAL DEMONSTRATIONS
Shivansh Patel
Shraddhaa Mohan'
Hanlin Mai!
Unnat Jain
Svetlana Lazebnik"
Yunzhu Li
UIUC
UC Irvine
SColubia University
ABSTRACT
This work introduces Robots Imitating Generated Videos (RIGVid), system that enables robots to perform complex manipulation tasks-such as pouring, wip- ing, and mixing-purely by imitating Al-generated videos, without requiring any physical demonstrations or robot-specific training. Given a language command and initial scene image, video diffusion model generates potential demon- stration videos, and a vision-language model (VLM) automatically filters out results that do follow the command. 6D pose tracker then extracts ob- ject trajectories from the video, and the trajectories are retargeted to the robot an embodiment-agnostcc fashion. Through extensive real-world evaluations. we show that filtered generated videos are effective as real demonstrations, and that performance improves with generation quality. We also show that re- lying on generated videos outperforms more compact alternatives such as key- point prediction using VLMs, and that strong 6D pose tracking outperforms other ways to extract trajectories, such as dense feature point tracking. These find- ings suggest that videos produced by a state-of-ther -ortt---rttttt off-the-shelf model can offer an effective source of supervision for robotic manipulation. Project page: httts::/riivii: robot q t.ub. io/
INTRODUCTION
Videos offer a rich and expressive source training data for robotic manipulation, and numer- methods have successfully leveraged them for supervision. Such methods typically fall into two categories: (1) Learning from publicly available large-scale datasets of real-world videos (Bahl) eet al. 2023] Ye et al., 2024). and (2) Imitation of specific demonstrations captured under controlled conditions that closely match the execution setting (Bahl et al. [2022] Kareer et al., 2024). Unfor- tunately, both of these strategies come with challenges that limit broad deployment. Large-scale video datasets often introduce domain gaps (Zhou et al. 2024 Gao et al. 2025) and require adapta- tion specific robot embodiments and tasks (Bahl al.) 2023; O' Neill al. 2023). On the other hand, video-based imitation involves laborious data collection that must ensure close alignment in viewpoints, morphologies, and interaction modalities (Bahl et al., 2022 Dasari & Gupta 2021).
Motivated by recent advances in video generation, we explore a new paradigm: can single gener- ated video, generated to exactly match our input environment and task description, be used as the sole source of supervision for robotic manipulation? Recently released models like SORA (Brooks) al. 2024) and Kling (Kli 2024) have demonstrated impressive capabilities in producing realistic videos from language and image inputs. At the same time, it has been shown that such videos suffer from distorted object geometries (Liu et al. 2024| Zhang et al. 22024), physically implausi- ble interactions (Motamed et al., 2025) Yang et al. 2025). and unrealistie scene dynamics (Bansal let al. 2024; Guo et al. (025). Consequently. while the idea of synthesizing video demonstrations enticing, usefulness the robotics setting is yet be convincingly established. Prior work incorporating video generation into robotics typically relies on additional supervision, such as task- specific training (Du et al. 2024) fine-tuning on offline robot trajectory datasets (Bharadhwaj et al. 2024b a). By contrast, we ask whether robot can perform real-world manipulation tasks solely by imitating generated dilos-witthut any additional supervision or task- speecffii training.
"denotes equal advising.

---

Published as a conference paper at ICLR 2026
Pour Plant
Generated Video
Object Trajectory
Grasp & (Object Robott Retanee Retarget
Image
Video
6D Onier T
Monocular Depth Estimation
Death Map
Scale Factor
Prodicted Dapth
Figure 1: RIGVid overview. Given an initial scene image and depth, we generate a video conditioned on a language command. A VL.M-based automatic filtering step (not shown) can be used to reject videos that fail to follow the prompt. A monocular depth estimator recovers depth for each frame of the generated video, and these depth maps combined with the corresponding RGB frames produce 6D Object Pose Trajectory. After grasping, the trajectory is retargeted to the robot execution.
To this end, we introduce Robots Imitating Generated Videos (RIGVid), a framework that con- nects video generation models real-world robotic execution. Fig. D gives oveview of the method. Given an input RGB-D image of the scene and a free-form language command (e.g., "pour water on the plant"), we use a state-of-the-art video diffusion model generate candidate video of the task. The generated video is not guaranteed to accurately follow the language command but we show that a VLM can be used to automatically filter out unsuccessful generations with high ac- curacy. Next, we estimate per-frau per-frame depth on the video, segment the manipulated object, and track its 6D object pose trajectory across the frames using the FoundationPose tracker (Wen et al. 2023b).
While this tracker relies on pre-computed object mesh, preliminary experiments (App. C indicate that our method is compatible with mesh-free approaches, though their inference speed is currently infeasible for real-time deployment. Finally, the extracted 6D object pose trajectory is retargeted to the robot for execution. The retargeting process only requires the transformation between the end-
effector and the object, it can be easily applied across platforms. During deployment, RIGVid performs real-time object tracking and dynamically adjusts the robot's actions to handle disturbances and execution-time variations, promoting robust and adaptive behavior.
We evaluate RIGVid on four real-world manipulation tasks: spatula on pan, and sweeping trash. These tasks span diverse manipulation challenges, including range of depth variation (minimal in pouring vs. significant in lifting). thin and partially occluded objects (spatula, sweeping brush), and diverse object geometries and actions. Our results show that, when paired with our filtering mechanism, generated videos are as effective as human videos for vi- sual imitation, enabling robots to act entirely from synthetic supervision. Moreover, the performance of RIGVid improves with video quality, suggesting favorable trend where advances in generative models directly translate to stronger manipulation capabilities.
pouring water, lifting a lid, placing a
The main downside of video generation is its substantial computational cost. Also, a represen- tational level, one may wonder whether predicting video pixels is wasteful, and whether we should instead predict compact and minimal representation that can be efficiently translated to ex- ecutable trajectory. One example of this philosophy is the recent ReKep method (Huang et al. 2024a), which uses VLM generate relational keypoint constraints from task deseription and then solves for a 6D trajectory. We compare our approach to ReKep and demonstrate that video generation does, in fact, perform substantially better than the generation of more sparse and high- level representation. Next, given generated video, one may ask whether 6D object-level tracking is necessary, given its up-front requirement of an object mesh. To address this question, we compare against a broad range of alternative tracking approaches
sparse point tracking (Bharadhwaj et al.) 2024b), dense optical flow (Ko et al. (022). 3D feature fields (Kerr al., 2024), and generated goal supervision (Bharadhwaj et al. 2024a) and show consistently higher success rates.
In summary, our contributions are: (1) We propose a framework that enables robots to perform open- open-world manipulation tasks without any real-worke demonstrations, only generated videos. (2) We show high-quality generated videos perform on par with real videos for robotic imitation, es- tablishing that synthetic data can serve as an effective substitute for real data. (3) We demonstrate that our combination of video generation and 6D trajectory extraction outperforms competing SOTA methods based on VLMs, point tracking, optical flow, feature fields, and generated- -goal supervision.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Direct Imitation from Videos. This seeks match visual states in demonstration videos those of the learner, without requiring expert action labels or robot states (Dasari & Gupta 2021 Valas- sakis et al.) 2022). While effective, this approach demands paired demonstrations in the same set- ting. A common strategy is to leverage visual correspondences trrespondencee-ttackss (Bharadhwaj et al., 2024b) or optical flow (Argus et al. 2020 Xu et al. 2024)-10 to infer object trajectories. Bharadhwaj et al. (Bharadhwaj et al.) 2024b) predicts object tracks and uses PnP to recover poses for closed-loop task execution. Dense descriptor learning (Florence et al., 2018 Zhu et al. (024) has proven power- ful for handling variations in object geometry and appearance. Kerr et al. (Kerr et al. (2024) recover object part trajectories from monocular videos using feature fields. Crucially, these methods rely on demonstrations collected under conditions closely matching the target task. In contrast, our method removes this requirement by generating task and conditiond videos.
Imitation from Offfine Videos. This paradigm alleviates the need for paired demonstrations by leveraging offfine video data, and has consequently attracted significant attention (Smith et al. 2019 Liu et al. 2018). Many works focus learning affordance models from internet-scale video datasets (Bahl et al. 2023) Srirama et al., 2024). Here, affordances are defined as scene-centric predictions of where and how an agent can interact, often captured as contact-point heatmaps and short motion trajectories that can be translated into robot actions. For example, Bahl el al. (Bahl jet al. 2023) learn from large-scale human videos to output dense contact maps and trajectory way- points, which downstream imitation, exploration, reinforcement modules can transform into ex- ccutable robot motions. However, these methods suffer from domain gap between training videos and spec-fpe environments, and require additional mechanisms to obtain task-conditioned af- fordances. In contrast, our method does not explicitly predict affordances, but instead relies on generated, task- and scene-specific generated videos for imitation.
Video Generation for Robotics. Video generation has emerged as a promising avenue for robotics (Du et al., 2024 2023). A common limitation of these is their reliance on robot data, either to train the video generation model (Liang et al. 2004; [Sun et al.] al.] (2024). or to train policies (Bharad- hwaj et al.) 2024a). or both (Du et al.j [024]] 2023). Bharadhwaj et al. (Bharadhwaj et al. 2024a) leverages tracks on generated videos to condition policy learning. Albaba et al. (Albaba et al. 2025 uses generated videos to compute rewards for RL training. The most closely related work is Liang et al. (Liang et al. 2024), which executes robotic tasks by tracking tools attached to the robot's 's end effector. While effective, their method relies on 1,822 human-collected robot demonstrations for four tasks, and is confined to tasks executable only by tools. In contrast, our approach requires no such data collection. Instead of tools, our method tracks objects, enabling a significantly broader range of manipulation tasks without using any robot data.
Motion Retargeting for Object Manipulation. Early work in learning from demonstration es- tablished the foundation for motion retargeting (Gleicher) 1998 Calinon 2016). More recently, deep learning- based retargeting methods have emerged (Cheng et al. 2024 Choi et al., 2020), with some incorporating object-centric representations to bridge the gap between the demonstrator and the robot (Kerr et al., 2024 Li et al., 2024). Many approaches have applied retargeting to humanoid robots (Hu et al. 2014). Other works have extended these techniques to dexterous manipulation (Qin) et al. 2022; Lakshmipathy et al. 2024). Like most prior work, we assume fixed transformation between the end-effector and the object. While motion retargeting has traditionally relied on human demonstrations, RIGVid eliminates this dependency by leveraging generated videos.
OUR METHOD: ROBOTS IMITATING GENERATED VIDEOS
An overview of our method is shown in Fig. T It takes as inputs the initial scene RGB image, its corresponding depth map, and free-form human command. Our goal is to predict the robot's 6DoF end-effector trajectory. This section describes the key steps of RIGVid: (1) Generate a scene and task-conditioned video and predict its corresponding depth using a monocular depth estimator (Sec. 3.1: (2) Compute 6D pose rollout via an object pose tracker (Sec. 3.2): (3) Grasp the object
and retarget the pose trajectory to the robot, and execute the resulting trajectory (Sec. 3.3 (..c.3.3)
3.1 GENERATING VIDEOS AND CORRESPONDING DEPTH
Since the generated videos may not necessarily follow the language command or have other issues, need automatic filtering mechanism to discard inaccurate generations. We found that we can
3

---

Published as a conference paper at ICLR 2026
do the filtering reliably by prompting VLM - specifiiclllly GPT-4o (Achiam et al. 2023 assess whether the generated video depicts a successful execution of the command. As image input to GPT- 4o, sample four evenly spaced frames in the video and concatenate them vertically to create video summary. The VLM determines whether the action described in the command is performed by visible hand. App. [ provides the full prompt used for filtering and examples of video summaries with their corresponding VLM responses. If the response is negative, we regenerate the video and repeat the process for up to five attempts. If all attempts fail, we default the final attempt.
As input to the downstream tracking step, we also need to predict the depth for the generated video, using the predictor from Ke al. (Ke et al., 2024). One complication that the estimated depth is not grounded in real-world units, but subject to scale-shift ambiguity (Hartley & Zisserman 2003) Consistent with prior works adopting depth estimators in vision- basee robotics (Gervet et al. 2023 we compute an affine scale-and-shift transformation, aligning the predicted depth in the first frame with the initial real depth map around the active object (discussed in Sec. 3.2). This transformation is then applied to the entire predicted video to resolve the ambiguity.
3.2 IDENTIFYING ACTIVE OBJECT MASK AND 6D OBJECT POSE TRAJECTORY
To extract 6D pose rollout, we first identify the active object-the one being manipulated in the generated video. A binary mask for this object in the initial RGB image is essential for object tracking and determining which object to grasp. Given the initial image and the task command, we prompt GPT-4o to identify the object most likely to be manipulated. We then ground the predicted object category into a bounding box using Grounding DINO (Liu et al. )023), and further refine this into segmentation mask using SAM-2 (Ravi et al. 2024).
Once the active object is localized by the mask we track across the generated video using the scaled predicted depth. This yields the 6D pose rollout. Tracking objects in videos is rich area of research, and we experimented with several 6D pose trackers (Labbé et al.] 2022 Wen et al.) b0) For real-world deployment, we found FoundationPose (Wen et al., 2023b)
perform
the best. It requires an object mesh, which we pre-compute using BundleSDF (Wen et al. 2023a). For this, we record short RGBD video where the object is held and rotated in front of the camera to capture all sides. While straightforward, this process constrains our method to settings where a mesh can be precomputed. Nonetheless, as shown in App. C our method is also compatible Brrruuches-BuddleDDD. can jointly reconstruct and track the object-but current
with mesh-free inference speeds make these alternatives infeasible for real-time use. To ensure stable and realistic motion during execution, we apply an averaging filter to smooth abrupt pose changes, particularly in rotation. Additional details on this smoothing step are provided in App. D
3.3 OBJECT TO ROBOT MOTION RETARGETING
Once the object trajectory is obtained, we first grasp the object. We use an off-the-shelf grasper, AnyGrasp (Fang al. 2023), identify and execute the highest-scoring grasp within a defined boundary around the active object mask. After grasping, we retarget its trajectory to the robot's end- effector. Since the object remains firmly grasped, we assume fixed transformation between the robot's end-effector and the object. This transformation is obtained by composing two rigid-bou body transforms: (1) the pose of the object relative to the gripper at the moment grasped and (2) the offset between the gripper and the robot's s end-effector. By combining these two components, we obtain a single transformation from the end-effector to the object.
The corresponding end-effector trajectory is obtained by applying the fixed end-effictor-tt-objee transformation to the object's pose along the entire trajectory. This formulation ensures that the retargeted 6D pose rollout follows the object's 's motion while maintaining a stable grasp. These are executed on the physical robot, enabling it to reproduce the object's movement as observed in the generated video. A key strength of this approach is that it is robot-agnostic. Specifically, to accommodate a different robot or gripper, only the end-effector to the object transformation needs to be updated to reflect the new end-effector configuration.
3.4 CLOSED LOOP EXECUTION
A core strength of our approach is its ability to operate in a closed-loop manner, enabling robust execution despite disturbances unexpected changes during task exccution. During deployment, the system continuously tracks the object's 6D pose in real time using FoundationPoss: to update the robot's end-effector trajectory as the task progresses. This feedback allows the robot dynamically
4

---

Published as a conference paper at ICLR 2026
adjust its motions: if the object deviates from the planned trajectory due to external perturbations, such as a human pushing the robot or a slip after grasping, the system detects the discrepancy by comparing the current object pose to the precomputed trajectory. If the detected deviation exceeds a threshold of cm in position or 20 degrees in orientation, the robot backtracks to the last success- fully executed trajectory point and resumes execution from there (Fig- 2 This recovery mechanism enables RIG Vid to maintain stable task execution, realigning and successfully completing the ma- nipulation task despite perturbations. Additional examples of robustness are provided App. H
EXPERIMENTS
This section presents our experimental evalua- tion. We describe the robot setup. manipula- tion tasks, and evaluation protocol (Sec. 4.1). Then assess the impact of video generation models and filtering strategies downstream robotic performance (Sec. (42), Next, we com- pare RIGVid to SOTA VLM-based trajectory prediction method that allows zero-shot exe- cution (Sec. (.3). and to alternative tracking approaches for trajectory extraction (Sec. 4.44. Finally, test generalization across embodi- ments, extensions to new tasks, and robustness real-world disturbances (Sec. 4.5)
interupts
trijectory
Robot backtracks
Successful execurion
Figure 2: RIGVid is robust to perturbations, A hu- man pushes the robot during execution (image 1), caus- ing the object to deviale from the trajectory. When the deviation is detected (image 2). the robot backtracks to the last successfully executed trajectory point (image 3) and then resumes the planned motion (image 4).
4.1 ROBOT
SETUP, TASKS, AND EVALUATION
Pour the plant
Lift lid pot
Place spatula pan
Sweep dustpan
Figure 3: Evaluation tasks. We evaluate RIGVid on everyday manipulation tasks of varying difficulty.
We conduct experiments on an xArm7 robot
arm with a stationary Orbbec Femto Bolt camera, positioned next to the robot capture RGBD observations. We evaluate our method on four everyday manipulation tasks, which are illustrated in Fig. These span a diverse range robotic challenges, and their descriptions are as follows:
1. Pouring water requires the robot to position and tilt watering can above plant. While the depth the can relative to the camera remains largely constant, successful execution demands smooth trajectory spanning the pick-up, transport, and pouring phases. A trial is considered successful if the watering can't 's spout positioned above the plant at the end of the execution.
2. Lifting a lid requires the robot to lift a pot lid. Unlike pouring, where the camera is viewing the scene from the side, the camera here is looking down towards the pot. As result, this task involves significant variation in object depth, as the lid moves closer to the camera during execution. Success is achieved if the lid no longer in contact with the pot the end of the trial.
3. Placing a spatula on a pan requires the robot to place the head of a spatula into pan. The spatula has thin, elongated geometry and is often partially occluded during manipulation, which presents challenge for object tracking, particularly for non-mesh-based approaches. The task is considered successful if the spatula's 'ss head is in the pan the end of execution.
4. Sweeping trash requires the robot to sweep trash into a dustpan. This task is especially challenging as it combines the need for precise positioning to align the head of the sweeping brush with the trash, along with all the challenges encountered from the placing task. A trial is successful if the trash is touching the base of the dustpan at the end of the execution.
Task success is determined via human judgment based on the above criteria. though the procedure could be readily automated with VLM. The initial setup configuration is fixed across trials of the
5

---

Published as a conference paper at ICLR 2026
Qualitative comparison video generation for three models. Sora (top) drastically alters scene layout and object size. Kling v1.5 (middle) does not fully follow the prompt (water not poured over the plant) and exhibits physically implausible behaviors (water pouring out of the top of the kettle but not the spout). Kling v1.6 (bottom) produces the most consistent and realistic result.
Figure 4:
same task, and each trial has different generated video. All baselines the same videos for consistent comparison and reporting.
4.2 QUALITY AND FILTERING OF GENERATED VIDEOS
As discussed in Sec. 3.1 we experimented with Sora, Kling v1.5, and Kling v1.6 for video genera- tion. We also compared different video filtering strategies. Next, we present our key findings.
How do different video generation models compare for robotic imitation? Sora is known for creating visually impressive and cinematic videos. Unfortunately, these videos often prioritize aesthetics over following the human command. As seen in the top of Fig. 4 Sora frequently alters the camera viewpoint, changes object positions, or even swaps out objects mid-sequence. This lack of scene and object consistency makes Sora poorly suited for imitation. Kling v1.5 places more emphasis on following language instructions, generally preserves the original scene, and correctly depicts the target object. Nonetheless, it is still prone physically implausible behaviors and command following failures. In the second row of Fig. 4 the teapot is not positioned over the plant and the water pours out from the top, not the spout (in other failure cases, nothing all happens in the video, and the command is not even attempted). By contrast, Kling v1.6 (bottom row of Fig. 4 has greatly improved command following and physical plausibility, proving to be the most reliable video generator for us. More examples of generated videos are shown in App. Fig. 19
What are the filtering statistics for different video generation models?
Pour Water
Lift
Place Spatula
Sweep Trash
Filtering pass rate
80
Confirming the trends
60
described above, Fig. [5]
reports the pass rates of
each model across our four
20
tasks for the GPT-4o filter
described in Sec. 3.1 Sora fails all tasks 100% of the time. Kling v1.5 does better, successfully passing pouring 52.6% of the time, lifting 27.7%, placing 4%, and sweeping 2%. Kling V1.6 shows a substantial improvement across tasks, passing pouring 83%, lifting 66%, placing 55%, and sweeping 45% of the time. We noticed that, particularly for the placing and sweeping tasks, even Kling V1.6 frequently generated videos in which the command was not followed. In many cases, the video remained static, and no action was performed.
Som
Kling V1.5
kling V1.6
Sora
Kling v1.5
Kling V1.6
Sara
Kling V1.5
Kling V1.6
Som
Kling V1.5
Kling v1.
Figure 5: Filtering statistics. Kling V1.6 videos have the highest pass rate, demonstrating more accurate adherence to language commands.
How accurate is VLM-based video filtering, and there any simpler alternatives? In Tab. I report Pearson correlation coefficients between filtering metrics and human judgments of genera- tion correctness. Our VLM-basee filtering achieves strong agreement with human ratings across

---

Published as a conference paper at ICLR 2026
Filtering Metrics
Pour Water
Lift Lid
Place Spatula
Sweep Trash
Average
Video-text Consistency
0.06
0.47
0.70
0.11
0.34
12V Subject Consistency Querying GPT oll
0.35
0.58
-0.09
0.37
0.91
0.91
0.91
0.66
0.84
Table 1: Comparison of video filtering metrics. Pearson correlation coefficients measure each metric's effectiveness in assessing whether generated video follows the language command. Querying GPT ol proves be most effective.
Pour Water
Place Spatula
Lift Lid
Sweep Trush
100
60
40
20
0
Sara
Kling V1.SUF
Kling V1.6 uP
Kling Vi&EE
Real
Sara
Klong VL.SUF
Klmp V1.4 up
Kling Vies
Roal
Strr
Kling V1.5UP
Kling V1.6 UP
Kling V1.6B
Real
Sora
Kligg UF
king V16 UP
Klnng V
Real
Figure RIGVid performance vs. video quality. The dashed lines separate performance on generated videos from real videos. Kling V1.6 produces most reliable videos and leads to highest RIGVid success.
Filtered videos perform on par with real ones. UF denotes unfiltered and F denotes filtered.
all tasks, with high correlation values. Most errors made by the VLM-based filter are false nega- tives-oocasionally discarding usable videos, but almost never passing incorrect We also explore the most relevant metrics for our case from recent benchmark suite for evaluating video generation quality and instruction following. VBench++ (Huang al. 2024b): (i) video-text consis- tency measuring the aligmment between the command and the generated video (Wang et al. (023), and (ii) image-to- -o-idooo (I2V) subject consistency which evaluates whether subjects present in the input image persist throughout the video (Caron et al., (021). These metrics correlate only weakly or inconsistently with task success and are not reliable for filtering.
Does higher video quality lead to better robol performance? ? To quantify this, Fig. plots RIGVid 's task success across five video sources: unfiltered Sora, unfiltered Kling v1.5, unfiltered Kling v1.6. filtered Kling v1.6, and real human demonstration videos. For each source, use 10 videos per task. We observe a clear trend: as video quality improves, so does success rate. Sora's unfiltered videos lead to 0% success rate, Kling v1.5 performs better, and Kling v1.6 gives the highest results among all generated videos. Filtering dramatically improves reliability: after discarding failed gen- erations using
automatic approach, success rate with filtered Kling v1.6 videos improves from 80% to 100% on pouring, from 60% to 80% on lifting, from 50% to 90% on placing, and from 20% to 70% on sweeping.
Can generated videos replace real videos for imitation?? The results in Fig. indicate that, when using filtered Kling vl.6 videos, RIGVid 's performance is similar that achieved with real human demonstration videos. This finding suggests that, at current model quality, generated videos are already sufficient for visual imitation, substantially reducing the need for manual data collection.
What causes failure of imitation given high-quality videos? Aside from one case where the object slipped out of the gripper, all failures on filtered Kling v1.6 videos are attributed to errors in depth estimation. These errors result in inaccurate 6l) trajectories and lead to tracking failures. Notably, similar depth estimation issues are also observed in real videos, suggesting that limitation lies in the depth model itself. App. A provides detailed analysis of failure cases with qualitative examples.
4.3 RIGVID vs. VLM-BASER TRAJECTORY PREDICTION
Video generation is computationally expensive, prompting the question of whether more efficient alternatives can enable robot manipulation without any demonstrations. VLMs offer one poten- tial alternative by predicting simplified task abstractions-gou states (Huang et al.) 2023). con- straints (Huang et al. 2024a). reward functions (Patel et al.) 20tho-ithhuut generating full visual sequences. making them cheaper in computation and inference time. We take the state-of- the-art ReKep (Huang al. 2024a) method representative of this line of work, and compare against Fig. []] In our comparison, RIGVid achieves 85% vs. ReKep's 50% success over four tasks. App. E illustrates ReKep's failures, which attribute inaccurate keypoint predictions.

---

Published as a conference paper at ICLR 2026
This comparison suggests that, for tasks and experimental setup. VLM--genratte abstractions are compact and may lack the rich, necessary details for successful robot execution. Thus, despite its higher cost, video generation provides crucial supervision rather than being wasteful expense.
While this result highlights, for tasks and setup, the addi- tional detail in generated videos
Pour Water
Lift Lid
Place Spatula
Sweep Trash
100
80
supports more reliable execution than the current VLM-based al-
60
ternative, it does not rule out the
40
possibility that future or alter- native VI.M-based approaches could close this gap. Our find- ings suggest that, at present, video generation can provide richer supervision for manipula- tion compared to this specific VLM-based method, despite its higher computational cost.
0
ReKep (VLM-Based)
Ours
Figure 7: RIGVid ReKep Success Rates. RIGVid outperforms
SOTA VLM-based trajectory prediction method.
4.4 COMPARISON ALTERNATIVE TRAJECTORY EXTRACTION METHODS
We investigate the best way to extract trajectory information from videos for the purpose of visual imitation. To this end, adapted several competitive methods that use different types of tracking to imitate a video without demonstrations. For each method, we describe its inputs and outputs, core approach, our modifications, and the motivation for its inclusion (additional details in App. F
Track2Act (Bharadhwaj et al. 2024b) (tracks-Based) This method takes an RGBD image of the initial scene, and a single goal image that specifies the desired final configuration. Since we have no other way to get the goal image, we set it to the last frame of the generated video. Using only this pair of images, Track2Act uses learned model to predict dense grid of 2D point tracks, producing pixel-level correspondences hetween the initial and goal image. These tracks are then lifted to 3D) using the depth map from the initial frame and converted into a sequence of 3D object poses via the Perspective-n-Peee (PnP) algorithm. We do not finetune their track prediction network, and do not use their residual policy. Track2Act is an attractive alternative as it uses dedicated track prediction network that operates solely on the start and goal images, without requiring any intermediate frames. However, its main drawback is that the learned track prediction network may not generalize to all scenarios, as evidenced by our experiments and qualitative results.
(Flow-Based). Given an initial RGBD image, task description, and active object mask, AVDC predicts object motion by first generating task-conditioned video and then computing optical flow between frames. This optical flow is used in an optimization process to reconstruct the object trajectory. In adaptation, substitute AVDC's original video generator with our improved model, while preserving all downstream processing. Unlike Track2Act, AVDC leverages optical flow across the entire video, giving it dense temporal correspondences at every pixel and thus many more cues for tracking. It is attractive because offers a denser input for object tracking. Nevertheless, it is sensitive cumulative errors in flow estimation, which degrade the accuracy of the resulting object trajectories.
AVDC (Ko al. 2023)
4D-DPM (Kerr et al. 2024) (Feature Field-Based). This method takes a 3D Gaussian splatting field of the object and a real video of the task, and outputs object trajectories over time. A feature field, similar to NeRF representations,
continuous mapping from 3D space to high-dimensional feature vectors that capture both geometry and appearance. By aligning the feature field with indi- vidual video frames, the method can estimate object trajectory across the video. To build the field, 4D-DPM requires separate video where the object is captured from all sides. In our adaptation, since 4D-DPM expects a real human demonstration video, we instead use a generated video as the task input video. We modify this method from tracking object part poses to tracking single objects. This approach is compelling because applics semantic, feature-bast based reasoning to track objects, capturing entire object structure from video, without relying on explicit correspondences. However, it tends to produce unstable tracking in our experiments, limiting its practicality.
Gen2Act (Bharadhwaj et al. 2024a) (Generated Goal-Based). Gen2Act takes input an RGBD image of the scene and a task description, and outputs robot actions predicted by a learned pol- icy. In the original formulation, the extracted tracks on the generated video were used to supervise
8

---

Published as a conference paper at ICLR 2026
Pour Water
Lift Lid
Place Spatula
Sweep Trash
Average
100
80
60
40
20
ADC
AD-DPM
Art
Ours
TrackAAtt
AVDC
AD-DPM
Gen2An
Ours
Arack2Act
AV ADC
AD-DPM
Gen2An
Oure
Arack2Ast
AVDC
ID-DPM
Gea2Aet
Ours
Trel2Act
AVDC
AD-DPM
Gen2Adt
Ours
Figure 8: Comparative evaluation of trajectory extraction methods. RIGVid consistently achieves higher success across all four tasks; relative improvements higher tasks become harder (i.e., from left right).
large offline robotics dataset. In our adaptation, do not any policy learning. Instead, we extract object tracks from the generated video and directly estimate object poses from these tracks using the initial depth image. This removes any dependence demonstration data or learned policies. Gen2Act is notable for leveraging sparse correspondences extracted from the generated video, enabling task-relevant object motion be tracked and retargeted without requiring explicit actions. However, when large portions of the object become occluded or undergo significant rotations, many of the tracking points are lost, resulting too few correspon- dences to estimate object pose accurately and ultimately causing the tracking to fail.
schaviour-cloning
expert
Fig. 8 shows that RIGVid achieves a success rate 85.0%, compared 67.5% for Gen2Act and considerably lower rates for all other baselines. This margin grows with more complex tasks. Meth- ods such as Track2Act (7.5%), AVDC (32.5%), and 4D-DPM (35.0%) rely on point tracks or op- tical flow, but their performance remains limited, especially as object rotations occlusions severe. Gen2Act, which combines video generation with point-based tracking, closes part of the gap but consistently struggles when large portions of the object become untrackable. In contrast, RIGVid's use of 6D object pose trajectory enables robust execution, accounting for the 17.5% im- provement over Gen2Act. This advantage persists when more powerful tracking models like Co- Tracker3 (Krrevv et al. 2024) are used, as shown in App. G
Looking the task-wise breakdown in Fig. 8 RIGVid maintains high success rates depth varies significantly (such as in the lifting task) or when the objects are thin, small, or partially occluded (such in placing spatula or sweeping trash). Other methods frequently struggle in these
object
TrackZAct (Tracks)
AVDC (Flow)
GonZAct (Gencrated Gool)
4D:DPM (Feature Field)
RIGVid Object Pese Traj)
Robot Execution
Figure 9: Analyzing intermediate visual representations. Only Gen2Act and our 6D Object Pose Trajectory can correctly track the position and rotation of the watering can, leading to successful execution. Check the description in the main paper for detailed discussions of the failure modes of the alternative methods.

---

Published as a conference paper at ICLR 2026
settings, often failing to recover object trajectories when objects become partially hidden change distance rapidly. The advantage of RIGVid is most pronounced on the most challenging tasks: for both spatula placement and sweeping, RIGVid achieves success rates 20-25% above the next best baseline. These results suggest that the structured 6D pose trajectory not only enables robust tracking under depth changes and occlusion but also scales to scenarios where correspondence methods fail.
Visualizing the outputs in Fig. 9 for the same generated video, we observe the intermediate pre- dictions and resulting robot executions. For Track2Act, the predicted tracks diverge from the true object path, leading to failed execution. Often, the track2act track prediction does not follow the true motion paths, which the primary source of errors in our experiments. AVDC generates reasonable optical flow in individual frames, but when summed across the entire video, the resulting trajectory often physically implausible, and the execution fails. We often found that this summing up of object flow across the video leads to small errors that accumulate over the entire video, resulting in faulty object location across the video. Gen2Act yields plausible tracks and leads to successful manipulation. We often found that tracks were accurate, and the resulting trajectory after PnP was also accurate. 4D-DPM exhibits inconsistent tracking performance. While it accurately follows the object in certain segments, the example shown reveals incorrect tracking during the first half of the episode, which ultimately causes the rollout to fail. We often found that the tracking was unsta- ble and very jerky. In contrast, the 6D object pose trajectories produced by RIGVid remain stable throughout the episode and closely match the actual object motion, resulting in successful execution.
4.5 TESTING GENERALIZATION
Embodiment-Agnostic
Transfer. We test RIGVid's generalizability another embodiment by deploying it on the ALOHA robot for the pouring task (Fig. TO top left). On this setup, it achieves 80% success, compared to 100% on
Tuble
Shirt
Bimanual Place Shoes
Upright Ketchup
Unplug Charger
Spill Bearn
our default xArm setup ! RIGVid also generalizes to bimanual setup, simulta- neously placing a pair of shoes into a box using both arms (Fig. I0 bottom left).
Figure 10: RIGVid's emmodiment----gossicccc capabilities and examples on solving complex, open-world tasks. RIGVid can readily work on ALOHA setup (Zhao et al. 2023) as shown on top left. On the bottom left, RIGVid is retargeted the bimanual ALOHA setup. On the right, it generates jectories for diverse manipulation tasks including wiping, mixing, and iron- ing without using any physical demonstrations.
Extensions Additional Tasks. Besides our four main focus tasks, we also obtained promising preliminary results on a larger variety of diverse and challenging manipulation tasks shown in Fig. IO (right). These tasks include wiping, mixing, and ironing, uprighting ketchup bottle, unplugging charger, and rotating spoon to spill beans. Notably, the latter three tasks involve extreme rotations, which RIGVid can handle successfully.
5 CONCLUSIONS
We introduced RIGVid, the first method for robotic manipulation that works from just generated videos. By leveraging recent advances in generative vision and pose estimation, RIGVid enables robots execute complex tasks entirely from generated video. We extract 6D Object Pose Trajec- tory from the generated videos and retarget to the robot, demonstrating a data-efficient approach to robotic skill acquisition. Our analysis shows correlation between video quality and task success. Additionally, our comparisons with SOTA VLM-based manipulation methods confirm that leverag- ing dense visual cues from generated videos yields more reliable performance. We also show that RIGVid significantly outperforms competing trajectory extraction methods across a diverse set of tasks, and demonstrate the robustness of our approach to disturbances. Our work represents step toward enabling robots to learn from the visual knowledge in generative models, reducing reliance on time-consuning data collection.
TTe performance drop stems from inaccurate camera calibration of ALOHA's wrist cameras.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENT
We thank the members of the RoboPIL. lab, and UIUC vision and robotics labs for their valuable dis- cussions and feedback. Unnat would like to especially acknowledge Chen Bao, Homanga Bharad- hwaj, Shikhar Bahl, and friends at CMU and Skild for their insightful conversations on learning from videos. We also thank Justin Kerr for his assistance in reproducing the 4D-DPM baseline. This work is partially supported by the Toyota Research Institute (TRI), the Sony Group Corpora- tion, Google, Dalus AI, and an Amazon Research Award, Fall 2024. This article solely reflects the opinions and conclusions of its authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
REFERENCES
Kling ai. https://www wwtp://wwigga.c.... com// 2024. Accessed: 2024-02-10.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Mert Albaba, Chenhao Li, Markos Diomataris, Omid Taheri, Andreas Krause, and Michael Black. Nil: No-data imitation learning by leveraging pre-trained video diffusion models. arXiv preprint arXiv:2503. 10626, 2025.
Max Argus, Lukas Hermann, Jon Long, and Thomas Brox. Flowcontrol: Optical flow based visual servoing. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7534-7541 IEEE, 2020.
Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. arXiv preprint (xXiv:2277.0945/0 2022.
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from hu- man videos as a versatile representation for robotics. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13778-13780 2023.
Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical common- sense for video generation. arXiv preprint 24Xiv:2406.03520, 2024.
Homanga Bharadhwaj. Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. GenZact: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint a6X2v:2409.16283, 2024a.
Homanga Bharadhwaj. Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Pre- dicting point tracks from internet videos enables diverse zero-shot robot manipulation, 2024b.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL httts::/tpppettteeeeeeetpt video-gener a t on mwoemoolllll..oooooomooooo s- -s- d- www.slmmttoooo...o
com/ rewearcheeer/eeeeee
Sylvain Calinon. A tutorial on task paraametteeriz movement learning and retrieval. Intelligent service robotics, 9:1-29, 2016.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. arXiv preprint 2402 2024.
Sungjoon Choi, Matthew KXJ Pan, and Joohyung Kim. Nonparametrie motion retargeting for hu- manoid robots shared latent space. In Robotics: science and systems, 2020.

---

Published as a conference paper at ICLR 2026
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. In Conference Robot Learning. pp. 2071-2084. PMLR, 2021.
Carl Doersch, Yi Yang, Dilara Gokay, Pauline Luc, Skanda Koppula, Ankush Gupta, Joseph Hey- ward, Ross Goroshin, João Carreira, and Andrew Zisserman. Bootstap: Bootstrapped training for tracking-any ponnt arXiv preprint arXiv:2402.008820 2024.
Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, al. Video language planning. arXiv preprini arXiv:2310. 10625, 2023.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofrr Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Hao-Shu Fang, Chenxi Wang, Hongjie Fang. Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spatial and tem- poral domains. IEEE Transactions on Robotics, 2023.
Peter Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint 100.v:1806.08766, 2018.
Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503. 189388 2025.
Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot. Navigating to objects in the real world. Science Robotics, 2023.
Michael Gleicher. Retargetting motion new characters. Proceedings of the 25th annual con- ference on Computer graphics and interactive techniques, pp. 33-42, 1998.
Xuyang Guo. Jiayan Huo, Zhenmei Shi, Zhao Song, Jiahao Zhang, and Jiale Zhao. T2vphysbench: A first-principles benchmark for physical consistency in text-to-video generation. arXiv preprint arXivv25005.00337, 2025.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Kai Hu, Christian Ott, and Dongheui Lec. Online human walking imitation in task and joint space based on quadratic programming. In 2014 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 3458 3464. IEEE, 2014.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973. 2023.
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024a.
Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma. Nattapol Chan- paisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench++: Comprehensive and versatile benchmark suite for video generative models. arXiv preprint arXiv:2411.13503. 2024b.
Nikita Karaev, Iurii Makarov. Jianyuan Wang, Natalia Neverova. Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.1183) 2024.
Simar Kareer, Dhruv Patel, Ryan Punamiya. Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoff- man, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. arXiv preprint arXiv:2410.24221 2024.
12

---

Published as a conference paper at ICLR 2026
Bingxin Ke, Dominik Narnhofer, Shengyu Huang, Lei Ke, Torben Peters, Katerina Fragkiadaki, Anton Obukhov, and Konrad Schindler. Video depth without video models, 2024. URL https: / //axxiv.rr//aa.... http:///abb22111.99..
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 12 2023.
Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg, and Angjoo Kanazawa. Robot see robot do: Imitating articulated object manipulation with monocular 4d reconstruction. arXiv preprint 24X1vv2409.18121 2024.
Chung Min Kim, Mingxuan Wu, Justin Kerr, Ken Goldberg, Matthew Tancik, and Angjoo Kanazawa. Garfield: Group anything with radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10.21.5 21530-21539, 2024.
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua Tenenbaum. Learning to act from actionless videos through dense correspondences. arXiv preprint arXiv:2310.0833 2023.
Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Trem- blay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivic. Megapose: 6d pose esti- mation of novel objects via render & compare. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
Arjun S Lakshmipathy, Jessica K Hodgins, and Nancy S Pollard. Kinematic motion retargeting for contact-rich anthropomorphic manipulations. arXiv preprint arXiv:2402.04820 2024.
Jinhan Li, Yifeng Zhu, Yuqi Xie, Zhenyu Jiang, Mingyo Seo, Georgios Pavlakos, and Yuke Zhu. Okami: Teaching humanoid robots manipulation skills through single video imitation. In 8th Annual Conference on Robot Learning, 2024.
Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2023.
Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl Vondrick. Dreamitate: Real-world visuomotor policy learning via video genera- tion. arXiv preprini arXiv:2406. 16862, 2024.
Fangfu Liu, Wenqiang Sun, Hanyang Wang. Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. Reconx: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint urXiv:2408. 16767, 2024.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren. Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint (xXxi:2333.054000 2023.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1118-11 125. IEEE, 2018
Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models learn physical principles from watching videos? arXiv preprint arXiv:2501.09038, 2025.
Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri. Abhishek Gupta, Ab- hishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x- embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.0886 2023.
Shivansh Patel, Xinchen Yin, Wenlong Huang, Shubham Garg, Hooshang Nayyeri. Li Fei-Fei, Svet- lana Lazebnik, and Yunzhu Li. A real-to-sim-to-- approach to robotic manipulation with vlm- generated iterative keypoint rewards. arXiv preprint arXiv:2502. 08643, 2025.
13

---

Published as a conference paper at ICLR 2026
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference Computer Vision, pp. 570-587. Springer, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
https://arxxt www.://////aaooooooooooooooaaa 00020
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao-Yual Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam Segment anything in images and videos, 2024. URL bttps:
org/ 205//bb//00 2ttp://48..00000 00714
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv:19/2.04443; 2019.
Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, and Abhinav Gupta. Hrp: Human affordances for robotic pre-training. arXiv preprint vrXiv22407.18111 2024.
Yihong Sun, Hao Zhou, Liangzhe Yuan, Jennifer J Sun, Yandong Li, Xuhui Jia, Hartwig Adam, Bharath Hariharan, Long Zhao, and Ting Liu. Video creation by demonstration. arXiv preprint arXiv:2412.09551, 2024.
Eugene Valassakis, Georgios Papagiannis, Norman Di Palo, and Edward Johns. Demonstrate once, imitate immediately (dome): Learning visual servoing for one-shot imitation learning. In 2022 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8614-8621. IEEE, 2022.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal under- standing and generation. In The Twelfth International Conference on Learning Representations, 2023.
Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Muller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. CVPR, 2023a.
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. arXiv preprint arXiv:2312.083344, 2023b.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8121-8130, 2022.
Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint wxXi:22407.15208, 2024.
Xindi Yang, Baolu Li, Yiming Zhang, Zhenfei Yin, Lei Bai, Liqian Ma, Zhiyong Wang. Jianfei Cai, Tien-Tsin Wong, Huchuan Lu, al. Vlipp: Towards physically plausible video generation with vision and language informed physical prior. arXiv e-prints, pp. MeXiv-2503. 2025.
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu- Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410. 11758, 2024.
Qihang Zhang, Shuangfei Zhai, Miguel Angel Bautista, Kevin Miao, Alexander Toshev, Joshua Susskind, and Jiatao Gu. World-consistent video diffusion with explicit 3d modeling. arXiv preprint arXiv:24/2.01822 2024.
Zhengyou Zhang. A flexible new technique for camera calibration. JEEE Transactions on pattern analysis and machine intelligence, 22(11) 13300-13333 2000.
14

---

Published as a conference paper at ICLR 2026
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-graine grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
Jiaming Zhou, Teli Ma, Kun-Yu Lin, Zifan Wang, Ronghe Qiu, and Junwei Liang, Mitigating the human-robot domain discrepancy in visual pre-training for robotic manipulation. arXiv preprint 24t.:2240..122355. 2024.
Junzhe Zhu, Yuanchen Ju, Junyi Zhang, Muhan Wang, Zhecheng Yuan, Kaizhe Hu, and Huazhe Xu. Densematcher: Learning semantic correspondence for category-level manipulation from single demo. arXiv preprint arXiv:2412.05568, 2024.
15

---

Published as a conference paper at ICLR 2026
Appendix
We structure the supplement into the following subsections:
A Details best practices for video generation.
Overview of prompt and examples of video summaries with GPT responses used for video filtering.
Results and discussion on our method's s mesh-free object tracking version.
D
Details reducing noise in 6D) pose rollouts for stable and realistic motion.
Adaptation and implementation details of baseline methods.
Comprehensive example of Rekep Predictions and Execution.
Discussion of limitations of Tracking using point tracks.
Elaboration on our method's 's robustness.
Thorough analysis of eтrors caused by depth estimation.
Discussion regarding the choice between the use MegaPose and FoundationPose, focus-
ing trajectory stability.
Additional analysis of generated videos and human demos using VBench++ metrics.
We are attaching the source code in the supplementary materials for reproducibility.
A BEST PRACTICES FOR VIDEO GENERATION
We found that the following practices lead to reliable video generation: (1) having clean back- ground without visual distractions, (2) minimizing the number of distractor objects in the scene, (3) ensuring objects are reasonably large and viewed from natural, human-like perspective, (4) ensur- ing there is one clearly identifiable task that can be performed, (5) using simple and concise text prompts, and (6) setting the relevance factor to 0.7 with the negative prompt "fast motion" led to the most reliable video generations.
B PROMPTING FOR VIDEO FILTERING AND FILTERING STATISTICS
The prompt for GPT ol-based filtering is shown in Figure 16 We provide GPT with the prompt, a video summary created by vertically concatenating evenly sampled frames from the video and the language command (e.g., "pour water"). GPT ol then responds with "Yes" or "No" to indicate whether the task is successfully performed.
C MESH-FREE OBJECT TRACKING
We experiment with a mesh-free object tracking version of our method. Specifically, we use BundleSDF (Wen al. 2023a). which jointly performs 6-DoF object tracking and reconstruc- tion from RGBD observations. For the pouring task, we evaluate our method using trajectories obtained via BundleSDF over 10 trials and observe success rate of (90%), matching our default tracking setup. While the BundleSDF paper reports real-time capabilities, we found that its official implementation takes approximately 30 minutes to process each video in practice, which limits its applicability for real-time deployment. In contrast. default tracker operates in real-time, abling closed-loop execution and recovery from disturbances as discussed in Sec. [4.5] While the BundleSDF paper reports real-time capabilities, tice with the official implementation. We expect that future advances in model-free tracking will address these efficiency bottlenecks, allowing for real-tima mesh-free deployment.
observed significantly higher runtimes in prac-
D SMOOTHING OBJECT TRAJECTORIES
To reduce noise and jitter in the estimated object poses, we apply moving average filter with fixed sliding window (centered on each point) to the position and orientation components. Translations are smoothed independently along each axis, while orientation is processed similarly after converting from quaternions to rotation vectors. This approach mitigates abrupt changes, resulting in a more stable and realistic object trajectory with smoother transitions.
16

---

Published as a conference paper at ICLR 2026
E REKEP PREDICTIONS AND EXECUTIONS
A detailed example of ReKep's keypoint and VLM predictions for pouring task is shown in Fig. [7 The VLM first predicts grasping the watering can at keypoint 1. For the transport phase, instructs moving keypoint 8 above keypoint 15, while keeping its height above keypoint 7. For pouring action, keypoint 8 remains above 15 (to place the spout over the plant) and above keypoint 4 (to induce tilting). The resulting robot execution fails. We attribute most ReKep failures to inaccurate keypoint pre- dictions, shown in Fig. II In the lid im- age, no keypoint appears at the lid handle. In the placing task, keypoints cluster around pan corners. For the sweeping task, the keypoints are generally well-placed, and executions suc- ceeded. Suboptimal initial keypoints lead to in- accurate downstream VLM predictions.
A
F DESCRIPTION OF BASELINES
Examples ReKep's Keypoint Loca- tions. The keypoint placements are often suboptimal, except for sweeping task, where the keypoints are rea- sonable.
Figure 11:
Track2Act (Bharadhwaj et al. 2024b): We adapt Track2Act's procedure to our setup pre- serving its core idea of object-centric trajectory estimation from point tracks. Track2Act generates a future interaction plan by predicting 2D point trajectories (using DiT- diffusion model) between an initial image and goal image, then recovers a sequence of 3D object transforms via Perspective-and-Pee (PnP) (Zhang 2000).
To integrate this into pipeline, use their published checkpoint but modify the input wormulation-while the initial image remains identical to our real camera's view, the goal image taken from the last frame of generated video rather than being physically captured. We then PnP on the predicted point tracks along with the initial depth image to estimate the object's rigid motion across frames, thereby defining the end-effector trajectory. We use interpolation between consecutive poses because Track2Act generates only a sparse set of frames, and denser sampling is needed for smooth trajectory estimation and execution. However, we exclude Track2Act's ''s closed- loop residual policy correction, focusing solely open-loop 6D object-pose estimation and execu- tion. This adaptation allows us to directly evaluate how well vision-based, open-loop approach generalizes to our setting without additional corrections.
AVDC (Ko et al. driven video (using a trained text-conditioned video generation model) and using optical flow from GMFlow (Xu et al. 2022) estimate dense pixel correspondences. It then reconstructs 3D object motion using an optimization step that refines pose estimates based on the tracked flow and depth information. To improve robustness, AVDC also includes replanning mechanism that re-executes the pipeline when predicted motion stagnates.
2023): : The AVDC approach models action trajectories by synthesizing task-
Since the trained text-conditioned video generation model did not generalize well our setup, use the same generated video as in other experiments to ensure fair comparison. While we do not employ AVDC's replanning strategy, we predict object poses using similar optimization framework based on flow and depth information.
4D-DPM Kerr et al. 2024): 4D-DPM is designed to track 3D motion of articulated object parts from a single video. constructs a 3D Gaussian splatting (Kerbl et al. 2023) representation of the scene to capture object features, then applies GARField (Kim al. 2024) to cluster the Gaussians into discrete object components. In our adaptation, we modify this to operate on entire objects rather than individual parts. Specifically, set the clustering parameters to treat the object
17

---

Published as a conference paper at ICLR 2026
single entity, ensuring that motion estimation is performed at the object level rather than segmenting it into multiple parts. This allows us to track and execute trajectories for the whole object.
Gen2Act (Bharadhwaj et al. 2024a): Gen2Act introduces a video-conditioned policy learning framework that first generates a human video using video generation model from scene im- age and task description. It then extracts object tracks using BootsTAP (Doersch et al. 2024), and trains a policy using behavior cloning with an auxiliary track prediction loss and offline robot demonstrations. At inference, Gen2Act uses the generated video and the learned policy to predict robot actions.
Our approach presents a simplified adaptation of this framework that removes the need for behavior cloning, and offline demonstrations. Instead of using the extracted tracks as an auxiliary loss, we directly process them for pose estimation. To recover 3D object positions, we leverage depth image corresponding to the scene image, allowing us to obtain depth values for the extracted 2D tracks. We apply RANSAC filtering remove outlier track points and then use the Perspective- n-Point (PnP) (Zhang 2000) to estimate the object's 6DoF pose. This adaptation preserves the core idea of leveraging video and track-based signals while eliminating the need for supervised policy learning.
initial
G LIMITATION OF TRACKING WITH POINT TRACKS
All point tracks fail under extreme rotations, as initially visible points often become occluded. This is a fundamental limitation of any correspondence-basee tracking method relying solely on visible surface features. We show this failure in Fig. 12 As the object rotates, most initial points are lost, resulting in insufficient 2D-3D correspondences to solve a stable PnP problem. This degrades pose estimation quality, leading to large drift or abrupt jumps in estimated object motion. Such instability cascades into execution errors, often causing the robot to fail the task altogether. As a result, both variants of Gen2Act-despaa
stronger tracking backbones like CoTracker fail under large out-of-plane rotations. In contrast, RiG'sd's model-based 6D tracking handles these situations more robustly, as it uses full-object geometry and SE(3) filtering to maintain stable trajectories.
WethAat
Gen2Act
Figure 12: Gen2Act with BootsTAP, CoTracker, and RIGVid. Blue points denote the tracked points used for PnP; red points represent the reprojected 3D points. For good PnP solution, these should align, as seen in the first frame. For Gen2Act, the blue points drift significantly from the red ones in later frames, indicating failure in pose estimation due tracking loss, which leads to failed robot execution.
H ADDITIONAL ROBUSTNESS EXAMPLES
Examples of RIGVid's robustness are shown in Fig. 13 In the first row, the robot grasps the object, but due to a misaligned grasp, the object rotates unexpectedly. The robot compensates by rotating it back to the correct orientation and then resumes the planned trajectory, completing the task suc- cessfully. In the bottom row, a human perturbs the object during execution while it is held by the robot. RIGVid detects the resulting change in the relative transformation and automatically re-aligns the object before continuing. When the human intervenes a second time, RIGVid again corrects the deviation, resulting in successful task completion.
18

---

Published as a conference paper at ICLR 2026
Robot approaches gresp
Fault grasp rotates the object
Robot corrects the foulty grasp
Successful Execution
Human pushes object
Recovary
Human pushes again
Recovery & Successful Execution
Figure 13: Additional examples RIGVid's robustness. the top row, RIGVid recovers from a faulty initial grasp by reorienting the object before continuing exccution. In the bottom row, it corrects for external disturbances on the object when human pushes it mid-execctiin, realigning and successfully completing the task.
I ERRORS FROM DEPTH ESTIMATION
Poue Water
LA0
Plac Spatuls
Sweep Tradh
100
20
Genersted Videes
Videos with Predicted Real Videos with
Impact Depth Estimation Errors on RIGVid performance. Errors monocular depth estimation result in worse performance of generated and real videos. RIGVid achieves perfect success across all tasks with real videos and real depth.
Figure 14:
In Fig- 14, we isolate the impact depth estimation errors. Robot executions on real videos with real depth (captured using an RGBD camera) achieve a 100% success rate, whereas executions from real videos with generated depth result in an 85% average success. Similarly, executions from Kling V1.6-generated vidcos with generated depth also achieve 85% success, suggesting that the primary source error lies monocular depth estimation. Upon inspection, we observe two common undesirable behaviors in the predicted depth: inaceurate depth values and temporal flickering. An example of inaccurate depth is shown in Fig. 15a In the generated video, when the spatula is brought close the camera, the depth changes by only 6.8 cm, which is visibly inconsistent with the video and likely much smaller than the real-world change. Inaccuracies also occur in real videos, as shown the figure-tie the head of the spatula is estimated be far from the camera, despite appearing close, revealing another failure mode in monocular depth estimation. Flickering is shown in Fig. [5b] Although the position of the watering can relative to the camera remains nearly unchanged across three consecutive frames, the estimated depth varies significantly. The zoomed-in region on the right shows the can appearing much whiter than on the left, indicating a substantial change in predicted depth. The average depth of the can changes from 40.1 cm to 38.2 cm-a 1,9 cm difference over just 0.066 seconds which is physically implausible for the generated video. We find similar flickering behavior in real videos well, where the depth changes from 43.2 cm 40.9 cm in the given example-a a 2.3 cm difference. Since errors in the generated depth are the main source of failure, also tested removing it entirely by estimating object pose directly from the RGB frames of the
19

---

Published as a conference paper at ICLR 2026
(a) Generated Video
(a) Generated Video
Aug Dapth: 40.1 cm
Aug Depth: 38.227 em
Avg Depth: 58.6 cm
Avg Depth: 51.8 cm
(b) Real Vidco
(b) Real Video
Aug Depth: 43.275 cm
Avg Depth: 40.969
Avg Depth:
Avg Depth: cm
(b) Flickering in Depth Prediction. We show three consecutive frames of video and its correspond- ing predicted depth. The depth the watering can change noticeably across frames frames-appearing signifi- cantly whiter in the third frame despite minimal tual motion. We observe this hehavior in both gener- ated
(a) Errors in Monocular Depth Estimation. the generated video (top), the depth of the spatula changes only slightly despite large visual change. In the real video (hottom), the spatula's head is pre- dicted lie farther away, contradicting the visual ap- pearance.
videos.
Figure 15: Combined Figure: Comparing depth estimation emror (leff) and prediction flickering (right) in generated videos.
generated video using MegaPose. However, this approach leads to even unstable and noisy trajectories, as detailed in App.
J CHOICE BETWEEN MEGAPOSE AND FOUNDATIONPOSE
We compare trajectory stability from MegaPose (Labbé et al. 2022) and FoundationPose (Wen) al. 2023b) by computing the translational and rotational RMS jitter. For each method, apply
a Gaussian smoothing filter (a = 2 frames) to the raw SE(3) pose sequences, compute the residual between original and smoothed trajectories, and then calculate:
N
I N
jitter crans
12,12 jitter not Σ
0
N
where At, is the translational residual frame and 0, is the angular magnitude (in radians) of the relative rotation R Raw. converted to degrees. Metrics are averaged over ten pouring trajectories from generated videos.
MegaPose yields average translational RMS jitter of 0.0045m and rotational RMS jitter of 37.47% whereas FoundationPose achieves 0.0029m translational and 14.31 rotational jitter. This demon- strate that FoundationPose produces significantly smoother and more stable trajectories. Addition- ally, allows for real-time tracking during the execution, making RIGVid robust to external distur- bances.
20

---

Published as a conference paper at ICLR 2026
K COMPARING VIDEO GENERATIVE MODELS
To further assess video quality, we report VBench++ (Huang et al. 2024b) metrics in Table 2 and explain them below. The numbers in the table are scaled 100x for easier interpretation. We collect these metrics on 40 randomly selected and unfiltered videos per model, 10 for each of the four tasks. Kling v1.6 outperformed the other models on most metries but performed similarly worse in video-text consistency and dynamic degree. Human evaluations discussed in Sec. 4.2 suggest that the video-text consistency and I2V subject consistency are not reliable indicators whether generated video correctly follows a given command. Sora scored high on dynamic degree, likely due to tendency drastically alter the scene, resulting in exceptionally large motions. Generated videos from these models and their corresponding metrics are shown in Fig. 18 and further details on these metrics can be found the next section.
VBench++ Metric Definitions:
Subject Consistency, Subject consistency deseribes whether subjects' appearance remain consis- tent, which is computed by DINOv1 (Caron et al., (021) similarities across video frames.
Background Consistency. Background temporal consistency by CLIP (Radford al. 2021
similarities across frames.
Motion Smoothness. Evaluates smoothness of videos by utilizing video frame interpolation model AMT eLi al. 2023).
Dynamic Degree. Describes whether the video contains large motions binary metric.
Aesthetic Quality. Human perceived artistic and beauty value such as photo-realism, layout and color harmony.
Imaging Quality, Assesses the presence of distortion in a video, such as noisiness, blurriness, and over-exposure.
Video-Text Consistency. Text-to-video alignment score calculated by VICLIP Wang et al. 2023
I2V Subject Consistency. Similarity between subjects in input image and each video frame, well as similarity between consecutive frames. Features are extracted from DINOv1 (Caron et al. 1
Generation Models
Kling V1.6
Kling V1.5
Sora
Demos
Subjact Consistency
96.34
91.66
83.09
94.91
Buckground Comistency
96.64
93.97
89.34
95.00
Dynamic Degres
52.5
57.5
70.0
80.0
Acsthetic Quality
51.75
49.77
46.22
49,30
Imaging Quality
72,80
71.48
68.68
72.52
Consistency
12V Subject Consistency
97.88
93.96
89.09
97.89
Table 2: Video generation quality metrics for real human demonstration videos and different models. Higher values indicate better quality. Kling performs comparably to or surpasses other models on most metrics.
21

---

Published as a conference paper at ICLR 2026
Prompt
#* Instructions
You need to deternine whether specific action occurs the
given image.
#*# Inage Format
The image has rows.
The action progresses from top to botton.
#ø# Task
Exanine the image and the provided query.
Decide 1f the action described in the query clearly
depicted.
A hand should be performing the action. Without it, the action
actions are likely imp implausible executed by an actor.
The task has be done with the specified object in the first frane. The video cannot bring in a new object such that there are two of the same objects.
Output
Your response must be in the following format hand. Explain new object has been brought that is different from objects the first frane
Explain
Explain your reasoning
the given image.
irin_done: Yes or No
*e Query
Query Task: Pour water fram the watering can on the plant Query
GPT Response
holding tilting the watering visibly flowing onto the plant in the final frame. action_done: Yes
Prompt
eè Instructions
previous prompt. Instructions onitted simplicity.
se Query
Query Task: Place the spatula the Query Image:
CPT of Response
Hand present? Yes, hand is visible holding spatula.
2) New object introduced? No, there is na new object introduced; it is still the original spatula and the original pan. 3) Reasoning:
Although we see hand holding the spatula, there is no point at which the spatula is placed the pan. action_done donee No
the pan. The spatula instead ends resting the table
Figure 16: Examples prompting GPT ol to filter generated videos. We sample frames from the generated video and prompt GPT ol to assess whether the specified task is performed successfully in the video. The top example passes the filtering, while the bottom does not.
22

---

Published as a conference paper at ICLR 2026
IB
def stagel_subooll_ Subgoal__ooo Loonstrainttieed_eeiee keypoints);
"""Align end-effector watering (keypoint "m
handle_poine keypoints
cost np.
handte_point)
return
def stage2_pet pttp:ppaaaaahaaaiiieiaaaaa conntraintieed__efeettte keypoints):
"""The robot nust still be orasping the watering can handle (keypoint 1), "" return
def teathhccostranntleen_eerre
keypoints):
""""The watering can must stay upright to avoid spilling (spout above base). spout keypoints(a]
keypoints
Ensure spout is not lower than base cost nax(θ, spout[2] base]2]]
def
conttrainttedd_efffcttr,, effector, keypoints):
"h"The watering can spout (keypaint 8) needs to be 10cm above the plant (keypoint 15), -... spout keypoints
plant keypoints[15]
offsetted_pon feited__ointt__ plant np.array( 0.18)) 18cn above the plant np. Linalg. norm/spuut offsetted_pointt
return
stage3_path, Batpee__pah__omeeee constraint effeeeeeerrrrr keypoints):
The robat must still be grasping the watering can handle (keypoint 1). return get_graappngg_oo ctapgn_ooosttteoeeeeooo keyy.byt. idx(1)
Saagee_path_come (en effcctor, keypoints):
"h watering can spout (keypoint 8) is directly above the plant (keypoint 15) in x-y plane. ua,
spout keypoints
plant keypoints[15]
cost np. Baimalg.nommspputtt. return cost
plant [:2]]
Laagee__uuggoa_cocr wus.spoaa__nsttrrantleeeee (edd_effector,, keypoints):
"*"The watering can spout (keypoint needs to 10cn above the center of the plant (keypaint 15), "mm spout keypoints[8) keypoints[35]
offsetted_poinn plant np.array(le, 8, 0.1)) 10cm above the plant cost np. tinalgg.orm/spourt offsetted_point)
return cost
wwt_oostraaneeeedeeeiiooooo effector, keypoints): """The watering can spout (keypoint 8) nust be tilted below the base (keypoint 4) to pour, spout keypoints|8]
keypoints(4]
Ensure spout lower than base cost nax(e, spout(2] base(2)) return cost
Figure 17: ReKep's output for the pouring task the resulting robot execution (top-right). The VLM predicts to grasp at keypoint 1, move keypoint 8 above 15 and during transport, and above 15 and 4 for pouring leading to failed execution.
23

---

Published as a conference paper at ICLR 2026
SORA
Const: 0.267
12V Subj. Const: 0.887
Subj. Const: 0.808
Const: 0.221 Subj. Const:
Subj. Const: 0,746
12V Subj. Const: 0.930
Subj. Const: 0.915
12V Subj, Const: 0.977
Subj. Const: 0.839
Kling Al v1.5
VT Const: 0.244
12V Const: 0.989
Subj. Const 0.936
VT Const: 0.195
12V Subj. Const: 0.978 Subj.
VT Const: 0.231 Subj. Const 0.989
Subj. Const 0.982
12V Subj, Const 0.865
Subj. Const : 0.965
Kling Al v1.6
VT Const: 0.217
12V Subj. Const: 0. 0.995
Const: 0.975
VT Const: 0.208 Subj. Conat: 0.964
Subj. Const: 0.969
Conet: 0.245
12V Subj. Const: 0.9965 Subj. Const: 0.965
VT Const: 0.188
12V Subj, Const: 0,955 Subj. 0.951
Figure 18:
Qualitative Comparison Different Video Generative Models. Videos from three video
generation models are shown using evenly sampled frames, along with VBench++ (Huang et al. (2024)) met- rics: vidco-text consistency, image-to-video subject consistency, and subject consistency. Kling v1.6 scores highest on these metries, followed by Kling v1.5 and then Sora.
24

---

Published as a conference paper at ICLR 2026
SORA
Kine V1.5
King
SORA
Kiing V1.5
Kling V/1. 16
SORA
Kling V1
Kling V1
SORA
Klnng V1.5
King
Figure 19: Qualitative comparison of video generation. Sora-generated videos often alter the scene layout and objects. Kling V1.5 produces more plausible results but includes physically implausible elements. Kling V1.6 better preserves scene fidelity and closely follows the human command.
25


Reviews:
[{"review_id": "xtCXHaefkv", "reviewer": "Reviewer_Tugp", "summary": "This paper presents RIGVid, a demonstration-free robotic manipulation framework that enables a robot to execute tasks purely by imitating AI-generated videos. Given a textual command and a single RGB-D observation, the system generates a video via a diffusion model, filters it with a VLM (GPT-4o) for semantic correctness, estimates per-frame depth, extracts a 6D object pose trajectory using FoundationPose, and retargets this trajectory to the robot for execution. The method is evaluated on four real-world manipulation tasks (pouring, lifting, placing, sweeping) and compared against several strong baselines including ReKep, Track2Act, Gen2Act, and optical-flow or feature-field-based trackers. Experiments demonstrate that filtered generated videos can yield comparable success rates to real human videos, supporting the claim that synthetic visual demonstrations can replace physical ones under certain conditions.", "questions": "The current pipeline extracts trajectories based on human-hand motion in the generated videos.\nHave the authors considered directly generating robot gripper poses from Kling or similar models, which might align better with robotic control?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a demonstration-free robotic manipulation framework that relies solely on generated videos. While the approach is currently constrained by the capability of existing video generation models, it points toward a compelling direction: as world-model generators such as SORA and Veo continue to evolve, this paradigm could become a practical and scalable way to get rid of robot teleoperation or imitation data.\n\nThe system is solidly engineered. It combines existing modules for depth estimation and 6D pose tracking in a sensible way to get a high-level “pose flow” that drives the robot’s actions. While I don’t think pose is the final or ideal representation for manipulation (as I’ll mention later), this design makes sense at the current era of the  co-evolving embodied AI and generative world models. At least for the four manipulation tasks shown in the paper, the whole pipeline works coherently and delivers consistent results.\n\nThe experiments cover a diverse set of real-world manipulation tasks and include detailed ablations on video quality, filtering accuracy, and trajectory extraction methods. The results provide convincing quantitative and qualitative evidence for the framework’s effectiveness.\n\nThe paper is clearly written and easy to follow. The figures and descriptions effectively illustrate the overall pipeline", "weaknesses": "The use of FoundationPose requires access to the CAD or mesh models of target objects, or at least works significantly better when such models are provided. This requirement limits the applicability of the system in unstructured or open-world environments where object models are unavailable. In addition, while modern video generators such as Kling can potentially produce videos containing deformable objects, FoundationPose cannot reliably track non-rigid shapes, which further constrains the border usage of the proposed pipeline.\n\nLack of details on handling pre- and post-grasp phases. Since the method uses the object trajectory directly as the source for retargeting, the robot gripper is assumed to remain fixed at the grasp position. It is unclear how the system handles the pre-grasp phase (approaching and aligning for grasping) and the post-manipulation phase (safely releasing or retracting the gripper). These should ideally be integrated into the overall policy for realistic and safe execution.\n\n\nThe paper does not evaluate any explicit recovery behavior after execution failures. It would be helpful to know how long the recovery takes, and whether the system (or the video generator such as Kling) can re-generate corrective videos to handle failed trials or unexpected conditions.\n\n\nCurrent video generation models are limited by the duration of the generated content. The paper does not analyze how such limitations affect tasks with longer execution horizons, where the manipulation may take more time than the video generator can represent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqpkVTcA4y", "reviewer": "Reviewer_uft8", "summary": "This paper pilotly explores using purely AI-generated videos for robots to perform tasks. The pipeline is very simple but effective: 1. Generate videos according to task instructions; 2. Lift the videos into 3/4D; 3. Performation robot pose estimation on the generated point cloud sequences. Extensive experiments are conducted.", "questions": "Although the proposed method itself is not very impressive, the problem that the paper aims to explore is meaningful. The experiments are very comprehensive, making this paper a good *study* research, instead of a *method* paper.\n\nTherefore, as for the current submission, not only the title but also the way to present the paper should be adjusted to a \"study\"-style paper, as there is no very novel framework is proposed, and no finetuning is provided, but only different video-gen models are tested. \n\nMoreover, some more experiments mentioned in the weaknesses are expected to be provided to strengthen the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is insightful. In fact, the reviewer personally also believes in the proposed solution path.\n2. The experiments are superior, with extensive ablation studies and real-world evaluations.", "weaknesses": "1. Ablations on 3D reconstruction are missing. How about using other SOTA 4D reconstruction methods, instead of just monocular depth estimation? Will it improve the performance?  \n2. Similarly, the tracking method also needs to be ablated. Especially, what if we just use ICP on the generated sequences? This may help to better evaluate the quality of the generation. \n3. If some robot data is used to finetune the video-gen (instead of directly testing), will it improve the performance? \n4. Open question: If the quality of the previous video-gen is low, are there any possible designs to post-refine the result, maybe from another action policy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KQpdzsQsUX", "reviewer": "Reviewer_bPbv", "summary": "This paper proposes RIGVid, a system that enables robots to imitate AI-generated videos instead of real demonstrations. Using video generation, VLM-based filtering, and 6D pose tracking, the robot executes manipulation tasks directly from synthetic examples. Experiments show synthetic videos can match real demonstrations and outperform keypoint-based and optical-flow baselines. The work suggests high-quality generated video can serve as effective supervision for robotic manipulation, reducing the need for real data.", "questions": "How well does this method generalize? If applied to a scene that has not been tuned or calibrated, is it still likely to succeed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The method requires no robot-specific training or real demonstrations, showing strong potential for scalable imitation learning directly \nfrom generated videos.\n2. The experimental analysis is thorough and insightful, revealing key factors such as the critical role of depth estimation in successful video-based control.", "weaknesses": "1. The paper suffers from dense formatting and uneven space allocation: related work and baseline descriptions are disproportionately long, while the core experimental section is compressed and difficult to follow. This tight layout and aggressive space packing visibly violate ICLR formatting expectations and reduce readability. A clearer narrative structure and more balanced discussion would improve clarity and presentation quality.\n\n2. Although the method follows a different paradigm than Vision-Language-Action (VLA) models, a direct discussion or empirical comparison is still needed. This would contextualize the contribution and help readers understand how generated-video-based imitation currently stands relative to dominant VLA pipelines, clarifying strengths, limitations, and future directions for this alternative trajectory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pwD4bw7wUD", "reviewer": "Reviewer_Jtgy", "summary": "Summary:\n\nRIGVid enables a robot to execute real‑world manipulation by imitating AI‑generated videos—without physical demonstrations or robot‑specific training. Given a language command and a scene image, the system (1) generates a candidate video, (2) filters low‑quality generations with a vision‑language model (VLM), (3) extracts a 6‑DoF (6D) object‑pose trajectory, and (4) retargets that trajectory to different robot embodiments for closed‑loop execution.\n\nContributions:\n\nEnd‑to‑end from generated video: First framework to execute real‑world manipulation directly from generated videos, requiring no physical demos or robot‑specific training.\n\nGenerated videos as supervision: Empirically shows that high‑quality, VLM‑filtered generated videos can substitute for real demonstrations in visual imitation.\n\nSimple, effective recipe: Video generation → VLM filtering → 6‑DoF pose extraction → embodiment‑agnostic retargeting; this pipeline outperforms SOTA alternatives based on VLM abstractions, point/flow tracking, feature fields, or generated goal images.\n\nClosed‑loop, robot‑agnostic control: Robust to disturbances via real‑time re‑tracking and recovery.\n\nInsights:\n\nFiltering is effective. VLM‑based filtering correlates strongly with human judgments and, when it errs, tends to be conservative (false negatives), which is the safer failure mode.\n\nState extraction is key. The 6‑DoF pose trajectory extracted from generated videos serves as a compact, transferable dataset format and appears to be the minimal supervision needed for successful grasping.\n\nOverall assessment:\n\nThis is a strong paper with a few remaining questions. If the authors are willing to address them, I would be happy to raise my score accordingly.", "questions": "Question 1 :What do generated videos add to robotic manipulation?\n\nMy current understanding is that generated videos provide a tighter visual grounding between the object and the robot than trajectory-only supervision. If the robot is ultimately learning a pick-and-place trajectory, why not supervise directly on trajectories? My hypothesis is that VLM-based filtering removes irrelevant content and that generated videos interface more naturally with VLMs than trajectories do.\n\nRequest: Please show an experiment that visualizes which generations are filtered out and why. Even better, compare (a) VLM filtering on generated videos vs. (b) an LLM (text-only) filtering of candidate trajectories, holding downstream training constant. If (a) > (b), that would substantiate the claim—and I’m inclined to raise my score if I can see this result.\n\n\nQuestion 2:\nPlease include more qualitative failure cases of the full method (e.g., failure modes in re-tracking, depth errors on transparent/reflective objects, backtracking oscillations, or embodiment-transfer edge cases), along with brief diagnoses and suggested fixes.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Closed-loop robustness. Real-time re-tracking and an explicit backtracking rule (≥3 cm or ≥20° deviation) enable recovery from perturbations—practically valuable and often missing in imitation-from-video work.\n\nThorough comparative evaluation for trajectory extraction. Benchmarks span tracks (Track2Act), flow (AVDC), feature fields (4D-DPM), and generated-goal supervision (Gen2Act). The 6D-pose rollout consistently performs best, especially on harder cases (thin/occluded objects, depth discontinuities).\n\nUseful ablations on video quality and filtering. Better generators (e.g., Kling v1.6) and automatic VLM-based filtering materially improve success; filtered generations approach real-video performance—an actionable result for the community.\n\n Demonstrates portability (xArm7 → ALOHA, including a bimanual vignette) via an object-to-end-effector transform, suggesting the representation is robot-agnostic.", "weaknesses": "Scale and statistical power. Main quantitative results cover only four tasks and, for some comparisons, use 10 videos per task per source with human-judged success. That’s limited for strong claims like “on par with real demonstrations,” and no confidence intervals are reported.\n\nCompute and latency under-specified. The pipeline generates and filters videos (up to five attempts), runs depth estimation and 6D tracking, and then executes in closed loop. The paper acknowledges high cost but does not quantify wall-clock time, attempts-per-success, or throughput—metrics that matter for practical deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)