Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
Authors: Junbang Liang
Ruoshi Liu
Ege Ozguroglu
Sruthi Sudhakar
Achal Dave
Pavel Tokmakov
Shuran Song
Carl Vondrick
Venue: CoRL 2024 (2024)
Keywords: Imitation Learning
Visuomotor Policy
Video Generation
TL;DR: We introduce a visuomotor policy based on conditional video generation and 3D tracking which is much more generalizable in manipulation tasks than traditional behavior cloning methods.

Abstract: A key challenge in manipulation is learning a policy that can robustly generalize to diverse visual environments. A promising mechanism for learning robust policies is to leverage video generative models, which are pretrained on large-scale datasets of internet videos. In this paper, we propose a visuomotor policy learning framework that fine-tunes a video diffusion model on human demonstrations of a given task. At test time, we generate an example of an execution of the task conditioned on images of a novel scene, and use this synthesized execution directly to control the robot. Our key insight is that using common tools allows us to effortlessly bridge the embodiment gap between the human hand and the robot manipulator. We evaluate our approach on 4 tasks of increasing complexity and demonstrate that capitalizing on internet-scale generative models allows the learned policy to achieve a significantly higher degree of generalization than existing behavior cloning approaches.

Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
Junbang Liang"l
Ruoshi Liu Ege Ozgurogiu'
Sruthi Sudhakar'
Achal Dave
Pavel T
Shuran Song?
Carl Vondrick'
CColumbia University
Toyota Research Institute Stanford University
dreamitate columbia.
Generated Video
Desam
Image
Video Gonerator
Execcuio
VOLOHCRA Volky
Rabot Execution
Figure 1: Real-World Visuomotor Policy Learning via Video Generation. Dreamitate is a vi- suomotor policy learning framework that fine-tunes a video generative model synthesize videos (indicated by of humans using tools to complete a task. The tool's 's trajectory in the gener- ated video tracked, and the robot executes this trajectory to accomplish the task in the real-world.
Abstract: key challenge in manipulation learning a policy that can robustly generalize diverse visual environments. promising approach for learning ro- bust policies is to leverage video generative models, which are pretrained on large- scale datasets of internet videos. In this paper, propose visuomotor policy learning framework that fine-tunes a video diffusion model human demonstra- tions of given task. At test time, we generate an example of an execution of the task conditioned on images of a novel scene, and use this synthesized execution directly to control the robot. Our key insight is that using common tools allows to effortlessly bridge the embodiment gap between the human hand and the robot manipulator. We evaluate our approach on four tasks of increasing com- plexity and demonstrate that harnessing internet-scale generative models allows the learned policy achieve significantly higher degree of generalization than existing behavior cloning approaches.
Keywords: Visuomotor Policy Learning, Imitation Learning, Video Generation
Introduction
Learning visuomotor policies that decide actions from perception has been longstanding challenge in robotics because they need to generalize to diverse physical situations [1]. Recently, behavior cloning from human demonstrations [2, 3] has become the method of choice for many manipulation tasks, where visuomotor policy learning is formulated as a regression problem to map visual obser- vations to actions supervised by human demonstrations. However, behavior cloning requires ground truth robot actions, which makes scaling to diverse situations challenging.
Equal contribution.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

We aim to learn generalizable visuomotor policies by leveraging video generation. Trained on large-scale datasets of internet videos, video generation provides a promising avenue for general- ization because extensive priors from human behavior can be transferred into robot behavior. Many recent methods have also explored how to capitalize on these priors, such as synthesizing videos of human behavior [4] or synthesizing videos of robot behavior [5]. However, while videos of people capture the diversity of behavior, generated human actions are difficult to transfer to robots due to the embodiment gap. While directly synthesizing videos of robots would be more realizable physically, the scale of available data is many orders of magnitude smaller than videos of humans, making the learned policy less robust in an in-the-wild environment.
How else can we leverage video generation for policy learning? this work, we introduce Dreami- tate,' shown in Figure generation. Our key insight is that the end-effector is the most important part of the embodiment for manipulation: its interaction with diverse objects is challenging to model explicitly and can benefit most from common sense knowledge from video generation. The rest of robot embodiment can be solved through inverse kinematics. Given a visual observation, our policy synthesizes a video of a person performing tasks with a tool. To translate into robot behaviors, we 3D track the tool in the synthesized video, and simply transfer the trajectory into explicit robot actions.
visuomotor policy that controls robot behavior via conditional video
Our formulation offers several major advantages compared to traditional visuomotor policies:
Generalizability. Our underlying video generation model is pretrained on a massive amount of videos available on the internet, including a wide variety of human manipulation in all kinds of environments. By formulating policy learning as video model finetuning, the prior knowledge learned from pre-taining can be preserved, allowing our model to learn generalizable manipula- tion skills with the same amount of demonstration data.
Scalability. Our finetuning videos are recordings of human demonstrations instead of teleoper- ation, making our dataset collection more scalable.
Interpretability. During inference time, our video model predicts future execution plans in the form of videos before the actual robot executions are performed. Compared to black-box end- to-end policy, our formulation offers an intermediate representation of policy that is interpretable to humans, a key feature for human-robot interaction applications.
We evaluated our approach across four real-world tasks, including bimanual manipulation, precise 3D manipulation, and long-horizon tasks, using a small number of expert human demonstrations. We found that the video model consistently outperformed the baseline behavior cloning model in generalizing to unseen scenarios. Additionally, we analyzed the model's performance when scaling down the training dataset and found maintained strong generalization performance even with fewer demonstrations. We will release the code and data for reproducing our results.
Related Work
Behavior Cloning. Behavior cloning (BC) has emerged as a leading approach in manipulation tasks, learning policies from demonstrations through supervised learning. Early BC efforts involved end-to-end models mapping states to actions, but struggled with multimodal behaviors and high- precision tasks [6. 7, 8]. Subsequently, Energy-Based Models (EBMs) were explored, predicting actions by minimizing energy in sequence optimization [9, 10, 11, 12]. Recently, conditional gen- erative models have shown promise in capturing multimodal demonstration behaviors, enhancing task success rates [2, 3, 13]. Different from previous models, we integrate video prediction and 3D tracking to predict action prediction.
Visual Pretraining for Policy Learning. Prior works have extensively explored various ways to pre-train the perception model in visuomotor policy for learning more robust visual representation.
dream i-tate (verb) ,drêm tāt). To behave in way that imitates dream.

---

Human Comorihation Scau
"hakiig
Acoot Excoution Sotup
Toan seaaooo
040
Utfusonn
runing
Figure 2: Method Overview. For each task, (Step 1) we capture stereo camera recordings of human demonstrations tool use from a top and side view. (Step 2) The video model is fine-tuned to recreate these demonstrations from an initial scene image. (Step 3) In new, unseen scenario, stereo image pair is provided to the video model to generate ( ())) the manipulation. The tool's trajectory in the generated video is tracked and executed by the robot complete the task.
One of the most popular pertaining objectives has been video prediction [14, 15, 16. 17, 18]. By learning to predict future based on the current observation, model can learn the dynamic and causality of the world crucial for physical interaction. Contrastive learning [15, 19, 20, 21J as well as masked autoencoding [22, 23, 24] are two very popular self-supervised learning objectives for learning visual representation for robotics. Besides, another line of work [25, 26, 27] studies learning generalizable reward function through visual pertaining for reinforcement learning.
More recently, in light of the fast progress of text-to- video generative models [28, 29, 30, 31, 32], the hope to leverage internet-scale video pertaining for robotics is rekindled [33]. A line of work uses video generative model as a world simula- tor [34, 5, 28], which can predict future video conditioned on an action. Another line of work uses video-language model for longer-horizon planning [35, 36, 37]. Different from prior works on this topic which typically form planning and decision making, Dreamitate uses video prediction model part of visuomotor policy and directly uses the predicted video for action prediction.
Video Models for Decision-Making
video prediction models world model simulator in order to per-
Approach
3.1 Overview
Given a visual observation of the
Human Behowlrr
Human lools
Hozot locis
Hobot Hahavior
scene dy, our goal to plan and ex- ecute robot actions at E SE(3) via video generation. We approach this problem through the framework, vi- sualized in Figure 2:
Rotation Fig.
Scooping
Seeeppig
a, T(()) where (4r) = fo (b()
where fe(-) is generative video model with learnable parameters 0. We train ff to generate videos of hu- man doing the task using a tool that can track in 3D. T tracks the tra- jectory of the tool in the generated video, which we can directly use to control the end-effector state to per- form manipulation tasks. This com- position of synthesize- teeettheen-eee track facilitates cheap data collection without teleoperation, and aligns with pretaining video data, which composed mostly videos showing human behavior.
Puen-Shape
Fl..71
Figure 3: Aligning Human and Robot Behavior via Trackable Tools. We tools with known CAD mod- els for human demonstrations to facilitate precise 3D track- ing across four real-world manipulation tasks. We design unique tools (e.g., spoon for scooping) to show the gener- alizability of our approach for varying tasks.
3.2 Video Generation
Starting with a pre-trained video generator, fine-tune it on small video demonstration dataset in order to combine the priors from internet videos with behaviors from demonstration videos. To

---

collect videos for fine-tuning, we build tabletop setup with two calibrated cameras positioned 45 degrees apart for better visibility. We capture several pair of stereo videos (i², (²) E V of humans performing the task using a 3D printed tool (see Figure 3 for details).
We use these demonstration videos fine-tune the video generative model. Since will imitate the trajectory of the synthesized tool in the physical world, we generate stereo videos to make action possible in 3D. Let (6), o??) be the predicted stereo video frames at time and of)) be the ground truth stereo frames from demonstration videos. We optimize the video prediction objective:
min E
Σ [( - 11)1 + 11(0,² ([(o)-=")l1)
vf)1lz
for {6+} fe(w))
We initialize 0 to the pre-trained weights learned from large-scale internet video datasets (Stable Video Diffusion [29]). We train a separate fe for each task (e.g. sweeping or scooping).
Following the implementation from [38], the encoder and decoder are frozen such that only the spatial/temporal attention layers are fine-tuned. The per-frame image embedding input to the model modified based on the viewing angle of the output frame to facilitate stereo video generation. This ensures that the first half of the frames generate the first view and the half generates the second view, using the initial image embedding from the respective view.
At test time, we will be given stereo image pair of the scene that we wish the robot to manipulate. We apply the tune video model f(o) and obtain generated stereo video frames é of the task being performed.
3.3 Track then Act
The synthesized video frames { serve as the intermediate representation of our policy to obtain an action trajectory, {az). that we then execute on the robot. Actions 0 E SF(33 are represented as the 6D pose of the tool relative to the camera. By using a known CAD model, we can efficiently and accurately track the tool in each frame of the generated videos and therefore obtain precise locations of the tool. Each action a₂ in the action trajectory corresponds to one frame from the synthesized video. For 3D consistency, we use the generated stereo pair, along with the calibrated camera parameters to resolve 6D pose of the tool. For tracking the tool from an RGB image, we use Megapose [39] and operate on 768 x 448 resolution video frames with camera intrinsic derived from the default values the Intel Realsense camera. Finally, during execution, the customized tool is mounted on the robot arm and actions ag. .... or are executed by the robot with the trajectory smoothed using impedance control. The entire data-collection, training, and fine-tuning pipeline tool's as manipulators and track known tools in
agent-independent. This is possible because generated videos, removing the need for learning any agent-specific action spaces.
4 Experiments
In this section, we evaluate the performance of our video model in real-world robot manipulation tasks. The evaluation, summarized in Table 1, encompasses four distinct tasks including object rotation, granular material scooping, table top sweeping and shape pushing (see Figure 3 for visu- alization of the used tools). The details of each task are further explained in the corresponding sec- tions below. For an assessment of our model's capabilities, we benchmark its performance against Diffusion Policy [40], Action Chunking Transformer (ACT) [3] and Vector-Quantizer Behavior Transformer (VQ-BeT) [13] as baselines.
4.1 Experimental Setup
To evaluate the model's 'ssse generalizability, the data was collected by a human demonstrator different locations for training and evaluation using the same camera setup, though the exact tabletop and lighting conditions varied. To further test the model's robustness in real-world scenarios, ensured

---

mmage
Figure 4: Rotation Qualitative Results. For this task, provide input the image the left for each row, with an example successful rotation overlaid. Our video generation-based approach succeeds, while Diffusion Policy fails to select stable grasping points the object, causing it to slip during manipulation. Generated videos are indicated by
Iopu mage
Our
D Rusion Policy
Figure 5: Scooping Qualitative Results. We provide the input image on the left (without the overlays provided for illustration) to our model and Diffusion Policy, and show the output trajectory on the right. Our approach succeeds most trials, while Diffusion Policy often distracted by other objects in the scene, placing material in the wrong container and failing to scoop accurately. Generated videos are indicated by
that the sets of objects used in training and evaluation non-overlapping. To initialize tool tracking with Megapose [39], background subtraction and hand removal based on skin color were employed to obtain the tool bounding box. In rare cases where this process failed, human corrections were applicd to the bounding boxes (both for our method and for the baseline).
We used the same training data train the baselines with stereo image input. The videos were preprocessed using Megapose for tool tracking, providing target trajectories for training. For Dif- fusion Policy we used a pretrained ResNet- variant [40] as the baseline, as the pretrained CLIP encoder [41] variant showed lower performance in our tasks. We trained the baselines to predict the entire action trajectory for the next 12 time steps without replan as an open-loop system, similar to the video model.
Training Objects
Demonstrations
Test Objects
Test Trials
Rotation Scooping
371
17 Bowls, 8 Particles
368
8 Bowls, 4 Particles
40
Sweeping
Particles
356
6 Particles
40
Push-Shape
26 Letters
727
8 Shapes
32
Table Tasks Summary. We detail the train and test setup for each task above. For each task, use distinct set of objects during testing than training.
4.2 Object Rotation Task
We design rotation task, shown in Figure 4, to test our policy's ability to coordinate end- effectors and choose appropriate grasping points to manipulate real-world objects. In each training demonstration, we randomly place an object on the table, and use gripper, shown in Figure (top), to grasp the object at appropriate points and rotate counterclockwise up to 45 degrees.
Task.
Setup. We collect training data with 31 objects, with 14 real-world objects (such as boxes hammer) and 17 custom colored shapes made out of foam. We use 10 unseen real-worke objects for
5

---

Yopr Image
Our Fleccutor)
DPhuoo Polay Esecutor)
x
Figure 6: Sweeping Qualitative Results. We provide the input image on the left (minus the white and green circles overlaid for illustration) to our model and Diffusion Policy, and show the output trajectories on the right. Our approach again achieves high success rate even on this challenging, multi-modal task, whereas Diffusion Policy generates trajectories that often collide with obstacles and fail to sweep the target. Generated videos are indicated by
Rotation
Scooping
Sweeping
Push-Shape
Model
Success Rate
Success Rate
Success Rate
mloU
Rot. Error
ACT [3]
5 5/40
10 40
3/40
0.527
44.1"
"Q-BeT [13]
4/40
11 /40
0/40
0.477
47.4"
Diffusion Policy [2]
22 / 40
22 /0
5/40
0.550
48.2"
Ours
37 / 40
34/40
37/40
0.731
8.0"
Table 2: Quantitative Results. We compare our method to ACT, VQ-BeT and Diffusion Policy on four tasks quantitatively. We report success rates (successful trials total trials) for rotation, scooping, and sweeping tasks. For Push-Shape, we report mean dntersection-over-mme average rotation error (Rot. Error). Our approach performs well across all tasks, whereas baselines shows worse performance overall and degrades in the more challenging sweeping and Push-Shape scenarios.
(mloU) and
evaluation. We mark a trajectory as successful if the robot makes and maintains contact with the object throughout the rotation, and rotates it at least 25 degrees counterclockwise.
Results. As shown in Table 2, our policy significantly outperforms baselines (92.5% vs. 55% at best). Fig. 4 illustrates cases where our policy succeeds while Diffusion Policy fails. We observe that baselines fail to move the end-effector into contact with the object. In more challenging cases, baselines are prone to selecting unstable grasping points causing the object to slip during manipulation. In contrast, our policy consistently makes contact with the object, with limited failures when selecting appropriate grasping points for particularly challenging shapes, such as a transparent bag with toys inside.
4.3 Granular Material Scooping Task
Task. This task, shown in Figure 5, requires scooping granular material (e.g., beans) from full container an empty one while avoiding distractor objects. This task requires the policy to perform precise manipulation of scooping tool, identify the full and the empty containers, along with their precise locations. in arbitrary positions, and ignore distractors.
Setup. We collect demonstrations with 17 bowls and colored beans with only distraction object in the scene at time. At test time, we use an unseen set of bowls, new colored particles, and distractor objects per trial sampled from fixed set of 15 everyday objects. We mark a trajectory as successful if the robot transfers any particles to the empty bowl.
Results. As shown in Table 2 (middle), our policy significantly outperforms baselines (85% vs. 55% at best). This is a particularly challenging task for our approach, due to the small target (the scooper) for object tracking and video generation. This demonstrates that stereo video generation can accurately determine the object's 's pose perform 3D manipulation. By comparison, baselines often distracted by objects in the scene, placing material in the wrong container, and failing more frequently in scooping due to misjudging the bowl's i height on the table, as illustrated in Fig. 5.

---

Iipud
Step
Step
Step
Shep
Exccutori
X
Exccuori
Figure 7: Push-Shape Qualitative Results. We provide the image on the left, including the gray overlay indicating the target object position and orientation, show output trajectories on the right. Our method produces appropriate pushing actions to adjust the object's position. In contrast, Diffusion Policy fails to effectively rotate the object match the mask. Generated videos (top row, bottom right) are indicated by
model and Diffusion Policy, and
4.4 Table Top Sweeping Task
Task. The sweeping task, shown in Figure 6, requires policies to use brush to sweep randomly placed particles to target location marked by randomly placed star, while avoiding obstacles. This task is designed to test policy's 's ability to handle multi-modal distributions in the training data.
Setup. Training data includes colored particles and 25 distraction objects, with 5 to 6 distractions time. At test time, we use 6 new particles and 15 unseen distractions, with 2 to 4 beans and variations with multiple
distractions randomly placed on the table. The training data contains ways to achieve the goal, such choosing which particle to sweep in the scene. Due the small size of the particles, we remove the final pooling layer from Diffusion Policy ResNet-18 encoder to provide higher spatial resolution and improve baseline performance. We mark a trajectory as successful any particle is transferred to be within 50 mm from the target.
Results. In Table (right) observe that the video model maintains strong performance in this task with 92.5% success rate. In contrast, baselines generate trajectories that often collide with obstacles and fail to sweep to the target (see Fig. 6), achieving only 12.5% success rate at best with Diffusion Policy. This experiment demonstrates that the capitalizing on internet-scale pre-trained video generation models allows to better handle multi-modal demonstrations and achieve a much larger degree generalization in this challenging scenario.
4.5 Push-Shape Task (Long Horizon)
Task. Push-Shape, shown in Figure 7, is a challenging version of! the long-horizon Push-T task [2]: we place foam shape on the table, and task the robot with pushing the object to a specified target goal mask (given as input to the policy) over consecutive steps. This is challenging task as requires adjusting the position and orientation of the shape, requiring predicting the shape's move- ment, which depends the contact between the object and the end-effector, well material properties of the table.
Setup. We train with a set of 26 foam objects each in the shape of letter from the alphabet, and test on 8 unseen foam shapes (including digits and polygons). As this is a challenging, multi-step task, we score the best of rollouts for each trial, and report the mean intersect- -vveruuooo (mIoU) with the target mask well the average rotation error from the target.
Results. As shown in Table 2, over the 32 trials, the video model achieved significantly higher mloU of 0.731 compared to 0.550 mloU (at best with Diffuusion Policy), and an average rotation error of 8.0 degrees compared to 44.1 degrees (at best with ACT). We find that baselines tend to push the object to the target, but fails to effectively rotate the object to match the mask. By contrast, the video model produces appropriate pushing actions adjust the object's position and orientation. thereby reducing the rotation error. Qualitative examples of the model rollouts illustrated in Fig. 7.
7

---

4.6 Performance Scaling Curve
We studied the impact of training set size on the gen- cralization ability of our policy and compared it with Diffusion Policy using the rotation task. Both mod- els performed well with the full dataset. When re- trained with two-thirds and one-third of the dataset and tested over 40 trials, the results depicted in Fig- ure indicate that while Diffusion Policy's 's perfor- mance declines significantly with reduced data, our model remains stable, maintaining high success rates even with only one-third of the data. This highlights our model's superior generalization, which can be attributed to the extensive pre-traine pre-training Internet- scale video generation models.
Ours Diffusion Policy
0.8
Success Rate
0.6
0.4
0.2
0
1/3
2/3
Full
Training Dataset Fraction
Figure Number of Demonstrations. Our approach maintains strong generaliza- tion performance with less training data com- pared to Diffusion Policy.
4.7 Ablation Study
We investigated the impact of pretraining and stereo camera setups on model generalization using the ro- tation task. The results are shown Table 3. With- pretraining, the model produces blurry videos with significantly distorted tools, but the tracking model still tracks the tool's pose with stereo pro- jection correcting depth estimation. We also ex- plored the effects of the stereo camera setup. In the stereo input monocular output (SIMO) model, we track only one view of the stereo output in the full model. Additionally, we trained monocular input and monocular output (MIMO) model. The SIMO model showed performance decline due to greater depth estimation errors without stereo projec- tion. The MIMO model exhibited an even further reduction in performance, indicating the impor- tance of stereo input conditioning.
Rotation
Model
Success Rate
Full Model
37
Full Model w/o Pretraining Stereo In Monocular Out
18 / 40
30 / 40
Monocular In Monocular Out 14 40
Table 3: Ablation Results. We provide re-
sults the effect of pretraining and stereo camera setup on model performance.
5 Limitations
Our approach is limited by the need for visually trackable tools, and the model is task-specific, lacking conditional text embeddings to resolve ambiguities in input images. Future work could explore integrating text embeddings to train unified policy across tasks. The reliance on rigid tools and video generation restricts the model's use in tasks requiring fine-grained force control. Additionally, using tools with known CAD models and camera calibration reduces flexibility, while tracking small objects remains challenge for pose estimation models, affecting robot trajectory planning. Finally, the computational demands of video models make real-time control impractical, with current processing times of 33.5 seconds per video and 7.5 seconds for object pose tracking on an A 100 GPU.
Conclusion
In this study, we explore how video generative models can learn generalizable visuomotor policies. We propose to fine-tune a video diffusion model on human demonstrations to synthesize execution videos, which are then used control the robot. Our key insight that using common tools helps bridge the gap between human demonstrations and the robot manipulator. Experiments show that leveraging pretrained video diffusion models Intereet-scale data leads to greater generalization compared to tfte-of-theear behavior cloning methods.
8

---

Acknowledgments
We would like to thank Basile Van Hoorick and Kyle Sargent for providing the video model training code and for their valuable feedback. We also appreciate the insightful discussions with Paarth Shah and Dian Chen, and the reviewers for their helpful comments. This research was partially supported by the Toyota Research Institute and the NSF NRI Award #2132519.
References
[]] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, I. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mer- cat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q Vuong, H. R. Walke, B. Wulfe, Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le. Y. Li, K. Lin, R. Lin, Z. Ma, Maddukuri, Mirchandani, D. Morton, T. Nguyen, A. O'Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang. Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Ja- yaraman, J. J. Lim, J. Malik, R. Martín-Martin, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip. Y. Zhu, T. Kollar, S. Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset, 2024.
[2] C. Chi, Z. Xu, S. Feng. E. Cousineau, Y. Du, B. Burchfiel, Tedrake, and Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2024.
[3] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine tne-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. / 13705, 2023.
[4] H. Bharadhwaj, A. Gupta, V. Kumar, and Tulsiani. Towards generalizable zero-shot manip- ulation via translating human interaction plans. arXiv preprint arXiv:2312.00775, 2023.
[5] Y. Du, M. Yang. B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation, 2023.
[6] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems. 1. 1988.
[7] T. Zhang, 7. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality telcoperation. In 2018 IEEE international conference on robotics and automation (ICRA), pages 5628-5635. IEEE, 2018.
[8] Florence, L. Manuelli, and Tedrake. sulf-supervised correspondence in visuomotor policy learning, 2019.
[9] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy- learning. Predicting structured data, 1(0), 2006.
[10] Y. Du and I. Mordatch. Implicit generation and generalization in energy-based models, 2020.
[11] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023.
[12] Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and Tompson. Implicit behavioral cloning, 2021.
[13] S. Lee. Y. Wang, H. Etukuru, H.J. Kim, N. M. M. Shafiullah, and L. Pinto. Behavior generation 2024.
with latent actions. arXiv preprint

---

[14] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction, 2016.
[15] Sermanet, C. Lynch, Y. Chebotar, Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video, 2018.
[16] M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell, and S. Levine. Stochastic variational video prediction. arXiv preprint arXiv:1710.11252, 2017.
[17] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction, 2018.
[18] D. Suris, R. Liu, and C. Vondrick. Learning the predictability of the future. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12607-12617, June 2021.
[19] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, 2020.
[20] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation, 2022.
[21] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[22] Y. Seo, D. Hafner, H. Liu, F. Liu, S. James, K. Lee, and P. Abbeel. Masked world models for vi- sual control. In K. Liu, D. Kulic, J.d J. Ichnowski, editors, Proceedings The 6th Conference on Robot Learning. volume 205 of Proceedings of Machine Learning Research, pages 1332- 1344. PMLR, 14-18 Dec 2023. URL https: attps://proceedings www.mmlr.rreee.e presss/205/ss002333 html.
[23] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In K. Liu, D. Kulic, and J. Ichnowski, editors, Pro- ceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 416-426, PMLR, 14-18 Dec 2023. URL mlr. prress//205//raddovvviic.o
html.
[24] Radosavovic, B. Shi, L. Fu, K. Goldberg, Darrell, and J. Malik. Robot learning with sensorimotor pre-training. 2023.
[25] Y. J. Ma, W. Liang, V. Som, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-iman image representations and rewards for robotic control, 2023.
[26] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-wild" human videos, 2021.
[27] A. Escontrela, A. Adeniji, W. Yan, A. Jain, X. B. Peng, K. Goldberg, Y. Lee, D. Hafner, and P. Abbeel. Video prediction models rewards for reinforcement learning, 2023.
[28] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Tay- lor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh. Video genera- tion models as world simulators. 2024. URL https: https://openai. com/research/ dideo-generation -egeeerrtiioommmdllls-------e--e wawtssssswwooldsmmmmmml..
[29] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. En- glish, V. Voleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
10

---

[30] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023.
[31] S. Zhang, Wang. Y. Zhang, K. Zhao, H. Yuan, Z. Qin, X. Wang. D. Zhao, and Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded difffusion models, 2023.
[32] Ho, W. Chan, C. Saharia, J. Whang, Gao, A. Gritsenko, D. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: High definition video generation with diffusion models, 2022.
[33] S. Yang, J. Walker, J. Parker-Holder Y. Du, J. Bruce, A. Barreto, P. Abbeel, and D. Schuur- mans. Video as the new language for real-world decision making, 2024.
[34] Z. Yang, Y. Chen, J. Wang. S. Manivasagam, W.-C Ma, A. J. Yang, and R. Urtasun. Unisim:
A neural closed-loop sensor simulator, 2023.
[35] Y. Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, Sermanet, T. Yu, P. Abbeel, B. Tenenbaum, L. Kaclbling, A. Zeng, and J. Tompson. Video language planning, 2023.
[36] A. Ajay, S. Han, Y. Du, S. Li, A. Gupta, T. Jaakkola, J. Tenenbaum, Kaelbling, A. Srivastava, and P. Agrawal. Compositional foundation models for hierarchical planning, 2023.
[37] K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero- shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10633 2023.
[38] B. Van Hoorick, R. Wu, E. Ozguroglu, K. Sargent, R. Liu, P. Tokmakov, A. Dave, C. Zheng, and C. Vondrick. Generative camera dolly: Extreme monocular dynamic novel view synthesis. arXiv preprint arXiv:2405. /4868, 2024.
[39] Y. Labbé, L. Manuelli, A. Mousavian, S. Tyree, S. Birchfield, J. Tremblay, J. Carpentier, M. Aubry, D. Fox, and J. Sivic. Megapose: 6d pose estimation of novel objects via render & compare. arXiv preprint arXiv:2212.06811 2022.
Visuomotor policy learning via action diffusion. arXiv preprint 003:4:2303.04137, 2023. [40] C. Chi, Feng, Du, Z. Xu, E. Cousineau, Burchfiel, and S. Song. Diffusion policy:
[41] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS), 2024.
[42] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[43] R. Cadene, S. Alibert, A. Soare, Q Gallouedec, A. Zouitine, and T. Wolf. Lerobot: State- of-the-art machine learning for real-world robotics in pytorch. https://github.ce com/ huggingface/lerobot, 2024.

---

A Appendix
A.1 Video Model Implementation
We adapted the pretrained Stable Video Diffusion model [29], which generates 25-frame videos at time. our adaptation, the first 13 frames correspond the stereo view 1, and the last 12 frames correspond to the stereo view 2, captured from the two cameras. To condition the video model to generate stereo video, modified the per-frame image embedding based the viewing angle of each output frame. Since each frame of the stereo videos should be paired but the video model generates an odd number of frames, the first frame of the video model output is always the same as the input and discarded at test time. The model training hyperparameters are given Table 4. We conducted the finetuning experiments using four A A100 GPUs over span 40 hours. During inference, use 30 denoising steps with constant classifier-free guidance of 1.0. Additional qualitative results for the four tasks shown in Fig 9, 10, 11 and 12.
Hyperparameters
Resolution
Lr
Batch
Steps
Clip Duration
Fps
Motion Score
Rotation (Full DS)
768
le-5
4
16384
2.0
6
200
Rotation (2/3 DS)
768
le-5
16384
2.0
200
Rotation (1/3 DS)
768 x48
le-5
15360
2.0
200
Rotation (w/o Pretrain)
768x448
le-5
16384
2.0
6
200
Rotation (SIMO)
768x448
le-5
4
16384
2.0
6
200
Rotation (MIMO)
768x448
le-5
16384
2.0
6
200
Scooping
768x448
le-5
16384
3.0
5
200
Sweeping
768x448
le-5
4
16384
3.0
5
200
Push-Shape 768x448
lc-5
4
17408
2.0
6
200
4: Hyperparameters for Video Model Training. Resolution: image and video resolution, Lr.
learning rate, Batch: batch size, Steps: training steps for the evaluation checkpoint, Clip Duration: single demonstration video length in seconds, Fps: the video sub-sampling frame rate and model fps parameter, Motion Score: model motion score parameter.
A.2 Experimental Setup
The stereo camera setup consists of two Intel RealSense D435i cameras spaced approximately 660 mm apart at a 45 angle. The distance between the cameras and the table is about 760 mm. The real-world data collection and the robot experiment setups are shown in Fig. 17 and 18. The training videos are recorded at resolution of x280x720 and are then cropped and resized to the appropriate resolution for model input. The table surface used for data collection and experiments is covered with a black cloth, which introduces variations in friction and increases uncertainty the Push- Shape experiments. For the rotation and scooping experiments, UFACTORY xArm robots are used, while UR5 robots are used for the sweeping and Push-Shape experiments. For calculating the mloU in the Push-Shape experiment, the view from the stereo camera is used. In each trial with multiple steps, the resulting image with the highest IoU with the target is used to calculate the rotation error. In sweeping and push-shape experiments, the robot end-effector height is limited to avoid robot collision with the table top.
A.3 Data Collection
For all tasks, the first frame of the human demonstration video is an image of the scene. The subsequent frames include the human demonstrator using the tool to perform the manipulation. In the Push-Shape demonstration, object pushed location in multiple steps. The final position of the object is used mask and blended with the entire video for the target position. The objects used in training and testing for different tasks shown in Fig. 13, 14, 15 and 16.
12

---

A.4 Object Tracking
In the videos, the tool is tracked using MegaPose [39]. Utilizing a stereo setup, the center of the tracked object from each camera are projected into 3D space as a straight line. The translation component of the object in 3D space determined by finding the midpoint between the projected lines from the cameras. The rotation component of the object is obtained by averaging the object rotations from the two views. This refined object pose from the stereo setup enhances the accuracy of the object's depth measurement from the cameras. In the scooping task, only the handle of the scooper is tracked to avoid inaccuracies due to occlusion by particles. In the sweeping and Push-Shape tasks, the tool without the handle is tracked, as the handle occluded by the human hand. To obtain the tool trajectories for training Diffusion Policy, the same stereo tracking is applied the demonstration videos.
A.5 Diffusion Policy Baseline
We CNN-based Diffusion Policy baseline, employing two pretrained ResNet-18 [42] image encoders to process the stereo images of the scene. The input images have resolution of x84x222, similar to the original implementation resolution. We found that higher resolution input images did not improve model performance. We trained Diffusion Policy for 200 epochs to predict actions for the next 12 time steps.
A.6 ACT Baseline
We use the ACT implementation from LeRobot [43] as our baseline, utilizing single pretrained ResNet- 18 [42] image encoder process stereo images of the scene, the implementation supports multi-image conditioning by default. The input images have resolution of 768x448. We trained ACT for approximately 90k steps to predict actions for the subsequent 12 time steps.
A.7 VQ-BeT Baseline
We use the VQ-BeT implementation from LeRobot [43] as our baseline, modifying it to incorporate two pretrained ResNet- 18 [42] image encoders process stereo images of the scene, since the orig- inal implementation does not support multi-image conditioning. The input images have a resolution of 768x448. Through experimentation, 5k steps, followed by training the entire model for around 85k steps, yielded the best results in predicting actions for the subsequent 12 time steps.
found that training the VQ-BeT VAE for approximately
13

---

Irpul Image
Ours (Tra,ectory)
Diftusion Poicy (Trajectory)
x
End
Figure 9: Additional Rotation Qualitative Results. The trajectories of the end-effectors are pro- jected onto the input image.
Irput Image
Ours (Trjectory)
Difusion Policy (Trejectory)
X
Start
End
Figure 10: Additional Scooping Qualitative Results. The trajectory of the end-effector is projected onto the input image.
14

---

Irput Image
Ours (Trojactory)
Pollcy (Trejectory)
Figure 11: Additional Sweeping Qualitative Results. The trajectory of the end-effector is pro- jected onto the input image.
15

---

Input image
Step
Step
Slep
Final Image
Ours
Diltusion Pollyy
Oure
Difusion Potcy
Ou3
Dituuion Pollyy
foyoy
Start
End
Figure 12: Additional Push-Shape Qualitative Results. The trajectory the end-effector is pro- jected onto the input image.
Training Objects
Testing Objects
TJTU
YJTU
XL>C
57
DY27T
Figure 13: Rotation Objects. The training set includes real-world objects and custom colored shapes made out of foam. The testing includes 10 challenging real-world objects.
16

---

Training Objects
Testing Objects
Figure 14: Scooping Objects. The training set includes 17 bowls, 8 colored beans and 4 real- world objects. The testing set includes bowls, colored particles and 15 real-world distraction objects. During data collection, both the real-world objects and inverted bowls are used as the single distraction object the scene.
Training Objects
Testing Objects
Figure 15: Sweeping Objects. The training set includes 25 distraction objects and 6 colored beans, with 5 6 distractions in the scene time during data collection. The testing includes 15 real- world distraction objects and 6 colored particles. The star is used as the target in the experiments.
Training Objects
Testing Objects
ABCDEFGH ILKLMNOPO RSTUVWXYZ
4
7
Figure 16: Push-Shape Objects. The training set includes all 26 capital letters of the alphabet, while the testing set consists of 8 shapes, including digits and polygons.
17

---

Stereo Camera 1
Stereo Camera 2
Human Tool
Figure 17: Real-world Data Collection Setup. The data collection setup has the same camera arrangement as the robot experiment setup.
Stereo Camera
Stereo Camera 2
Robot Tools
Figure 18: Real-world Robot Experiment Setup. The robot experiment setup includes the 3 robots to perform the experiments.
18


Reviews:
[{"review_id": "8ItazQWJD5", "reviewer": "Reviewer_RxCE", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors demonstrate how to use video prediction models as a way to train robot policies. First, an off-the-shelf video model is finetuned on a few human demonstrations of a tabletop task. Stereo cameras are used to collect demonstrations and subsequently, the generated pairs of videos are used to estimate the 3D position of a tool handled by the human. This 3D position subsequently serve as robot action which are executed. The method is tested on table top tasks like shape pushing, sweeping and scooping tasks.", "summary_of_recommendation": "I think the paper presents excellent ideas. However, I believe the experiments section needs improvement. Baselines and ablations are essential and are currently missing, which is why I am recommending a weak reject. With these improvements, I think this paper will be useful to many in the community.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How does changing the camera angle affect the video model? Is it still robust to such differences?\n2. What is the speed of policy execution? What is the inference speed of the video model? \n3. Can you please describe the error modes of the work?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HhS1paBXXn", "reviewer": "Reviewer_gdSH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents Dreamitate, a novel approach to visuomotor policy learning that leverages large-scale video generation models. The key idea is to fine-tune a video diffusion model on human demonstrations with a tool, track the tool's trajectory in the generated video, and translate the tool trajectory to robot actions. The authors demonstrate their approach on four manipulation tasks, showing improved generalization compared to a state-of-the-art baseline.", "summary_of_recommendation": "This paper presents an approach that shows promise in leveraging internet-scale data for robot learning. While I have some concerns about its scalability and real-world applicability, the results are impressive and the ideas are exciting, and I believe it could inspire important follow-up work. I think the robot learning community would benefit from engaging with these ideas, and I will consider revising my recommendation if my questions and concerns are addressed.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Why did you choose to track the tool instead of the end-effector? Why not, for example, use something like the UMI gripper (https://arxiv.org/abs/2402.10329) for data collection? \n\n2. Why do some tasks have significantly more demonstrations than others? Did some tasks require more demonstrations for the video generation fine-tuning to work?\n\n3. Are all policies single-task? If not, how is the task specified for the model? \n\n4. What are the success criteria for the shape pushing task? \n\n5. The results in Figure 8 show decreased performance with 2/3 of the data. What could be the reason for this decline? Was there significant variance in policy performance? \n\n6. Are actions generated by simply transforming the tool pose from the camera frame to the robot frame? Is there any additional control mechanism, such as PID controllers or filters? Is the policy closed loop?\n\n7. Your method segments out the background and hands to obtain object poses using Megapose. However, the diffusion baseline does not seem to have a corresponding mechanism to encourage generalization to various backgrounds. Could this difference have impacted the fairness of the comparison? \n\n8. What computational resources are needed for training and evaluation? How long does the video generation take?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J194x4JWPV", "reviewer": "Reviewer_w1Yn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a framework for visuomotor policy learning by fine-tuning a video generative model on human demonstrations. This approach leverages large-scale datasets of internet videos to generalize robot manipulation tasks across diverse environments. The framework synthesizes execution videos of tasks based on initial scene images, allowing robots to track and execute these tasks directly. Evaluated on four tasks, the method demonstrates better generalization compared to traditional behavior cloning approaches. The key insight is that using common tools bridges the gap between human and robot manipulation, enabling effective learning from minimal demonstrations.", "summary_of_recommendation": "The paper is well-written and introduces a compelling approach to enhance robot training data through video generation models, which can synthesize new videos from a single initial state image. This method effectively combines behavior cloning's efficiency with video generation to create synthetic data, demonstrating improvement over current state-of-the-art techniques. However, the reliance on known tools could limit scalability in real-world scenarios.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "I think the authors should expand their limitations section to include that their method may not work effectively when there are multiple objects that can be interacted with similarly using the chosen tool. Additionally, their method may also fail with tasks that require precise force application, such as cutting or other force-dependent manipulations. Addressing these limitations would provide a more comprehensive understanding of the method's constraints and potential areas for improvement.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lRs7xB1vpU", "reviewer": "Reviewer_MoCz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present Dreamitate, a visuomotor policy learning framework that fine-tunes a video diffusion model on human demonstrations for dexterous manipulation tasks. The method generates videos of task execution conditioned on novel scenes and uses these videos to directly control robots. The key insight is using common tools to bridge the embodiment gap between human and robot manipulators. Dreamitate demonstrates superior generalization compared to behavior cloning methods, validated across four real-world tasks: rotation, scooping, sweeping, and shape pushing.", "summary_of_recommendation": "While the paper presents a novel approach to visuomotor policy learning using video generation and shows promising results, there are several areas that could benefit from further development. The focus on non-prehensile tasks and limited real-world complexity are significant weaknesses. Addressing these issues and expanding the evaluation to include prehensile and more realistic tasks would significantly strengthen the contribution.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Task Complexity: All the tasks are non-prehensile. How does the method perform on tasks that require picking or grasping?\n2. Failure Cases: The tasks can be solved by the proposed method, but what are the typical failure cases for your method and the diffusion policy?\n3. Performance Comparison: The Push-Shape task appears similar to the push-T task in the diffusion policy paper. The performance in the Push-Shape task is much worse. Why?\n4. Training Data Subsampling: What does \"open-loop system without subsampling the training data\" in line 155 mean?\n5. Task Difficulty: The hardest task, Push-Shape, has been well-solved with both learning-based and model-based methods. Can you perform more realistic and harder tasks?\n6. Robustness to Perturbations: How does the system handle perturbations during the trajectory execution?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
+ Dreamitate is a novel approach for using video-diffusion models in manipulation.
+ The approach is scalable, interpretable, and agent-agnostic. 
+ The approach outperforms Diffusion Policies on real-world tasks. 
+ The paper is well written.

Weaknesses:
- Missing video-conditioned baselines such as UniPi.
- Evaluations are missing prehensile tasks like grasping.
- Some details on the evaluation setup are missing.
- The method uses custom tools during data collection, which might hinder general applicability. 
- The limitations section could benefit from a discussion on interactions with multiple objects, robustness to perturbations, and failure cases.

Post rebuttal:  
Overall, the reviews are mixed; some key concerns regarding the generalizability of the approach remain. However, "video diffusion for robotics" is a relatively new field, so limitations could be addressed in future works.
Decision: Accept