Learning to Act from Actionless Videos through Dense Correspondences
Authors: Po-Chen Ko
Jiayuan Mao
Yilun Du
Shao-Hua Sun
Joshua B. Tenenbaum
Venue: ICLR 2024 spotlight (2024)
Keywords: Video-Based Policy
Video Dense Correspondence
Primary area: applications to robotics, autonomy, planning
TL;DR: We developed a robot policy using images, eliminating action annotations. It's trained on RGB videos, effective for table-top tasks and navigation. Our framework allows rapid modeling with just 4 GPUs in a day.

Abstract: In this work, we present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments from few video demonstrations without using any action annotations. Our method leverages images as a task-agnostic representation, encoding both the state and action information, and text as a general representation for specifying robot goals. By synthesizing videos that "hallucinate" robot executing actions and in combination with dense correspondences between frames, our approach can infer the closed-formed action to execute to an environment without the need of any explicit action labels. This unique capability allows us to train the policy solely based on RGB videos and deploy learned policies to various robotic tasks. We demonstrate the efficacy of our approach in learning policies on table-top manipulation and navigation tasks. Additionally, we contribute an open-source framework for efficient video modeling, enabling the training of high-fidelity policy models with four GPUs within a single day.

Published as a conference paper at ICLR 2024
LEARNING TO ACT FROM ACTIONLESS VIDEOS THROUGH DENSE CORRESPONDENCES
Po-Chen Kot National Taiwan University
Jiayuan Mao MIT CSAIL
Yilun Du MIT CSAIL
Shao-Hua Sun National Taiwan University
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL
ABSTRACT
In this work, we present an approach to construct a video-based robot policy ca- pable of reliably executing diverse tasks across different robots and environments from few video demonstrations without using any action annotations, Our method leverages images as a task-agnostic representation, encoding both the state and action information, and text as general representation for specifying robot goals. By synthesizing videos that "hallucinate" robot executing actions and in combi- nation with dense correspondences between frames, our approach can infer the formed action to execute to an environment without the need of any explicit action labels. This unique capability allows us to train the policy solely based on RGB videos and deploy learned policies to various robotic tasks. We demonstrate the efficacy of our approach in learning policies navigation tasks. Additionally, we contribute an open-source framework for effi- cient video modeling, enabling the training of high- high-fidelity policy models with four GPUs within a single day.
table-top manipulation and
INTRODUCTION
A goal of robot learning is to construct a policy that can successfully and robustly execute diverse tasks across various robots and environments. A major obstacle is the diversity present in different robotic tasks. The state representation necessary fold cloth differs substantially from the one needed for pouring water, picking and placing objects, or navigating, requiring a policy that can process each state representation that arises. Furthermore, the action representation to execute each task varies significantly subject to differences in motor actuation, gripper shape, and task goals, requiring a policy that can correctly deduce an action to execute across different robots and tasks.
One approach to solve this issue is to use images as a task-agnostic method for encoding both the states and the actions to execute. In this setting, policy prediction involves synthesizing a video that depicts the actions robot should execute (Finn& & Levine, 2017; Kurutach et al., 2018; Du et al., 2023), enabling different states and actions to be encoded in modality-agnostee manner. However, directly predicting an image representation a robot should execute does not explicitly encode the required robot actions to execute. To address this, past works either learn an action-specific video prediction model (Finn & Levine, 2017) spsk-spe dnverse-dnamiie model to predict actions from videos (Du et al., 2023). Both approaches rely on task-specific action labels which can be expensive to collect in practice, preventing general policy prediction across different robot tasks.
This work presents a method that first synthesizes a video rendering the desired task execution; then, it directly regresses actions from the synthesized video without requiring any action labels or task-specific inverse- inverse-dynamics model, enabling us to directly formulate policy learning as a video generation problem. Our key insight is that action inference from video in many roboties tasks can be formulated as solving for a rigid 3D transform of objects or points in the generated video. Such a transform can be robustly inferred using off-the-shelf optical flow and segmentation networks, and actions can then be executed from these transforms using off-the- off-the-shelf inverse kinematics and motion planners. We illustrate the efficacy of our method across various robotics tasks ranging from table-top assembly, ego-centric object navigation, and real-world robot manipulation in Figure 1.
1 Work done while Po-Chen Ko is visiting student MTT. Project page: https://tow-diffusion. wtt.ffuuiioioigguiiooo/o

---

Published as a conference paper at ICLR 2024
Assembly
Find toaster
Put the peach into the bowl
Figure Diverse Task Execution without Action Labels. Our approach can execute policies given only synthesized video, without any action labels, across various manipulation, navigation, and weal-world tasks.
Another limitation of existing approaches that formulate policy prediction as a video prediction problem is that they suffer from high computational costs during training, requiring the use of over 256 TPU pods (Du et al., 2023), with limited availability of the underlying source code. As contribution, we provide an open-source codebase for training video policy models. Through a series of architectural optimizations, our framework enables the generation of high-fidelity videos for policy execution, with training accomplished on just 4 GPUs in a single day.
Concretely, this work contributes the following: (1) We propose a method to infer actions from video prediction without the need of any action labels by leveraging dense correspondences in a video. (2) We illustrate how this approach enables us to learn policies that can solve diverse tasks across both table-top manipulation and navigation. (3) We present an open-soune spen-source framework which enables efficient video modeling that enables us learn policies efficiently on 4 GPUs in single day.
2 RELATED WORK
Robot Learning from Videos. A large body of work has explored how to leverage videos for robot learning (Sun al., 2018; Pari al., 2022; Nair al., 2022; Shao et al., 2021; Chen et al., 2021; Bahl et al., 2022; Sharma et al., 2019; Lee & Ryoo, 2017: Du et al., 2023; Chethan et al., 2023; Karamcheti al., 2023). One approach relies upon using existing video datasets to construct effective visual representations (Pari et al., 2022; Nair et al., 2022; Karamcheti et al., 2023). A Alternatively, goal or subtask information for robotic execution may be extracted for videos (Shao et al., 2021; Chen et al., 2021; Chethan et al., 2023; Bahl et al., 2022: Sharma et al., 2019; Lee & Ryoo, 2017; Sivakumar et al., 2022) or used as a dynamics model for planning (Finn & Levine, 2017; Kurutach et al., 2018). The absence of rewards and action labels distinguishes our work from offline RL (Levine et al., 2020), Most similar to our work, in UniPi (Du et al., 2023), policy prediction may directly be formulated text-conditioned video generation problem. Our approach extends UniPi and illustrates how dense correspondences enable action inference without any explicit action labels. Another work with a similar high-level idea ours (Bharadhwaj al., 2023) predicts hand poses from videos and uses them directly for control, while we infer actions from object-centric trajectories. While hand poses contain more details of manipulator-objec interactions, object-centric actions may help eesoddmnbd transfer.
Leveraging Dense Correspondences. Dense correspondences have emerged as an effective implicit parameterization of actions and poses (Florence et al., 2018; Manuelli et al., 2022; Yen-Chen et al., 2022; Simeonov et al., 2022; 2023; Chun al., 2023; Sundaresan et al., 2020: Ryu al., 2023). Given dense correspondences in 2D (Florence et al., 201 8; Manuelli et al., 2022; Sundaresan et al., 2020; Yen-Chen et al., 2022) of 3D (Simeonov et al., 2022; 2023; Chun et al., 2023; Ryu et al., 2023) both object and manipulator poses may be inferred by solving for rigid transforms given correspondences. Our approach uses dense correspondences between adjacent frames of synthesized videos to calculate object of scene transformations and then infer robot actions.
Learning from Observation. In contrast to imitation learning (learning from demonstration: Osa et al., 2018; Kipf et al., 2019; Ding et al., 2019; Fang et al., 2019; Mao et al., 2022; Wang et al., 2023), which assumes access to expert actions, learning from observation methods (Torabi et al., 2019b; 2018; 201 2019a; Lee al., 2021; Karnan et al., 2022) learn from expert state sequences (e.g..
2

---

Published as a conference paper at ICLR 2024
(a) Input
Pick the
Step Video Pred.
Synthesized (b)
Video
blue bowl
imgo (imput)
inng:
iung:
img:
imga
Step 2: Flow Est.
Depth State
Computed
Step 3: SF(3) Est. (Flow Depth)
flow
flows
flow
flow
Optical Flow
(c)
Object Pose
(d)
& Robot Command
Figure Overall framework AVDC. (a) Our model takes the RGBD observation the current environmental
state and textual goal description as its input. (b) It first synthesizes a video of imagined execution of the
lask using diffusion model. (c) Next, it estimates the optical flow between adjacent frames in the video. (d)
Finally, leverages the optical flow dense correspondences between frames and the depth of the first frame compute SE(3) transformations of the target object, and subsequently, robot arm commands.
video frames). Action-free pre-training methods (Baker et al., 2022; Escontrela et al., 2023) extract knowledge from unlabeled videos and learn target tasks through RI.. extcolorblackFor example, recent approach involves learning value functions by pre-training on existing video datasets (Chethan et al., 2023). Despite encouraging results, these methods require interacting with environments, which may be expensive even impossible. In contrast, our proposed method does not require environmental interactions and therefore more applicable.
3 ACTIONS FROM VIDEO DENSE CORRESPONDENCES
The architecture of our proposed framework, Actions from Video Dense Correspondences (AVDC), is depicted in Figure 2. AVDC consists of three modules. Given the initial observation (i.e., an RGBD image of the scene and textual task description), we first employ video synthesis model to generate video that implicitly captures the sequence of required actions (Section 3.1). Then, we use flow prediction model to estimate the optical flow of the scene and objects from the synthesized video (Section 3.2). Finally, leveraging the initial depth map and predicted optical flows, we reconstruct the movements of objects for manipulation or robots for navigation, described in Section 3.3.
3.1 TEXT-CONDITIONED VIDEO GENERATION
Our text-conditioned video generation model is a conditional diffusion model. The diffusion model takes the initial frame and text description its condition and learns to model the distribution of possible future frames. Throughout this paper, our video generation model predicts a fixed number of future frames (T' 8 in our experiments).
The diffusion model aims to approximate the distribution p(img1:r (ingg:rlimgg txt), where img1:1 repre- sents the video frames from time step to img, denotes the initial frame, and txt represents the task description. We train a denoising function that predicts the noise applied to img given the perturbed frames. Given the Gaussian noise scheduling 3,, our overall objective is,
MASE = || <- (VT- Brimg -vi-niimg + Bie,, ax))
where is sampled from multivariate standard Gaussian distribution, and / is a randomly sampled diffusion step t. A main practical challenge with training such video diffusion models is that they are usually computationally expensive. For example, the closest work us, UniPi (Du et al. (2023)), requires over 256 TPU pods to train. In this paper, we build a high-fidelit video generation model that can be trained on GPUs a single day through a series of architectural optimizations, Section G presents complexity analyses and how the process can be significantly accelerated.
Our model modified version of the image diffusion model proposed by Dhariwal & Nichol (2021), built upon U-Net (Ronneberger et al., 2015), as illustrated in Figure 3a. The U-Net consists of the same number of downsample blocks and upsample blocks. enhance consistency with the initial frame, we concatenate the input condition frame img, to all future frames img1:r- To encode the text, we use a CLIP-Text (Radford et al., 2021) encoder to obtain vector embedding and combine it into the video generative model additional inputs to individual downsampling and upsampling blocks.
Importantly, we use a factorized spatial- temporal convolution similar to the model from Ho et al. (2022), within each ResNet block (He et al., 2016). As shown in Figure 3b, in approach, the 5D input feature map with shape (B, H, W, T, C.), where B is the batch size, H and W represent the spatial dimensions, T' is the number of time frames, and C denotes the number of channels,
3

---

Published as a conference paper at ICLR 2024
Input frames: clean initial frame T-1 noisy subsequent frames HxxxTx(2x3)
Output: denoised video
BxHxWxTXC
HxWxTx3
Conv 2D (Spatial)
Res. Block
(BxT)xIIXWXC
Res. Block
Conv ID (Temporal)
Res. Black
(BXHXW)XTXC
Res. Block
Reshape
Goal (text) Pick the blue bowl
BXHXWXTXC
CLIP
(a) U-Net architecture for our base diffusion model
(b) Factorized spatial-temporal ResNet block
Figure 3: Network architecture of our video diffusion model. (a) We use U-Net architecture follow- ing Dhariwal & Nichol (2021) but extending it videos. (b) We use factorized spatial-temporal convolution kernel Sun et al. (2015) as the basic building block. Dashed lines both figures represent residual connec-
tions (He et al., 2016).
undergoes two consecutive convolution operations. First, we apply spatial convolution identically and independently to each time step / = 2, ................................ .. T. Then, we employ a temporal convolution layer identically and independently at each spatial location. This factorized spatial-temporal convolution replaces conventional 3D convolution methods, leading to significant improvements in training and inference efficiency without sacrificing generation quality. More details on the model architecture and training can be found in Section F.
3.2 FL.WW PREDICTION
To regress actions from predicted videos, we leverage flow prediction as an intermediate representation. We employ off-the shelf GMFlow, transformer architecture specifically designed for optical flow prediction (Xu et al., 2022). Given two consecutive frames img, and img,+) predicted by the video diffusion model, GMFlow predicts the optical flow between two images which is essentially a pixel-level dense correspondence map between two frames. This allows us to track the movement of each input pixel with simple integration of this vector field over time.
vector field the image,
Alternatively, one could train diffusion models to directly predict the flow by first preprocessing training videos with the flow prediction model. However, in our experiments, we encountered challenges in optimizing such models and observed that they failed to match the performance achieved by the two-stage inference pipeline. We conjecture that this difficulty arises from the lack of spatial and temporal smoothness in flow fields. For instance, the flow field is sparse when only single object moves. Consequently, the Gaussian diffusion model may not be the optimal model for flow distributions. We empirically compare two alternatives in subsequent experiments.
3.3 ACTION REGRESSION FROM FLOWS AND DEPTHS
Based on the predicted flow, which essentially gives us a dense prediction of pixel movements, we can reconstruct object movements and robot movements in the video. Our key insight is to, given the 3D information (depth) of the input frame and dense pixel tracking, reconstruct a sequence of 3D rigid transformations for each object. In this work, we explore two different settings: predicting object transformations assuming a fixed camera (fixed-camea object manipulation) and predicting camera (robot) movement assuming static scene (visual navigation).
Predict object-centric motion. We first consider predicting 3D object motions in videos assuming a fixed camera. We represent each object as a set of 3D points (x:). The points corresponding to the object of interest will be extracted by external segmentation methods, such pretrained image segmentation model, or simply specified by the human. Given the camera intrinsic matrix and the input RGBD image, we can compute the initial 3D positions of these points. Let T; denote the rigid body transformation of the object at time step relative to the initial frame. We can express the projection of 3D point onto the image plane time step as KT, (u, Vt, de), where K is the camera intrinsic matrix. Furthermore, the projected 2D point on frame is thus (u//d,, Vt/d,
The optical flow tracking provides with the projection of the same point in frame t, specifically w//d, de and v,/d. By tracking all points in {r:). we can find the optimal transformation T that minimizes the following 1.2 loss:
LTaans E₁ luf
(KTaa); 2 (KTaa))
(KTia:) (KT)a))
+ 22
4

---

Published as a conference paper at ICLR 2024
where (ug, v/) is the corresponding pixel of point 2; in frame i, and (KT,z;), denotes the i-th entry of the vector. It's worth noting that even if we do not directly observe de, this loss function remains well-formed based on the assumption that T, represents rigid body transformation.
During execution, we first extract the mask of the object to manipulate and use the dense correspon- dences in predicted videos to compute the sequence of rigid body transformations for the object. Next, given inferred object transformations, we can use existing off-the-shelf robotics primitives to generalizably infer actions in the environment. In particular, if the object is graspable, we randomly sample a grasp on the object and then compute the target robot end-effector pose based on the target object pose and the grasping pose. When the object is not directly graspable (e.g.. a door), we similarly sample contact point and use push action to achieve the target object transformation.
We treat the grasp/contact point as the first subgoal. Then, we iteratively apply the computed transformation on the current subgoal to compute the next subgoal until all subgoals are computed. Next, we use a position controller to control the robot to reach the subgoals one by one. More details on inferring robot manipulation actions can be found in Section H.I. In contrast to our approach, directly learning explicitly regress actions using a learned inverse dynamics requires a substantial number of action labels so that neural network can learn existing knowledge such as inverse dynamics, grasping and motion-planning
Inferring Robot Motion. The similar algorithm can also be applied to predict robot (i.e., the camera) motion assuming all objects are static. Due to the duality of camera motion and object motion, we use exactly the same optimization algorithm find Tj (object-centric motion), and the camera motion C₁= = (Tt) Concretely, we make the following modifications to adapt AVDC to navigation tasks. (1) The video diffusion model is trained duplicate the last frame once the object is found. (2) Instead of tracking objects, we utilize the optical flow of the whole frame to estimate the rigid transformations between frames. (3) Based on the calculated rigid transformations, we simply map the transformations to the closest actions, detailed in Section H.2.
Depth Estimation. We can reconstruct 3D object or robot trajectories solely from the depth map of the initial frame (i.e., the subsequent depth maps is not required). By leveraging dense correspondences between frames and assuming rigid object motion, we can reconstruct accurate 3D trajectories. This holds significant advantages as it enables us to train video prediction models exclusively using RGB videos, allowing for learning from online sources like YouTube, and only requires an RGB-D camera (or monocular depth estimator) at execution time. By eliminating the dependence on depth maps from subsequent frames, our system is significantly more adaptable to various data sources.
Replanning Strategy. After inferring the object or robot trajectories, we can execute the trajectory using a position controller in an open-loop manner. Yet, it can suffer from accumulated errors. As the planning horizon increases, the accuracy of predicted object locations diminishes due combined errors in video synthesis and flow prediction. To mitigate this issue, we propose a replanning strategy. If the robot movement is smaller than Imm over 15 consecutive time steps while the task has not been fulfilled, we re-run our video generation and action prediction pipeline from the current observation.
EXPERIMENTS
We describe the baselines and the variants of our proposed method AVDC in Section 4.1. Then, we compare AVDC to its variants and the baselines on simulated robot arm manipulation tasks in Meta- World (Figure 4a) in Section 4.2 and simulated navigation tasks in iTHOR (Figure 4b) in Section 4.3. Note that although it is possible to obtain ground-truth actions from demonstrations in these two domains, our method does not use these actions; instead, these actions are only used by the baselines to provide an understanding of the task difficulty. Then, Section 4.4 evaluate the ability of AVDC to control robots by learning from oft-of-domain human videos without actions, illustrated in Figure 4c. In Section 4.5, we leverage the Bridge dataset (Figure 4d) and evaluate AVDC on real-world manipulation tasks with Franka Emika Panda robot arm (Figure 4e). Extended qualitative results can be found in Section B and additional experimental details can be found in Section H.
4.1 BASELINES AND VARIANTS OF AVDC
Baselines. We compare AVDC multi-task behavioral cloning (BC) bascline given access to set of expert actions from all videos (15, 216 labeled cume-acciion pairs in Meta-Woo and 5, 757 in iTHOR), which are unavailable to method. This baseline encodes the RGB observation to
5

---

Published as a conference paper at ICLR 2024
(a) Woold
(b) iTHOR
(c) Visual Pusher
(d) Bridge
(e) Panda Arm
Figure 4: Environments & Tasks. (a) Weea-World is simulated benchmark featuring various tasks with Sawyer robot arm. (b) ITHOR is simulated benchmark for embodied common sense reasoning. We adopt its object navigation task, requiring navigating to target objects located in different rooms. (c) Visual Pusher a real-world video dataset with 195 human pushing videos. (d) Bridge is real-world video dataset comprised of 33, 078 robot demonstrations conducting various kitchen tasks. (e) Panda Arm is a real-world pick-and-plast tabletop environment with Franka Emika Panda robot arm.
feature vector with a ResNet- 18 (He et al., 2016). Then, the feature vector is concatenated with a one-hot encoded camera ID and task representation encoded by the CLIP-Text model (Radford et al., 2021). The concatenated representation is then fed to a 3-layer MI.,, which produces an action. We explore initializing the weights of ResNet-18 from scratch (BC-Scratch) from the pre-trained parameters of R3M (Nair et al., 2022) (BC-R3M).
Additionaly, we experimented with Diffusion Policy (Chi et al., 2023), which also leverages denoising diffusion, but directly predicts actions instead of video frames like we did. We followed the setting used by most of the experiments in the original paper. More details are described in Section H.1.4.
We also implement UniPi (Du et al., 2023), a learning- frimiii--roo-vvieoo method that learns an inverse dynamics model to generate actions from videos, as a baseline. Specifically, UniPi infers actions from the videos synthesized by AVDC. Since the exact number of steps between two generated frames in our model may vary across different episodes, we modify the inverse dynamics model to output an additional binary label indicating whether to switch to the next frame of synthesized video plans. This predictor can be trained with the demonstrations (with actions) used to train the BC baselines.
AVDC and its Variants. We compare AVDC to its variants that also predict dense correspondence.
AVDC (Flow) learns to directly predict the optical flow between frames as described in Sec- tion 3.2. We include this variant to justify our 2-stage design, which synthesizes a video and then infers optical flows between each pair of frames.
AVDC (No Replan) is the opened-loop variant of our proposed method, which synthesizes a video, infers flows, produces a plan, executes it, and finishes, regardless of it fails or succeeds. We include this variant to investigate whether our replanning strategy is effective.
AVDC (Full) is our proposed method in full, employing the 2-stage design and can replan.
Additional Ablation Studies and Experiments. We also include additional ablation studies on the effect of first-frame conditioning in video generation and different text encoders (e.g.. CLIP and T5) in Section E, study of extracting object mask with existing segmentation model in Section D.1, an experiment training BC with more data in Section D.2, using object masks extensively as proxy for actions in Section D.3, and a quantitative quality analysis on the synthesized videos in Section D.4.
4.2 HETA-WORLD
Setup. Meta- Woold (Yu et al., 2019) is a simulated benchmark featuring various manipulation tasks with Sawyer robot arm. We include 11 tasks, and for each task, we render videos from 3 different camera poses. The same set of camera poses is used for training and testing. We collect demonstrations per task per camera position, resulting in total 165 videos. To isolate the problem of learning object manipulation skills, for our methods and all its variants, we provide the ground- truth segmentation mask for the target object. We include additional study on using external segmentation models in Appendix D.I.
Each policy is evaluated each task with 3 camera poses, each with 25 trials. A policy succeeds if it reaches the goal state within the maximum environment step and fails otherwise. The positions the robot arm and objects are randomized when each episode begins. The result is reported in Table 1.
Comparison to Baselines. Our method AVDC (Full) consistently outperforms the two BC baselines (BC-Scratch and BC-R3MI and UniPi all the tasks by large margin. Furthermore, AVDC

---

Published as a conference paper at ICLR 2024
door-open
door-close
basketball
shelf-place
btn-press
btn-press-toy
BC-Scratch
21.3%
36.0%
0.0%
0.0%
34.7%
12.0%
BC-RAM
1.3%
58.7%
0.0%
0.0%
36.0%
4.0%
UniPi (With Replan)
0.0%
36.0%
0.0%
0.0%
6.7%
0.0%
Diffusion Policy
45.3 %
45.3 %
8.0 %
0.0 %
40.0 %
18.7 %
AVDC (ID)
0.0%
36.0%
0.0%
0.0%
0.0%
0.0%
AVDC (Flow)
0.0%
0.0%
0.0%
0.0%
1,3%
40.0%
AVDC (No Replan)
30.7%
28.0%
21.3%
8.0%
34.7%
17.3%
AVDC (Full)
72.0%
89.3%
37.3%
18.7%
60.0%
24.0%
faucct-close
faucet-open
handle-press
hammer
assembly
Overall
BC-Scratch
18.7%
17.3%
37.3%
0.0%
1.3%
16.2%
BC-R3M
18.7%
22.7%
28.0%
0.0%
0.0%
15.4%
UniPi (With Replan)
4.0%
9.3%
13.3%
4.0%
0.0%
6.1%
Diffusion Policy
22.7%
58.7%
21.3%
4.0%
1.3%
24.1%
AVDC (ID)
4.0%
9.3%
13.3%
4.0%
0.0%
6.1%
AVDC (Flow)
42.7%
0.0%
66.7%
0.0%
0.0%
13.7%
AVDC (No Replan)
12.0%
17.3%
41.3%
0.0%
5.3%
19.6%
AVDC (Full)
53.3%
24.0%
81.3%
8.0%
6.7%
43.1%
Table Meta-World Result. We report the mean success rate across tasks. Each entry the table shows the average success rate aggregated from 3 camera poses with 25 seeds for each camera pose.
Synthesized
Video
Goal
Success Rate (%)
Assembly
State
40
Predicted
Camers
Optical
Camera
Depth
Flow
20
Canera
Number of Replans
Inferred
Actions
Figure 6: Number of Replanning Steps vs. Success Rate. Our method AVDC achieves higher success rates across all viewing angles with planning trials. justifying effective- ness our replanming strategy.
Figure S: Qualitative Results Meta-Worke AVDC can reliably syathesize a video, predict optical flow between frames, and infer and execute actions to fulfill the assembly task. Current subgoals (+) and next subgoals (+) are rendered in inferred action visualizations.
(Full) also outperforms the Diffusion Policy in 10 out of 11 tasks and in overall performance by sigmificant margin. This indicates that the tasks are still very challenging, even with access to expert actions. Note that AVDC (Full) is able to solve the task "hammer," which involves using tools, with performance surpassing all baselines. This is done by predicting actions based on tool motions.
and achieves reasonable performance on faucet-close and handle-press, while performing very poorly on the rest of the tasks. As described in Section 3.2, the diffusion model employed in this work may not be optimal for flow prediction. Also, AVDC (Full) consistently outperforms AVDC (No Replan), justifying the effectiveness of closed-loop design, enabling replanning when the policy fails.
Comparing AVDC Variants. AVDC (Flow) performs the best on certton-pres--oopllee
Intermediate Outputs. To provide insights into the pipeline of AVDC, we visualized the synthesized video, predicted optical flow, and inferred actions (i.e., motion planning) in Figure 5, Our diffusion model synthesizes reasonable video showing the robot arm picking up the nut and placing it onto the peg. The optical flow predicted from video frames accurately captures the robot arm's 's motions. Then, based on the predicted flow, the inferred actions can reliably guide the arm to fulfill the task.
We investigate how varying the maximum number of replanning step affects the performance of AVDC. As presented in Figure 6, the success rate consistently increases with more replanning trials, demonstrating the effectiveness our proposed replanning strategy.
Effect of Replanning Trials.
Failure Modes. The primary failure mode we observed is the errors made by the optical flow tracking model, partially because these models are not trained on any in-domain data. Since the prediction resolution is not very high in experiments, small pixel-level errors in tracking small objects would
7

---

Published as a conference paper at ICLR 2024
Synthesized
Video
BC-Seratch
Goal Television
Kitchen 1.7%
0.0%
26.7%
Living Room
3.3%
0.0%
23.3%
State
Bedroom 1,7%
1.7%
38.3%
Overall
2.1%
0.4%
31.3%
Predicted Optical Flow
Table 2: iTHOR Result. We report the mean success rate, aggregated from types of objects per room with 20 episodes per object. Both the BC baselines fail to achieve meaningful per- formance on the iTHOR object naviga- tion tasks. On the other hand, AVDC performs reasonably with 31.3% av- erage success rate.
Depth
Rocatmaight
Acceforvard
Sorstotete
Dote
Inferred
Actions
Figure 7: Qualitative Results iTHOR. AVDC reliably syn- thesize a video, predict optical flow between frames, and infer and exccute actions to navigate to the television.
result in large errors in the 3D space. We believe that by directly increasing the resolution of video synthesis or by training an in-domain optical flow model, we can improve the performance.
4.3 iTHOR
Setup. iTHOR (Kolve et al., 2017) a simulated benchmark for embodied common sense reasoning. We consider the object navigation tasks for evaluation, where agent randomly initialized into scene learns to navigate to an object of a given type (e.g., toaster, television). At each time step, the agent observes 2D scene and takes of the four actions: GoveForward Rot notateLer ft, RotateRi ght, and Done. We chose 12 different objects to be placed at 4 type of rooms (e.g.. kitchen, living room). No object segmentation is required in this navigation task.
Each policy is evaluated on 12 object navigation tasks distributed in 4 different types of rooms (3 tasks for each room). policy succeeds if the target object is the agent's sight and within 1.5m distance within the maximum environment step when Done is predicted and fails otherwise. The position of the agent is randomized at the beginning of each episode. The result reported in Table 2.
Comparison to Baselines. Our proposed method AVDC can find target objects in different types of rooms fairly often (31.3%), while the two BC baselines fail entirely. BC-R3M with a pre-trained ResNet-18 18 performs worse than BC-Scratch, which can be attributed to the fact that R3M is pre- trained on robot manipulation tasks and might not be suitable for visual navigation tasks.
Intermediate Outputs. The intermediate outputs produced by AVDC are presented in Figure 7. The diffusion model can synthesize video showing an agent navigating to the target object. Then, desired agent movements ease of mapping the flow to MoveF o rrward, aotateLeffa or Rot attateRion ght. When no flow is predicted, it indicates the agent has found the object and selects Done the predicted action.
be easily inferred from the predicted optical flow, resulting in the
4.4 CHOSS-EMBBDDMENT LEARNING: FROM HUMAN VIDEOS TO ROBOT EXECUTION
We aim examine if AVDC achieve cross-embodiment learning, e.g., leverage human demon- stration videos to control robots to solve tasks.
(a) Synthesized Video
Setup. We evaluate our method with Visual Pusher tasks (Schmeckpeper et al., 2021; Zakka et al., 2022). Specifically, we learn video diffusion model from only actionless human pushing data (198 videos), with the same U-net architecture used in Meta-Worle experiments and trained the model for 10k steps. Then, we evaluate AVDC on simu- lated robot pushing tasks without any finettuning
(b) Robot Execution
Results. AVDC exhibits strong zero-shot transfer capability, achieving 90% zero-shot success rate out of 40 runs. This indicates that AVDC can per- form cross- eess-immodiment learning utilizing out- of-domain human videos to achieve reliable robot execution. A synthesized video and the corresponding robot execution illustrated in Figure 8.
Figure 8: Qualitative Results on Visual Pusher. AVDC can (a) synthesize video plans by watching human demonstrations and (b) infer actions to con- trol the robut without any fine-tuning.
8

---

Published as a conference paper at ICLR 2024
Goal Put apple in platz
State
Depth
Qualitative Results Bridge. AVDC can reliably infer cur- rent subgoals (+) and next subgoals (+) for woold robot manipulation tasks.
Figure 9:
Figure 10: Qualitative Results Franka Emika Panda. AVDC can reliably synthesize a video, predict optical flow between frames, and infer and exccute actions to fulfill the assembly task. Current subgoals (+) and next subgoals (+) are rendered in the bottom row.
4.5 REAL-WoOL WORLD FRANKA EMIKA PANDA ARM WITH BRIDGE DATASET
We aim to investigate if our proposed framework AVDC can tackle real-world robotics tasks. To this end, we train our video generation model on the Bridge dataset (Ebert et al., 2022), and perform evaluation on real-world Franka Emika Panda tabletop manipulation environment.
Setup. The Bridge dataset (Ebert et al., 2022) provides 33, 078 teleoperated WidowX 250 robot demonstrations of various kitchen tasks captured by web camera without depth information. Our real-world setup comprises a Franka Emika Panda robot arm and an Intel Realsense D435 RGBD camera mounted a fixed frame relative to the table. Due to the differences in camera FOVs and the environmental setup, directly applying the video generative model trained on Bridge to our setup does not generalize well. We thus fine-tuned the diffusion model with 20 human demonstrations collected with our setup. In our real-world evaluation, we assume that the target object can be grasped using a top-grasp that no reorientation of the target object is needed. Note that both the Bridge dataset and our human demonstration datasets do not contain any action label relevant to our robot: Bridge is based on different robot model and our tabletop videos are human hand manipulation videos.
Zero-Shot Generalization of Bridge Model. We found that the video diffusion model trained on Bridge videos can reasonably generalize real scenes without fine-tuning, as discussed in Section C.
Results. The predicted object motion qualitative results on the Bridge dataset are presented in Fig- ure 9. AVDC can reliably synthesize videos, predict optical flow, identify target objects, and infer actions. Figure 10 presents the visualizations of planned robot trajectories, showcasing the successful deployment of our system. More qualitative results can be found Section B. We also quantitatively evaluated the entire pipeline. To this end, we set up 10 scenes with different initial object configura- tions and tasks. Each task requires a pick-and-place of an object of a specified category (e.g., apple) to container (e.g., plate). The results are detailed in Section H.3.
DISCUSSION
Limitations. The proposed AVDC, while being successful in diverse simulated and real-world settings, faces several challenges. First, the algorithm may lose track of objects heavily occluded by the robot arm struggle with optical flow prediction when there are rapid lighting changes or significant object movements. Additionally, our current implementation is not adept at handling tasks with deformable objects, requiring future work to develop new strategies for tracking representing these objects, such as key-point- based tracking. Real-world manipulation tasks, which often require predicting grasps or contact points, are also challenging due the disparity between human hands and different robot hands, necessitating the integration of specialized manipulation algorithms such as grasp prediction modules (Sundermeyer al., 2021). Lastly, force information RGB videos is unobtainable. Future work may consider leveraging real-world interaction data to address this.
Conclusion. This work presents an approach to learning to act directly in environments given only RGB video demonstrations by exploiting dense correspondence between synthesized video frames. We illustrate the general applicability of our approach in both simulated and real-world manipulation and navigation tasks and embodimem settings. We further present open-soure implementation for fast and efficient video modeling. We hope our work inspires further work on learning from videos, which can be readily found the internet and readily captured across robots.

---

Published as a conference paper at ICLR 2024
Acknowledgement. We thank anonymous reviewers for their valuable comments. We gratefully acknowledge support from ONR MURI grant N00014-16 1-000-1----0077; from the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF- SEF-12212166, from NSF grant 2214177; from Air Force Office Scientific Research (AFOSR) grant FA9550-22 1--50022-11- 2221-0029; from ONR MURI grant from the MIT-IBM Watson AI Lab; from the MIT Quest for Intelligence; and from the Boston Dynamics Artificial Intelligence Institute. This project was partially supported by the National Science and Technology Council in Taiwan (NSTC Shao-Hus Sun was partially supported by the Yushan Fellow Program by the Ministry of Education, Taiwan, Any opinions. findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors.
N00014-22-1-274 1-2740; from ARO grant W91 INF-23-1--000000
REFERENCES
Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. In Robotics: Science and Systems, 2022. 2
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video Pre'rriining (VPT): Learning to Act by Watching Unlabeled Online Videos. In Neural Information Processing Systems, 2022.
Homanga Bharadhwaj, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Zero-Shot Robot Manipulation from Passive Human Videos. arXiv:230..02011, 2023.
Annie S Chen, Suraj Nair, and Chelsea Finn. Learning Generalizable Robotic Reward Functions from "In-The-Ve Willd Human Videos. In Robotics: Science and Systems, 2021.
Bhateja Chethan, Guo Derek, Ghosh Dibya, Singh Anikait, Tomar Manan, Vuong Quan, Chebotar Yevgen, Levine Sergey, and Kumar Aviral. Robotic Offline RL from Internet Videos via Value- Function Pre-Training. arXhv:2309.1304/ 2023. 2,
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Roboties: Science and Systems (RSS), 2023.
Ethan Chun, Yilun Du, Anthony Simeonov, Tomas Lozano-Perez, and Leslie Kaelbling. Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation. In IEEE International Conference on Robotics and Automation, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. In Neural Information Processing Systems, 2021. 3,4
Yiming Ding, Carlos Florensa, Picter Abbeel, and Mariano Phiclipp. Goal-Conditioned Imitation Learning. Neural Information Processing Systems, 2019.
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofrr Nachum, Joshua B Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning Universal Policies via Text-Guided Video Generation. In Neural Information Processing Systems, 2023. 2,
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge Data: Boosting Generalization of Robotic Skills with
Datasets. In Robotics: Science and Systems, 2022. 9
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Young- woon Lee, Danijar Hafner, and Pieter Abbeel. Video Prediction Models as Rewards for Reinforce- ment Learning. arXiv:2305. 14343, 2023. 3
Bin Fang, Shidong Jia, Di Guo, Muhua Xu. Shuhuan Wen, and Fuchun Sun. Survey of Imitation Learning for Robotic Manipulation. International Journal of Intelligent Robotics and Applications, 2019. 2
10

---

Published as a conference paper at ICLR 2024
Chelsea Finn and Sergey Levine. Deep Visual Foresight for Planning Robot Motion. In IEEE International Conference on Robotics and Automation, 2017. 1,2
Peter R Florence, Lucas Manuelli, and Russ Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. In Conference on Robot Learning, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016. 3, 4, 6
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video Diffusion Models. In Neural Information Processing Systems, 2022.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General Perception with Iterative Attention. In International Conference on Machine Learning, 2021. 17
Siddharth Karamcheti, Suraj Nair, Annie S. Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv:2302.12766, 2023.
Haresh Karnan, Faraz Torabi, Garrett Warnell, and Peter Stone. Adversarial Imitation Learning from Video using a State Observer. In IEEE International Conference on Robotics and Automation, 2022. 2
Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. ComplLE: Compositional Imitation Learning and Execution. In International Conference on Machine Learning, 2019. 2
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Lo, Piotr Dollár, and Ross Girshick. Segment Anything. arXiv:2304.02665
2023. 15
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. 17
2017. 8
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart Russell, and Pieter Abbeel. Learning Plannable Representations with Causal InfoGAN. In Neural Information Processing Systems, 2018. 1,
Jangwon Lee and Michael S Ryoo. Learning Robot Activities from First-Person Human Videos
Using Convolutional Future Regression. In CVPRW, 2017. 2
Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph J. Lim. Generalizable Imitation Learning from Observation via Inferring Goal Proximity. In Neural Information Processing Systems, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXivv2005.01643, 2020. 2
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-set Object Detection. 23Xiv:2303.05499 2023. 15
Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation. In ISRR, 2022. 2
Jiayuan Mao, Tomás Lozano-Pérez Joshua B Tenenbaum, and Leslie Pack Kaelbling, PDSketch: Integrated Domain Programming, Learning, and Planning. In Neural Information Processing
Systems, 2022. 2
Luca Medeiros. Language Segment-Anything, 2023. URL https://gthuub.co///ueemmee
tang--semmen-anythinng GitHub repository. 15
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: Universal Visual Representation for Robot Manipulation. In Conference Robot Learning, 2022. 2,
11

---

Published as a conference paper at ICLR 2024
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An Algorithmic Perspective on Imitation Learning. Foundations and Trends@ in Robotics, 2018.
Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Aruncchalam, and Lerrel Pinto. The Surprising Effectiveness of Representation Learning for Visual Imitation. In Robotics: Science and Systems, 2022. 2
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, 2021. 3, 6
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, 2015.
Hyunwoo Ryu, Jeong-Hoon Lee, Hong-in Lee, and Jongeun Choi. Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning. In International Conference on Learning Representations, 2023.
Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models. In International Conference on Learning Representations, 2022. 19
Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Reinforcement learning with videos: Combining offline observations with interaction. In Conference on Robot Learning, 2021. 8
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2Robot: Learning Manipulation Concepts from Instructions and Human Demonstrations, IJRR, 2021. 2
Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third-person visual imitation learning via
decoupled hierarchical controller. In Neural Information Processing Systems, 2019. 2
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural Descriptor Fields: Squiv-aqu Object Representations for Manipulation. In IEEE International Conference on Robotics and Automation, 2022. 2
Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Pulkit Agrawal. Equivaaqui Relational Rearrangement with Neural Descriptor Fields. In Conference on Robot Learning, 2023. 2
Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic Telekinesis:
Learning Robotic
Hand Imitator by Watching Humans on Youtube. In Robotics: Science and Systems, 2022. 2
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In
International Conference Learning Representations, 2021. 19
Lin Sun, Kui Jia, Dit-Yan Yeung, and Bertram E Shi. Human Action Recognition using Factorized Spatio-Temporal Convolutional networks. In International Conference Computer Vision, 2015.
Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural Program Synthesis from Diverse Demonstration Videos. International Conference on Machine Learning, 2018.
Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan, Ashwin Balakrishna, Michael Laskey, Kevin Stone, Joseph Gonzalez, and Ken Goldberg. Learning Rope Manipulation Policies Using Dense Object Descriptors Trained Synthetic Depth Data. In IEEE International Conference Robotics and Automation, 2020. 2
Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, and Dieter Fox. Contact-Grasple Efficient 6-DoF Grasp Generation in Cluttered Scenes. In IEEE International Conference on Robotics and Automation, 2021.
12

---

Published as a conference paper at ICLR 2024
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral Cloning from Observation. In International Joint Conference on Artificial Intelligence, 2018.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative Adversarial Imitation from Observation.
In Imitation, Intent, and Interaction (I3) Workshop, 2019a. 2
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent Advances in Imitation Learning from Observation. In IJCAI, 2019b. 2
Hsiang-Chun Chun Wang, Shang-Fu Fu Chen, and Shao-Hua Sun. Diffusion Model-Angmented Behavioral Cloning. arXiv:2302.13335, 2023.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. GMFlow: Learning Optical Flow via Global Matching. In IEEE/CVF Conference Computer Vision and Putlern Recognition, 2022. 4
Lin Yen-Chen Pete Florence, Jonathan T Barron, Tsung- Yi Lin, Alberto Rodriguez, and Phillip Isola. NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields. In IEEE International Conference Robotics and Automation, 2022. 2
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. WortaWWrrd: A Benchmark and Evaluation for Multi-Tash and Meta Reinforcement Learning. In Conference on Robot Learning, 2019.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. XIRL: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, 2022.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
Table of Contents
A Code
14
B Extended Qualitative Results
C Zero-Shot Generalization Real-World Scenes
14
D Extended Experiments
15
D.1 Object mask with segmentation models
15
BC with more data
D.3 Object Mask as Proxy for Action
16
D.4 Quality Analysis on Synthesized Videos
16
E
Additional Ablation Studies
16
First-Frame Conditioning
F.2 Text Encoder.
17
E.3
Factorized Spatial- Temporal Convolution
17
F
Model Architecture and Training Detail
17
F.I
Text Encoder...
F2 Video Diffusson Model
G Hardware and complexity analysis
18
G.1
Training cost
18
G.2 Inference cost
18
G.3 Replanning cost
18
G.4 Improving Inference Efficiency with Denoising Diffusion Implicit Models
19
H Details on Experimental Setup
19
H.l Meta-Worla Experimental Setup
19
ITHOR Experimental Setup
20
II.3
Franka Emika Panda Arm Experimental Setup
21
A
CODE
The code for reproducing our results is included Avoeebasee.VDCC directory the attached supplementary zzip file.
B EXTENDED QUALITATIVE RESULTS
Our supplementary website presents additional qualitative results, including
Synthesized Videos: Meta-World, iTHOR, Visual Pusher, and Bridge.
Task Execution Videos: Meta- World, iTHOR, Visual Pusher, and the real-world Franka Emika Panda tasks.
C ZERO-SHOT GENERALIZATION ON REAL- WORLD SCENES
While most tasks in the Bridge data were recorded in toy kitchens, we found that the video diffusion model trained this dataset already can generalize to complex real-world kitchen scenarios, produc- ing reasonable vidcos given RGB images and textual task descriptions. Examples the synthesized videos can be found supplementary website.
14

---

Published as a conference paper at ICLR 2024
Successful mask prediction
Input text: small faucet handle
Input text: gray door
Failed mask prediction
Input text: small faucet handle
Input text: gray door
Figure Object Mask with Segmentation models. Successful failed object masks extracted by Language Segment Anything.
D EXTENDED EXPERIMENTS
D. OBJECT MASK WITH SEGMENTATION MODELS
We experimented with utilizing existing segmentation methods to extract the object mask for our action regression algorithm. We employed Language Segment-Anything (Medeiros, 2023), which is based on GroundingDINO (Liu et al., 2023) and Segment-Anything (Kirillov et al., 2023). The experiment was conducted in Meta-Workd setting, using the predicted object mask instead of the GT object mask. In this setup, the achieved success rate averaged over all 11 tasks is 34.5%, which 8.6% lower than the success rate using the GT object mask (43.1%). The performance drop is attributed to incorrect object masks produced by the object segmentation model. Qualitative object segmentation results are presented Figure 11.
D.2 BC WITH MORE DATA
We experimented with training plain BC with more data. The results are reported below.
Door-Open Door-Close
Basketball
Shelf-Place
Btn-Press
Btn-Presss Press-Top
AVDC-165
72.0%
89.3%
37.3%
18.7%
60.0%
24.0%
BC-165
21.3%
36.0%
0.0%
0.0%
34.7%
12.0%
BC-330
21.3%
65.3%
0.0%
0.0%
45.3%
21.3%
BC-660
61.3%
72.0%
0.0%
1.3%
77.3%
49.3%
BC-1650
96.0%
81.3%
0.0%
0.0%
96.0%
85.3%
Faucet-Close
Faucet-Open
Handle-Press
Hammer
Assembly
Overall
AVDC-165
53.3%
24.0%
81.3%
8.0%
6.7%
43.1%
BC-165
18.7%
17.3%
37.3%
0.0%
1.3%
16.2%
BC-330
44.0%
29.3%
29.3%
2.7%
0.0%
23.5%
BC-660
77.3%
77.3%
62.7%
10.7%
0.0%
44.5%
BC-1650
93.3%
94.7%
86.7%
12.0%
0.0%
58.7%
Table 3: BC with More Data
Table shows that plain BC needs 20 videos per view (in total 660 videos with action labels, compared to AVDC trained with 165 videos without action labesls) to perform similarly to our method AVDC (43.1 1 % overall). That said, BC needs around 4 times videos and action labels than method,
15

---

Published as a conference paper at ICLR 2024
which highlights the efficiency of our proposed method. More importantly, BC still cannot learn most tasks that require grasps (e.g., pick and place, use tools) even with 50 demonstrations per view (BC-1650).
It's important to emphasize that we included the performance BC to calibrate the difficulty of the tasks, and BC has access to action labels that not accessible to our proposed method; therefore, this is not a fair comparison.
D.3 OBJECT MASK AS PROXY FOR ACTION
AVDC have access to object masks only at test time. However, it's also possible to obtain object mask during training time by running the existing object segmentation model on the training dataset. Such extensive use of object mask can serve as proxy for actions. To provide an idea of the performance of using object masks as a proxy for actions, we have conducted experiments with the following setting: We trained a model that takes in segmented object and directly predicts optical flow within the segmentation (without diffusion). Then, we used the same procedure as AVDC to calculate actions. The results presented follows.
Door-Oper
Door-Closs
Basketball
Shelf-Place
Btn-Press
Btn-Press-Top
Object Mask Proxy AVDC (Full)
1.3%
20.0% 89.3%
0.0% 37.3%
0.0% 18.7%
12.0% 60.0%
2.7%
72.0%
24.0%
Faucet-Closs
Faucet-Open
Handle-Press
Hammer
Assembly
Overall
Object Mask Proxy AVDC (Full)
25.3%
9.3% 24.0%
17.3% 81.3%
2.7% 8.0%
0.0% 6.7%
8.2%
53.3%
43.1%
Table 4: Object mask as Proxy for Action
Table shows that our proposed AVDC outperforms the method that predicts the object masks as proxy for the actions.
D.4 QUALITY ANALYSIS SYNTHESIZED VIDEOS
We further quantitatively compared the synthesized videos the ground truth videos regarding PSNR, SSIM, MSE, and LPIPS. Specifically, we synthesized videos with our trained Meta-Worke video model given unseen first frames (Unseen initial configurations). and compared every synthesized video frame to the corresponding ground truth video frame. We report average PSNR, SSIM, MSE, and LPIPS (AlexNet) scores over 15 videos for each view, totaling 15*3*1 1=495 videos being evaluated. Also, we report the scores comparing the last frames of synthesized videos and ground truth videos. The following table summarizes the result.
PSNR
SSIM
MSE
LPIPS L
Last Frame
25.46
0.8920
0.0050
0.0525
Whole Video
25.02
0.8847
0.0057
0.0557
Table Quantative Quality Analysis on Synthesized Videos
Table 5 shows that proposed video diffusion model can reliably synthesize videos of task execution.
E ADDITIONAL ABLATION STUDIES
E. FIRST-FRAME CONDITIONING
To study the effectiveness of our first-frame conditioning strategy, we conducted quantitative experiment that compares our RGB-chhanel-wiis concatenating strategy with trivial baseline: frame-wis wise concatenate. which concatenates the input frame before the first frame of the noisy video. We calculated the mean squared error (MSE) between the last frame the ground video and the last frame of the synthesized video to evaluate the quality of synthesized videos. We show the results
16

---

Published as a conference paper at ICLR 2024
cat C+P (Ours)
cat C+CLIP
Mean Squared Error (MSE)
0024
cat c+T5 base
0.200
Training Steps
Figure 12: Additional Ablation Studies on the First Frame Condition Strategy and Text Encoders. We ablate the first frame conditioning strategy and compare the performance of different text encoders. Specifically, we calculate the MSE between last frame ground truth video and last frame a synthesized video. cat.c (Ours): the first frame concatenated with noisy video in RGB dimension, which is our proposed method, cat_t: the first frame is concatenated with noisy video in time dimension. CLIP: CLIP text encoder (63M). T5: T5 base encoder (1 10M).
Figure 12. Each data point is an average MSE calculated with 4000 samples of video generation, and the error bar shows the standard error of MSEs. Our method (cat.c) consistently outperforms frame-wist wise concatenating (cat_t) in the early stages of training on the Bridge dataset. Some qualitative generation results be found
supplementary website.
E.2 TEXT ENCODER
We compare the video generation quality of our CLIP-text encoder (63M parameters) with the same model but with T5-base encoder (110M parameters), dubbed AVDC (TS- We used the same pixel-level MSE error as the evaluation metric. Figure 12 shows the result. The difference between the performance of the two text encoders is not significant.
E.3 FACTORIZED
CONVOLUTION
We compare the minimum VRAM requirement to train video model in our setting with and without spatial-temporal convolution. The minimum requirement for training with batch size of 1 in our Meta-World World setting is 13139MB with factorized convolution and 16708MB without it. Our Bridge setting requires 8625MB with factorized convolution and 12606MB without it. In our Bridge experiments, we fine-tuned our video diffusion model on human data on a 3080-Ti GPU (12GB VRAM), which only possible with factorized convolution.
F
MODEL ARCHITECTURE AND TRAINING DETAIL
F.1 TEXT ENCODER
We used fixed pre-trained CLIP-Text encoder for encoding text descriptions. After encoding a text description with the encoder, we employ (Perceiver; Jaegle et al., 2021) as our attention-pooling network to aggregate the output tokens from the CLIP-text encoder into single vector and added it to the time embedding of the diffusion model. This simple condition mechanism has been successful in experiments. We did not cross-attention text inputs throughout this work. The hyperparameters of Perceiver are listed in Table
F.2
VIDEO DIFFUSION MODEL.
For all models, dropout=0, www..head.chamells-33... ccannees==32, train/ifference timesteps=100, training objective=predict v, beta schedule=cosine, loss function=12, min snr gamma=5, learning rate=le-4, ema update steps=10, 10, decay=0.999. We list all other hyperparameters in Table 7.
17

---

Published as a conference paper at ICLR 2024
Parameter
Value
layers
num
num num_head eeaanelss
64
compoutput.cmm output.tokens tokens
64
num, cum_output_em output_tokens.eu pooled
4
max www.xssqiuueee len
512
expansion factor
Table Model parameters for our Perceiver.
Meta-World
iTHOR
Bridge
num parameters
201M (128, 128)
109M
166M
resolution casenehaanels num_res.. block
(64, 64)
(48, 64)
128
128
160
3
attention attention.resolution resolutions
(8, 16)
(4, 8)
(4, 8)
channel_me mmult
(1, 2, 3, 5)
(1, 2, 4)
(1, 2, 4)
bateh_size
16
32
training timesteps
80k
180k
Table 7: Comparison of configuration parameters for Meta-World, iTHOR, and Bridge.
G HARDWARE AND COMPLEXITY ANALYSIS
G.1 TRAINING COST
We train all models on 4 V100 GPUs with 32GB memory each. For environment-speccfie datasets like those used in Meta- World and iTHOR experiments, the training of video policies can be finished within day. As for a much larger dataset like Bridge, have to train for longer to obtain consistent results. Despite the large size of Bridge data compared to the datasets we used in the other two experiments, we can generate high-quality and consistent results with just days of training.
Meta-World about 24 hours of training (165 videos)
iTHOR: about 24 hours of training (240 videos)
Real-world experiment: about 48 hours of pre-training on around 40k Bridge videos and 4 hours of fine-tune tne-tuning on 20 human videos.
G.2 INFERENCE COST
We conducted our experiments (inference) on a machine with an RTX 3080Ti GPU. We provide detailed run time breakdown on Meta-Worlh experiment of each step of our method below.
Cext-Conditioned Video Generation: Synthesizing video of predicted execution the most Lume-consuming step of our method, which takes roughly 10.57 seconds (1.51 seconds per video frame on average).
Flow Prediction: Predicting optical flow between pair of two subsequent frames takes 0.28
seconds on average.
Action Regression from Flows and Depths: Inferring the action from optical flow prediction
1.31 seconds on average.
Action Execution: Running inferred action using the controller in the environment takes
1.53 seconds on average.
G.3 REPLANNING COST
In Meta- World experiments, AVDC used about 18 seconds for each round action planning. Since the maximum number of replans is set 5, the number of action planning rounds within episode varies from to 6. Therefore, the total planning cost ranges from about 18 to 108 seconds on a Nvidia GeForce RTX 3080Ti GPU.
18

---

Published as a conference paper at ICLR 2024
G.4 IMPROVING INFERENCE EFFICIENCY WITH DENOISING DIFFUSION IMPLICIT MODELS
We can incorporate various techniques into our method to improve its inference efficiency. To speed up the video synthesis step, we can progressively distill the diffusion models for faster sampling (Salimans & Ho, 2022). Also, we can leverage lighter-weight optical flow prediction models to increase efficiency. To accelerate action prediction from flows, we can design more sophisticated techniques for sampling and optimizing actions or parallelizing them using GPUs.
This section investigates the possibility of accelerating the sampling process using Denoising Dif- fusion Implicit Models (DDIM; Song et al., 2021). To this end, instead of iterative denoising 100 steps, as reported in the main paper, we have experimented with different numbers of denoising steps (e.g., 25, 10, 5, 3) using DDIM. The qualitative results of synthesized videos are presented on our supplementary website.
We have found that reducing the number of denoising steps to 10 still leads satisfactory generated video quality while resulting in a 10x speedup in video generation. Specifically, the overall mean success rate across tasks in Meta-World with 10-step DDIM 37.5% %, which is competitive with our original method with 100 denoising steps with an overall success rate of 43.1 %. That said, when the task running cime-critical, we can speed up the video generation step by ten times with only 5.6 % drop in task performance.
H DETAILS ON EXPERIMENTAL SETUP
This section describes experimental details, including learning diffusion models, inferring actions, replanning strategies, etc.
H.1 WORLD EXPERIMENTAL SETUP
This section describes the details of the Meta- World experiments.
H.1.1 LEARNING THE DIFFUSION MODEL
We aim to learn a video diffusion model that can synthesize video, showing a robot fulfilling task, from an initial frame and a the task described in natural language. We found that most goal-conditioned manipulation tasks, the final frame of the whole video is often highly correlated to the text description when the current (first) frame is given. In other words, the model can easily synthesize the last frame given the current frame and text description, while the model is often more uncertain about intermediate frames and therefore performs poorly in synthesizing intermediate frames.
To take advantage of this finding, we propose an adaptable frame sampling technique to sample frames from the whole video for training. In particular, we first randomly sample frame from the whole video dataset as the current frame. We then uniformly sample T frames from the current (i.e., initial) frame the final frame from the same video. Then, we use these T' frames (1 current/initial frame, T' 2 intermediate frames, and 1 final frame) to train our video diffusion model. We empirically found that this adaptable frame sampling technique significantly improves the learning efficiency of the video diffusion model, enabling the training to finish within a single day using just GPUs.
H.1.2 CALCULATING OBJECT RIGID TRANSFORMATIONS AND INFERRING ACTIONS
Given the predicted optical flow between each pair of frames of a synthesized video and the initial frame, we aim to infer a robot's actions to follow the synthesized video.
Tracking Object and Determining Contact Point. Since World focuses manipulating objects, we propose to track an object of interest by extracting an object mask. To determine the contact point for the robot to grasp an object, we simply sample N = 500 points from the object mask and compute the centroid of an object as the contact point. Note that more sophisticated methods for determining contact points can be employed to further improve the proposed method.
Calculating Object Rigid Transformation Object and Computing Subgoal. Given the optical flow computed from the synthesized video frames, we can use it to compute the 2D correspondence
19

---

Published as a conference paper at ICLR 2024
between two frames. We use the RANSAC algorithm to find optimal 2D transformation that produces the most inliners from the 2D correspondences. We only use these inliner points for the computations in the current and the following steps for better robustness. We apply our method as described in Section 3.3 to obtain a sequence of 3D rigid transformations. Then, we apply these transformations on the sampled grasp sequentially to obtain sequence of subgoals, indicating how the object should be moved.
Inferring Actions. Given each subgoal, we decide whether use "grasp" action or "push" action to interact with the object by checking if the maximum magnitude of vertical displacement exceeds 10cm based on the heuristic that pick-and and-p tasks usually require the robot lift the object, which produces a vertical displacement; on the other hand, optimal object trajectories of pushing tasks do not exhibit such vertical displacement.
Once we decide if the robot should "grasp" or "push" the object, we determine the robot arm's s action as follows. For the grasp mode, we simply control the robot to take a grasping action (closing the grippers) at the point and then move toward the subgoals. For the push mode, we put the robot arm in a specific direction to the object that allows pushing before moving the robot towards the subgoals. We calculate such direction by extrapolating the line between the push point and the first subgoal more than I 10cm away from the grasp. Here, we consider the sampled grasp location as the contact point for the push action, referred to as the "push point".
H.1.3 REPLANNING STRATEGY
In Detad, we replan (i.e., perform the closed-loop control) by synthesizing a video with the current observed state as the initial frame for the video diffusion model. Then, we use the current object mask and depth information to compute a new sequence of subgoals. Note that we do not re-decide the interaction mode. For the grasp mode, we simply move the gripper toward the new subgoals. For the push mode, we re-initialize the gripper as described above. In specific, when re- re-planning is triggered, we re-imitialize the gripper by 1) syntheiszing a video plan, 2) sampling grasp and calculating subgoals, and 3) calculating the direction for placing the gripper by extrapolating the line between the grasp point and the first subgoal. Once the re-initilization is done, we can start the robot execution. After the re-initialization, we start to move the gripper toward the new subgoals.
H.1.4 DETAILS OF BASELINES
Diffusion Policy Following the original paper, the image observation is encoded with the adapted ResNet-18 with group norm and spatial softmax pooling. For the diffusion backbone model, we experimented with the ID convolutional FiLM U-net architecture proposed in the paper. The backbone model is adapted to take in task embeddings, i.e., the CLIP-Text task embeddings are concatenated with the observation embeddings. The hyperparameters To (the number of past frames used as a condition), Tр (the number of future actions to predict), and Ta (the number of actions to execute before replanning, 0,7a, i=Tp), were set to align with the configurations used in most of the paper's experiments. Specifically, we set (T', Tp T.) (2, 16, 8). We used a batch size of 4096 and evaluated the checkpoints at 15k, 25k and 35k training steps. We picked the checkpoint with the best overall performance as the Diffusion model baseline.
H.2
ITHOR EXPERIMENTAL. SETUP
This section describes the details of the iTHOR experiments.
1.2.1 LEARNING THE DIFFUSION MODEL
We aim to learn a video diffusion model that can synthesize a video that shows an agent navigating to target object in first-person point of view in iTTHR indoor scenes. To sample video segments for training, we first randomly sample a frame from the video demonstration dataset, and then we retrieve T - consecutive future frames from the same video. We do not skip any intermediate frames (i.e., we do not apply the adaptable frame sampling technique used in Meta- (erl-,Wrld), as iTHOR environment uses discrete actions such as moving and rotating for a constant distance or angle. If the number of subsequent frames is less than T 1 in the sampled video, duplicate the last frame to compensate for missing frames. This allows the model to recognize that the target object is found and the agent should stop moving.
20

---

Published as a conference paper at ICLR 2024
H.2.2 CALCULATING SCENE TRANSFORMATIONS AND INFERRING ACTIONS
Tracking Scene and Calculating Scene Transformation. In the navigation setup, instead of tracking the correspondences of a particular object, we track the correspondences of the entire scene. To this end, instead of generating an object mask, we initialize scene mask by thresholding points (magnitude of optical flow > 1). Then, to calculate the scene transformation, we apply similar procedure to the Meta- World experiment for computing the object transformation. However, we do not use the RANSAC algorithm to obtain inliners; instead, at each time step, we simply remove the key points that move out-of-bound (i.e., outside the image) and keep the rest as the inliner points to calculate the scene transformation.
moving
Inferring Actions. Given calculated the scene transformation at each step, we design a procedure to infer action (MoveF orward, Rotat eee ft, Rot RotateRight or Done). We propose to observe an imaginary point located meter in front of the robot. We apply the calculated scene transformation on this imaginary point. We then decide the action based on the translation of this imaginary point before and after applying the transformation. Specifically. if the translation is close to 0 (<< 1mm), since the agent stays still, choose Done. Otherwise, of this imaginary point. If the magnitude of the horizontal displacement is less than 25em, we choose MoveF orward. Otherwise, we select RotateLefe or RotateRi ght depending on the direction of the displacement.
check the horizontal displacement
H.2.3 REPLANNING STRATEGY
In iTHOR, we replan when we lose track of most correspondences from the initial frame. Specifically, if the number of inliners we keep is less than 10% of the original number we sampled, we re-sytthesize a video, predict optical flow, and infer actions.
H.3 REAL-WORLD FRANKA EMIKA PANDA ARM EXPERIMENTAL SETUP
This section describes the details of the real-world Franka Emika Panda arm experiments.
Hardware. Our arrangement comprises a Franka Emika Panda arm and an Intel Realsense D435 RGBD camera mounted at fixed frame relative to the table. The robot arm is equipped with parallel motion two-jaw gripper, and the robot arm is in joint position control mode. Meanwhile, the camera has calibrated intrinsic and extrinsic. Therefore, any object motion predicted in the camera frame can directly transformed into the world frame.
Dataset Collection. After training on the Bridge dataset, we fine-tuned the model with 20 human demonstrations in a real-world tabletop manipulation setting. These videos are collected by humans using their hands to move objects on the table and accomplish tasks. Our object set includes plates, bowls, a few categories of fruits (apples, oranges, bananas, peaches, and mangoes), and utensils such as forks and knives distractors. The task is to pick up fruits from their initial locations and place them in the specified container, a plate or a bowl.
Action Prediction and Execution. In real-world evaluation, assume that the target object can be grasped using a top-grasp and that no re-orientation of the target object needed. Therefore, in order compute the target object poses, first manually specify the segmentation of the object (in principle, it can be done using other object segmentation models, too), extract the corresponding optical flow, and compute the sequence of the object poses. We extract its initial pose (in the first frame) and the target pose (in the last frame), and generate robot arm trajectory using an inverse- kinematics (IK) solver. In practice, we found that since we are using only a small set of tracking points (objects are small in our camera view), the reconstruction of 3D rotations is not robust. This can potentially addressed by leveraging higher-resolution video generative model or simply, different camera configurations.
Failure Mode Analysis In our real-world experiments, we found that our approach failed of the 10 tested trials. We found that 75% of the failures were caused by the wrong plan from the video diffusion model. either picked the wrong object or placed at the wrong target. The other 25% of the failures were caused by the discontinuity of video generation. The generated plan seems to be correct, but the object disappeared in some intermediate frame, which eventually led to the failure.
21


Reviews:
[{"review_id": "pk855MWulF", "reviewer": "Reviewer_QCWJ", "summary": "The goal of this paper is to learn robot policies from action-free video data. The motivation is that there exists a lot of video data, but very little action data. Video prediction methods are often over-dependent on actions but have the benefit of being task agnostic. AVDC aims to solve this challenge by learning a video generation model (via diffusion) on the robot video data. From the generated sequence of images, the optical flow is estimated, which conditioned on some 3D knowledge as well as masks/segmentations of different objects gives an idea of the actions that are taken. The robot actions are then taken. To avoid accumulating errors, AVDC allows for replanning after a few actions. The approach is tested on manipulation (Meta-World) and navigation (iThor) setups, as well as qualitative results on a cross-embodiment visual pusher dataset and some robot arm data. Experiments and ablations find that (1) AVDC outperforms inverse dynamics+video prediction and BC baselines (2) all individual components are important.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important problem of learning from action-free videos \n- The method, to my knowledge is novel\n- The approach significantly outperforms baselines on many different tasks \n- The ablations are well analyzed \n- The paper is easy to follow and well written", "weaknesses": "- I think one of the main limitations is the setting: AVDC needs videos of robots performing the task. I believe this is a contrived setting as it is very likely that if video + 3D information is available, then this was a robot demonstration, and one can just collect action data. To me, it is unclear how this approach will scale beyond robot data. \n\n- I am concerned by the reported results for the BC baseline. Due to action data being available, as well as the robot data being in-domain for the task a simple BC or kNN baseline should work very well. There are many cases where the results are < 5% success. This should be addressed. I would be willing to increase my score if this weakness is addressed. \n\n- AVDC relies on object/robot masks - a simple baseline would be to use those as a proxy for the actions. One could get pseudo action labels from the videos and train a policy. \n\n- AVDC assumes that all objects are going to be directly manipulated by the robot directly but this is not the when one object as a tool. \n\n- Navigation approaches have many action free baselines which should be explored as well\n\n- It would be good to see real world experiments\n\n- It would be good to have more of an analysis on the quality of the video prediction model. I suspect it has a", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WWTeOBq2O3", "reviewer": "Reviewer_cxfp", "summary": "The paper presents an approach for learning video-based policies in robot manipulation settings. The key benefit of the approach is training on actionless video data across human and robot embodiments. The method, termed Actions from Video Dense Correspondences (AVDC), consists of three stages: (1) diffusion-based video prediction given a text-based goal and starting image, (2) optical flow prediction creating dense correspondences, and (3) executed on a robot platform using off-the-shelf inverse kinematics and motion planners. AVDC uses the ability to project a 3D point onto the image plane both from depth and optical flow to compute the transformation of rigid objects across the predicted video frames. These transformations allow AVDC to infer actions in the environment. Then off-the-shelf robotics primitives can be used to enact the planned trajectory. The approach is benchmarked on the Meta-World and iTHOR simulation platforms and on a real-world robot platform, outperforming the considered baselines.", "questions": "1. In the related work, you mention that RL based methods often have to interact with the environment. However, offline RL-based methods avoid this issue (e.g., [A]). What is the downside of such approaches compared to the proposed method?\n2. Was the choice of the factorized spatial-temporal ResNet block ablated?\n3. I did not quite understand in Sec. 3.3 'Predict object-centric motion', what happens to achieve subsequent subgoals after the first grasp-contact point is reached. Do you pick the next one in the subsequent predicted video frame?\n4. In the replanning strategy, why would a smaller robot movement necessarily be indicative of failure? What if the inaccuracy in compounding error results in large, but inaccurate robot movements?\n5. Is there a reason not to use a receding horizon-style replanning strategy as in [B]?\n6. Do you have a sense as to why AVDC (Full) underperformed in the 'btn-press-top' task in Table 1?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The general problem of making use of actionless human video data is of interest and importance to the research community.\n* The problem is well-motivated and the literature review does a good job of contextualizing the paper in prior work.\n* The paper is strong, well-written and easy to follow.\n* The use of geometry to reconstruct the transformation of the predicted objects (stationary camera) or embodiment (moving camera) which can be derived simultaneously from the optical flow and depth camera during deployment is clever. This allows the training data for the video prediction and optical flow not to require depth, with depth only being necessary during deployment. The transformations of either the objects or the embodiment then can be used in conjunction with off-the-shelf inverse kinematics, motion planners, grasp point predictors, etc. This also allows for learning from human videos and then zero-shot deploying to the robot, which is very impressive.\n* The figures are informative and effectively illustrate the benefits of the proposed approach.\n* The experiments consider both simulation and real robot evaluation, as well as an ablation study, demonstrating AVDC's superior performance as compared to the considered baselines and support for AVDC's design choices. In particular, I appreciated the discussion and later the results for why not to directly predict the optical flow without the intermediate step of video prediction.\n* The discussion did a good job of describing the weaknesses and failure modes of the proposed method.", "weaknesses": "* The literature review is missing a number of relevant works.\n  * V-PTR: similar high-level motivation of using video-based, prediction-focused pre-training and then action-based finetuning. This should have likely served as a baseline for the proposed method. \n  * [A] Bhateja, Chethan, et al. \"Robotic Offline RL from Internet Videos via Value-Function Pre-Training.\" arXiv preprint arXiv:2309.13041 (2023).\n  * Diffusion policy: diffusion policy has shown very good results in terms of multi-task, low-data regime performance. \n    * [B] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" arXiv preprint arXiv:2303.04137 (2023). \n    * [C] Ha, Huy, Pete Florence, and Shuran Song. \"Scaling up and distilling down: Language-guided robot skill acquisition.\" arXiv preprint arXiv:2307.14535 (2023).\n* In particular, my biggest concern with the paper is the lack of comparison to a strong BC baseline. As the AVDC method uses diffusion to predict images, it seems natural to baseline against a diffusion policy (e.g., [B, C]). R3M is a fairly old representation at this point (e.g., Voltron [D] would be a better representation) and the simple MLP-based BC policy would strongly underperform diffusion policy. This is confirmed by, for example, the very poor baseline performance in Tables 1 and 2. In Sec. 4.3, it is mentioned that since 'R3M is pretrained on robot manipulation tasks ... it might not be suitable for visual navigation tasks'. Something like [E] could be a better baseline here.\n  * [D] Karamcheti, Siddharth, et al. \"Language-driven representation learning for robotics.\" arXiv preprint arXiv:2302.12766 (2023).\n  * [E] Shah, Dhruv, et al. \"ViNT: A Foundation Model for Visual Navigation.\" arXiv preprint arXiv:2306.14846 (2023).\n* In Sec. 4.5 'Results', the paper states that Fig. 10 presents screenshots of robot trajectories, but I believe that is Fig. 9? Fig. 10 shows human predicted trajectories.\n\nSome typos and points of confusion are listed below:\n1. Page 3 - 'Unipi'.\n2. Sec. 4.1:  'compare AVDC to its [variants] that also predict dense correspondence'.\n3. Sec. 4.2: 'maximum number of planning affects' -> 'maximum number of replanning steps affects'.\n\n**Post-rebuttal: Most of my concerns have been addressed! I am raising my score as such.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SwtG9uXKJ6", "reviewer": "Reviewer_itAQ", "summary": "The paper introduces a method for constructing a video-based robot policy, capable of performing diverse tasks across different environments. This approach doesn't require action annotations but uses images for a task-agnostic representation. Text is employed for specifying robot goals. By synthesizing videos to predict robot actions and employing dense correspondences between frames, the model infers actions without explicit training labels. It can leverage the large-scale RGB videos on the internet for training, and use this knowledge for robotic manipulation. The paper showcases the effectiveness of this approach in tabletop manipulation and navigation tasks and also provides an open-source framework for efficient video modeling.", "questions": "1. In the appendix H.1.2, the authors say \"We calculate such direction by extrapolating the line between the grasp point and the\nfirst subgoal more than 10cm away from the grasp\" Should it be push point?\n2. In Sec. H.1.3, \"For the push mode, we re-initialize the gripper as described above \" is not clear. What does the re-initialization refer to?\n3. When learning the diffusion model for the IThor environment, why not apply the adaptable frame sampling technique  in this case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed a new correspondence based method to obtain robot action in forecasted robot videos. It proves that a latent dynamic model is not needed if the forecasted video has good quality.\n2. The authors proposed a new method to generate future videos using a diffusion model, which achieves efficient training. It provides a promising toolbox for the community.\n3. The method is evaluated on two tasks, table-top manipulaion and in-door navigation, demonstrating its effectiveness in different domains.\n4. The paper is well-written and solid.", "weaknesses": "1. The selected robot tasks are relatively toy, and the potential of such kind of video prediction method is not evaluated. However, this is not the weakness of this paper, but a common practice for video prediction based robot control.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1Gl2d3GzV", "reviewer": "Reviewer_Hwk5", "summary": "The paper presents a method called Actions from Video Dense Correspondences (AVDC) that learns to perform robotic tasks without ever accessing labels of robot actions. The method first trains a text-conditioned diffusion video prediction model on videos of robot data. It then computes optical flows for adjacent frames of the prediction, and by segmenting out the object of interest and finding a rigid body transformation that best fits the corresponding points, computes an object (manipulation) or robot (navigation) transformation that enables the robot to follow the plan. Experiments are conducted on several environments: simulated meta-world and iTHOR tasks, visual pusher for evaluating learning from a human embodiment, and bridge data/a real Panda arm.", "questions": "- The video generation performance is seemingly quite good even when very few training trajectories are provided (just 165 videos for Meta-World). Can you comment about how overfitting can be avoided or provide some intuition?\n- The performance of the UniPi baseline is surprisingly poor. Could you please provide an explanation for the common failure modes or visualizations? Same goes for the BC baselines. Is this due to the low amount of data provided, thus causing action prediction models to overfit? If so, would it be possible to report results with greater number of demonstrations (like 50, or even 15, rather than 5)?\n- Are the camera poses that are used for evaluating the policies the same as the ones in the training data? I assume that they are but it would be good to have confirmation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Learning from action-free videos is an important and challenging problem, which could enable data-driven robotics to access scale through sources like Youtube.\n- The proposed method makes sensible design decisions, including the video model and correspondence computation strategy. \n- The evaluation is very thorough in terms of the number of environments considered, and AVDC appears to yield consistent performance gains.\n- The authors have committed to making their video model, which requires significantly fewer resources than prior works, open source. I think this is also a valuable contribution (although less related to the main message of the work).\n- The presentation is quite good, the writing is clear, and main ideas are communicated directly.", "weaknesses": "- The most apparent weakness of this work is that not all tasks that a robot might want to solve can be solved by a trajectory of target object poses. For example, it is unclear how to plan a task that would require the robot to use another object as a tool. It is also unclear how deformable objects could be handled. There also may be tasks such as pressing a button on a microwave, which do not involve the microwave moving, but require a particular amount of force to be applied, for which this may not be applicable.\n- The work “Zero-Shot Robot Manipulation from Passive Human Videos” by Bharadhwaj et al. presents very similar (although not identical) ideas and is not discussed in the prior work or cited. Could the authors discuss and ideally perform a comparison to some of the ideas in that work?\n- Related to the previous point, the ideas presented in this paper are not entirely novel. However, I believe that this particular combination/instantiation of them, as well as the evaluation and exploration of them that is provided, is a valuable contribution to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)