Grounding Video Models to Actions through Goal Conditioned Exploration

## Abstract

Abstract Large video models, pretrained on massive amounts of Internet video, provide a rich source of physical knowledge about the dynamics and motions of objects and tasks.
However, video models are not grounded in the embodiment of an agent, and do not describe how to actuate the world to reach the visual states depicted in a video.
To tackle this problem, current methods use a separate vision-based inverse dynamic model trained on embodiment-specific data to map image states to actions.
Gathering data to train such a model is often expensive and challenging, and this model is limited to visual settings similar to the ones in which data are available.
In this paper, we investigate how to directly ground video models to continuous actions through self-exploration in the embodied environment – using generated video states as visual goals for exploration.
We propose a framework that uses trajectory level action generation in combination with video guidance to
enable an agent to solve complex tasks without any external supervision, e.g., rewards, action labels, or segmentation masks.
We validate the proposed approach on 8 tasks in Libero, 6 tasks in MetaWorld, 4 tasks in Calvin, and 12 tasks in iThor Visual Navigation.
We show how our approach is on par with or even surpasses multiple behavior cloning baselines trained on expert demonstrations while without requiring any action annotations.

### 1 Introduction

Large video models (Brooks et al., ; Girdhar et al., ; Ho et al., ) trained on a massive amount of Internet video data capture rich information about the visual dynamics and semantics of the world for physical decision-making. Such models are able to provide information on how to accomplish tasks, allowing them to parameterize policies for solving many tasks (Du et al., ). They are further able to serve as visual simulators of the world, allowing simulation of the visual state after a sequence of actions (Brooks et al., ; Yang et al., ), and enabling visual planning to solve long-horizon tasks (Du et al., ).

However, directly applying video models zero-shot for physical decision-making is challenging due to embodiment grounding. While generated videos provide a rich set of visual goals for solving new tasks, they do not explicitly provide actionable information on how to reach each visual goal. To ground video models to actions, existing work has relied on training an inverse dynamics model or goal-conditioned policy on a set of demonstrations from the environment (Black et al., ; Du et al., ; ). Such an approach first requires demonstrations to be gathered in the target environment and embodiment of interest, which demands human labor or specific engineering (e.g. teleoperation or scripted policy). In addition, the learned policies may not generalize well to areas in an environment that are out-of-distribution of the training data.

Recently,  Ko et al. () proposes an approach to directly regress actions from video models, without requiring any action annotations. In  Ko et al. (), optical flow between synthesized video frames is computed and used, in combination with a depth map of the first image, to compute a rigid transform of objects in the environment. Robot actions are then inferred by solving for actions that can apply the specified rigid transform on an object. While such an approach is effective on a set of evaluated environments, it is limited in action resolution as the inferred object transforms can be imprecise due to both inaccurate optical flow and depth, leading to a relatively low success rate in evaluated environments (Ko et al., ). In addition, it is difficult to apply this approach to many robotic manipulation settings such as deformable object manipulation, where there are no explicit object transforms to compute.

Figure: Figure 1: Grounding Video Models to Actions. Our approach learns to ground a large pretrained video model into continuous actions through goal-directed exploration in the environment. Given a synthesized video, a goal-conditioned policy attempts to reach each visual goal in the video, with data in the resulting real-world execution saved in a replay buffer to train the goal-conditioned policy.
Refer to caption: x1.png

We propose an alternative manner to directly ground a video model to actions without using annotated demonstrations. In our approach, we learn a goal-conditioned policy, which predicts the actions to reach each synthesized frame in a video. We learn the policy in an online manner, where given a specified task, we use each intermediate synthesized frame as a visual goal for a goal-conditioned policy from which we obtain a sequence of actions to execute in an environment. We then use the image observations obtained from execution in the environment as ground-truth data to train our goal-conditioned policy. We illustrate our approach in Figure 1.

In practice, directly using synthesized images as goals for exploration often leads to insufficient exploration. Agents often get stuck in particular parts of an environment, preventing the construction of a robust goal-conditioned policy. To further improve exploration, we propose to generate chunks of actions to execute in an environment given a single visual state. By synthesizing and executing a chunk of actions we can explore the environment in a more directed manner, enabling us to achieve a more diverse set of states. We further intermix goal-conditioned exploration with random exploration to further improve exploration.

Overall, our approach has three contributions: (1) We propose goal-conditioned exploration as an approach to ground video models to continuous actions. (2) We propose a set of methods for effective exploration in the environment, using a combination of chunked action prediction for temporally coherent exploration along with periodic randomized actions for robust state coverage.
(3) We illustrate the efficacy of our approach on a set of simulated manipulation and navigation environments.

### 2 Related Work

###### Video Model in Decision making

A large body of recent work has explored how video models can be used in decision making (Yang et al., ; McCarthy et al., ). Prior work has explored how video models can act as reward functions (Escontrela et al., ; Huang et al., ; Chen et al., ), representation learning (Seo et al., ; Wu et al., ; ; Yang et al., ), policies (Ajay et al., ; Du et al., ; Liang et al., ), dynamics models (Yang et al., ; Du et al., ; Zhou et al., ; Brooks et al., ; Rybkin et al., ; Mendonca et al., ), and environments (Bruce et al., ).
Our work explores that given video generations, how we can learn a policy and infer the actions to execute in an environment without any action labels, while existing works (Baker et al., ; Bharadhwaj et al., ; Black et al., ; Wen et al., ; Zhou et al., ; Wang et al., ) requires domain specific action data.
Most similar to our work is AVDC (Ko et al., ), which uses rigid object transformations
computed by optical flow between video frames as an approach to extract actions from generated videos. We propose an alternative unsupervised approach to ground video models to continuous action by leveraging video-guided goal-conditioned exploration to learn a goal-conditioned policy.

###### Learning from Demonstration without Actions.

A flurry of work has studied the problem of robot learning from demonstrations without actions.
One line of work studies the problem of extrapolating the control actions assuming that the expert state trajectories are provided (Torabi et al., ; Radosavovic et al., ; Li et al., ), though collecting such ground-truth state-level data is expensive and hard to scale.
Some recent work explores learning from image/video robotic data without actions (Seo et al., ; Wu et al., ; Mendonca et al., ; Ma et al., ; Wang et al., ; Ma et al., ; Schmeckpeper et al., ), either by constructing a world model, reward model, or representation model. However, these methods usually need additional finetuning on data or rewards from specific downstream environments.
By contrast, our method uses action-free demonstration videos to train a video generative model and leverages the generated frames as goals to learn a goal-conditioned policy to complete a task.
As a result, our policy learning requires neither action labels nor environment awards which can be challenging to obtain.

Robotic Skill Exploration.
Typical robot skill exploration is formulated as an RL problem (Haarnoja et al., ; Hafner et al., ) and assumes some form of environment rewards, but the design of reward functions is highly task dependent and demands human labor.
To this end, some recent work explores robotic skill exploration without any rewards.
One typical class of methods is developed upon computing intrinsic exploration rewards to promote rare state visit. These methods can be based on
prediction error maximization (Pathak et al., ; Henaff, ; Shyam et al., ),
disagreement of an ensemble of world models (Hu et al., ; Sekar et al., ; Sancaktar et al., ),
entropy maximization (Pong et al., ; Pitis et al., ; Jain et al., ; Eysenbach et al., ),
counting (Bellemare et al., ),
or relabeling (Ghosh et al., ).
However, discrepancy, counting, or relabeling based methods are limited to simple environments, short-horizon tasks, or low-dimensional state-space.
We propose to use video models for direct exploration, as large pretrained video models are a rich source of task-specific information.
With efficient guidance from the video model,
we show that our method is able to collect high-quality data from the environment and accomplish challenging long-horizon tasks conditioned on language instruction at test time.

### 3 Method

In this section, we describe our method to ground video models to actions by unsupervised exploration in the environments.
First, in Section 3.1, we describe the pipeline of policy execution conditioned on video frame and hindsight relabeling via environment rollouts.
Next, in Section 3.2, we introduce a periodic random action bootstrapping technique to secure the quality of video-guided exploration.
Finally, in Section 3.3, we propose a chunk-level action prediction technique to further enhance the coverage, stability, and accuracy of the goal-conditioned policy.
The pseudocode of our method is provided in Algorithm 1.

#### 3.1 Learning Goal-Conditioned Policies through Video-Guidance and Hindsight Relabeling

A key challenge in unsupervised skill exploration is that
the possible underlying environment states are enormous, making it difficult for an agent to discover and be trained on every valid state, especially in the high-dimensional visual domain.
Many relevant states for downstream task completion, such as stacking blocks on top of each other or opening a cabinet, require a very precise sequence of actions to obtain, which is unlikely to happen from random exploration.

Video models have emerged as a powerful source of prior knowledge about the world, providing rich information about how to complete various tasks from large-scale internet data. We leverage the knowledge contained in these models to help guide our exploration in an environment to solve new tasks.
To this end, we propose a novel method that uses a pre-trained video model $f_{\theta}(x_{\text{start}},c)$
to guide the exploration and shrink the search space only to task-relevant states
($x_{\text{start}}$ denote the initial observation and $c$ denote the corresponding task description).
This concurrently benefits sides: the goal-conditioned policy obtains task-relevant goals so as can perform efficient exploration centered around the task-relevant state space; on the other hand, the underlying information from the video model is extracted to end effectors and enables effective decision-making and control in embodied agents.
Specifically, during exploration, we first leverage the video model to generate a sequence of images based on the given image observation $x_{\text{start}}$ and language task description $c$,

$$ $\mathtt{pred_v}=f_{\theta}(x_{\text{start}},c),\>\hskip 5.0pt\text{where} \hskip 5.0ptx_{\text{start}}\sim\mathcal{O}\hskip 5.0pt\text{and}\hskip 5.0ptc \sim\mathcal{T}$ (1) $$

where $\mathcal{O}$ is the observation space and $\mathcal{T}$ is the task space.
We then execute the actions predicted by the goal-conditioned policy $a^{\text{pred}}=\pi(a|x_{\text{start}},x_{\text{goal}})$ in the environment, where the goal $x_{\text{goal}}$ is set to each predicted video frame $\mathtt{pred_v}_{i}$ sequentially.
Hence, we can obtain an episode of image action pairs $(x_{1:t}^{\text{env}},a_{1:t}^{\text{pred}})$ of length $t$ from the environment and these rollout data will be added to a replay buffer $R$ for policy training.

At the beginning of training, these data might not necessarily reach the exact goals given by the video model, but are still effective to indicate the task-specific region, since the rollout results of $a_{1:t}^{\text{pred}}$ are relabeled and can reflect the ground-truth environment dynamics. Empirically, we observe the proposed video-guided exploration scheme is very effective, and we visualize the curve of number of success versus total number of video-guided rollouts in Figure 8.
Our overall training objective is

$$ $\max_{\phi}\>\>\mathbb{E}_{(x_{i:i+h},a_{i:i+h})\sim R}\big{[}\log\pi_{\phi}(a _{i:i+h}|x_{i},x_{i+h})\big{]}$ (2) $$

where $i$ is a randomly sampled temporal index inside a rollout episode, $h$ denotes the horizon of the policy, and $\phi$ represents the parameters of the goal-conditioned policy. Note that provided a video model $f_{\theta}$ trained on internet scale data, the video predictions can be easily generalized to a broad observation space $\mathcal{O}$ and task space $\mathcal{T}$, enabling the proposed guidance scheme directly applied to various unseen scenarios.

#### 3.2 Periodic Random Action Bootstrapping

When a goal-conditioned policy is initialized from scratch, it is unable to effectively process the frames provided by a video model and use them to guide exploration, as it is unable to process input frames.
Empirically, we found that a randomly initialized policy would often instead preferentially output particular actions (ı.e. only move up) independent of goals given by the video model.

This significantly compromises the exploration because the policy will not explore the task-related states as we expect.
More importantly, though we can obtain ground-truth environment dynamics via hindsight relabeling, the actions in the replay buffer $R$ are output from the scratch policy itself. This might result in an undesirable loop where only the previous outputs are used as the ground-truth and these actions are irrelevant to completing the task.

To this end, inspired by random action sampling in RL setting (Sutton, ), we propose a novel periodic random action bootstrapping method
for grounding video model to actions.
Specifically, we first conduct random action exploration and append the resulting data to the replay buffer $R$ before the training starts,
and periodically conduct extra random exploration during training. The process can be denoted by

$$ $R\>\>\leftarrow\>\>a_{1:t^{r}}^{i_{r}}\sim\mathtt{Uniform}\big{[}a_{\text{low} },a_{\text{high}}\big{]}\quad\text{for}\>\>i^{r}\>\>\text{in}\>\>[1,2,...,n_{r}]$ (3) $$

where $n_{r}$ denotes the number of random action episodes, $t^{r}$ denotes the length of a random action episode, $a_{\text{low}}$ and $a_{\text{high}}$ are the action limits.
This proposed bootstrapping method can enhance the exploration in two ways: the initial random actions serve as the basic world dynamics information which enables the policy to reach the vicinity of goal states specified by video frames, ensuring effective video-guided exploration; the periodic extra random exploration can further expand the agent’s discovered state space and stabilize policy training.
Please see Section 4.3 for ablation studies.

Figure: Algorithm 1 Grounding Video Models to Actions

#### 3.3 Chunk Level Action Prediction and Exploration

In current robot exploration frameworks, the policy usually predicts a single action as output and explores the environment with the predicted action. However, single-action prediction can hinder diverse exploration, as the agent can easily stuck in a single state, outputting the same repeated action.
In addition, trajectories generated in such manner are often incoherent, as repeatedly sampled actions may reverse and undo actions generated previously in the trajectory.
Finally, exploration with single action prediction requires one model forward pass for each action, resulting in a larger computational burden and higher latency.

To tackle these issues, we propose to predict a sequence of actions with horizon $h$ and explore the environments using the predicted chunks of actions.
On one hand, by modeling behavior over a longer horizon, chunk-level action prediction can encourage more coherent action sequences and
mitigate myopic actions and compounding errors, especially when the temporal distance to the goal is large.
On the other hand, exploration with action chunks can in turn collect more coherent data from the environment, further facilitating the policy to capture the underlying environment dynamics.
In addition, we utilize chunk-level actions during random exploration. Specifically, we sample an action mean $a^{m}$ from a uniform distribution, and based on $a^{m}$, we sample a chunk of actions $a^{c}$ of length $l_{c}$ from Gaussian distribution, where the $i$-th action is represented as $a_{i}^{c}\sim\mathcal{N}(a^{m},\sigma)$.
This ensures consistent exploration and avoids the zero-mean random action issue that confines the agent to a small vicinity near the start state.

Recent works have also employed action chunking (Bharadhwaj et al., ; Zhao et al., ) in behavior cloning.
However, our application is different and we propose to use action chunking to enable more effective unsupervised exploration. We illustrate how, in combination with video-guided exploration, this action chunking allows for more coherent exploration, as well as enabling models to make consistent plans to achieve video goals.
In Section 4.3, we provide a study comparing chunk-level prediction versus single action prediction in the Libero environment and empirically show that the chunk-level design can substantially improve the resulting goal-reaching policy.

Figure: Figure 2: Environment Demonstrations. We evaluate our method on three robotic manipulation environments: Libero, Meta-World, Calvin, and one visual navigation environment: iThor, with a total of 30 tasks.
Refer to caption: x2.png

### 4 Experiments

We present our experiment results across four simulated environments shown in Figure 2.
In Section 4.1, we describe our results on three robotic manipulation environments: 8 tasks on Libero (Liu et al., ), 6 tasks on MetaWorld (Yu et al., ), and 4 tasks on Calvin (Mees et al., ).
Following this, in Section 4.2, we show evaluation results on 4 different scenes and 12 targets on iTHOR (Kolve et al., ) visual navigation environment.
Finally, in Section 4.3, we present ablation studies on the proposed chunk level action prediction and random action bootstrapping methods.
We provide more experiment results in Appendix A and B.
We use the same demonstration data to train the baseline methods and our video model, except that training the video model only requires the image sequences of demonstrations, while most baseline methods need the corresponding action annotations (highlighted by an asterisk).
In evaluation, for manipulation tasks, the initial robot state and object positions are randomized; for visual navigation tasks, we randomize the robot start position.

###### Implementation.

For the video model, the inputs are the observation image and the task description. We follow the lightweight video model architecture introduced in (Ko et al., ) and train one video model from scratch for each environment.
We deem that finetuning large text-conditioned video models (Yang et al., ; Chen et al., ; Saharia et al., ) or designing task-specific video model might be an interesting future research direction.
For the goal-conditioned policy, we implement it with a CNN-based Diffusion Policy (Chi et al., ), which takes as input the observation image and the goal image and outputs a chunk of actions.
For detailed implementation of our method and each baseline, please refer to Appendix D.

**Table 1: Quantitative results on 8 tasks of two different scenes in Libero. Note that methods marked with an asterisk ‘*’ require ground-truth action demonstrations to train, while other methods do not. Ours w/ SuSIE uses an image-editing model to generate subgoals guidance while Ours uses a video model.**
|  | put-red-mug-left | put-red-mug-right | put-white-mug-left | put-Y/W-mug-right | Overall |
| --- | --- | --- | --- | --- | --- |
| BC* | 8.8$\pm\mathtt{5.3}$ | 15.2$\pm\mathtt{7.8}$ | 32.0$\pm\mathtt{12.9}$ | 21.6$\pm\mathtt{12.3}$ | 19.4$\pm\mathtt{9.6}$ |
| GCBC* | 2.4$\pm\mathtt{2.0}$ | 0.8$\pm\mathtt{1.6}$ | 16.0$\pm\mathtt{7.2}$ | 7.2$\pm\mathtt{5.3}$ | 6.6$\pm\mathtt{4.0}$ |
| DP BC* | 33.6$\pm\mathtt{3.2}$ | 33.6$\pm\mathtt{8.2}$ | 59.2$\pm\mathtt{7.8}$ | 57.6$\pm\mathtt{5.4}$ | 46.0$\pm\mathtt{6.2}$ |
| DP GCBC* | 24.8$\pm\mathtt{4.7}$ | 22.4$\pm\mathtt{7.4}$ | 16.0$\pm\mathtt{8.8}$ | 3.2$\pm\mathtt{3.0}$ | 16.6$\pm\mathtt{6.0}$ |
| SuSIE* | 18.4$\pm\mathtt{2.0}$ | 32.0$\pm\mathtt{8.4}$ | 43.2$\pm\mathtt{4.7}$ | 25.6$\pm\mathtt{11.5}$ | 29.8$\pm\mathtt{6.6}$ |
| AVDC | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ |
| Ours w/ SuSIE | 23.2$\pm\mathtt{3.0}$ | 60.0$\pm\mathtt{6.7}$ | 68.8$\pm\mathtt{4.7}$ | 67.2$\pm\mathtt{8.9}$ | 54.8$\pm\mathtt{5.8}$ |
| Ours | 38.4$\pm\mathtt{15.3}$ | 40.8$\pm\mathtt{7.8}$ | 51.2$\pm\mathtt{3.9}$ | 38.4$\pm\mathtt{8.6}$ | 42.2$\pm\mathtt{8.9}$ |

Figure: Figure 3: Qualitative Results of task ‘put the yellow and white mug to the right plate’ in Libero environment. The start states of the robot and objects are randomized in test time. Only a subset of the predicted video frames are shown due to the space limit. Our goal-conditioned policy shown in the bottom right is able to follow the video prediction and finish the task. BC cannot accurately locate the target while AVDC can move to the mug but without the skill of grasping concave objects.
Refer to caption: x3.png

#### 4.1 Manipulation

In this section, we aim to evaluate the goal-conditioned policy learned by the proposed unsupervised exploration in tabletop manipulation environments with continuous action space.
To better understand the capability of the method, we investigate multi-task policy learning in Libero and Calvin, and single-task policy learning in Meta-world.
Note that methods highlighted by an asterisk require ground-truth action labels to train, while our method only requires image observations for training the video model with the potential of zero-shot generalization if trained on internet-scale data.

Libero (Liu et al., ) is a tabletop simulation of a Franka Robot, which features several dexterous manipulation tasks. For each task in Libero, the agent is required to achieve the final state described in a corresponding sentence, which identifies the target object and task completion state.
The action space consists of the delta position and orientation of the end effector and the applied force on the gripper, resulting in a total dimension of 7.
To improve the efficiency of exploration, we further add an exploration primitive for grasping, which we discuss in detail in Appendix D.2.

We include 8 tasks from two scenes in Libero as the testbed.
In this environment, we aim to evaluate the multi-task learning capability of our method, hence we only train one policy model for all 8 tasks. The video model is trained on the visual image sequences of the demonstrations provided in Libero, where we use 20 episodes per task, thereby 160 demonstrations in total.
We train BC, GCBC, SuSIE (Black et al., ) on the same visual image sequences but with expert actions corresponding to each image.
Following the setting in AVDC, we provide it with the segmentation masks of the target objects, while our method does not require such privileged information.

We present the quantitative results in Table 1 and qualitative results in Figure 3 and 6.
Across 8 tasks, our method is able to outperform baselines by a margin, though without any access to expert action data.
We observe that BC-based methods tend to memorize and overfit to the training data, where they fail to locate the target objects, while our method is more robust in test-time.
In addition, though knowing the exact positions of target objects in test-time, AVDC is unable to achieve any meaningful results, probably because that AVDC uses hard-coded action primitives and planning procedural, making it difficult to generalize to more complex manipulation tasks, for example, grasping concave objects such as mugs.
Note that our method can also be integrated with various forms of generative models, as shown by Ours w/ SuSIE where we use an image-editing model to predict the subgoals for exploration.
Please refer to Appendix A for more results and Appendix C for failure analysis.

**Table 2: Quantitative Results of 6 tasks in Meta-World. Methods marked with an asterisk ‘*’ require ground-truth action demonstrations to train. AVDC uses the segmentation mask of the target object to compute actions. Our method learns the policy by video-guided self-exploration in the environment without any external supervision.**
|  | door-open | door-close | handle-press | hammer | assembly | faucet-open | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| BC* | 64.0$\pm\mathtt{4.4}$ | 76.0$\pm\mathtt{0.0}$ | 49.6$\pm\mathtt{3.2}$ | 4.8$\pm\mathtt{3.9}$ | 8.0$\pm\mathtt{2.5}$ | 88.8$\pm\mathtt{3.0}$ | 48.5$\pm\mathtt{2.8}$ |
| GCBC* | 64.8$\pm\mathtt{6.9}$ | 93.6$\pm\mathtt{2.0}$ | 50.4$\pm\mathtt{6.5}$ | 4.8$\pm\mathtt{1.6}$ | 8.0$\pm\mathtt{2.5}$ | 95.2$\pm\mathtt{1.6}$ | 52.8$\pm\mathtt{3.5}$ |
| DP BC* | 73.6$\pm\mathtt{4.8}$ | 94.4$\pm\mathtt{2.0}$ | 52.0$\pm\mathtt{8.4}$ | 7.2$\pm\mathtt{3.0}$ | 0.8$\pm\mathtt{1.6}$ | 82.4$\pm\mathtt{2.0}$ | 51.7$\pm\mathtt{3.6}$ |
| DP GCBC* | 68.0$\pm\mathtt{0.0}$ | 96.0$\pm\mathtt{4.0}$ | 36.0$\pm\mathtt{0.0}$ | 14.0$\pm\mathtt{2.0}$ | 5.6$\pm\mathtt{2.0}$ | 80.0$\pm\mathtt{0.0}$ | 49.9$\pm\mathtt{1.3}$ |
| AVDC | 52.0$\pm\mathtt{0.0}$ | 97.3$\pm\mathtt{1.9}$ | 76.0$\pm\mathtt{5.7}$ | 4.0$\pm\mathtt{3.3}$ | 8.0$\pm\mathtt{0.0}$ | 66.7$\pm\mathtt{5.0}$ | 50.7$\pm\mathtt{2.6}$ |
| Ours | 76.0$\pm\mathtt{4.9}$ | 85.6$\pm\mathtt{2.0}$ | 94.4$\pm\mathtt{2.0}$ | 43.2$\pm\mathtt{6.4}$ | 16.8$\pm\mathtt{3.0}$ | 87.2$\pm\mathtt{3.0}$ | 67.2$\pm\mathtt{3.6}$ |

Figure: Figure 4: Qualitative Results of task Door-Open in Meta-World environment. The position of the box and robot are randomized in test-time. Only a subset of the predicted video frames are shown due to space limit. Our goal-conditioned policy can follow the subgoals given by the video frames and successfully finish the task. BC misses the handle probably due to the out of training distribution box position and starts to predict random actions. AVDC can move to the handle thanks to the exact given handle location. However, it begins to close the door halfway, probably because of the incorrect flow prediction due to error accumulation or occlusion.
Refer to caption: x4.png

Meta-World (Yu et al., )
is a simulated robotic benchmark of a Sawyer robot arm with a set of tabletop manipulation tasks that involve various object interactions and different tool use.
The action space consists of the delta position of the end effector and the applied force on the gripper, resulting in a total dimension of 4.

We consider 6 tasks: door-open, door-close, handle-press, hammer, assembly, faucet-open.
We directly utilize the video model checkpoint provided in AVDC.
To further validate the proposed exploration approach, we conduct single-task exploration in this environment, where we train one policy model for each task. Each learning-based baseline model is trained on the same demonstrations used to train the video model, namely, 15 episodes per task.
We present the quantitative results in Table 2 and qualitative results in Figure 4.
We observe that our policies can successfully learn various manipulation skills and outperform all the counterparts in the average success rate despite the absence of action labels for training. In addition, we also compare with training a single-task RL with a zero-shot reward function in Appendix A.7.

Calvin (Mees et al., ) is a robotic simulation environment with multiple language-conditioned tasks. This environment contains a 7-DOF Panda robot arm and various assets including a desk with a sliding door, a drawer, an LED, and a lightbulb. The agent is required to complete tasks in the environment given by a corresponding language description. The action space we use is same as Libero.

We consider 4 tasks: turn on lightbulb, turn on led, move slider left, and open drawer.
These tasks involve different manipulation skills and different operation areas in the workspace, allowing us to validate skill learning and spatial coverage capability of our goal-conditioned policy.
For example, to move the slider left, the agent has to drag the handle from the central area to the upper left corner. We present the quantitative results in Table 3 and qualitative results in Figure 5. We do not report AVDC in the above table, as we found it performed very poorly in the above environments. Even without any action data or rewards, our policy is able to cover the majority of the space and outperform all the baselines, especially for the challenging open drawer task, where the agent must move downwards below the table and pull toward the bottom right boundary of the environment.

Figure: Figure 5: Qualitative Results of task Open Drawer in Calvin environment. Our goal conditioned policy successfully completes the task by following the subgoals in the video, while both BC and GCBC cannot put the end effector to the handle and thus fail to open the drawer.
Refer to caption: x5.png

Figure: Table 3: Quantitative Results of 4 Calvin tasks. Each task requires the agent to manipulate objects located in different regions, especially in open drawer where the policy needs to cover the bottom right boundary of the environment.
Refer to caption: x6.png

Figure: Figure 7: Qualitative Results of Navigation to Bread in iTHOR environment. The start position of the agent is randomized in test time. Only a subset of the predicted video frames are shown due to the space limit. In navigation tasks with a moving camera, the video model can still generate realistic frames that respect the the actual scene layout. Our goal-conditioned policy is able to follow the video frames and reach the bread shown in the middle of the frame. However, BC turns to the other way, and AVDC cannot correctly infer the required actions and miss the target.
Refer to caption: x7.png

**Table 4: Quantitative Results on iThor Navigation. We report the success rates across 4 different scenes and 12 targets. Though without access to action labels, our method is on par with the BC and GCBC trained on expert demonstrations and outperforms AVDC which also does not require expert action labels.**
|  | Kitchen | Living Room |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Toaster | Spatula | Bread | Overall | Painting | Laptop | Television | Overall |
| BC* | 48.0$\pm\mathtt{2.4}$ | 56.0$\pm\mathtt{3.7}$ | 36.0$\pm\mathtt{6.6}$ | 46.7$\pm\mathtt{4.3}$ | 36.0$\pm\mathtt{12.4}$ | 45.0$\pm\mathtt{11.0}$ | 49.0$\pm\mathtt{4.9}$ | 43.3$\pm\mathtt{9.4}$ |
| GCBC* | 52.0$\pm\mathtt{5.1}$ | 49.0$\pm\mathtt{6.6}$ | 39.0$\pm\mathtt{5.8}$ | 46.7$\pm\mathtt{5.9}$ | 27.0$\pm\mathtt{8.1}$ | 49.0$\pm\mathtt{5.8}$ | 58.0$\pm\mathtt{8.7}$ | 44.7$\pm\mathtt{7.6}$ |
| AVDC | 10.0$\pm\mathtt{4.1}$ | 13.3$\pm\mathtt{4.7}$ | 13.3$\pm\mathtt{2.4}$ | 12.2$\pm\mathtt{3.7}$ | 8.3$\pm\mathtt{2.4}$ | 13.3$\pm\mathtt{8.5}$ | 20.0$\pm\mathtt{4.1}$ | 13.9$\pm\mathtt{5.0}$ |
| Ours | 45.0$\pm\mathtt{3.2}$ | 56.0$\pm\mathtt{4.9}$ | 44.0$\pm\mathtt{5.8}$ | 48.3$\pm\mathtt{4.6}$ | 29.0$\pm\mathtt{4.9}$ | 42.0$\pm\mathtt{6.0}$ | 57.0$\pm\mathtt{5.1}$ | 42.7$\pm\mathtt{5.3}$ |

#### 4.2 Visual Navigation

In addition to tabletop-level robot arm manipulation tasks, we evaluate the proposed method in a room-level visual navigation setting with discrete actions.

iTHOR (Kolve et al., ) is a room-level vision-based simulated environment, where agents can navigate in the scenes and interact with objects. We adopt the iTHOR visual object navigation benchmark, where agents are required to navigate to the specific type of objects given by a nature language input. The action space consists of four actions: Move Ahead, Turn Left, Turn Right, and Done. We incorporate 4 different scene types: Kitchen, Living Room, Bedroom, and Bathroom, with 3 targets in each scene. Following (Ko et al., ), the video model is trained on 20 demonstration image sequences per task. BC and GCBC are also trained on the same demonstrations while with access to corresponding actions.
Since the navigation actions are discrete and episode lengths (typically $<20$) are much shorter than robot arm manipulation tasks, we find that predicting a single action suffices. Our method uses the same ResNet+MLP model architecture as BC-based baselines.

We present the quantitative results in Table 4.
Our method outperforms the no expert data baseline AVDC by 36% on average and surpasses all baselines in the Kitchen scene while performing on par with BC-based baselines in other scenes.
Qualitative results are shown in Figure 7 and 16.
Our goal-conditioned policy is able to reliably synthesize discrete actions that follow the generated video plan.
In contrast, the actions inferred by AVDC are incorrect: the agent keeps rotating right and misses the target, whereas the underlying actions in the video should be moving ahead and then rotating right, probably because that the drastic change in observation poses challenges for optical flow prediction.
BC predicts a wrong action at the first step, where the agent directly rotates to the opposite side and navigates to the target Spatula, likely due to its limited generalizability – directly mimicking the similar actions in the training demonstrations at this position.

#### 4.3 Ablation Studies

In this section, we ablate the effectiveness of design choices described in Section 3. We provide additional studies in Appendix A, including training with different amounts of data, video exploration frequency $q_{v}$, and different horizons of the video model and goal-conditioned policy.

###### Random Action Bootstrapping.

We conduct ablation studies on the importance of the random action bootstrapping technique.
We first present the training-time exploration efficiency in Figure 8.
As shown by w/o rand, the no random action method tends to collapse and can hardly achieve any task success during the exploration. Since the task completion data cannot be collected from the environment, w/o rand cannot obtain any meaningful results in test time, which is reflected in Table 5.
In contrast, as shown by the blue, red, and orange lines in Figure 8, the model is able to obtain significantly more task success after we apply random action bootstrapping.
The performance of w/o extra rand decreases through time in Scene 1, showing that extra random exploration is able to stabilize training, and its performance is similar to ours in Scene 2, probably because the manipulation region for Scene 2 focuses on the center area of the table (see Figure 12) and the initial random actions suffice.
Besides, we observe that the slope of the curve gradually becomes stable after just 250 rollout attempts, which means that the success rate of exploration can easily converge, showing the efficiency of the proposed video-guided exploration scheme.

###### Chunk-level Action Prediction.

We compare the performance of the proposed chunk-level action prediction model v.s. single action prediction model. We use the same setup as BC (ResNet18+MLP) for the single action prediction baseline.
We first present a line chart of the number of success during exploration v.s. total number of exploration rollouts in Figure 8.
Both the chunk-level method and the single action method achieve non-trivial numbers of success during exploration.
However, we can see that the chunk-level model consistently obtains higher success rates (i.e., steeper slope) in the exploration phase, which facilitates the policy learning because the training data contains more trajectories that successfully complete the tasks.
In Table 5, we present the test-time success rate of the two methods in Libero. Similar to the exploration phase, the chunk-level action prediction model achieves higher success rates by 25.8% and 21.0%, respectively.

###### Video Guided Environment Exploration.

We further compare the performance of purely random action exploration versus with video-guided exploration, as shown by w/o video in Table 5,
where we replace the video-guided exploration in Algorithm 1 by random exploration.
Unsurprisingly, the purely random exploration baseline fails across all tasks.
We hypothesize that without guidance from the video model, the agent can hardly discover the necessary states to complete the tasks, probably due to the long temporal distance between the start and goal states and the fact that the relevant state space for task completion only accounts for a tiny subset of the infinite possible states.

Figure: Figure 8: Line Chart of Number of Success versus Number of Total Exploration Rollouts During Training. Both ours, w/o extra rand, single action can effectively complete tasks during video-guided exploration and thus is able to collect high-quality data for further policy training.
Refer to caption: x8.png

### 5 Discussion

###### Limitations.

Our approach has several limitations. First, since the approach relies on goal-conditioned random exploration and without access to action labels, for tasks that require very precise manipulation (i.e. stacking a block tower in millimeter precision), our random exploration procedure may not find the precise set of actions. In such settings, having a random exploration primitive (i.e. stacking a block on top of another) on which we do goal-conditioned policy learning may help us find such precise actions. In addition, purely random exploration in the physical world might pose additional requirements for the workbench,
as sampled actions or the initial actions outputted by the learned goal-conditioned policy may cause undesired contacts between the robot and the external environment. This can be partially mitigated by integrating hard-coded safety constraints and is also a further direction of future work.
Finally, our method usually requires dozens of video-guided rollouts to obtain a competent policy for a task. While resetting an environment is easy in simulation, this might demand additional work in real-world experiments.
Combining our approach with a policy that can reset environment states would be an interesting direction for the future.

###### Conclusion.

In this paper, we have presented a self-supervised approach to ground generated videos into actions. As generative video models become increasingly more powerful, we believe that they will be increasingly useful for decision-making, providing powerful priors on how various tasks should be accomplished. As a result, the question of how we can accurately convert generated video plans to actual physical execution will become increasingly more relevant, and our approach points towards one direction to solve this question, through online interaction with the agent’s environment.

##### Acknowledgments

We thank the Center for Computation and Visualization at Brown
University and Chen Sun for providing the computational resources.
We thank Ronald Parr and Haotian Fu for the helpful discussion.

### Appendix

## 

### Appendix A Additional Quantitative Results

In this section, we provide additional quantitative experiment results.
In Section A.1,
we compare our methods with BC trained on additional demonstration data.
Next, in Section A.2,
we investigate the effect of training the video model with different amounts of demonstration data.
In Section A.3,
we experiment our proposed method with video models of different horizons.
Then, in Section A.4,
we study how varying the video exploration frequency $q_{v}$ affects exploration performance.
In Section A.5,
we conduct ablation studies of using goal-conditioned policies with different horizons.
Following this, in Section A.6, we evaluate the policy after different numbers of training iterations.
In Section A.7, we compare our method with a reinforcement learning baseline with zero-shot reward.
Finally, in Section A.8, we conduct ablation studies on the task success rate over the number of video-guided exploration rollouts.

#### A.1 BC with More Data

We provide comparison of our policy learned by unsupervised exploration and plain BC with increasing amount of training data.
To train these BC baselines, We use the official demonstrations provided by Libero. For example, BC 10 means that we use 10 demonstrations per task to train the BC model (resulting in 80 demonstrations in total).
The reported results are average across 5 checkpoints.
Note that our policy does not need any action labels and directly learn how to ground the goals given by the video model via exploration in the environment. The video model is trained with 20 demonstrations, which only requires image sequences.
That said, this is not a completely fair comparison, but we include this comparison to better illustrate the difficulty of the tasks.

We report the results in Table 6. The success rate of our method even outperforms BC trained with 50 demonstrations per task, while our policy is learned purely by exploration, indicating the efficacy of the proposed exploration method. We also observe that though the success rate of BC generally increases over more training data, BC 20 is able to slightly outperform BC 30 in Scene 1 while BC 30 slightly outperforms BC 40 in Scene 2.
Since the BC model is trained on 8 tasks, the heterogeneous task structures can result in fluctuation in the success rate (the model might bias towards some specific tasks).

**Table 6: BC with Different Amount of Data in Libero Environment.**
|  | put-red-mug-left | put-red-mug-right | put-white-mug-left | put-Y/W-mug-right | Overall |
| --- | --- | --- | --- | --- | --- |
| BC 10 | 2.4$\pm\mathtt{3.2}$ | 8.8$\pm\mathtt{6.9}$ | 23.2$\pm\mathtt{12.0}$ | 5.6$\pm\mathtt{4.1}$ | 10.0$\pm\mathtt{6.5}$ |
| BC 20 | 8.8$\pm\mathtt{5.3}$ | 15.2$\pm\mathtt{7.8}$ | 32.0$\pm\mathtt{12.9}$ | 21.6$\pm\mathtt{12.3}$ | 19.4$\pm\mathtt{9.6}$ |
| BC 30 | 9.6$\pm\mathtt{4.1}$ | 14.4$\pm\mathtt{9.0}$ | 28.0$\pm\mathtt{9.1}$ | 15.2$\pm\mathtt{11.1}$ | 16.8$\pm\mathtt{8.3}$ |
| BC 40 | 8.0$\pm\mathtt{5.1}$ | 11.2$\pm\mathtt{5.3}$ | 44.8$\pm\mathtt{19.8}$ | 21.6$\pm\mathtt{12.0}$ | 21.4$\pm\mathtt{10.6}$ |
| BC 50 | 12.8$\pm\mathtt{8.9}$ | 20.0$\pm\mathtt{12.9}$ | 40.0$\pm\mathtt{12.1}$ | 18.4$\pm\mathtt{6.0}$ | 22.8$\pm\mathtt{10.0}$ |
| Ours | 38.4$\pm\mathtt{15.3}$ | 40.8$\pm\mathtt{7.8}$ | 51.2$\pm\mathtt{3.9}$ | 38.4$\pm\mathtt{8.6}$ | 42.2$\pm\mathtt{8.9}$ |

#### A.2 Video Model with Different Amounts of Data

We investigate the effects of training the video model with different amounts of demonstration data.
For the result shown in Table 1 in the main paper, we use 20 demonstrations per task to train the video model.
In this section, we provide two ablation studies that use 10 demonstrations per task and 50 demonstrations per task, as shown in Table 7 below.

The performance of our method increases in both scenes as with more training data.
While the performance uniformly increases in Scene 1, we observe that in some specific tasks of Scene 2, the performance slightly decreases. This might be due to the learning capacity of the underlying goal-reaching policy, since we only train one policy model for all eight tasks.
In general, we believe that scaling up the data size, data diversity, and model complexity can further buttress the performance.

**Table 7: Training the Video Model with Different Amount of Data.**
|  | put-red-mug-left | put-red-mug-right | put-white-mug-left | put-Y/W-mug-right | Overall |
| --- | --- | --- | --- | --- | --- |
| train size 10 | 18.4$\pm\mathtt{8.2}$ | 31.2$\pm\mathtt{4.7}$ | 22.4$\pm\mathtt{2.0}$ | 44.8$\pm\mathtt{5.9}$ | 29.2$\pm\mathtt{5.2}$ |
| train size 20 | 38.4$\pm\mathtt{15.3}$ | 40.8$\pm\mathtt{7.8}$ | 51.2$\pm\mathtt{3.9}$ | 38.4$\pm\mathtt{8.6}$ | 42.2$\pm\mathtt{8.9}$ |
| train size 50 | 40.0$\pm\mathtt{10.7}$ | 54.4$\pm\mathtt{9.0}$ | 69.6$\pm\mathtt{14.9}$ | 57.6$\pm\mathtt{9.7}$ | 55.4$\pm\mathtt{11.1}$ |

#### A.3 Video Model with Different Horizons

In this section, we study the effect of different video model horizons.
We follow the training procedure of the video model in Ko et al. (): during training, given a start image observation, we uniformly sample $h$ images between the start image and the final task completion image and use these $h$ images as supervision signals.
That said, a longer video horizon $h$ will generate a denser subgoal sequence.
Following Ko et al. (), we set horizon $h=7$ across all our experiments except for this ablation study in Table A.3.

As shown in Table A.3, our method is able to maintain a similar performance across different video horizons. We observe that performance decreases when video horizon is set to 9.
One potential reason is that the modeling complexity increases when we increase the prediction horizon, while we keep the same video model architecture, suggesting that we should learn video models over a sparser set of video frames.

**Table 8: Video Model with Different Horizons.**
|  | door-open | door-close | handle-press | hammer | assembly | faucet-open | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DrQ + LIV | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 8.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 1.3$\pm\mathtt{0.0}$ |
| Ours | 76.0$\pm\mathtt{4.9}$ | 85.6$\pm\mathtt{2.0}$ | 94.4$\pm\mathtt{2.0}$ | 43.2$\pm\mathtt{6.4}$ | 16.8$\pm\mathtt{3.0}$ | 87.2$\pm\mathtt{3.0}$ | 67.2$\pm\mathtt{3.6}$ |

#### A.4 Different Video Exploration Frequency

In this section, we study the effect of video exploration frequency. For example, when $q_{v}$ is set to 200, we conduct one video-guided exploration for each task every 200 training iterations.
The trade-off between smaller and larger $q_{v}$ is that: if $q_{v}$ is small, the agent will conduct exploration in a higher frequency, where the replay buffer will refresh faster and contain more latest rollout data. If $q_{v}$ is large, the agent conducts exploration less frequently, which might enable the agent to better fit and digest the existing data in the replay buffer. We provide ablation studies on 5 different $q_{v}$ on Libero environment, as shown in Figure 9.

Figure: Figure 9: Different Video Exploration Frequency $q_{v}$. $q_{v}=200$ indicates that we conduct one video-guided exploration for each task every 200 training iterations. Thus, the agent will conduct more exploration with lower $q_{v}$ while less exploration with higher $q_{v}$. Our method performs steadily across different values of $q_{v}$.
Refer to caption: x9.png

Figure 9 shows that
our method performs steadily across various $q_{v}$, ranging from 100 to 800.
Note that when $q_{v}$ is set to 800, the video exploration is four times smaller than our default setting ($q_{v}=200$), which means that our method can achieve a comparable performance with much fewer environment interactions.

#### A.5 Different Policy Horizon for our Goal Conditioned Policy

In this section, we study the effect of using different horizons for the diffusion policy based goal-conditioned policy.
We set the policy horizons to 12, 16, 20, and 24 with the same CNN-based architecture.
We observe that the policy learned by the proposed exploration pipeline is robust to the different horizons value, and obtain the highest success rate when horizon = 16, which also conforms to the reported results in Diffusion Policy (Chi et al., ).
Please refer to Table A.5 for quantitative results.

**Table 9: Different Horizons for the Goal-Conditioned Policy.**
|  | door-open | door-close | handle-press | hammer | assembly | faucet-open | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DrQ + LIV | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 8.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 1.3$\pm\mathtt{0.0}$ |
| Ours | 76.0$\pm\mathtt{4.9}$ | 85.6$\pm\mathtt{2.0}$ | 94.4$\pm\mathtt{2.0}$ | 43.2$\pm\mathtt{6.4}$ | 16.8$\pm\mathtt{3.0}$ | 87.2$\pm\mathtt{3.0}$ | 67.2$\pm\mathtt{3.6}$ |

#### A.6 Different Training Timesteps

In this section, we investigate the model performance at different training timesteps on Libero environment, specifically, when the model is trained for $40k$, $80k$, $120k$, $160k$, and $200k$ steps. We set the video exploration frequency $q_{v}$ to 200 in this experiment.
We observe that the overall success rate of the two scenes stabilize after $80k$ training steps. However, the per-task success rate oscillates through further training. This might be caused by the recent collected data in the replay buffer (which is used for training) and the task level interference.

**Table 10: Performance at Different Training Steps in Libero Environment.**
|  | door-open | door-close | handle-press | hammer | assembly | faucet-open | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DrQ + LIV | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 8.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 1.3$\pm\mathtt{0.0}$ |
| Ours | 76.0$\pm\mathtt{4.9}$ | 85.6$\pm\mathtt{2.0}$ | 94.4$\pm\mathtt{2.0}$ | 43.2$\pm\mathtt{6.4}$ | 16.8$\pm\mathtt{3.0}$ | 87.2$\pm\mathtt{3.0}$ | 67.2$\pm\mathtt{3.6}$ |

#### A.7 Comparison to RL with Zero-Shot Reward

In this section, we compare our method with a reinforcement learning baseline.
Since our method does not have access to environment rewards, we leverage a foundational zero-shot robotic model to generate the rewards.
Specifically, we adopt DrQ  (Kostrikov et al., ; Yarats et al., ) and LIV (Ma et al., ) as the RL method and foundation model respectively.
We use the potential-based reward defined in LIV.

We present the results in Table 11.
We see that DrQ+LIV fails to make meaningful progress in five out of six tasks, achieving only an 8% success rate in the handle-press task.
We observe that one probable cause is that the generated reward signals are ambiguous, which hinders the RL method to learn. For instance, it is challenging for the model to generate a significant positive reward upon task completion and the rewards may oscillate even when positive progress is being made.

**Table 11: Quantitative Comparison to DrQ+LIV on 6 tasks from Meta-World.**
|  | door-open | door-close | handle-press | hammer | assembly | faucet-open | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DrQ + LIV | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 8.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 0.0$\pm\mathtt{0.0}$ | 1.3$\pm\mathtt{0.0}$ |
| Ours | 76.0$\pm\mathtt{4.9}$ | 85.6$\pm\mathtt{2.0}$ | 94.4$\pm\mathtt{2.0}$ | 43.2$\pm\mathtt{6.4}$ | 16.8$\pm\mathtt{3.0}$ | 87.2$\pm\mathtt{3.0}$ | 67.2$\pm\mathtt{3.6}$ |

#### A.8 Success Rate v.s. Number of Video-Guided Rollouts

In this section, we conduct ablation studies on the task success rate over the number of video-guided rollouts.
We show the performance curve of the Libero task put the red mug on the plate in Figure 10, where the y-axis represents the success rate of 25 evaluation episodes and the x-axis denotes the number of conducted video-guided exploration rollouts.

Specifically, we first warm-start the policy by training it on randomly sampled actions, which corresponds to zero video-guided rollouts.
Then, the model begins to perform video-guided exploration, and we set the video exploration frequency to 800 (i.e., the agent will conduct 5 video-guided exploration every 800 training iterations).
We save a model checkpoint after every 10 video exploration episodes and evaluate each checkpoint on 25 test-time problems.
Note that to better demonstrate the correlation of success rate and # of video-guided exploration, we train the policy model just on a single task, while in Table 1, the policy model is jointly trained on 8 tasks.

As shown in Figure 10, the policy is unable to accomplish the task without video-guided exploration episodes (when the x-axis is 0). However, the success rate rapidly increases with the start of video-guided exploration episodes, reaching a success rate of 40% after only 100 episodes. After approximately 200 episodes, the success rate gradually saturates, stabilizing around 80%.

Figure: Figure 10: Performance Curve of task put the red mug on the plate in Libero. The y-axis represents the success rate over 25 unseen test-time problems and the x-axis denotes the number of conducted video-guided exploration episodes. The task success rate increases rapidly with the start of video-guided exploration, while the agent struggles to complete the task without it.
Refer to caption: x10.png

### Appendix B Additional Qualitative Results

In this section, we present additional qualitative results in both tabletop manipulation environments and visual navigation environments.
For extra results and side-by-side qualitative comparison, please refer to our [website](https://video-to-action.github.io/).

In Section B.1, we provide qualitative results for each task in Libero environment.
In Section B.2, we show qualitative results for Meta-World tasks. Next, in Section B.3, we present qualitative results on Calvin environment. Finally, in Section B.4, we present a long-horizon visual navigation planning rollout in iTHOR environment.

#### B.1 Libero

In this section, we provide additional qualitative results of our method on 8 tasks in Libero environment, as shown in Figure 11 and 12.
For each episode, we show the generated video in the first row and the rollout results of our goal conditioned policy learned by self-supervised exploration in the second row.

Figure: Figure 11: Additional Qualitative Results of the Generated Videos and Our Policy Rollout on Libero Scene 1. We present qualitative results of four tasks in Libero Scene 1. For each task, results are displayed in 2 rows: the first row of images are the generated video from the video model conditioned on the start observation image and task description, and the second row of images are the rollout of our policy conditioned on the corresponding frames from the generated video. The task description provided to the video model are shown below the respective images.
Refer to caption: x11.png

Figure: Figure 12: Additional Qualitative Results of the Generated Videos and Our Policy Rollout on Libero Scene 2. We present qualitative results of four tasks in Libero Scene 2. For each task, results are displayed in 2 rows: the first row of images are the generated video from the video model conditioned on the start observation image and task description, and the second row of images are the rollout of our policy conditioned on the corresponding frames from the generated video. The task description provided to the video model are shown below the respective images.
Refer to caption: x12.png

#### B.2 Meta-Wolrd

In this section, we provide additional qualitative results of our method on 6 tasks in Meta-World environment, as shown in Figure 13 and 14.
For each episode, we show the generated video in the first row, and the rollout results of our goal-conditioned policy learned by self-supervised exploration in the second row.

Figure: Figure 13: Additional Qualitative Results of the Generated Videos and Our Policy Rollout of 4 tasks on Meta-World. We present qualitative results of four tasks in Meta-World Environment. For each task, results are displayed in 2 rows: the first row of images are the generated video from the video model conditioned on the start observation image and task description, and the second row of images are the rollout of our policy conditioned on the corresponding frames from the generated video. The task prompts provided to the video model are shown below the respective images.
Refer to caption: x13.png

Figure: Figure 14: Additional Qualitative Results of the Generated Videos and Our Policy Rollout of 2 tasks on Meta-World. We present qualitative results of four tasks in Meta-World Environment. For each task, results are displayed in 2 rows: the first row of images are the generated video from the video model conditioned on the start observation image and task description, and the second row of images are the rollout of our policy conditioned on the corresponding frames from the generated video. The task prompt provided to the video model are shown below the respective images.
Refer to caption: x14.png

#### B.3 Calvin

In this section, we provide additional qualitative results of our method on 4 tasks in Calvin environment, as shown in Figure 15.
For each episode, we show the generated video in the first row, and the rollout results of our goal conditioned policy learned by self-supervised exploration in the second row.

Figure: Figure 15: Additional Qualitative Results of the Generated Videos and Our Policy Rollout of 4 tasks on Calvin. We present qualitative results of four tasks in Calvin Environment. For each task, results are displayed in 2 rows: the first row of images are the generated video from the video model conditioned on the start observation image and task description, and the second row of images are the rollout of our policy conditioned on the corresponding frames from the generated video. The task prompt provided to the video model are shown below the respective images.
Refer to caption: x15.png

#### B.4 iThor Visual Navigation

Figure: Figure 16: Four Consecutive Policy Rollout for one Evaluation Episode in a Living Room Scene of iTHOR. In each rollout, the first row of images are the generated video from the video model conditioned on the start observation image (shown in the leftmost column) and task description (name of the target). The second row of images are the rollout of our policy conditioned on the corresponding frames from the generated video. The names of the scene and target are shown at the bottom. Due to the long spatial distance, the agent takes more than 20 actions to navigate to the television, which corresponds to the four videos above. The last observation of the previous rollout is fed to the video model to generate future subgoals, as indicated by the dashed arrows.
Refer to caption: x16.png

In this section, we provide additional qualitative results of the generated videos and our goal conditioned policy learned by self-supervised exploration in a Living Room Scene (FloorPlan201) of iTHOR visual navigation environments, as shown in Figure 16.
Starting at Start Obs 0, the agent is tasked to navigate to the television at the other end of the room, which typically requires more than 20 actions to reach.
While the horizon of the video model is only 7, we consecutively generate future subgoals by feeding the last observation image to the video model as condition (indicated by the dashed arrows). With four video predictions and 26 actions, the agent successfully navigates to the television, showing the long and consecutive rollout capability of our policy.

### Appendix C Failure Mode Analysis

In the previous section, we demonstrate that our method is able to achieve non-trivial performance in various robotic environments. However, the method still poses some limitations, which lead to failure. In this section, we provide analysis into the failure mode of our method. We categorize the causes to video model based and policy learning based.

#### C.1 Video Model

The performance of the video model is one influential factor of the success rate because that if an incorrect subgoal is given, even though the policy is accurate enough, the task cannot be completed.

Hallucination is a common issue for generative models (Yang et al., ; Ji et al., ). We also observe some extent of hallucination in our video model. We present some visualizations in Figure 17.

In Table 1, Ours and Ours w/ SuSIE only achieve 25.6% and 36.0% on the task ‘put the white mug on the plate’.
We observe that one major cause of this relatively low performance is the
heavy hallucination in the generated video of this task, as shown in the first row of Figure 17. We hypothesize that this might partially due to the dataset imbalance problem, as we have two tasks involving chocolate pudding while only one for white mug in this scene, considering that only 20 demonstrations are provided for each task.

Figure: Figure 17: Failure Mode: Video Model Hallucination. In the first row, the given prompt to the video model is ‘put the white mug on the plate’. Though the first four frames generated by the video model are on the right path for achieving the goal, the white mug is suddenly replaced by the chocolate pudding in the following frames, which will subsequently confuse the policy. In the second row, the given prompt to the video model is ‘put the yellow and white mug on the right plate’. Similarly, the agent in the video frames first correctly approaches the yellow and white mug, but the yellow and white mug is replaced by a red mug in the intermediate frames.
Refer to caption: x17.png

Mismatch of Task and Generated Videos Given a specific task description, we observe that the video model might generate a video that is actually for completing another task. For example, given the task description of ‘put the chocolate pudding to the left of the plate’, the video model might generate a video that ‘put the chocolate pudding to the right of the plate’. One potential cause is the relatively close embedding distance between these two sentences as well as the corresponding videos, making the model generate a similar but incorrect video.
We provide some visualizations in Figure 18.

We believe that these issue can be mitigated by multiple ways, such as improving the video model architecture, scaling up the training data, finetuning from existing pre-trained large video model, or using the environment interaction feedback to correct the video model, which might be interesting future research directions.

Figure: Figure 18: Failure Mode: Mismatch of the Task and the Generated Video. In the first row, the given prompt to the video model is ‘put the white mug on the left plate’, while the generated video puts the red cup to the left plate. In the second row, the given prompt to the video model is ‘put the chocolate pudding to the left of the plate’, while the generated video puts the chocolate pudding to the right of the plate.
Refer to caption: x18.png

Figure: Figure 19: Failure Mode: Precision, Libero. A failure case of ‘put the red mug on the left plate’ in Libero Scene 1. We only show the results of the last three subgoals. Though the goal conditioned policy is able to successfully put the red mug on the left plate in the last frame, a slight displacement is introduced (see the rightmost column), which results in task failure.
Refer to caption: x19.png

#### C.2 Policy Learning

Policy learning is particularly challenging in such unsupervised setting since we assume no action demonstrations nor rewards. Generally, we observe that the learned policy is able to follow the video model, but there remain challenges for the policy to handle fine-grained tasks that require precise control. For example, our policy achieves 94.4% at handle press task, while only achieves 16.8% at assembly task in Meta-World. In Figure 19 and Figure 20, we provide some qualitative results of failure cases in Libero and Meta-World.

In Figure 19,
we present a failure case of the task ‘put the red mug on the left plate’ in Libero. In this task, the agent needs to put the red mug at the center of the left plate within a small tolerance.
One major reason of the policy’s failure is that the mug is not precisely placed at the center, as shown in the rightmost column.

In Figure 20,
we present a failure case of the task assembly in Meta-World. In this task, the agent needs to pick up a nut and precisely place it onto a peg. While the policy seems to exactly follow the given subgoals, it fails to insert the nut at the last frame where the nut undesirably touches the peg.

One potential way to increase the performance is to design some replanning strategies.
Another interesting direction to improve the policy is to better leverage the geometric information. Currently, our policy only takes one 2D observation image as input.
Using a complementary 3D point cloud for observation input is likely to be beneficial (Ze et al., ; Ling et al., ) for fine-grained tasks.
In addition, incorporating some action primitive in the random exploration phase, learning or defining some manipulation primitives for precise control task might also be helpful.

### Appendix D Implementation details

In this section, we describe the implementation of our proposed method and baselines.

###### Software:

The computation platform is installed with Red Hat 7.9, Python 3.9, PyTorch 2.0, and Cuda 11.8

###### Hardware:

For each of our experiments, we used 1 NVIDIA RTX 3090 GPU or a GPU of similar configuration.

#### D.1 Model Architectures

For the architecture of the video model, we adopt the same design from AVDC (Ko et al., ), which uses 3D UNet as video denoiser and CLIP to encode language features.
In tabletop manipulation tasks, We instantiate the goal-conditioned policy by a diffusion policy (Chi et al., ), which uses ResNet18 as image encoder and 1D UNet to denoise the action trajectory. Following the default setup in (Chi et al., ), we set the horizon to 16 across all other experiments.
Finally, for iTHOR environment, since the action space is discrete and the required number of actions to reach goal is smaller, we use a ResNet18 with a 3-layer MLP with ReLU activation as the goal-conditioned policy with an output horizon of 1.

#### D.2 Training Details

###### Training Pipeline

In training, we randomly sample sequences of image-action pairs of horizon $h$ from the replay buffer.
For Diffusion Policy setup, we mainly follow the hyperparameters suggested in the original paper.
We provide detailed hyperparameters for training our model in Table 12 and 13.
We do not apply any hyperparameter search or learning rate scheduler.
For Libero environment, the training time of our model is approximately 36 hours for a full 200k training steps. However,
the performance can gradually saturate with much fewer steps. For the Meta-World and iThor environments, the training time of our model is approximately one day; for Calvin, the training time of our model is around 15 hours.
For these environments, we also observe that the performance can gradually saturate within less training time.

###### Exploration Grasping Primitive.

To improve the exploration efficiency of grasping, we further add a simple grasping primitive that encourages the agent to attempt a grasp when the end effector is positioned closely above an object.
Specifically, we measure the distance from the wrist camera to the nearest obstacle and compare the distance to the current height of the gripper above table.
The agent is prompted to grasp if the difference is high (indicating a nearby object).
This heuristic can be applied across various robots and greatly reduces the search space necessary to learn grasping.

**Table 12: Hyperparameters of our Goal-conditioned Policy in Libero, Meta-World, and Calvin.**
| Hyperparameters | Value |
| --- | --- |
| Action Prediction Horizon | 16 |
| Action Horizon | 8 |
| Diffusion Time Step | 100 |
| Iterations | 200K |
| Batch Size | 64 |
| Optimizer | Adam |
| Learning Rate | 1e-4 |
| Input Image Resolution | (128,128) |

**Table 13: Hyperparameters of our Goal-conditioned Policy in iTHOR.**
| Hyperparameters | Value |
| --- | --- |
| Horizon | 1 |
| Image Encoder | ResNet18 |
| MLP size | [(1024, 256), (256, 128), (128, 4)] |
| Activation | ReLU |
| Iterations | 100K |
| Batch Size | 64 |
| Optimizer | Adam |
| Learning Rate | 1e-4 |

#### D.3 Implementation of Baselines

###### Behavior Cloning (BC).

For BC, the observed image is first fed to a ResNet-18 (He et al., ) to encode visual information. The vision feature vector is then fed into a 3-layer MLP to predict the next action. For multi-task BC, we concatenate the vision feature with task language description feature encoded by CLIP, and feed the concatenated feature to a 3-layer MLP to predict an action. We use MSE as loss and train for 100k steps with a batch size of 64.

###### Diffusion Policy Behavior Cloning (DP BC).

For DP BC, following Chi et al. (), we use a 1D convolutional neural network as trajectory denoiser and use a ResNet-18 to encode the image observation. We set the diffusion denoising timestep to 100, horizon to 16, and train for 200k iterations with a batch size of 64.

###### Goal-conditioned Behavior Cloning (GCBC).

For GCBC, we concatenate the start image and goal image and feed it to ResNet-18 to encode the visual observation. Similarly, the visual feature is then fed to a 3-layer MLP to predict the next action. We use MSE as loss and train for 100k steps with a batch size of 64. In test-time, given a task, similar to our proposed method, we use the same video model to generate subgoals for GCBC to complete the task.

###### Diffusion Policy Goal-conditioned Behavior Cloning (DP GCBC).

For DP GCBC, we use an additional ResNet-18 image encoder to encode the goal image, and otherwise the model architecture is the same as DP BC.
Note that the model for this baseline is also identical to the model we use in our proposed method.
The goal image feature will then be concatenated with observation image features to condition the denoiser network. We set the diffusion denoising timestep to 100, horizon to 16, and train for 200k iterations with a batch size of 64.

###### AVDC.

For AVDC, we directly adopt the official codebase (Ko et al., ). In Meta-world, we directly use the video model provided in the codebase, which is also the same video model for our method. In AVDC paper, the results are averaged across three different camera views, while our method only uses one camera view. Thus, for AVDC we report results on the same camera view as our method. In iTHOR, we notice that the video model resolution in the codebase is only (64, 64), which is too blurry to identify the target objects. Therefore, we train our video model with a resolution of (128, 128).
Since AVDC does not experiment with Libero, we train our video model in this environment.

###### SuSIE.

For SuSIE, we first train an image-editing diffusion model using the same video demonstration data to train the video model. The image-editing diffusion model also takes as input an observation image and a task description but outputs the next subgoal image. We then use this image-editing model to generate subgoals to guide our exploration, keeping all other hyperparameters the same.

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Would the computational efficiency be improved by using distilled video diffusion models, which could generate videos in a very fast manner? Would the distilled models affect the performance of the proposed method?
2. Given the reliance on the pretrained video generation model, how does the framework handle cases where the video model’s generalization is limited? Are there specific techniques that could be employed to improve the model’s adaptability to novel or dynamic environments?
soundness: 3
strengths: 1. The unsupervised grounding of video models to actions eliminates the dependency on expensive action annotations, efficiently addressing the problem of mapping video-based observations to actionable policies.
2. The proposed method achieves strong performances across multiple evaluation environments, outperforming supervised methods in quantitative and qualitative resutls. Besides, the method show adaptability across different domains, from robotic manipulation to visual navigations, demonstrating the robustness and generalization ability.
3. The ablation studies are comprehensive, demonstrating the effectiveness of the each proposed components.
4. Video-guided exploration allows the agent to focus on task-relevant state spaces, resulting in more targeted exploration and more effective data collection. This targeted approach enables the model to gather high-quality training data efficiently, especially for complex, long-horizon tasks that would otherwise require extensive manual annotation.
summary: This paper presents a framework for grounding large pretrained video models to actions within an embodied environment through self-exploration. By generating visual goals from video states, the authors propose a goal-conditioned exploration strategy that allows agents to solve complex tasks without needing external supervision like rewards or action labels. A set of methods, including chunked action prediction and exploration with randomized exploration, are proposed to enable robust exploration. The proposed method is validated on several simulated environments, such as Libero, MetaWorld, Calvin, and iTHOR, where it demonstrates performance on par with or exceeding that of behavior cloning baselines trained on expert demonstrations.
weaknesses: This paper is well-written with strong motivation and comprehensive evluation results. I only have some minor weakness and questions.

1. The reliance on random exploration may not be able to achieve high performance in tasks requiring high precision, such as tasks involving fine-grained manipulation or exact positioning. This approach may struggle to find optimal actions in environments where precise control is crucial, limitations is applications.
2. The proposed method is highly rely on the quality and generalization ability of the pretrained video generation models. If these models fail to generalize well to new environments, the effectiveness of the approach could be significantly constrained, especially in dynamic or unfamiliar settings.
3. The computational requirements might be high due to the chunk-level action prediction.
4. In algorithm 1, line 10-12 lacks an indentation, making it hard to read what is the condition and what is the content of the end-if sentence.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weakness section.
soundness: 3
strengths: The paper is clearly written and easy to follow, the method proposed achieves state-of-the-art results on several commonly used datasets with consistent improvement. The ablation study does show the effect of each designed module.
summary: This paper introduces a method to "distill" a video-diffusion model to a goal-conditioned  policy for robot manipulation. The authors proposed to leverage the conditional generation capability of video generative models to sample future trajectories as goals for refining the goal-conditioned policy, which can gradually align its action prediction with the actual ground truth. The resulting method achieves state-of-the-art results on several commonly-used benchmarks.
weaknesses: One concern about the proposed method lies in the selection of video-guidance. As assumed in this paper, a good enough conditional video generative model is key in improving the goal-conditioned policy in the iterative refining process. This leads to questions on the availability of such models for specific tasks with limited demonstrations. Though the authors mentioned leveraging pre-trained text-to-video models could be discussed in the future, it seems necessary even at the current scope (or if there is other work arounds or model performance guarantees on the data needed for a good enough task specific video generator).

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: Key questions that should be addressed:
1. Maybe I have misunderstood, but how is it possible that without any ‘good’ actions ever given, the model can find actions that satisfy the goal eventually? Is it relying on the fact that at some point during random action bootstrapping there must be a few success cases? If so, how many successes in random action bootstrapping is required to learn the policy? Also, what is the performance of the policy with just random action bootstrapping (after line 3 of Algo 1) compared to performance after fine tuning with the video model in the self-supervised learning stage?
2. It is mentioned that random action bootstrapping is also performed at periodic intervals during training, but this is not indicated in Algorithm 1 (Algo 1 only shows random action bootstrapping before the loop starts). Please add that to the algorithm.  
3. How is random action bootstrapping implemented in experiments? Specifically, what values are used a_low and a_high in equation 3 for all the different experiments? These details seem important to include in main or supplemental given that w/o random has a 0% success rate. 
4. How frequently is the random action bootstrapping done? Would be a good point to add to the method (Sec 3.2) if this is a consistent number, or to the experiments if this is a hyperparameter.
5. Please provide details on how 'random extra' is performed. How many times extra do you do the random action bootstrapping? 
6. For each experiment, it would be good to note the number of iterations of self-supervised exploration used (N in Algorithm 1). 
7. If time permits, it would be good to understand the dependency of the policy during training on the model: What happens if the video model changes the sample rate it produces frames after the policy has been trained? For example, once the policy is trained, if the video model that is used changes to a pretrained model, does the policy produce viable actions? A simple experiment to run for this would be to just use the current trained policy, and replace the video model at test time, and see if the policy can still work. 

Additional minor comments:
1. Would be good to include more details in the caption for Figure 2. What are these pairs of images showing (the start and goal image of the task or something else)?
2. Please mention the metric you use to evaluate the ‘success’ or ‘failure’ of a rollout? Is there some check in the simulation environment? 
3. Typo line #147: should say ‘benefits both sides’
4. Typo line #423: should say ‘natural’ not ‘nature’
5. Line #314: Sure the video-model is zero-shot but the policy requires self-supervised training afterwards, so just change this sentence so it doesn't over-claim.
soundness: 3
strengths: The paper has several strengths including:
1. The work is well-motivated, well-written, and clear
2. The method is unique in proposing video models as a way to enable strong goal conditioning for the policy without needing action labels. 
3. The idea of self-learning is unique compared to current RL approaches
4. Good set of ablations performed in the main paper and supplement to analyze the method well 
5. The authors sufficiently addressed most of the limitations of their work. Despite these limitations, the method is still very interesting.
summary: This paper proposes leveraging video guidance to learn goal-conditioned policies without access to any external supervision. By leveraging pre-trained task conditioning video generative models, they propose sampling a sequence of frames demonstrating how to complete a task. These frames serve as sub-goal supervision for training a downstream goal-conditioned policy. The policy is initialized via behavior cloning with random action bootstrapping, and then proceeds to improve by sampling the video model to generate more conditioning data for the policy. The self-supervised learning proceeds by collecting the rollouts in the replay-buffer and fine-tuning the policy on the replay-buffer. The video generative model is trained with just image pairs and no action labeled data and can be replaced with large internet-scale pre-trained models in the future. 

Overall, this paper could be a significant contribution in combining self-supervised learning with video generative models to enable agents to solve complex tasks. But currently, there is confusion regarding the success of this method when random action bootstrapping fails. If addressed sufficiently, the rating can be increased.
weaknesses: Some weaknesses of the paper include:
1. There is a lot of confusion around random action bootstrapping. Mainly, I am confused if it is possible to learn a policy without any successful actions (assuming random action bootstrapping does not result in any successful trajectories). Please see the first 6 points under ‘questions’ to know what needs to be added to the paper to address this weakness. 
2. The cost of training this kind of model compared to BC is not mentioned in the limitation section. (The number of rollouts and iterations during self-training is high compared to the number of expert demonstrations for a BC policy). 
3. A real world experiment is not provided. Seems that such an experiment would be high cost, maybe that is why it is not shown? Would be good to discuss if there is a way then to transfer easily from sim2real, but I acknolwedge sim2real is an existing open question, so possibly ok to not address this.
4. Unsure if 3.3 is a novel contribution. Diffusion policy generates action chunks already. What is the new ‘mean sampling’ proposing beyond the standard diffusion policy action generation scheme? Lines 239-242 are unclear.

Meta-review: The paper proposes a novel approach to grounding video models to actions through goal-conditioned exploration, which is a good contribution to the field of embodied AI. The authors have addressed most of the concerns raised by the reviewers in a satisfactory manner.  The proposed method has shown good performance and potential for further development. Minor remaining issues do not outweigh the paper's strengths and contributions. Therefore, the paper is recommended for acceptance.

Decision: Accept (Spotlight)