Learning Universal Policies via Text-Guided Video Generation
Authors: Yilun Du
Sherry Yang
Bo Dai
Hanjun Dai
Ofir Nachum
Joshua B. Tenenbaum
Dale Schuurmans
Pieter Abbeel
Venue: NeurIPS 2023 spotlight (2023)
Keywords: sequential decision making
general-purpose agent
video diffusion
TL;DR: We cast sequential decision making as a text-conditioned video generation problem and propose UniPi which can generalize to combinatorial and multi-task environments and be able to utilize broad internet-scale text-video datasets.

Abstract: A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress in text-guided image synthesis has yielded models with an impressive ability to generate complex novel images, exhibiting combinatorial generalization across domains. Motivated by this success, we investigate whether such tools can be used to construct more general-purpose agents. Specifically, we cast the sequential decision making problem as a text-conditioned video generation problem, where, given a text-encoded specification of a desired goal, a planner synthesizes a set of future frames depicting its planned actions in the future, after which control actions are extracted from the generated video. By leveraging text as the underlying goal specification, we are able to naturally and combinatorially generalize to novel goals. The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space of images, which, for example, enables learning and generalization across a variety of robot manipulation tasks. Finally, by leveraging pretrained language embeddings and widely available videos from the internet, the approach enables knowledge transfer through predicting highly realistic video plans for real robots.

Learning Universal Policies via Text-Guided Video Generation
Yilun Du* Mengjiao Yang *38 Bo Dai 29 Hanjun Dai 2 Ofir Nachum
Joshua B. Tenenbaum Dale Schuurmans a1 Pieter Abbeel
MIT
Google DeepMind UC Berkeley Georgia Tech University of Alberta"
http://wwerrraaplloooggi.o...... github.io/
Abstract
A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress text-guided image synthesis has yielded models with an impressive ability to generate complex novel images, exhibiting combinatorial generalization across domains. Motivated by this success, we investigate whether such tools can be used to construct more general-pune purpose agents, Specifically, we cast the sequential decision making problem as a text-conditioned video generation problem, where, given text-encoded specification of a desired goal, planner synthesizes set of future frames depicting its planned actions in the future, after which control actions are extracted from the generated video. By leveraging text as the underlying goal specification, we are able naturally and combinatorially generalize novel goals. The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space images, which, for example, enables learning and generalization across variety of robot manipulation tasks. Finally, by leveraging pretrained language embeddings and widely available vidcos from the internet, the approach enables knowledge transfer through predicting highly realistic video plans for real robots?.
Introduction
Building models that solve diverse set of tasks has become dominant paradigm in the domains of vision and language. In natural language processing, large pretrained models have demonstrated remarkable zero-shot learning of new language tasks [1, 2, 3J. Similarly, in computer vision, models such as those proposed in [4, 5] have shown remarkable zero-shot classification and object recognition capabilities. A natural next step to use such tools construct agents that can complete different decision making tasks across many environment
However, training such agents faces the inherent challenge of environmental diversity, since different environments operate with distinct state action spaces (e.g., the joint space and continuous controls in MuloCo are fundamentally different from the image space and discrete actions in Atari). Such diversity hampers knowledge sharing, learning, and generalization across tasks and environments. Although substantial effort has been devoted to encoding different environments with universal tokens in a sequence modeling framework [6], it is unclear whether such an approach can preserve the rich knowledge embedded in pretrained vision and language models and leverage this knowledge to transfer to downstream reinforcement learning (RL) tasks. Furthermore, it is difficult to construct reward functions that specify different tasks across environments.
In this work, we address the challenges environment diversity and reward specification by lever- aging video (i.e., image sequences) universal interface for conveying action and observation
denotes equal contribution. Correspondence to yilundu@mit. t. edu and sherryy@berkeley.com SSe video visualizations
polllyplllc....oooo github. io.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Text-Condiine Conditioned Video Generation
Combinatorial anguage
Multi Task Generalization
Lang-Horion Planning
Wipe Brown Bas Before Pick up Red Block Then Red
Block Beowa
[Rock Ihnwnn Blo
with Yellow Spunge
Internet-Scall Scalle Knowledge Transfer
Wipe the Counter with
Prnsgpple Fow Stayo
Robot Arn and Red Cloch
Figure 1: Text-Conditional Video Generation as Universal Policies. Text-conditional video generations enables us to train general purpose policies on wide sources of data (simulated, real robots and YouTube) which may be applied to downstream multi-task settings requiring combinatorical language generalization, long-horizon planning, or intemet-scale knowledge.
behavior in different environments, and text as universal interface for expressing task descriptions. In particular, we design a video generator as a planner that sequentially conditions on current image frame and a text passage describing a current goal (i.e., the next high-level step) to generate trajectory in the form of an image sequence, after which an inverse dynamics model is used to extract the underlying actions from the generated video. Such an approach allows the universal nature of language and video to be leveraged in generalizing to novel goals and tasks across diverse environments. Specifically, we instantiate the text- conditondt video generation model using video diffusion. set of underlying actions are then regressed from the synthesized frames and used to construct a policy to implement the planned trajectory. Since the language description of task is often highly correlated with control actions, text-conditioned video generation naturally focuses generation on action relevant parts of the video. The proposed model, UniPi, is visualized in Figure 1.
We have found that formulating policy generation via text-conditioned video synthesis yields the following advantages:
Combinatorial Generalization. The rich combinatorial nature of language be leveraged to synthesize novel combinatorial behaviors in the environment. This enables the proposed approach to rearrange objects new unseen combinations of geometric relations, as shown in Section 4.1.
Multi-task Learning. Formulating action prediction as a video prediction problem readily enables learning across many different tasks. We illustrate in Section 4.2 how this enables learning across language-condtionned tasks and generalizing to new ones at test time without finetuning.
Action Planning. The video generation procedure corresponds to a planning procedure where sequence of frames representing actions is generated to reach the target goal. Such a planning procedure is naturally hierarchical: a temporally sparse sequence of images toward a goal can first be generated, before being refined with more specific plan. Moreover, the planning procedure is steerable, in the sense that the plan generation can be biased by new constraints introduced at test-time through test-time sampling. Finally, plans are produced in video space that is naturally interpretable by humans, making action verification and plan diagnosis easy. We illustrate the efficacy of hierarchical sampling in Table and steerability in Figure
Internet-Scale Knowledge Transfer. By pretraining a video generation model on a large-scale text-video dataset recovered from the internet, one can recover a vast repository of "demonstrations" that aid the construction of a text-conditioned policy in novel environments. We illustrate how this enables the realistic synthesis of robot motion videos from language instructions in Section 4.3.
The main contribution of this work is to formulate text-conditioned video generation as a universal planning strategy from which diverse behaviors can be synthesized. While such approach departs from typical policy generation in RI., where subsequent actions to execute are directly predicted from current state, illustrate that UniPi exhibits notable generalization advantages over traditional policy generation methods across a variety of domains.
2 Problem Formulation
We first motivate a new abstraction, the Unified Predictive Decision Process (UPDP), an alternative to the Markov Decision Process (MDP) commonly used in RI., and then show an instantiation of UPDP with diffusion models.

---

2.1 Markov Decision Process
The Markov Decision Process [7] is a broad abstraction used to formulate many sequential decision making problems. Many RL algorithms have been derived from MDPs with empirical success [8, 9, 10], but existing algorithms are typically unable to combinatorially generalize across different environments. Such difficulty can be traced back certain aspects of the underlying MDP abstraction:
i) The lack of a universal state interface across different control environments. In fact, since different environments typically have separate underlying state spaces, one would need to construct a complex state representation to represent all environments, making learning difficult. ii) The explicit requirement of real-valued reward function in an MDP. The RL problem is usually defined as maximizing the accumulated reward in an MDP. However, in many practical applications, how design and transfer rewards is unclear and different across environments. iii) The dynamics model in an MDP is environment and agent dependent. Specifically, the dynamics model T a) that characterizes the transition between states (s, s') under action is explicitly dependent to the environment and action space of the agent, which can be significantly different between different agents and tasks.
2.2 Unified Predictive Decision Process
These difficulties inspire us to construct an alternative abstraction for unified sequential decision making across diverse environments. Our abstraction, termed Unified Predictive Decision Pro- cess (UPDP), exploits images as a universal interface acToss environments, texts as task specifiers to avoid reward design, and task-agnostic planning module separated from environment-dependeen control to enable knowledge sharing and generalization.
Formally, we define a UPDP to be a tuple g = (X', C, H. H), where X' denotes the observation space of images, C denotes the space of textual task descriptions, H E A is finite horizon length, and p(-1zoo c) : X' x C + " is a conditional video generator. That is, p(-)z,, c) E A(X")) is conditional distribution over Ht-ste image sequences determined by the first frame To and the task description c. Intuitively, P synthesizes H-step image trajectories that illustrate possible paths for completing target task c. For simplicity, we focus on finite horizon, episodic tasks.
Given a UPDP G. we define a trajectory-task conditioned policy क (-1( ) x x C - A(A to be a conditional distribution over H -tep action sequences A Ideally, x x-11<2, 1 specifies a conditional distribution of action sequences that achieves the given trajectory {xx 1H in the UPDP G for the given task To achieve such an alignment, will consider offline RL scenario where we have access to a dataset of existing experience D ((x,, ot) TH. e) from which both p(-1 c) and π (-H
c) can be estimated.
In contrast to an MDP, a UPDP directly models video-based trajectories and bypasses the need to specify reward function beyond the textual task description. Since the space of video observations X and task descriptions C are both naturally shared across environments and easily interpretable by humans, any video-based planner p( [x),,)) c) Another benefit of UPDP over an MDP is that UPDP isolates the video-based planning with p(- c) from the deferred action selection using < (-M ( - c). This design choice isolates planning de- cisions from action-specific mechanisms, allowing the planner to be environment and agent agnostic.
be conveniently reused, transferred and debugged.
UPDP can be understood as implicitly planning over MDP and directly outputting an optimal trajectory based on the given instructions. Such a UPDP abstraction bypasses reward design, state extraction and explicit planning, and allows for non non-Markoviar modeling of image- based state space. However, learning a planner in UPDP requires videos and task descriptions, whereas traditional MDPs do not require such data, so whether depends on what types of training data are available. Although the mon-Markovian model and the requirement of video and text data induce additional difficulties in UPDP comparing to MDP, it is possible to leverage existing large text-video models that have been pretrained on massive, web-scale datasets alleviate these complexities.
MDP UPDP is more suitable for given task
2.3 Diffusion Models for UPDP
Let r [ #H] H denote a sequence of images. We leverage the significant recent advances in diffusion models for capturing the conditional distribution P(T|20, c), which we will
3

---

leverage text and ffrmml conditioned video generator in UPDP. We emphasize that the UPDP formulation is also compatible with other probabilistic models, such as a variational autoen- coder [1 I. energy-based model [12, 13], or generative adversarial network [14]. For completeness we briefly cover the core formulation at high-level, but defer details to background references [15].
We start with an unconditional model. A continuous-time diffusion model defines forward process (T<Tx.T)
auT, where & [0, 1] and az: of are scalars with predefined schedules. A generative process p(7) is also defined, which reverses the forward process by learning a de- noising model s(Tk. k).. Correspondingly
can be generated by simulating this reverse process with an ancestral sampler [16] or numerical integration [17]. In our case, the unconditional model needs be further adapted condition on both the text instruction c and the initial image z0- Denote the conditional denoiser as s(T.. klc, zo). We leverage classifier-free guidance [18] and use 8(T,, kile, 2o) = (1 + ()s(TT, kilc, .o) ws(Tk: k) as the denoiser the reverse process for sampling, where w controls the strength of the text and first-frame conditioning.
3 Decision Making with Videos
Next we describe the proposed approach UniPi in detail, which is a concrete instan- tiation of the diffusion UPDP. UniPi in- corporates the two main components dis- cussed in Section 2, shown in Figure 2: (i) diffusion model for the universal video- based planner p(- c), which synthesizes videos conditioned on the first frame and task descriptions; and (ii) task-specific action generator ) J c), which infers actions sequences from generated videos through inverse dynamics modeling.
Universal Policy (UniPi)
Inverse Dynamies
Tillng
Text
Rotate
Video Diffasion
Robot Actions [Ax, 4y] Rolale - Aww
H
Temporal Super
Resolution
Inverse Dynomles
3.1 Universal Video-Based Planner
Figure 2: Given an input observation and text instruction, we
Encouraged by the recent success of text- to vo-video models [19], we seek to construct video diffusion module as the trajectory planner, which can faithfully synthesize future image frames given an initial frame and textual task description. However, the desired planner departs from the typical setting in text-to-video models [20, 19] which normally generate unconstrained videos given a text description. Planning through video generation is more challenging as requires models to both be able to generate constrained videos that start at specified image, and then complete the target task. Moreover, to ensure valid action inference across synthesized frames a video. the vidco prediction module needs to be able to track the underlying environment state across synthesized video frames.
plan set of images representing agent behavior. Images are converted to actions using an inverse dynamies model.
Conditional Video Synthesis. To generate valid and executable plan, a text-to-viee tovvo model must synthesize a constrained video plan starting at an initial image that depicts the initial configuration of the agent and environment. One approach to solve this problem is to modify the underlying test-time sampling procedure of an unconditional model, by fixing the first frame of the generated video plan to always begin at the observed image, as done in [21]. However, we found that this performed poorly and led to subsequent frames in the video plan to deviate significantly from the original observed image. Instead, we found more effective to explicitly train a constrained video synthesis model by providing the first frame of each video as explicit conditioning context during training.
Trajectory Consistency through Tiling. Existing text-to-video models typically generate videos where the underlying environment state changes significantly during the temporal duration [19]. To construct an accurate trajectory planner, it is important that the environment remain consistent across all time points. To enforce environment consistency in conditional video synthesis. we provide, as additional context, the observed image when denoising each frame in the synthesized video. In particular, we re-purpose temporal super-resolution video diffusion architecture, and provide as context the conditioned visual observation tiled across time, as the opposed to a low temporal-resolutee video for denoising at each timestep. In this model, we directly concatenate each

---

intermediate noisy frame with the conditioned observed image across sampling steps, which serves as strong signal to maintain the underlying environment state across time.
Hierarchical Planning. When constructing plans in high dimensional environments with long time horizons, directly generating set of actions to reach goal state quickly becomes intractable due to the exponential blow-up of the underlying search space. Planning methods often circumvent this issue by leveraging natural hierarchy planning. Specifically, planning methods first construct coarse plans operating on low dimensional states and actions, which may then be refined into plans in the underlying state and action spaces. Similar to planning, our conditional video generation procedure likewise exhibits natural temporal hierarchy. We first generate videos at coarse level by sparsely sampled videos ("abstractions") of the desired behavior along the time axis. Then we refine the videos to represent valid behavior in the environment by reppr-resol videos across time. Meanwhile, coarse-to-fine super-resolution further improves consistency via interpolation between frames.
Flexible Behavioral Modulation. When planning a sequence of actions to a given sub-goal, one can readily incorporate external constraints modulate the generated plan. Such test-time adaptability can be implemented by composing prior lu(T) during plan generation to specify desired constraints across the synthesized action trajectory [21], which also compatible with UniPi. In particular, the prior ((T)) can be specified using a learned classifier on images to optimize a particular task, or as Dirac delta on a particular image to guide a plan towards a particular set of states. To train the rext-conditioned video generation model, we utilize the video diffusion algorithm in [19], where pretrained language features from T5 [22] are encoded. Please see Appendix A for the underlying architecture and training details.
3.2 Task Specific Action Adaptation
Given a of synthesized videos, may train small task-specific inverse-dynamics model to translate frames into a set of actions as described below.
Inverse Dynamics. We train small model to estimate actions given input images. The training of the inverse dynamics is independent from the planner and can be done on a separate, smaller and potentially suboptimal dataset generated by simulator.
Action Execution. Finally, we generate an action sequence given Fo and e by synthesizing H image frames and applying the learned inverse-dynamic model to predict the corresponding H actions. Inferred actions can then be executed via closed-loop control, where we generate H new actions after each step of action execution (i.e., model predictive control), or via open-loop control, where we sequentially execute each action from the intially inferred action sequence. For computational efficiency, we use an open-loop controller in all our experiments in this paper.
4 Experimental Evaluation
The focus of these experiments is to evaluate UniPi in terms of its ability to enable effective, generalizable decision making. In particular, we evaluate
(1) the ability to combinatorially generalize across different subgoals in Section 4.1,
(2) the ability to effectively learn and generalize across many tasks in Section 4.2,
(3) the ability leverage existing videos the internet to generalize complex tasks in Section 4.3.
See experimental details in Appendix A. Additional results are given in Appendix B and videos in the supplement.
4.1 Combinatorial Policy Synthesis
First, we measure the ability of UniPi to combinatorially generalize to different language tasks.
Setup. To measure combinatorial generalization, we use the combinatorial robot planning tasks in [23]. In this task, a robot must manipulate blocks an environment to satisfy language instructions, i.e. pul red block right of cyan block. To accomplish this task, the robot must first pick up a white block, place it in the appropriate bowl to paint a particular color, and then pick up and place the block in plate that it satisfies the specified relation. In contrast to [23] which uses
5

---

Seen
Novel
Model
Place
Relation
Place
Relation
State + Transformer BC [24]
19.4 + 3.7
8.2 2.0
11.9 + 4.9
3.7 + 2.1 7.3 2.6
Image Transformer BC [24]
9.4 1 2.2
11.9 - 1.8
9.7 17740 4.5
Image TT [25]
17.4 2.9
12.8 + 1.8
13.2 4.1
9.1 i 2.5
Diffuser [21]
9.0
11.2
9
UniPi (Ours)
59.1 2.5
2.0
60.1
3.9
46.1 + 3.0
Table 1: Task Completion Accuracy in Combinatorial Environments. UniPi generalizes to seen and novel
combinations of language prompts in Place (c.g., place X in Y) and Relation (e.g.. place X to the left of Y) tasks.
Input Frame
Synthesized Frames
Block im the Brown Box
PutAn Orange Block Left of Red Block
Put Red Block on Purple Block
Figure 3: Combinatorial Video Generation. Generated videos unseen language goals test time.
pre- pre-prorrammed pick and place primitives for action prediction, predict actions in the continuous robotic joint space for both the baselines and our approach.
We split the language instructions in this environment into two sets: one set of instructions (70%) that seen during training, and another set (30%) that only seen during testing. The precise locations of individual blocks, bowls, and plates in the environment are fully randomized in each environment iteration. We train the video model on 200k example videos of generated language instructions the train set. Details of this environment can be found in Appendix A. We constructed demonstrations of videos in this task by using a scripted agent.
Baselines. We compare the proposed ap- proach with three separate representative ap- proaches. First, we compare to existing work that uses goal-conditioned transformers learn across multiple environments, where goals can be specified as episode returns [26]. expert demonstrations [6], or text and images [24]. To represent these baselines,
Synthoizud Frames
Bluck
Orange Block
construct trans- former behavior cloning (BC) agent to predict the subsequent action execute given the task description and either the visual observation (Im- age Transformer BC) or the underlying robot joint state (State Transformer BC). Second, given that our approach regresses sequence of actions to execute, we further compare with trans- former models that regress sequence of future actions to execute, similar to the goal-conditioned behavioral cloning of the Trajectory Transformer [25] (Image TT). Finally, to highlight the im- portance of the video-as- as-oocyy approach, we compare UniPi with learning diffusion process that, conditioned on an image observation, directly infers future robot actions in the joint space (as opposed to diffuusngg future image frames), corresponding [21, 27]. For both our method and each baseline, we condition the policy on encoded language instructions using pretrained T5 embeddings. Note that do in this setting, existing offline reinforcement learning baselines not directly applicable not have access to the reward functions in the environment.
Exceuted
Figure 4: Action Execution. Synthesized vidco plans and executed actions in the simulated environment. The two video plans roughly align with each other.
To compare UniPi with baselines, we measure final task completion accuracy across new instances of the environment and associated language prompts. We subdivide the evaluation along two axes: (1) whether the language instruction has been seen during training and (2) whether the
Metries.

---

Inpul Frame
Synthesized Frames
Put the Red Blocks Gray
Pack All the Poecelain Salid Plate Objects in the Brown Box
Pack and Blue Dlocks tnto the Brown Bax
Figure 5: Multitask Video Generation. Generated video plans test tasks in the multitask setting. UniPi
able to synthesize plan across a set environments.
Frame
Frame Consistency
Temporal Heirarchy
Condition
Place
Relation
No
No
No
13.2 - 3.2
12.4 + 2.4
Yes
No
No
52.4 -:
34.7 1 2.6
Yes
Yes
No
53.2 = 3.0
39.4 + 2.8
Yes
Yes
Yes
59.1 = 2.5
53.2 : 2.0
Table Task Completion Accuracy Ablations. Each component of UniPi improves its performance. Perfor- mance reported on the seen place and relation tasks.
language instruction specifies placing block in relation some other block opposed direct pick-and-place.
Combinatorial Generalization. In Table find that UniPi generalizes well to both seen and novel combinations of language prompts We illustrate our action generation pipeline in Figure and different generated video plans using approach in Figure 3.
we ablate UniPi on seen language instructions and in-relation-to tasks. Specifically, we study the effect of conditioning the video generative model on the first observa- tion frame (frame condition), tiling the observed frame across timesteps (frame consistency) and super-resolving video generation across time (temporal hierarchy). In settings where frame consistency is not enforced, we provide zeroed out image as context the non-start frames in video. We found that all components of UniPi are crucial for good performance. We found that frame conditioning and consistency enabled videos to be consistent with the observed image and temporal hierarchy enabled more detailed video plans.
Ablations. In Table 2,
Intarmaliate
Gridance
Symthesized l'mmes
Right
Cyan
Orange
Block
Put Left
Cyan Block
Orange
Figure 7: Adaptable Planning. By guiding test-time sampling towards intermediate image, adapt particular block.
planning procedure
Adaptability. We next assess the ability of UniPi adapt test time new constraints. In Figure 7, we illustrate the ability to construct plans which color and move one particular block to specified geometric relation.
4.2 Multi-Enviromment Transfer
We next evaluate the ability UniPi to effectively learn across a set of different tasks and generalize, at test time, to new set of unseen environments.
Setup. To measure multi-task learning and transfer, we use the suite of language guided manipula- tion tasks from [28]. We train our method using demonstrations across a set of 10 separate tasks from [28), and evaluate the ability of our approach to transfer to 3 different test tasks. Using scripted oracle agent, we generate set of 200k videos of language execution in the environment. We report the underlying accuracy which each language instruction completed. Details of this environment can be found in Appendix A.

---

Input Frame
Synthesized Frames
Flip Pot Upright in Sink
Turn Faucet Left
Pick Up Sponge Wipe Plate
Put Big Spoon from Basket Tray
Figure 6: High Fidelity Plan Generation. UniPi can generate high resolution video plans across different language prompts.
Place Bowl
Pack Object
Pack Pair
Model
State Transformer BC
9.8 2.6
21.7 3.5
1.3 0,9
Image + Transformer BC
5.3 1,9
5.7 2.1
7.8 2.6
Image + TT
4.9 2.1
19.8 + 0.4
2.3
1.6
Diffuser
14.8 + 2.9
15.9
+ 2.7
10.5 + 2.4
UniPi (Ours)
51.6 + 3.6
75.5
3.1
45.7
+ 3.7
Table 3: Task Completion Accuracy on Multitask Environment. UniPi generalizes well to new environments when trained on set of different multi task environments.
Baselines. We use the same baseline methods as in Section 4.1. While our environment setting is similar that of [28], this method is not directly comparable actions to the existing primitives of pick and place as opposed to using the joint space of robot. CLIPort is also designed solve the significantly simpler problem of inferring only the poses upon which to pick and place objects (with no easy manner to adapt to our setting).
approach, CLIPort abstracts
In Table 3 we present results of UniPi and baselines across new tasks. The UniPi approach is able to generalize and synthesize new vidcos and decisions different language tasks, and can generate videos consisting of picking different kinds of objects and different colored objects. We further present video visualizations of our approach in Figure
Multitask Generalization.
4.3 Real World Transfer
Finally we evaluate the extent to which UniPi can generalize to real world scenarios and construct complex behaviors by leveraging widely available videos the internet.
Setup. Our training data consists an intemet-scale pretraining dataset and smaller robotic dataset. The pretraining dataset uses the same data as [19], which consists of 14 million video-text pairs, 60 million image-text pairs, and publicly available LAION-400M image-text dataset. The robotic dataset is adopted from the Bridge dataset [29] with 7.2k video-text pairs, where we use the task IDs as texts. We partition the 7.2k video-text pairs into train (80%) and test (20%) splits. We pretrain UniPi the pretraining dataset followed by finetuning the train split of the Bridge data. Architectural details can be found in Appendix A.
Video Synthesis. We are particularly interested in the effect of pretraining on internet-ccale video data that is not specific to robotics. We report the CLIP scores, FIDs, and FVDs (averaged across frames and computed on 32 samples) of UniPi trained on Bridge data, with and without pretraining. As shown in Table 4, UniPi with pretraining achieves significantly higher FID and FVD and marginally better CL.I score than UniPi without pretraining, suggesting that pretraining on non-robot data helps with generating plans for robots. Interestingly, UniPi without pretraining often synthesizes plans that fail to complete the task (Figure 6), which is not well reflected in the CLIP

---

Synthesized Frames
Input Trame
Synthesized
hick Up Yellow Com (Scratch)
Pick Up Yellow Coca (Pratminaa))
Curno On Rumner (Scratc))
On Buncer (Prutiine)
Figure 8: Pretraining Enables Combinatorial Generaliza- Figure 9: Robustness to Background Change. tion. Internet pretraining enables UniPi to synthesize videos of tasks not seen in training. In contrast, model trained from scratch incorrectly generates plans of different tasks.
UniPi learns to be robust to changes of underlying background, such as black cropping or the addition of shoppe objects.
Model (24x40)
CLIP Score
FID
FVD 1
Success t
No Pretrain
24.43 + 0.04
17.75 + 0.56
288.02 + 10,45
72.6%
Pretrain
24.54 0.03
14.54 + 0.57
264.66 13.64
77.1%
Table 4: Video Generation Quality of UniPi on Real Environment. The use of existing data on the internet improves video plan predictions under all metrics considered.
score, suggesting the need for better generation metrics for control-specific tasks. To tackle the lack of such a metric, we develop a surrogate metric for evaluating task success from the generated videos. Specifically, we train success classifier that takes in the last frame of generated video and predicts whether the task is successful or not. We find that training a model from scratch achieves 72.6% success while finetuning from pretrained model improves performance to 77.1% %. In both settings, generated videos are able to successfully complete most tasks.
Generalization. We find that internet-scale pretraining enables UniPi to generalize to novel task commands and scenes in the test split not seen during training, whereas UniPi trained only on task- specific robot data fails to generalize. Specifically, Figure 8 shows the results of novel task commands that do not exist in the Bridge dataset. Additionally, UniPi is relatively robust to background changes such as black cropping or the addition of photo-shopped objects as shown in Figure 9.
5 Related Work
Learning Generative Models of the World. Models trained to generate environment rewards and dynamics that can serve as "world models" for model-based reinforcement learning and planning have been recently scaled to large-scale architectures developed for vision and language [9, 25, 30, 31]. These works separate learning the world model from planning and policy learning, and arguably present mismatch between the generative modeling objective of the world and learning optimal policies. Additionally, learning a world model requires the training data be strict action reward format, which is incompatible with the largely available datasets on the internet, such as YouTube videos. While methods such as VPT [32] can utilize internet-scale data through learning an inverse dynamics model to label unlabled videos, an inverse dynamics model itself does not support model-based planning or reinforcement learning to further improve learned policies beyond imitation learning. Our text-conditioned video policies can be seen as jointly learning the world model and conducting hicrarchical planning simultancously, and is able to leverage widely available datasets that are not specifically designed for sequential decision making.
Diffusion Models for Decision Making, Diffusion models have recently been applied different decision making problems [21, 27, 33, 34, 35, 36]. Most similar to this work, [21] trained an unconditional diffusion model to generate trajectories consisting of joint-based states and actions. and used a separately trained reward model to select generated plans. On the other hand, [27] trained a conditional diffusion model to guide behavior synthesis from desired rewards, constraints or agent skills. Unlike both works, which learn task-specific policies from scratch, our approach of text-condition video generation as a universal policy can leverage internet-scale knowledge to learn generalist agents that can be deployed to variety of novel tasks and environments. Additionally, [37]

---

applied web-scale text- eoxciiondt image diffusion to generate goal image to condition policy on, whereas our work uses video diffusion to learning universal policies directly.
Learning Generalist Agents. Inspired by the success of large-scale pretraining in vision and language domains, large-scale sequence and image models have recently been applied learning generalist decision making agents [6, 26, 38]. However, these generalist agents can only operate under environments with the same state and action spaces (e.g., Atari games) [26, 38], or require studious tokenization [6] that might seem unnatural in scenarios where different environments have distinct state and actions spaces. Another downside of using customized tokens for control is the inability to directly utilize knowledge from pretrained vision and language models. Our approach, on the other hand, uses text and images universal interfaces for policy learning so that the knowledge from pretrained vision and language models can be preserved. The choice of diffusion as opposed to autoregressive sequence modeling also enables long-term and hierarchical planning.
Learning Text- Coat--onditioeed Policies. There has been a growing amount of work using text commands as way to learn multi-task and generalist control policies [39, 40, 24, 41, 42, 43]. Different from our framing of indeo-as-policies existing work directly trains language-come connrgeeoondiiooeed control policy in the action space of some specific robot, leaving morssooopyyy multi entti-environmentt learning of generalist agents as an unsolved problem. We believe this paper is the first to propose images as universal state and action space to enable broad knowledge transfer across environments, tasks, and even between humans and robots.
6 Conclusion
We have demonstrated the utility of representing policies using text-conditioned video generation, showing that this enables effective combinatorial generalization, multi-task learning, and real world transfer. These positive results point to the broader direction using generative models and the wealth of data the internet powerful tools to generate general-purpose purpose decision making systems.
Limitations. Our current approach has several limitations. First, the underlying video diffusion process can be slow, it can take a minute to generate highly photorealistic videos. This slowness can be overcome by distilling the diffusion process into a faster sampling network [44], which in our initial experimentation resulted in 16x speed-up. UniPi may further be sped up with by faster speed samplers in diffusion models. Second, the environments considered in this work are generally fully observed. In partially observable environments, video diffusion models might make hallucination of objects or movements that are unfaithful or not the physical world. Integrating video models with semantic knowledge about the world may help resolve this issue, and the integration of UniPi with LLMs would be an interesting direction of future work.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. I
[2] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint 220.::22204002111,,. 2022.
[3] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages PMLR, 2021.
10

---

[5] Jean-Baptiste Alayrac, Jeff Donahue. Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. NeurIPS, 2022.
[6] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. generalist agent. arXiv preprint arXiv:2205.061 0675 2022. 1, 6, 10,
[7] Martin I. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
[8] Tuomas Haarnoja, Aurick 7hou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018. 3
[9] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193. 2020. 3,9
[10] Tianjun Zhang, Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Bo Dai. Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning, pages 26447-26466. PMLR, 2022. 3
[11] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
[12] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-baset models. arXiv preprint arXiv: (903.08689) 2019. 4
[13] Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. Exponential family estimation via adversarial dynamics embedding. Advances in Neural Information Processing Systems, 32, 2019. 4
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2014.
[15] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilidrium thermodynamics. In International Conference on Machine Learning, 2015. 4
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[17] Jamming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.00 20200000000 2020. 4
[18] Jonathan Ho and Tim Salimans. arXiv:2207.12598 2022.
Classifier-free diffusion guidance. arXiv preprint
[19] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02333, 2022. 4, 5,
[20] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399) 2022.
[21] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning, 2022. 4, 5, 6,
14
11

---

[22] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., (1(44))116) 2020. 5, 14
[23] Jiayuan Mao, Tomas Lozano-Pereet Joshua B. Tenenbaum, and Leslie Pack Kaelbing. PDSketch: Integrated Domain Programming. Learning, and Planning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[24] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06688 2022. 6, 10
[25] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021. 6, 9, 14
[26] Kuang- Huei Lec, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Ser- gio Guadarrama, Ian Fischer, Eric Jang, Hemryk Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205. 2022. 10, 14
[27] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision- arXiv preprint arXiv:2211.15657, 2022. 6,9 9
[28] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages
PMLR, 2022. 7,8 8
[29] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danilidiss, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109, 13396, 2021. 8
[30] Michael R Zhang. Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, ziyu wang. and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2021.
[31] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. arXiv preprint arXiv:2209.0000 00588, 2022. 9
[32] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint a7Xiv:2206.11795, 2022.
[33] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208. 06193, 2022. 9
[34] Edwin Zhang, Yujie Lu, William Wang, and Amy Zhang. Lad: Language augmented diffusion for reinforcement learning. arXiv preprint arXiv:2210. 15629, 2022. 9
[35] Julen Urain, Niklas Funk, Georgia Chalvatzaki, and Jan Peters. Se diffusiioooissss: Learning cost functions for joint grasp and motion optimization through diffusion. arXiv preprint arXiv:2209.03855, 2022. 9
[36] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer. Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. arXiv preprint arXiv:22 17366, 2022. 9
[37] Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffusion models to robotics. arXiv preprint 2rXiv:2210.024:: 2022. 9
[38] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline learning on diverse multi-task data both scales and generalizes. arXiv preprint arXiv:2211.15144
2022. 10
12

---

[39] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.055008 2022. 10
[40] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishman, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01000 2022. 10
[41] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.076443 2020. 10
[42] Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specitication via grounded natural language. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022. 10
[43] Hao Liu, Lisa Lee, Kimin Lee, and Pieter Abbeel. Instruction-followinng agents with jointly pre-trained vision-language models. arXiv preprint arXiv:2210.1344 2022. 10
[44] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005111 2022. 10
[45] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprini arXiv:2204.03456 2022. 14
[46] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 2205v 1487, 2022. 14
13

---

A Architecture, Training, and Evaluation Details
A.1 Video Diffusion Training Details
We use the same base architecture and training setup [45] which utilizes Video U-Net architecture with 3 residual blocks of 512 base channels and channel multiplier [1. 4), attention resolutions [6, 12, 24], attention head dimension 64, and conditioning embedding dimension 1024. We noise schedule log SNR with range [-20, 20J. We make modifications Video U-Net to support first-frame conditioning during training. Specifically, we replicate the first frame to be conditioned future frame indices, and apply temporal super resolution model condition on the replicated first frame by concatenating the first frame channel-wise the noisy data similar [46]. We temporal convolutions as opposed to temporal attention to mix frames across time, to maintain local temporal consistency across time, which has also been previously noted in [19]. We train each of our video diffusion models for 2M steps using batch size 2048 with learning rate lc-4 and 10k linear warmup steps. We use 256 TPU-v4 chips for our first-frame conditioned generation model and temporal super resolution model.
at all
We use T5-XXL [22] process input prompts which consists of 4.6 billion parameters. For combinatorial and multi-task generalization experiments on simulated robotic manipulation, we train first-frame conditioned video diffusion models on 10x48x64 videos (skipping every 8 frames) with 1.7B parameters and temporal super resolution of 20x48x64 (skipping every 4 frames) with 1.7B parameters. The resolution of the videos are chosen so that the objects being manipulated (e.g.. blocks being moved around) are clearly visible in the video. For the real world video results, we finetune the 16x40x24 (1.7B), 32x40x24 (1.7B), 32x80x48 (I.4B), and 32x320x 192 (1.2B) temporal super resolution models pretrained on the data used by [19].
A.2 Inverse Dynamics Training Details
UniPi's inverse dynamics model is trained to directly predict the 7-dimensional controls of the simulated robot arm from an image observation mean squared error. The inverse dynamics model consists of 3x3 convolutional layer, 3 layers of 3x3 convolutions with residual connection, mean-pooling layer across all pixel locations, and an MLP layer of (128, 7) channels to predict the final controls. The inverse dynamics model is trained using the Adam optimizer with gradient norm clipped at and learning rate le-4 for total of 2M steps where linear warmup is applied to the first 10k steps.
A.3 Baselines Training Details
We describe the architecture details of various baselines below. The training details (e.g., learning rate, warm up, gradient clip) of each baseline follow those of the inverse dynamics model detailed above.
Transformer BC [6, 26]. We employ the same transformer architecture as the 10M model of [26] with 4 attention layers of heads each and hidden size 512. We apply layers of 3x3 convolution with residual connection to extract image features, which, together with T5 text embeddings, are used as inputs to the transformer. We additionally experimented with vision transformer style linearization of the image patches similar to [26]. but found the performance to be similar. We use a context length of 4 and skip every 4 frames similar to UniPi's inverse dynamics. We tried increasing the context length of the transformer to 8 but it did not help improve performance.
Transformer TT [25]. We use similar transformer architecture as the Transformer BC baseline detailed above. Instead of predicting the immediate next control in the sequence as in Transformer BC, we predict the next 8 controls (skipping every 4 controls similar other baselines) the output layer. We have also tried autoregressively predicting the next 8 controls, but found the errors to accumulate quickly without additional discretization.
State-Based Diffusion [21]. For the state-based diffusion baseline, we use similar architecture as UniPi's first-frame conditioned video diffusion, where instead of diffusing and generating future
14

---

image frames, we replicate future controls across different pixel locations and apply the same U-Net structure as UniPi learn state-based diffusion models.
A.4 Details of the Combinatorial Planning Task
In the combinatorial planning tasks, we sample random DOF poses for blocks, colored bowls, the final placement box. Blocks start off uncolored (white) and must be placed in bowl obtain color. The robot then must manipulate and move the colored block to have the desired geometric relation in the placement box. The underlying action space of the agent corresponds values of robot plus a discrete contact action. When the contact action is active, the nearest block on the table attached the robot gripper (where for methods that predict continuous actions, thresholded action prediction >> 0.5 to correspond to contact). Given individual action predictions for different models, we simulate the next state of the environment by running the joint controller in Pybullet to reach the predicted joint state (with timeout of 2 seconds due to certain actions being physically infeasible). As only a subset of the video dataset contained action annotations, we trained the inverse-dynamics model on action annotations from 20k generated videos.
joint
A.5 Details of the CLIPort Multi-Enviree hutti-Environment Task
In the CLIPort environment, we use the same action space the combinatorial planning tasks and execute actions similarly using the built in joint controller in Pybullet. As our training data, we use
scripted agent on put-
seq, aasaanlllng-ttsssss--sseee--ooo
org.reacco.......aaaaaa.aaeeaeee tower-of-
san org.aaa.e..a..........oooeeeeeer packing-sene google-objects-seg
objects-group. As our test data, we used the environments htw.://eeeee.....i-ooooooooeeeee sasseen-goole--ne trained the inverse dynamics action annotation across the 200k generated videos.
group, berkingg-bxess www.s-ppirr-uuseen--oooo
We
15

---

B Additional Results
B.1 Additional Results on Combinatorial Generalization
Input
Synthesized Framess
Por A Brown on An Orange Block
Pot A Red Block Right of Orange Block
Put Yelloo block the Brown
Figure 10: Combinatorial Video Generation. Additional results UniPi's generated videos for unseen language goals test time.
B.2 Additional Results on Multi-Environment Transfer
Input Frme
Synthesize Frimes
Put the Giay Blocks
in Brown Bowl
Pack All the Purple
Red Blocks into the Brown
Pack All the Papei
Max Box Objects
Figure 11: Multitask Video Generation. Additional results UniPi's generated video plans different new tasks in the multitask setting.
16

---

B.3 Additional Results on Real- World Transfer
Input
Symthesized Frumes
Close small box Eaps
Lift bowl
Put potato on plate
Put distractors
Tum lever vertical
Figure 12: High Fidelity Plan Generation. Additional results on UniPi's high resolution video plans across different language prompts.
17


Reviews:
[{"review_id": "D3Twu7ZrxW", "reviewer": "Reviewer_voju", "summary": "A general framework, the Unified Predictive Decision Process (UPDP), was proposed in this paper. It leverages images as a universal interface, texts as reward specifiers and an independent planning module for policy synthesis. Powerful diffusion model was adopted into the framework of UPDP to generate authentic future video frames and inverse dynamics was used to generate downstream policies.", "questions": "1. In Figure 7, are there images generated by the diffusion model or videos taken during the action excution? The reflection on the microoven in \"Turn Faucet Left\" row looks too consistent to be something generated by diffusion while this reflection is not shown in the input frame.\n\n", "limitations": "Authors talked about the potential improvement of generation speed for diffusion model part.\n\nIn general, I think this is a solid and intriguing work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written, clear and easy to understand.\n2. A very intriguing introduction of UPDP framework. UPDP enables better utilization and knowledge transfer of large-scale generative pretraining which may demonstrate a broader impact in the future applications.\n3. Extensive experiments demonstrate the effectiveness of the proposed method, including high-quality video generation,  combinatorial generalization and multi-environment transfer.", "weaknesses": "1: Diffusion model: The adapation of diffusion model to UPDP lacks novelty. Temporal superresolution and tiling the context frame are commonly used tricks in video diffusion model. Although this can be regarded as \"simple yet effective\", this paper can be further improved if authors can include more domain specific design into the instantiation of UPDP with diffusion models.\n\n2. Metrics: For table 4, first there is a typo in line 302 saying \"higher FID and FVD\" while actually it's lower FID and FVD. Besides, FID and FVD are also not informative in this case because they only measure the distance between the distribution of generation and real data, which cannot tell if the model follow the text prompt correctly. Beyond training a classifier, a better metric could be measuring IoU of bounding boxes between groundtruth and generations, which are produced by a pretrained object detector, or human evaluation.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review. Please find our response below.\n\n> Novelty of diffusion models for UPDP\n\nWe agree that frame conditioning and temporal super-resolution are not new in video diffusion models, but adapting them to control and hierarchical planning have not been done before. To our knowledge, UniPi is the first to extend video diffusion to perform planning in the image space, and the implications are significant given the continual development of text-to-video foundation models and the ability to leverage internet-scale video data to improve decision making. Furthermore, we note that the introduction of UPDP as a more practical alternative to the more restrictive MDP is also novel. We will update the manuscript to include these discussions.\n\n> IoU metric for evaluation\n\nThank you for the suggestion. We will try this evaluation metric for the final version of the manuscript, although we suspect that using IoU of bounding boxes might have more false negatives due to the stochasticity of the generated plans (i.e., there are multiple ways to complete a task and the final frame might look different between the ground truth and generated plans, despite generated plans also complete the task).\n\n> Figure 7 clarification\n\nThe videos in Figure 7 are generated by the diffusion model. We have attached a few more generated videos conditioned on the same first frame and noted the generated videos are diverse.\n\n> Typos on line 302\n\nWe have updated the manuscript to fix the typo.\n"}, {"review_id": "ClFS9xqB9n", "reviewer": "Reviewer_eKLN", "summary": "The authors utilize the enhanced capabilities of text-guided image synthesis, a recent advancement in deep learning, to engineer general-purpose embodied agents capable of sequential decision-making. \n\nThe proposed method involves using language instructions as inputs to a text-conditioned video generation model, specifically video diffusion. Actions are then inferred from the generated video using a learned inverse dynamics model. \n\nAs agent learning from video play has been a well-established approach in reinforcement learning and imitation learning for complex tasks, translating language instructions into video for general-purpose agents can be viewed as a universally applicable strategy. This approach addresses the challenges of diversity and heterogeneity in agent environments, offering a broader, more versatile solution that can leverage also pretrained large-scale language and language-video models.", "questions": "Could the authors explain the benefits of the proposed method, specifically in comparison to other instruction-following agents that can use pretrained multi-modal models?\n\nIf each video frame generated can be seen as sub-goal representations in the task, and each sub-goal is dealt by the inverse dynamics model, hierarchical planning in Section 3.1 seems to be critical for the effective handling of long-horizon tasks. Could the authors more describe the procedure of sampling videos? \n\nIn line 215, the open-loop controller is chosen for computation efficiency. Could the author explain more? What if the stochastic environment has some randomness? \n\nCould the authors explain why Transformer-BC and TT achieve low performance in the experiments?\n\nIn Figure 5, the adaptable planning scenario is not clear. \n", "limitations": "The conclusion has specified the limitations including computation load of video diffusion.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper demonstrates how to harness the potential of text-guided image synthesis for agent learning, providing evaluations through multiple experimental scenarios. \n\nThe paper is insightful, and it can offer several meaningful perspectives for readers interested in using LLMs and multi-modal pretrained models for agent learning. ", "weaknesses": "The proposed method in this paper can be seen novel, marking its originality, and showcasing insights on the utilization of multi-modal models.  However, I also think that the method description in Section 3.1 falls short on illustrating clear technical contributions. It seems to only incorporate the existing diffusion model and the inverse dynamics model in the proposed framework. \n\nDespite the innovative nature of the broader concept, a more detailed explanation and comprehensive analysis regarding how to improve and generalize the proposed method can enhance the clarity and overall influence of the paper; e.g., comparing with language instruction-following agents with multi-modal capabilities, and comparing with skill-based hierarchical RL approach that generate latent skill sequences (that can be also translated into actions later). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback on our work! We address your questions as follows.\n\n> Highlighting contribution in method section\n\nWe will update our method sections to highlight the contributions, which includes (1) how to re-purpose text-to-video models designed for media and entertainment to be a useful tool for control through frame-conditioning, (2) how to overcome difficulties around generating consistent frames across time, and (3) how to conduct hierarchical planning effectively. We believe that these contributions indeed address challenging problems in both video diffusion models and planning and control.\n\n> Procedure of sampling videos to address temporal hierarchy and long-horizon planning\n\nGenerating videos hierarchically at the right granularity across time is indeed important to ensure UniPi’s performance. To sample a video plan in the simulated robotics experiments, we first sample 10 frames with a larger frame-skip between each frame (8). Then conditioned on these 10 frames, we fill in the intermediate frames, resulting in 20 frames with frame-skip 4. We found frame-skip 4 results in the right granularity for training an effective inverse dynamics model.\n\n> Advantage over language instruction following agent pretrained on multimodal data\n\nMost instruction follow agents using multimodal data are typically initialized from pretrained VLM models. These models are typically trained with captioned image and text-pairs, but typically do not contain much motion / physics information about the environment. In contrast, in our approach, we can train on a wealth of existing language annotated video data, which captures much more information about how to act in environments and the physics of the environments. This allows our approach to transfer world-knowledge about how to do particular tasks, such as precise visual motions to open a door handle which are not available in pretrained VLM models.\n\n\n> Open loop control\n\nA limitation of UniPi is the high computational cost of video diffusion to generate a video plan, making it costly to perform open loop control. In stochastic environments, we could choose to regenerate video plans in the case in which observations do not match the ones in our plan,  or wecan batch sample video plans ahead of time switch between generated plans to address changes in the environment. There is also a wealth of work on improving the video sampling speed of diffusion models.\n\n> Poor performance of Transformer-BC and TT\n\nWe believe the poor performance of Transformer-BC and TT is due to the long task horizon of demonstrations and the inability of these agents to accurately synthesize and follow the long horizon plans in this environment. As the different steps in the environment are executed, errors accumulate in the observation space and the agents fail to correctly finish the task.\n\n> Figure 5 adaptable planning clarification\n\nFigure 5 shows that in addition to using text to guide plan generation, UniPi can further utilize intermediate frames by fixing a particular future frame during sampling to guide generated plans towards moving either the left or the right block. We have updated the caption to make this clear. Please let us know if you have additional confusion."}, {"review_id": "ndoBZoOyMA", "reviewer": "Reviewer_VoKS", "summary": "This paper frames the sequential decision-making problem as a text-conditioned video generation problem. Given a text-encoded specification of a desired goal and the first frame with the initial configuration, a planner generates a set of future frames that depict planned actions. The generated video is then used to extract control actions. This framework facilitates leveraging pretrained language embeddings and large-scale internet videos to enable combinatorial generalization and knowledge transfer across diverse tasks.", "questions": "1) I wonder how much of the combinatorial generalization exhibited by the model is dependent on the quality of the text embedding. For example, as shown in [1], using something like CLIP language embeddings can make it difficult to capture more abstract notions like \"left\" or \"right\" relative spatial positioning, affecting combinatorial generalization in some of the tasks shown in Section 4.1. Could the authors include an sensitivity analysis on the T5 embeddings?\n\n2) The authors claim this approach allows for more universality when it comes to environment diversity or reward specification. But it seems to me that the inverse dynamics model discussed in Section 3.2 would have to be retrained for different tasks anyways. How do the parameter/data scales for this (smaller) model compare to the diffusion component? \n\n3) How does performance for combinatorial generalization in Section 4.1 scale compared to something like BC? i.e. is there is a ceiling to training inverse dynamics on generative data -- could the authors demonstrate how performance scales with number of examples compared to something like [1] or [2]?\n\n[1] [Programmatically Grounded, Compositionally Generalizable Robotic Manipulation](https://arxiv.org/abs/2304.13826)\n\n[2] [CLIPort: What and Where Pathways for Robotic Manipulation](https://arxiv.org/abs/2109.12098)", "limitations": "Yes, the authors have discussed some limitations in their concluding remarks.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) Casting sequential decision-making as an synthesis problem naturally allows us to leverage a growing wealth of existing research in large-scale image/video/language generative models that encapsulate valuable world models for robotics.\n\n2) Strong experiments: both simulation and real-world tasks are included.\n\n", "weaknesses": "1) Some overlap with prior work: the bulk of the framework seems to borrow heavily from Decision Diffuser[1]. Specifically, [1] also uses classifier-free guidance diffusion to generate a sequence of states, followed by some inverse dynamics modeling to identify the action to execute. The main innovations of the work are centered around substituting engineering components in prior works; e.g. replacing unconditional generation from something like Diffuser[2] with conditional generation\n\n[1] [Is Conditional Generative Modeling all you need for Decision Making?](https://arxiv.org/abs/2211.15657)\n\n[2] [Planning with Diffusion for Flexible Behavior Synthesis](https://arxiv.org/abs/2205.09991)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments – please see our response below. Feel free to let us know if you have additional questions or comments.\n\n> Overlap with Prior Work.\n\nWe believe the primary novelty of our work over past work such as Decision Diffuser is the construction of a large-scale model for decision making that can be directly learned from Internet data. While Decision Diffuser, similar to our work, uses diffusion models, in combination with classifier free guidance, to predict states and actions, it assumes the presence of existing datasets of states and actions in a domain of interest. In contrast, our work is able to use the existing video information on the internet to learn to make decisions, by casting the decision making problem is that of generating a video of actions to execute conditioned on a text description of the actions you wish to take.\n\n> Effect of Language Embedding on Combinatorial Generalization.\n\nWe ran an ablation experiment where we tested the combinatorial generalization ability of our approach with different sizes of language embeddings from T5. While we found that different sizes of language embeddings did really affect the video quality, we indeed found that larger language embedding sizes substantially improved combinatorial generalization (as measured by CLIP score on Bridge).\n\n| | CLIP| FID|\n| ---- | ---- | ---|\n|T5_small  |   22.75  |    15.53  |   \n|T5_large  |   22.78   |   16.07   | \n|T5_XXL   |   24.54   |   14.54   | \n|T5_XXL (No Pretrain)   |   24.43  |   17.75   | \n\n\n> Inverse Dynamics Model.\n\nThe inverse dynamics model is very small (<1M parameters) and trained on substantially less resources (1 TPU for 12 hours)  while the video model which is large (5B parameters) and trained on substantially more resources (256 TPUs for several days). While the inverse dynamics would need to be trained per environment, the model only needs to learn to predict the action that can transition between two frames which is substantially easier to learn.\n\n> Scaling of Combinatorial Generalization.\n\nIn principle, we believe that both our text-conditioned video generation approach and language conditioned BC would scale well with increased amounts of data. However, while there are plentiful amounts of videos on the internet (and many commercial large text-to-video models like Gen2 that demonstrate very good combinatorial generalization) there is much less labeled action data, making it much easier to get a combinatorially generalized version of our approach. \n\nWe don’t believe there is a limit to training inverse dynamics models on generated data and believe it's actually much easier to train an inverse dynamics model than a BC policy that generalizes well as also theoretically shown in [1]. This is because the inverse dynamics model only has a task to  infer an action given both present and future states while a policy must anticipate the next action to take to maximize reward across all future states [1]. If the reviewer desires, we are happy to add an additional experiment demonstrating scaling of the performance of the inverse dynamics model with number examples in the final version of the paper. \n\n[1] Brandfonbrener et al. Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation\n"}, {"review_id": "5tsjdFEl7F", "reviewer": "Reviewer_5Y9t", "summary": "This paper introduces a novel approach to text-conditioned video generation task  by treating it as a goal-conditioned RL, where the text is formulated as the goal and the multi-step image sequences in the video are the consecutive observations. The proposed method, termed Unified Predictive Decision Process (UPDP), aims to learn effective control strategies. Compared to traditional MDPs, the sequences in UPDP are determined by both the initial frame and the task description. This enables bypassing reward design and facilitates non-Markovian modeling. To model decision-making within UPDP, this paper introduces UniPi, which consists of two key components: i) a diffusion model planner and ii) a task-specific action generator. Empirical evaluations on various control tasks demonstrate the effectiveness of the proposed method.", "questions": "1. The specific difference between the super-resolution and low-temporal-resolution architectures mentioned in lines 179-181 appears to be that the former employs the observation tiled as a condition, while the latter does not. It is unclear whether the super-resolution and low-temporal-resolution architectures refer to the hierarchical structure mentioned in lines 184-194. To enhance clarity, I suggest the authors restructure lines 174-183 accordingly. Alternatively, if these architectures are not the hierarchical structure, the authors should provide a clear explanation or visual representation, such as a model figure, to elucidate the architectures for improved readability.\n\n2. In Appendix A.3, it is unclear whether \"replicate future controls across different pixel locations\" serves the purpose of concatenating state-action pairs. Further clarification is needed to understand the specific role and functionality of this mechanism.\n\nThe other questions are above.\n\n", "limitations": "As the authors showed, the video diffusion process is very slow, even with some fast sampling methods. It's not suitable for real-time robotic control for now, while might be an interesting domain to explore in future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is clear to follow. \n2. The proposed method is effective and seems reasonable.\n3. A large number of experiments showing performance gain.\n4. Synthesized Frames are illustrative and interesting.", "weaknesses": "1. While the high-level idea presented in this paper is intuitive and reasonable, the detailed descriptions of certain specific modules require further emphasis. For instance, the video-based planner $\\rho(\\cdot|x_0,c)$ is employed to generate image sequences (or videos). However, the role of the action in this context is not clearly defined. If the planner is unrelated to the action, it raises questions regarding how the action can influence the state. From the statement in lines 115-116 showing that \"This design choice isolates planning decisions from action-specific mechanisms, allowing the planner to be environment and agent agnostic\", it seems suggest that this module is indeed a forecaster rather than a planner.\n\n2. Text-conditioned generation has gained recent popularity. Although the paper states that planning through video generation poses challenges due to the need for specific initial images to complete tasks, guided generation (conditioned on an initial image and a language description) is a common choice that has been extensively studied. Similarly, inverse dynamics is also a well-explored method that has been proven effective. Therefore, the combination of these techniques may not provide significant novelty from a model design perspective.\n\n3. Some details in the paper may lead to misunderstandings. For example, in section 3.1, the author proposes tiling as a means to ensure trajectory consistency. However, two questions arise: i) Does tiling involve concatenating the generated frame with the initial image of the video or the ground-truth image at a specific timestep? ii) How exactly does tiling provide trajectory consistency? I recommend the authors extend this paragraph with more details.\n\n4. Figure 2 appears somewhat irregular, as the widths of the noises applied to video diffusion are equal, whereas the noises applied to Temporal Super Resolution vary in width. Furthermore, the outputs of the video diffusion module also exhibit different widths. Ensuring consistency in the widths of these elements would enhance the clarity of the figure.\n\n5. The disentanglement between the planner module and the policy network is presented as a strength of the model. However, it remains unclear how the policy network can impact the planner or interact with the environment in such cases. Section 3.2 attempts to explain this relationship, but the details provided are not easy to follow. It is hypothesized that the video planner solely provides synthesized frames for reference (denoted as $\\bar{o}$). However, it is unclear whether, when the environment provides an observation $o$, the inputs for inverse dynamics should be $(o_t, \\bar{o}_{t+1})$ or $(\\bar{o}\\_t, \\bar{o}\\_{t+1})$. Further clarification is needed in this regard.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments – please see our response below. Feel free to let us know if you have additional questions or comments.\n\n> Detailed Description of Modules.\n\nWe will clarify the detailed description of the modules. Our video planner seeks to construct a sequence of states that represent a plan going from our original start state to the desired final goal state. Since the planner only has to generate a set of feasible states transitions to go from the start to goal state, it does not need to worry about the precise actions to generate to reach each feasible state, which can environment specific (i.e. some environments require you to execute a sequence of actions to open a door while other let you go directly through the door).\n\n> Novelty of Text-Conditioned Generation. \n\nMost existing works in text-to-video generation have focused primarily on their application to AI-CG and most large scale text-to-video models (Phenalki, Make-A-Video, Imagen-Video) typically only generate frames conditioned on natural language. The primary novelty of our approach is formulating the problem of acting as a text-to-video generation problem, where given a language description of goal, we synthesize a set of frames of video denoting how we will act. This further requires some domain specific designs, such as how to condition video generation on observed images, where we propose trajectory observation tiling.\n\n> Tiling to Ensure Trajectory Consistency.\n\nThe tiling referred to in this section corresponds to tiling the first observed image to each generated frame in the video. This operation ensures that the background details in the first observed image are more readily captured across each frame of the synthesized video as when denoising each frame in the video, the full conv architecture sees (starting with the first convolution) the direct concatenation of an intermediate noisy video with the first observed frame of the video. We will clarify this in the paper.\n\n> Figure 2.\n\nWe have attached a modified version of Figure 2 with equal width in image outputs to the main rebuttal response PDF.  Feel free to let us know any other modifications we can do on the figure to improve clarity.\n\n> Clarification on Inverse Dynamics.\n\nTo infer an action using the inverse dynamics model, we take as input $o_t, \\overline{o}_{t+1}$, the current observed frame and the next synthesized frame. We will clarify this in the paper.\n\n> L179-L181.\n\nThe trajectory consistency through observation tiling section refers to the base video diffusion model in Figure 2, where we want to synthesize a low resolution video given the observed image and text-instruction. The temporal super-resolution model we refer to is not related to the one discussed between L184-L194, but rather the temporal super-resolution architecture from Imagen Video. We will clarify this reference in the main paper.\n\n> Baseline Details.\n\nIn our state based diffusion model, we want to diffuse a sequence of states to accomplish the instructed language task. To apply our video-diffusion model architecture to this setting, we take each D dimensional state and tile it to form a HxWxD image. We then train a video diffusion to diffusion on this “state” video and use this to regress our future states to execute. We will include these details in the paper.\n"}]

Author rebuttal:
We thank reviewers for their positive reviews and feedback. Reviewers noted that the paper was well written (Reviewer voju, 5Y9t), insightful (Reviewer eKLN), and had strong experiments (Reviewer VoKS). Reviewers voju, VoKS, 5Y9t had some concerns about novelty which we address below.

## Novelty

The main novelty of our work lies in our formulation that enables us to leverage internet video as an effective data modality for decision making. Most recent works on leveraging internet data have used multimodal models, primarily VLMs, are trained on captioned image-text pairs, to construct agents. While such agents will have rich prior knowledge about the underlying semantics of individual images, they lack knowledge about motion of objects or their physics, and for example would not know how visually one should go around opening something like a door handle. In contrast, learning from video allows us to learn the motions of people and the world, enabling us to have prior knowledge about the mechanisms of opening a door, or what types of object dynamics or movement are possible.

To leverage internet video for decision making, we propose the UPDP decision making formulation, which casts decision making in different environments to be very closely related to video generation, enabling us to transfer this wealth of knowledge. While our resultant instantiation of UPDP shows some similarity to Decision Diffuser and existing video models, these are particular instantiations of our approach, and our formulation to could be also applied to newly discover or other existing generative models of video. 


Meta-review: The paper provides a technique that use generative model with diffusion models to perform planning. The scores from all reviewers are positive and most reviewers find the approach interesting and the results are convincing. The AC agrees with the reviewers on accepting the paper.
Decision: Accept (spotlight)