Video Language Planning
Authors: Yilun Du
Sherry Yang
Pete Florence
Fei Xia
Ayzaan Wahid
brian ichter
Pierre Sermanet
Tianhe Yu
Pieter Abbeel
Joshua B. Tenenbaum
Leslie Pack Kaelbling
Andy Zeng
Jonathan Tompson
Venue: ICLR 2024 poster (2024)
Keywords: Planning
Hierarchical Planning
Language Models
Video Models
Long-Horizon Planning
Primary area: applications to robotics, autonomy, planning
TL;DR: We present a way to solve long horizon decision making tasks by planning through the space of video and language.

Abstract: We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data.  To this end, we present video language planning (VLP), an algorithm that consists of a tree search procedure, where we train (i) vision-language models to serve as both policies and value functions, and (ii) text-to-video models as dynamics models. VLP takes as input a long-horizon task instruction and current image observation, and outputs a long video plan that provides detailed multimodal (video and language) specifications that describe how to complete the final task. VLP scales with increasing computation budget where more computation time results in improved video plans, and is able to synthesize long-horizon video plans across different robotics domains -- from multi-object rearrangement, to multi-camera bi-arm dexterous manipulation. Generated video plans can be translated into real robot actions via goal-conditioned policies, conditioned on each intermediate frame of the generated video. Experiments show that VLP substantially improves long-horizon task success rates compared to prior methods on both simulated and real robots (across 3 hardware platforms).

Published as a conference paper at ICLR 2024
VIDEO LANGUAGE PLANNING
Yilun Du 8 Mengjiao Yang', Pete Florence', Fei Xia', Ayzaan Wahid', Brian Ichter', Pierre Sermanet', Tianhe Yu', Pieter Abbeel', Joshua B. Tenenbaum', Leslie Kaelbling Andy Zeng'. Jonathan Tompson
Google Deepmind", Massachusetts Institute of Technology UC Berkeley https: language-plante ing. github. .io/
ABSTRACT
We are interested in enabling visual planning for complex long-horizon tasks the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data. To this end, we present video language planming (VLP), an algorithm using tree search procedure, where we train (i) vision-language models to serve as both policies and value functions, and (ii) text-to-video models as dynamics models. VLP takes as input long-horizon task instruction and current image observation, and outputs long video plan that provides detailed multimodal (video and language) specifications that describe how to complete the final task. VLP scales with increasing computation budget where more computation time results in improved video plans, and is able synthesize long-horizon video plans across different robotics domains from multi- object rearrangement, to multi-camera bi-arm dexterous manipulation. Generated video plans can be translated into real robot actions via coal-connitionee policies, conditioned on each intermediate frame of the generated video. Experiments show that VLP substantially improves long- long-horizon task success rates compared to prior methods on both simulated and real robots (across 3 hardware platforms).
INTRODUCTION
Intelligently interacting with the physical world involves planning over both (i) high-level semantic abstractions of the task (i.e., what do next), well the (ii) low-level underlying dynamics of the world (i.e.. how the world works). Factorizing the planning problem into two parts, one driven by task-specific objectives and the other task-agnostic modeling of state transitions, is an idea that is pervasive and fundamental. This factorization drives much of the classic work in robotics from integrating task and motion planning (Cambon et al., 2009; Wolfe et al., 2010; Kaelbling & Lozano- Pérez, 2011) 1) to deriving control policies that can perform complex manipulation tasks over long time horizons such as tidying a dining table or rearranging collection of objects to build new structures.
Pre-trained large language models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022) have shown be capable of generating high-level step-by-step plans for long-horizon tasks over symbolic (often linguistic) abstractions of the task (Huang et al., 2022a; Ahn et al., 2022), but this is only part of the solution. LLMs are restricted by what they can represent in text, and struggle with grounding i.e.. reasoning over shapes, physics, and constraints of the real world (Tellex et al., 2020; Huang et al., 2023). L.MS can be integrated into larger vasion-language models (VLMs) (Driess et al., 2023) that, when trained on sufficient data, can respect physical constraints observed in image inputs to generate more feasible plans that may be less likely command the robot to perform impossible actions, or manipulate inaccessible objects. However, existing VLMs are predominantly trained on static image captioning and Q&A datasets - consequently, they continue to struggle to reason over dynamics e.g.. how objects may move or collide with one another over time.
Meanwhile, recent text-to-video models trained on the wealth of videos on the Internet (Villegas et al., 2022; Ho et al., 2022), have demonstrated an ability to learn the dynamics and motions of objects by synthesizing video predictions of the future (Du et al., 2023b). Existing video models can only generate short time horizon clips without losing visual fidelity, and whether they can be applied for long-horizar planning remains unclear. Nevertheless, they exhibit properties that are complementary to VLMs in that they (i) can model the low-level visual dynamics of objects in ways that are more information rich than text, and (ii) can absorb another source of Internet data e.g.. YouTube videos. This leads the natural question of how to build planning algorithm that can

---

Published as a conference paper at ICLR 2024
Video Language Planning
Long Horizon Visual Planning with Tree Search
VLM as Text Policles
make a horizontal Iine of blocks
Video Model Dynamics Model
image text actions
text Image - video rollouts of the future
PaLM-E
push red star groan
Text-to-Videe
vdea
See
Action 1, push rod star to left
Action 2. green
Action 3. move arm to yellow.. ..
Actions from Goal- Coal-Conditioned Policies
VLM as Heuristic Functions image ception steps left
Swp Eroeart
Sby Jeplet
Policies
PaLM-E
Experiments across Multiple Rabots
Execute plan from tree search
steps to
Figure I: Video Language Planning uses forward search via vision-language models and text-to-video models to construct long-horizon video plans. From image observation, the VLM policy (top left) generates next-step text actions, which a video model converts into possible future image sequences (top right). Future image states are evaluated using a VLM heuristic function (bottom left), and the best sequence is recursively expanded with tree search (middle). Video plans be converted action execution with goal-conditioned policies (bottom right).
leverage both long- hong-horizon abstract planning from LLMs/ / VLMs and detailed dynamics and motions from text-to- wwwev-idddeeeeee
In this work, we propose to integrate vision-language models and text-to-video models to enable video language planning (VLP), where given the current image observation and language instruction, the agent uses a VLM to infer high-level text actions, and a video model to predict the low-level outcomes of those actions. Specifically, VLP (illustrated in Fig. 1) synthesizes video plans for horizoorizon tasks by iteratively: (i) prompting the VLM as a policy to generate multiple possible next-step text actions, (ii) using the video model as dynamics model to simulate multiple possible video rollouts for each action, and (iii) using the VLM again but as a heuristic function to assess the favorability of each rollout in contributing task progress, then recursively re-p re-planning with (i).
The combination of both models enables forward tree search over the space of possible video se- quences discover long-hone horizoorizon plans (of hundreds of frames) that respect visual dynamics. partic- ular, VLP offers advantages in that it (i) can generate higher quality plans at inference time by expand- ing the branching factor of the search, allowing plan quality to scale with increasing compute budget, and (ii) benefits from training on incomplete language-labell horizon snippets (that can be re-composed and sequenced into videos with missing language labels (but contain dynamics that the video model can still learn from). Experiments in both simulated and real settings (on 3 robot hardware platforms) show that VLP generates more complete and coherent multimodal plans than baselines, including end-to-end models trained directly to generate long videos conditioned on long-horizon task instructions. VLPs exhibit improved grounding in terms of the consistency of scene dynamics in video plans, and when used in conjunction with inverse dynamics models or goal-conde condionnitio policies to infer control trajectories (Du et al., 2023b), can be deployed a variety of objects over countertop settings, to pushing groups of blocks that rearrange them into new formations. In terms of execution success, VLP-based systems are far more likely achieve task completion for long-horizon instructions than state-of-the-art alternatives, including PaLM-E (Driess al., 2023) and RT-2 (Brohan al., 2023) directly fine-tuned for long-horizon tasks. We also observe that when co-trained on Internet-scale data, VLP generalizes to new objects and configurations.
video data, which may contain short-
ones), or segments of
robots to perform multi-step tasks from picking and stowing
Our main contributions are: (i) video language planning, scalable algorithm for generating long- horizon video plans by synergising vision-language models and text-to-video models, (ii) experiments
2

---

Published as a conference paper at ICLR 2024
that show how VLP enables both simulated and real robots to perform complex long-horizon manipulation tasks, exhibiting task completion rates that often exceed that of the next best available approach by significant margin, and (iii) ablations that study modes of generalization, as well as how VI.P scales with more compute. VLP presents a modern re-examination of visual planning by integrating large generative models pretrained on Internet-scale data, and is not without limitations.
2 VIDEO LANGUAGE PLANNING
Our planning system, VLP, takes a visual observation Zy of a scene and a natural language goal g and infers video plan { h:r,, where each image 2; is sub-goal to accomplish g. We assume that each image Ze serves as an accurate representation of world state and use an image goal-conditioned policy controller infer low level control actions to reach each image state.
Below, we first discuss how vision-language and video models are used in VLP as planning sub- modules in Sec. 2.1. Next, we talk about tree-search algorithm using vision- language and video building blocks in See. 2.2. Finally, in Sec. 2.3, we discuss how we can convert video plans into policies accomplish each long-horizon task.
2.1 USING VISION-LANGUAGE AND VIDEO MODELS AS PLANNING SUBMODULES
We first discuss how to use vision- -lnguage and video models as sub- modules in VLP to synthesize long horizon plans. At high level, we use the multimodal processing power of VLMs to propose abstract text actions a' to execute given goals and images. We then use the dynamics knowledge of video models to accurately synthesize possible future world states zंr when abstract actions are executed. Finally, we use VLM to process possible future world states ri. and assess which sequence 21:7 and associated actions are the most promising to complete a task.
Vision-Language Models Policies. Given high-level goal g. VLP searches over a space of possible abstract actions; these text actions a are generated by a VLM policy TVLM 9) + that conditioned both on the goal g and an image I of the current state. We implement this policy following Driess et al. (2023) and query the VLM for natural language action to take given as context the natural language goal and a tokenized embedding of the current image (Fig. Top Left). We experiment with two different strategies for constructing this policy. In the first, we provide the VLM a set of example text action labels and ask the VLM to predict possible actions to accomplish a goal. In the second, we finetune the PaLM-E model on randomly selected short trajectory snippets 21.s labeled with abstract actions inside a long trajectory T1:H that accomplishes a long horizon goal g.
Video Models as Dynamics Models. In order to perform the high-level search, given an image of a current state and language description of an abstract action, we need to predict the concrete resulting state. In addition, to generate low-level controls that instantiate this abstract action, we need a feasible sequence of low-level states that "interpolate" between the current state and the resulting state. We obtain both of these things from text-to-vides model fv a), which takes image and a short horizon text instruction a and outputs a short synthesized video Ø1.s starting at the image observation zo (Fig. Top Right) following Du et al. (2023b). We construct this text-to- -ivvoo model by training on a set of short image trajectory snippets F1.7 and associated language labels a.
Vision-I LinguaeLanguage use a VLM to implement a heuristic function H (ar, g) which takes as input an image observation and natural language goal description g and outputs scalar "heuristic" predicting the number of actions required to reach a state satisfying goal from current state x (Fig. Bottom Left). To construct this heuristic function, which accomplish a long horizon goal g, and train it to predict, given an image in the subtrajectory Tr, the number of steps left until the end of the trajectory snippet. The negated number of predicted steps to goal completion from VLM is used to implement HVIM(E,)) g) (so that high heuristic value corresponds to being close to goal completion).
Language Models as Heuristic Functions. To effectively prune branches in search, we
finetune a PaLM-E model using long trajectory snippets 21:H
2.2 PLANNING WITH VISION-LANGUNE MODELS AND VIDEO MODELS
Given combination of modules discussed in Sec. 2.1, directly applying the TVLM to infer text actions a to reach goal is not sufficient, as TyLM is not able to perform sufficiently accurate reasoning select actions that are helpful in the long run. Furthermore, there many possible low-level image sub-sequences that correspond to different ways to perform o, but it is critical to select one that is consistent with the rest of the actions that must be taken to reach g.
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Decision Making with VLP
1: Input: Current visual observation za, Language goal g
2: Functions: VLM Policy TVLM (x. g). Video Model [var(z, a). VLM Heuristic Function (2LM (2,
3: Hyperparameters: Text- Branching factor A, Aideo-Branching factor D. Planning Beams B, Planning
horizon H
4: plans + [pra] V E {1 {1....B) BB
# Initialize B Different Plan Beams
5: for h = 1... H do
6: for b 1... Bo do
7:
x +- [1]ams[b[[[1]]
Get the Latest Image State in the Plan Beam # Generate A Different Text Actions
8: 01:4 <- (1,
9:
video iidenbbras <- [f, a.) fori (1... A) for j in (1...D)] happbbaapppedddaagaa//idooo.o branchhs, HVLM)) # Add Video with Highest Value to Plan
10:
end for
12: max idx, iix + aramax(plans, HVLM), argmin(plans, HVLM)
13:
plans[min_idy) idx] < plans[max idx]
# Periodically Replace the Lowest Value Plan
14:
end for
15:
plan 4- argmax(plans, HVLM)
# Return Highest Value Plan
Instead, we propose to search for a sequence of actions to reach g, corresponding to finding a hong-horizon video plan F1:H which optimizes
Fi.H
arg max HVLM(ZH, g).
(I)
F1.21 FVL.
A separate procedure is then used to instantiate control actions to enact the optimized video plan Fi... To sample long-horizon video plans #1 we first synthesize a short horizon video plan #1.s from starting image through 21.5 fvm TVLM g)) and autoregressively extend to a full long-horizon video plan by recursively applying (VM TVLM (I, g)) on the final synthesized image state Fg. To optimize across video plans in Eqn (1), we
rea-he procedure based parallel hill climbing (Selman & Gomes, 2006) (illustrated in Algorithm 1).
Our planning algorithm initializes set of B parallel video plan beams. At each step of the planning horizon, for each video beam, we first sample a set of actions using TVLM(I, g). and for each action we synthesize D different videos using fv a). We then use our heuristic function HVLM( (x,, .9) to select the generated video with the highest heuristic among the A x D generated videos and extend the corresponding video plan beam with this generated video. Over the course of plan generation, certain video plan beams will obtain high heuristic value and be more promising to explore. Therefore, every 5 steps, we discard the beam with the lowest value and replicate its video plan with the beam with the highest value. Our final full long horizon video plan corresponds to the beam with highest heuristic value at the end of planning.
Preventing Exploitative Model Dynamics. When our planning procedure optimizes the VLM heuristic function HVLM(2,, (z.9) .9 it can exploit irregularities in the dynamics model fvM a) to get artificially high estimates. For instance, the planning procedure can exploit videos from fVM( a) where key objects have teleported desired locations where the final image observation obscures undesirable portions of world state. To prevent over-exploitation of HVLM g). during the planning procedure in Algorithm we discard generated videos from fvM(z, a) if they increase the the heuristic estimate HVLM(H, g) above a fixed threshold.
2.3 ACTION REGRESSION FROM VIDEO THROUGH GOAL- COAL-CONDTT POLICIES
Given a synthesized video plan Ti.H, to execute tasks, we must infer control actions u to reach synthesized images. Prior work infers actions from synthesized videos by using an inverse-dynamies model on each synthesized frame (Du et al., 2023b).
In many settings, a single action may not be sufficient to directly reach the next synthesized image, Le. if you need to remove the cap off toothbrush, and even in settings in which this the case, it may be difficult to precisely predict the correct action to reach the next frame. To reduce the burden on the inverse dynamics model, we propose to use a short-horizon goal-conditioned policy Toontrol (z, ).). which given the current image observation and next frame in video plan Ty outputs a low level control action u that makes progress towards Zg. For each frame in our video plan T1:H. the goal-come coldooiiiiooeed policy is executed for fixed pre-specified number of timesteps. We train Teonnol using paired image and low level control snippets ai. and ui.r, where we sample a random timestep a corresponding state 24, and future state th and train Toontral (Iz, Fe+h) to predict ut-
4

---

Published as a conference paper at ICLR 2024
group blocks by color
Spparste dluwe eebbe
Move ueeelbe the pentiggon
Move bile exbr hee crescentt
Move the atpiloo the fne-leeko credoeet
the yelicte
sellow Bawards
Push the fellow pentagon info the vellow star
Move the orm--oWo HoWords the coffeer
Push the prcenifaar zarinnt green dube
9. Mevee tow-oo the greeen blocks
10, Pushiedd siihtty City the down
make horizontal line
yol
the
blae telingle
Mite the cardle fovreedie yellow heart
More the tru touettnnggn the of roncie
Stede e towardd the blue singln
Prostr the cleser the bine capee
/Moree yellow trieggnn ripnser the caroe
Figure 2: Long Horizon Video Plan. Long horizon video plans generated by VLP on both simulated and real images. VLP is only given the initial image and language goal. Language subplans and other image frames directly synthesized.
Sim Environment
Real Environment
Move Area
Group Color
Make Line
Move Area
Group Color
Make Line
Model
UniPi
2%
4%
2%
4%
12%
4%
VLP (No Value Function)
10%
42 %
8%
20%
64%
4%
VLP (Ours)
58 %
98%
66% %
78%
100%
56%
Table 1: Accuracy of Generated Video Plans. The percentage VLP and baselines are able to synthesize a full video plan which can fully complete tasks in simulation and real environments. VLP substantially outperforms both UniPi and directly combining the VLM policy
Replanning. Given a very long- task, is both difficult to use Teontrol to accurately execute the full video plan T1.Hf (due accumulating error) and difficult to fully synthesize a plan that completely finishes a long-horizon task given a fixed planning horizon. To circumvent this issue, we use receding horizon control strategy (Kwon & Han, 2005), where we generate videos plans with fixed horizon (that might not fully complete the task), and then repeatedly regenerate/replan video plans with the same horizon after fixed number of action executions.
EXPERIMENTAL RESULTS
We first evaluate the ability of VLP to synthesize long-horizon video plans for different tasks in Sec. 3.1. We then investigate VLP's ability to execute generated video plans in various environments in Sec. 3.2. Finally, we further investigate generalization capabilities of VLP in Sec. 3.3.
3.1 LONG-HORIZON VIDEO SYNTHESIS
Baselines. We compare our approach with two other approaches for synthesizing long-horizon video plans. First, consider training text-to-video model fvM long horizon text goals, in UniPi (Du et al., 2023b), omitting the entire VLP planning process. Next, we consider synthesizing long horizon video plans by chaining TyLM policy with fvM. without the heuristic function.
Object Rearrangement. We first illustrate video plans in the Language Table environment (Lynchh et al., 2023), which consists of dataset of long-horte monghooizoon demonstrations and text labels with incom- plete short horizon labels (Appendix A.3). give as input to VLP a random image and randomly chosen language goal. We then visualize the generated VLP plans (Fig. 2). We report the quantitative success of synthesizing long-horizon videos given random starting images for each task in Language Table in Tab. 1. For each reported number, generated total of 50 videos from each method and visually assessed the percentage time the video successfully solved the given task. VLP substantially outperforms the baseline of directly synthesizing videos given long-horizon prompt,

---

Published as a conference paper at ICLR 2024
Language Branch
Video Branch
Line Performance
Beam
Branch
Beams
4%
10%
Beam 2, Branch 16
4
22%
2
4
4
56%
Figure 3: Video Accuracy vs Planning Budget. Left: VLP scales positively with more compute budget; it is better able to synthesize plans to solve tasks with more planning (i.e. with higher beam- branching factor). Success percentage reported on the make line task. Right: Qualitative illustration of vidco plans for making a line generated without planning (Beam Branch 1) compared to extensive planning (Beam 2, Branch 16),
X
X
Au Puts
Acran Face
X
X
Figure 4: Planning Tree on 7DoF Mobile Manipulator. VLP is able prune unlikely language and video branches synthesize coherent long-hou -orizon video plan.
indicating the importance hierarchical structure. VLP further outperforms the ablation of only using VLM policy with a video model, pointing the effectivenesss of the VLP planning procedure and including the value function.
Effect of Search of Video Synthesis. We analyze the effect of search in generating long-horizon videos in Fig. (left). We consider increasing the video branching, language branching and the beams in the search procedure. We find that each increase of branching factor in search substantially increases the success of synthesized long horizon plans. A qualitative illustration of the difference of generated plans with small and large branching factor is illustrated in Fig. 3 (right).
Planning on 7DoF Mobile Manipulators. We qualitatively illustrate how we can generate plans higher-Dof DoFF 7DoF Mobile Manipulator in Fig. 4. Our planning system is able generate videos of actions that both open and close drawers in order to satisfy specified text prompts.
Planning Multicamera 14DoF Bi-Manual Manipulators. We further illustrate how our ap- proach can generate multi-view 4-camera videos dexterous manipulation on the 14DoF bi-manual ALOHA (Zhao et al., 2023) platform in Fig. 5, Our video model outputs videos across views simulta- neously (by concatenating each view channelwise), while our VLM policy and heuristic function takes as input top and side views. While short horizon text labels are incomplete (Appendix A.3), approach synthesizes multiview consistent plans which stack bowls, cups, and utensils.
3.2
HARG--HOIZON EXECUTION
We next evaluate the ability of VIP to not only generate plans as in Sec. 3.1, but to actually use planning (and replanning) to execute long-horizon tasks in closed-loop environments.
Baselines. We compare our approach to a set of approaches to solve long-horizon tasks. (i) We consider using VLM directly plan, using PaLM-E (Driess al., 2023) to plan short horizon text snippets to execute, which are converted to actions using text-conditioned policy, conditioned on generated text snippets from PaLM-E. We also (ii) compare with UniPi (Du et al., 2023b), where videos are directly generated by text-to-video model trained on long-horizon text goals and converted to actions using our goal-conditioned policy. NexI, we consider (iii) directly learning language-conditioned behavioral cloning policy on long-horizon text and actions, using the codebase and architecture of the LAVA model Lynch et al. (2023). Finally, we (iv) compare with multi-modal transformer architecture predict actions, and finetune the RT2 model Brohan et al. (2023) long-how horizon text and actions. Note that methods that plan in language space (Hao et al., 2023) are not applicable, they cannot simulate detailed visual dynamics (Appendix A.

---

Published as a conference paper at ICLR 2024
Pick up the bumd Figure 5: Multiview Video Plans for Dexterous Manipulation. Long horizon video plans (and associated language subgnals) generated by VLP for solving the long horizon task of stacking everything in table together. VLP is able to synthesize multiview video plans task completion. first generated language subgoals goals are illustrated as well final generated goal
Place pink bowl in blue bowl
Pick up
Stock green blue
Pick up orange
Push the bowl to the
cameras, that consistent with each other and with
image. VLP is only given first image.
Move to Area
Group by Color
Make Line
Model
Reward
Completion
Reward
Completion
Reward
Completion
UniPi (Du et 2023b)
30.8
0%
44.0
4%
44.0
4%
LAVA (Lynch et al., 2023)
59.8
22%
50.0
2%
33.5
RT-2 (Brohan et al., 2023)
18.5
0%
46.0
26%
36.5
2%
PALM-E (Driess et al., 2023)
36.5
0%
43.5
2%
26.2
0%
VLP (Ours)
87.3
64%
95.8
92%
65.0
16 %
Table 2: Execution Performance Long Horizon Tasks. VLP is able to accurately execute actions for different long-horizon synthetic language table tasks. VLP substantially outperforms all existing methods.
Move blocks the botlom left comer
Group Blocks By Color
Maks Horizontal Line in the
Figure 6: Simulation Execution. Illustration of execution of VL.P on different simulated environments. VI.P able accomplish different long horizon goals.
We evaluate each approach quantitatively on moving all blocks to different areas of the board, grouping blocks by color, or making blocks in line (details on quantitative evaluation in Appendix). We report quantitative results in Tab. 2, and find that our approach substantially outperforms all baseline methods. As the task horizon of each task is very long (around 1500 steps), we found that many baseline methods would become "stuck" and stop acting effectively. We illustrate example executions using VLP Fig.
Quantitative Results.
Effect of Planning. Next, we analyze the effect of the amount of planning on execution success rates in Tab. We find that increasing both the planning horizon and the branching factor of planning substantially improves the success of task execution (at the cost of inference time).
Ablations of Goal-Condite Condiionid Policy We further conduct experiments different approaches to extracting actions from videos in Tab, 4. We find that using a goal-conditioned policy conditioned on each intermediate frame in a synthesized video leads to the best overall performance (outperforming using a goal-conditioned policy sparsely on the end frames of each short-horie thort-horizon video).
Real Execution. We provide executions of VLP multiple real-world robots in Fig. and Fig. 8. As in Fig. 7, VLP is able to effectively execute each shown long-horizon task on a real Language Table robot. We further provide executions of generated video plans of our approach the FDoF mobile manipulator in Fig. 8. Similarly we find that goal-conditioned policy can realize plans.
7

---

Published as a conference paper at ICLR 2024
Move all Blocks to the Right
Group Blocks By Color
Make Horizontal Line
Figure 7: Real Execution. Illustration of execution VLP on real world robots. VLP is able to accomplish different long horizon goals when executed on real robots.
Open up
fade top drawer
Place apple top drower
Closs sop duawer
Figure 8: 7DoF Mobile Robot Execution. VLP is able execute complex, long horizon plans mobile robot.
Group Color
Group Color
Planning Horizon
Branching Factor
Line Score
Line Completion
Beams
Action Inference
Score
Completion
Inverse Dynamics
89.7
80%
Goal Policy (Last)
85.0
66%
16
8%
Goal Policy (Every) 95.8
92%
16
65.0
16%
Table 4: Extracting Actions From VLP Video Plans. Comparison of using inverse dynamics applying goal-conditioned policy either last frame or ev- ery frame of synthesized mort-horiioon video. Success percentage reported on the group by color task.
Table 3: Execution Accuracy Planning Budget. VLP to more accurately execute video plans to solve tasks with a larger amount of planning. Success percentage reported on the make line task.
3.3 GENERALIZATION
Generalization to Lighting and Objects. In VLP, policy execution is abstracted into visual goal generation followed by goal-conditioned controller. With this abstraction, video model can simply focus on capturing the visual dynamics objects, while a goal-conditioned policy needs to focus only on relevant visual details to achieve the next (nearby) goal. We found that this enables VLP to generalize well, as the video model is able to visually generalize to new images, while the policy is able generalize well nearby new visual goals. In Fig. 9 (top), VLP is able generalize the task of putting all objects in top right corner, to three new objects, a rubber donut and cupcake and wooden hexagon. Fig. (bottom) VLP is further able to generalize lighting conditions substantially different than the ones the model was trained on and can be successfully deployed to new robot a different building location.
Generalization to New Tasks In VLP, both VLM and text-to-video models may be pre-trained on a vast amount of Internet data. Fig. 10, we train both VLM and text-to-video models on large mix of datasets and illustrate how further generalizes and executes new tasks on unseen objects.
RELATED WORK
There is great deal of recent work in leveraging large pretrained foundation models for decision- making agents (Yang et al., 2023b) in particular, using LLMs (and the commonsense knowledge they store) to infer actions represented in text. For example, given language instructions, LLMs can be prompted to generate high-level step-by-step plans (Huang et al., 2022a; Ahn et al., 2022; Wang et al., 2023a) each step mapped robot actions, invoked with pre- trained policies or existing APIs (Liang al., 2023). Using existing VLMs (Chen et al.. 2022), plans on visual inputs, represented with language descriptions (Zeng et al., 2022; Huang et al., 2022b) token embeddings co-trained with L.LMs (Driess al., 2023). Nevertheless, VLMs often
also be conditioned
8

---

Published as a conference paper at ICLR 2024
New Objects
New Lightin
Figure 9: Generalization to Objects and Lighting. VLP generalize execution scenes with new objects (top) consisting of wooden yellow hexagon, rubber donut and rubber cupcake and new office (bottom) with different lighting conditions (a lot more lighting on the right side of the board) and similarly execute tasks.
Nova Pick mancere
Novel
greae
Figure 10: Task Generalization VLP can generalize new tasks on unseen objects using internet knowledge.
pretrained static image datasets (i.e., image-in, text-out), and subsequently (i) are limited plans that can be expressed in text, and (ii) may struggle to reason about the dynamics of the world. Video models, on the other hand, exhibit potential to generate informative image sequences of the future, and there exists a body of work using them as dynamics models to capture object motion across image states (Finn et al., 2016; Xue et al., 2016; Oprea et al., 2020; Oh et al., 2015). Generating high-quality videos over long time horizons a known challenge (Babaeizadeh et al., 2021; Saxena al., 2021); however, recent progress in larger models and text-conditioning give rise generation of text-to-video models that can generate detailed image frames (Ho et al., 2022; Villegas al., 2022), that can drive policies for decision making (Du al., 2023b). However, these works focus predominantly on short-horizon videos. Our work investigates using text-to-video models together with VLMs produce long-horizon horizon video plans (potentially predicting hundreds of frames into the future), leveraging the scalability of tree search to plan in the space of videos and language.
Our work can be viewed bringing together two families of generative foundation models to compose new capabilities a strategy shown to be effective for applications such as 2D image synthesis (Du al., 2020; Liu al., 2021; Nie et al., 2021; Liu et al., 2022: Wu et al., 2022; Du et al., 2023a; Wang et al., 2023b), 3D synthesis (Po & Wetzstein, 2023), video synthesis (Yang et al., 2023a), trajectory planning (Du al., 2019; Urain et al., 2021: Gkanatsios et al., 2023; Yang et al.. 2023c) and multimodal perception (Li et al., 2022: Zeng et al., 2022). Most similar to our work, HiP (Ajay al., 2023) combines language models, video models, and action models for hierarchical planning. In contrast, our work uses forward search procedure to combine the strengths of a VLM and video model. This composition through forward search enables us simulate and reason about long horizons of future actions, while HiP only generates and acts on plans one step at a time.
LIMITATIONS AND CONCLUSION
Limitations. Our planning approach leverages images as world state representation. In many tasks, this is insufficient as does not capture the full state and cannot encode latent factors such physics mass. Limitations can be partly remedied by generating multi-view videos or by using heuristic function with the full video plan as input. In addition, we observed that our video dynamics model does not always simulate dynamics accurately. In several situations, we observed that synthesized videos would make objects spontaneously appear, change shape, or teleport to new locations. We believe that larger video models or explicit reinforcement learning feedback for physics (Black et al., 2023) could help solve these problems. Finally, our video generation is slow, and it takes 30 minutes synthesize full long-horizon video plan, which be accelerated by distillation (Salimans & Ho, 2022) or alternative video generation procedures (Villegas et al., 2022).
Conclusion. We have presented VLP, an approach to long-horizon decision-making by combining VLMs with text-to-video models. We illustrate the power of test-time composition, in this case planning, to generate behaviors much more complex than its components. While experiments largely explore VLP in the context of robotics, there may be downstream applications in other areas too, such steerable media generation as well additional methods compose models.

---

Published as a conference paper at ICLR 2024
Acknowledgements. We would like thank Tomas Lozamo-Peree Perez for providing helpful comments in the project and Kamyar Ghasemipour for help in setting up experiments on the Language Table real robot. We gratefully acknowledge support from NSF grant 2214177; from AFOSR grant FA9550-22- 1-0249; from ONR MURI grant N0001
from ARO grant W91 INF-23- 1F-23-1-0034; from the MIT-IBM Watson Lab; from the MIT Quest for Intelligence; and from the Boston Dynamics Artificial Intelligence Institute. Yilun Du is supported by a NSF Graduate Fellowship.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. URL
https:// aar/iiiiiiv..////vva...aaa.aaata org/ abs 2204. 01 09691. 1,8,16
Anurag Ajay, Seungwook Han, Yilun Du, Shaung Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaclbling. Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. arXiv preprint arXiv:2339. 089088,7 2023.
Mohammad Babacizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Du- mitru Erhan. Fitvid: Overfitting pixel-level video prediction. arXiv preprint arXiv:2 13195,
2021. 9
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023. 9
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.066/77 2022. 15, 16
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision Vision-language-actiinn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 2023. 2, 6, 15, 16
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 1
Stephane Cambon, Rachid Alami, and Fabien Gravot. A hybrid approach to intricate motion, manipulation and task planning. The International Journal of Robotics Research, 28(1):104-126, 2009. 1
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adan Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209. 06794, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022. 1, 16
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pp. 720-736, 2018. 15
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:230..03378, 2023. 2, 6, 7, 16
Yilun Du, Toru Lin, and Igor Mordatch. Model based planning with energy based models. CORL, 2019. 9
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models.
Advances in Neural Information Processing Systems, 33:6637-6647, 2020. 9
10

---

Published as a conference paper at ICLR 2024
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. arXiv preprint arXtv:2302.11552, 2023a. 9
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv e-prints, pp. arXiv-2302, 2023b. 1, 3, 4, 5, 6, 7, 15, 16
Frederik Ebert, Yanlai Yang. Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danillidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv: 13396, 2021. 15
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016. 9
Nikolaos Gkanatsios, Ayush Jain, Zhou Xian, Yunchu Zhang, Christopher Atkeson, and Katerina Fragkiadaki. Energy- models as zero-shot planners for compositional scene rearrangement. arXiv preprint 2rXiv:2304.443333 2023. 9
Kristen Grauman, Andrew Westbury, Eugene Byree Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Bgo4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022. 15
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14994 2023. 6
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 2xXi:2210.02300: 2022. 1,
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pp. 91 9118-91.7. PMLR, 2022a. 1,8 8
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 2xXi:22077005008 2022b. 8
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprini 2rXiv:23033.08855 2023.
Leslie Pack Kaelbling and Tomás Lozano Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pp. 1470-1477. IEEE, 2011.
Wook Hyun Kwon and Soo Hee Han. Receding horizon control: model predictive control for state models. Springer Science & Business Media, 2005. 5
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 15:31199-31211 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 9493-950X IEEF, 2023. 8
Nan Liu, Shuang Li, Yilun Du Josh Tenenbaum, and Antonio Torrala. Learning to compose visual relations. Advances in Neural Information Processing Systems, 14-23166-23177 2021. 9
11

---

Published as a conference paper at ICLR 2024
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. arXiv preprint arXi::2006.07714, 2022. 9
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong. and Pete Florence. Interactive language: Talking to robots real time. IEEE Robotics and Automation Letters, 2023. 5, 6, 16
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34, 2021.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard I. Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances in neural information processing systems, 28, 2015. 9
Sergiu Oprea, Pablo Martinez-Goonvalez, Alberto Garcia-Garcia, John Alejandro Castro-Varger Sergio Orts-Escolano, Jose Garcia-Rodriguu and Antonis Argyros. A review deep learning techniques for video prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6))2806-288) 28222, 2020.
Ryan Po and Gordon Wetzstein. Compositional 3d scene generation using locally conditioned diffusion, arXiv preprint arXiv:2303. 12211288,, 2023."
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0m 00512,, 2022.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders. Advances in Neural Information Processing Systems, 34:29246-29257, 2021. 9
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25295 25294, 2022. 15
Bart Selman and Carla P Gomes. Hill-climbing search. Encyclopedia of cognitive science, 81:82, 2006. 4
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
Julen Urain, Angi Li, Puze Liu, Carlo D D'Eramo, and Jan Peters. Composable energy policies for reactive motion generation and reinforcement learning. arXiv preprint arXiv:2105.04962 2021.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399 2022. 9
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01566 2023a.
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for text-controlled vision models. arXiv preprint arX1v:2302.03693, 2023b. 9
Jason Wolfe, Bhaskara Marthi, and Stuart Russell. Combined task and motion planning for mobile manipulation. In Proceedings of the International Conference Automated Planning and Scheduling, volume 20, pp. 254-257 2010.
Tailin Wu, Megan Tjandrasuwita. Zhengxuan Wu, Xuelin Yang, Kevin Liu, Rok Sosie, and Jure Leskovee. Zeroc: A neuro-symbolic model for zero-shot concept recognition and acquisition at inference time. Advances in Neural Information Processing Systems, 35:9828-998 98409, 2022.
12

---

Published as a conference paper at ICLR 2024
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. Advances in neural information processing systems, 29, 2016. 9
Mengjiao Yang, Yilun Du, Bo Dai, Dale Schuurmans, Joshua B Tenenbaum, and Picter Abbeel. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.01877 2023a. 9
Sherry Yang, Ofrr Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129, 2023b. 8
Zhutian Yang, Jiayuan Mao, Yilun Du, Jiajun Wu, Joshua B 'Teeebbuu, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Compositional diffusion-based continuous constraint solvers. arXiv preprint arXiv:2309.0099000 2023c.
Andy Zeng, Adrian Wong. Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lec, Vincent Vanhoucke, et al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00590 00598, 2022. URL httpt::://tattttee
org/ abs/ 2204. 00598. 8,9
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint wxXii:2304.13703. 2023,
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
In the Appendix, we provide additional results in Sec. A.1. We provide evaluation details in Sec. A.2, dataset details in Sec. A.3, training details in Sec. A.4, and planning details in Sec. A.5.
A.1
ADDITIONAL RESULTS
Below, we provide additional experimental results.
Failure Cases. We observed several failures cases when applying our approach across tasks. First, when using world knowledge from other text-to-video datasets to generalize and execute new tasks, found that sometimes the video model would incorrectly interpret tasks illustrated in Fig. XI, where it interprets the manipulator as an octopus. In addition, we found that our approach sometimes synthesized videos with inconsistent physics, where objects would infrequently appear or disappear as illustrated in Fig. XII.
Robustness of Conditoodd Policy We found that decomposition of execution in VLP to visual goal generation and subsequent execution allowed for strong generalization in the goal- conditioned policy. This was because the goal-conditioned policy can discard most information in image goals and focus only on the portion required for immediate execution of the policy. We illustrate this robustness in Fig. XIII. Even when the generated goals are misaligned with the observed board, with many artifacts in the generated boundaries, our goal-conditioned policy is still able to execute the visual task.
Additional Video Plans. We provide additional video plans synthesized by our approach on block arrangement in Fig. XIV. Our approach is able to robustly synthesize both language and video plans. Planning with Language Models. To directly plan with language models, we would need language model accurately simulate the visual dynamics of an image serialized to text. In Fig. XV, we find that existing LLMs are unable to accurately simulate visual dynamics, missing object collisions, identities and gripper movement, preventing them from being effectively used to plan.
A.2 EVALUATION DETAILS
Video Evaluation. To evaluate whether a generated video satisfied a long horizon task, we manually assessed whether at any point in the generated video, the image state in the video satisfied the specified long-horizon goal. Due to the computational cost of generating long-horizon video (taking approximately 30 minutes per video), we generated a total of 50 videos for each goal across each task and method.
Execution Evaluation. We used the ground truth simulation state of each block in the Language Table environment to compute rewards and task completion thresholds for each task in the paper. To compute the reward function for "group blocks by color". same color that were within 0.1 units of each other. To compute the reward for "move blocks to an area", we assessed the number of blocks within 0.2 units and 0.27 y units specified corner. To compute the reward for "move blocks into a line in the center of a board", we computed the number of blocks with 0.05 units of the center of the board. We gave all baselines and VLP a total of 1500 timesteps of execution in each environment, with performance measured at the end of 1 1,500 timesteps. an intermediate timestep the task was complete, we terminated execution early.
assessed the percentage of blocks of the
We evaluated each method on total of 50 environments on each task due to slowness in execution of both our method and baselines (VLP took approximately 1 hour per environment, while RT2 baselines took upproximate 0.5 hours per environment). For VLP, we generated a plan of horizon length 2 with beam width 2 and branching factor 16, where each video in a planning horizon corresponded to 16 frames. We called the goal-conditioned policy a total of 4 times for each of the first 16 frames of the synthesized video plan in simulated environments. On real world evaluation tasks, we called the goal-conditioned policy on the first 10 frames of the synthesized video plan more finely account for differences in physics of the real world execution compared to those used to gather the data.
A.3 DATASET DETAILS
Language Table. We trained VLP on approximately 10000 long horizon trajectories in both simulation and real across a set of several hundred different long horizon goals. We chose total of goals to run experiments on as they allowed easy automated evaluation. Data in simulation and real combined in total is roughly 20000 trajectories and with approximately 400000 labels. Since long- oan-hooiizon goal demonstrations are gathered all at once, without any intermediate short-horizon text goals, short-horizon text is labeled in hindsight long-horizon demonstrations. A
text
14

---

Published as a conference paper at ICLR 2024
regille
regime
Figure XI: Failure in Transferring Web Knowledge. VLP is instructed to "pick up octopus toy". In the low-data regime, the model has only the octopus toy in training data few times, and the generated frames roughly reconstructed the shape of toy. but fails at accurately recover dynamics, the model has never seen in training data, and without any training data VLP transfers the wrong web knowledge and makes the gripper octopus arms.
no-data regmie,
Figure XII: Failure in Physics. When synthesizing videos, VLP will sometimes make objects dissappear or reappear. Creating long videos that respect object permanence remains important future direction for work.
set of human raters are asked to go through long-horizon demonstrations and select sub- sub-snippets that correspond interpretable short-horizon text goals. As many robot motions do not easily map text actions, these sub-snippets do not capture all frames in demonstrations and are often overlapping in time, leading to short-horizon text labels to be incomplete.
7DoF Mobile Manipulator. For our 7DoF mobile manipulator planning and execution experiments, we trained VLP on the dataset from RT-1 (Brohan et al., 2022). For our generalization experiments of VLP, we train large rext-to-video diffusion model using a mix of data from our particular 7DoF Mobile Manipulator, Bridge (Ebert et al., 2021), RT-2 (Brohan et al., 2023), Ego4D (Grauman et al., 2022), EPIC-KITCHEN (Damen et al., 2018 8), and LAION-400M (Schuhmann et al., 2022).
14DoF Bi-Manual Manipulation. For our 14DoF Bi-Manual Manipulation, we train VLP on approximately 1200 teleoped demonstrations on kitchen stacking task, where operators were first asked to stack bowls on top each other, then cups on top of bowls, and then utensils on top of cups. Each demonstration was annotated with roughly 20 language instructions, leading roughly 25k short-horizon text labels. Similar to the Language Table setting, short-horizon text labels are incomplete and generated through hindsight relabeling of the long teleoped trajectories.
A.4 TRAINING DETAILS
Video Models. To train video diffusion models, we follow the model architecture and training approach from (Du et al., 2023b). We train a base text-conditioned video generation model at 24 40 resolution and upsample it to 48 x 80 resolution and then 192 320 resolution, where videos at each resolution are set to generate a temporal video of length 16 frames. We use base channel width of 256 across models and train base eotdioniit video model using 64 TPUv3 pods for 3 days and higher resolution superresolution models for day. We train separate text-to-vided models per domain.
15

---

Published as a conference paper at ICLR 2024
Synthesized Frames
Goal Execution
Figure XIII: Goal Policy Robustness to Synthesized Goals The goal-conditioned policy robust to noise in synthesized goals. In the above example, given synthesized goals in the top row, the policy can directly execute control actions in the real environment in the bottom
VLM Models. To train VLMs, we follow the architecture and codebase of PaLM-E (Driess et al., 2023). We fine-tune a single 12B PaLM-E (Driess et al., 2023) jointly to both predict heuristics values and policies. We formulate heuristic value prediction as a sequence prediction problem, allowing the heuristic functions to predict multiple different values for multi-modal demonstrations, We finetune the VI.M model using 64 TPUv3 pods for day on data in each domain and use separate models per domain.
Conl-Conditioned Policy. To train our goal-conditioned policy, we use the LAVA model (Lynch al., 2023) architecture, where the text-encoder from CLIP replaced with ResNet encoder for the goal image. We train our goal-conditioned policy in each domain using 16 TPUv3 pods for 1 day. Baselines. We compare approach with set of multiple baselines. For the LAVA baseline, we follow the exact settings from the LAVA paper (Lynch et al., 2023), where we train the model conditioned on long-horizon goal text and trained the model for day use 16 TPUv3 pods. For our UniPi (Du et al., 2023b) baseline, we followed the same training architecture and approach as the text-conditioned video model in VLP, where the model is trained on mongoorizon video elements. For the RT-2 baseline, we follow the architecture and codebase of the original paper (Brohan et al., 2023), where we use the 12B RT2-PaLMMEE model. We trained RT-2 using TPUv3 pods for 3 days.
A.S PLANNING DETAILS
Language Table. To generate video plans, we planned with a horizon 16, a beam width of 2, language branching factor of 4, and video branching factor of 4. To enable fast video generation, we used the DDIM sampler, with a total of 64 timesteps of sampling at the base resolution and timesteps of sampling at the higher resolution samples, with classifier-fre guidance scale of 5 for the base model. We generated 16 videos in parallel at once use a 4 TPU inference pod.
We queried the VLM policy to generate different text actions given an image with a temperature of 0.3. Our VLM heuristic function decoded the number of steps left until task-completion with temperature of 0.0. We set our heuristic function clipping threshold during planning to be 50 and removed videos if the improvement was larger than 50 after one video rollout. This number was heuristically determined based off generated plans at different thresholds, with the aim of finding the highest threshold such that long-horizon plans still looked physically plausible. Our total planning procedure across horizon of 16 took approximately 30 minutes.
7DoF Mobile Manipulator. In the 7DoF mobile manipulator experiments we made slight tweaks to the approach. For VLM Planning, we used PaLM-E (Driess et al. (2023)) to generate scene captions, and few-shot prompted PaLM (Chowdhery al. (2022)) to generate plans following the prompts in SayCan (Ahn et al. (2022)). The beam search has beam width of 3. The video diffusion model the same as above, except that it has different resolution for the base model (64 x 80) and super resolution model (256 x 320), and is finetuned on RT-1 (Brohan et al. (2022)) data only. The goal-conditioned policy is using the last frame of the generated video segment only.
14DoF Bi-Manual manipulation. We followed the same planning setup in Language Table. set our heuristic function clipped threshold during planning to 15.
16

---

Published as a conference paper at ICLR 2024
make horizantal line at the center the board
Push the green be inlo the
Push the green zuoe info blue crescent
3. Plsh the green cube mo mo
Move the green gla tee left the blue crescent
Potn your am fog the red Pentagon
blue crescant
ael lowi star of
mventeggron
prentegem
go
pentagen
all blocks the bottom right
Move the red carcle the left of the you heccccon
Mitse the green cirele clase to the red cincle
Doint your arth the grown allir
Moswr the grewnn fowwordds the center
Mawe the preen ofar Inmends The rnd st
More thergncin chole gieen gtor
onn bluy closer arcict
triahaio green star
Fasah sito
yourr
OF
circ'e
Figure XIV: Additional Long Horizon Video Plans. Additional long hotizon video plans generated by VLP on both simulated and real images. VLP is only given the initial image and language goal. Language subplans and other image frames are directly synthesized.
17

---

Published as a conference paper at ICLR 2024
at [-0.1, 0.1]. pentagon [0.1. 0.1]. yellow cube [- 10.11. yellow heart at [0.2. 0.2]. green star at [0.3, 0.3]. green pentagon at [-0.3,-0.3], blue cylinder at [0.4, 0.4). blue triangle at [0.5, 0.5]. The robot gripper at [0, 0j. What is the resultant state of the world after the command, push the red pentagon next the green star? The robot gripper must physically push blocks. Make sure to simulate the dynamics of gripper and block collision well as block on block collision. List the state of all blocks and the robot gripper.
Prompt:
Consider robot table with
Simulating the dynamics of the gripper and block collisions, well block-on-bleck collisions, involves series of steps. I'll describe state the world after the command "push pentagon next to the green star":
Response:
Initial state:
Red cube at [-0.1, 0.1]
Red pentagon at 10.1, 0.1]
Yellow cube at [-0.1, -0.1]
Yellow heart [0.2, 0.2]
GGreen star at [0.3, 0.3]
Green pentagon at [-0.3, -0.3]
Blue cylinder [0.4, 0.4]
Blue triangle at [0.5, 0.5]
Roboo gripper at [0, 0]
Steps:
1.The robot gripper moves to red pentagon.
2.ehe gripper picks the red pentagon.
3. gripper moves towards the green star.
The gripper releases red pentagon the green star.
Resultant state:
Red cube at [-0.1, 0.1]
Ycllow cube at [-0.1, -0.1]
Yellow heart at [0.2, 0.2]
Green star at [0.3, 0.3]
Green pentagon [-0.3, -0.3]
Blue cylinder [0.4, 0.4]
Blue triangle at [0.5, 0.5]
Robot gripper [0, 0]
Red pentagon [0.45, 0.45] (approximately, based on gripper's release position)
Note:
The final position of the red pentagon next to the green star depends on the specific dynamics of the gripper and the release mechanism. The above description assumes successful execution of the push command without any unexpected behavior or collisions that might alter the final positions.
Figure XV: Simulating Dynamics with GPT3.5. LLMs are unable to accurately simulate the visual dynamics of serialized image representation. In the above example, provide simple serialization of an image in object a single point).
Language Table trepresenting
LLM is unable to accurately preserve the object the scene, does not accurately simulate movement of the gripper and does not simulate the collisions that would happen when the red pentagon move to the green star (collision with the yellow heart).
18


Reviews:
[{"review_id": "klXykK3FMt", "reviewer": "Reviewer_MAiq", "summary": "The paper proposes a framework for planning in the space of videos and language by leveraging recent text-to-video models (over vision-language models (VLMs) that are trained on static images) for incorporating scene dynamics and object motions. They take as input an initial observation and a long-horizon language instruction and output a plan in the form of video and language specifications, by first prompting a VLM for language actions and rolling out a language-conditioned video model whose outputs are subsequently assessed by the VLM again for favourability towards task progress using a learned heuristic.", "questions": "Are there any limitations induced by video prediction models, which as the authors identified can generate out-of-manifold frames during long-horizon video prediction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1) The paper is easy to follow and well-structured with the main contributions listed clearly in the introduction alongwith grounding in related works and building blocks that make up this method (VLM, a video prediction model, and an inverse-dynamics model).\n2) The authors present strong model performance compared to 4 baseline methods on object rearrangement tasks, as well as deploy their method on multiple real-robot platforms.", "weaknesses": "While the tasks chosen for this paper are claimed to be long-horizon, they are not challenging enough to showcase a big leap in this realm of tasks. For example, when grouping blocks by color, while the task may seem long-horizon given there are large number of blocks on the table to manipulate, there is no dependency between subgoal successes/failures. This makes the task solvable without any need to retain information for long horizons. Hence, I believe the chosen suite of tasks does not evaluate the model adequately for solving long-horizon tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3d7alSwxMU", "reviewer": "Reviewer_uBLi", "summary": "The authors propose a general-purpose framework solving for robotics tasks in which planning is done using a video generative model, and then using the sampled video as a target, a low-level controller selects actions to be executed by the robot in order to reproduce the video. To improve the quality of long-term video generation, the authors propose a hierarchical approach in which a list of commands are generated in text-format in addition to the video, and together optimized to minimize an estimate of how close the task is to being completed. The authors demonstrate impressive performance for multiple synthetic and real-world robotics tasks. Importantly, due to the use of internet-scale pre-training for the vision-language model (VLM) and video generative model, the approach generalizes well in a zero-shot setting to novel lighting conditions, objects, and tasks.", "questions": "In addition to the two main points mentioned in the weaknesses section, I have a few other minor questions:\n\n- Section 2.2 \"Vision-Language Models as Heuristic Functions.\" - is this model trained with regression, or the tokenized representation of the number of steps with the standard language modeling objective? If it's with the language modeling objective, do the authors choose the mode of the distribution over predicted number of steps when using the heuristic to plan in Algorithm 1?\n- Section 2.3 \"Replanning\" - is there a way to make this re-planning rate dynamic? Ideally, it would be possible to sense when reality has diverged from the video plan in a significant way that is not recoverable.\n- A.4 \"Video Models\" - what's the frequency of these videos, e.g. how many seconds does 16 frames correspond to?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "A great challenge for the machine learning community is to determine how recent advances in generative modeling can be used to advance robotics. I think the method proposed in this paper makes great strides towards this end. Both vision and language foundation models are combined to produce an impressive model-based goal-conditioned planner for robotics. Additionally, I found the paper very well written; I'm not an expert in robotics, but I found the paper easy and interesting to read. The discussion of the limitations at the end is especially insightful, as well as the visuals of video generative model failures in the appendix. The method is stated clearly and its efficacy is backed up with substantial qualitative and quantitative results.", "weaknesses": "I think the main weakness of the current submission is that it does not include measurements of runtime for the proposed approach, for instance in Table 3. Planning in video space seems very expensive which could limit how well this approach could be used for robotics tasks that need to be executed quickly. Of course, these runtimes can always be improved, but it would be helpful to see what the runtime is with today's hardware.\n\nRelatedly, I'm curious if the authors have considered optimizing only the text sequence with planning in Algorithm 1, then following with per-clip video generation? It seems much more efficient to me to search through text instead of pixels. Based on Table 3, it seems there are large gains to be had from a more exhaustive search, which I'm guessing is much easier to do if the need to render pixels is eliminated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "we64bdap7v", "reviewer": "Reviewer_4qts", "summary": "This paper addresses the challenge of long-horizon visual planning tasks by utilizing the robust generative capabilities of large vision-language models (VLM) and text-to-video models. The authors introduce the Video Language Planning (VLP) algorithm, which takes both a visual observation and a natural language goal as inputs and subsequently processes a series of instructions, video frames, and low-level controls. The effectiveness of their proposed heuristic value function and tree search procedure is well demonstrated through extensive long-horizon robot manipulation tasks across three hardware platforms.", "questions": "- Table 1 clearly illustrates that the Value Function utilized in VLP significantly enhances result accuracy. However, in many robotic tasks, it is common to have multi-modal policies. For instance, in the \"make a horizontal line\" task shown in Figure 2, there could be multiple ways to manipulate objects, resulting in various possible remaining steps, especially for images that are far from the final goal. It would be helpful to see whether the PaLM-E model can be fine-tuned to accommodate multi-modal trajectories.\n- While the paper properly discusses a few limitations, the limitation of the task horizon is unclear in this paper. In the appendix, the authors provided both VLP and other baselines with 1500 timesteps to complete a task. Does this imply that VLP may not be suitable for handling longer-horizon tasks? For example, a long-distance task, such as moving an object over a long distance (e.g., 10 meters) from one location to another.\n- In the introduction, the authors emphasize the potential benefits of VLP when working with incomplete videos that lack corresponding language labels. It would be helpful to see more detailed descriptions and accompanying experimental results to further demonstrate this capability, which are not found in this paper.\n- The authors have highlighted the issue of overestimation with the Value Function. It would be interesting to see if out-of-distribution images or goals might also contribute to this problem. If so, could the application of certain offline reinforcement learning algorithms, such as CQL, potentially offer a solution for addressing this issue?\n- Minor issues that may need careful proofreading:\n  - Sec. 2: a image goal-conditioned -> an\n  - as planning submodules Sec. 2.1 -> in Sec. 2.1", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The long-horizon tasks pose significant challenges and matter in the field of robot manipulation. The proposed VLP algorithm has the potential to significantly improve the success rate of these tasks, as demonstrated in both simulated and real-world experiments in this paper\n- Collecting large-scale robot manipulation datasets can be a costly endeavor. VLP, on the other hand, harnesses the impressive generative and generalization capabilities of the latest VLMs and text-to-video models that are pre-trained on Internet-scale data. This approach can serve as valuable inspiration for other researchers looking to tackle challenging robot manipulation tasks without the need for expensive data collection.\n- The experimental results regarding the relationship between execution accuracy and planning budget provide valuable insights into the efficiency of the proposed VLP algorithm.", "weaknesses": "The primary concern regarding this work is its potential for reproduction and adaptation to, e.g., other hardware platforms and low-cost budgets.\n- While the study tests the VLP algorithm on three hardware platforms, they are either relatively simple (i.e., having just one end effector) or self-designed. There is a question as to whether the well-trained VLP model could generalize or adapt quickly to other popular platforms. This limitation may reduce the paper's overall impact.\n- While video model inference is computationally expensive, its scalability is also a concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MJEdoyogFt", "reviewer": "Reviewer_AYpb", "summary": "The authors describe an algorithm for video language planning (VLP) that takes as input a long-horizon task instruction and current image observation, and outputs a long video plan that provides detailed multimodal (video and language) specifications that describe how to complete the final task. VLP leverages (i) vision-language models to serve as both policies and value functions, and (ii) text-to-video models as dynamics models. VLP is evaluated on simulated and real-world robotic platforms.", "questions": "- Unclear how long of a video does the video models produce. \n- Authors say that they train separate models per domain but it is unclear what is a domain. Is tabletop manipulation in sim and real world considered same or different domain?\n- In table. 1, why not ablate VLP without the video dynamics model? Given my concerns in the weakness section, would have loved to see this ablation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Overall well-written paper, which is easy to follow\n2. Great to see real-world evaluation despite the simplicity of tasks", "weaknesses": "1. One of the key contributions of VLP is the use of text-to-video models as dynamics models. But authors experiments and baselines are not convincing about the value of using these models as dynamics models. \n* Authors use relatively simple tasks to evaluate VLP. Why not use a text-based LLM or something like this: https://arxiv.org/abs/2106.00188 as a dynamics model instead of the text-to-video model given the simplicity of the evaluation tasks? Would have been great to see such a baseline to really understand the edge that video dynamic models provide.\n * It would have also been great to see VLP being applied to more complex tasks e.g., procedural planning (https://arxiv.org/abs/1907.01172). Generating a walk-through plan for a procedural task is a convincing use case for video dynamics model as compared to say an LLM-based dynamics model.\n2. It is also not clear which use-cases will really require VLP as compared to existing LLM-based planning models or more generalized agent architectures such as GATO (https://arxiv.org/abs/2205.06175) or decision transformers. To that end, it would have been useful to see a broader set of baselines in the paper. Some baselines to consider -- RAP: https://arxiv.org/abs/2305.14992 (uses MCTS + LLM dynamics model similar to VLP's tree search+dynamics model), Visual Language Planner: https://arxiv.org/pdf/2304.09179.pdf (learns a multimodal dynamics model and policy by finetuning an LLM), VIMA: https://arxiv.org/pdf/2210.03094.pdf (no dynamics model but multimodal planner, could truly bring out the value of video-based dynamics via such a comparison?)\n* Authors also say HiP is the closest to their work in terms of an approach but do not provide a performance comparison in their evaluation. Why? Would love to see it.\n3. VLP's inference is slow.  Given that VLP primarily focuses on robotic applications right now, it is unclear whether VLP is suitable for real world deployment. Slow runtime should also be reported as limitations in the main paper (rather than in the appendix).\n4. It is unclear if the authors are considering open-sourcing the code and the models. Without that the reproducibility and value of the work for the community reduces. I encourage the authors to discuss open-sourcing plans in their rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)