Video Language Planning
Authors: name: Yilun Du; name: Mengjiao Yang; name: Pete Florence; name: Fei Xia; name: Ayzaan Wahid; name: Brian Ichter; name: Pierre Sermanet; name: Tianhe Yu; name: Pieter Abbeel; name: Joshua B. Tenenbaum; name: Leslie Kaelbling; name: Andy Zeng; name: Jonathan Tompson
Published: 2023-10-16 17:48:45

Abstract: We are interested in enabling visual planning for complex long-horizon tasks
in the space of generated videos and language, leveraging recent advances in
large generative models pretrained on Internet-scale data. To this end, we
present video language planning (VLP), an algorithm that consists of a tree
search procedure, where we train (i) vision-language models to serve as both
policies and value functions, and (ii) text-to-video models as dynamics models.
VLP takes as input a long-horizon task instruction and current image
observation, and outputs a long video plan that provides detailed multimodal
(video and language) specifications that describe how to complete the final
task. VLP scales with increasing computation budget where more computation time
results in improved video plans, and is able to synthesize long-horizon video
plans across different robotics domains: from multi-object rearrangement, to
multi-camera bi-arm dexterous manipulation. Generated video plans can be
translated into real robot actions via goal-conditioned policies, conditioned
on each intermediate frame of the generated video. Experiments show that VLP
substantially improves long-horizon task success rates compared to prior
methods on both simulated and real robots (across 3 hardware platforms).