This&That: Language-Gesture Controlled Video Generation for Robot
  Planning
Authors: name: Boyang Wang; name: Nikhil Sridhar; name: Chao Feng; name: Mark Van der Merwe; name: Adam Fishman; name: Nima Fazeli; name: Jeong Joon Park
Published: 2024-07-08 00:28:41

Abstract: We propose a robot learning method for communicating, planning, and executing
a wide range of tasks, dubbed This&That. We achieve robot planning for general
tasks by leveraging the power of video generative models trained on
internet-scale data containing rich physical and semantic context. In this
work, we tackle three fundamental challenges in video-based planning: 1)
unambiguous task communication with simple human instructions, 2) controllable
video generation that respects user intents, and 3) translating visual planning
into robot actions. We propose language-gesture conditioning to generate
videos, which is both simpler and clearer than existing language-only methods,
especially in complex and uncertain environments. We then suggest a behavioral
cloning design that seamlessly incorporates the video plans. This&That
demonstrates state-of-the-art effectiveness in addressing the above three
challenges, and justifies the use of video generation as an intermediate
representation for generalizable task planning and execution. Project website:
https://cfeng16.github.io/this-and-that/.