Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable
  Robot Manipulation
Authors: name: Homanga Bharadhwaj; name: Debidatta Dwibedi; name: Abhinav Gupta; name: Shubham Tulsiani; name: Carl Doersch; name: Ted Xiao; name: Dhruv Shah; name: Fei Xia; name: Dorsa Sadigh; name: Sean Kirmani
Published: 2024-09-24 17:57:33

Abstract: How can robot manipulation policies generalize to novel tasks involving
unseen object types and new motions? In this paper, we provide a solution in
terms of predicting motion information from web data through human video
generation and conditioning a robot policy on the generated video. Instead of
attempting to scale robot data collection which is expensive, we show how we
can leverage video generation models trained on easily available web data, for
enabling generalization. Our approach Gen2Act casts language-conditioned
manipulation as zero-shot human video generation followed by execution with a
single policy conditioned on the generated video. To train the policy, we use
an order of magnitude less robot interaction data compared to what the video
prediction model was trained on. Gen2Act doesn't require fine-tuning the video
model at all and we directly use a pre-trained model for generating human
videos. Our results on diverse real-world scenarios show how Gen2Act enables
manipulating unseen object types and performing novel motions for tasks not
present in the robot data. Videos are at https://homangab.github.io/gen2act/