RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning
Authors: name: Charles Xu; name: Qiyang Li; name: Jianlan Luo; name: Sergey Levine
Published: 2024-12-13 04:57:55

Abstract: Recent advances in robotic foundation models have enabled the development of
generalist policies that can adapt to diverse tasks. While these models show
impressive flexibility, their performance heavily depends on the quality of
their training data. In this work, we propose Reinforcement Learning Distilled
Generalists (RLDG), a method that leverages reinforcement learning to generate
high-quality training data for finetuning generalist policies. Through
extensive real-world experiments on precise manipulation tasks like connector
insertion and assembly, we demonstrate that generalist policies trained with
RL-generated data consistently outperform those trained with human
demonstrations, achieving up to 40% higher success rates while generalizing
better to new tasks. We also provide a detailed analysis that reveals this
performance gain stems from both optimized action distributions and improved
state coverage. Our results suggest that combining task-specific RL with
generalist policy distillation offers a promising approach for developing more
capable and efficient robotic manipulation systems that maintain the
flexibility of foundation models while achieving the performance of specialized
controllers. Videos and code can be found on our project website
https://generalist-distillation.github.io