RACER: Rich Language-Guided Failure Recovery Policies for Imitation
  Learning
Authors: name: Yinpei Dai, name: Jayjun Lee, name: Nima Fazeli, name: Joyce Chai

Developing robust and correctable visuomotor policies for robotic
manipulation is challenging due to the lack of self-recovery mechanisms from
failures and the limitations of simple language instructions in guiding robot
actions. To address these issues, we propose a scalable data generation
pipeline that automatically augments expert demonstrations with failure
recovery trajectories and fine-grained language annotations for training. We
then introduce Rich languAge-guided failure reCovERy (RACER), a
supervisor-actor framework, which combines failure recovery data with rich
language descriptions to enhance robot control. RACER features a
vision-language model (VLM) that acts as an online supervisor, providing
detailed language guidance for error correction and task execution, and a
language-conditioned visuomotor policy as an actor to predict the next actions.
Our experimental results show that RACER outperforms the state-of-the-art
Robotic View Transformer (RVT) on RLbench across various evaluation settings,
including standard long-horizon tasks, dynamic goal-change tasks and zero-shot
unseen tasks, achieving superior performance in both simulated and real world
environments. Videos and code are available at:
https://rich-language-failure-recovery.github.io.