RLIF: Interactive Imitation Learning as Reinforcement Learning
Authors: Jianlan Luo,Perry Dong,Yuexiang Zhai,Yi Ma,Sergey Levine
Presentation: ICLR 2024,Poster

Abstract: Although reinforcement learning methods offer a powerful framework for auto-
matic skill acquisition, for practical learning-based control problems in domains
such as robotics, imitation learning often provides a more convenient and accessible
alternative. In particular, an interactive imitation learning method such as DAgger,
which queries a near-optimal expert to intervene online to collect correction data for
addressing the distributional shift challenges that afflict naïve behavioral cloning,
can enjoy good performance both in theory and practice without requiring manually
specified reward functions and other components of full reinforcement learning
methods. In this paper, we explore how off-policy reinforcement learning can
enable improved performance under assumptions that are similar but potentially
even more practical than those of interactive imitation learning. Our proposed
method uses reinforcement learning with user intervention signals themselves as
rewards. This relaxes the assumption that intervening experts in interactive imita-
tion learning should be near-optimal and enables the algorithm to learn behaviors
that improve over the potential suboptimal human expert. We also provide a uni-
fied framework to analyze our RL method and DAgger; for which we present the
asymptotic analysis of the suboptimal gap for both methods as well as the non-
asymptotic sample complexity bound of our method. We then evaluate our method
on challenging high-dimensional continuous control simulation benchmarks as
well as real-world robotic vision-based manipulation tasks. The results show that it
strongly outperforms DAgger-like approaches across the different tasks, especially
when the intervening experts are suboptimal. Additional ablations also empirically
verify the proposed theoretical justification that the performance of our method is
associated with the choice of intervention model and suboptimality of the expert.
Code and videos can be found on the project website: https://rlif-page.github.io