SERL: A Software Suite for Sample-Efficient Robotic Reinforcement
  Learning
Authors: name: Jianlan Luo; name: Zheyuan Hu; name: Charles Xu; name: You Liang Tan; name: Jacob Berg; name: Archit Sharma; name: Stefan Schaal; name: Chelsea Finn; name: Abhishek Gupta; name: Sergey Levine
Published: 2024-01-29 10:01:10

Abstract: In recent years, significant progress has been made in the field of robotic
reinforcement learning (RL), enabling methods that handle complex image
observations, train in the real world, and incorporate auxiliary data, such as
demonstrations and prior experience. However, despite these advances, robotic
RL remains hard to use. It is acknowledged among practitioners that the
particular implementation details of these algorithms are often just as
important (if not more so) for performance as the choice of algorithm. We posit
that a significant challenge to widespread adoption of robotic RL, as well as
further development of robotic RL methods, is the comparative inaccessibility
of such methods. To address this challenge, we developed a carefully
implemented library containing a sample efficient off-policy deep RL method,
together with methods for computing rewards and resetting the environment, a
high-quality controller for a widely-adopted robot, and a number of challenging
example tasks. We provide this library as a resource for the community,
describe its design choices, and present experimental results. Perhaps
surprisingly, we find that our implementation can achieve very efficient
learning, acquiring policies for PCB board assembly, cable routing, and object
relocation between 25 to 50 minutes of training per policy on average,
improving over state-of-the-art results reported for similar tasks in the
literature. These policies achieve perfect or near-perfect success rates,
extreme robustness even under perturbations, and exhibit emergent recovery and
correction behaviors. We hope that these promising results and our high-quality
open-source implementation will provide a tool for the robotics community to
facilitate further developments in robotic RL. Our code, documentation, and
videos can be found at https://serl-robot.github.io/