Advancements in Deep Reinforcement Learning: Algorithms and Implementations

The field of deep reinforcement learning sparked to life in 2013, when the first end-to-end learning system achieved near-human level performance playing Atari games from screen images. Ever since then, new achievements have been accruing at an astounding rate. By now, deep RL methods have achieved super-human performance in advanced, modern video games, and are also providing promising results in robotic control, to include self-driving cars. Deep RL is a confluence of multiple supporting theories and technologies, drawing for example from optimization, statistics, and animal learning, while relying on sophisticated computing hardware and software to run immense experiments. It has progressed largely using simulations, along three main branches of research separated by distinct learning rules. After introducing their principles, this thesis includes contributions to all three lines of research. Of the many possible ways to advance this multi-faceted field, this thesis follows a largely empirical path which begins with improving software implementations and ends with unlocking a new algorithmic paradigm./nIn the first implementation effort, techniques to scale deep RL algorithms to modern, multi-GPU supercomputers are introduced, thereby reducing the wall-clock time of some experiments from hours to minutes. Thereafter, the best of these techniques are incorporated into a new code base which for the first time incorporates all three families of algorithm over a unified and highly optimized infrastructure for RL. Offered freely to the public, the code base has already reached well over a thousand researchers, and it was used to conduct most of the following research./nThe remaining contributions are in the form of new algorithms. The first is a novel recurrent architecture and associated auxiliary training losses which significantly improve learning in complex environments requiring long-term memory. Examples include 3-D visual environments such as maze navigation and games of capture the flag. The second algorithmic contribution is in the area of safe RL, in environments where the agent must learn to avoid certain hazards, or ``costs'', while learning to maximize the usual rewards. The new algorithm is an elegant enhancement that brings PID control to Lagrangian methods, a staple in constrained optimization. The effect is to balance costs against rewards more responsively, greatly reducing constraint violations during learning. The third and final algorithmic contribution is a more dramatic departure from prior art. While the sparking achievement of deep RL was the ability to learn visual features from rewards in an end-to-end fashion, that method suffers limitations such as: 1) slow feature learning under sparse rewards, 2) learning features which do not generalize well for use in other tasks, and 3) difficulty in isolating the phenomena of representation learning and strategy learning from each other. The final chapter in this thesis circumvents such limitations by introducing unsupervised learning techniques for training visual feature encoders. Without having used any rewards for training, these encoders can be used as-is for downstream RL tasks. This achievement opens new possibilities for research and flexible application of deep RL./nAltogether, deep RL is an exciting, multi-dimensional technology, much too broad to give full coverage in this thesis. The research arc presented here does carve a wide swath, however, which advances deep RL on several frontiers. Hopefully, the presentation will leave the reader with appreciation for some of the major new possibilities, along with the effort required to realize them.

Advisor: Abbeel, Pieter
DeWeese, Michael