Sample-Efficient Automated Deep Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( RL ) algorithms are often sensitive to the choice of internal hyperparameters ( Jaderberg et al. , 2017 ; Mahmood et al. , 2018 ) , and the hyperparameters of the neural network architecture ( Islam et al. , 2017 ; Henderson et al. , 2018 ) , hindering them from being applied out-of-the-box to new environments . Tuning hyperparameters of RL algorithms can quickly become very expensive , both in terms of high computational costs and a large number of required environment interactions . Especially in real-world applications , sample efficiency is crucial ( Lee et al. , 2019 ) . Hyperparameter optimization ( HPO ; Snoek et al. , 2012 ; Feurer & Hutter , 2019 ) approaches often treat the algorithm under optimization as a black-box , which in the setting of RL requires a full training run every time a configuration is evaluated . This leads to a suboptimal sample efficiency in terms of environment interactions . Another pitfall for HPO is the non-stationarity of the RL problem . Hyperparameter settings optimal at the beginning of the learning phase can become unfavorable or even harmful in later stages ( François-Lavet et al. , 2015 ) . This issue can be addressed through dynamic configuration , either through self adaptation ( Tokic & Palm , 2011 ; François-Lavet et al. , 2015 ; Tokic , 2010 ) or through external adaptation as in population-based training ( PBT ; Jaderberg et al. , 2017 ) . However , current dynamic configuration approaches substantially increase the number of environment interactions . Furthermore , this prior work does not consider adapting the architecture . In this work , we introduce a simple meta-optimization framework for Sample-Efficient Automated RL ( SEARL ) to address all three challenges : sample-efficient HPO , dynamic configuration , and the dynamic modification of the neural architecture . The foundation of our approach is a joint optimization of an off-policy RL agent and its hyperparameters using an evolutionary approach . To reduce the amount of required environment interactions , we use a shared replay memory across the population of different RL agents . This allows agents to learn better policies due to the diverse collection of experience and enables us to perform AutoRL at practically the same amount of environment interactions as training a single configuration . Further , SEARL preserves the benefits of dynamic configuration present in PBT to enable online HPO and discovers hyperparameter schedules rather than a single static configuration . Our approach uses evolvable neural networks that preserve trained network parameters while adapting their architecture . We emphasize that SEARL is simple to use and allows efficient AutoRL for any off-policy deep RL algorithm . In a case study optimizing the popular TD3 algorithm ( Fujimoto et al. , 2018 ) in the MuJoCo benchmark suite we demonstrate the benefits of our framework and provide extensive ablation and analytic experiments . We show a 10× improvement in sample efficiency of the meta-optimization compared to random search and PBT . We also demonstrate the generalization capabilities of our approach by meta-optimizing the established DQN ( Mnih et al. , 2015 ) algorithm for the Atari benchmark . We provide an open-source implementation of SEARL.1 Our contributions are : • We introduce an AutoRL framework for off-policy RL which enables : ( i ) Sample-efficient HPO while training a population of RL agents using a shared replay memory . ( ii ) Dynamic optimization of hyperparameters to adjust to different training phases ; ( iii ) Online neural architecture search in the context of gradient-based deep RL ; • We propose a fair evaluation protocol to compare AutoRL and HPO in RL , taking into account the actual cost in terms of environment interactions . • We demonstrate the benefits of SEARL in a case study , reducing the number of environment interactions by up to an order of magnitude . 2 RELATED WORK . Advanced experience collection : Evolutionary RL ( ERL ) introduced by Khadka & Tumer ( 2018 ) and successors PDERL ( Bodnar et al. , 2020 ) and CERL ( Khadka et al. , 2019 ) combine Actor-Critic RL algorithms with genetic algorithms to evolve a small population of agents . This line of work mutates policies to increase the diversity of collected sample trajectories . The experience is stored in a shared replay memory and used to train an Actor-Critic learner with fixed network architectures using DDPG/TD3 while periodically adding the trained actor to a separate population of evolved actors . CERL extends this approach by using a whole population of learners with varying discount rates . However , this line of work aims to increase a single configuration ’ s performance , while our work optimizes hyperparameters and the neural architecture while training multiple agents . SEARL also benefits from a diverse set of mutated actors collecting experience in a shared replay memory . Schmitt et al . ( 2019 ) mix on-policy experience with shared experiences across concurrent hyperparameter sweeps to take advantage of parallel exploration . However , this work neither tackles dynamic configuration schedules nor architecture adaptation . ApeX/IMPALA : Resource utilization in the RL setting can be improved using multiple actors in a distributed setup and decoupling the learner from the actor . Horgan et al . ( 2018 ) extends a prioritized replay memory to a distributed setting ( Ape-X ) to scale experience collection for a replay memory used by a single trainer . In IMPALA ( Espeholt et al. , 2018 ) , multiple rollout actors asynchronously send their collected trajectories to a central learner through a queue . To correct the policy lag that this distributed setup introduces , IMPALA leverages the proposed V-trace algorithm for the central learner . These works aim at collecting large amounts of experience to benefit the learner , but they do not explore the space of hyperparameter configurations . In contrast , the presented work aims to reduce the number of environment interactions to perform efficient AutoRL . Neural architecture search with Reinforcement Learining : The work of Zoph & Le ( 2016 ) on RL for neural architecture search ( NAS ) is an interesting counterpart to our work on the intersection of RL and NAS . Zoph & Le ( 2016 ) employ RL for NAS to search for better performing architectures , whereas we employ NAS for RL to make use of better network architectures . AutoRL : Within the framework of AutoRL , the joint hyperparameter optimization and architecture search problem is addressed as a two-stage optimization problem in Chiang et al . ( 2019 ) , first shaping the reward function and optimizing for the network architecture afterward . Similarly , Runge et al . ( 2019 ) propose to jointly optimize algorithm hyperparameters and network architectures by searching 1Please find the source code on GitHub : github.com/automl/SEARL over the joint space . However , they treat the RL training as a black-box and do not focus on online optimization nor sample-efficiency . In contrast to black-box optimization , we jointly train the agent and dynamically optimize hyperparameters . Faust et al . ( 2019 ) uses an evolutionary approach to optimize a parametrized reward function based on which fixed network topologies are trained using standard RL algorithms , treating the RL algorithm together with a sampled reward function as a black-box optimizer . In this work , we do not use parametrized reward functions , but instead , directly optimize the environment reward . The main difference to this line of work is sample efficiency : While they train and evaluate thousands of configurations from scratch , we dynamically adapt the architecture and RL-algorithm hyperparameters online , thereby drastically reducing the total amount of interactions required for the algorithm to achieve good performance on a given task . We propose an evaluation protocol taking into account the aspect of sample-efficiency in RL in section 4.3 . Self-Tuning Actor-Critic : Zahavy et al . ( 2020 ) propose to meta-optimize a subset of differentiable hyperparameters in an outer loop using metagradients . This however , does not extend to non-differentiable hyperparameters and thus does not allow for online tuning of e.g . the network architecture . As a results , such hyperparameters can not be meta-optimized in their framework . HOOF : Paul et al . ( 2019 ) proposes sample-efficient hyperparameter tuning for policy gradient methods by greedily maximizing the value of a set of candidate policies at each iteration . In contrast to our work , HOOF performs HPO for on-policy algorithms that do not achieve comparable performance on continuous control tasks while requiring more interactions and not considering architecture optimization . Population-Based Training ( PBT ) : PBT ( Jaderberg et al. , 2017 ) is a widely used dynamic and asynchronous optimization algorithm . This approach adapts a population of different hyperparameter settings online and in parallel during training , periodically replacing inferior members of the population with more promising members . Similarly to SEARL , PBT can jointly optimize the RL agent and its hyperparameters online , making it the most closely related work . Recent work has improved upon PBT by using more advanced hyperparameter selection techniques ( Parker-Holder et al. , 2020 ) . In contrast to SEARL , PBT and follow-ups do not optimize the architecture and , more importantly , do not share experience within the population . As our experiments show , these advances lead to speedups of up to 10x in terms of sample efficiency . 3 SAMPLE-EFFICIENT AUTORL . In this section , we introduce a Sample-Efficient framework for Automated Reinforcement Learning ( SEARL ) based on an evolutionary algorithm acting on hyperparameters and gradient-based training using a shared experience replay . First , we discuss relevant background that describes SEARL building blocks before giving an overview of the proposed AutoRL framework , followed by a detailed description of each individual component . 3.1 BACKGROUND . Evolvable neural network : Using evolutionary algorithms to design neural networks , called Neuroevolution ( Floreano et al. , 2008 ; Stanley et al. , 2019 ) , is a long-standing approach . Some approaches only optimize the network weights ( Such et al. , 2017 ) , while others optimize architectures and weights jointly ( Zhang & Mühlenbein , 1993 ; Stanley & Miikkulainen , 2002 ) . To evolve the neural network in SEARL , we encode RL agent ’ s neural architectures by the number of layers and the nodes per layer , similar to ( Miikkulainen et al. , 2019 ) . When adding a new node to a layer , existing parameters are copied , and newly added parameters are initialized with a small magnitude . This is a common technique to preserve already trained network weights ( Wei et al. , 2016 ) . Shared experience replay : Replaying collected experiences ( Lin , 1992 ; Mnih et al. , 2015 ) smooths the training distribution over many past rollouts . The experience replay acts as a store for experience collected by agents interacting with the environment . Deep RL algorithms can sample from this storage to calculate gradient-based updates for the neural networks . It has been used and extended in various flavors , often to make use of diverse experience or experience collected in parallel ( Horgan et al. , 2018 ; Khadka & Tumer , 2018 ; Bodnar et al. , 2020 ; Khadka et al. , 2019 ) . SEARL employs a shared experience replay , which stores the diverse trajectories of all differently configured RL agents in the population so that each individual can benefit from the collective experiences during training . 3.2 FRAMEWORK . In SEARL , each individual in our population represents a deep reinforcement learning agent consisting of a policy and value network and the RL training algorithm hyperparameters , including the neural network architecture . The training and meta-optimization of these individuals take place in an evolutionary loop that consists of five basic phases ( initialization , evaluation , selection , mutation , and training ) , as shown in Figure 1 . During one epoch of this evolutionary loop , all individual properties could change through different mutations and training operators . This happens independently for each individual and can be processed in parallel . A novel feature of our approach is that rollouts of each individual are not only used for evaluation and selection purposes but also serve as experience for off-policy training of all agents and are stored in a shared replay memory . When changing the architecture , we follow the approach of Lamarckian evolution ( Ross , 1999 ) , where the updated weights of an agent during training are not only used in the evaluation phase but are preserved for the next generation . In the following , we describe the different phases of our evolutionary loop in detail ; we refer the reader to Appendix B for detailed pseudocode of the algorithm . Initialization : SEARL uses a population pop , of N individuals , each consisting of an RL agent Ai , and its hyperparameter settings θi . Thus , we can represent popg at each generation g as follows : popg = ( { A1 , θ1 } g , { A2 , θ2 } g , ... , { AN , θN } g ) ( 1 ) The individual ’ s hyperparameter setting θi is composed of architecture hyperparameters , such as the number of layers or the layer sizes , and algorithm hyperparameters , such as the learning rate . To arrive at a minimum viable neural network size , we start with reasonably small neural network architecture and enable its growth by using mutation operators . Other hyperparameters are set to some initial value , as they are subsequently adapted in the evolutionary loop . In our experiments we observed that using random initialization of hyperparameters for SEARL did not lead to large performance differences , see Appendix E. This suggests that there is very little domain-specific knowledge required to effectively use SEARL . Evaluation : After initialization and after each training phase , we evaluate each individual in the population using the RL agent , Ai , for at least one episode or a minimum number of steps in the environment . This ensures a minimum amount of new experience from each agent and keeps the stored trajectories in the shared replay memory diverse . The evaluation can be performed in parallel since each agent acts independently . We use the mean reward of the individual ’ s evaluation as fitness value fi . Selection : We use tournament selection with elitism ( Miller & Goldberg , 1995 ) for each prospective slot in the population of the new generation . For each tournament , k individuals are randomly chosen from the current population popg , and the individual with the largest fitness value fi is selected for the slot . We repeat this tournament N − 1 times to fill all slots . The size k allows us to control how greedily the selection mechanism picks candidates for the new population . We reserve one spot in the new population for the current population ’ s best- performing individual , thus preserving it across generations . Mutation : To explore the space of network weights and hyperparameters , we use different singleparent mutation operators . We apply one of the following operators uniformly at random to each member : ( 1 ) Mutation of the weights of the neural networks by adding Gaussian noise . ( 2 ) Change of activation function of the neural network . ( 3 ) Change of the neural network size by either adding additional nodes to a given layer or adding a new layer altogether while reusing the trained weights and initializing new weights randomly . ( 4 ) Change of algorithm hyperparameters . ( 5 ) No operation . We refer the reader to Appendix A for more details . Training : Using each individual ’ s current hyperparameters , we train it by sampling from the shared replay memory . Each individual is trained for as many steps as frames have been generated in the evaluation phase , as is common practice in deep RL . Optionally , the training time could be reduced by using only a fraction j of the steps to adapt to computational constraints . Since the neural network size could be subject to mutation between two training phases , the target network of the RL algorithm needs to be adapted too . Furthermore , the optimizer weight-parameters connected to individual network weights can not remain the same across generations . We address these issues by creating a new target network and re-initializing the optimizer at the beginning of each training phase . Our experiments show that this re-creation and re-initialization does not harm the performance of the considered RL algorithm . Please find more details in Appendix C. Like the evaluation phase , the training can be performed in parallel since every individual is trained independently . Shared experience replay : A shared experience replay memory collects trajectories from all evaluations and provides a diverse set of samples for each agent during the training phase . This helps to improve training speed and reduces the potential of over-fitting .

Summary: This paper propose a population-based AutoRL framework for hyperparameter optimization of off-policy RL algorithms. The framework optimizes both the hyperparameters and also the neural architecture in a one-shot manner, e.g., search and train at the same time. A shared experience replay buffer is used across the population, which as demonstrated in the experiments, substantially increase the sample efficiency compared to PBT and random search.