Sample-Efficient Automated Deep Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( RL ) algorithms are often sensitive to the choice of internal hyperparameters ( Jaderberg et al. , 2017 ; Mahmood et al. , 2018 ) , and the hyperparameters of the neural network architecture ( Islam et al. , 2017 ; Henderson et al. , 2018 ) , hindering them from being applied out-of-the-box to new environments . Tuning hyperparameters of RL algorithms can quickly become very expensive , both in terms of high computational costs and a large number of required environment interactions . Especially in real-world applications , sample efficiency is crucial ( Lee et al. , 2019 ) . Hyperparameter optimization ( HPO ; Snoek et al. , 2012 ; Feurer & Hutter , 2019 ) approaches often treat the algorithm under optimization as a black-box , which in the setting of RL requires a full training run every time a configuration is evaluated . This leads to a suboptimal sample efficiency in terms of environment interactions . Another pitfall for HPO is the non-stationarity of the RL problem . Hyperparameter settings optimal at the beginning of the learning phase can become unfavorable or even harmful in later stages ( François-Lavet et al. , 2015 ) . This issue can be addressed through dynamic configuration , either through self adaptation ( Tokic & Palm , 2011 ; François-Lavet et al. , 2015 ; Tokic , 2010 ) or through external adaptation as in population-based training ( PBT ; Jaderberg et al. , 2017 ) . However , current dynamic configuration approaches substantially increase the number of environment interactions . Furthermore , this prior work does not consider adapting the architecture . In this work , we introduce a simple meta-optimization framework for Sample-Efficient Automated RL ( SEARL ) to address all three challenges : sample-efficient HPO , dynamic configuration , and the dynamic modification of the neural architecture . The foundation of our approach is a joint optimization of an off-policy RL agent and its hyperparameters using an evolutionary approach . To reduce the amount of required environment interactions , we use a shared replay memory across the population of different RL agents . This allows agents to learn better policies due to the diverse collection of experience and enables us to perform AutoRL at practically the same amount of environment interactions as training a single configuration . Further , SEARL preserves the benefits of dynamic configuration present in PBT to enable online HPO and discovers hyperparameter schedules rather than a single static configuration . Our approach uses evolvable neural networks that preserve trained network parameters while adapting their architecture . We emphasize that SEARL is simple to use and allows efficient AutoRL for any off-policy deep RL algorithm . In a case study optimizing the popular TD3 algorithm ( Fujimoto et al. , 2018 ) in the MuJoCo benchmark suite we demonstrate the benefits of our framework and provide extensive ablation and analytic experiments . We show a 10× improvement in sample efficiency of the meta-optimization compared to random search and PBT . We also demonstrate the generalization capabilities of our approach by meta-optimizing the established DQN ( Mnih et al. , 2015 ) algorithm for the Atari benchmark . We provide an open-source implementation of SEARL.1 Our contributions are : • We introduce an AutoRL framework for off-policy RL which enables : ( i ) Sample-efficient HPO while training a population of RL agents using a shared replay memory . ( ii ) Dynamic optimization of hyperparameters to adjust to different training phases ; ( iii ) Online neural architecture search in the context of gradient-based deep RL ; • We propose a fair evaluation protocol to compare AutoRL and HPO in RL , taking into account the actual cost in terms of environment interactions . • We demonstrate the benefits of SEARL in a case study , reducing the number of environment interactions by up to an order of magnitude . 2 RELATED WORK . Advanced experience collection : Evolutionary RL ( ERL ) introduced by Khadka & Tumer ( 2018 ) and successors PDERL ( Bodnar et al. , 2020 ) and CERL ( Khadka et al. , 2019 ) combine Actor-Critic RL algorithms with genetic algorithms to evolve a small population of agents . This line of work mutates policies to increase the diversity of collected sample trajectories . The experience is stored in a shared replay memory and used to train an Actor-Critic learner with fixed network architectures using DDPG/TD3 while periodically adding the trained actor to a separate population of evolved actors . CERL extends this approach by using a whole population of learners with varying discount rates . However , this line of work aims to increase a single configuration ’ s performance , while our work optimizes hyperparameters and the neural architecture while training multiple agents . SEARL also benefits from a diverse set of mutated actors collecting experience in a shared replay memory . Schmitt et al . ( 2019 ) mix on-policy experience with shared experiences across concurrent hyperparameter sweeps to take advantage of parallel exploration . However , this work neither tackles dynamic configuration schedules nor architecture adaptation . ApeX/IMPALA : Resource utilization in the RL setting can be improved using multiple actors in a distributed setup and decoupling the learner from the actor . Horgan et al . ( 2018 ) extends a prioritized replay memory to a distributed setting ( Ape-X ) to scale experience collection for a replay memory used by a single trainer . In IMPALA ( Espeholt et al. , 2018 ) , multiple rollout actors asynchronously send their collected trajectories to a central learner through a queue . To correct the policy lag that this distributed setup introduces , IMPALA leverages the proposed V-trace algorithm for the central learner . These works aim at collecting large amounts of experience to benefit the learner , but they do not explore the space of hyperparameter configurations . In contrast , the presented work aims to reduce the number of environment interactions to perform efficient AutoRL . Neural architecture search with Reinforcement Learining : The work of Zoph & Le ( 2016 ) on RL for neural architecture search ( NAS ) is an interesting counterpart to our work on the intersection of RL and NAS . Zoph & Le ( 2016 ) employ RL for NAS to search for better performing architectures , whereas we employ NAS for RL to make use of better network architectures . AutoRL : Within the framework of AutoRL , the joint hyperparameter optimization and architecture search problem is addressed as a two-stage optimization problem in Chiang et al . ( 2019 ) , first shaping the reward function and optimizing for the network architecture afterward . Similarly , Runge et al . ( 2019 ) propose to jointly optimize algorithm hyperparameters and network architectures by searching 1Please find the source code on GitHub : github.com/automl/SEARL over the joint space . However , they treat the RL training as a black-box and do not focus on online optimization nor sample-efficiency . In contrast to black-box optimization , we jointly train the agent and dynamically optimize hyperparameters . Faust et al . ( 2019 ) uses an evolutionary approach to optimize a parametrized reward function based on which fixed network topologies are trained using standard RL algorithms , treating the RL algorithm together with a sampled reward function as a black-box optimizer . In this work , we do not use parametrized reward functions , but instead , directly optimize the environment reward . The main difference to this line of work is sample efficiency : While they train and evaluate thousands of configurations from scratch , we dynamically adapt the architecture and RL-algorithm hyperparameters online , thereby drastically reducing the total amount of interactions required for the algorithm to achieve good performance on a given task . We propose an evaluation protocol taking into account the aspect of sample-efficiency in RL in section 4.3 . Self-Tuning Actor-Critic : Zahavy et al . ( 2020 ) propose to meta-optimize a subset of differentiable hyperparameters in an outer loop using metagradients . This however , does not extend to non-differentiable hyperparameters and thus does not allow for online tuning of e.g . the network architecture . As a results , such hyperparameters can not be meta-optimized in their framework . HOOF : Paul et al . ( 2019 ) proposes sample-efficient hyperparameter tuning for policy gradient methods by greedily maximizing the value of a set of candidate policies at each iteration . In contrast to our work , HOOF performs HPO for on-policy algorithms that do not achieve comparable performance on continuous control tasks while requiring more interactions and not considering architecture optimization . Population-Based Training ( PBT ) : PBT ( Jaderberg et al. , 2017 ) is a widely used dynamic and asynchronous optimization algorithm . This approach adapts a population of different hyperparameter settings online and in parallel during training , periodically replacing inferior members of the population with more promising members . Similarly to SEARL , PBT can jointly optimize the RL agent and its hyperparameters online , making it the most closely related work . Recent work has improved upon PBT by using more advanced hyperparameter selection techniques ( Parker-Holder et al. , 2020 ) . In contrast to SEARL , PBT and follow-ups do not optimize the architecture and , more importantly , do not share experience within the population . As our experiments show , these advances lead to speedups of up to 10x in terms of sample efficiency . 3 SAMPLE-EFFICIENT AUTORL . In this section , we introduce a Sample-Efficient framework for Automated Reinforcement Learning ( SEARL ) based on an evolutionary algorithm acting on hyperparameters and gradient-based training using a shared experience replay . First , we discuss relevant background that describes SEARL building blocks before giving an overview of the proposed AutoRL framework , followed by a detailed description of each individual component . 3.1 BACKGROUND . Evolvable neural network : Using evolutionary algorithms to design neural networks , called Neuroevolution ( Floreano et al. , 2008 ; Stanley et al. , 2019 ) , is a long-standing approach . Some approaches only optimize the network weights ( Such et al. , 2017 ) , while others optimize architectures and weights jointly ( Zhang & Mühlenbein , 1993 ; Stanley & Miikkulainen , 2002 ) . To evolve the neural network in SEARL , we encode RL agent ’ s neural architectures by the number of layers and the nodes per layer , similar to ( Miikkulainen et al. , 2019 ) . When adding a new node to a layer , existing parameters are copied , and newly added parameters are initialized with a small magnitude . This is a common technique to preserve already trained network weights ( Wei et al. , 2016 ) . Shared experience replay : Replaying collected experiences ( Lin , 1992 ; Mnih et al. , 2015 ) smooths the training distribution over many past rollouts . The experience replay acts as a store for experience collected by agents interacting with the environment . Deep RL algorithms can sample from this storage to calculate gradient-based updates for the neural networks . It has been used and extended in various flavors , often to make use of diverse experience or experience collected in parallel ( Horgan et al. , 2018 ; Khadka & Tumer , 2018 ; Bodnar et al. , 2020 ; Khadka et al. , 2019 ) . SEARL employs a shared experience replay , which stores the diverse trajectories of all differently configured RL agents in the population so that each individual can benefit from the collective experiences during training . 3.2 FRAMEWORK . In SEARL , each individual in our population represents a deep reinforcement learning agent consisting of a policy and value network and the RL training algorithm hyperparameters , including the neural network architecture . The training and meta-optimization of these individuals take place in an evolutionary loop that consists of five basic phases ( initialization , evaluation , selection , mutation , and training ) , as shown in Figure 1 . During one epoch of this evolutionary loop , all individual properties could change through different mutations and training operators . This happens independently for each individual and can be processed in parallel . A novel feature of our approach is that rollouts of each individual are not only used for evaluation and selection purposes but also serve as experience for off-policy training of all agents and are stored in a shared replay memory . When changing the architecture , we follow the approach of Lamarckian evolution ( Ross , 1999 ) , where the updated weights of an agent during training are not only used in the evaluation phase but are preserved for the next generation . In the following , we describe the different phases of our evolutionary loop in detail ; we refer the reader to Appendix B for detailed pseudocode of the algorithm . Initialization : SEARL uses a population pop , of N individuals , each consisting of an RL agent Ai , and its hyperparameter settings θi . Thus , we can represent popg at each generation g as follows : popg = ( { A1 , θ1 } g , { A2 , θ2 } g , ... , { AN , θN } g ) ( 1 ) The individual ’ s hyperparameter setting θi is composed of architecture hyperparameters , such as the number of layers or the layer sizes , and algorithm hyperparameters , such as the learning rate . To arrive at a minimum viable neural network size , we start with reasonably small neural network architecture and enable its growth by using mutation operators . Other hyperparameters are set to some initial value , as they are subsequently adapted in the evolutionary loop . In our experiments we observed that using random initialization of hyperparameters for SEARL did not lead to large performance differences , see Appendix E. This suggests that there is very little domain-specific knowledge required to effectively use SEARL . Evaluation : After initialization and after each training phase , we evaluate each individual in the population using the RL agent , Ai , for at least one episode or a minimum number of steps in the environment . This ensures a minimum amount of new experience from each agent and keeps the stored trajectories in the shared replay memory diverse . The evaluation can be performed in parallel since each agent acts independently . We use the mean reward of the individual ’ s evaluation as fitness value fi . Selection : We use tournament selection with elitism ( Miller & Goldberg , 1995 ) for each prospective slot in the population of the new generation . For each tournament , k individuals are randomly chosen from the current population popg , and the individual with the largest fitness value fi is selected for the slot . We repeat this tournament N − 1 times to fill all slots . The size k allows us to control how greedily the selection mechanism picks candidates for the new population . We reserve one spot in the new population for the current population ’ s best- performing individual , thus preserving it across generations . Mutation : To explore the space of network weights and hyperparameters , we use different singleparent mutation operators . We apply one of the following operators uniformly at random to each member : ( 1 ) Mutation of the weights of the neural networks by adding Gaussian noise . ( 2 ) Change of activation function of the neural network . ( 3 ) Change of the neural network size by either adding additional nodes to a given layer or adding a new layer altogether while reusing the trained weights and initializing new weights randomly . ( 4 ) Change of algorithm hyperparameters . ( 5 ) No operation . We refer the reader to Appendix A for more details . Training : Using each individual ’ s current hyperparameters , we train it by sampling from the shared replay memory . Each individual is trained for as many steps as frames have been generated in the evaluation phase , as is common practice in deep RL . Optionally , the training time could be reduced by using only a fraction j of the steps to adapt to computational constraints . Since the neural network size could be subject to mutation between two training phases , the target network of the RL algorithm needs to be adapted too . Furthermore , the optimizer weight-parameters connected to individual network weights can not remain the same across generations . We address these issues by creating a new target network and re-initializing the optimizer at the beginning of each training phase . Our experiments show that this re-creation and re-initialization does not harm the performance of the considered RL algorithm . Please find more details in Appendix C. Like the evaluation phase , the training can be performed in parallel since every individual is trained independently . Shared experience replay : A shared experience replay memory collects trajectories from all evaluations and provides a diverse set of samples for each agent during the training phase . This helps to improve training speed and reduces the potential of over-fitting .

Summary: In this paper, the authors intend to propose an efficient automated reinforcement learning (RL) framework. To achieve this goal, they integrate three technologies, i.e., evolutionary RL for hyperparameter search, evolvable neural network for policy network design, and shared experience replay for improving data usage. The paper uses a case study on MuJoCo to demonstrate the claimed advantages over baselines.