Sample-Efficient Automated Deep Reinforcement Learning
Authors: Jörg K.H. Franke
Gregor Koehler
André Biedenkapp
Frank Hutter
Venue: ICLR 2021 Poster (2021)
Keywords: AutoRL
Deep Reinforcement Learning
Hyperparameter Optimization
Neuroevolution

Abstract: Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters. This sensitivity can partly be attributed to the non-stationarity of the RL problem, potentially requiring different hyperparameter settings at various stages of the learning process. Additionally, in the RL setting, hyperparameter optimization (HPO) requires a large number of environment interactions, hindering the transfer of the successes in RL to real-world applications. In this work, we tackle the issues of sample-efficient and dynamic HPO in RL. We propose a population-based automated RL (AutoRL) framework to meta-optimize arbitrary off-policy RL algorithms. In this framework, we optimize the hyperparameters and also the neural architecture while simultaneously training the agent. By sharing the collected experience across the population, we substantially increase the sample efficiency of the meta-optimization. We demonstrate the capabilities of our sample-efficient AutoRL approach in a case study with the popular TD3 algorithm in the MuJoCo benchmark suite, where we reduce the number of environment interactions needed for meta-optimization by up to an order of magnitude compared to population-based training.

Published as a conference paper at ICLR 2021
SAMPLE-EFFICIENT AUTOMATED DEEP REINFORCEMENT LEARNING
Jörg K.H. Franke', Gregor Köhler², André Biedenkapp' & Frank Hutter DDepartment of Computer Science, University of Freiburg, Germany "German Cancer Research Center, Heidelberg, Germany
Bossh Center for Artificial Intelligence, Renningen, Germany franke jecs uni www.bbrrgg... de
ABSTRACT
Despite significant progress in challenging problems across various domains, ap- plying sate-of-the-ar deep reinforcement learning (RL) algorithms remains chal- lenging due to their sensitivity to the choice of hyperparameters. This sensitivity can partly be attributed to the non-stationarity of the RL problem, potentially re- quiring different hyperparameter settings various stages of the learning process. Additionally, in the RL setting, hyperparameter optimization (HPO) requires a large number of environment interactions, hindering the transfer of the successes in RI to real-world applications. In this work, we tackle the issues sample-efficient and dynamic HPO in RL. We propose population-based automated RL (AutoRL) framework to meta-optimize arbitrary off-policy RL algorithms. In this framework, we optimize the hyperparameters and also the neural architecture while simultane- ously training the agent. By sharing the collected experience across the population, we substantially increase the sample efficiency of the meta-optimizztion. We demonstrate the capabilities of our sample-efficient AutoRL approach in case study with the popular TD3 algorithm in the MuloCo benehmark suite, where we reduce the number of environment interactions needed for opttimzttiiiiiionnn by up to an order of magnitude compared to population-base bassd training.
INTRODUCTION
Deep reinforcement learning (RL) algorithms are often sensitive to the choice of internal hyper- parameters (Jaderberg et al. 20177] Mahmood et al. 2018), and the hyperparameters of the neural network architecture (Islam al. [ et al. 2018), hindering them from being applied out-of-the-box to new environments. Tuning hyperparameters RL algorithms can quickly become very expensive, both in terms of high computational costs and large number of required environment interactions. Especially in real-world applications, sample efficiency is crucial (Lee et al. 2019).
Hyperparameter optimization (HPO; Snoek al. 2012. Feurer & Hutter 2019) approaches often treat the algorithm under optimization as a black-bon box, which in the setting of RL requires a full training run every time configuration is evaluated. This leads a suboptimal sample efficiency in terms of environment interactions. Another pitfall for HPO is the non-stationarity of the RL problem. Hyperparameter settings optimal at the beginning of the learning phase can become unfavorable or even harmful in later stages (François- Lavet et al. 2015). This issue can be addressed through dynamic configuration, either through self adaptation (Tokic & Palm 2011 François-Lavet al., 2015) Tokic 2010) through external adaptation in population-based training
al. 2017). However, current dynamic configuration approaches substantially increase the number of environment interactions. Furthermore, this prior work does not consider adapting the architecture.
In this work, we introduce a simple meta-optimization framework for Cample-Efficierr Automated RL (SEARL) to address all three challenges: sample-efficient HPO, dynamic configuration, and the dynamic modification of the neural architecture. The foundation of our approach joint optimization of off- off-policy RL agent and its hyperparameters using evolutionary approach. To reduce the amount of required environment interactions, we use a shared replay memory across the population of different RI. agents. This allows agents to learn better policies due to the diverse

---

Published as a conference paper at ICLR 2021
collection of experience and enables to perform AutoRL at practically the same amount of environment interactions as training a single configuration. Further, SEARL preserves the benefits of dynamic configuration present in PBT to enable online HPO and discovers hyperparameter schedules rather than a single static configuration. Our approach uses evolvable neural networks that preserve trained network parameters while adapting their architecture. We emphasize that SEARL is simple use and allows efficient AutoRL for any off-po deep RL algorithm.
In case study optimizing the popular TD3 algorithm (Fujimoto al. 2018 in the MuJoCo benchmark suite we demonstrate the benefits of our framework and provide extensive ablation and analytic experiments. We show a 10x improvement in sample efficiency of the neta-oppimizztion compared to random search and PBT. We also demonstrate the generalization capabilities of our approach by meta-optimizing the established DQN (Mnih et al.) 2015) algorithm for the Atari benchmark. We provide an open-source implementation of SEARL Our contributions are:
We introduce an AutoRL framework for off- RL which enables: (i) Sample-efficient HPO while training a population of RL agents using shared replay memory. (ii) Dynamic optimization of hyperparameters to adjust to different training phases; (iii) Online neural architecture search in the context of gradient- based deep RI.;
We propose fair evaluation protocol to compare AutoRL and HPO in RL, taking into account the actual cost in terms of environment interactions.
We demonstrate the benefits of SEARL in case study, reducing the number of environment interactions by up to an order of magnitude.
2 RELATED WORK
Advanced experience collection: Evolutionary RL (ERL) introduced by Khadka & Tumer (018)
and successors PDERL (Bodnar et al. (2020) and CERL (Khadka et al., (209)) combine Actor-Critic RL algorithms with genetic algorithms to evolve small population of agents. This line of work mutates policies to increase the diversity of collected sample trajectories. The experience is stored in shared replay memory and used train an Actor-Critic learner with fixed network architectures using DDPG/TD3 while periodically adding the trained actor to separate population of evolved actors. CERL extends this approach by using a whole population of learners with varying discount rates. However, this line of work aims to increase a single configuration's performance, while our work optimizes hyperparameters and the neural architecture while training multiple agents. SEARL also benefits from diverse set of mutated actors collecting experience shared replay memory. Schmitt et al. (2019) mix on-policy experience with shared experiences across concurrent hyperparameter sweeps take advantage of parallel exploration. However, this work neither tackles dynamic configuration schedules nor architecture adaptation.
ApeX/IMPAAA: Resource utilization in the RL setting can be improved using multiple actors in distributed setup and decoupling the learner from the actor, HHrgan et al. (2018) extends a prioritized replay memory distributed setting (Ape-X) to scale experience collection for replay memory used by a single trainer. In IMPALA (Espeholt et al.] 2018), multiple rollout actors asynchronously send their collected trajectories central learner through queue. To correct the policy lag that this distributed setup introduces, IMPALA leverages the proposed V-trace algorithm for the central learner. These works aim at collecting large amounts of experience to benefit the learner, but they do not explore the space of hyperparameter configurations. In contrast, the presented work aims to reduce the number of environment interactions to perform efficient AutoRL.
Neural architecture search with Reinforcement Learining: The work of Zoph & Le 2016) on RL for neural architecture search (NAS) is an interesting counterpart to our work on the intersection of RL and NAS. Zoph& & Le (2016) employ RL for NAS to search for better performing architectures, whereas we employ NAS for RL to make use of better network architectures.
Within the framework of AutoRL, the joint hyperparameter optimization and architecture search problem is addressed as a two-stage optimization problem in Chiang et al. (2019), first shaping the reward function and optimizing for the network architecture afterward. Similarly, |Runge et al.) (2019) propose to jointly optimize algorithm hyperparameters and network architectures by searching
AutoRL:
Please find the source code on GitHub: gituuh.com/autom//EAARI
2

---

Published as a conference paper at ICLR 2021
over the joint space. However, they treat the RL training black-box and do not focus online optimization nor sample-effiiinney. In contrast to black-bot optimization, we jointly train the agent and dynamically optimize hyperparameters. Faust et al. (2019) uses an evolutionary approach to optimize a parametrized reward function based on which fixed network topologies are trained using standard RI algorithms, treating the RL algorithm together with sampled reward function black-box optimizer. In this work, we do not use parametrized reward functions, but instead, directly optimize the environment reward. The main difference to this line of work is sample efficiency: While they train and evaluate thousands of configurations from scratch, we dynamically adapt the architecture and RL-algorithm hyperparameters online, thereby drastically reducing the total amount of interactions required for the algorithm to achieve good performance on a given task. We propose an evaluation protocol taking into account the aspect of sample-efficiency in RI, section 4.3
Self- Self-Tuning Actor-Critic: Zahavy et al. 2020) propose to meta-optimize subset of differen- tiable hyperparameters in an outer loop using metagradients. This however, does not extend to non-differentiable hyperparameters and thus does not allow for online tuning of e.g. the network architecture. As a results, such hyperparameters can not be meta-optimized in their framework.
HOOF: Paul et al. (2019) proposes sample-efficient hyperparameter tuning for policy gradient methods by greedily maximizing the value of a set of candidate policies at each iteration. In contrast to our work, HOOF performs HPO for on-policy algorithms that do not achieve comparable performance on continuous control tasks while requiring more interactions and not considering architecture optimization.
Population-Based Training (PBT): PBT (Jaderberg et al. 2017 is a widely used dynamic and asynchronous optimization algorithm. This approach adapts a population of different hyperparameter settings online and in parallel during training. periodically replacing inferior members of the popula- tion with more promising members. Similarly to SEARL, PBT can jointly optimize the RL agent and its hyperparameters online, making it the most closely related work. Recent work has improved upon PBT by using more advanced hyperparameter selection techniques (Parker-Holdd et al. 2020). In contrast to SEARL, PBT and follow-ups do not optimize the architecture and, more importantly, do not share experience within the population. As our experiments show, these advances lead to speedups of up to 10x in terms of sample efficiency.
SAMPLE-EFFICIEN AUTORL
In this section, we introduce a Sample-Efficient framework for Automated Reinforcement Learning (SEARL.) based on an evolutionary algorithm acting on hyperparameters and gradient-based training using shared experience replay. First, we discuss relevant background that describes SEARL building blocks before giving an overview of the proposed AutoRI. framework, followed by a detailed description of each individual component.
3.1 BACKGROUND
Evolvable neural network: Using evolutionary algorithms to design neural networks, called Neuroevolution (Floreano et al., 2008 [Stanley et al., (019). is long-standing approach. Some approaches only optimize the network weights (Such et al.| (207). while others optimize architectures and weights jointly (Zhang & Mühlenbein, 1993] Stanley & Müikkulainen) 2002). To evolve the neural network in SEARL, we encode RL agent's 's neural architectures by the number of layers and the nodes per layer, similar to (Mükkulainenn et al. 2019). When adding a new node to a layer, existing parameters are copied, and newly added parameters are initialized with a small magnitude. This is a common technique to preserve already trained network weights (Wei et al. 2016).
1992 Mnih et al. 2015) smooths the training distribution over many past rollouts. The experience replay acts as a store for experience collected by agents interacting with the environment. Deep RL algorithms can sample from this storage to calculate gradient-based updates for the neural networks. It has been used and extended in various flavors, often to make use of diverse experience or experience collected in parallel (Horgan eet al. 2018: Khadka & Tumer 2018 Bodnar et al.] [2020) Khadka et al. 2019). SEARL employs shared experience replay, which stores the diverse trajectories of all differently configured RL agents in the population that each individual can benefit from the collective experiences during training.
Shared experience replay: Replaying collected experiences (Lin
3

---

set to Published as a conference paper at ICLR 2021
3.2 FRAMEWORK
In SEARL, each individual in our population represents a deep reinforcement learning agent consisting of policy and value network and the RL training algorithm hyperparameters, including the neural network architecture. The training and meta-optimization of these individuals take place in an evolutionary loop that consists of five basic phases (initialization, evaluation, selection, mutation, and training), as shown in Figure During one epoch of this evolutionary loop, all individual properties could change through different mutations and training operators. This happens independently for each individual and can be processed in parallel.
novel feature of our approach is that rollouts of each individual are not only used for evaluation and selection purposes but also serve experience for off-policy training of all agents and are stored shared replay memory. When changing the architecture, we follow the approach of Lamarckian evolution [ooss 1999), where the updated weights of an agent during training are not only used in the evaluation phase but are preserved for the next generation. the following, we describe the different phases of our evolutionary loop in detail; refer the reader to Appendix B for detailed pscudocode of the algorithm.
Initialization: SEARL uses population pop, of N individuals, each consisting of RL agent A, and its hyperparameter settings 0.. Thus, we can represent pop, at each generation as follows:
popg ([A1, 01) )1 {22, 02) .... {AN, )g)
The individual's hyperparameter setting 0; is composed architecture hyperparameters, such as the number of layers or the layer sizes, and algorithm hyperparameters, such as the learning rate. To arrive at a minimum viable neural network size, we start with reasonably small neural network architecture and enable its growth by using mutation operators. Other hyperparameters some initial value, as they are subsequently adapted in the evolutionary loop. In our experiments observed that using random initialization of hyperparameters for SEARL did not lead to large performance differences, see Appendix E. This suggests that there is very little domain-specific knowledge required to effectively use SEARL.
Evaluation: After initialization and after each training phase, we evaluate each individual in the population using the RL agent, A;, for at least one episode or minimum number of steps in the environment. This ensures minimum amount of new experience from each agent and keeps the stored trajectories in the shared re- play memory diverse. The evaluation can performed in parallel since each agent acts in- dependently. We use the mean reward of the individual's evaluation as fitness value fr.
Initialize Population
Transitions
Evaluation Environment
Tournament Selection
Selection: We use tournament selection with elitism Millerr & Goldberg 1995) for each prospective slot in the population of the new generation. For each tournament, & individuals are randomly chosen from the current popula- tion popg- and the individual with the largest fitness value f; is selected for the slot. We re- peat this tournament N times to fill all slots. The size l allows us to control how greedily the selection mechanism picks candidates for the new population. We reserve spot in the new population for the current population's 's best- performing individual, thus preserving it across generations.
Mutations
Traneition Samples
Replay Merorr
RL-Training Steps
Figure 1: Overview of Sample-efficient AutoRL (SEARL).
To explore the space of network weights and hyperparameters, we use different single- parent mutation operators. We apply one of the following operators uniformly at random to each member: (1) Mutation the weights of the neural networks by adding Gaussian noise. (2) Change of activation function of the neural network. (3) Change of the neural network size by either adding
Mutation:
4

---

Published as a conference paper at ICLR 2021
additional nodes to given layer adding a new layer altogether while reusing the trained weights and initializing new weights randomly. (4) Change of algorithm hyperparameters. (5) No operation. We refer the reader to Appendix for more details.
Using each individual's current hyperparameters, we train by sampling from the shared replay memory. Each individual is trained for many steps frames have been generated in the evaluation phase, as is common practice in deep RL. Optionally, the training time could be reduced by using only a fraction j of the steps to adapt to computational constraints. Since the neural network size could be subject to mutation between two training phases, the target network of the RL algorithm needs to be adapted too. Furthermore, the optimizer weight-parametee network weights cannot remain the same across generations. We address these issues by creating a new target network and re-initializing the optimizer at the beginning of each training phase. Our experiments show that this re-creation and re-initialization does not harm the performance of the considered RL algorithm. Please find more details in Appendix C Like the evaluation phase, the training can be performed in parallel since every individual is trained independently.
Training:
connected to individual
Shared experience replay: A shared experience replay memory collects trajectories from all evaluations and provides a diverse set of samples for each agent during the training phase. This helps to improve training speed and reduces the potential of over- fittiiiting.
4 CASE STUDY: META-OPTIMIZATION OF TD3
In this section, we demonstrate the capabilities and benefits of SEARL. We simulate the meta- optimization of the Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3; Fujimoto al. 2018) 2018)) on the widely used MuJoCo continuous control benchmark suite [Todorov et al. 2012). well-performing hyperparameter configuration on MuloCo is known, even though the optimization of TD3's hyperparameters is unreported. This makes up a useful setting to investigate meta-optimization with SEARL compared to unknown algorithms or environments.
To study SEARL, we assume zero prior knowledge about the optimal hyperparameter configuration and compare SEARL against two plausible baselines. All meta-algorithms are tasked with optimizing network architecture, activations, and learning rates of the actor and the critic. Following Fujimoto et al. (2018), evaluate all methods on the following MuJoCo environments: Ha fCheetah, h,, Walker2D, Ant, Hopper r and Reacher. In general, other off-policy RL algorithms could also be optimized using SEARL due the generality of our framework.
4.1 BASELINES
Since there is directly comparable approach for efficient AutoRL (see Section 2}. compare SEARL to random search and modified version of PBT that allows for architecture adaptation. As RL is sensitive to the choice of hyperparameters, the design of the configuration space matters. To allow for fair comparison, we keep the search spaces for all methods similar; see Appendix D
Random Search: Random search makes no assumptions on the algorithm and is a widely used baseline for HPO (Feurer& & Hutter 2019). We sample 20 random configurations and train TD3 using each configuration without any online architecture hyperparameters changes for 1M frames. We select the configuration that resulted in the best training performance and evaluate it with 10 different random seeds to obtain its validation performance.
Modified Population- Based Training: To fairly compare PBT with our proposed approach, we modify PBT be able to adapt the neural architecture of the networks. We perform the same mutations to the activation function and neural network size as in SEARL (Section 3.2). This modification is required for a fair comparison since using an arbitrary fixed network size could compromise PBT's performance. On the other hand, using the network size reported by Fujimoto et al. (2018) would not take into account any potential HPO due to an unknown budget. In other aspects, follow Daderberg et al. (2017), e.g., initializing the population by sampling from the configuration space uniformly at random. We train and evaluate a population of 20 agents for 100 generations. In every generation, the agents are trained and evaluated for 10 000 steps each.
5

---

Published as a conference paper at ICLR 2021
12:00
5000
4000
8000
Random Search modified PBT
6000
3000
SEARL (ours)
4000
2000
1000
90-
10F
10
10
10
10
207
Tonal Enstatmmettt Stape
Total Environment Steps
(a) HalfCheetah
(h) Walker2D
3500
0.0
1000
2.5
4000
3000
1500
-10.0
1000
3000
500
-1.0
9
10"
12
10
10
10º
125
10
107
Total Instromenn Steps
Total Enstronment Steps
TostEnmmmenn Steps
(c) Ant
(d) Hopper
(c) Reacher
Figure 2: Performance comparison of SEARL, modified PBT, and random search regarding the required total steps the environment. Random search and modified PBT require an order of magnitude more steps in the environment to reach the same reward as SEARL. The blue line depicts SEARI,, the green line of modified PBT, and the red line of random search. All approaches are evaluated over 10 random seeds. The x-axis shows the required environment interactions in log- log-scale. Dashed lines are performances of the different seeds and the shaded area of the standard deviation.
4.2 SEARL FOR TD3
Since evolve actor and critic networks over time, we start with one-layer neural network with 128 hidden nodes, which represents the lower bound the search space defined in Appendix D In an experiment investigating random initialization, we found no benefit of a more diverse starting population; please find details in Appendix E For the remaining hyperparameters, we chose a commonly used configuration consisting of ReLU activation. We use the Adam default value (1e-9) (Kingma & Ba 2015) as an initial learning rate. Similar to the PBT baseline, we used 20 workers.
We evaluate using one complete episode for rollouts or at least 250 steps in the environment. This results in at least 5 000 new environment interactions stored in total in the shared replay memory each generation. To select a new population, we use a tournament size of ie 3. After mutation, we train the individuals with samples from the shared replay memory for j = 0.5 times the number of environment interactions of the whole population during evaluation in this generation. We run SEARL for a total of 2 million environment interactions. This results in 1 million training updates for each individual the population, similar to random search and PBT. We provide a detailed table of the full configuration in Appendix
4.3 FAIR EVALUATION PROTOCOL FOR HPO IN RL
Standard evaluation of RL agents commonly shows the achieved cumulative reward in an episode after interacting with the environment for N steps. It is common practice to only report the best configuration's 's rewards found by an offline or online meta-optimization algorithm. However, this hides the much larger number of environment interactions needed to find this well-performing configuration. In contrast, we argue that the interaction counter should start ticking immediately as tackle new task or environment. Such a fair reporting of mntaioopti could also increase the comparability of approaches (Feurer & Hutter 2019).
As there is convention on how display the performance of meta-learning while taking into account the total amount of environment interactions as described above, we opt for the following simple presentation: When reporting the meta-optimization effort, every environment interaction

---

Published as a conference paper at ICLR 2021
12000
5000
4000
3000
TD3 sample contggs
6030
SEARL (ours)
2000
2038
Total Envirnment Stepe Millon
Total Enrrenment Stuza Million
HalfCheetah
(b) Walker2D
3500
3000
30:0
2100
60:00
2005
1020
1505
1305
300
0.0
0.5
0.3
0.0
0.5
1.0
Tokal Enviommen Steps Milion
Total Environment Steps in Milloon
Endromnntn Steas in Millon
(c) Ant
(d) Hopper
(c) Reacher
Figure 3: Comparison of the mean performance over 10 random seeds of SEARL (blue line) against 10 randomly sampled configurations of TD33, evaluated over 10 random seeds (red) over 2M environment frames. The shaded area represents the standard deviation.
used for the meta-optimization process should be counted when reporting the performance at any given time. In the case of offline meta-optimization efforts as in random search trials, this translates to multiplying the number of interactions of the best configuration's performance curve by the number of trials used to arrive at this configuration.
RESULTS & DISCUSSION
Comparison baselines: Figure 2 we compare the mean performance of SEARL random search and our modified version of PBT when training the TD3 agent and tuning its algorithm and architecture hyperparameters. We report the results following the evaluation outlined
SEARL outperforms random search and modified PBT in terms of environment interactions with up to 10x faster convergence in each task, which indicates a significant gain in the sample efficiency of our approach. We want to stress that individuals within the population do not get samples more often during training compared with the modified PBT setup. The substantial increase in sample efficiency for cettaoopimizztion is mainly due to the fact that individual experience samples get re-used more often during the training phase across the whole population. This gained efficiency of the individual learning agents compensate the overhead of the meta-optimization, which allows us to get strong hyperparameter configuration in only two times the environment interactions compared the reported final performance in the TD3 paper.
single experience
Furthermore, in Figure 2 we see that SEARL outperforms random search in two and modified PBT in four of five tasks performance. In the other tasks, SEARL is on par with the baselines, and we often see a smaller standard deviation in performance, which could indicate that SEARL is robust random seeds. However, SEARI. and PBT struggles Reacher r in comparison to random search. The lower performance on Reache could attributed the very short episode length of this environment. TD3 cope with this setup due to frequent training updates. However, in SEARL and other evolutionary approaches like ERL, where fixed number of samples is collected, this could harm the performance since the training updates become less frequent.
of the final performance, while clearly dominating in terms of any
Performance in the Search Space: In Figure B] we evaluate the single-run performance SEARL in the search space of TD3 random search. We plot the mean performance for a single SEARL run and 10 sampled TD3 configurations over 10 random seeds each. The TD3 configurations run for 2M environment frames similar SEARL and in contrast to 1 1M frames in
10 random seeds

---

Published as a conference paper at ICLR 2021
the upper experiment. We observe that SEARL matches the performance of the favorable part of the search space even within the same number of environment steps as single TD3 run, again only struggling on Reacher. This indicates that SEARL evolves effective configuration schedules during training, even when starting with population consisting of a single hyperparameter setting using a small one-layer neural network architecture.
Dynamic Configuration: We show exemplary hyperpa- rameter schedules discovered by SEARL on the HalfChee- tah task in Figure 4 In the first 500 000 environment interactions, SEARL quite quickly increases the network size of the TD3 agents to values, which are more in line
Hidd nodes
1000
500
with commonly reported network architectures used for strong results on MuloCo's HalfCheetah task. Afterward, the architecture size is increased more slowly, before sat- urating after 1M environment interactions. This shows SEARL can adapt the network architecture starting from a relatively small network size and prevent further increases in size once the network capacity is sufficient. The learn- ing rates shown in the lower part of Figure f both the actor and the critic networks, and can be seen to slowly decrease over time. Different schedules can be observed on other environments, see Appendix G The need for in- dividual adaptation in each environment is also reflected by the final neural network architectures of the best- rest-performing agent, as shown in Appendix
0.0
0.5
1.0
1.5
2,0
Learning Rate
0.002
0.001
0.000
0.0
0.5
1.0
1.5
2.0
Total Environment Steps Million
Figure 4: SEARL changes of network size (top) and actor/critic learning rate (bottom, red/blue) across the population in the HalfCheetah environment.
While the final actor networks in Hal HalfCheetah have on average 2.8 layers and more than 000 nodes, for other environments such as Walker2d or Ant smaller network sizes perform better. This suggests that SEARL could be capable of automatically adapting the network size to the task complexity. Another advantage of the dynamic configuration in SEARL. lies in the possibility to achieve on-par performance reported in Fujimoto et al. (2018)[TD3) with significantly smaller network architectures. Experiments with smaller static network architectures in TD3 do not achieve comparable performance. This suggests that growing networks like in SEARL is helpful for achieving strong performance with small network sizes.
Ablation Study: To determine how the individual parts of our proposed method influence perfor- mance, we conducted ablation study, which is shown in Figure 5 For the ablation experiment without architecture evolution we set the hidden layer size to [400, 300] The ablation study clearly shows that disabling shared experience replay harms the performance most severely. Also, the other aspects of SEARL are beneficial in the mean. The various environments benefit from the individual SEARL features different extents. While some tasks benefit strongly from architecture adaptation, others benefit more from hyperparameter changes. For example, removing the architecture adap- tation in HalfCheetah leads drop in performance while is beneficial in Walker2D, while the hyper-parameter adaptations exert the opposite behavior.
To test the generality of SEARL, we conducted a held-out experiment. SEARL performs better or on-par on four out of five environments compared with a held-out tuned TD3 configuration, demonstrating the generalization capabilities of SEARL across different environments. Details can be found in Appendix H
Meta-optimization of DQN: To demonstrate the generalization capabilities of our proposed frame- work, we also use SEARL to automatically tune the widely used Rainbow DQN algorithm (Hessel ct al. 2018) on five different environments of the Arcade Learning Environment (Bellemare et al. 2013 and assume zero prior knowledge about hyperparameters. Following the approach outlined in Section 3 we initialized the neural networks starting from a small size and allowed changes in the CNN & MLP activation functions as well as the learning rate during the mutation phase. Please find a detailed description in Appendix [ We compare a SEARL-optimized and a RS-optimized Rainbow DQN and observe similar results as in the case study on TD3, where SEARL shows a significantly as well as a strong final performance, surpassing the best random search configuration's 's performance in all but one environment (see Figure [00 Appendix[], This suggests that SEARL is capable of optimizing off-polic, RL algorithms across various environments with different input modalities, as well as different RI. algorithm setups.
increased sample-efficieng,
8

---

Published as a conference paper at ICLR 2021
12000
5300
SEARL
Baterpe Roturn
0000
no shared experiance replay
no learning rate adaption
architecture adaption
40:0
no network weight mutation
1000
5.0
0.6
1.0
0.0
0.2
0.4
0.6
3.0
100
Tosal Enatronment Staps len
Total Environment Steps Million
(b) HalfCheetah
(c) Walker2D
3500
6100
-2.5
-5.0
2000
-7.5
3000
1300
-2.5
500
0.0
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
2.6
0.8
1.
Total Envirommnn Steas Milion
Tocal Envtronment Stops Milloo
Total Environment Steos Bion
(d) Ant
(e) Hopper
(f) Reacher
Figure 5: Comparison of the performance impact when removing different SEARL features individu- ally. All results show mean performance across five different random seeds. The standard deviation is shown as shaded areas.
SEARL for On-Policy Deep RL: The mettaooptimzztionn of on-policy algorithmss using SEARL would most likely not show the same benefits in sample efficiency due to the high impact of the shared experience replay. Nonetheless, we expect that the dynamic configuration of the algorithm and network architecture hyperparameters would still positively impact on-policy algorithms. Another option would be to incorporate the choice of the deep RL algorithm being optimized as an additional hyperparameter for each individual, thus allowing multiple RL algorithms be optimized.
Computational Effort: SEARL requires less compute to optimize an agent than a random search or PBT. This has two reasons: (A) As shown in Figure 2 SEARL requires up 10 times fewer environment interactions. (B) SEARL evolves neural network starting with a small network size, which reduces computational costs the beginning of the meta-optimization. Even though all compared methods can run in parallel, random search and PBT can run asynchronously. while SEARL has an evolutionary loop that requires synchronicity, at least in our current implementation.
CONCLUSION
Advances in AutoRL dispose of the required manual tuning of an RL algorithm's 's Besides the general benefits of automated optimization, like fair comparability of different algorithms (Feurer & Hutter 2019), more automation the HPO of RL algorithms leads to improved sample efficiency. This could make algorithms more viable in real-worid applications.
hyreeitaaaahppppppaaaaaa hyperparameters.
Following these desiderata, we proposed SEARL, a sample-efficient AutoRL framework. SEARL performs an efficient and robust training of any off-policy RL algorithm while simultancously tuning the algorithm and architecture hyperparameters. This allows SEARL to discover effective configuration schedules, addressing the non-stationarity of the RL problem. We demonstrated the effectiveness of our framework by meta-optimizing TD3 in the MuJoCo benchmark suite, requiring up an order of magnitude fewer interactions' in the environment than random search and PBT while matching or improving their final performance in 4 out of 5 environments.
The improvement in sample efficiency be attributed mainly the of shared replay experiences when training the population of agents and allows us to perform AutoRL almost for free. SEARL required only twice the number of environment interactions single TD3 run (without HPO) while simultaneously training agents and performing a dynamic hyperparameter optimization.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENT
The authors acknowledge funding by the Robert Bosch GmbH, as well support by the state of Baden-Va Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG. This work has also been supported by the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme under grant no. 716721.
REFERENCES
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
C. Bodnar, B. Day, and P. Lió. Proximal distilled evolutionary reinforcement learning. In Proceedings of the Conference Artificial Intelligence (AAAI20), pp. 3283-3290. AAAI Press, 2020.
H. L. Chiang, A. Faust, M. Fiser, and A. Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Aulomation Letters, 4(2):2007-2014, 2019.
J. Dy and A. Krause (eds.). Proceedings of the 35th International Conference Machine Learning (ICML' 18), volume 80, 2018. Proceedings of Machine Learning Research.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley,
I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. Dy & Krause (2018). pp. 1407-1416.
A. Faust, A. Francis, and D. Mchta. Evolving rewards to automate reinforcement learning. 19X5v:1905.07628 [cs.LG), 2019.
M. Feurer and F. Hutter. Hyperparameter optimization. In F Hutter, L Kotthoff, and Vanschoren (eds.), Automated Machine Learning: Methods, Systems, Challenges, pp. 3-33. Springer, 2019. Available for free at http://automl.orgbbodl
D. Floreano, P. Dür, and C. Mattiussi. Neuroevolution: from architectures learning. Evolutionary intelligence, 1(1):77-2, 2008.
V. François-Lavet, R. Fonteneau, and D. Ernst. How discount deep reinforcement learning: Towards new dynamic strategies. In NIPS 2015 Workshop Deep Reinforcement Learning, 2015.
S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In Dy & Krause (0018), pp. 1582-1591.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. Mcllraith & Weinberger (2018).
M. Hessel, J. Modayil, H. Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. G. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Mcllraith & Weinberger (2018). pp. 3215-3222.
D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver. Dis- tributed prioritized experience replay. Proceedings of the International Conference on Learning Representations (ICLR' 18), 2018. Published online: iclr.cc.
R. Islam, P. Henderson, M. Gomrokchi, and D. Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv:1708.0411 04133 [cs.LG), 2017.
M. Jaderberg, V. Dalibard, S. Osindero. Dumning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population based training of neural networks. 1711:/711.090 [cs.LG), 2017.
W. Czamecki. J. Donahue, A. Razavi, O. Vinyals, T. Green,
10

---

Published as a conference paper at ICLR 2021
S. Khadka and K. Tumer. Evolution Evolution-guided policy gradient in reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS' 18), 2018.
S. Khadka, S. Majumdar, T. Nassar, Z. Dwiel, E. Tumer, S. Miret, Y. Liu, and K. Tumer. Collaborative evolutionary reinforcement learning. In K. Chaudhuri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning (ICML' 19), volume 97. Proceedings of Machine Learning Research, 2019.
D. P. Kingma and J. Ba. Adam: method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
S. Lee, C. Sung-lk, and S. Chung. Sample-efficient deep reinforcement learning via episodic backward update. In Wallach et al. (2019), pp. 2110-2119.
L. Lin. Self-improving reactive agents based reinforcement learning, planning and teaching. Machine Learning, 8:293-321, May 1992.
A. R. Mahmood, D. Korenkevych, G. Vasan, W. M., and Bergstra. Benchmarking reinforcement learning algorithms on real-world robots. In Proceedings of the 2nd Annual Conference Robot Learning (CoRL), volume 87, pp. 561-591, PMLR, 2018.
S. Mcllraith and K. Weinberger (eds.). Proceedings of the Conference on Artificial Intelligence (AAAI'/8), 2018. AAAI Press.
R. Mikkkulainnn, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, H. Shahrzad, A. Navruzyan, N. Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293-312. Elsevier, 2019.
B. Miller and Goldberg. Genetic algorithms, tournament selection, and the effects of noise. Complex Systems,
1995.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529-533 2015.
Jack Parker-Holder Vu Nguyen, and Stephen J Roberts. Provably efficient online hyperparameter optimization with population-basee bandits. Advances in Neural Information Processing Systems, 33, 2020.
S. Paul, V. Kurin, and S. Whiteson. Fast efficient hyperparameter tuning for policy gradient methods. In Wallach al. 2019). pp. 4618-4628.
B. J Ross. A lamarckian evolution strategy for genetic algorithms. Practical handbook of genetic algorithms: Complex coding systems, 3:1-16, 1999.
F. Runge, D. Stoll, S. Falkner, and F. Hutter. Learning to design RNA. In Proceedings of the International Conference on Learning Representations (ICLR'I9) 19), 2019. Published online: iclr
T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In Proceedings of the International Conference on Learning Representations (ICLR' 16), 2016. Published online:
iclr.ec
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off- actor-critic with shared experience replay. arXiv preprint arXiv: 1909.11909.11. 2019.
Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger (eds.). Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 12), pp. 2960 2968, 2012.
11

---

Published as a conference paper at ICLR 2021
K. O. Stanley and R. Mikkullanen. Evolving neural networks through augmenting topologies. Evol. Comput., 10(2):99-11)) 127, June 2002. ISSN 1063-6560.
K. O. Stanley, J. Clune, J. Lehman, and R. Mikkullanen. Designing neural networks through neuroevolution. Nature Machine Intelligence, 1(1):24-35, 2019.
P. Such, V. Madhavan, E. Conti, Lehman, K. Stanley, and J. Clune. Deep neuroevolution: Genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning. 0rXiv:1712.065633
[cs.NE], 2017.
E. Todorov, Erez, and Y. Tassa. MuJoCo: A physics engine for model- control. In Interna- fional Conference on Intelligent Robots and Systems (IROS' /2), pp. 5026-0026333 IEEE, 2012.
M. Tokic. Adaptive epsilon-greedy exploration in reinforcement learning based on value difference. In KI 2010: Advances in Artificial Intelligence, 33rd Annual German Conference on AI, Karlsruhe, Germany, September 21-24, 2010. Proceedings, pp. 203-210, 2010.
M. Tokic and Palm. Value-difference based exploration: Adaptive control between epsilon-greedy and softmax. In KI 2011: Advances in Artificial Intelligence, 34th Annual German Conference on AI, Berlin, Germany, October 4-7,201 Proceedings, pp. 335-346, 2011.
H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché Buc, E. Fox, and R. Garnett (eds.). Proceed- ings of the 32nd International Conference Advances in Neural Information Processing Systems (NeurlPS' 19), 2019.
T. Wei, C. Wang, Y. Rui, and C. W. Chen. Network morphism. In M. Balcan and K. Weinberger (eds.), Proceedings of the 33rd International Conference on Machine Learning (ICML' volume 48 pp. 564-572. Proceedings of Machine Learning Research, 2016.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. self-tuning actor-critic algorithm. Advances in Neural Information Processing Systems, 33, 2020.
B. Zhang and H. Mühlenbein. Evolving optimal neural networks using genetic algorithms with occam's 's razor. Complex systems, 7(3):199-220, 1993.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2016
12

---

Published as a conference paper at ICLR 2021
APPENDIX
A MUTATION OPERATORS
For each individual, we select one of the following operators uniformly at random:
This operator adds Gaussian noise to a subset of the RL agent's neural network weights. The additive noise is only added the actor network, in similar fashion, described in Khadka & Tumer (2018) Algorithm 3, using the same hyperparameters.
Gaussian Mutation:
Activation Function: For each network independently. new activation function for the hidden layers is chosen randomly from a pre-defined set with equal probability. The set contains the activation functions (rclu, clu, tanh) excluding the current used activation function.
Network Size: This operator changes the network size, either by adding new layer with probability of 20% or new nodes with a probability of 80%. When adding a new layer, the last hidden layer will be copied, and its weights are re-initialized. This leaves the features learned in the carlier layers unchanged by the mutation operator. When adding new nodes, we first select one of the existing hidden layers at random and add either 16. 32 64 additional nodes to the layer. The existing weight matrix stays the same, and the additional part of the weight matrix is initialized.
Hyperparameter Perturbations: This operator adapts the hyperparameters of the training algo- rithm online. We make use of hyperparameter changes as proposed in PBT Jaderberg et al. (2017). For the TD3 case study, only change the learning rates of the actor and critic network. To this end, we randomly decide to increase or decrease the learning rate by multiplying the current rate by factor of 0.8 1.2.
No-Operator: No change is applied to the individual. This option allows more training updates with existing settings.
13

---

Published as a conference paper at ICLR 2021
B SEARL ALGORITHM
Algorithm 1: SEARL algorithm
Input: population size N, number of generations G, training fraction j Initialize replay memory R
Initialize start population popu (fA1, 01)u, {A2, 02)o. {AN, IN to)
for y to G do Set fitness list F =
Set selected population pop selectedd ()
Set mutated population pop
Set transition count 7 =
Evaluatien uuutiiininn of previous population foreach {A; 0: E popg-1 do
Jg. transitions <- EVALUATR(( 1-)
10
store transitions in R T len (taansiiiiss
11
add fÿ to Fg end
12
13
/+ Tournament selection with elitism
14
best 1
selected
15
add best pop
16
for 1 N winner, + TOURIAMEENTSELLET((001111111 selected add winner; to pop
18
19
end
Mutation of tournament selection foreach {A, E pop of fffrringg MUTATE({A, 0,)) mutated
essselected do
29
22
add opriing to pop end
23
/+ RL-Training steps
foreach {A,,0) 0,} E pop mapttateted do sample transitions from R
24
26
trained, + RL-T TRAINING( A,, 0,). transitions, training_steps= * () add trained, popg
28
end
29 end
14

---

Published as a conference paper at ICLR 2021
C IMPACT OF TARGET NETWORK RE- TE-CREATION AND OPTIMIZER RE-INITIALIZATION
Due the changing network architecture in the mutation phase of SEARL, each training phase requires a change of the value target network and a re-initialization of the optimizers used during training. To deal with this, SEARL copies the individual's current value network after the mutation phase and uses this copy as the frozen value target network for the training phase. SEARL also re- initializes the optimizers to deal with the changing network architectures. We analyze the performance impact of this target network re-creation and re- re-initializatio by simulating the effects in the context of the TD3 algorithm the MuJoCo suite. this end, we re-create the target network and re- initialize the optimizer after every episode in the environment. We note that in the original TD3 setting, the target network is updated using soft updates. The target re-creation case we analyze corresponds hard update of the target network. The results of the described experiments shown in Figure where each curve represents the mean performance of 20 TD3 runs with different random seeds, and the shaded surprisingly, re-creation and re-initialization do not seem to hurt performance and even improve performance in out of environments.
represents the standard deviation. We observe that, perhaps
12000
5000
Default
8000
Optimizer reintdalizaaionn
Target re-creation
4000
Re-creation and rein itial izatiio
1000
8.0
0.4
1.0
0.0
0.7
0.4
2.6
2.8
1.0
Tosal Erafrommant Stope Ban
Total Environment Stops in Million
(b) HalfCheetah
(c) Walker2D
4000
-2.5
-5.0
2000
-7.5
1000
1500
2000
-2.5
0.0
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
3.6
0.8
1.
Total Enviromment Stepsin
Toxal Enstronment Steps Million
Total Environment Steas Bion
(d) Ant
(c) Hopper
(f) Reacher
Figure 6: This plot shows the impact of target re-creation and optimizer re-initialization after each episode in the TD3 algorithm. The default configuration uses soft target update and only initializes the optimizer once at the beginning of the training. The plots show the mean performance across 20 random seeds on the MuloCo suite, and the shaded area represents the standard deviation.
15

---

Published as a conference paper at ICLR 2021
D HYPERPARAMETER SEARCH SPACES
The hyperparameter search space is inspired by the PBT-UNREAL experiments and can be found in Table We choose the initial hyperparameter space for PBT smaller than random search since PBT can increase or decrease parameters during the optimization.
Parameter
PBT
Random Search
learning rate
{5 x 10-3 10-6) (relu, lanh, elu}
activation
layers
[1,2]
{1. 2. (64, 500)
units
(128, 384)
Table 1: Considered hyperparameter search spaces. The random search space is slightly larger since the configurations fixed, whereas PBT can explore network sizes.
16

---

Published as a conference paper at ICLR 2021
E RANDOM HYPERPARAMETER INITIALIZATION IN SEARL
One may hypothesize that a more diverse starting population coming from random initialization of each individual might help to find well-performing configurations faster. In this experiment initialize the SEARL population with the same search space as in PBT (Appendix D). Figure 7 shows the mean performance of 10 runs with different random seeds. The shaded area represents the standard deviation. We observe that using more diverse starting population does not provide substantial benefits any environment. But in cases where less task-specific information is available, we show that using random initialization does not seem to harm performance significantly.
5000
10036
4000
SEARL
6020
SEARL (default initialization)
6030
1000
5.0
0.5
0.0
1.0
20
Tosal Endronment Steps Willoo
Tool Envionment Steas Milloon
(a) HalfCheetah
(b) Walker2D
3102
5000
3009
-2.5
doenge Retum
3020
2500
-5.0
2000
-7.5
1302
2000
1000
-125
1000
-15.0
Total Envionment Staso i Milioo
Total Envrrnment Steps Willonn
Total Driiion munt Steps Millon
(c) Ant
(d) Hopper
(c) Reacher
Figure 7: The mean performance over 10 random seeds of SEARL using the default configuration for hyperparameter initialization (blue line) compared with random initialization (orange) over 2M environment steps. The shaded area represents the standard deviation.
17

---

Published as a conference paper at ICLR 2021
F SEARL CONFIGURATION
Table shows the configuration SEARL for the TD3 case study.
Parameter
Value
Max. frames in environment
000 000
Replay memory size
000 000
Min. frames per evaluation
250
Population size
20
Selection tournament size
3
New layer probability
20%
New nodes probability
80%
Parameter noise standard deviation
0.1
Training batch size
100
Fraction of eval frames for training
0.5
Default learning rate
0.001
Optimizer
Adam
TD3 Gamma
0.99
TD3 Tau
0.005
TD3 policy noise
0.2
TD3 noise clip
0.5
TD3 update frequency
2
Default activation function
relu [128]
Start network size
Table 2: The configuration of SEARL in the TD3 case study.
18

---

Published as a conference paper at ICLR 2021
G CHANGE OF HYPERPARAMETERS OVER TIME
We visualize a set of hyperparameters for a SEARL optimization run to generate more insights about the online hyperparameter adaptation. We show the sum of hidden nodes and the learning rates of both the actor and the critic for all individuals in the population in Figure While for runs in environments like HalfCheetah and Ant, the learning rate seems to decrease over time for most individuals. In other environments, the learning rate seems to stay more constant. This suggests that no single learning rate schedule seems optimal for all environments, thus showing the need for online hy hyprpaarmeeter adaptation. We also observe that different environments require different network sizes to achieve strong performance. This in line with previous findings, e.g., Henderson et al. (2018).
Laieing mokes
Rete Hoten
1020
500
Sum of hidden nodes
Actor leamingg rate
0.002
Critic learning rate
0.034
0.002
0.000
0.000
Envfronment
Millos
(b) HalfChectah
(c) Walker2D
Hodes 00905
1550
og Bate Hidtes 00de5
1500
1000
1020
500
1.0
Loaning Rudz
0.002
2,002
0.001
0.022
0.000
0.000
2.000
1.5
0.0
1.0
Stapa
(d) Ant
(e) Hopper
(f) Reacher
Figure 8: Change of the total network size (as the sum of hidden nodes in all layers, orange) and the learning rates (actor learning rate in red, critic learning rate in blue) over time. A plot contains all population individuals configurations over 2M steps for one SEARL run.
19

---

Published as a conference paper at ICLR 2021
H SEARL COMPARED TO TD3 HELD-OUT OPTIMIZATION
We test the generality SEARL against held-out tuned TD3 configuration. Therefore, we tune TD3 hyperparameters in the artificial setting of four similar environments with 2M interactions in total to find the hyperparameter setting that is best on average across these four environments and evaluate on the held-out environment for 2M steps. We compare this seave-one-enviromeen to single fixed hyperparameter configuration for SEARL across all environments running for 2M environment interactions to demonstrate that SEARL capable of robust joint HPO and training of learning agent. As shown in Figure SEARI. outperforms the TD3-tuned hyperparameter setting in three environments, performs on par "Walker2d" and slightly worse in "Reacher". These results demonstrate the generalization capabilities of SEARL across different environments.
performance
5000
12000
4200
3000
TD3 theld-out tuned)
0000
SEARL
2000
4030
2000
5.0
0.5
1.5
2.4
0.5
1.5
Stape
Toral Environment Staps
(b) HalfCheetah
(c) Walker2D
3500
-2.5
3000
-7.5
3000
1500
10.00
2000
1000
12.5
-15.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
2.0
0.0
1.9
1.5
Tota Insirommnt
(d) Ant
(c) Hopper
(1) Reacher
Figure 9: The comparison of SEARL performance against held-out tuned TD3 configuration, tuned with the same budget of SEARL run (2M environment interactions), and evaluated over 2M steps. All results show mean performance across 10 different random seeds. The standard deviation is shown shaded areas.
20

---

Published as a conference paper at ICLR 2021
I FINAL ARCHITECTURE FOUND BY SEARL IN TD3 MUJOCO EXPERIMENTS
T the average across ten different seeded runs of the architecture of the best performing individual. While the actor networks in Hal fCheetah have 2.8 layers and over 000 nodes on average the networks are much smaller in other environments like Walker2 or Ant. This could indicate that different environments require different network capacities. SEARL could be capable of automatically adapting to the network size to the task complexity.
MuloCo
Avg. Layer
Avg. total nodes
Avg. nodes per layer
HalfCheetah
2.8 (0.1)
1019.2 (80.3)
360.8 (22.3)
Walker2d
1.7 (0.2)
689.6 (113.3)
422.7 (34.4)
Ant
1.7 (0.2)
723.6 (119.5)
432.59 (16.6)
Hopper
1.3 (0.1)
462.0 (75.5)
357.0 (27.0)
Reacher
2.1 (0.3)
|
840.0 (145.9)
394.1 (38.7)
Table Comparing the average final architecture across 10 random seeds different MuJoCo environments. The standard error in brackets.
21

---

Published as a conference paper at ICLR 2021
J OPTIMIZING RAINBOW DQN WITH SEARL ON ATARI
To demonstrate the generalization capabilities of the proposed SEARL framework to different RL settings, we perform AutoRL tuning of the widely used Rainbow DQN algorithm (Hessel et al. 2018)
in four Atari environments (Bellemare al. 2013). Rainbow DQN is combination of different extensions to the DQN algorithm (Mnih et al. 2015). We use all the extensions proposed in Rainbow DQN except for the prioritized replay memory (Schaul et al. (016)) due to the shared experience collection in SEARL. We compare SEARL to random search to evaluate SEARL's performance in the search space of this setup. We report the results in Figure T0
For the random search experiments, we select the best performance out of 10 sampled configurations. For each configuration, we randomly choose the CNN layer count as well as the channel size, stride and kernel size, the MLP layer count and size, the activation function, and learning rate, please find details in Table Compared to the TD3 case study section 4 the SEARL Rainbow DQN setup differs only in the agents' initial network size. Here we use small MLP [128] and small CNN (two-layer, channel size [32, 32). kernel size: [8, 4), stride: [4, 2). To evolve the agents, we allow changes to the MLP by adding layer or nodes to existing layer (add {8, 16, 32)), activation function ({ (relu,, tanh, elu)). and the learning rate during the mutation phase, similarly to the TD3 case study, In addition, we mutate the CNN by adding new convolution layers or changing the channel size (add {8, 16, 32)), kernel size ((3, 4, 5,5 or stride ( {1, 2, 4} and in sum 8).
1500
20
1250
1.0
1000
Avenge
750
Random Search SEARL (aurs)
-10
500
-20
10P
10"
16
Enviromenn Steps
(b) Pong
(c) Enduro
40
120
1.00
80
Average
60
80
60
20
-20
00
1.a
10a
10'
30
10'
30'
10'
16
10'
10F
10.
Toral Endronnmnnt Stops
Total Envtrorment Steps
Total Environment Steps
(d) Boxing
(e) Freeway
(f) RoadRunner
Figure 10: Performance comparison of SEARL and random search regarding the required total steps in the environment. Random search requires an order of magnitude more steps in the environment to reach the same reward level as SEARL. The blue line depicts SEARL and the red line random search. SEARL is evaluated over 5 random seeds and random search over runs with 10 configurations each. The x-axis shows the required environment interactions in icalsc Dashed lines are performances of the different seeds and the shaded areas show the standard deviation.
We run both SEARL and each random search configuration for 2M frames in the environment. We report results in the same fashion as for the TD3 case study. For each environment, we perform 5 SEARL runs with different random seeds and 5 runs of random search (which choose the best configuration out of 10 random configurations each). We report the best of these runs scaled by the total required step cost. As previously observed in the TD3 case study, we observe that SEARL again outperforms the random search baseline significantly in terms of sample-efficiency. In Pong and RoadRunner, the SEARL optimized performance is par with the best random search configuration, and in Freeway, the SEARL optimized final performance is slightly worse. Still, in both Endure and Boxing, SEARL outperforms the best random search configurations.
22

---

Published as a conference paper at ICLR 2021
These results demonstrate the general usefulness of SEARL across different RL algorithms and environments, even covering different types of network architectures and input modalities.
Parameter
Random Search
learning rate
{5 10-3 10-5) {relu, tanh, elu}
activation
MLP layers
[1, 2}
MLP nodes
64, 384
CNN layers
{1, 2,3
CNN channels
[64, 384
CNN stride
(1,2,4) and Σ layer stride;
CNN kernel
[3, 4,5
Table 4: Considered hyperparameter search spaces for DQN experiments.
23


Reviews:
[{"review_id": "sGq2JUEqn7Y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe authors in this paper propose to optimize the hyperparameters and also the neural architecture while simultaneously training the agent. They evaluate the proposed method with TD3 in the MuJoCo benchmark suite. \n\nOverall, the proposed method is well-motivated and well-written, and they provide enough experiment/implementation details to reproduce the results. More importantly, they provide the source code. \n\n# Strength\n\n- Tuning architecture and hyperparameters with PTB for off-policy RL has not been studied before. Compared to baselines, the proposed method is much more sample-efficient.\n- The evaluation on MuJoCo is comprehensive, especially the ablation study in Section 4.\n\n# Weakness\n\n- They only test on a single benchmark with one method, TD3. MuJoCo is arguably simple. Actually, the visual world in MuJoCo is quite limited, so the encoder of the RL agent does not have to be huge. It could be more convincing if the authors can test on another benchmark, e.g. ProcGen. \n- I think compared to computer vision tasks with huge neural networks, the search space for the architecture of RL models is much smaller, which can be observed in the ablation study. Without architecture adaption, there is no observable difference. It seems that combining the tuning of architecture and hyperparameters is not that useful.", "extra_scores": {}, "extra_text": {}}, {"review_id": "kdhMor3nwqu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by the sensitivity of RL algorithms to the choice of hyperparameters and the data efficiency issue in training RL agents, the authors propose a population-based automated RL framework which can be applied to any off-policy RL algorithms. In the framework, they optimise hyperparameters together with neural architectures. The authors use TD3 on MuJoCo environments as a showcase to demonstrate the advantages of the proposed method. They reduced the number of environment interactions significantly compared to baselines like random search and a modified population-based training algorithm.\n\nPros:\n+ The authors propose an important novel component to PBT-like framework, which is sharing experience among agents in the population. This innovation itself leads to 10x improvement on sample efficiency;\n+ Very clear description of the motivations, related work, the details of the training framework, and the experiments. The paper reading has been super pleasant;\n+ Results of TD3 on MuJoCo environments, i.e., the ones in Figure 2, are very promising. In the five environments tested, SEARL algorithm shows an order of magnitude improvement on the sample efficiency.\n\nCons:\n- The proposed sharing experience among agents in the population is limited to off-policy RL algorithm, as also noted by the authors;\n- There are a few lines of AutoRL and AutoML research are missing in references or in discussions. For example, the architecture search lines of work from Zoph et al. , and for gradient-based meta parameters optimisation like \"A Self-Tuning Actor-Critic Algorithm\";\n- For the main contribution in the paper, shared experience replay within the population, there's a recent work by Schmitt et al. called \"Off-Policy Actor-Critic with Shared Experience Replay\" which also demonstrates that sharing experience replay in off-policy learning can improve sample efficiency dramatically. Though they are not in the setting of meta learning hyperparameters or architectures, I think it worths discussions;\n- From the ablations in Figure 5, it looks like the architecture adaption does not contribute much to the final performance. It might be because MuJoCo environments do not require complex neural architectures. To show that the neural architecture adaptation is a crucial part of the framework (which is a major difference between the proposed method and PBT), the authors might have to move to a complex domain of environments to demonstrate that.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eZwoC2vi9JF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper propose a population-based AutoRL framework for hyperparameter optimization of off-policy RL algorithms. The framework optimizes both the hyperparameters and also the neural architecture in a one-shot manner, e.g., search and train at the same time. A shared experience replay buffer is used across the population, which as demonstrated in the experiments, substantially increase the sample efficiency compared to PBT and random search. \n\nStrengths:\n- The idea is simple and intuitively makes sense. By sharing the experiences across the population each experience sample gets re-used more often during the training hence the increase in sample efficiency.\n\nWeakness:\n- Only apply to off-policy RL. \n- Random search is not a very compelling baseline. PBT is not optimized for sample efficiency. It seems that the shared replay buffer can also be applied to PBT? What about differentiable HPO methods? Those are often shown to be much more sample efficient compared to evolution based approaches. \n- There is little discussions on what the search results look like, e.g., the learning rate schedule or the neural architecture found by the search method. How are they compared to the SOTA learning rate schedule or neural architecture? Or maybe not even SOTA just compare to some simple heuristics that gradually decrease the learning rate or increase the network size. It would be of limited value if the use of AutoRL can only achieve marginally better performance compared to those simple heuristics. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "V5xH1ySgVf", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary：\nIn this paper, the authors intend to propose an efficient automated reinforcement learning (RL) framework. To achieve this goal, they integrate three technologies, i.e., evolutionary RL for hyperparameter search, evolvable neural network for policy network design, and shared experience replay for improving data usage. The paper uses a case study on MuJoCo to demonstrate the claimed advantages over baselines.\n\nSome pros:\n1. The motivation is good. The automation of reinforcement learning is beneficial to the research community, especially for researchers who are not familiar with RL but in need of it.\n2. The framework is general and friendly to users. This framework is general and can be regarded as a plug-in module for a series of reinforcement learning methods. Besides, as we all know, both of the autoML and RL have a heavy computational burden, the adaption of evolvable neural network and shared experience replay greatly alleviate this dilemma. \n3. The organization of this paper is easy to follow. We can follow the authors from why they want to deal with the problem, to how they are inspired by existing work, and then to how the algorithms are design based on the questions to be answered and the existing technologies. \n\nSome cons:\n1. The experiment is far from enough. Actually, this paper only has a case study. First, the author claims that the framework can optimize arbitrary off-policy RL algorithms, why only try on TD3? From the perspective of robustness, the authors need to compare more off-policy algorithms with and without the proposed method. Second, the paper claims there is no directly comparable approach for efficient AutoRL, which I do not agree with. In its own related work, many AutoRL baselines are listed, e.g., H. L. Chiang et al.  'Learning navigation behaviors end-to-end with autorl', F. Runge et al. 'Learning to design RNA'. For these baselines, they can either take the same exploration steps and compare performance with the proposed method, or compare the performance/computational cost when reaching the same performance. Anyway, the readers expect to see more comparisons with more baselines from more perspectives. Third, the authors claim that to tackle the non-stationarity of the RL problem, existing studies can substantially increase the number of environment interactions, implying the proposed framework has advantages on non-stationarity RL environments, but still, no experimental results are given.\n2. Some technical details are missing. The logic is clear, the solution is reasonable, but the details are ignored. It is a good idea to keep the writing compact, but the lack of details may harm the readability of the paper. For example, since this is a general framework, how should we design hyperparameter settings of SEARL in initialization for different algorithms? In the training part, why should individual be trained for as many steps as frames have been generated in the evaluation phase, and why the training time could be reduced by using only a fraction j of the steps? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles a very important topic in deep RL, namely automatic (non-differentiable) hyper-parameter tuning. It does so by combining ideas from genetic algorithms and neural architecture search with shared experience replay in order to obtain the key property of sample efficiency.  The proposed solution is communicated clearly, and the results are compelling (often 10x improvements), as well as qualitatively interesting.

Unfortunately for the authors, their original submission contained only part of the intended results, hence the borderline scores by some reviewers. In the meanwhile, a second suite of experiments have been added, which I think are compelling enough evidence to validate the paper's approach.
Decision: Accept (Poster)