Evolving Reinforcement Learning Algorithms
Authors: John D Co-Reyes
Yingjie Miao
Daiyi Peng
Esteban Real
Quoc V Le
Sergey Levine
Honglak Lee
Aleksandra Faust
Venue: ICLR 2021 Oral (2021)
Keywords: reinforcement learning
evolutionary algorithms
meta-learning
genetic programming

Abstract: We propose a method for meta-learning reinforcement learning algorithms by searching over the space of computational graphs which compute the loss function for a value-based model-free RL agent to optimize. The learned algorithms are domain-agnostic and can generalize to new environments not seen during training. Our method can both learn from scratch and bootstrap off known existing algorithms, like DQN, enabling interpretable modifications which improve performance. Learning from scratch on simple classical control and gridworld tasks, our method rediscovers the temporal-difference (TD) algorithm. Bootstrapped from DQN, we highlight two learned algorithms which obtain good generalization performance over other classical control tasks, gridworld type tasks, and Atari games. The analysis of the learned algorithm behavior shows resemblance to recently proposed RL algorithms that address overestimation in value-based methods.

Published as a conference paper at ICLR 2021
EVOLVING REINFORCEMENT LEARNING ALGORITHMS
John D. Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Sergey Levine, Quoe V. Le. Honglak Lee, Aleksandra Faust" Research at Google, Mountain View, CA 94043, USA
ABSTRACT
We propose a method for meta-learning reinforcement learning algorithms by searching over the space of computational graphs which compute the loss function for value-based model-free RL agent optimize. The learned algorithms are Hamain-agnostie and can generalize to new environments not seen during train- ing. Our method can both learn from scratch and bootstrap off known existing algorithms, like DQN, enabling interpretable modifications which improve perfor- mance. Learning from scratch on simple classical control and gridworld tasks, our method rediscovers the semporal-different DQN, we highlight two learned al algorithms which obtain good generalization per- formance over other classical control tasks, gridworld type tasks, and Atari games. The analysis of the learned algorithm behavior shows resemblance to recently pro- posed RL algorithms that address overestimation in value-based methods.
(TD) algorithm. Bootstrapped from
INTRODUCTION
Designing new deep reinforcement learning algorithms that can efficiently solve across a wide va- riety of problems generally requires a tremendous amount of manual effort. Learning design reinforcement learning algorithms or even small sub-components of algorithms would help ease this burden and could result in better algorithms than researchers could design manually. Our work might then shift from designing these algorithms manually into designing the language and optimization methods for developing these algorithms automatically.
Reinforcement learning algorithms can be viewed as procedure that maps an agent's 's experience to a policy that obtains high cumulative reward over the course of training. We formulate the problem of training an agent one of meta-learning: an outer loop searches over the space of computational graphs or programs that compute the objective function for the agent to minimize and an inner loop performs the updates using the learned loss function. The objective of the outer loop is to maximize the training return of the inner loop algorithm.
Our learned loss function should generalize across many different environments, instead of being specific to a particular domain. Thus, we design a search language based on genetic programming (Koza] 1993) that can express general symbolic loss functions which can be applied to any envi- ronment. Data typing and a generic interface to variables in the MDP allow the Iearned program to be domain agnostic. This language also supports the use of neural network modules subcompo- nents of the program, so that more complex neural network architectures can be realized. Efficiently searching over the space of useful programs is generally difficult. For the outer loop optimization, we use regularized evolution (Real et al. 2019) recent variant of classic evolutionary algorithms that employ tournament selection (Goldberg & Deb 199T). This approach can scale with the num- ber of compute nodes and has been shown to work for designing algorithms for supervised learning (Real et al.) 2020). We adapt this method automatically design algorithms for reinforcement learning.
While learning from scratch is generally less biased, encoding existing human knowledge into the learning process can speed up the optimization and also make the learned algorithm more inter- pretable. Because our search language expresses algorithms as a generalized computation graph, we
@ google.com

---

Published as a conference paper at ICLR 2021
Mutator
Train Ervs.
Randomized Algorithirs
Environments (E)
Mutated Algoritim
known Algarithins
Algorithm Evaluation
Population of
[oe)) f)
Algorithms (L)
Figure I: Method overview. We use regularized evolution to evolve a population of RL algorithms. mutator alters top performing algorithms to produce new algorithm. The performance of the algorithm is evaluated over set of training environments and the population is updated. Our method can incorporating existing knowledge by starting the population from known RI. algorithms instead of purely from scratch.
embed known RL algorithms in the graphs of the starting population of programs. We compare starting from scratch with bootstrapping off existing algorithms and find that while starting from scratch can learn existing algorithms, starting from existing knowledge leads new RL algorithms which can outperform the initial programs.
We learn two new RL algorithms which outperform existing algorithms in both sample efficiency and final performance on the training and test environments. The learned algorithms are domain agnostic and generalize to new environments. Importantly, the training environments consist of suite of discrete action classical control tasks and gridworld style environments while the test environments include Atari games and are unlike anything seen during training.
The contribution of this paper is method for searching over the space of RL algorithms, which we instantiate by developing formal language that describes broad class of value- model-free reinforcement learning methods. Our search language enables us to embed existing algorithms into the starting graphs which leads to faster learning and interpretable algorithms. We highlight two learned algorithms which generalize completely new environments. Our analysis of the meta- learned programs shows that our method automatically discovers algorithms that share structure to recently proposed RI. innovations, and empirically attain better performance than deep Q-learning methods.
2 RELATED WORK
Learning to learn is an established idea in in supervised learning, including meta-learning with genetic programming (Schmidhuber, 1987 Holland 1975] Koza 1993), learning neural network update rule (Bengio al.) 1991). and self modifying RNNs (Schmidhuber 1993). Genetic program- ming has been used to find new loss functions (Bengio et al. 1994. Trujillo & Olague 2006). More recently, AutoML (Hutter al. 2018) aims to automate the machine learning training process. Au- tomated neural network architecture search (Stanley Mükkulainen (2002) Real et al. 2017 2019) Liu et al. 2017 Zoph & Le 2016 Elsken et al.; 2018 Pham al. (208)) has made large improve- ments in image c classification. Instead of learning the architecture, AutoML-Zert (Real et al.) 2020) learns the algorithm from scratch using basic mathematical operations. Our work shares similar ideas, but is applied to the RL setting and assumes additional primitives such as neural network learn computational graphs with the goal of automating
modules. In contrast AutoML-Zero, RL algorithm design. Our learned RL algorithms generalize to new problems, not seen in training.
Automating RL. While RL is used for AutoML (Zoph & Le 2016: Zoph et 2018 Cai et al., 2018 Bello et al.] 2017). automating RL itself has been somewhat limited. RL requires different design choices compared to supervised learning, including the formulation of reward and policy update rules. All of which affect learning and performance, and are usually chosen through trial and error. AutoRL addresses the gap by applying the AutoML framework from supervised learning to the MDP setting in RL. For example, evolutionary algorithms are used to mutate the value actor network weights (Whiteson & Stone 2006 Khadka & Tumerr 2018)), learn task reward (Faust et al, 2019). tune hyperparameters (Tang& & Choromanskil 2000 2000 Franke al. 2020), or search for neural network architecture (Song et al.) 2020; Franke et 2000). This paper focuses on aggostii RL update rules in the value-based RI. setting which are both interpretable and generalizable.
2

---

Published as a conference paper at ICLR 2021
Meta-learning in RL. Recent work has focused few-shot task adaptation. Finn al. 2011 7
Finn & Levine (2018) meta-learns initial parameters which can quickly adapt to new tasks, while RL (Duan et al. 2016 and concurrent work (Wang et al. 2017), formulates RL itself
learning problem that is learned with an RNN. The meta-learned component of these works is tuned to a par- ticular domain environment, in the form of NN weights which cannot be used for completely new (Reed & De Fre- Pierrot et al. 2019) overcome the environment generalization challenge by learning hi- erarchical neural programs with domain-specific encoders for different environments. Here, the computational graph has a flexible architecture and generalizes across different environments.
domains with potentially different sized inputs. Neural "ngraamee-Innerpreete
itas 2015
Learning RL algorithms or their components, such as reward bonus or value update function. has been studied previously with meta-gradients (Kirsch et al.) 2020, Chebotar et al.. 2019 Oh et al.) 2020), evolutionary strategies (Houthooft et al.) (018), and RNNS (Duan et al. 2016). Although our work also learns RL algorithms, the update rule is represented as computation graph which includes both neural network modules and symbolic operators. One key benefit is that the resulting graph can be interpreted analytically and can optionally be initialized from known existing algo- rithms. Prior work that focuses learning RL losses, generalizes to different goals and initial conditions within a single environment (Houthooft et al. (2018), or learns a domain invariant policy update rule that can generalize to new environments (Kirsch at. 2020). Another approach searches over the space of curiosity programs using a similar language of DAGs with neural network modules (Alet et al., (020aa) and performs the veta-ttainnng on single environment. In contrast, our method is applied to learn general RL update rules and meta-trained over diverse set of environments.
LEARNING REINFORCEMENT LEARNING ALGORITTMSS
In this section, first describe the problem setup. An inner loop method Eval(L, E) evaluates learned RI. algorithm L on a given environment E. Given access to this procedure, the goal for the outer loop optimization is learn RL algorithm with high training return over set of training environments. We then describe the search language which enables the learning of general loss functions and the outer loop method which can efficiently search over this space.
3.1 PROBLEM SETUP
We assume that the agent parameterized with policy T0(ae(s)) outputs actions a, at each time step an environment & and receives re- ward r and next state 81+1. Since we are focusing on discrete action value-based RL methods, 0 will be the parameters for Q- value function and the policy is obtained from the Q-value function using an e-greedy strat- egy. The agent saves this stream of transitions (st, St-1, 0z, re) to replay buffer and contin- ually updates the policy by minimizing a loss function L(sr, de, Tt, St41, 0, 7) over these tran- sitions with gradient descent. Training will oc- for a fixed number of M training episodes where in each episode m, the agent cams episode return Rm Σ 2t=0 The perfor- mance of an algorithm for given environment is summarized by the normalized average train- ing return, M and Rmax are the minimum and maximum re- turn for that environment. We assume these are known ahead of time. This inner loop evaluation procedure Lv E) is outlined in Algorithm |
A:S - [.se/9]
S
#'s Msst(R
List(R]
NN: List [8]
Selectlisi
F
MaxUsl
Cutput
Figure 2: Visualization of a RL algorithm, DQN, as a computational graph which com- putes the loss L (Q(st, a.) (rr + y max, (sarg(Sr41,01) 22) Input nodes are in blue, parameter nodes in gray, operation nodes in ange, and output in green.
1 Em=1 Rmas- Ri where Rmin
R. RR-RRmin_
To score an algorithm, use the normalized av- erage training return instead of the final behavior policy return because the former metric will factor in sample efficiency well.
3

---

Published as a conference paper at ICLR 2021
The goal of the meta-leamnr is to find the optimal loss function ((s;, "t, Ty with maximal normalized average training return over the set of training environments. The full objective for the meta-learner is:
Ft, (441,0,7) to optimize
L' = arg max
Σ Eval(L,
L is represented computational graph which we describe in the next section.
3.2 SEARCH LANGUAGE
Our search language for the algorithm L should be expressive enough to represent existing algo- rithms while enabling the learning of new algorithms which can obtain good generalization perfor- mance across a wide range of environments. Similar to Alet et al. (2020a), we describe the RL algorithm as general programs with a domain specific language, but we target updates the policy rather than reward bonuses for exploration. Algorithms will map transitions (st, Qz, $t)1,T1), policy parameters 0, and discount factor y into scalar loss to be optimized with gradient descent. We express L as a computational graph or directed acyclic graph (DAG) of nodes with typed inputs and outputs. See Figure for a visualization of DQN expressed in this form. Nodes are of several types:
Input nodes represent inputs to the program, and include elements from transitions (St, ag $141, and constants, such as the discount factor 7,
Parameter nodes are neural network weights, which can map between various data types. For example, the weights for the Q-value network will map an input node with state data type to list real numbers for each action.
Operation nodes compute outputs given inputs from parent nodes. This includes applying param- eter nodes, as well basic math operators from linear algebra, probability, and statistics. A full list of operation nodes provided in Appendix A By default, we set the last node the graph to compute the output of the program which is the scalar loss function to be optimized. Importantly, the imputs and outputs of nodes are typed among (state, action, vector, float, list, probability). This typing allows for programs to be applied to any domain. It also restricts the space of programs to ones with valid typing which reduces the search space.
Algorithm Algoritha Evaluation, Ev)
Algorithm Evolving RT. Algorithms
1: Input: RL. Algorishm L. Environment E. training episodes M
Training envirnments (E). burdle enviromment &s. hurdle threshold optional existing algonism
Input:
Initialive: Q-value parameters target parameters o' empty replay
butter
2: Initialize: Population of RL. algocithms [L), history manom-
3: for to de
ired inpurs 7. If hoststrpppng, initialize P with A.
for U to
3: Score L with [[]]]] Σg Exa((L,E) do 5: Sample toumament T ~ -nnffar((())
probahility c, select random action
4: for =
otherwise a, arg max, Q(a,.)) 4)
7:
Step em incemmnnt sust.rs alo...)
6:
Purent algorithm I. highest score algorithm in
D DJ [s.. (t-111
Child algorithm Hash((/((T))
MuistelL.)
HTE
Update targot
parameters
9:
If [7[ hash was new and Exa((E) > then H|1/]score +- Σe Eval(1/
11:
end for
10:
12: Compute spisode return Ri,, Er.
end If
and
Add to population
Output:
13:
Remove oldest 6. from population
15: Normalized training performace DM Touce Paats
14:
end for
Oufput: Algorithm L with highest
3.3 EVOLUTIONARY SEARCH METHOD
Evaluating thousands of programs over a range of complex environments is prohibitively expensive, especially done serially. We adapt a genetic programming (Koza) 1993) method for the search method and use regularized evolution (Real et al.] 2019). a variant of classic evolutionary algo- rithms that employ tournament selection (Goldberg & Deb, 1991). Regularized evolution has been shown to work for learning supervised learning algorithms (Real et al. 2020) and can be parallelized across compute nodes. Tournament selection keeps a population of algorithms and improves the population through cycles. Each cycle picks tournament of T < P algorithms random and selects the best algorithm in the tournament as a parent. The parent is mutated into a child algorithm which gets added to the population while the oldest algorithm in the population is removed. We use
4

---

Published as a conference paper at ICLR 2021
single type of mutation which first chooses which node in the graph to mutate and then replaces it with a random operation node with inputs drawn uniformly from all possible inputs.
There exists combinatorially large number of graph configurations. Furthermore, evaluating a sin- gle graph, which means training the full inner loop RL algorithm, can take up a large amount of time compared to the supervised learning setting. Speeding up the search and avoiding needless compu- tation are needed to make the problem more tractable. We extend regularized evolution with several techniques, detailed below, to make the optimization more efficient. The full training procedure is outlined in Algorithm 2
Functional equivalence check (Real al., 2020 Alet et al. 2020b). Before evaluating a program,
we check if it is functionally equivalent to any previously evaluated program. This check is done by hashing the concatenated output of the program for 10 values of randomized inputs. If mutated program is functionally equivalent to an older program, we still add it to the population, but use the saved score of the older program. Since some nodes of the graph do not always contribute to the output, parts of the mutated program may eventually contribute to functionally different program.
Early hurdles (So et al. 2019). We want poor performing programs to terminate early so that we can avoid unneeded computation. We use the CartPole environment as an early hurdle environment ε, by training a program for a fixed number of episodes. If an algorithm performs poorly, then episodes will terminate in short number of steps (as the pole falls rapidly) which quickly exhausts the number of training episodes. We use Eval( Ex) < a as the threshold for poor performance with a chosen empirically.
Program checks. We perform basic checks to rule out and skip training invalid programs. The loss function needs to be a scalar value so we check if the program output type is a float (type(L) R). Additionally, we check if each program is differentiable with respect to the policy parameters by checking if a path exists in the graph between the output and the policy parameter node.
Learning from Scratch and Bootstrapping. Our method enables both learning from scratch and learning from existing knowledge by bootstrapping the initial algorithm population with existing algorithms. We learn algorithms from scratch by initializing the population of algorithms randomly. An algorithm sampled by sampling each operation node sequentially in the DAG, For each node, an operation and valid inputs to that operation are sampled uniformly over all possible options.
While learning from scratch might uncover completely new algorithms that differ substantially from the existing methods, this method can take longer to converge to a reasonable algorithm. We would like to incorporate the knowledge we do have of good algorithms to bootstrap our search from a better starting point. We initialize our graph with the loss function of DQN (Mnih et al.) (2033) so that the first nodes represent the standard DQN loss, while the remaining nodes are initialized randomly. During regularized evolution, the nodes are not frozen, such that it is possible for the existing sub- graph to be completely replaced if a better solution is found.
4 LEARNED RL ALGORITHM RESULTS AND ANALYSIS
We discuss the training setup and results of our experiments. We highlight two learned algorithms with good generalization performance, DONClipped and DQNReg, and analyze their structure.
4.1 TRAINING SETUP
Meta- PeeannTa details: We search over programs with maximum 20 nodes, not including inputs or parameter nodes. A full list of node types is provided in Appendix A We use population size of 300, tournament size of 25, and choose these parameters based on the ones used in (Real et al. 2019). Mutations occur with probability 0.95. Otherwise new random program is sampled. The search is done over 300 CPUs and run for roughly 72 hours, at which point around 20, 000 programs have been evaluated. The search is distributed such that any free CPU is allocated to a proposed individual such that there are no idle CPUs. Further meta-training details are in Appendix B
Training environments: The choice of training environments greatly affects the learned algorithms and their generalization performance. At the same time, our training environments should be not too computationally expensive to
will be evaluating thousands of RL. algorithms. We use
5

---

Published as a conference paper at ICLR 2021
Trainin Parfomanco
Training Rerformance Histogram
NL1100 Nrpuaro
669
Scrath Ene
Fram Scraish 2 tni
Fram Scrasc
19009
riterr of Aiw.ipiteess Folkaree
(a) Learning curve
(b) Performance histogram
Figure 3: Left: Meta-training performance over different number of environments from scratch, and boot- strapping. Plotted as RI. evaluation performance (sum normalized training return across the training en- vironments) over the number of candidate algorithms. Shaded region represents one standard deviation over 10 random seeds. More training environments Icads to better algorithms. Bootstrapping from DQN speeds up convergence and higher final performance. Right: Meta-training performance histogram for bootstrapped training. Many of top programs have similar structure (Appendix
range of 4 classical control tasks (CartPole, Acrobat, MountainCar, LunarLander) and a set of 12 multitask gridworld style environments from MiniGrid (Chevalier-Boiseer et al., 2018). These environments are computationally cheap to run but also chosen cover diverse set of situations. This includes dense and sparse reward, long time horizon, and tasks requiring solving a sequence of subgoals such picking up a key and unlocking door. More details are in Appendix C
The training environments always include CartPole as an initial hurdle. If an algorithm succeeds on CartPole (normalized training performance greater than 0.6), it then proceeds to harder set of train- ing environments. For our experiments, we choose these training environments by sampling a set of 3 environments and leave the rest test environments. For learning from scratch also compare the effect of number of training environments on the learned algorithm by comparing training on just CartPole versus training CartPole and Lunarl
RL Training details: For training the R.. agent, we use the same hyperparameters across all training and test environments except as noted. All neural networks are MLPs of size (256, 256) with ReLU activations. We use the Adam optimizer with a learning rate of 0.0001. is decayed linearly from 0.05 over 1c3 steps for the classical control tasks and over 1e5 steps for the MiniGrid tasks.
4.2 LEARNING CONVERGENCE
Figure 3a shows convergence over several training configurations. We find that at the end of training roughly 70% of proposed algorithms are functionally equivalent to a previously evaluated program, while early hurdles cut roughly another 40% of proposed non-duplicate programs.
Varying number of training environments: We compare learning from scratch with a single training environment (CartPole) versus with two training environments (CartPole and LunarLander). While both experiments reach the maximum performance on these environments (Figure 3a). the learned algorithms are different. The two-environment training setup learns the known TD loss
LDQN = (Q(st, a.) (rr + max (aarg(11,00))
while the single-enviromment training setup learns a slight variation I. (Q(s), 0z) (rr + max, Qtarg (St, a²) that does not use the discount, indicating that the range of difficulty the training environments important for learning algorithms which can generalize.
Learning from scratch versus bootstrapping: In Figure 3a we compare training from scratch versus training from bootstrapping on four training environments (CartPole, KeyCoriddoS3RII DynamicObstacle-6xx DoorKey-5x5). The training performance does not saturate, leaving room for improvement. Bootstrapping from DQN significantly improves both the convergence and per- formance of the meta-training, resulting in 40% increase in final training performance.

---

Published as a conference paper at ICLR 2021
4.3 LEARNED RL ALGORITHM:: DONCLIPPED AND DQNREG
In this section, we discuss two particularly interesting loss functions that were learned by our method, and that have good generalization performance on the test environments. Let
v = r2 + y max (targ($t, a), and 0 = Qs1,, a+) - Yz-
The first loss function DQNClippedd is
LiONClipped max [Q az), 6 + Yi] + max [Q(se, 0e) - Ye, (max Qtarg (st, a))²
DoNNClipped was trained from bootstrapping off DQN using three training environments (LunarLan- der, SetrrriiDynammmOsssiiie
It outperforms DQN and double- DQN, DDQN, (van Hasselt et al. 2013) on both the training and unseen environments (Fig- ure The intuition behind this loss function that, if the Q-values become too large (when (s a,) > 8² + Y,), the loss will act to minimize (ss,, az)) instead of the normal 6² loss. Altern- tively, we can view this condition as å = Q(8t: ar) - Yt > 88. This means when is small enough then (sx., ux) are relatively close and the loss is just to minimize ((8,, 0z).
MiniGrid-LavvaappS5)
134
C45 OZL 053
124 11 1N 3M
Masi Seortepdooo
-14
-30
J007
Figure 4: Performance of learned algorithms (DQNClipped and DQNReg) versus baselines (DQN and DDQN) on training and test environments as measured by episode return over 10 training seeds. A dashed line indi- cates that the algorithm was meta-trained on that environment while solid line indicates test environment. DQNReg can match or outperform the baselines on almost all the training and test environments. Shaded regions correspond to standard deviation.
The second learned loss function, which we call DQNReg, is given by
LDQNReE 0.1 * Q(s a.) +
DQNReg was trained from bootstrapping off DQN using three training environments (KeyCorri- dorS3RI, Synamii-Obstacles-bxxe
DoorKey-5x5). In comparison to DQNClipped, DQNReg di- rectly regularizes the Q values with a weighted term that is always active. We note that both of these loss functions modify the original DQN loss function to regularize the Q-values to be lower in value. While DQNReg is quite simple, it matches or outperforms the baselines on all training and test en- vironments including from classical control and Minigrid. It does particularly well on few test
7

---

Published as a conference paper at ICLR 2021
environments (SimpleCrossingSNNI, DoorKey-6x -enrr,, and Unlock) and solves the tasks when other methods fail to attain any reward. It is also much more stable with lower variance between seeds, and more sample efficient on test environments (LavaGapS5, Empty-6x6, Hampty-Random-555)
In Table Il we evaluate DQNReg on a set of Atari games. We use the same architecture in DQN (Mnih et al.] 2013) and use the same no-op evalua- tion procedure which evaluates a trained policy ev- ery million training steps over 200 test episodes. Even though meta-training was on computationally simple, non-image based environments, we find that DQNReg can generalize to image-based envi- ronments and outperform baselines. The results for the baselines are taken from their respective papers
Env
DQN
DIXON
DQNReg
Asteroid
1364,5
734.7
2097.5
2390,4
Bowling
50.4
68.1
40.1
80.5
Boxing
88.0
91.6
94.6
100.0
RoadRunner
39544.0
44127.0
35466.0
65516.0
Table 1:
Performance of learned algorithm DQNReg against baselines on several Atari games, Baseline numbers taken from reported papers.
MMnih al. 2013 van Hasselt et al. 2015 Schulman et al. 2017).
These algorithms are related to recently proposed RL algorithms, conservative Q-learning (CQL) (Kumar et al.] (020) and M-DQN (Vieillard et al.) (020). CQL learns a conservative Q- function by augmenting the standard Bellman error objective with a simple Q-value regularizer: log >. exp (Q(s),))) Q(s,, aq) which encourages the agent to stay close to the data distribution while maintaining maximum entropy policy. DQNReg similarly augments the standard objective with a Q-value regularizer although does in different direction by preventing overestimation. M-DQN modifies DQN by adding the scaled log-policy (using the softmax Q-values) to the imme- diate reward. Both of these methods can be seen as ways to regularize value-based policy. This resemblance indicates that our method can find useful structures automatically that are currently being explored manually, and could be used to propose new areas for researchers to explore.
We discover that the best performing algorithms from the experiment which learned DQNReg are consistent, and in the form L 8² + k Q(5,, az). This loss could use further analysis and investigation, possibly environment-specific tuning of the parameter k. See Appendix for details.
4.4 ANALYSIS OF LEARNED ALGORITHMS
Mie Dec S0545
MinO CateyCCrraeCcCcCCCCooooo 3311-40
Figure 5: Overestimated value estimates is generally problematic in value-based RL. Our method Icarns algo- rithms which regularize the Q-values helping with overestimation. We compare the estimated Q-values for our learned algorithms and baselines with the optimal ground truth Q-values across several environments during training. Estimate is for taking action zero from the initial state the environment. While DQN overestimates the Q-values, our learned algorithms DQNClipped and DQNReg underestimate the Q-values.
We analyze the learned algorithms understand their beneficial effect on performance. In Figure 5 we compare the estimated Q-values for each algorithm. We see that DQN frequently overestimates the Q values while DDQN consistently underestimates the Q values before converging to the ground truth Q value which are computed with manually designed optimal policy. DONClipped has similar performance to DDQN, in that it also consistently underestimates the Q values and does so slightly more aggressively than DDQN. DQNReg significantly undershoots the Q values and does not converge to the ground truth. Various works (van Hasselt et al., 2015 Haarnoja et al. 2018 Fujimoto et al., 2018) have shown that overestimated value estimates is problematic and restricting the overestimation improves performance.
The loss function in DQNClipped is composed of the sum of two max operations, and so we can analyze when each update rule is active. We interpret DQNClipped as max(v1, 02) + max(u2, 03) with four cases: 1) 01 > U/ and U3 U4 2) 11 U2 and Uy < va 3) t1 < U2 and Us < va 4) V1 < 82 and By > Ug. Case 2 corresponds to minimizing the Q values. Case 3 would correspond to the normal DQN loss of 82 since the parameters of Qtorg are not updated during gradient descent.
8

---

Published as a conference paper at ICLR 2021
In Figure 6 plot the proportion of when each case is ac- tive during training. We see that usually case 3 is gener- ally the most active with small dip in the beginning but then stays around 95%. Meanwhile, case 2, which regular- izes the Q-values, has small increase in the beginning and then decreases later, matching with our analysis in Figure 6 which shows that DQNClipped strongly underestimates the Q-values in the beginning of training. This can be seen as constrained optimization where the amount of Q-value regu- larization is tuned accordingly. The regularization is stronger the beginning of training when overestimation problem- atic (Q(,,, a,) > 82 x²+Y)) and gets weaker as 52 gets smaller.
Figure 6: DQNClipped, can he broken down into four update rules where each rule active under certain conditions. Case 3 corresponds to normal TD learning while case 2 corresponds minimizing the Q-values. Case 2 more active in the beginning when value overestima- tion is a problem and then becomes less active as it is no longer needed.
Our learned algorithm.
5 CONCLUSION
In this work, we have presented a method for learning reinforcement learning algorithms. We design general language for representing algorithms which compute the loss function for value-based model-free RI. agents to optimize. We highlight two learned algorithms which although relatively simple, can obtain good generalization performance over a wide range of environments. Our analysis of the learned algorithms sheds insight on their benefit as regularization terms which are similar to recently proposed algorithms. Our work is limited to discrete action and value-based RL algorithms that are close to DQN, but could easily be expanded to express more general RL algorithms such as actor-critic or policy gradient methods. How actions are sampled from the policy could also be part of the search space. The set of environments we use for both training and testing could also be expanded to include a more diverse set of problem types. We leave these problems for future work.
ACKNOWLEDGEMENTS
We thank Luke Metz for helpful early discussions and feedback the paper, Hanjun Dai for early discussions on related research ideas, and Xingyou Song, Krzysztof Choromanski, and Kevin Lee for help with infrastructure. We also thank Jongwook Choi for help with environment selection.
REFERENCES
Ferran Alet, M. F. Schneider. Tomas Lozano-Peree and L. Kaelbling, Meta-learning curiosity algo- rithms. ArXiv, abs/2003.05325 2020a.
Ferran Alet, Martin F. Schneider, Tomas Lozano-Perez and Leslie Pack Kaelbling. Meta-learning curiosity algorithms. In International Conference on Learning Representations, 2020b. URL
https:://ppenrevil/
net /f
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoe Le. Neural optimizer search with reinforce- ment learning. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 459-468. JMLR. org, 2017.
S. Bengio, Yoshua Bengio, and J. Cloutier. Use of genetic programming for the search of a new learning rule for neural networks. Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence, pp. 324-327 vol.1, 1994.
Yoshua Bengio, S. Bengio, and J. Cloutier. Learning synaptic learning rule. IJCNN-91-Seatle International Joint Conference on Neural Networks, ii:969 vol.2-, 1991.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Proceedings the Thirty-Second AAAI Conference on Artificial Intel- ligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18) New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2787-2794, 2018
Yevgen Chebotar, Artem Molchanov, Sarah Bechtle, Ludovic Righetti, F. Meier, and Gauray S. Sukhatme. Meta-learning via learned loss. ArXiv, abs/ Abs/1990.05374, 2019.

---

Published as a conference paper at ICLR 2021
Maxime Chevalier Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. hatpps: : http//gtttb..tt/ com/ wwtp//ww.wiccacacccaacacccccccaa http///gmmmmiiiiii.... 2018.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and P. Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. ArXiv, hts//611..22779 2016.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, arXiv 2018.
preprint
Alcksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards automate reinforcement learning. arXiv preprint arXiv:1905.07628 2019.
Chelsea Finn and S. Levine. Meta-leaming and universality: Deep representations and gradient descent can approximate any learning algorithm. ArXiv, abs/1710.11622 1622, 2018.
Chelsea Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ArXiv, abs/1703.033400 2017.
Jörg K. H. Franke, Gregor Köhler, André Biedenkapp. and Frank Hutter. Sample-efficient automated deep reinforcement learning. 2020.
Scott Fujimoto, H. V. Hoof, and David Meger. Addressing function approximation error in actor- critic methods. ArXiv, abs/1802.09977, 2018.
David E Goldberg and Kalyanmoy Deb. A comparative analysis of selection schemes used in genetic algorithms. FOGA, 1991.
Toumas Haamojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICMI,, 2018.
John H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor, MI, 1975. second edition, 1992.
Rein Houthooft, Richard Y. Chen, Phillip Isola, Bradly C. Stadie, F. Wolski, Jonathan Ho, and P. Abbeel. Evolved policy gradients. ArXiv, abs/ 1801888822.11 2018.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning: Meth- ods, Systems, Challenges. Springer, 2018. In press, available at http://automl.orgbbool
Shauharda Khadka and Kagan Tumer. Evolution-guided policy gradient in reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa Resa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 1188-1221 1200. Curran Associates, Inc., 2018.
Louis Kirsch, Sjoerd van Steenkiste, and J. Schmidhuber. Improving generalization in meta rein- forcement learning using learned objectives. ArXiv, abs/1910.0400 2020.
John Koza. Genetic programming the programming of computers by means of natural selection. In Complex adaptive systems, 1993.
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. ArXiv, his/2006600479,, 2020.
Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy, Progressive neural architecture search. CoRR, abs/1712.00559, 2017. URL
00559
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
Junhyuk Oh, Matteo Hessel, Wojeiech Marian Czamecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. ArXiv, abs/2007.08794, 2020.
10

---

Published as a conference paper at ICLR 2021
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Thomas Pierrot, Guillaume Ligner, Scott Reed, Olivier Sigaud, Nicolas Perrin, Alexandre Laterre, David Kas, Karim Beguir, and Nando de Freitas. Learning compositional neural programs with recursive tree search and planning. NeurlPS, 2019.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 2902-2911. JMLR.org, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In AAAI, volume abs/1 0ess/8802.1548, 2019.
Esteban Real, Chen Liang, David So, and Quoe V. Le. Automl-zero: Evolving machine learning algorithms from scratch. In ICML, volume Abs/22003033344 2020.
Scott Reed and Nando De Freitas.
Neural programmer-nneeppeeerr. arXiv preprint
arXiv:1511.0627,, 2015.
Juergen Schmidhuber. Evolutionary principles in self-referential learning. 1987.
Juergen Schmidhuber. A self-referential weight matrix. 1993.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063477 2017.
D. So. Chen Liang, and Quoc Le. The evolved transformer. ArXiv, abs/1901.11117, 2019.
Xingyou Song, Krzysztof Choromanski, Jack Parker-Holder, Yunhao Tang, Wenbo Gao, Aldo Pac- chiano, Tamas Sarlos, Deepali Jain, and Yuxiang Yang. Reinforcement learning with chromatic networks for compact architecture search. 2020.
Kenneth O Stanley and Risto Mükkkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2):99-127, 2002.
Yunhao Tang and Krzysztof Choromanski. Online hyper-paramen tuning off-policy learning via evolutionary strategies, 2020.
L. Trujillo and G. Olague. Synthesis of interest point detectors through genetic programming. In GECCO '06, 2006.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. AAAI, 2015.
Nino Vieillard, Olivier Pietquin, and M. Geist. Munchausen reinforcement learning. ArXiv, abs/2007. 14430, 2020.
Jane X. Wang, Zeb Kurth-Nelson. Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, D. Kumaran, and Matt M. Botvinick. Learning to reinforcement learn. ArXiv, abs/1611.05705 2017.
Whiteson and P. Stone. Evolutionary function approximation for reinforcement learning. Mach. Learn. Res., 7:877-917, 2006.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In ICLR, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pp. 8697-8710, 2018
11

---

Published as a conference paper at ICLR 2021
A SEARCH LANGUAGE DETAILS
Inputs and outputs to nodes the computational graph have data types which include state S, action Z. float IR, list Lis. probability F. vector V. The symbol X indicates it can be of S, IR or V. We assume that vectors are of fixed length 32 and actions are integers. Operations will broadcast so that for example adding float variable state variable will result the float being added each element of the state. This typing allows the learned program to be domain agnostic. The full list of operators listed below.
Operation
Input Types
Output Type
Add
X, X
X
Subtract
X, X
Max
X, X
X
Min
X, X
X
DotProduct
X. X
Div
X, X
X
L2Distance
X, X
R
MaxList
List [i
MinList
List I
R
ArgMaxList
List R
2
SelectList
List
Meanl.st
List
VarianceList
List X
X
Log
X
X
Exp
X
X
Abs
X
X
(C)NN:3 -> List
S
List
R
(CNNN:
S
(CNNS + V
V
Softmax
List
R
KLDiv
P
Entropy
P
R
Constant
0.5, 0.2, 0.1, 0.01
Multiply' Tenth
X
X
Normal(O, 1)
R
Uniform(O, 1)
B TRAINING DETAILS
We describe the training details and hyperparameters used. For all environments we use the Adam optimzier with a learning rate of 0.0001.
Common RL training details. All neural networks are MLPs of size (256, 256) with ReLU activa- tions. For optimizing the Q-function parameters we use the Adam optimizer with a learning rate of 0.0001. Target update period 100. These settings are used for all training and test environments.
Classical control environments. The value of c is decayed linearly from 1 to 0.05 over 1000 steps. CartPole, Acrobat, and MountainCar are trained for 400 episodes and LunarLander trained for 1000 episodes.
MiniGrid environments. The value of decayed linearly from to 0.05 over 10 steps. During meta-training, MiniGrid environments trained for 10 steps.
Atari environments. We use the same neural network architecture as in Mnih et al. (2013). Target update period is 1, 000. The value of is decayed linearly from 1 to 0.1 over 10t steps. For evaluation, we use the no-op start condition as in Mnih et al. (2013) where the agent will output the no-op action for steps where x is random integer drawn between [1, 30). The evaluation policy uses an 0.001 and evaluated every 8 steps for 100 episodes. The best training snapshot is reported.
12

---

Published as a conference paper at ICLR 2021
C ENVIRONMENT DETAILS
We describe the classical control environments below. CartPole and LunarLander are dense reward while Acrobat and MountainCar are sparse reward.
Task ID
Description
The agent must balance a pole on top of a cart by applying a force +1 or -1 the cart. A reward of +1 is provided for each timestep the pole remains upright.
CartPole-vO
The agent controls a lander by firing one of four thrusters and must land the landing pad.
LunarLander-ve
The goal is swing 2-link system upright to a given height by applying 1, 0, or -1 torque on the join between the two links.
Acrobat-vl
The goal is to drive up the mountain on the right by first driving back and forth to build up momentum.
MountainCar-v0
We describe the MiniGrid environments below. The input to the agent a fully observed grid which encoded
NxNx3 size array where N the grid size. The 1st channel contains the index of the object type at that location (out of 11 possible objects), the 2nd channel contains the color of the object (out of 6 possible colors), and the 3rd channel contains the orientation of the agent out of 4 cardinal directions. This encoding is then flattened and fed into an MLP. There are 7 possible actions (turn left, turn right, forward, pickup, drop, toggle, done).
Unless stated otherwise, all tasks are sparse reward tasks with a reward of for completing the task. Max steps set to 100. size such 5x5 the environment name refers grid size with width and height of 5 cells.
Task ID
Description
The agent has find key hidden in one room and then use it to pickup object be- hind a locked door in another room. This tests sequential subgoal completion.
KeyComidorS3RI-v0
The agent has to reach the green goal square without touching the lava which will termi- nate the episode with zero reward. This tests safety and safe exploration.
LavaGapS5-v0
The agent must open a door to get to the green goal square in the next room.
MultiRoom-N2-44-00
13

---

Published as a conference paper at ICLR 2021
The agent has reach the green goal square on the other corner of the room and navigate around walls.
SimpleCrossingS9NI1 vO
The agent has reach the green goal square in an empty room.
Empty-vO
The agent has to reach the green goal square in empty room but is initialized to ran- dom location.
EmptyRandom-v0
The agent has to reach the green goal square without colliding with any blue obstacles which move around randomly. If the agent collides with an obstacle it receives reward of -1 and the episode terminates.
Dynamic-Obstacles-vv
The agent must navigate in maze com- posed of four rooms. Both the agent and goal square are randomly placed in any of the four rooms.
FourRooms-v0
The agent must pick up key to unlock a door to enter another room and get to the green goal square.
DoorKey-v0
14

---

Published as a conference paper at ICLR 2021
D GRAPH DISTRIBUTION ANALYSIS
We look at the distribution of top performing graphs and find similarities in their structure. This is summarized in Table where we describe the equations of learned algorithms for differing ranks (if sorted by score). The best performing algorithms from the experiment which learned DQNReg are all variants of adding ((st, az) the standard TD loss in some form, 6² k Q(s,, We think this kind of loss could use further investigation and that while we did not tune the value of k, this could also be tuned per environment.
Figure 3b we show the distribution of scores for all non- duplicate programs that have been evaluated. We provide a full list of top performing algorithms from a few of our experiments https: /gu. com/ jooreyes evelv nnrr
Raw Equation
Simplified Equation
Score
Rank
8 + 0.1 Q(3r 0z) 17 Qtara 0.1 (.).
8 + 0.2 [=)
3.905
2
+ 0.1 Q(s.. 77 7 Qtarg
8 ((5,a))
3.904
62 (7 Qtarg 0.1 Q(s.a)))
+ 0.1 (()
82 + Quay +0.1 Q(s..)))
82 + 0.1 ((s.a))
3.902
5
8 (0.1 a(s.)) - Y2
(0.1 (().))) 12)
3.898
6
+ +r Qtory + Q(s) max(1, Q(n.ae))
Quaa 0.1 Q().
NA
3,846
11146
83 + (8 + 0.1 o())
NA
3.65
12146
+ a()..))
+ Q(a a.)
2.8
12446
62
13246
Table 3: Other programs learned in learning DQNReg which is rank with score 3.907. Rank is
if scores sorted in decreasing order. Score is the sum of normalized RL training performance across four environments. The simplified equations contains only the relevant parts for minimizing the equation output. Qtarg refers to max, Qtarg (5)
E REPEATABILITY OF META TRAINING
In Figure 7 plot the meta-training performance for bootstrapping from DQN with four training environments (CartPole, KeyCorridorS3RII Dynamic-OOstacles-66e DoorKey-5x3 over ten trials. Four out of the ten trials reach the max training performance. Two out of 10 of these trials learns the same algorithm DQNReg while the other top two trials find other less interpretable algorithms.
Meta- Mrainig Performance Over Time
4.0
Maximum RL Evalustion Performance
3.5
Trial
Trial
Trial
2.0
Trial
15
Trial
Trial
1.0
Trial 7
Trial
0.0
2500
5000
7500
10000 12500 15000 17500 20000
Number of Algorithms Evallated
Figure 7: Meta-training performance for boot-strapping on 4 training environments for 10 random seeds.
15


Reviews:
[{"review_id": "EeNCiI0tlSC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper introduces learning of RL algorithms via evolution. Over here, RL algorithms are represented as computational graphs  where the graph outputs the objective of the algorithm to be minimized. Also, the graph  comprises pre-defined symbols and operations. \nGiven these computational graphs or RL algorithms, they use Regularized Evolution(RE) to evolve a RL algo. For evolution, each RL algo is evaluated with a set of training envs. Also, there are various checks introduced to avoid incorrect algos or re-evaluation of the same algo. For the mutation, the algo which performs best across all training envs is selected and it’s mutations are reintroduced in the population. \nThese computational graphs can be initialized randomly or existing RL algos could be induced into them for bootstrapping.\n \n##########################################################################\n\nReasons for score: \n\nI vote for marginal acceptance. I believe the notions introduced in this work could be useful for the community and their results seem to indicate a promising direction.\n\n##########################################################################\n\nPros: \n\n- They learn two new RL algos  “DQN clipped” &  “DQNreg”  which happen to be better  sample efficient  and have better final  performance than DQN. As per authors, these algorithms also share resemblance to recent CQL and M-DQN.\n- Promise of better algorithms than researchers can design given the coverage of all reasonable operators/symbols in the computational graphs.\n- They performed ablation study of with/ without bootstrapping of RL algorithms.\n\n#########################################################\n\nCons:\n- Large number of CPUs required.\n\n#########################################################\n\nQuestions:\n\n- How to make a choice of environments for meta-training?\n- Why are the authors using training return for performance of the Algorithm?  After training for M episodes, they can simply evaluate the greedy policy for “k” episodes and that metric should be used for performance measure. Otherwise, we are using an epsilon-greedy metric for comparison which could be really low as that’s dependent on epsilon and environment. \n- It’s not clear if they are using constant epsilon or decaying epsilon?\n \nOther comments:\n\nSection 4.2, 2nd para:\n“The two-environment training setup is learns the known … “ => remove “is”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ccu-qH-9ZJE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method of discovering, through evolutionary search, programatically defined RL algorithms.  \n\nPros:\n\n\n-- Novel (to my knowledge) in the search language it uses, which renders the resulting learned algorithms very interpretable and generalizable (as compared to, say, using a meta-learned loss function parameterized by a neural network)\n\n-- Interesting analyses of particular meta-learned algorithms\n\n-- Useful discussion of key implementation details (e.g. early hurdles, functional equivalence check)\n\nCons\n\n-- It would be nice to see more analysis of the algorithms obtained \"from scratch\" rather than starting from DQN.   Are any other (variants of) standard RL algorithms discovered?  Are there any new but interpretable algorithms that perform well?\n\n-- Comparison to other approaches to meta-learned loss functions (e.g. Kirsch et al., or Oh et al.) would be helpful\n\n-- I realize it may be computationally infeasible to meta-train on many more environments, or more difficult tasks, but it would at least be good to see how the learned algorithms generalize to environments considerably unlike any seen during meta-training (e.g. not gridworld and not \"classical control\" -- perhaps Atari?)\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1XCJWKC70of", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to develop new Reinforcement Learning (RL) algorithms through a population-based method very reminiscent of Genetic Programming (GP). The authors \"evolve\" loss functions that can be used across different RL scenarios and providing good generalization.\n\nPros:\n- The paper is well written, well structured and clear at all times. There are few typos here and there but nothing that affects the readability of the paper.\n- The results are relevant: the fact that the method re-discovers recently proposed learning models is remarkable, as well as it finds new models indicates that it is a line of research worthy of further exploration.\n- The other minor contributions are also noteworthy, specifically the Functional equivalence check, which IMO is a brilliant idea, as well as the early hurdles approach.\n- The interpretation of the approach is also correct: the fact that authors make a clear distinction between learning from scratch and bootstrapping tells me that they truly understand the overall framework they based their method on.\n\nCons:\n- The main idea behind this type of meta-learning is always very interesting to revisit. Nevertheless, and truth to be told, the idea of using GP to find new loss (or \"objective/fitness\", as known in evolutionary computation (EC)) functions is quite old [1,2]. At some points the work presented here feels somewhat \"old-fashioned\", or a mere re-edition or adaptation of those early seminal works.\n- The results can be easily rebutted; if I understand correctly, the authors are presenting the results of a SINGLE run for every scenario they tested their method on (different no. of environments) , which tell us nothing about the average behavior of their proposed approach. Although this fact is understandable given such long training times (3 days with 300 cpus for a single run), many people will not accept the presented results arguing that they could have been the result of lucky runs. On the bright side, such practice is somewhat usual in the deep learning community, so many people may overlook it for now. I'd suggest the authors make a statement arguing why they feel confident that they approach may present a low variance, or why they still consider these results relevant, even if they require twenty or 30 runs to find results this good. They could argue that even if their method is currently statistically unreliable, it could probably be stabilized with many methods proposed in the EC community (such as spatially distributed populations [3], etc.)\n- The thing that bugs me the most of this work is that the proposed method is clearly a flavor of GP; however the authors never really acknowledge it by its name; instead they claim it to be something called \"Regularized Evolution\" which is supposedly introduced in a previous paper; nevertheless, it is really just GP: they are evolving tree-shaped programs, which is the hallmark of GP (there exist variants of GP that are not even EC-based [4]), so I don't see a reason for not calling it for what it is. I wish the authors clearly state why their approach cannot be considered part of the GP framework, or if it indeed is, then make such statement clearly.\n- In this vein, it is also interesting to note that they do not use a crossover operation. In the GP framework, crossover is generally regarded as a more powerful operator than mutation, and the combination of both operators can give the best results. My guess is that crossover is difficult to implement given the \"search language\" (or 'primitives', as known in GP), and then crossover would result in many invalid programs. Still, it would be nice if authors could explain a little bit on this issue.\n\nOther comments:\nIt draw to my attention that you use populations of size 300 as well as a 300 cpu system. It would seem that you intend to have one cpu for each individual evaluation; however, given the advances presented by the functional equivalence check and the early hurdles technique, I wonder what happens when a individual is no longer being evaluated.. is that cpu left idle? or is it reassigned to contribute to the evaluation of another individual (which sound complex to do, given the RL nature of the problems being treated)?\n\nReferences:\n1. Bengio, S., Bengio, Y., & Cloutier, J. (1994, June). Use of genetic programming for the search of a new learning rule for neural networks. In Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence (pp. 324-327). IEEE.\n2. Trujillo, L., & Olague, G. (2006, July). Synthesis of interest point detectors through genetic programming. In Proceedings of the 8th annual conference on Genetic and evolutionary computation (pp. 887-894).\n3. Tomassini, M. (2006). Spatially structured evolutionary algorithms: Artificial evolution in space and time. Springer Science & Business Media.\n4. Hooper, D. C., Flann, N. S., & Fuller, S. R. (1997). Recombinative hill-climbing: A stronger search method for genetic programming. Genetic Programming, 174-179.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a meta-learning algorithm for reinforcement learning. The work is very interesting for the RL community, it is clear and well-organized. The work is impressive and it contributes to the state-of-the-art. 
Decision: Accept (Oral)