Meta-learning curiosity algorithms
Authors: Ferran Alet*
Martin F. Schneider*
Tomas Lozano-Perez
Leslie Pack Kaelbling
Venue:  (2020)
Keywords: meta-learning
exploration
curiosity

Abstract: We hypothesize that curiosity is a mechanism found by evolution that encourages meaningful exploration early in an agent's life in order to expose it to experiences that enable it to obtain high rewards over the course of its lifetime. We formulate the problem of generating curious behavior as one of meta-learning: an outer loop will search over a space of curiosity mechanisms that dynamically adapt the agent's reward signal, and an inner loop will perform standard reinforcement learning using the adapted reward signal. However, current meta-RL methods based on transferring neural network weights have only generalized between very similar tasks. To broaden the generalization, we instead propose to meta-learn algorithms: pieces of code similar to those designed by humans in ML papers. Our rich language of programs combines neural networks with other building blocks such as buffers, nearest-neighbor modules and custom loss functions. We demonstrate the effectiveness of the approach empirically, finding two novel curiosity algorithms that perform on par or better than human-designed published curiosity algorithms in domains as disparate as grid navigation with image inputs, acrobot, lunar lander, ant and hopper.

Published as a conference paper at ICLR 2020
META-LEARNING CURIOSITY ALGORITHMS
Ferran Alet', Martin F. Schneider', Tomás Lozano-Pérez Leslie Pack Kaelbling Computer Science and Artificial Intelligence Laboratory Massachusetts Institute Technology Cambridge, MA 02139, USA {alet, mef.martinff tlp,
edu
ABSTRACT
We hypothesize that curiosity is mechanism found by evolution that encourages meaningful exploration early in an agent's life in order to expose it experiences that enable it to obtain high rewards over the course of its lifetúme. We formulate the problem of generating curious behavior loop will search over space of curiosity mechanisms that dynamically adapt the agent's reward signal, and an inner loop will perform standard reinforcement learning using the adapted reward signal. However, current meta-RL methods based on transferring neural network weights have only generalized between very similar tasks. To broaden the generalization,
one of meta-learning: an outer
instead propose to meta-leam algorithms: pieces of code similar to those designed by humans in ML papers. Our rich language of programs combines neural networks with other building blocks such as buffers, nearest-neighbor modules and custom loss functions. We demonstrate the effectiveness of the approach empirically, finding novel curiosity algorithms that perform on par or better than human-designed published curiosity algorithms in domains as disparate as grid navigation with image inputs. acrobot, lunar lander, ant and hopper.
INTRODUCTION
When a reinforcement-lermiing agent is learn- ing to behave, is critical that it both explores its domain and exploits its rewards effectively. One way to think of this problem is in terms of curiosity or intrisic motivation: construct- ing reward signals that augment or even replace the extrinsic reward from the domain, which in- duce the RL agent explore their domain in way that results in effective longer-term learn- ing and behavior (Pathak et al.) 2017; Burda cet al., 2018) Oudeyer] 2018). The primary diffi- culty with this approach is that researchers hand-designing these strategies: is difficult for humans to systematically consider the space of strategies or to tailor strategies for the distri- bution of environments agent might be ex- pected to face.
RL algorithm (fixed: PPO)
curiosity algorithm (meta-learmed)
S
r
a
environment
Figure 1: Our RL agent is augmented with cu- riosity module, obtained by meta-learning over a complex space of programs, which computes pseudo-reward F at every time step.
We take inspiration from the curious behavior observed in young humans and other animals and hypothesize that curiosity ploration early in an agent's 's life. This exploration exposes it to experiences that enable it to learn to obtain high rewards over the course of its lifetime. We propose to formulate the problem gen- outer loop, operating at "evolutionary" scale
mechanism found by evolution that encourages meaningful ex-
erating curious behavior one of meta-learning: will search over a space algorithms for generating curious behavior by dynamically adapting the
"qqual contribution.

---

Published as a conference paper at ICLR 2020
agent's reward signal, and an inner loop will perform standard reinforeement learning using the adapted reward signal. This process is illustrated in figure I note that the aggregate agent, outlined in gray, has the standard interface of an RL agent. The inner RL algorithm is continually adapting to its input stream of states and rewards, attempting to learn a policy that optimizes the discounted sum of proxy rewards Σx 7 The outer "evolutionary" search is attempting to find program
for the curiosity module, objective like the mean performance on the last few trials.
to optimize the agent's 's lifetime return Σ t== Tt, another global
In this meta-learning setting, our objective is to find a curiosity module that works well given distribution of environments from which we can sample at meta-learning time. Meta-RL has been widely explored recently, in some cases with a focus on reducing the amount of experience needed by initializing the RL algorithm well (Finn et al. 2017 Clavera et al.] 2019) and, in others, for efficient exploration (Duan al. 2016 Wang et al. (017). The environment distributions in these cases have still been relatively low-diversity. mostly limited to variations of the same task, such as exploring different mazes or navigating terrains of different slopes. We would like to discover curiosity mechanisms that can generalize across a much broader distribution of environments, even those with different state and action spaces: from image-based games, to joint-based robotic control tasks. To do that, we perform meta-learning in a rich, combinatorial, open-ended space of programs.
This paper makes three novel contributions.
We focus on a regime of rentar-eiinorreeeet--eeerr agent might face are dramatically disparate and in which the agent's lifetime is very long. This is substantially different setting than has been addressed in previous work on meta-RL. and it requires substantially different techniques for representation and search.
in which the possible environments the
We propose to do meta-learning in a rich, combinatorial space of programs rather than transferring neural network weights. The programs are represented in a domain-specijp lan- guage (DSL) which includes sophisticated building blocks including neural networks complete with gradient-descent mechanisms, learned objective functions, ensembles, buffers, and other regressors. This language is rich enough to represent many previously reported hand-designed exploration al- gorithms. We helieve that by performing meta-RI. in such a rich space of mechanisms, we will be able to discover highly general, fundamental curiosity- based exploration methods. This generality means that a relatively computationally expensive meta-learning process can be amortized over the lifetimes of many agents in a wide variety of environments.
We make the search over programs feasible with relatively modest amounts of computation. It is a daunting search problem to find a good solution in a combinatorial space of programs, where evaluating single potential solution requires running an RL algorithm for up to millions of time steps. We address this problem in multiple ways. By including environments of substantially dif- ferent difficulty and character, we can evaluate candidate programs first on relatively simple and short-horizon domains: if they don't perform well in those domains, they pruned early, which saves a significant amount of computation time. In addition, we predict the performance of an al- gorithm from its structure and operations, thus trying the most promising algorithms early in our search. Finally, we also monitor the learning curve of agents and stop unpromising programs before they reach all T environment steps.
We demonstrate the effectiveness of the approach empirically, finding curiosity strategies that per- form on par or better than those in published literature. Interestingly, the top algorithms, to the best of our knowledge, had not been proposed before, despite making sense in hindsight. We con- jecture the first one (shown in figure (3) is deceptively simple and that the complexity of the other one (figure in the appendix) makes it relatively implausible for humans to discover.
2 PROBLEM FORMULATION
2.1 META-LEARNING PROBLEM
Let us assume we have an agent equipped with an RL algorithm (such as DQN or PPO, with all hyperparameters specified), A, which receives states and rewards from and outputs actions to an vironment ε, generating a stream of experienced transitions e(A; E), = (s,, az T2s st4 1). The agent continually learns a policy (t)= 8, - at, which will change in time as described by algorithm A;
2

---

Published as a conference paper at ICLR 2020
so (t) = A(er.e1) and thus di A(e):t-))ss.). Although this need not be the case, we can think of A as an algorithm that tries to maximize the discounted reward Σ, r're+s: Y < 1 and that, at any time-step t, always takes the greedy action that maximizes its estimated expected discounted reward.
To add exploration to this policy, we include a curiosity module C that has access to the stream of state transitions e, experienced by the agent and that, at every time-step t, outputs proxy reward Fz- We connect this module so that the original RL agent receives these modified rewards, thus observing cA4, C; 8), (tt Oe; Fr C(era-1): (iia-i),ss+4 Sc+1) i). without having access the original rr- Now, even though the inner RL algorithm acts in a purely exploitative manner with respect to 7, it may efficiently explore in the outer environment.
Our overall goal to design a curiosity module C that induces the agent to maximize Σ rt. for some number of total time-steps 7 or some other global goal, like final episode performance. In an episodic problem, T will span many episodes. More formally, given a single environment &, RL algorithm A. and curiosity module C, as a dynamical system that induces state transitions for the environment, and learning updates for the curiosity module and the agent. Our objective is to find C that maximizes the expected original reward obtained by the composite system in the environment. Note that the expectation is over two different distributions different time scales: there is "outer" expectation over environments E, and in "inner" expectation over the rewards received by the composite system in that environment, so our final objective is:
see the triplet (environment, curiosity module, agent)
max
Eg E
Σ
rt
2.2 PROGRAMS FOR CURIOSITY
In science and computing, mathematical language has been very successful in describing varied phenomena and powerful algorithms with short descriptions. As Valiant points out: "the power (of mathematics and algorithms] comes from the implied generality, that knowledge of one equation alone will allow one to make accurate predictions about host of situations not even conceived when the equation was first written down" (Valiant 2013). Therefore, in order to obtain curiosity modules that can generalize over very broad range of tasks and that are sophisticated enough to provide exploration guidance over very long horizons, we describe them in terms of general programs in a domain- language. Algorithms in this language will map history of (st, St+1, ae, re) tuples into a proxy reward Fz.
Inspired by human-designed systems that compute and use intrinsic rewards, and to simplify the search, we decompose the curiosity module into two components: the first, J, outputs an intrin- sic reward value it based on the current experienced transition (st, 0e, St+1) (and past transitions ($1:1-1, (1:e-1) indirectly through its memory); the second, X. takes the current time-step t, the actual reward ra, and the intrinsic reward i, (and, if it chooses store them, their histories) and combines them to yield the proxy reward fr. To ease generalization across different timescales, in practice, before feeding into X we normalize by the total length of the agent's 's lifetime, T..
Both programs consist of a directed acyclic graph (DAG) of modules with polymorphically typed inputs and outputs. As shown in figure 2 there are four classes of modules:
Input modules (shown in blue), drawn from the set {st, Qr, $1+1} } for the I component and from the set {úr, for the x component. They have no inputs, and their outputs have the type corresponding to the types of states and actions in whatever domain they are applied to, or the reals numbers for rewards.
Buffer and parameter modules (shown in gray) of two kinds: FIFO queues that provide as output finite list of the most recent inputs. and neural network weights initialized at random at the start of the program and which may (pink border) or may not (black border) get updated via back- pro-ppropagttion depending on the computation graph.
Functional modules (shown in white), which compute output values given the inputs from their parent modules.
3

---

Published as a conference paper at ICLR 2020
0[1): S -
s(t+1)
0(2): S + F
Program Input
Network
NN: - + Detach
NN: S -
Inverse features (Pathak et al., 201
Operation
Program Output
L2Distance
function
AddToLoss
without backprop.
Disagreement (Pathak al., 2019)
Random Network Distillation (Burda et al., 2018b)
Input with backprop.
Figure 2: Example diagrams of published algorithms covered by our language (larger figures in the appendix). The green box represents the output of the intrinsic curiosity function, the pink box is the loss to be minimized. Pink arcs represent paths and networks along which gradients flow back from the minimizer update parameters.
Update modules (shown in pink), which functional modules (such as k-Nearest- Neighbor) that either add variables to buffers or modules which add real-valued outputs a global loss that will provide error signals for gradient descent.
single node in the DAG designated as the output node (shown in green): the output of this node is considered to be the output of the entire program, but it need not be leaf node of the DAG.
On each call to a program (corresponding to one time-step of the system) the current input values and parameter values are propagated through the functional modules, and the output node's output given the RL algorithm. Before the call terminates, the FIFO buffers are updated and the adjustable parameters are updated via gradient descent using the Adam optimizer (Kingma & Ba, 2014]. Most operations are differentiable and thus able to propagate gradients backwards. Some operations are not differentiable, including buffers (to avoid backpropagating through time) and "Detach" whose purpose stopping the gradient from flowing back. In practice, we have multiple copies of the same agent running at the same time, with both shared policy and shared curiosity module. Thus, we execute multiple reward predictions on a batch and then update on a batch.
Programs representing several published designs for curiosity modules that perform internal gra- dient descent, including inverse features (Pathak et al.] 2017). random network distillation (RND) (Burda al. 2018), and ensemble predictive variance (Pathak et al., 2019). shown in figure (bigger versions can be found in appendix (A.3). We can also represent algorithms similar to novelty search (Lehman & Stanley 2008) and EX (Fu et al., 2017). which include buffers and nearest neighbor regression modules. Details on the data types and module library are given in appendix A
crucial, and possibly somewhat counter-intuitive, aspect of these programs is their use of neural network weight updates via gradient descent as form of memory. In the parameter update step, all adjustable parameters are decremented by the gradient of the sum of the outputs of the loss modules, with respect to the parameters. This type of update allows the program to, for example, learn make some types of predictions, online, and use the quality of those predictions a state to modulate the proxy reward for visiting that state (as is done, for example, in RND).
Key to our program search are polymorphic data types: the inputs and outputs to each module are typed, but the instantiation of some types, and thus of some operations, depends the environment. We have four types: reals R, state space of the given environment S, action space of the given en- vironment A and feature space F, used for intermediate computations and always set to 32 in current implementation. For example, a neural network module going from S to F will be instanti- ated convolutional neural network if is image and fully connected neural network of the
4

---

Published as a conference paper at ICLR 2020
appropriate dimension if S is vector. Similarly, if we measuring an error in action space A we use mean arean-squared error for continuous action spaces and negative log- loglliillhood for discrete action spaces. This facility means that the same curiosity program can be applied, independent of whether states are represented as images or vectors, or whether the actions are discrete or continuous, or the dimensionality of either.
This type of abstraction enables our meta-learning approach to discover curiosity modules that gen- eralize radically, applying not just new tasks, but tasks with substantially different input and output spaces than the tasks they were trained on.
To clarify the semantics of these programs, we walk through the operation of the RND program in figure 2 Its only input is $t+1, which might be an image or an input vector, which is processed by two NNs with parameters θ₃ and 22, respectively. The structure of the NNs (and, hence, the dimensions of the ④;) depends on the type of 8 if Sc+-1 is an image, then they are CNNs, otherwise fully connected networks. Each NN outputs a 32-dimensional vector; the L2 distance between these vectors is the output of the program on this iteration, and is also the input to loss module. So, given an input 8t41, the output intrinsic reward is large if the two NNs generate different outputs and small otherwise. After each forward pass, the weights in O2 are updated to minimize the loss while O₃ remains constant, which causes the trainable NN to mimic the output of the randomly initialized NN. As the program's ability predict the output of the randomized NN on an input improves, the intrinsic reward for visiting that state decreases, driving the agent to visit new states.
To limit the search space and prioritize short, meaningful programs we limit the total number of modules of the computation graph to 7. Our language is expressive enough to describe many (but far from all) curiosity mechanisms in the existing literature, as well as many other potential alternatives, but the expressiveness leads to a very large search space. Additionally, removing or adding a single operation can drastically change the behavior of a program, making the objective function non- smooth and, therefore, the space hard to search. In the next section we explore strategies for speeding up the search over tens of thousands of programs.
3 IMPROVING THE EFFICIENCY OF OUR SEARCH
We wish to find curiosity programs that work effectively in wide range of environments, from simple to complex. However, evaluating tens of thousands of programs in the most expensive envi- ronments would consume decades of GPU computation. Therefore, we designed multiple strategies for quickly discarding less promising programs and focusing computation grams. In doing so, we take inspiration from efforts in the AutoML community (Hutter et al. 2018).
few promising pro-
We divide these pruning efforts into three categories: simple tests that are independent of running the program in any environment, "filtering" by ruling out some programs based on poor performance in simple environments, and Mesta--met-RLL) learning to predict which curiosity programs will produce good RI. agents based on syntactic features.
3.1 PRUNING INVALID ALGORITHMS WITHOUT RUNNING THEM
Many programs are obviously bad curiosity programs. We have developed two heuristics to imme- diately prune these programs without an expensive evaluation.
Checking that programs are not duplicates. Since our language is highly expressive, there are many non-obvious ways of getting equivalent programs. To find duplicates, we de- signed a randomized test where we identically seed two programs, feed them both identical fake environment data for tens of steps and check whether their outputs are identical.
Checking that the loss functions cannot be minimized independently of the input data. Many programs optimize some loss depending on neural network regressors. If we treat inputs as uncontrollable variables and networks sible function, then for every variable, we can determine whether neural networks can be optimized to minimize it, independently of the input data. For example, if our loss function is Nos)) 2 the neural network can learn to make it by disregarding s and optimizing the weights to O. We discard any program that has this property.
having the ability to become any pos-
5

---

Published as a conference paper at ICLR 2020
3.2 PRUNING ALGORITHMS CHEAP ENVIRONMENTS
Our ultimate goal is to find algorithms that perform well on many different environments, both simple and complex. We make two key observations. First, there may only tens of reasonable programs that perform well on all environments but hundreds of thousands of programs that per- form poorly. Second, there are some environments that are solvable in a few hundred steps while others require tens of millions. Therefore, a key idea in our search is to try many programs in cheap environments and only a few promising candidates in the most expensive environments. This was inspired by the effective use of sequential halving (Karnin et al. 2013) in hyper- parameter optimiza- tion (Jamieson & Talwalkar 2016).
By pruning programs aggressively, may be losing multiple programs that perform well com- plex environments. However, by definition, these programs will tend to be less general and robust than those that succeed in all environments. Moreover, sake, but also to ease the search since, even if we only cared about the most expensive environment, performing the complete search only in this environment would be impractical.
seek generalization not only for its own
3.3 PREDICTING ALGORITIMM PERFORMANCE
Perhaps surprisingly, we find that we can predict program performance directly from program struc- ture. Our search process bootstraps initial training of (program structure, program perfor- mance) pairs, then uses this training set to select the most promising next programs to evaluate. We encode each program's 's structure with features representing how many times each operation is used, thus having as many features as number of operations in our vocabulary. We use a k-nearest- neighbor regressor, with k 10. We then try the most promising programs and update the regressor with their results. Finally, we add an <-greedy exploration policy to make sure we explore all the search space. Even though the correlation between predictions and actual values only moderately high (0.54 on a holdout test), this is enough to discover most of the top programs searching only half of the program space, which is our ultimate goal. Results are shown in appendix C
We can also prune algorithms during the training process of the RL agent. In particular, any point during the meta-search, we use the top current best programs as benchmarks for all time- steps. Then, during the training of a new candidate program time t with the performance at time of the top K programs and stop the run if its performance is significantly lower. If the program is not pruned and reaches the final time-step T with of the top K performances, it becomes part of the benchmark for the future programs.
compare its current performance at
4 EXPERIMENTS
Our RL agent uses PPO (Schulman al. 2017) based on the implementation by Kostrikov (2018) in PyTorch (Paszke et al. 2017). Our code (https: htt::/gittttbaa wwtmcoommmmmcmcmecmm Ptta-learnin www.rniinnccuiiooo www.cuusttyyaaaooo all.ossttlllooiriieoooooooo gorittmstt can take in any OpenAl gym environ-
ment [Brockman et al. 2016) with specification of the desired exploration horizon T'.
We evaluate each curiosity algorithm for multiple trials, using a seed dependent on the trial but independent of the algorithm, which leads to the PPO weights and curiosity data-structures being initialized identically on the same trials for all algorithms. As common in PPO, we run multiple rollouts (5, except for MuJoCo which only has 1), with independent experiences but shared policy and curiosity modules. Curiosity predictions and updates are batched across these rollouts, but not across time. PPO policy updates are batched both across rollouts and multiple timesteps.
4.1 FIRST SEARCH PHASE IN SIMPLE ENVIRONMENT
We start by searching for good intrinsic curiosity program in a purely exploratory environment, designed by Chevalier- Boisvert et al. 2018)) which is image-based grid world where agents navigate in an image of 2D room either by moving forward in the grid or rotating left or right. We optimize the total number of distinct cells visited across the agent's lifetime. This allows us to evaluate intrinsic reward programs in a fast and simple environment, without worrying about combining it with external reward.

---

Published as a conference paper at ICLR 2020
To bias towards simple, interpretable algorithms and keep the search space manageable, search for programs with at most 7 operations. We first discard duplicate and invalid programs, as described in section 3.1 resulting in about 52.000 programs. We then randomly split the programs across machines, each with 8 Nvidia Tesla K80 GPUs for 10 hours; thus a total of 13 GPU days.
Each machine finds the highest-scoring 625 programs in its section of the search space and prunes programs whose partial learning curve is statistically significantly lower than the current top 625 programs. To do so, after every episode of every trial, we check whether meaAprogrom (step) S mea72206625 (slep) 25ldzop625 sldprogram- Thus, we account for both inter-program variabil- ity among the top 625 programs and intra- program variability among multiple trials of the same program.
We use n--nearest-neighbor regressor to pre- dict program performance and choose the next program to evaluate with an c-greedy strategy, choosing the best predicted program 90% of the time and a random program 10% of the time. By doing this, we try the most promising pro- grams early in our search. This is important for
s(t+1)
0[1]: S F
s(t)
a(t)
NN: S
NN: S
two reasons: first, only try 26,000 programs, half of the whole search space, which we esti-
target
prediction
mated from earlier results (shown in figure g in the appendix) would be enough to get 88% of the top 1% of programs. Second, the earlier we run our best programs, the higher the bar for later programs, thus allowing us to prune them earlier, further saving computation time. Searching through this space took a total of 13 GPU days. As shown in figure in the ap- pendix, we find that most programs perform rel- atively poorly, with a long tail of programs that are statistically significantly better, comprising roughly 0.5% of the whole program space.
Action Prediction Loss
L2 Distance
Add To Loss
Figure 3: Fast Action-Space Transition(FAST): hop-performing intrinsic curiosity algorithm dis- covered in our phase search.
The highest scoring program (a few other programs have lower average performance but are statisti- cally equivalent) is surprisingly simple and meaningful, comprised of only operations, even though the limit was 7. This program, which we call FAST (Fast Action-Space Transition), is shown in fig- ure 3 it trains single neural network (a CNN MIP depending the action from st41 and then compares its predictions based on St41 with its predictions based on St, generating high intrinsic reward when the difference is large. The action prediction loss module either computes softmax followed by NLL loss or appends zeros to the action to match dimensions and applies MSE loss, depending on the type of the action space. Note that this is not the same as rewarding taking a different action in the previous time-step. The network predicting the action is learning to imitate the policy learned by the internal RL agent, because the curiosity module does not have direct access to the RI agent's 's internal state.
the type of state) predict
Of the top 16 programs, 13 are variants of FAST, including versions that predict the action from instead of 1.+1. The other 3 are variants of more complex program that is hard to understand at first glance, but we finally determined to be using ideas similar to cycle-consistency in the GAN literature Zhu al. (2017) (we thus name it Cycle-consistency intrinsic motivation); the diagram and explanation are in figure 10 in the appendix. Interestingly, to the best of our knowledge neither algorithm had been proposed before:
conjecture the former too simple for humans to believe would be effective and the latter too hard for humans to design, as it was already very hard to understand in hindsight.
4.2 TRANSFERRING TO NEW ENVIRONMENTS
Our reward combiner developed in lunar lander (the simplest environment with meaningful extrinsic reward) based on the best program among preliminary set 16,000 programs (which re- sembled Random Network Distillation; its computation graph is shown in appendix E. Among set
7

---

Published as a conference paper at ICLR 2020
ll start clear
Percanntle 90.5;
start signitaonce
100
sightliconoe
16
120
16
-100
-200
Performanc Lurae Lander
-200
Performant on Acroboot
-200
-250
-000
-250
-400
-45
-000
400
GridWorld
Performance GridWoorld
Figure 4: Correlation between program performance in gridworld and in harder environments (lunar lander on the left, acrobot the right), using the top 2,000 programs in gridworld. Performance is evaluated using mean reward across all learning episodes, averaged over trials (two trials for acrobot / lunar lander and five for gridworld). The high number of algorithms performing around -300 in the middle of the right plot is an artifact of averaging the performance of two seeds and the mean per- formance in Acrobot having two peaks. Almost all intrinsic curiosity programs that had statistically significant performance for grid world also do well on the other two environments. In green, the performance of three published works; in increasing gridworld performance: disagreement (Pathak et al. 2019), inverse features (Pathak et al. 2017) and random distillation (Burda et al. 2018).
2,500 candidates (with 5 fewer operations) the best reward combiner discovered by our search was Fr = (1+i. (iT)), tT, Notice that for i << 1 (usually the case) this is approximately f ~ i (1 t/TTi + (t/T)rr, which is down-scaled version of intrinsic reward plus linear interpolation that ranges from all intrinsic reward at 0 to all extrinsic reward = T. future work, we hope to co-adapt the search for intrinsic reward programs and combiners as well as find multiple reward combiners.
14%,
Given the fixed reward combiner and the list of 2,000 selected programs found in the image-based grid world, we evaluate the programs on both lunar lander and acrobot, in their discrete action space versions. Notice that both environments have much longer horizons than the image-based based grid world (37,500 and 50,000 vs 2,500) and they have vector vector-based, based, rather than image-based, inputs. The results in figure 4 show good correlation between performance on grid world and on each of the new environments. Especially interesting is that, for both environments, when intrinsic reward in grid world is above 400 (the lowest score that statistically significantly good), performance on the other two environments is also good in more than 90% of cases.
Finally, we evaluate on two MuJoCo environments (Todorov et al. 2012): hopper and ant. These environments have more than an order of magnitude longer exploration horizon than acrobot and lunar lander, exploring for 500K time-steps, as well as continuous action-spaces instead of discrete. We then compare the best 16 programs on grid world (most of which also did well on lunar lander and acrobot) to four weak baselines (constant 0,-1,1 intrinsic reward and Gaussian noise reward) and three published algorithms expressible in our language (shown in figure 2b. We run two trials for each algorithm and pool all results in each category to get confidence interval for the mean of that category. All trials used the reward combiner found on lunar lander. For both environments find that the performance our top programs is statistically equivalent to published work and sig- nificantly better than the weak baselines, confirming that we meta-learned good curiosity programs.
Note that we meta- eettattriind our intrinsic curiosity programs only on one environment (GridWorld) and showed they generalized well to other very different environments: they perform better than pub- lished works in this meta-train task and one meta-test task (Acrobot) and on par in the other 3 tasks meta-test tasks. Adding more meta-training tasks would be simple as standardising the perfor-
8

---

Published as a conference paper at ICLR 2020
Class Baseline algorithms
Ant
Hopper [318.5, 525.0]
1-95.3,-39.9
Meta-learned algorithms
[+67.5, +80.0]
[589.2, 650.6]
Published algorithms
[+67.4, +98.8]
[627.7, 692.6]
Table 1: Meta-leamndd algorithms perform significantly better than constant rewards and statistically equivalently to published algorithms found by human researchers (see 2). The table shows the con- fidence interval (one standard deviation) for the mean performance (across trials, across algorithms) for each algorithm category. Performance is defined as mean episode reward for all episodes.
mance within each task (to make results comparable) and then selecting the programs with best mean performance. We chose to only meta-train on single, simple, task because it (surprisingly!) already gave great results, highlighting the broad generalization of meta-learning program representations.
RELATED WORK
In some regards our work is similar to neural architecture search (NAS) (Stanley & Miikkulainen 2002 Zoph & Le [
Elsken et al. 2018 Pham et al. (2018) or hyperparameter optimization for deep networks (Mendoza et al. 2016), which aim finding the best neural network architecture and hyper-parameters for particular task. However, in contrast to most (but not all, see Zoohh et al.) (2018) NAS work, we want to generalize to many environments instead of just one. Moreover, search over programs, which include non-neural operations and data structures, rather than just neural-networt architectures, and decide what loss functions to use for training. Our work also re- sembles work in the AutoML community (Hutter et al., 2018) that searches in space of programs, for example in the case of SAT solving (KhudaBukhsh et al. 2009) or auto-sklearn (Feurer et al. 2015) and concurrent work learning loss functions to replace cross-entropy for training fixed architecture on MNIST and CIFAR (Gonzalez & Miikkulainen, 2019
2020). Although we took
inspiration from ideas in that community (Jamieson & Talwalkar 2016 Li et (016), our algo- rithms specify both how to compute their outputs and their own optimization objectives in order to work well in synchrony with an expensive deep RL algorithm.
There has been work on meta-learning with genetic programming (Schmidhuber 1987 searching over mathematical operations within neural networks (Ramachandran et al.) [207]] Gaier & Hap 2019), searching over programs to solve games (Wilson et al. 2018 Kelly & Heywood 2017 Silver al. 2019) and to optimize neural networks (Bengio et al. 1995| Bello et al., 2017), and neural networks that learn programs (Reed & De Freitas, | et al. 2019). Our work uses neural networks as basic operations within larger algorithms. Finally, modular meta-learning (Alet et al.) 2018 2019) trains the weights of small neural modules and transfers to new tasks by searching for a good composition of modules; as such, it can be seen as a (restricted) dual of our approach.
There has been much interesting work in designing intrinsic curiosity algorithms. We take inspiration from many of them to design our domain-specific language. In particular, we rely on implicit memory, which scales well to millions
the idea of using neural network training of time-steps, as well as buffers and nearest-neighhour regressors. As we showed in figure 2 we can represent several prominent curiosity algorithms. We can also generate meaningful algorithms similar to novelty search (Lehman & Stanley 2008) and E (Fu et al., 2017); which include buffers and nearest neighbours. However, there are many exploration algorithm classes that do not cover, such as those focusing on generating goals (Srivastava et al. 2013 Kulkarni et al. 2016 Florensa et al., 2018). learning progress (Oudeyer al. 2007] SScmiihhubrr 20088 Azar et al. 2019), generating diverse skills (Eysenbach et al. 2018), stochastic neural networks (Florensa et al., 2017 Fortunato et al. (017). count-based exploration (Tang et al. 2017) or object-based curiosity measures (Forestier & Oudeyer 2016). Finally, part of our motivation stems from Taīga et al. (2019) showing that some based curiosity algorithms have trouble generalising to new environments.
There have been research efforts meta-learning exploration policies: Duan et al. 2016); Wang et al. (2017) learn an LSTM that explores an environment for one episode, retains its hidden state and is spawned in second episode in the same environment; by training the network maximize the reward in the second episode alone it learns to explore efficiently in the first episode. (Stadic et al. 2018) improves their exploration and that of Finn et al. (2017) by considering the importance of

---

Published as a conference paper at ICLR 2020
sampling in RL policies. Gupta et al. (2018) combine gradient- based meta-learning with learned latent exploration space in which they add structured noise for meaningful exploration. Closer to our formulation, [Zheng et al. (0188) parametrize an intrinsic reward function which influences policy- gradient updates in a differentiable manner, allowing them to backpropagate through a single step of the policy-gradiens update to optimize the intrinsic reward function for single task. In contrast all three of these methods, we search over algorithms, which will allows us to generalize more broadly and to consider the effect of exploration on up 10y 10c time-steps instead of the 10 10 of previous work. Finally, Siraally,Chinng CChiag et al. (2019): y: Faust et al. (2119) have setting similar to ours where they modify reward functions over the entire agent's ' lifetime, but instead of searching over intrinsic curiosity algorithms they tune the parameters of hand-designed reward function.
Related work on meta-learning (Schmidhuber 1987 Thrun & Pratt 1998 Clune 2019) and efforts to increase its generalization can be found in appendix [ Closest to our work, evolved policy gra- dients (EPG, [Houthooft et al. (2018)) use evolutionary strategies to meta-learn a neural network that acts loss function and is used to train policy network. EPG generalizes by meta-training with target locations east of the start location and meta-testing with target locations to the west. In contrast, showed that by meta-learning programs, can generalize between radically differ- ent environments, not just goal variations of a single environment. Concurrent to our work, Kirsch et al. (209)) also show generalization capabilities between environments similar ours (lunar lan- der, hopper and half-cheetah). Their approach transfers a parametric representation, for which is unclear how to adapt the learned neural losses to an unseen environment with different observation space. Their approach thus does not encode states into the loss function, which is critical for efficient exploration. In contrast, our algorithms can leverage polymorphic data types that adapt the neural networks to the environment they are running in, adapting both the size and the type of network (CNN vs MLP) running in each environment.
CONCLUSIONS
In this work, we proposed to meta-learn algorithms and show that by transferring programs we can generalize between tasks much more varied than previously possible in meta-RI, even between those with different input or output spaces. In many settings, however, the input and output space remain the same as change tasks. This opens the possibility of getting the best of both worlds by meta-learning weights along with structure, thus simultancously transferring domain- speciffcc knowledge in the weights and higher-level algorithmic knowledge in the architecture. In addition, we note that the approach of meta-learning programs instead of network weights may have further applications beyond finding curiosity algorithms, such as meta- meta-learning optimization algorithms or even meta-learning meta-learning algorithms. Our relatively modest compute (2 GPU-weeks) and a simple search method restricted could search over significantly bigger spaces. It thus may be possible to automatically search for new machine learning algorithms from more fundamental building blocks for wide variety of problems.
medium-sized search space, but we expect that future work
ACKNOWLEDGMENTS
We thank Kelsey Allen, Peter Karkus, Kevin Smith, Josh Tenenbaum and the rest of the Honda- CMM MIT team for their insightful feedback. We thank Chris Lu for his idea on what the algorithm in figure i0 computing. We also want to thank Bernadette Bucher, Chelsea Finn, Abhishek Gupta, Deepak Pathak, Lerrel Pinto, Oleh Rybkin, Karl Schmeckpeper and Joaquin Vanschoren for valuable conversations. Finally, we also want thank Maria Bauza and Tej Chajed for their feedback on early drafts and Clement Gehring for his help setting up the experiments.
We gratefully acknowledge support from NSF grants 1523767 and 1723381, AFOSR grant FA9550- 17-1-0165, ONR grant N00014- 8- 1-2847, the Honda Research Institute, SUTD Temasek Labora- tories and the MIT Quest for Intelligence. Any opinions, findings, and conclusions or recommenda- tions expressed in this material do not necessarily reflect the views of our sponsors.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Ferran Alet, Tomas Lozano-Perez, and Leslie P. Kaelbling. Modular meta-lcarning. In Proceedings The 2nd Conference on Robot Learning, pp. 856-868, 2018.
Ferran Alet, Erica Weng, Tomas Lozano-Perez, and Leslie Kaclbling. Neural relational infer- ence with fast modular meta-learning. In Advances in Neural Information Processing Systems (NeurIPS) 32. 2019.
Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo Avila Pires, Jean-Bastian Grill, Florent Altché, and Rémi Munos. World discovery models. arXiv preprint vrXiv:1902.07685 2019.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoe V Le. Neural optimizer search with reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 459 468. JMLR. org, 2017.
Samy Bengio, Yoshua Bengio, and Jocelyn Cloutier. On the search for new learning rules for anns. Neural Processing Letters, 2(4))26-30, 1995.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.013: 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 00/8//0.22280 /2894, 2018.
Maxime Chevalier- for openai gym. attps: //tp:::///tttuu com wwtp//wwwwimm/mem/e//mmm//.cwcct wwwp.i//m///ymmmmiii/e/a
Lucas Willems, and Suman Pal. Minimalistic gridworld environment http:///www....////////e//////// gym minnirin 2018.
Hao-Tien Lewis Chiang, Aleksandra Faust, Marek Fiser, and Anthony Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 442):2007-2014, 2019.
Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-learning for model-based control. In International Conference on Learning Representations, 2019.
Jeff Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv: 1905. 2019.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint
2016.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, arXiv preprint 1r08.:/808.055
2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint 10XXv:/802.0070, 2018.
Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards automate reinforcement learning. arXiv preprint 1oXivv:905.07661 2019.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 2017.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2962-2970. Curran Associates, Inc., 2015. URL http: /ttp:er/pppreeee/ mitrssiipppp. nips...... CC/ cttppper//pp/ paper/ 5872- fifficient... and robust aut oma ted mach ne machine--earningg learning.ppi..
Chelsea Finn. Learning to Learn with Gradients. PhD thesis, EECS Department, University of California, Berkeley, Aug 2018. URL TechRpts/20133 B/EECS 201 8- 105. html
eecs www.eley..e.y...... edu/Pubs/

---

Published as a conference paper at ICLR 2020
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. arXiv preprint arXiv: 2017.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint 1704::::704.00000 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research. pp. 1515-1528, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http:/ http:///cccceeiiggs... mlr. press/ http://////ooreeaa....am.
Sébastien Forestier and Pierre-Yves Oudeyer. Modular active curiosity-driven discovery of tool use. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp- 3965-3972. IEEE, 2016.
Meire Fortumato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv: I 706. 10295, 2017.
Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. Advances in Neural Information Processing Systems, pp. 2577-2587, 2017.
Adam Gaier and David Ha. Weight agnostic neural networks. arXiv preprint arXiv:1906.04358, 2019.
Santiago Gonzalez and Risto Miikkulainen. Improved training speed, aceuracy, and data utilization through loss function optimization. arXiv preprint arXiv: 1528, 2019.
Santiago Gonzalez and Risto Mikkuulainen. Evolving loss functions with multivariate taylor poly- nomial parameterizations, 2020.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311, 2018. 2018.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Advances in Neural Information Processing Systems, pp. 5400-5409, 2018.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning: Meth- ods, Systems, Challenges. Springer, 2018. In press, available at http://automllorg/bbookk
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics, pp. 240-248, 2016.
Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed bandits. In International Conference on Machine Learning, pp. 1238-1246, 2013.
Stephen Kelly and Malcolm Heywood. Multi-task learning in atari video games with emergent tan- gled program graphs. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 195-202. ACM, 2017.
Ashiqur R KhudaBukhsh, Lin Xu, Holger H Hoos, and Kevin Leyton-Brown. Brown. Satenstein: Auto- matically building local search sat solvers from components. In Twenty-First International Joint Conference on Artificial Intelligence, 2009.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6990, 2014.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. arXiv preprint 1911v:1910.04098. 2019.
12

---

Published as a conference paper at ICLR 2020
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https:// -oapaa--pppaakt--aiii. 2018.
github.
Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016.
Joel Lehman and Kenneth Stanley. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp. 329-336, 2008.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv: 1603.06560, 2016.
Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. To- wards automatically-tume neural networks. In Workshop on Automatic Machine Learning, pp- 58-65, 2016.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv:/804.0372 2018.
Pierre-Yves Oudeyer. Computational theories of curiosity-driven learning.
arXiv preprint
1802./0000/5 2018.
Pierre- Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-266, 2007.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning, arXiv preprint arXiv: 1511:055111.0344 2015.
Adam Paszke, Sam Gross, and Adam Lerer. Automatic differentiation in PyTorch. In International Conference Learning Representations, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Auriosity-driver exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. arXiv preprint 1r0.v:1906.044 2019.
Hicu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: / 008.002...2... 2018.
Thomas Pierrot. Guillaume Ligner, Scott Reed, Olivier Sigaud, Nicolas Perrin, Alexandre Laterre, David Kas, Karim Beguir, and Nando de Freitas. Learning compositional neural programs with recursive tree search and planning. arXiv preprint arXiv: /2941, 2019.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint 1rXiv:1710.0594 2017.
Scott Reed and Nando De Freitas. arXiv:1511.0677.. 2015.
Neural programmer-interpreeess. arXiv preprint
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:1606.04677 2016.
Jürgen Schmidhuber.
Evolutionary principles in self-referential learning, learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Jürgen Schmidhuber. Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. Workshop on anticipatory behavior in adaptive learning systems, pp. 48-76. Springer, 2008.
13

---

Published as a conference paper at ICLR 2020
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Tom Silver, Kelsey Allen, Alex K Lew, Leslie Pack Kaelbling, and Josh Tenenbaum. Few-shot bayesian imitation learning with logic over programs. arXiv preprint
2019.
Rupesh Kumar Srivastava, Bas R Steunebrink, and Jürgen Schmidhuber. First experiments with powerplay. Neural Networks, 41:130-136, 2013.
Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations learning to explore via meta-reinforcement learning. arXiv preprint arXiv:1803.01 1800:01180..111 018, 2018.
Kenneth Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation,, 10(2):99-177, 2002.
Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, and Mare G Bellemare. Benchmarking bonus- based exploration methods on the arcade learning environment. arXiv preprint arXiv: /908. 02388, 2019.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Chen, Yan Duan, John Schul- man, Filip DeTurck, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753- 2762, 2017.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-500660 5033. IEEE, 2012.
Leslic Valiant. Probably Approximately Correct: NatureOs Algorithms for Learning and Prospering in Complex World. Basic Books (AZ), 2013.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. arXiv preprint 0ttiv:/999.04444 2019.
JX Wang, Z Kurth-Nelson. D Tirumala, H Soyer, JZ Leibo, R Munos, C Blundell, D Kumaran, and M Botivnick. Learning to reinforcement learn. arxiv 1611.05763, 2017.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poct): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 2019.
Dennis Wilson, Sylvain Cussat-Bland Hervé Luga, and Julian Miller. Evolving simple programs for playing atari games. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. ACM, 2018
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in neural information processing systems, pp. 2396-2407, 2018.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Sergey Levine, and
Chelsea Finn. Meta-world: benchmark and evaluation for multi-task and meta-reiifforeement learning, 2019. URL / //tithub.. com/ wwtp://wwwwworooo......... wwwwrkgroo///ettaorr....
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pp. 4644 2018.
Jun-Yan Zhu, Tacsung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv: (Tl::////.1177 2016.
14

---

Published as a conference paper at ICLR 2020
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
15

---

Published as a conference paper at ICLR 2020
A DETAILS OF OUR DOMAIN-SPECIHE LANGUAGE FOR CURIOSITY
ALGORITHMS
We have the following types. Note that S and A get defined differently for every environment.
IR: real numbers such as ry or the dot-product between two vectors.
R+: numbers guaranteed be positive, such the distance between two vectors. The only difference to our program search between IR and R+ is in pruning programs that can optimize objectives without looking the data. For R+ check whether they can optimize down to O, for R we check whether they can optimize to arbitrarily negative values.
state space S: the environment state, such as a matrix of pixels or a vector with robot joint values. The particular form of this type is adapted each environment.
action space either 1-hot description of the action the action itself. The particular form of this type is adapted to each environment.
feature-space F R32: space mostly useful to work with neural network embeddings.
For simplicity, we only have a single feature space.
List[X]: for each type we may also have a list of elements of that type. All operations that take a particular type as input can also be applied to lists of elements of that type by mapping the function to every element in the list. Lists also support extra operations such as average variance.
A.1 CURIOSITY OPERATIONS
Operation
Input type(s)
State
Output type
Add
R
IR
RunningNorrm
R
R
R
VariableAsBuffer
X
List X
List X
NearestNeighhorRegresse
F,
List
SubtractOne Tenth
NormalDisttibution
13
Subtract
R. IR
13
Sgrt(Abs(x))
R
NN NN F.F
F
+F
OF
NN F.F F
F.
OF.F+t
NNF F A
F
O7-hh
A
NN A + IF
A
WA-BB
F
(C)NN
S
OS-FF
(C)NN, Detach
S
O5-FF
I
(Sseeeeee
S
5x0g-FF
List F
NN Ensemble F F F
F
5x0x
List F
NN Ensemble F.F + F
F,
5x0x,E-4F
List
NN Ensemble F, A F
5X0A,F
List
MinimizeValue
R
Adam
L2Norm
X
R+
L2Distance
X, X
ActionSpaceLoss
X. A
DotProduct
X. X
Add
X. X
X
Detach
X
X
Mean
[R
Variance
List X
Mean
List X
Mapped L2 Norm
List %
List [
Average Distance
Lish X
X
Minus
List X, X
List X
16

---

Published as a conference paper at ICLR 2020
Note that X stands for the option of being F A. NearestNeighborRReeessor takes query and target, automatically creates a buffer the target (thus keeps list as a state) and answers based on the buffer. RunningNorm keeps track of the variance of the input and normalizes by that variance.
A.2 REWARD COMBINER OPERATIONS
Operation
Input type(s)
State
Output type
Constant (0.01,0.1.0.5.1
R
NormalDistribution
R
Add
R, R
R
Max
IR
R
Min
R. RR
R
WeightedNormallzedSSm
R. R. IR
R
Running
IR
R
Buuferr
R
List IR]
List [R]]
Subtract
R. I
R
Multiply
R, lk
R
Sqrt( Abs(x))
R
R
Mean
List R
Note that W Weighted wew.radooooaaiiieSneeeeee c,
ob+ [a|tlc]
RunningNorm keeps track of the vari-
ance of the input and normalizes by that variance.
A.3 TWO OTHER PUBLISHED ALGORITHMS COVERED BY OUR DSL
s((+1)
0(4):
s(O)
a(O)
0[1): F
0[2): FxF + A
NN: S + F
NN: S + F
0[3): F x F
NN: A F
target
NN: F + A
NN: F F F
prediction
Action Prediction Loss
L2 Distance
Add
Add To
Figure 5; Curiosity by predictive error on inverse features by Pathak al. (2017). In pink, paths and networks where gradients flow back from the minimizer.
17

---

Published as a conference paper at ICLR 2020
s(t)
0[1): F
s(++)
NN: S + F + Detach
a(t)
0[2]: F x A + FF
NN: S - F + Detach
NN: : - [F]
Variance
Average Distance
AddToLoss
Figure 6: Curiosity by ensemble predictive variance Pathak al. (2019). In pink, paths and net- works where gradients flow back from the minimizer.
B RELATED WORK ON META-RL AND GENERALIZATION
Most work on meta-RL. has focused on learning transferable feature representations or parameter values for quickly adapting to new tasks (Finn al., 2017 Finn 2018 Clavera et al. 2019) or im- proving performance on a single task (Xu et al.) 2018 Veeriah et al. 2019). However, the range of variability between tasks typically limited to variations of the same goal (such moving dif- ferent speeds or to different locations) or generalizing to different environment variations (such as different mazes or different terrain slopes). There have been some attempts to broaden the spectrum of generalization, showing transfer between Atari games thanks to modularity (Fernando et al. 2017]] Rusu et al., 2016) or proper pretraining (Parisotto et al., 2015). However, as noted by Nichol et al. (2018), Atari games are too different get big gains with current feature-transfer methods; they instead suggest using different levels of the game Sonic to benchmark generalization. Moreover, Yu al. (2019) recently proposed benchmark of many tasks. [Wag al. (2019) automatically gen- erate different terrains for a bipedal walker and transfer policies between terrains, showing that this more effective than learning a policy on hard terrains from scratch; similar our suggestion in section 3.2 In contrast to these methods, we aim at generalization between completely different environments, even between environments that do not share the same state and action spaces.
18

---

Published as a conference paper at ICLR 2020
C
PREDICTING ALGORITHM PERFORMANCE
Performance Regressor on the Test Set
Adhv0 eoforeeece
400
300
200
100
230
300
400
500
Predkcted Per Partoremance
Figure 7: Predicting algorithm performance from the structure of the program alone. Comparison between predicted and actual performance test set; showing a correlation of 0.54. In black, the identity line.
0.01 Programs Found Evalusted
programc
2000 4003
6000
30000
12000 20000 18000
avallated
Figure 8: Predicting algorithm performance allows find the best programs faster. We investigate the number the top 1% of programs found vs. the number of programs evaluated, and observe that the optimized search (in blue) finds 88% of the best programs after only evaluating 50% of the programs (highlighted in green). The naive search order would have only found 50% of the best programs that point.
19

---

Published as a conference paper at ICLR 2020
D PERFORMANCE ON GRID WORLD
600
500
Number of states visited
400
300
200
100
5000
10000
15000
20000
25000 0
1000
2000
Program index, sorted by mean performance
Figure 9: In black, mean performance across 5 trials for all 26,000 programs evaluated (out of their finished trials). In green mean plus one standard deviation for the mean estimate and in red one minus one standard deviation for the mean estimate. On the right, you can program means form roughly a gaussian distribution of very big noise (thus probably not significant) with a very small (between 0.5% and 1% of programs) long tail of programs with statistically significantly good performance (their red dots are much higher than almost all green dots), composed of algorithms leading to good exploration.
20

---

Published as a conference paper at ICLR 2020
E
INTERESTING PROGRAMS FOUND BY OUR SEARCH
0[1]: S - F
s(t)
0(3): S F
s(t+1)
NN: S + F
NN: S F
0[2): F + F
NN: S - F
target
query
target
query
Predict Target From Query
Predict Target From Query
L2 Distance
Figure 10: Cycle-Consistee Consistency Intrinsic Motivation algorithm, found by our search of the top 16 programs on grid world are variants of this program). The purple Predict Target From Query boxes feed the query to a neural network, return the prediction as output and add the prediction loss the optimization, back-propas propagating to the network and the query, but not the target. Notice that 0, is not getting trained because no loss back-propagates there; thus producing a random feature embedding et) from st). The algorithm combines several concepts seen in the literature, such an untrained network like RND Burda et al. (2018) and predicting another state in feature space like Pathak et al.) |017|[2099 but also includes weight sharing between both predictions, which makes the algorithm hard to interpret at first sight, see below for an in-depth explanation.
One can give meaning to the role of all 3 neural networks by considering how they contribute to minimizing the loss. To so, let us name the networks: [(1} (as labeled in the figure) as re, (for random embedding). ([2) as be, (for backwards) and 0[3} fro, (for forward and random embedding) and look at the algorithm in equation form:
i = llbe, (fre,(st)) bor (fre, (s
01 is kept at its random initialization
θ 02 = 02 1-- a0, (1lbe (Fres (st) ro, (s+)l++
llbs, (fre,(51+1) - free(
θ (Ilbe (froz(se)) -r, (st)l)
03 = 03 " 20g
We can see that ro, will indeed be random embedding because the network is randomly initialized and is not trained. Then, we observe that the second term the loss for 02, which does not involve 0g and thus O2 has to minimize alone, is [lb, (Sr+,(5(+1) a transformation of $141 and has to make very similar to the same transformation applied to st therefore, this term is similar to cycle- wele--oonsisteccy found in some other parts of machine learning Zhu et al.] (2017) and bø, must act like a backward model. Finally, looking at the minimization of 0g receives the original 8, and has to output vector such that the backward model will bring it close
freg(s:)ll ||. In this term, bø, receives
21

---

Published as a conference paper at ICLR 2020
the random embedding of st. Therefore fg must learn forward model composed with the random embedding of 01. Finally, we see that the algorithm outputs llbø, (fre, (sx)) bø, (fre, (5++1)) I|. going forward and backward for both $++1 and 8 and comparing the difference. In summary, this distance combines errors in the cycle-consistency of predictions (which will be higher unvisited parts of the state) with distance in the random embedding space between s(t) and s(t 1), i.e. moving to a very different state.
0[1] S + [F]
s(++1)
0[2]: S + F
NN: S [F]
NN: S - F Detach
NN: S + F
Minus
Average Distance
L2 Norm
Sum
Sqrt(Abs(x))
AddToLoss
Figure 11: Top variant in preliminary search on grid world; variant on random network distillation using an ensemble of trained networks instead of a single one.
22


Reviews:
[{"review_id": "H1gsicx6FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Instead of hand-designing exploration bonuses and intrinsic reward, the paper proposes to view the curiosity algorithms as programs described with domain-specific language (DSL), then search programs which allows RL agents to optimize the environment reward combined with the curiosity reward generated by the program. The search produces programs similar to algorithms proposed in literature and also some strategies which generalize well. In order to make the search tractable, the authors develop a few criteria: 1) evaluate the program on relatively simple and short-horizon domains; 2) predict the performance of the program and rank them; 3) stop the agents if the learning curves do not look good after long enough of training steps. \n\nThis is a very interesting idea, and it's partially inspired by the architecture search line of research. It would be great if the authors could provide more information about \"predicting algorithm performance\" section. \n\nI find it very interesting and exciting to see programs like Figure 3. They are quite interpretable. The results on Table 1 are really exciting, they show that the searched programs could be generalized into other unseen tasks. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gaST1RYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to meta-learn a curiosity module via neural architecture search. The curiosity module, which outputs a meta-reward derived from the agent’s history of transitions, is optimized via black box search in order to optimize the agent’s lifetime reward over a (very) long horizon. The agent in contrast is trained to maximize the episodic meta-reward and acts greedily wrt. this intrinsic reward function. Optimization of the curiosity module takes the form of an epsilon-greedy search, guided by a nearest-neighbor regressor which learns to predict the performance of a given curiosity program based on hand-crafted program features. The program space itself composes standard building blocks such as neural networks, non-differentiable memory modules, nearest neighbor regresses, losses, etc. The method is evaluated by learning a curiosity module on the MiniGrid environment (with the true reward being linked to discovering new states in the environment) and evaluating it on Lunar Lander and Acrobot. A reward combination module (which combines intrinsic and extrinsic rewards) is further evaluated on continuous-control tasks (Ant, Hopper) after having been meta-trained on Lunar Lander. The resulting agents are shown to match the performance of some recent published work based on curiosity and outperforms simple baselines.\n\nThis is an interesting, clear and well written paper which covers an important area of research, namely how to find tractable solutions to the exploration-exploitation trade-off. In particular, I appreciated that the method was clearly positioned with respect to recent work on neural architecture search, meta-learning approaches to curiosity as well as forthcoming about the method’s limitations (outlining many hand-designed curiosity objectives which fall outside of their search space). There are also some interesting results in the appendix which show the efficacy of their predictive approach to program performance.\n\nMy main reservation is with respect to the empirical validation. Very few existing approaches to meta-learning curiosity scale to long temporal horizons and “extreme” transfer (where meta-training and validation environments are completely different). As such, there is very little in the way of baselines. The paper would greatly benefit from scaled down experiments, which would allow them to compare their architecture search approach to recent approaches [R1, R2], black-box optimization methods in the family of evolution strategies (ES, NES, CMA-ES), Thompson Sampling [R3] or even bandits tasks for which Bayes-optimal policies are tractable (Gittins indices). These may very well represent optimistic baselines but would help better interpret the pros and cons of using neural architecture search for meta-learning reward functions versus other existing methods. Conversely, the paper claims to “search over algorithms which [...] generalize more broadly and to consider the effect of exploration on up to 10^5, 10^6 timesteps” but at the same time does not attempt to show this was required in achieving the reported result. Pushing e.g. RL2 or Learning to RL baselines to their limits would help make this claim.\n\nAlong the same line, it is regrettable that the authors chose not to employ or adapt an off-the-shelf architecture search algorithm such as NAS [R4] or DARTS [R5]. I believe the main point of the paper is to validate the use of program search for meta-learning curiosity, and not the details of the proposed search procedure (which shares many components with recent architecture search / black-box optimization algorithms). Using a state-of-the-art architecture search algorithm would have made this point more readily.\n\nAnother important point I would like to see discussed in the rebuttal, is the potential for cherry-picking result. How were the “lunar lander” and “acrobot” environments (same question for “ant” and “hopper”) selected? From my understanding, it is cheap to evaluate learnt curiosity programs on downstream / validation tasks. A more comprehensive evaluation across environments from the OpenAI gym would help dispel this doubt. Another important note: top-16 results reported in Figure 4 and Table 1 are biased estimates of generalization performance (as they serve to pick the optimal pre-trained curiosity program). Could the authors provide some estimate of test performance, by e.g. evaluating the performance of the top-1 program (on say lunar lander) on a held-out test environment? Alternatively, could you comment on the degree of overlap between the top 16 programs for acrobot vs lunar lander? Thanks in advance.\n\n[R1] Learning to reinforcement learn. JX Wang et al..\n[R2] RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. Yan Duan et al.\n[R3] Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search. Guez et al.\n[R4] Neural Architecture Search with Reinforcement Learning. Barrett et al.\n[R5] DARTS: Differentiable Architecture Search. Liu et al.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1lw3b8TFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an algorithm to generate curiosity modules for reinforcement learning. The authors define a program language which can represent many possible curiosity modules that include training neural networks, replay buffers, etc. It also presents an approach to searching for the best curiosity module in this set, which is some various ways to do pruning and to determine which methods to try.\n\nThe paper is very novel - the idea of developing a domain specific language full of building blocks to represent various curiosity modules is unique and interesting.\n\nThe search over curiosity modules is a bit mis-represented I think. In the introduction, it gives the impression that part of the algorithm is to search over these curiosity modules, and also that it's to find the best one that works across a wide set of tasks. Instead the search method is a separate procedure outside of the algorithm and most of the search steps are performed on individual tasks instead of over a set of tasks.\n\nIn Sec 3.3, you say that \"perhaps surprisingly, we find that we can predict performance directly from program structure,\" but you never provide any evidence of doing so. \n\nThe simple environment that you used is a bit contrived, rather than taking a normal task, the goal itself is to do complete exploration (maximize the total number of pixels visited). It seems like the combination of the intrinsic curiosity program here with the reward combiner is that the intrinsic curiosity program should be only about complete exploration, and then the combiner is responsible for balancing that with task rewards. You should be more explicit that in this first part of the search you're only looking at the intrinsic curiosity program, without the combiner, and therefore do not want a task with extrinsic rewards. This breakdown of searching for the intrinsic curiosity program first and the combiner later seems like another important aspect of making your search efficient. \n\nThe main drawback of this paper is that there are little comparisons to related work. The only methods compared to are ones where the curiosity method is expressible in the language of the method.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes meta-learning auxiliary rewards as specified by a DSL. The approach was considered innovative and the results interesting by all reviewers. The paper is clearly of an acceptable standard, with the main concerns raised by reviewers having been addressed (admittedly at the 11th hour) by the authors during the discussion period. Accept.
Decision: Accept (Poster)