Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes to meta-learn a curiosity module via neural architecture search. The curiosity module, which outputs a meta-reward derived from the agent’s history of transitions, is optimized via black box search in order to optimize the agent’s lifetime reward over a (very) long horizon. The agent in contrast is trained to maximize the episodic meta-reward and acts greedily wrt. this intrinsic reward function. Optimization of the curiosity module takes the form of an epsilon-greedy search, guided by a nearest-neighbor regressor which learns to predict the performance of a given curiosity program based on hand-crafted program features. The program space itself composes standard building blocks such as neural networks, non-differentiable memory modules, nearest neighbor regresses, losses, etc. The method is evaluated by learning a curiosity module on the MiniGrid environment (with the true reward being linked to discovering new states in the environment) and evaluating it on Lunar Lander and Acrobot. A reward combination module (which combines intrinsic and extrinsic rewards) is further evaluated on continuous-control tasks (Ant, Hopper) after having been meta-trained on Lunar Lander. The resulting agents are shown to match the performance of some recent published work based on curiosity and outperforms simple baselines.

This is an interesting, clear and well written paper which covers an important area of research, namely how to find tractable solutions to the exploration-exploitation trade-off. In particular, I appreciated that the method was clearly positioned with respect to recent work on neural architecture search, meta-learning approaches to curiosity as well as forthcoming about the method’s limitations (outlining many hand-designed curiosity objectives which fall outside of their search space). There are also some interesting results in the appendix which show the efficacy of their predictive approach to program performance.

My main reservation is with respect to the empirical validation. Very few existing approaches to meta-learning curiosity scale to long temporal horizons and “extreme” transfer (where meta-training and validation environments are completely different). As such, there is very little in the way of baselines. The paper would greatly benefit from scaled down experiments, which would allow them to compare their architecture search approach to recent approaches [R1, R2], black-box optimization methods in the family of evolution strategies (ES, NES, CMA-ES), Thompson Sampling [R3] or even bandits tasks for which Bayes-optimal policies are tractable (Gittins indices). These may very well represent optimistic baselines but would help better interpret the pros and cons of using neural architecture search for meta-learning reward functions versus other existing methods. Conversely, the paper claims to “search over algorithms which [...] generalize more broadly and to consider the effect of exploration on up to 10^5, 10^6 timesteps” but at the same time does not attempt to show this was required in achieving the reported result. Pushing e.g. RL2 or Learning to RL baselines to their limits would help make this claim.

Along the same line, it is regrettable that the authors chose not to employ or adapt an off-the-shelf architecture search algorithm such as NAS [R4] or DARTS [R5]. I believe the main point of the paper is to validate the use of program search for meta-learning curiosity, and not the details of the proposed search procedure (which shares many components with recent architecture search / black-box optimization algorithms). Using a state-of-the-art architecture search algorithm would have made this point more readily.

Another important point I would like to see discussed in the rebuttal, is the potential for cherry-picking result. How were the “lunar lander” and “acrobot” environments (same question for “ant” and “hopper”) selected? From my understanding, it is cheap to evaluate learnt curiosity programs on downstream / validation tasks. A more comprehensive evaluation across environments from the OpenAI gym would help dispel this doubt. Another important note: top-16 results reported in Figure 4 and Table 1 are biased estimates of generalization performance (as they serve to pick the optimal pre-trained curiosity program). Could the authors provide some estimate of test performance, by e.g. evaluating the performance of the top-1 program (on say lunar lander) on a held-out test environment? Alternatively, could you comment on the degree of overlap between the top 16 programs for acrobot vs lunar lander? Thanks in advance.

[R1] Learning to reinforcement learn. JX Wang et al..
[R2] RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. Yan Duan et al.
[R3] Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search. Guez et al.
[R4] Neural Architecture Search with Reinforcement Learning. Barrett et al.
[R5] DARTS: Differentiable Architecture Search. Liu et al.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents an algorithm to generate curiosity modules for reinforcement learning. The authors define a program language which can represent many possible curiosity modules that include training neural networks, replay buffers, etc. It also presents an approach to searching for the best curiosity module in this set, which is some various ways to do pruning and to determine which methods to try.

The paper is very novel - the idea of developing a domain specific language full of building blocks to represent various curiosity modules is unique and interesting.

The search over curiosity modules is a bit mis-represented I think. In the introduction, it gives the impression that part of the algorithm is to search over these curiosity modules, and also that it's to find the best one that works across a wide set of tasks. Instead the search method is a separate procedure outside of the algorithm and most of the search steps are performed on individual tasks instead of over a set of tasks.

In Sec 3.3, you say that "perhaps surprisingly, we find that we can predict performance directly from program structure," but you never provide any evidence of doing so. 

The simple environment that you used is a bit contrived, rather than taking a normal task, the goal itself is to do complete exploration (maximize the total number of pixels visited). It seems like the combination of the intrinsic curiosity program here with the reward combiner is that the intrinsic curiosity program should be only about complete exploration, and then the combiner is responsible for balancing that with task rewards. You should be more explicit that in this first part of the search you're only looking at the intrinsic curiosity program, without the combiner, and therefore do not want a task with extrinsic rewards. This breakdown of searching for the intrinsic curiosity program first and the combiner later seems like another important aspect of making your search efficient. 

The main drawback of this paper is that there are little comparisons to related work. The only methods compared to are ones where the curiosity method is expressible in the language of the method.


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Instead of hand-designing exploration bonuses and intrinsic reward, the paper proposes to view the curiosity algorithms as programs described with domain-specific language (DSL), then search programs which allows RL agents to optimize the environment reward combined with the curiosity reward generated by the program. The search produces programs similar to algorithms proposed in literature and also some strategies which generalize well. In order to make the search tractable, the authors develop a few criteria: 1) evaluate the program on relatively simple and short-horizon domains; 2) predict the performance of the program and rank them; 3) stop the agents if the learning curves do not look good after long enough of training steps. 

This is a very interesting idea, and it's partially inspired by the architecture search line of research. It would be great if the authors could provide more information about "predicting algorithm performance" section. 

I find it very interesting and exciting to see programs like Figure 3. They are quite interpretable. The results on Table 1 are really exciting, they show that the searched programs could be generalized into other unseen tasks. 
review_title: Official Blind Review #1

Meta-review: This paper proposes meta-learning auxiliary rewards as specified by a DSL. The approach was considered innovative and the results interesting by all reviewers. The paper is clearly of an acceptable standard, with the main concerns raised by reviewers having been addressed (admittedly at the 11th hour) by the authors during the discussion period. Accept.
Decision: Accept (Poster)