Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: (Originally my score was a weak reject.)

This paper aims to study whether a learned reward function can serve as a locus of knowledge about the environment, that can be used to accelerate training of new agents. The authors create an algorithm that learns an intrinsic reward function, that when used to train a new agent over a “lifetime” (which consists of multiple episodes), leads to the best cumulative reward over the lifetime. As a result, the learned intrinsic reward is incentivized to quickly “teach” the agent when and where to explore to find out as-yet unknown information, and then exploit that information once there is no more to be had. Experiments on gridworlds demonstrate that these learned intrinsic rewards: 1. switch between early exploration and later exploitation, 2. explore only for information that is relevant for optimal behavior, 3. capture invariant causal relationships, and 4. can anticipate and adapt to changes in the extrinsic reward within a lifetime.

I very much appreciated the design of the environments to test for specific properties within the learning algorithm: I think these experiments provide a very useful conceptual analysis of what learned intrinsic rewards can do.

My main qualm with the paper is with its significance -- the authors claim that the goal is to find out whether reward functions can be loci of knowledge, but we already know the answer is yes: the whole point of reward shaping is to improve training dynamics by building in knowledge into the reward function. It is not a surprise that learned reward functions can be loci of knowledge if our hand-designed reward functions already do so.

To me, the more interesting aspect of this paper is how much benefit we can get by learning intrinsic reward functions, relative to other ways of improving training dynamics. The authors do show that by allowing the intrinsic reward to be recurrent (and so dependent on past episodes), it is able to first incentivize exploration and later exploitation, which standard reward shaping cannot do (since usually reward shaping still maintains the assumption that the reward is a function of the state). However, given this motivation, it would be important to see comparisons between the proposed method of learning intrinsic rewards, and other methods for fast adaptation in the literature, such as MAML, which as I understand also has many of the properties highlighted in this paper.

Ideally there would also be experiments on more complex environments: the environments in the paper have 104, 25, and 49 states. If we in the ABC environments if you count “whether or not reward(object) is known” as part of the state, that multiplies it by 2^3 = 8 giving 200 and 392 states, if you then further add the ordering of r(A), r(B), and r(C), that multiplies by a factor of 3! = 6 giving 1200 and 2352 states. These environments are excellent for demonstrating the properties of learned intrinsic rewards and I am glad the authors have done these experiments and analyzed the results. However, given that the paper aims to scale the optimal reward problem, it would have been useful to see examples where the state space cannot be fully enumerated to evaluate scalability.

Questions:

In Figure 5, in episode 1, why is the learned intrinsic reward heavily penalizing the path to C, but not penalizing the path to B? In the initial episode, the intrinsic reward should only know that B is to be avoided; it doesn’t yet know whether A or C is the better object.  I would expect the learned intrinsic reward to put similar positive rewards on the path to C and the path to A, and negative reward on the path to B. (It is slightly more likely that C is the best object. This probably changes things slightly, but not significantly.)

Also in Figure 5, by episode 3, shouldn’t the final states (A or C) have intrinsic rewards of larger magnitude? Otherwise the agent can go back and forth on the path to collect lots of intrinsic reward without terminating the episode, even though this wouldn’t get extrinsic reward.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
The paper proposes a meta-learning approach to learn reward functions for reinforcement learning agents. It defines an algorithm to optimize an intrinsic reward function for a distribution of tasks in order to maximise the agent’s lifetime rewards. The properties of this reward function and meta-learning algorithm  are investigated through a number of proof-of-concept experiments. 

The meta-learning algorithm and the corresponding empirical investigation are the main contributions of the paper. The algorithm seems to be similar to previous meta-learning approaches, but differs by introducing a lifetime value function. While I thought the paper raises some interesting possibilities, I am currently leaning towards rejection. The proposed algorithm does not seem like a major innovation over cited previous work. The empirical evaluation provides a number of proof-of-concept ideas, but no in depth investigation of the properties of the approach. The theoretical properties of the approach are barely discussed.

Detailed remarks:

* The main addition to the meta-learning algorithm is the lifetime value function. The authors mention multiple times that this is crucial to learning, but the properties of this value function are not really investigated or discussed in depth:

- The authors mention that the value function must take into account changing future policies, but do not discuss this further. The value function update seems to be a standard on-policy TD update with the lifetime return and the complete history as input. The policy for this value function, however, is still a standard policy with only state as input (but it will be non-stationary over the agent lifetime). It would be good to discuss this learning problem in more detail.
- The algorithm uses an n-step return. Is this important? What effect does n have on learning?

* Another issue which I would have liked being discussed in more detail is the non-stationarity of the learning problem in general. Most of the approaches discussed in related work (e.g. shaping)  are aimed at learning/designing more informative reward functions. These reward functions still fit in the MDP framework, however, and map from states and actions to rewards. In the case of shaping approaches guarantees can be given that this does not alter the learning problem. The intrinsic reward functions used in this paper map the full life-time history of the agent to rewards. While this is a richer framework that can express more complicated tasks (like exploration over multiple episodes), it also invalidates many of the basic assumptions of reinforcement learning. The rewards are now no longer Markovian when only observing the current state. Moreover, the reward function will change over time. To what extent does this require non-stationary / history-based policy and value function learning to solve these issues? While some of these issues also apply to count based exploration strategies, (Strehl and Littman,2008 ) provided results that the  exploration bonuses result a Bellman Equation that accounts for uncertainties. No real guarantees seem to exist here.

* The empirical contribution focuses on trying to answer a number of questions regarding the properties of the learnt intrinsic rewards. I found these questions to be very broad, while the answers are mostly anecdotal evidence through proof-of-concept examples.  These examples do show potential benefits of meta-learning intrinsic rewards, but I was somewhat disappointed that there was no more systematic investigation. For example, questions like ‘how does the distribution of tasks affect intrinsic rewards’ or ‘does intrinsic reward generalise’ are not really answered by providing metrics of performance or generalisation in controlled experiments, but by providing some example cases. Several of these questions (including optimising exploration and dealing with non-stationarity) also seem to have been investigated to some extent in the original Optimal reward papers (Singh, 2009/2010). It would be good to clearly indicate what we have learned beyond these previous results. 

* There seems to be a bit of a mismatch between the learning objective for intrinsic rewards in the optimal reward framework and the results shown in the experiments. The learning objective aims to optimise lifetime rewards for a distribution of tasks. Most of the experiments seem to analyse episodic reward performance and compare against single-task (or task agnostic) methods.

Minor comments:

- The architecture / parameterization of the lifetime value function does not seem to be defined anywhere. Given that it takes histories as input I assume this is another RNN?
- There seems to be some small overloading in the notation with \eta occasionally being used to denote the parameters of the reward function r_eta or the reward function itself.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary 
The paper evaluates the intrinsic reward as a way of storing information about episodes. It adopts the optimal intrinsic reward setting (Singh'09), and extends its recent policy gradient implementation, LIRPG, to lifetime settings. The task in the lifetime setting is to learn an intrinsic reward such that when trained with it, the agent maximizes its total return over its lifetime. A lifetime is defined as a sequence of episodes, where the agent does not have memory of previous episodes, however, the function computing the intrinsic reward does. In proof-of-concept experiments, the paper demonstrates that the learned intrinsic reward captures properties of several gridworld environments and induces meaningful behavior in the agent, successfully transferring information from previous episodes. Interestingly, a state-based reward function also generalizes to agents with perturbed action spaces, showing that this way of storing information is agnostic to the agent’s action space.

Decision
The paper proposal is interesting and adequately evaluated, however, the impact of the paper might be limited by its limited technical novelty and lack of comparisons to strong baselines. I recommend marginal accept. 

Pros
- The paper is well-motivated.
- The paper is well-written and the method is clearly explained. The literature review is thorough.
- The experimental evaluation demonstrates several interesting and potentially promising phenomena.

Cons
- The novelty of the paper is limited as it is a somewhat straightforward extension of prior work.
- The impact of the paper is hard to judge as the experimental evaluation does not focus on potential usecases. 

Questions. Here, I will focus on scientific questions, answering which would significantly improve the quality of the paper.
- The biggest drawback of the paper is that the proposed method has an unfair advantage as it has a way of transmitting information across episodes, which the baselines do not (as stated on the bottom of page 5). While the findings of this paper are interesting, it is unclear how it compares to methods that have memory of previous episodes, such as agents with non-episodic recurrent policies, or meta-learning agents such as Duan’16, Finn’17. Is it possible that the proposed method e.g. scales better than recurrent policies due to compact representations or provides better generalization to things like action space changes?
- How does the method compare to hand-designed intrinsic rewards on hard exploration games (such as montezuma’s revenge or pitfall Atari games)? Since it can only learn to explore on games that it previously successfully solved, it is possible that a hand-designed intrinsic reward such as RND (Burda’19) would perform better on these hard games. On the other hand, it is possible that the method will in fact perform better on these games due to more directed exploration.
- How does the method compare to hand-designed intrinsic reward on out-of-distribution tasks? Intuitively, the method should perform the worse the further from the training distribution the task is, while the hand-designed rewards will always perform similarly. However, what is the extent to which the proposed method generalizes? It is possible that this method would be very useful in practice if it generalized well.

Other potentially related work.
- Xu’18, Learning to Explore with Meta-Policy Gradient, is a relevant work that proposes a meta-learning framework for training an exploration policy.
- Metz’19, Meta-Learning Update Rules for Unsupervised Representation Learning, is a conceptually relevant work that proposes to meta-learn loss functions for unsupervised learning (and there is more recent related work on this topic too).

review_title: Official Blind Review #2

Meta-review: The authors present a metalearning-based approach to learning intrinsic rewards that improve RL performance across distributions of problems.  This is essentially a more computationally efficient approach to approaches suggested by Singh (2009/10).  The reviewers agreed that the core idea was good, if a bit incremental, but were also concerned about the similarity to the Singh et al. work, the simplicity of the toy domains tested, and comparison to relevant methods.  The reviewers felt that the authors addressed their main concerns and significantly improved the paper; however the similarity to Singh et al. remains, and thus the concerns about incrementalism.   Thus, I recommend this paper for rejection at this time.
Decision: Reject