Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes to meta learn the objective function of a policy gradient algorithm using second order gradients of the objective function w.r.t the state-action value Q. 

This is an interesting approach, however, I think the experimental evidence is not sufficiently convincing. 

- In particular, I think the most important baseline that is compared against is not RL2, but DDPG: RL2 is not designed to generalize but to learn quickly on new tasks from the training-task distribution. Because the proposed algorithm does not depend on the observed states, it generalizes much better, but is also much slower than RL2. On the other hand, it shares a lot of design choices with DDPG: Using TD3 and Double Q-learning, as well as using as objective function the Q-values. 
Looking at Figures 2, it is not clear that the proposed algorithm is substantially better than DDPG. 
- Cheetah, Hopper and Lunar Lander are very simple environments. Evaluation on (slightly) larger scale environments would show that the algorithm can scale. 
- The authors claim that the algorithm allows sharing of exploration strategies, which I don't believe can be the case based on it's current design.
- Lastly, I have a question about Fig 5a vs. Fig 2b: Shouldn't the performance of MetaGenRL be the same in both? It appears to perfom much better in Figure 2b.

Minor remark/question (didn't influence score):
The authors claim in the very first paragraph (and in the 4th paragraph) that inductive biases in humans are learned by natural evoluation through "distilling the collective learning experiences of many learners" by "learning from learning experiences". I'm not familiar with the relevant literature, but this seems like a strong statement which I believe should be supported by a citation. 

Edit because I can't make my response visible to authors anymore:
Thank you for your response to my review and apologies for my delayed answer.

After reading your responses I agree that PPO is a fairer comparison than DDPG and that you are outperforming PPO is promising.
I further agree that it is relevant to show that RL2 overfits (although I personally don't find that very surprising - see below). 

However, I still don't think that RL2 is a relevant baseline for this approach. There's a fundamental trade-off between the speed of adaptation and the amount of overfitting. 
If I want to adapt very quickly (like RL2 does), I need to leverage as much task-information as possible, thereby overfitting to the task-distribution. 
On the other hand, MetaGenRL is slow, it has training speed comparable with gradient-based approaches (by generalizes better by construction because it e.g. doesn't receive states as inputs). 
Consequently, because MetaGenRL doesn't offer any speed improvements over gradient-based approaches, it should be compared to them, and not RL2.

Taken both the positive results vs. PPO and the negative results vs. DDPG, together with the fact that there's no learning speed advantage for MetaGenRL, I would see it as an interesting, and promising, research direction, but so far without proof that it can advance state of the art, as it looses to RL2 in terms of speed and DDPG in terms of final performance.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper presents a novel meta reinforcement learning algorithm capable of meta-generalizing to unseen tasks. They make use of a learned objective function used in combination with DDPG style update. Results are presented on different combinations of meta-training and meta-testing on lunar, half cheetah, and hopper environments with a focus on meta-generalization to vastly different environments.
Motivation:
The work is well motivated and is tackling an important problem. There are a number of design decisions presented and only some are validated experimentally. Given the complexity of many existing meta-rl methods this seems fine but could obviously be improved upon either with more empirical work or with some guiding theory.
Experiments:
Overall the experiments are not convincing to me. Given that this is the majority of your paper is empirically based this is my main criticism. More detailed comments follow.
Figure 2a concerns me that just meta-training on lunar performs worse than ddpg (what your algorithm is based on). This suggests that this added complexity is not aiding in capacity and or hurting training. Can you comment on this? This result also casts doubt onto figure 2b, which, in isolation seems like an extremely promising example of meta-generalization. This makes me fear there is something indirect and not interesting occurring (e.g. the learned loss modifies with the DDPG algorithm which happens to increase noise in generated samples which improve performance only on some environments and hurts in others for example.)
Table 1 should include hand designed algorithms imo. Given how weak EPG is (as you stated for number of frames) and how RL^2 will never generalize across these different tasks it's hard to get a sense of the numbers. Your appendix does include a figure like this which shows ddpg performs quite well. Additionally, I don't understand why meta-training on lunar and transferring to hopper does better than meta-training on hopper (table 1, middle column). Can you comment on this?
While figure 3 is cool, I would appreciate if it put the meta-test performance on the same graph as meta-train performance. From eyeballing the curves it looks like it decreases at 100k iterations then finally increases again at 200k. This is strange. This also seems fraught from an empirical comparison point of view. How do you select when to test these algorithms? Ideally you would have a meta-validation set of tasks then only meta-test on the selected task but I see no mention of this.
Key details such as meta-training are also not discussed in depth nor ablated. From the details and curriculum scheme presented in the appendix this seems like quite a feat. Further study of these factors could be useful.
Hyperparameters of your baseline do not appear to be tuned (taken from appendix) where as for your method has a number of choices. How are you tuning these choices? Once again a meta-validation set would be the principled thing to tune against.
Finally, the experimental setup presented here is quite complicated. There are a ton of factors at play -- exploration, meta-generalization, meta-training, inner-training, instability of ddpg, so on. These all complicate the resulting picture. Having some simplified / more controlled setup to demonstrate these pieces would be greatly appreciated.
Other Suggestions:
Section 3 generalization: I think you mean meta-generalization.
Please include what error bars are for all plots.
 
Rating:
I am borderline leaning towards reject on this paper. I enjoyed reading this work and found the ideas interesting but the empirical comparisons are confusing and not convincing. I hope the authors continue to work to improve this! 

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper proposes a meta reinforcement learning algorithm called MetaGenRL, which meta-learns learning rules to generalize to different environments. The paper poses an important observation where learning rules in reinforcement learning to train the agents are results of human engineering and design, instead, the paper demonstrates how to use second-order gradients to learn learning rules to train agents. Learning learning rules in general has been proposed and this paper is another attempt to further generalize what could be learned in the learning rules. The idea is verified on three Mujoco domains, where the neural objective function is learned from one / two domains, then deployed to a new unseen domain. The experiments show that the learned neural objective can generalize to new environments which are different from the meta-training environments. 

Overall, the paper is a novel paper and with clear motivation, I like the paper a lot! Hope that the authors could address the following concerns and make the paper even better:

1. The current experiment setup is a great proof-of-concept, however it seems a bit limited to support the claims in the paper. The meta-training has only at most two environments and the generalization of the neural objective function is only performed at one environment. It would be great if the authors could show more results with more meta-training environments (say, 10 meta-training environments) and more meta-testing environments (the current setup is only with one);

2. The paper states a hypothesis that LSTM as a general function approximator, it is in principle able to learn variance and bias reduction techniques. However, in practice, due to learning dynamics and many other factors, it's not necessary true, i.e., how many samples are required for an LSTM to learn such technique is unclear. At the same time, at Page 8, Section "Dependence on V" actually acts as an example of LSTM couldn't figure out an effective variance-reduction method during the short meta-training time. The authors may want to put more words around the learnability of variance-bias trade-off techniques.

Notation issues which could be further improved:
1.  Page 2, "Notation" section and all of the following time indexing. Note that in Equation (1), r(s_1, a_t) has discount gamma^1, which is not true, I'd recommend the authors to follow the time indexing starting from 0, so that the Equation (1) is correct. (Alternatively, the authors could change from gamma^t into gamma^{t-1});
2. Section "Human Engineered Gradient Estimators" is missing the formal introduction of the notation \tau;
3. Overall, the authors seem to use \Phi and \theta interchangeably, it's better to use a unified notation across the paper;
4. In the paper, the authors choose \alpha to represent the neural net for learning the objective function, to make it clearer for the readers, the authors could consider to change \alpha into \eta, because \alpha is often considered as learning rate notation;
5. I'd suggest the authors to rewrite the paragraph in Page 3 "MetaGrenRL builds on this idea of ....,  using L_\alpha on the estimated return". This describes a key step in the algorithm while at the moment it's not very clear to the readers what's going on there;
6. Section 3.1 is missing a step to go from Q into V;
7. The authors could consider to describe the details of the algorithms in a more general actor-critic form, instead of starting from DDPG formulation. It would make the methods more general applicable (for example, extension to discrete action space).  

 
review_title: Official Blind Review #1

Meta-review: This paper proposes a meta-RL algorithm that learns an objective function whose gradients can be used to efficiently train a learner on entirely new tasks from those seen during meta-training. Building off-policy gradient-based meta-RL methods is challenging, and had not been previously demonstrated. Further, the demonstrated generalization capabilities are a substantial improvement in capabilities over prior meta-learning methods. There are a couple related works that are quite relevant (and somewhat similar in methodology) and overlooked -- see [1,2]. Further, we strongly encourage the authors to run the method on multiple meta-training environments and to report results with more seeds, as promised. The contributions are significant and should be seen by the ICLR community. Hence, I recommend an oral presentation.

[1] Yu et al. One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
[2] Sung et al. Meta-critic networks
Decision: Accept (Spotlight)