Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper deals with the problem of few-shot meta learning, i.e. where one explicitly optimizes for the performance of doing one (or more) steps of SGD on a new task. This was intially introduced through the MAML method. However, the authors argue that MAML only discusses gradient descent while in practise, approximations lead to doing stochastic gradient desecent instead, thus making the original formulation of the problem biased. To that end, this paper proposes an SGD based meta learning objective, which ensures simpler convergence analysis as well as better performance in two benchmark problems from the RL setting. 
main_review: Originality: The motivation of the work is solid in my opinion. The main question the authors ask is indeed important and the authors do a good job at highlighting it. 

Quality: The overall quality of the work is quite high. The explanations of the SG-MRL theory are well detailed and intuitive.

Clarity: The paper reads quite clearly overall. The flow of content is pretty nice as well.

Significance: I believe the paper contributes in both fronts, theoretically and empirically. In the theory part, it sheds important insight on the issues with prior work and how those can be fixed, in the case when we do not have access to exact gradients. In the practical part, the results are quite strong and show that a simple fix can provide better gains. It would be even better in my opinion if there were some ablations on how much the gradient is biased by (across training) and how/if it matches with the reward curves. 

Questions/Clarifications:

- Going through the MAML paper, it seems like the authors proposed an objective that can be optimized by auto-diff end-to-end. Therefore, they did not use the inner/outer loops explicitly to explain the application of gradients in their work. Is it accurate to say that SG-MRL can also be used with auto-diff directly and that the inner/outer loop computations happen the same way as described in the paper for both SG-MRL and MAML. Or is it that because MAML induces a biased gradient, it can work with auto-diff but SG-MRL cannot, since it requires computing independent averages at both the inner and outer loops?

- The empirical results look really convincing, in that computing an unbiased estimate of the stochastic gradient can have considerable difference in performance. However, the overall story of the paper leans heavily towards the theoretical contributions. Is there any specific reason why? Is the implementation more complicated than MAML in any sense? 
limitations_and_societal_impact: Suggestions: My sole primary suggestion would be to test this method in the supervised learning setting as well, just as in the MAML paper, so as to really drive the point home that correcting for the bias can have considerable empirical gains. This will again highlight that the paper does not only contribute theoretically. 

I do not think there any explicit negative societal impacts of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the Model-Agnostic Meta-Reinforcement-Learning (MAMRL) from a theoretical perspective. In particular, it proposes a modified MAMRL objective along with its unbiased gradient estimator, establishing a new Meta-RL algorithm called SR-MRL. It theoreticallly proves the convergence of SR-MRL and experimentally shows its comparable performance to the original MAML algorithm.
main_review: Overall, alghouth the techniques used in this paper do not seem to be very novel, I consider its results as a significant progress of Meta-RL because it is the first provably convergent Meta-RL algorithm, which meanwhile also achieves comparable performance to the MAML algorithm. The whole paper is also very well-written with clear logic flow.

One concern I have is about the assumptions this paper use. It seems that the convergence does not rely on any assumptions about underlying MDPs at all, which looks a little bit wired to me. Did I miss anything?

Questions:
- Is the exponential dependency of $V_{\zeta}$'s smoothness parameter in $\zeta$ fundamental? Is there any intuitive explanation on why its dependency is exponential?
- For the Remark 2, is there any benefits of increasing $\zeta$ from a theoretical perspective?
- Does the convergence result rely on any assumptions about the transition kernel of $\mathcal{M}_i$ or it can be completely arbitrary?

Minor suggestions on writing:
- At line 301 on page 8, *Theorem 1* -> *Proposition 1*

---

Score increased after rebuttal.
limitations_and_societal_impact: The authors adequately addressed the limitations of their work in Section 6 by comments on their assumptions. The potential negative social impact is not addressed, which will generally be all potential malicious usage of any reinforcement learning algorithms.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims to tackle the non-diminishing bias in MAMRL (MAML for RL) due to the stochastic policy gradient update for subtasks. Such bias hinders the first-order optimality when solving MAML with gradient descent. The authors compute the exact gradient of the MAMRL objective and identify the additional term that arises in RL. The authors then estimate such a gradient with an unbiased approximation based on samples and propose SG-MRL that utilizes such approximation to solve MAMRL. The authors show that SG-MRL converges to first-order stationary points under the standard assumptions and further conduct experiments on SG-MRL against baselines in the MuJoCo environment.
main_review: This paper is a fairly solid work on MAML, with both theoretical insights and empirical improvements over baselines. The algorithm is simple and well-motivated, and the theory is reasonable and well-explained. Experiments demonstrate the improvement of the proposed SG-MRL against baselines, which also partially corroborates the theory. 

It would be better if the authors could provide a more comprehensive comparison of SG-MRL against MAML on more tasks. Still, this is a minor issue. The main point of this work is on the theoretical side, for which the experiments are sufficient to me.

Some inquiries:

1. Is the bias correction term only adaptable to the update proposed in (6)? As the authors pointed out, the MAML gradient in (16) is biased and would also require a similar term to the bias correction term of SG-MRL. Does the difference in the objective (between (6) and (14)) affect the formulation of the bias correction term?

2. In lines 235-244, the authors mentioned that having a biased gradient requires careful selection of stepsize for MAML, while SG-MRL does not suffer from such stepsize selection. Could the authors provide more details regarding this point?

3. The theory involves the inner learning rate $\alpha$ as a smoothness parameter of the objective and requires $\alpha$ to be small for convergence. Can the theory say something about the effect of $\alpha$ on convergence and the overall performance of SG-MRL? 
limitations_and_societal_impact: See inquiries in the Main review. I do not have concerns regarding the societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a new debiased model-agnostic meta-reinforcement learning algorithm. The main improvement is that the paper considers a more accurate setting where one does not have accurate estimate on the gradient for each task, which is assumed by previous works. One has to estimate the gradient from samples in each task, and the samples themselves depend on the global parameter $\theta$, there will be an extra term for the global gradient.  The authors hence write the exact gradient and show the convergence. Numerical results show good performance.
main_review: The paper makes a solid improvement over the previous methods by considering a more accurate setting, i.e. the gradient of each task is estimated from batch data. By considering the more accurate setting, they are also able to achieve a better numerical performance.

The paper is well-written and easy to follow.
limitations_and_societal_impact: Despite the improvement shown in the numerical experiments, it would be sound if the authors can theoretically show the advantage of the proposed method versus the methods in the literature. To be more specific, how much does assuming the knowledge of the second-order information of the expected reward hurt the actual performance, when the information is not available. Will it cause bias or increased variance?
review_title: null

Meta-review: All reviews find that this paper is a solid contribution making a reasonable contribution to the empirical as well as the theoretical state-of-the-art of MAML, so that I recommend to accept it. Please take into account the points raised in the reviews when preparing the final version.
Decision: Accept (Poster)