Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a theoretical analysis of two sources of bias in the gradient estimations of meta-reinforcement learning methods that perform a two level optimization of an outer meta-learner and an inner gradient-based RL learner. The authors derive an upper bound to the first source of bias, named compositional bias as well as an analysis of the bias of the estimated Hessians coming from the use of auto-diff systems. Finally, the paper presents an empirical evaluation in tabular MDPs that supports the theoretical findings.
strengths_and_weaknesses: =========Update after Rebuttle====
My main concern on the validity of the 2 sources of bias has now been addressed and I agree that the bias terms exists and are strictly higher than 0. For this I will gladly raise my score from 3 to 6.

The paper proposes a general study on gradient estimation meta-reinforcement learning.
While the setting itself is interesting to the community I have some major concerns regarding the theoretical results which are listed as follow.
A) On the gradient expression. (Proposition 3.1)
I have some concerns regarding the correctness of the proof.
In particular, I would ask the authors to derive step by step  Equation 18 as according to my computation, the total derivative has been simplified with the gradient, which, in general is not correct.
B) On the compositional bias. (Section 4.1)
To the best of my understanding, the authors did not show that this term actually exists, instead they provide an upper bound on it (i.e., one might take a general unbiased estimator and upper bounds its bias with any positive number).
The second concern regarding the compositional bias is the fact that, as Lemma 4.4 shows, the compositional bias is upper bounded by the empirical variance divided by the number of trajectories taken to estimate the inner loop gradients.
This term tends to zero as the number of trajectories tends to infinity. Similar results holds (in high probability) for standard policy gradient setting (see for instance "Adaptive Batch Size for Safe Policy Gradients" (Papini 2017) or "Smoothing Policies and Safe Policy Gradients" (Papini 2019)): the error (not bias) in the gradient comes from a term that depends on the variance and the number of samples used (this can also be seen as a consequence of the Chebyshev inequality). This seems very similar to what the authors claim. In this sense, I am a bit confused on the value of Lemma 4.4 (which has a direct impact on the main theoretical result). More specifically, it seems to me that rather than a bias we are just considering an error in the gradient estimation (that, of course, occurs with finite samples).

C) On the multi-step hessian bias. (Section 4.2)
It is not clear to me where this bias comes from and seems simply to be a bug of differentation software. The discussion of Section 4.2 is  unclear to me.

D) On the trade-off (Section 4.3)
Generally speaking, Theorem 4.5 seems to be auto-referential.
The contribution of multi-step hessian bias is direct: it just states that this error exists and it is accounted in the computation.
The same holds for the compositional bias.
Regarding the variance, instead, the results seems incredibly loose since it does not tend to 0 (V1 * (K-1)) (more than a worst case solution).
Moreover, generally speaking, Theorem 4.5 shows very little insights and its contribution seems to be minor w.r.t. Lemma 4.4.
questions: Given the previous doubts on the compositional, I would ask the authors to:
1) Prove that the compositional bias is actually strictly greater than 0.
2) Comment on my second concern, related to the fact that this in not actually a bias, but a finite-sample error that goes to zero with more samples
3) How does the correlation plotted in Figure 1 relate to the compositional bias identifies. It seems to me again, that the plotted measure is related to MSE instead of the bias, reinforcing my doubts on the analysis performed.
limitations: The authors have discussed their limitations
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Gradient-based meta-learning (GMRL) has been a topic of interest in the NeurIPS community recently. The authors derive formal bounds on the bias and variance of the estimated meta-gradient in commonly-used algorithms for gradient-based meta-reinforcement learning, identifying two sources of meta-gradient bias in particular: "compositional bias" (accumulated across inner loop steps estimated with small batch sizes) and "multi-step Hessian bias", caused by using approximate adaptation in the inner loop. Using these observations, the authors applying existing techniques for variance reduction to new GMRL settings, showing that they can reduce bias and improve GMRL performance in some settings.
strengths_and_weaknesses: Strengths:
- Analysis of a relevant topic of interest to the NeurIPS community (gradient-based meta-RL)
- Intuitive theoretical result motivating application of existing techniques to new meta-RL settings
- Overall clear presentation

Weakness:
- Unclear what the extent of the impact of the theoretical results are
- Little methodological novelty (which could be okay)
- Experimental evaluations are okay, but the choice of methods to evaluate seems arbitrary and perhaps limited (see final question below)
questions: How specific are the conclusions in this paper to gradient-based meta-RL? To what extent are they useful for gradient-based supervised meta-learning?

Ideally, the authors might evaluate horizons that are deeper than only 5 gradient steps. Does LVC continue to provide effective mitigation at greater depth?

Most importantly, the experimental evaluations are relatively limited. It's personally difficult for me to judge the extent of the impact of the theoretical results alone, and I therefore would have found it quite interesting to consider other methods than just LVC. Could the authors justify why LVC is a particularly representative method to use for most of the evaluations?
limitations: More discussion of the limitations of the theoretical results would be welcome (e.g., how realistic are assumptions 1-3?).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the bias of gradient-based meta-RL algorithms. The authors provide a bias bound that is made up of a compositional bias and the multi-step Hessian estimation bias, which also suggests ways of understanding current algorithms. 
strengths_and_weaknesses: pros: 1. The paper is well written and easy to follow. Both the background and the analysis are thorough. 

2. The bias theorem may also guide future algorithmic designs of meta-RL algorithms, considering where the bias comes from.

3. The experiments also provide evidence supporting the theorem.

cons: 1. In experiments Fig. 2 and Fig. 3, it seems that the curves have not converged.

2. I'm not an expert in meta-RL. Are there other existing works that give bias bounds? I am only aware of the algorithmic designs mentioned in the paper, but I didn't see the related work in the theory aspect.

3. It seems that the authors only evaluate existing modifications to support the theorem instead of proposing new algorithms. Are there ways to combine some of them together to reach the best theoretical and experimental results? 
questions: Could the authors run the experiments for more iterations? I would also like to see if both sources of bias can be combined together or will they suggest new algorithmic designs. Besides, more related work on the theory side is encouraged. 
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper comprehensively investigates the gradient bias in Meta-Reinforcement Learning. The authors point out two source of bias: the compositional bias and the multi-step Hessian bias. The paper then gives a generic bound on the bias and variance of the final meta-gradient. The paper mitigates the two biases using off-policy training and LVC estimator respectively. A debased algorithm is tested on 8 Atari games.
strengths_and_weaknesses: Strengths:
I believe the paper is significant in the following ways
1. This is one of the first paper to give a generic analysis on gradient bias, which applies to most of the algorithms.
2. The GMRL framework proposed in the paper applies to a number of important meta-RL frameworks.
3. The analysis that decomposes the bias terms is novel and interesting.

Weakness:
1. I am concerned about the actual contributions of the paper. The paper provide theoretical analysis that decompose the gradient bias into several terms. However,  the experiments are only weakly dependent on the theories.

To be specific, the paper adapts off-policy training to mitigate the compositional bias. Enlarging the number of data to mitigate bias is a trivial solution opposed to actual bias correction. As the paper mentioned, off-policy training suffers the shifts in the data collection policies and importance sampling process has to be used. LVC is a method existing in the literature. Thus, the paper is not able to propose a novel algorithm that actually balances the bias and variance based on Theorem 4.5 to achieve a better performance. 

The experiments in 5.1 simply show that the performance is higher when replacing sample-based estimator with exact solution, which is quite obvious in my opinion.

The paper does not clearly state the main contribution over the literature. There are papers [33, 6] discussing bias in meta-RL. I believe [6] also considers the compositional bias. I don't understand the main contribution here except for proposing a more general framework that applies to different cases.

2. The paper is not well-organized and it can be hard to follow. Different sections are not strongly connected.



 


questions: Line 52: not clear what does the impact mean here 
Line 204 typo: an extra "the"
Line 216: It seems there are multiple bias terms in (9). Why is $\hat \Delta_J$ less important than $\Delta_C$ and $\Delta_H$? It would be better to use $\Delta_C$ instead of $E[\|\hat \theta^K - \theta^K\|]$ here as it has already been defined.
Line 218: What does the variance term imply here? Is there any algorithm in the literature that can balance bias and variance? How well do they work in the decomposition in (9)?
Line 219. The use of punctuation is bad here. What set is the maximization taken over?
Line 318. What is a DiCE estimator?

limitations: The authors did not address any potential negative societal impact. As a theoretical paper, I don't think it is necessary.
review_title: null

Meta-review: metareview: All reviewers are positive about this paper and recommend either weak (3x) or borderline (1x) accept. We had some good discussions, through which some of the reviewers (RjbJ and H2bz) decided to increase their initial scores. The reviewers consider this work a relevant topic, providing intuitive and novel theoretical results, well-written, and with the possibility of guiding future algorithmic design for meta-RL. I believe this is a good paper, and I recommend its acceptance.
confidence: Certain
Decision: Accept