Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers meta-RL with gradient-based adaptations, which relies heavily on the estimation of the Hessians of value functions. Though there are previous approaches that focus on unbiased/biased estimations of Hessian. This paper gives a unified view for estimating higher-order derivatives of value functions, through the lens of off-policy evaluation. This subsumes several previous works, also sheds some light on the bias-variance tradeoff of the estimate. Empirically, the new proposed second-order estimate is also incorporated in meta-RL algorithms to showcase the empirical efficiency. 
main_review: (Originality): The main contributions I see from this paper are: (1). a unified framework for estimating higher-order derivatives of value functions, through OPE methods. (2). the use of Taylor expansion in the value function, which controls the number of IS ratios in the value estimate, hence sheds some light on the bias-variance trade-off. (3). also a new method arising from this framework.

(Quality): The motivation is clear, and the method is clearly stated. Some comments here include:
[1]. For the bias-variance trade-off with the Taylor expansion, how to select the optimal $K$? Also, how does this bias-variance trade-off translate into the downstream meta-gradient estimations? Any comments on this?
[2]. In experiment section 5.1, the bias-variance trade-off examination seems a little bit confusing to me. It seems the argument here is the second-order method achieves a better trade-off through high correlation with the ground truth. Is there any way to empirically examine the bias and variance in this tabular setting?
[3]. For the continuous control problem, it seems the improvement from the new proposed approach is pretty marginal since the bias might be a dominating issue here. Are there any environments/MDPs/settings that we can expect a larger improvement of it? It would be great to include some discussions/new experiments on this.

(Clarity): The paper is well-written and easy to follow.

limitations_and_societal_impact: Limitations are discussed in Section 3, while it seems it does not have any societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper uses Huang and Jiang (2020) to talk about gradient-based meta learning in reinforcement learning using the language of off-policy evaluation. The logical connection behind those concepts stands from the fact that policy gradient methods in reinforcement learning are in fact instances of the likelihood ratio derivative estimator, which itself stems from important sampling. Proposition 3.2 and 3.3 are obtained from Huang and Jiang. Proposition 3.1 tries to extend their results to talk about higher order derivatives. The author's finally propose to use ideas from Taylor policy optimization from Tang et al. (2020) as a variance reduction method within the doubly robust policy gradient framework of Huang and Jiang.



main_review: In my opinion the paper doesn't go deep enough into the individual ideas being put forward. The doubly robust estimator of Huang and Jiang is at the core of this paper.  In the specific context of meta learning, what does it allow us to say or do that we couldn't achieve before? Why is doubly robust necessary? The doubly robust being a variance reduction technique why this VRT in particular and not another one, in the meta learning context? Regarding the Taylor approximation, why is it related to the doubly robust estimator? The paper does not do a convincing job at showing that these two components strongly depend on each other: conceptually, or experimentally. Overall, the paper lacks cohesion and depth.

On line 83 the notation suggests that the expectation is not taken over x and only over the task g. While technically correct, you don't really have to write the objective as an expectation over the value function. Your objective should simply be the expected discounted or on discounted return, written plainly as a sum of rewards.

In equation 2, I would suggest using a different notation when you're writing nabla of theta prime. Theta prime should be a function, but the way that it's written suggests that it's a variable instead.

Another suggestion for notation would be to write "v of theta" (instead of a superscript) in equation 2. This way it would be clearer how you have to apply the chain rule later on.

Reading more carefully on line 92 and equation 1 now I realize that you assume a fixed initial distribution starting from x0. This is confusing because on line 83 you simply write v of x and it looks like x should be a random variable: your initial state. Perhaps you should state explicitly that you assume (without loss of generality) a given initial state.

On line 92, the expression the "Hessian of the value function" is misleading because really the Hessian that you're taking is that of the expected return objective, not that of the value function.

I think that the unnumbered equation below line 106 is redundant: it simply says that the two quantities are unbiased estimators. Words suffice here. Instead, I would suggest that you explain the nature of the issue. The correlation problem that you're describing comes from taking the expectation over a product of two random variables. The fact that it's not simply the product of the individual expectations and that here is also a covariance term.

I'm confused by proposition 3.1 especially around line 137 when you introduce a new notation for the estimator nabla hat v. What you have in this proposition should be something about a derivative estimator: score function/REINFORCE, reparametrization. Now you don't seem to be defining how you obtain this nabla theta m  v hat, and in that sense proposition 3.1 does not quite stand on its own.

On line 171 you allude to the fact that the variance of policy learning is essentially due to having a large number of importance sampling terms. While this is a reasonable intuition to have, note that we can show formally that PDIS-type estimators need not have larger variance than let's say stationary or marginal IS.

limitations_and_societal_impact: The limitation of the framework per se are not quite explored. The authors talk about what can be said within that framework (dice) but not quite what it cannot say. Experimentally the paper looks at the performance of the combination of doubly robust plus Taylor approximation as an algorithm of its own. It does not quite try to understand what the doubly robust formulation allows us to address specifically in meta learning. It would be interesting to gain more insights into the value of the proposed method by taking a more ablative approach. Since the authors pitch a framework as the main contribution, performance results matter much less than experimental insights at the conceptual level.  I think that the paper could be restructured around the algorithm and to de-emphasize the framework part. To go all in on the algorithm (which might be how things happened chronologically anyways).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper uses the recently developed framework of Taylor expansions for value functions (ref [29] in the paper), expands the results, and shows how they could be applied to Off-Policy Evaluation, and by a non-trivial extension to meta-learning. The main algorthmic contribution of the paper is the application of the ideas to meta-learning, but I feel the main contribution is a conceptual one, further developing the ideas from [29] and drawing theoretical connections between OPE and meta-learning.

The paper includes an experimental section demonstrating marginal improvement over baselines in several domains.
main_review: Overview: Despite some flaws, I think this is a good paper and leaning towards acceptance. I think the two main limitations of the paper are:
(1) It draws very heavily on the theory and novelty of [29] and therefore its own novelty is a bit limited.
(2) The empirical improvements obtained for meta-learning (the problem the paper sets out to advance) are limited.
Additionally, there are some changes that should be made to make the paper clearer, but I think those can be easily fixed for the camera ready if the paper is accepted. Despite the drawbacks, the reason I support acceptance is that I feel the additional contributions on top of the already interesting results of [29] are non-trivial, and this is a potentially interesting research direction which could provide insight into many aspects of RL. For example, viewing meta-learning through the lens of Taylor expansions for value functions allows for a very clear connection and unification of derivatives estimations of value functions and OPE.

A general suggestion for the authors which I feel might improve the overall story of the paper, is to lean more heavily on the theoretical results and their ability to shed light on interesting insights such as the connection between OPE and meta-learning, and OPE to high order derivatives. At least my personal experience reading the paper was expecting significant novelty and improvement of results on meta-learning, which set me up for a disappointment. It was only on a second read that I appreciated the theoretical insights in the first half of the paper more. Additionally, I would set up the connection with [29] more - the authors do a good enough job of properly citing [29], but not enough to make it clear that the reader should be familiar with [29] to fully appreciate this paper. After emphasizing the theory and insights, I think the meta-learning algorithm and result can be presented as a cool implication (this is obviously a very subjective suggestion).

Some more specific comments:
- In lines 120-122 regarding how the approximation maintains the higher-order dependencies on \theta. I feel like I'm missing something and the sentence is failing to convey an idea it is trying to. Obviously when you make an approximation in some parameter you maintain higher order terms in the small parameter, and choose how many terms to keep. So this statement seems trivial and I'd like to know if there's something deeper which the authors are trying to convey and I'm missing.
- In section 3.2 and Proposition 3.1 - Is K a typo and should be m? (if yes, it is a consistent typo, which is why I'm confused). If it isn't a typo, I couldn't see where K is defined, and also can't follow the proof of Proportion 3.1, so this is a point I would really like clarification on.
- In section 5.1, the authors claim to investigate the bias/variance tradeoff of their method, but only show results for the overall MSE of estimating the derivative. I think the authors should show explicit plots of the decomposed variance and bias of each estimator shown.
- very minor - the y-labels in Figure 2 a-b are cut.
limitations_and_societal_impact: The authors do a good job of discussing the limitations of their method compared to alternative methods.

The paper is mostly theoretical and I don't see a potential negative soceital impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper is concerned with meta reinforcement learning. The author(s) introduced a framework for learning high-order derivatives of value functions using off-policy evaluation. Specifically, the author(s) proposed to adopt existing off-policy evaluation methods to derive the value estimator and then take the derivative with respect to the target parameters to construct the estimate. This technique is further adopted to implement meta reinforcement learning. 
main_review: I have a few major comments regarding the proposed procedure.

1. The author(s) proposed to adopt importance sampling and doubly-robust estimators to evaluate the value function. These estimators suffer from the curse of horizon, leading to the sub-optimality of the learned optimal policy. Given that the author(s) considered an MDP setting, there are more efficient estimators for value evaluation (see, e.g., https://arxiv.org/abs/1810.12429; https://arxiv.org/abs/1909.05850).

2. Instead of constructing estimators for the value, it might be more efficient to directly evaluate the gradient of the value and its Hessian matrix. In the single agent setting, there are efficient off-policy policy gradient estimators developed in the literature (see, e.g., https://arxiv.org/abs/2002.04014).

3. I think the author(s) focused on offline meta reinforcement learning problems. It remains unclear to me how the proposed method would work when the positivity assumption (e.g., the policy ratio shall be bounded away from infinity) in the offline dataset is violated. In the single agent setting, the pessimism principle is commonly adopted to mitigate this issue. Would it be useful to combine the pessimism principle with the proposed method?
limitations_and_societal_impact: The authors have discussed the limitations of their work on Page 6.
review_title: null

Meta-review: The expert reviewers for the most part appreciated the paper and were mostly positive. One issue that arose in discussion is that the intent and contribution of the paper is not sufficiently clearly stated, leading to some potential misunderstandings; the authors should very carefully use the feedback of all reviewers, both positive and negative, to more clearly frame the idea of bridging of meta-learning gradients and OPE. An addition, given the significant limitations of sequential IS (including as part of DR), the authors should offer a clear discussion of more efficient methods for OPE and gradient estimation such as pointed out by the reviewers and what might the implications of these potentially be in the authors' setting; this is an important discussion even if fully studying these implications in depth may be beyond the scope of the paper. The authors should similarly address the other points raised in the reviews and implement fixes suggested in their response.
Decision: Accept (Poster)