Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on designing PAC meta RL algorithms. While prior works took a model-free approach of learning regularized history conditioned policies, this work first estimates the underlying task distribution from the training task and then trains a meta RL algorithm on the learned task distribution. The resulting approach obtains state-of-the-art PAC bounds and shows that the approach can be used to improve practical meta RL algorithms (such as VariBAD).
strengths_and_weaknesses: **Strengths:** 
  1. The paper uses KDE techniques to improve the PAC bounds
  2. The resulting approach is simple which can be used to improve existing meta RL algorithms such as VariBAD

**Weaknesses:**
  1. While the approach obtains better PAC bounds, my concern is that KDE estimation will only work on tasks which have some low-dimensional embedding (such as the reaching task). I don’t think this approach of estimating task distribution and then learning from the task distribution will scale up to more complex settings (say, meta-world ML10 benchmark, Yu et al., 2021, https://arxiv.org/pdf/1910.10897.pdf).

Overall, I do think that the strength outweighs the weakness of the paper. However, I should say that PAC bounds for meta RL isn’t exactly my area of expertise and hence my opinion should be taken with a grain of salt.

questions: N/A
limitations: The paper doesn't explicitly discuss its limitations
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper can be separated into two quite different sections. 

The theoretical part derives complexity bounds for an algorithm that would have access to the MDP’s parameters (noted theta), that would run a PCA in order to reduce the dimension of that parameter’s space, and learn the density as modeled by an independent gaussian distribution (presented as an example of KDE). 
Then, the Bayes-optimal policy can be computed from this learned distribution under better PAC criteria than by using the naive density approximation in the high-dimensional space.

The experimental part considers data augmentation of the training set of MDPs by using the latent space learned by VariBAD. One can link this approach to the theoretical part by comparing the space reduction performed by the VAE of VariBAD with a PCA, 
where the high-dimensional parameter is the history itself (through enough episodes to reduce uncertainty), and the low-dimensional projection is the latent state at the end of an episode.

strengths_and_weaknesses: The theoretical part looks correctly driven, and brings up an interesting proven result: it is worth reducing the dimension to concentrate on the low-dimensional manifold that structures the set of MDPs. 

However, the experimental part is rather disappointing, as it does not directly verify the first part’s claims, and treats a very different setting (where theta is not observed) with quite a different algorithm, despite the fact that this separated part reveals another interesting result: one can use the VAE’s latent space to augment the data and improve generalisation with less examples. Also, this part is missing higher dimensional MuJoCo environments in order to fairly compare with VariBAD.

In my opinion, this paper could even be divided into two papers that would be worth separated publication: 

1) the first including this theoretical part with a good experimental study: where theta is observed, using both discrete and continuous environments (including the half-circle with function approximation), 
And comparing to naive baselines (ppo/sac with concatenated state + theta observation, KDE without PCA dimension reduction)

2) the second including the experimental part with a proper method / algorithm description, and adding larger envs (mujoco). Eventually, a theoretical explanation that would rather look at the improvement as the result of a data augmentation.

questions: L 263: why projecting back? It should be sufficient and even easier to train an agent using the reduced representation of priors. 
limitations: Not applicable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, PAC bounds for meta reinforcement learning with finite training tasks are derived. The derived bounds is applicable to both continuous and discrete state and action spaces. The derived bounds have exponential dependence on the dimensionality of the task space. To address this, the analysis is extended to the case with dimensionality reduction using principal component analysis. To support the theoretical results, the technique to improve the existing meta-RL method was evaluated in a simple environment.
strengths_and_weaknesses: Strength:
- Novel and practical PAC bounds for meta-RL are derived
- Unlike the bound proposed by Tamar et al. (2021), the derived bound can be applied to both continuous and discrete state action spaces
- In addition to theoretical results, experimental results are reported

Weakness
- The work is largely based on the prior work by Tamar et al. (2021), and one might think that the contribution is incremental
- The experiments seem simple, while I think that the theoretical results are strong enough

questions: - In practice, how should we determine the optimal bandwidth of the Gaussian kernel for KDE? How did the authors determined the optimal bandwidth for the experiment?
- I did not clearly understand how practical the problem setting of meta RL with finite training tasks is. I would appreciate it if the authors provide examples in the real world where meta RL with finite training tasks should be considered.
limitations: I think that the authors clearly described the limitation of their theoretical analysis. For example, it is clearly described that analysis regarding the dimensionality reduction is limited to PCA, which is a linear method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes an imaginary task generation method for meta-RL tasks with a limited number of training tasks using the prior estimated by kernel density estimation (KDE). The authors provide a theoretical bound for the regret of the estimation and empirically evaluate the method in conjunction with variBAD [1] on the HalfCircle environment. 

[1] Zintgraf et al., Varibad: A very good method for bayes-adaptive deep RL via meta-learning.
strengths_and_weaknesses: ***Strengths***

Overall, I think the strength of this study lies in its novelty and theoretical analysis.

1. Novelty

There are some previous works that generate imaginary tasks using the latent model as LDM [2]. However, this is the first work that directly estimates the density using KDE and uses the density to sample new tasks. This novel idea sounds reasonable and effective for parametric meta-RL tasks with finite tasks.

2. Solid theoretical bound

I appreciate the authors for the solid derivations for the regret bounds in section 2.2. Although the conclusion that the bound is proportional to the dimension of the parameter space is somewhat natural, it has a great meaning that it was shown through a solid proof. Especially the use of PCA to reduce dimension is well supported by this theory.

***Weaknesses***

On the other hand, I think there are weaknesses in practical application and experimentation. I think the contribution of this work can be more significant when the empirical evaluation and analysis are conducted as follows. 

1. Ambiguous link between theory and algorithm

I felt the transition from section 4 to section 6 to be somewhat abrupt. Maybe providing a pseudo-code of variBAD+KDE would prevent confusion. It is unclear whether the proposed method would work on the latent space generated by the encoder of VAE, which may deviate from the assumptions made. 

I know that the VAE of variBAD [1] also does some implicit density estimation based on multivariate Gaussian distribution initialized to standard normal distribution. Due to the nature of the VAE, the latent space generated by variBAD may already be regular (continuous and complete). Maybe applying the KDE on top of this latent space makes the latent space more regular?  One possibile way to show this is an empirical comparison on the latent space for unseen tasks as the t-SNE plot in [2], Figure 9. 

Also, I'm curious how regular and general the generated tasks will be given unseen context or states without regularizations as a dropout in [2]. I appreciate the visualization in Figure 4, but I'm not sure if the reward decoder would generate high rewards for states that were not rewarding during training for any training tasks. Maybe 20~30 tasks are enough to contain all states in a light blue circle in Figure 1?

As the authors claim, it could have been because of the special geometry of the HalfCircle's task distribution. Therefore I consider testing only one environment as a weakness of this work.

2. Limited experiment domains

The HalfCircle environment in this work is a very meaningful and intuitive domain. However, it is limited to a one-dimensional task space, therefore it is hard to conclude that the proposed method would outperform generally. Maybe the proposed method works well by overfitting the policy for the learned task prior. To emphasize the contribution of this work for finite training tasks, I strongly recommend evaluating at least tasks evaluated by variBAD [1] or out-of-distribution tasks evaluated by LDM [2].


[1] Zintgraf et al., Varibad: A very good method for bayes-adaptive deep RL via meta-learning.

[2] Lee et al., Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture.
questions: 1. Is there any additional methods the authors tried to improve the generalization of the generated tasks? Because it seems that the policy may overfit the KDE-estimated prior. E.g., generating contexts with perturbation on the prior. 

2. It would be great if the authors provide a table of all notations used in this paper in Appendix, as some of the notations are confusing and similar. 

3. It would be great if the authors review the reference and update them to the recent conference proceedings. 
e.g., [18] arXiv->ICLR2019, [19] CorR->NeurIPS 2021, [20] arXiv->NeurIPS 2020...


limitations: This work is based on in-distribution parametric meta-RL with finite training tasks.
Although it is not the focus of this work in the current state, it is hard to generalize to non-parametric, non-stationary, and out-of-distribution meta-RL tasks [3].

[3] Yu et al., Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
review_title: null

Meta-review: metareview: The paper provides a theoretical PAC analysis of meta-learning when the task distribution is directly learned. Under the assumption that the task distribution can be approximated in a low-dimensional space, this approach leads to improved bounds wrt to previous literature. The authors further illustrate how the lessons learned in the theory can be translated into a practical algorithm by integrating their kernel density estimation into the VariBAD algorithm.

There is general consensus that the paper makes several interesting contributions. First the authors devise a new algorithmic approach for meta-learning and derive PAC bounds that clearly improve over current results. The results are rigorous and their interpretation is insightful. Furthermore, the authors made a considerable effort in translating their method into an actual algorithm with a non-trivial empirical validation. While more work may be needed to have a full picture of the empirical merits and limitations of the proposed method, I am confident the paper can be built upon by other researchers in the area.

I strongly suggest the authors to integrate their rebuttal into the final version of the paper, in particular the discussion on the dependency on the dimensionality.
confidence: Certain
Decision: Accept