Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers the task of curriculum reinforcement learning (CRL), and propose an algorithm, GRADIENT, which leverages ideas from gradual domain adaptation (GDA) and optimal transport (OT). Essentially, GRADIENT focuses on the probability distribution $\rho(c)$ over the context variable $c$, and fine-tunes it from the source distribution to the target distribution gradually by treating it as a Wasserstein barycenter between the source and target (moving gradually from the source to target). The authors performed some brief theoretical analysis and conducted experiments over several benchmarks. The empirical results show GRADIENT has better sample complexity compared with the considered baselines.
strengths_and_weaknesses: Strengths:

1. The paper applies gradual domain adaptation (GDA) to reinforcement learning in a reasonable way. It seems that the task of curriculum reinforcement learning (CRL) involves a directional adaptation (i.e., from a source to a target) of probability distribution in a continuous way , which is indeed what GDA could help with. It is good to see that GDA is becoming useful in more and more application scenarios.

2. The authors provide a theoretical analysis of the proposed algorithm to corroborate it, which is nice. It looks that the derived RL bounds, Eq. (6)(8), look very similar to the bound in Lemma 1 of [1], which are derived in the supervised learning setting for GDA. Overall, I think the theoretical analysis is elegant. (Notice to area chair: I didn't check the proof since I'm not an expert of RL theory)

Weaknesses:

1. Novelty: The application of GDA and Wasserstein barycenter is quite straightforward, as long as one knows both GDA and CRL. So from the perspective of GDA, this paper is not very novel in terms of methodology.

2. [**IMPORTANT**] Missing discussions on a concurrent/prior work: This paper briefly mentioned a concurrent work [2] in Sec. 2 in one sentence. However, [2] actually has lots of similarity with this work, and it appeared online in Oct 2021, which seems more like a prior work rather a concurrent one. [2] also applied optimal transport (OT) and Wasserstein barycenters to CRL, thus in my opinion, it should be discussed in this paper more carefully. Reviewers need to know the difference between this paper and [2] clearly.

2. [**IMPORTANT**] Missing baseline in experiments: as I mentioned above, [2] is prior work, and it has released code. So there is no good reason for the authors not to compare GRADIENT against [2]. 


[1] Wang et al. Understanding Gradual Domain Adaptation: Improved Analysis, Optimal Path and Beyond. ICML 2022 (https://arxiv.org/abs/2204.08200)

[2] Klink et al. Metrics Matter: A Closer Look on Self-Paced Reinforcement Learning. 2021 (https://openreview.net/forum?id=lKcq2fe-HB)


-------------------------------------
**Post-Rebuttal**
The authors' responses addressed most of my concerns above, so I increased my rating.
questions: See Weaknesses above.
limitations: See Weaknesses above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers the problem of building curricula of tasks for reinforcement learning in order to learn the target task more easily. The authors propose to approach this problem by finding the interpolations between tasks using the Optimal Transport framework, thus replacing previous methods which heavily relied on f-divergences and the KL divergence in particular. Additionally, the paper introduces the $\pi$-contextual-distance in order to measure the distance between different contexts, particularly in settings where the context space is discrete. The authors provide theoretical proof that under certain assumptions the performance gap between each pair of neighboring tasks is small. Finally, they test the proposed method empirically and show that it outperforms competitors in settings where the context cannot be well approximated with a Gaussian distribution.
strengths_and_weaknesses: Strengths:
- The proposed approach is intuitive and follows naturally from the previous line of research on applying KL divergence to find a good sequence of tasks.
- The authors provide interesting and plausible scenarios when their approach is especially useful. In particular, Figure 5 illustrates this point visually by plotting the context space.
- The results in settings where the context cannot be effectively modeled as a Gaussian distribution are quite good.
- The paper is well written, the description of the method and experiments is mostly clear, and the mathematical notation is well defined.

Weaknesses:
- The empirical evaluation could be extended, as two out of three tasks (Maze and PointMass) are rather toyish. It would be useful to have another more complicated benchmark, such as the Ant locomotion environment that was used e.g. in the SPDL paper.
- Although the checklist at the end of the paper says otherwise, the limitations are not properly described. For example, I would suggest discussing the following limitations:
    - The method is only tested in settings where the context space is nice and relatively low-dimensional. 
    - The assumptions for Theorem 4.1 are quite strict (the context cannot change the transition dynamics nor the reward function).
    - The $\Delta\alpha$ hyperparameter seems to be very important, but there is no practical guidance on how to set it.
- This is subjective, but I'm not entirely convinced about the level of novelty of the proposed approach. The main idea is to take an approach that works well with the Kullback-Leibler divergence and use another way to measure the similarity of distributions, namely the Wasserstein distance. In the same way, the $\pi$-contextual-distance which the authors list as a contribution seems to be a straightforward extension of the distance introduced in [1]. On the one hand, this seems a bit iterative, but on the other I still think overall the contribution is significant, so treat this as minor criticism.

In the end, although I have some doubts about novelty and empirical evaluation, I think the contributions in this paper are significant. For the moment I decided to go with "Weak accept".

Various comments:
Figure 3(b) (and the rest of the figures with performance): Just to make sure, does the x-axis shows the number of steps of training on the target environment or the whole training process (i.e. starting with pre-training on the auxiliary task)?
Line 111: unnecessary parenthesis. 

[1] Castro, Pablo Samuel. “Scalable methods for computing state similarity in deterministic Markov Decision Processes.” AAAI (2020).
questions: The only suggestion I think is reasonable within the scope of the rebuttal period is adding the limitations to the paper. Other than that, please refer to the "Weaknesses" and "various comments" sections above. 
limitations: The paper does not discuss the potential negative societal impact and I don't think such discussion is required.

There is no discussion on the limitations of this work, which I think would be very useful for the reader. I listed a few possible limitations in the  Weaknesses section above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: this paper propose GRADIENT, a novel curriculum reinforcement learning method based on optimal transport. Therefore, GRADIENT improves the learning efficiency and asymptotic performance in a wide range of tasks compared with state-of-the-art baselines. 

For example, in the maze example, GRADIENT algorithm can better interpolates the environment task distribution from source to target env compared to linear interpolation. 
strengths_and_weaknesses: Strengths:

OT in CRL is novel

the proposed method deals with both discrete and continuous context space.

empircally,this paper shows its proposed method's learning efficiency

this paper provides a theorectical bound for policy transfer.


questions: Q: dealing with sparse rewards problem, there are some other useful RL algorithms, such as HER. have the authors conducted some experiments compared to these baselines? and why GRADIENT?

I appreciate the authors add HER as a baseline. Considering the discussion and revision, I increased my score
limitations: No
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes GRADIENT, a method for curriculum learning, by interpolating the initial context distribution and the target context distribution using the Wasserstein distance. This paper provides a theoretical justification for the proposed algorithm that the optimal value function under two adjacent contexts can be bounded by the Wasserstein distance between the contextual distributions of these adjacent contexts. This paper also demonstrates the practicality of the GRADIENT method in several goal-condition tasks.
strengths_and_weaknesses: Pros:
1. The main body of this paper is very well written and the Wasserstein distance indeed serves good metric for measuring distributions.
2. The experimental results are very interesting as they clearly reflect the interpolation of the Wasserstein Barycenter.
3. The experimental results seem promising -- as in all demonstrated tasks, they outperform the prior arts by a large margin.

Cons (Updated Aug 3rd):
1. (Major; clarified) The connection between the theoretical results and the experiments is not carefully addressed. The paper would be much better if the author can address how the main assumption (4.1) is reflected in the experiments. Since all experiments are low-dimensional, so it should not incur the "curse of dimensionality of the Wasserstein distance", hence it would be nice to address how the assumption is actually reflected in the selected experiments.
2. (Major; draft improved) The proof of theorem 4.1 is hard to follow. The readability of the paper would be largely improved if the author can summarize the notations/assumptions in the appendix, as the proof adopt many non-standard RL theory notations.
3. (Minor; draft improved) There are some minor citation issues in the related work section -- when using a paper to start a sentence, it would be better if the author used \citet{} instead of \citep{} (e.g, "[6] proposes Self-PacedReinforcementLearning(SPRL)"; "Concurrent to our work,[19] also propose to replace KL divergence").

The reviewer would like to improve the rating if the author can improve the readability of the proof.
questions: 1. It seems that the context dimension of all experiments conducted so far is either discrete or dimensional <=2, is it possible to run the GRADIENT algorithm on some non-goal-conditioned tasks?
2. It seems that assumption 4.1 assumes the state space is a subset of the context space ($\mathcal{S}\subset\mathcal{C}$), but in the FetchPush environment, the context is a 2-dimensional vector representing the goal while the state is a 28-dimensional vector, perhaps the author can clarify this?
limitations: N/A
review_title: null

Meta-review: metareview: The submission makes a strong case  for using optimal transport-based curriculum learning method principled theoretical insights and convincing experimental study.  At the same the there is a significant overlap between this paper and prior work on using optimal transport for curriculum learning and self-paced learning which needs to be recognized and discussed more extensively in the related work.
confidence: Less certain
Decision: Accept