Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses learning in goal-conditioned RL problems, and presents two contributions. The first is a simple algorithmic technique that continuously resamples goals within an episode, instead of only between episodes, to speed up learning. The second is an automatic curriculum learning method driven by the agent’s value function called VDIFF. VDIFF samples goals for which the agent is demonstrating learning progress. The effectiveness of these techniques, both separately and combined, is supported by empirical experiments in simulated robotic environments.
strength_and_weaknesses: The main strength of this paper is that it presents two clear and testable algorithmic ideas. The ideas are also relatively simple, which would make them easy to implement. Thus, this paper could be of interest to a broad range of RL practitioners.

The main weakness of this paper is the presentation and discussion of the empirical results.

With the first set of experiments, for example, the learning curves for only 9 of the 17 environments are shown. In 6 of the 9 shown, VDIFF outperforms the baselines, but in reality VDIFF outperforms the baseline only 6 of 17 environments (after a close look at the appendix). This disparity is misleading, and is not adequately highlighted. However, it is worth noting that VDIFF is never worse than the baselines (except maybe in 1 case.)

It would be useful to summarize all the data somehow in the main text, instead of including only cherry picked plots.

If possible, it would be useful to increase the number of seeds used in the experiments to at least 10, but 30 would be ideal. More seeds would help to support stronger conclusions.

How is the 95% confidence interval used in the plots computed? Is the distribution of the learning curves assumed to be normal? Is this actually the case?

Minor: The colors used in the plots make the lines hard to distinguish. Consider changing the colors to something with more contrast or even label the lines directly instead of using a legend.

In Figure 1, what is the algorithm labeled Random? It not clear to me from the text.

The paper uses the term “vanilla” throughout to mean “plain”. I don’t think that this term is useful or descriptive and can be confusing. It would be better to simply label each algorithm based on its components, e.g. “SAC with HER”.

In the last paragraph of Section 3.2.1 the paper says that using the smoothed updates provide a cleaner, less noisy LP signal, and that in practice the gain outway the slowdown introduced. Could this be further discussed? Are there experiments that compare the two cases? How big is the difference in practice?

clarity,_quality,_novelty_and_reproducibility: The content provided in the paper is mostly clear. The descriptions of the algorithmic ideas are clear. Nevertheless, the presentation of the experiments and associated discussion could be improved (see weaknesses above.)

The algorithmic ideas are simple, but I believe that they are novel.

Given the information provided in the paper, I believe the results should be reproducible.

summary_of_the_review: Overall, the paper presents two clear new algorithmic ideas, but the presentation and discussion of the experiments is lacking. Therefore I would argue to reject.

The paper could be improved by cleaning up the presentation of the experiments and expanding the discussion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to accelerate and improve goal-conditioned RL. The main contribution is the continuous goal sampling technique proposed. The idea is very simple. Instead of sampling a goal only at the start of an episode, we can sample a new goal every fixed number of timesteps within an episode. The second contribution is the automatic curriculum learning algorithm proposed, which uses learning progress estimated through value functions to create a self-paced curriculum. These two techniques are both not tied to the base RL algorithms.  
strength_and_weaknesses: - Strengths:
  - The proposed continuous goal sampling technique and automatic curriculum learning algorithm are very simple and not tied to the base RL algorithms. 
  - The paper is well-written and easy to follow overall. 
- Weaknesses:
  - In continuous goal sampling, just sampling a new goal every *fixed* number of timesteps seems to have some obvious drawbacks/limitations (and the paper dose not seem to discuss it). It does not really take into account the agent’s current learning progress or the state it is in, but just blindly changing goals when the time meets the pre-designed fixed condition. I could easily imagine some scenarios where this might hurt the sample efficiency, e.g., when the agent is close to reach the current goal and receive the highest reward, the goal (reward function) is suddenly changed. Then what the agent learns before could be wasted.
  - The technical novelty of the paper does not seem significant enough (e.g., sampling goals based on the agent's learning progress has been done before as mentioned in the paper). 
  - There seems to be some problems with the hyperparameters settings used in the baseline methods in the experiments. Specifically, in the experiments, the paper mentions "All baseline methods also use SAC with identical hyperparameter settings to enable a fair comparison." I do not understand why the baseline methods have to use the *same* hyperparameter settings as the proposed method to enable a fair comparison (if I understand it correctly). It is common to do separate hyperparameter tuning for different methods when comparing their performance in deep learning. 
clarity,_quality,_novelty_and_reproducibility: The paper has good clarity overall. 

The novelty of the paper dose not seem significant enough, given that the main proposed technique (continuous goal sampling) is extremely simple and seems to have some obvious drawbacks/limitations, and the other automatic curriculum learning algorithm proposed essentially uses a similar idea from existing works. 
summary_of_the_review: This work is well-motivated and the proposed methods are simple and easy to understand. Based on the experimental results, the main proposed technique continuous goal sampling seems to be quite effective in a variety of tasks tested, even though its idea is so simple. However, I'm not sure if the proposed method is compared against baseline methods in a fair way as the authors claimed, making it hard for me to judge how effective the proposed method really is. Also, I think the limitations of continuous goal sampling need to be clearly discussed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on goal conditioned RL and introduces two algorithmic contributions:
* goal resampling within an episode (called continuous goal resampling)
* a automatic curriculum learning method called VDIFF that samples appropriate goals based on learning progress (defined as the change in predicted (target) value for the same state and goal after some number of updates)
The paper demonstrates empirically that goal resampling is beneficial for VDIFF and other goal-conditioned RL methods. VDIFF is similar to other methods when hindsight experience replay (HER) is used but performs better without HER.
strength_and_weaknesses: Strength:
* goal resampling seems like a generally applicable and useful idea.
* VDIFF-R is empirically promising in the setting without HER

Weaknesses:
* There are two algorithmic contributions that are orthogonal. The first figure is misleading and I would prefer it if the results of figures 1 & 2 are combined in one figure. Since the resampling can be applied to other methods those should be present in the first comparison
* Related to the first weakness, I think the abstract is misleading and contains not enough nuance. "continuous goal sampling and VDIFF work synergistically and result in performance gains over current state-of-the-art methods" does not seem quite right when VDIFF-R does not outperform HER-R or VDS-R.
* I am not particularly convinced that settings without HER are very interesting. What is a setting where HER cannot be used? If that is where VDIFF shines then there should be an evaluation in such a setting directly rather than evaluating in a setting where HER can be used.
* I think it would be good to include the algorithms in the main text if possible. Similarly, I think the selective resampling mentioned in the appendix ought to be mentioned in the main text as it adds complexity.
clarity,_quality,_novelty_and_reproducibility: I think the paper is written in ways that could be misleading. Writing should be clarified and improved. I have no concern about reproducibility. Resampling goals has been used previously by practitioners designing tasks to elicit particular behaviours but to my knowledge it has not been explicitly incorporated into an algorithm previously. Similarly learning progress has been considered for curriculum generation but not in this exact formulation.
summary_of_the_review: The paper proposes two orthogonal contributions. To my mind it is written in a slightly misleading way and does not do enough to justify why the setting in which VDIFF-R performs well (no HER) is interesting and relevant. If there is an interesting setting where HER cannot be used then the evaluation should at least partially be in that setting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed two methods to improve auto curriculum learning strategies in the RL settings where multiple goals are available with no change to the environment dynamics:

1. Continuous goal sampling, where the goal of the agent changes every R < episode length steps
2. A curriculum based on "learning progress", which chooses to sample tasks (described here by a tuple (initial_state, goal) proportionally to the improvement in the value function within dT episodes, and to smooth these VFs using Polyak averaging.

The method is evaluated on a number of manipulation and maze navigation tasks, with the performance compared with VDS and HER. In the basic version, authors propose to use their method (called VDIFF) together with HER, but they also provide ablations on the effect of using HER with the conclusions that in some of the tasks VDIFF improves over HER and VDS, and in others VDIFF makes the same gains (ie. performance of HER is comparable to VDIFF which is better than the lack of both).
strength_and_weaknesses: ## Strengths
Continuous goal sampling is a simple, but a reasonable idea. It fits well within the Zone of Proximal Development (ZPD) framework[1] which states that tasks presented to the agent should be neither too easy nor too difficult. I see the standard "termination at the reward" as satisfying the "not too easy" part of the ZPD motivation, and authors "change the goal every R steps" naturally rejecting "too hard" tasks (if there is no reward in R steps, the goal gets rejected).

The experiments are performed in many (current, external to the paper) domains and the baselines of variants of VDS and HER are chosen appropriately. The paper tells a coherent story, supported by many experimental results.

## Weaknesses
Learning progress seems to only differ from SPaCE through the use of Polyak averaging. Authors claim that "SPaCE is targeted towards contextual RL and is only applicable in discrete task settings.", however I think SPaCE could be a reasonable additional baseline to try. Alternatively, the addition of Polyak averaging could be ablated, to understand the impact of the novel part proposed by this paper.

## Smaller suggestions

I would like to see HER performance on the Figure 3. Authors suggest that in many tasks, a further progress is difficult compared to a baseline of HER; it would be good to confirm that (where) the gains of HER are reproduced by VDIFF. It would make it clear whether the conclusion is:

1. Sometimes HER is better, sometimes their method, best to use both, or
2. VDIFF (with no HER) is always better, but sometimes HER is enough.

## Longer-term, future work suggestions

I find changing goals without resetting the environment only partially symmetric to "finishing episode at the reward"[2], as ending the episode on reward resets the environment, whereas for changing the goal the env is not reset. There are two alternatives which would treat both constraints in the same way:

1. Change "goal sampling" to reset the environment when there is no reward in R steps, or
2.  Don't reset the environment at the reward. Rather, continue running the environment, but cut the bootstrapping beyond the first reward, to make the agent only optimize "time until the first reward" and not keeping the reward satisfied later.

It's unclear how the proposed method deals with the situation where some tasks are inherently harder (=> require more time to solve optimally) than others, and thus require different Rs.

[1] Vygotsky, [Interaction Between Learning and Development](https://innovation.umn.edu/igdi/wp-content/uploads/sites/37/2018/08/Interaction_Between_Learning_and_Development.pdf)

[2] I understand authors made no claims to the symmetry here, but for the future work, I find it interesting to analyse the situation in more detail, which would make the results even stronger in my mind.
clarity,_quality,_novelty_and_reproducibility: I find the writing clear, and the results to be of high quality. As mentioned above, I find the continuous sampling part to be novel (yet simple), whereas the learning progress part of the contribution to be only marginally novel compared to SPaCE. I find the results reproducible.
summary_of_the_review: The authors propose two improvements to the curriculum learning for goal-conditoned RL. One of them is simple but novel, the other one is a marginal improvement over previous work (SPaCE). The experimental results are solid and convincing. 

I think the goal sampling idea would be an interesting addition to the conference, so I lean towards acceptance. The strength of my sentiment is weak due to the second, less novel part and the fact that the idea is something many would have tried even if they didn't read the paper (yet I haven't managed to locate it among previous literature).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper presented two contributions for goal conditioned episodic RL problems.  The first is sampling goals continuously throughout the episode (instead of at the end in the manner of Hindsight Experience Replay).  The other is an automated curriculum method which is informed by learning progress across time (VDIFF).  The method shows improvements on some problems in the experiments.

Strengths:
The core contribution is presented clearly, the problem setting is well motivated, and the results look promising to the reviewers.

Weaknesses:
The reviewers found the evidence in the paper to be lacking, and the mixture of the two contributions were muddled.  The revisions to the paper by the authors did partially rectify the reviewer concerns, but not fully.

Additional experimental limitations were noted by a separate examination of the paper, after the AC-reviewer meeting.
- "All baseline methods also use SAC with identical hyperparameter settings to enable a fair comparison." This is not fair, it is misleading.
- Is the initial exploration period of SAC included, as in the SAC paper?  This is crucial for good performance.
- Standard normal CI must be justified in the small sample regime (6 seeds). Student-t CI would be much better.
- The original HER paper used DDPG, its problematic this paper compares against SAC+HER, same issue for VDS
- A better configuration of SAC is well known and widely cited (https://arxiv.org/abs/1812.05905)
- It is unclear if all differences can be explained by untuned baselines. These changes from DDPG to SAC are confusing.
justification_for_why_not_higher_score: This paper was on the borderline. The reviewers appreciated the simplicity of the work and the two ideas. However, after a discussion with the reviewers, there was a consensus that a better experimental design and qualitative analysis was required for this work to be a clear contribution to the research community. The reviewers felt that a better explanation (via experiments and analysis) of how the two ideas combine would have made the paper acceptable (and that would be a good direction for a future revision). Additional effort on the experiment design would also make the resulting analysis more convincing.
Decision: Reject