Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an automated goal generation method, Curriculum selfplay (CuSP) which aims to generate goals of diversity and of increasing difficulty in sample efficient way, as well as to prevent catastrophically forgetting previously solved goals. The method is based on a 4-player game formulation, where cooperation and competition are balanced between two off-policy student learners and two regret-maximizing teachers, who plays the role of balancing progressive exploration and anti-catastrophic forgetting. The method is evaluated on a arrange of continuous control tasks and is shown to outperform a few baselines.
main_review: Strength:
1. The idea is well-motivated and clearly presented
2. Comprehensive experiments are performed to verify the effectiveness of the proposed method  

Weakness:
1. The curriculum generation is solely based on the concept of goals. The method would be more powerful if it can be used for non-goals based settings, such as those discussed in [1]. If would be better if the authors can provide some discussion about the general applicability of the proposed method.  

2. The work is mostly based on empirical study, the paper could be strengthened if some theoretical analysis, such as sample/computation complexity, convergence etc. can be provided. Since this method needs to consider multiple agent learning, the computation overhead could be a concern.

Other comments:
1. For “Goal Generator Design”, It is unclear to me why a latent noise variable z need to be introduced.

2. For the objective of goal generator policy, how is the exploitation term be constructed, especially under one-step optimization, does it include all the immediate rewards encountered in one episode?

[1] Portelas et al., Automatic Curriculum Learning for Deep RL: A Short Survey.

summary_of_the_review: In all, this paper is well written and provided a novel formulation for goal-conditional automatic curriculum learning. The paper could be strengthened if its use cases can be extended to more general curriculum learning settings (beyond goal-generation). It would also be better if more theoretical analysis, such as sample complexity/convergence and be provided.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a modification to asymmetric self play where we have two goal generators and two solvers. Each goal generator is incentivized to propose goals that one solver can solve but the other cannot and each solver is incentivized to solve goals given to them. In experimental results the method seems to perform a bit better than domain randomization. 

main_review: The authors say this is a 2-player zero-sum game but (1) they do not solve this game i.e. find a Nash equilibrium. Self play does not converge to a Nash equilibrium. (2) there exist Nash equilibria of this game where Alice and Bob are identical in which case the two goal generators get zero reward no matter what goals they propose. (3) even if one solved a Nash of this game that wasn’t degenerate, it’s not clear to me why that would be a good curriculum. 
So it seems that the method mainly relies on the training dynamics, not the static equilibrium properties of this setup. But even then, let’s say Alice and Bob start out as identical and the two goal proposers also start out as identical. Then shouldn’t Alice and Bob remain identical and the two goal proposers also remain identical because they all have the same updates? 
I don’t see how the given method satisfies the desiderata (progressive diversity, progressive feasibility and anti-forgetting). It seems like the first three rows are not a core part of the method but are used to heuristically improve the results. The last row refers to the actual newly proposed objective where you have symmetric goal setters and solvers. But as mentioned above, I don’t think this symmetric objective gives the desiderata like claimed.  
Why is SAC used to propose goals? Isn’t a goal a single action? 
The results look decent, but do not perform much better than uniform (domain randomization) on most tasks. I would be interested to see how well PAIRED does on these tasks. 

summary_of_the_review: I like this direction but I am worried that the main objective has problems that can prevent it from being an effective curriculum, even if they don't show up in these particular results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new curriculum generating algorithm for goal conditioned agents. In their setup, they have two students and two teachers; each teacher's job is to find a task their student can do better than the other student. They find that this symmetric curriculum self play (CuSP) outperforms asymmetric goal generating algorithms such as asymmetric self play and goal gan in a variety of goal based simulated robotics tasks.
main_review: Overall I found the paper written well and easy to understand. I thought the dynamic regret updates to the goal generating SAC buffer to be intuitive and appreciated the toy example they use to show it can better optimizing a non-stationary objective.

The main question for me that arose was: Where does the difference in Alice and Bob's performance come from? They both get to train on the same tasks and are the same algorithms. Is it from their different initializations? Is it from the fact that they first train on goals in different orders? Or is it simply that they are stochastic policies and when a task is somewhat hard one is likely to fail? If it is this last option, then I'd think you could come up with a similar and more compute efficient algorithm where a single generator is trying to pick goals to maximize the variance over returns of a single student. So in this setup you could have your single student just do 2 rollouts and use the difference in returns as the reward for the generator.

It feels like the H.E.R baseline should be included since you are using SAC to optimize your policies.

Why does the uniform goal distribution policy do so poorly on the reach task (Fig 7) with an extra goal dimension? I don't have a good intuitive sense for why CuSP should do better in this situation.
summary_of_the_review: Overall I feel the paper is slightly above the acceptance threshold. It does a good job of explaining it's algorithmic choices with toy experiments and shows a reasonable bump over other curriculum methods on a series of goal conditioned tasks. However, I'm left with the question of why the method works at all and if there's a better formulation that achieves the same thing (see my first question in the main review). If this were clarified and some analysis included in the paper I think it would be a much stronger submission and would be a clear accept; in this case I would be more than happy to increase my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces and evaluates a new unsupervised reinforcement learning method for curriculum learning. It aims to training agents in continuous environments with increasingly difficult tasks with a 4-agent algorithm with two goal-agents or teachers that have their respective "favourite" student that they want to excel over the other teacher's favourite.
main_review: Main strengths

* I found interesting the idea of multiple "teachers" and agents to account for catastrophic forgetting while exploring for new goals.
* I really liked the idea of updating the regrets from the replay buffer using the new evaluations from the critic, as far as I know this is new and quite nice
* Authors include quick and nice illustrations in Figures 1 and 2 illustrating the advantages of SAC and using updated regrets in the replay buffer
* Multiple benchmarks used

Main weaknesses

My bigger concerns with this work come from not well detailed parts/assumptions, the narrow comparison with AMIGO, the fact of highlighting the benefits of using a symmetric method with 4 agents but the lack of analysis of using larger numbers (e.g., 6 or 8) and leaving that part for future work. I detail all of these below:

---
* I didn't find a clear explanation of what is the action space of the goal-generation agents, authors comment that they need to be predefined and embeded with human priors of which goals are acceptable. This is an important limitation for which I found little discussion, an explanation of how these are defined in the experiments would be very beneficial.

* How are critic and actor implemented, separated? do they share the same networks? since here the critic plays an important role updating the replay buffer it would be beneficial to have a closer look on this

* In section 2 is mentioned that the goal distribution is separated into train and test but it is never clearly stated how this is done

* In page 4, first paragraph you mention that for the goal generators you add a latent noise variable to the state, why?

* In your method there is not really a "main solver" or agent, when reporting your results which one you pick? why?

---
* You cite the recent work from Campero et al. 2021 about AMIGO which also uses a goal generator agent and a goal achiever agent, it could be interpreted as a 2-agent version of what is presented here but in discrete spaces, I missed then a more in depth comparison with their work in benchmarks or , at least, theoretically.

---
* Since your work is not the first one exploring the idea of an agent looking for achievable yet challenging goals using a goal-generator agent and a reacher, the main contribution seems to be in the introduction of multiple of these and how they help each other, but then why stop in four? From the very beginning your work raises that question, thus is kind of disappointing that it is just left for future work.
 

---

### Post-discussion and updated version

I believe that authors have done a good work in the discussion and answering my concerns. I would have liked to see an analysis of the effect and equilibrium with systems of larger number of agents, but the current work has more than sufficient analysis and contributions on its own as reflected in the additional ablations and discussion incorporated in the new version. I have updated my scores accordingly
 

summary_of_the_review: The work introduces nice concepts and ideas that are not completely well supported nor fully explored. Thus although I like the  idea I am reluctant to give an accept at its current state

### Post-discussion and updated version
I think that the updated version supports much better author's statements and reflects better their contributions
review_title: null

Meta-review: While one reviewer remained concerned about the possibility of convergence to bad equilibria and felt that the proposed method appears to be four minor changes from prior work (PAIRED), the authors demonstrate empirically that the proposed changes make a significant difference in their evaluation. Other reviewers were positive about this work and all others rated this work as an accept. Post rebuttal the most positive reviewer increased their score to an 8 and felt did a good job answering their concerns. They wanted to see an analysis of systems with larger numbers of agents, but felt that the current manuscript was more than sufficient to warrant acceptance, and fell into the category of a good paper with the additional ablations provided during the rebuttal.

The AC recommends accepting this paper.
Decision: Accept (Poster)