Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors a method that lies in the intersection of two subfields of RL, namely open-endedness and unsupervised skill discovery. Specifically, they introduce a method called Rewardless Open-Ended Learning (ROEL) which extends a recent method called POET to perform skill discovery in a reward-free setting rather than traditional supervised RL. To this end, they use mutual-information-based skill discovery techniques which have recently become a popular approach for unsupervised RL. The authors empirically demonstrate that ROELD is able to learn identifiable skills in bipedal walker environment.
main_review: ## Pros:

1. The paper addresses an important problem: unsupervised skill discovery in (kind of) open-ended RL environments. This is exactly the kind of problem where specifying a concrete reward limit what can be achieved in terms of skill discovery. To me, the problem is real and practical.
2. The proposed framework of skill discovery in open-ended domains is novel, although combines two existing methods with each other.
3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed method.

## Cons:

1. I don't quite understand how exactly was the DADS baseline trained. The paper says it was trained "using the original bipedal default environment". There are two versions of the walker environment, according to its OpenAI gym implementation, namely normal (with slightly uneven terrain) and hardcore (ladders, stumps, pitfalls). Which one was it trained on? If DADS is trained on a single environment, it is not really fair to compare it with one trained on multiple environments for generalisation purposes.
2. Have authors tried training DADS using Domain Randomization (DR), where the training environment is sampled from a parameterized distribution, i.e., randomly sampling hand-picked obstacle hyperparameters, such as stump shape, surface roughness and gap width? The authors should show that ROEL works better than DADS trained on DR, otherwise, it is hard to argue about the effectiveness of the method.
3. I'm not convinced with the statement in Section 4.1 claiming that ROEL "outperforms" DADS in off-sample environments. What does "outperform" even mean in this particular experiment and how is it measured? All I see is a few cherry-picked environments where some skills of ROEL were able to pass through an obstacle where some skills of DADS weren't able to do this. First, I don't understand how the skills are selected in these experiments. How is z chosen? Have you tried all z's? Have you tried more example environments? Please elaborate on exactly what is going on here.
4. The structure of the paper can also be improved. Particularly, it is a bit unusual to see Implementation details (Section 3.3) in what seems like the Method (Section 3). The authors could consider adding an Experimental Setting section (after section 3) where such details will be described. Also, various future directions for work are described throughout the paper. Moving all those ideas to the Conclusion would improve the readability of the paper.

## Minor issues

- In PAIRED, a separate adversary generates the environments for the protagonist and antagonist. The antagonist doesn't do this.

## Additional issues

The issues below did not influence my score, but please address them in the updated version of the manuscript.

### Citations format:
- There are two basic citations commands in the `natbib` package
    - `\citet` for **textual** citations. This prints *Hinton et al. (2006)*
    - `\citep` for **parenthetical** citations. This prints *(Hinton et al.,* 2006*).*
- In the entire paper, authors make use of the `\citet` command (or generic `\cite` which can be problematic, see [natbib documentation](https://ctan.org/pkg/natbib), page 6). However, the majority of them should actually use the `\citep` command. According to the [ICLR 2022 formatting instructions](https://github.com/ICLR/Master-Template/raw/master/archive/iclr2022.zip):
    - "Citations within the text should be based on the natbib package and include the authors’ last names and year (with the “et al.” construct for more than two authors). When the authors or the publication are included in the sentence, the citation should not be in parenthesis using `\citet{}` (as in “See Hinton et al. (2006) for more information.”). Otherwise, the citation should be in parenthesis using `\citep{}` (as in “Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007).”). "

### Academic language and abbreviations:
- Please DO NOT use "&" instead of "and", "w/" instead of "with", etc, throughout the paper

### Punctuation errors:
- There are large number punctuation mistakes in sentences all over the paper. Especially, commas are very often missing. Please fix these.

# Post-rebuttal update

I thank the authors for their response and improving the manuscript based on my suggestions. Specially, I appreciate the authors running DADS on domain randomised version of the bipedal walker environment and making cosmetic changes to the paper.

However, after taking a look at the current version of the paper, I can't yet recommend that it is accepted in its current form. It seems like the qualitative and some quantitative analysis is still compared against DADS on the normal version of the bipedal environment. The performance gains against this baseline are expected and the comparison is unfair, as far as I'm concerned.

I therefore recommend that the authors continue improving the paper and submit it for another review process at one of the upcoming conferences.
summary_of_the_review: While I think the paper addresses an important problem, I have major concerns about the claims the authors make given the experiments they have. Specifically, I've raised questions on the choice of baseline agents and comparison with them. Furthermore, the writing can be improved significantly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes combining unsupervised skill-discovery with open ended learning, where the environment and agents are continually evolved together. Specifically the proposed method (ROEL) is a combination of POET[1] with the intrinsic reward for skill-discovery in DADS[2], and is evaluated on a simulated bipedal walker environment. 

main_review: Strengths 

1. Relevance of problem setting - 
The paper considers the very important setting of reward-free learning, since it is difficult to specify reward functions that align with our intended objectives and further cumbersome to provide rewards for real systems. The paper considers leveraging open-ended learning to try to learn a more diverse set of skills, since automatically generating challenging environments can enable new skills to be discovered. 

Weaknesses 

1. Experimental Evaluation  - missing evaluation on DADs environments

The proposed method can be viewed as DADs, augmented with a scheme to train on autonomously generated diverse environments (from POET). In the experiment section, the authors compare the performance of ROEL to that of DADs on held-out environment variations, and argue that ROEL is superior. However, they only use the bipedal walker which has very low action space (only 4 dimensions), and don’t test on any of the environments used in the DADs papers [2,3] which include control of much more complex agents such as the gym-ant, humanoid or the d-kitty robot. Since the skill-discovery from DADs has been shown to enable effective control in these agents, we need to evaluate if open-ended learning can scale to these agents as well. This is quite critical for general adoption of open-ended learning in unsupervised learning.  

2. Experimental Evaluation - issues with current evaluation

For the only included environment (bipedal-walker), performance comparison on the out of distribution environment would be much more compelling if the authors included quantitative measures of performance such as a success metric. Instead the authors compare performance by showing visualization of 1 run for each method (Fig.2), but this could be susceptible to noise, it’s unclear if this behavior can be reliably replicated, and further from the visualizations, it’s unclear if the ROEL agent actually cross the gap successfully. 

The authors do include some quantitative analysis of the prediction error of the observation trajectory from the skill network (Fig. 4), and show that DADs skill network overfits. But this is an argument for ROEL discovering more diverse skills, not whether these skills actually enable better performance on a given OOD environment, for which success metrics should be used.

In Fig.3 the authors plot returns from behavior resulting from different skill priors, and find that the return plot for ROEL is more diverse. They argue that diverse returns can be used as a heuristic to imply the corresponding skills are diverse, but I’m not sure this is necessarily true. For example, different skill priors could essentially learn to perform the same behavior but to varying degrees of proficiency, thus resulting in a diverse return plot without having skills that are really diverse. 

3. Novelty 

The novelty of the approach is very limited, as it essentially takes the POET algorithm for open-ended learning, and replaces the external environment reward with the exact reward used in DADs. It would have been interesting to consider how the DADs skill conditioned dynamics model that is learned could be used for ranking or selecting environments/agents in the open-ended learning part.  

summary_of_the_review: While the premise of generating new environments to enable more diverse skills to be learned is intriguing, the experiments do not convincingly demonstrate this. The paper lacks evaluation on more complex environments where DADs is known to work well, and also doesn’t include quantitate performance metrics for the simpler bipedal walker considered.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a method for learning skills in an open-ended, reward-free setting using coevolution of agents and environments trained to maximize the mutual information between a skill latent code and state transition.
main_review: ### Strengths
- Open ended learning and the unsupervised discovery of diverse skills is a very interesting problem and the paper suggests an extension of prior work to this intersection of approaches.  
- The paper is well written, and presents a complicated approach clearly.


### Concerns

The evaluations of the presented algorithm are unconvincing.  

- Primarily, the baseline is an algorithm trained on a single environment.  It is not surprising that a method trained on multiple environments (co-evolved or not) will generalize better to unseen environments.  If the contribution is meant to highlight the improved performance of co-evolution over a static distribution of tasks (or some other naive multi-task method), it would be nice to see that comparison.  The differences observed in Figures 3 and 4 could reasonably be explained by single vs multiple environment training, rather than the POET-like curriculum learned through ROEL.

- Second, the quantitative evaluation of unsupervised learning methods is a difficult problem in itself, but it would be nice to see some evaluation of fine-tuning to a particular “downstream” task with extrinsic reward (again, comparing to sensible baselines).

Without proper experimental comparisons, the significance and impact of the ROEL algorithm is difficult to assess.

## Post rebuttal update

Thank you for the response.  The evaluation against DR is a step in the right direction.  However, the evaluation still needs to be more mature (see my second point above) before I would be comfortable accepting this paper.  I encourage you to keep working on it and submit to another conference.
summary_of_the_review: Although the method is interesting and tackles a relevant problem, the experimental evaluations are not mature enough to provide a clear indication of the method's merit.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Rewardless Open-Ended Learning (ROEL) is fundamentally a combination between two Paired Open-Ended Trailblazer (POET) and Dynamics-Aware Unsupervised Discovery of Skills (DADS).
POET presents a framework for "open-ended learning" which automatically generating progressively more difficult environments to elicit agents with novel behaviors.
DADS introduces a method of "unsupervised RL" which learns a variety of skills without rewards solely based on mutual information between state transitions and skill-determining latent variable.
The result is an RL framework which generates environments to elicit controllable behaviors without a reward function from the environment.

The following empirical evaluations are performed:
- Qualitative assessment of skills learned by ROEL
- Qualitative comparison or ROEL and DADS
- Measurement of rewards of different skills learned by ROEL and DADS
- Measurement of state predictability of skills in different environments by ROEL and DADS

The primary contributions of the paper are:
- Combining open-ended RL and unsupervised RL
- ROEL yields a more robust set of policies in a bipedal-walker environment than the DADS baseline.

main_review: ## Strengths
- The combination of open-ended and rewardless learning is very natural and synergistic -- it moves strongly in the direction of learning without hand-crafted structures.
- The algorithm is clearly presented and represents a paradigm with wide applicability.

## Weaknesses
- One of the main selling points of ROEL is that it is rewardless (unlike POET), which means that is should easily transfer to other environments. This is not evaluated, though I understand that the original version of POET is restricted to one environment.
- It is difficult to see a significant qualitative difference between ROEL and DADS as illustrated in Figure 2.
    A video would likely better serve this end.
- The quantitative analysis is primarily indirect; it supports the hypothesis but not in a definitive way.
- A comparison against POET would be particularly salient as it is the open-ended learning baseline.


## Recommendation
I recommend "accept", although I could be convinced either way upon further discussion.


## Justification
Although I point out a number of weaknesses with the evaluation, in particular, I believe the evaluation is an intrinsically difficult step for this type of work
The evaluation that is performed supports the claims of the paper, and the overall quality is still sufficient for acceptance.
Furthermore the novelty of the algorithm and its clear presentation lay a good foundation for future work.

Specific criteria:
- Correctness: 3
- Technical Novelty and Significance: 3
- Empirical Novelty and Significance: 2

## Questions
- What were the particular difficulties in combining these two algorithms/paradigms?


## Additional Comments
A caveat of my review is that while I am familiar with reinforcement learning, I am not up-to-date on the literature surrouding POET and DADS.
I can judge the work on its internal consistency and how much further it develops these two approaches, but if there are other similar approaches, I might not be aware of them.

- s1 p6: "We hypothesis" -> "We hypothesize"
- s4.1 Figure 2: This sequence of frames is not altogether convincing of the point being made. A video might be necessary.
- s4.1 p2: "this is a good qualitative example of how ROES is able to leverage experience gained through mutation & goal-switching" -- this claim does not seem adequately defended as I think the example only shows that ROEL might be more robust than DADS rather than specifically leveraging experience.
- s4.1 p2: "mutation & goal-switching" -> "mutation and goal-switching"
- s4.2 p2: "the reward function is based on the ability of the agent to reach the end of the episode" --
    Does reach the end of the episode mean that the agent moves rightward or simply that it does not fall before some number of timesteps?
- s4.2: "in sample" -> "in-sample", "out of sample" -> "out-of-sample"
- Figure 4: Highlight the fact that the in-sample plots have a highly zoomed-in y-axis (which is hard to read).
    I did not realize this at first and thought ROEL was somehow highly noisy compared to DADS in-sample, but in reality, the scale is too small to make a difference.
    Maybe it would just be better to keep the in-sample and out-of-sample plots at the same scale to demonstrate that ROEL is very similar to DADS in-sample, relatively speaking.
- s4.2: Probably beyond the scope of this paper (no expectation of implementing it), but a behavior/skill embedding space would be a great way to measure the diversity of behaviors, although this might be very difficult to implement.
- s4.2 last paragraph: "predictable & useful skills" -> "predictable and useful skills"

summary_of_the_review: ROEL is a logical and synergistic combination primarily of rewardless learning (DADS) and open-ended learning POET.
The algorithm has a clear applicability and a good presentation.
The experiments are reasonable and support the primarily claims of the paper, although the experiment measure indirect aspects of the claims as direct experimentation is intrinsically (to the claims) difficult.
review_title: null

Meta-review: I thank the authors for their submission and active participation in the discussions. This papers is borderline with the majority of three reviewers leaning towards rejection and one leaning towards acceptance. All four reviewers unanimously agree on the empirical validation being a major weakness of this paper with reviewer kQnm putting less emphasis on this shortcoming. Overall, I side with reviewers Bne6, 2GeS and HJjY, and believe the paper needs to a more thorough set of experiments. I therefore recommend rejection.
Decision: Reject