Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: For unsupervised reinforcement learning, this paper proposes (1) a set of metrics to evaluate the exploration and skill learning of the agent without specific downstream tasks; (2) an automatic curriculum to train the agent with different intrinsic rewards in different stages, formulated by multi-armed bandit.
strength_and_weaknesses: Strength:
* The paper is well-written and easy to understand.
* The approach is novel and well motivated.
* The experimental results are clearly showcased.

Weaknesses:
* The experiments are only based on a single 2D maze environment, which makes the results not convincing enough. There are other more realistic environments to further test on, e.g., https://arxiv.org/pdf/2110.15191.pdf.
* There is no results to show if the proposed unsupervised RL method can really help in downstream applications, although the proposed approach doesn't require assumptions on specific downstream task.
clarity,_quality,_novelty_and_reproducibility: The paper has great clarity. The proposed approach is novel, and the experiment settings are clear.
summary_of_the_review: The paper is well-written and the proposed approach is novel, but the experiments are only based on a single 2D environment makes the results not fully convincing. Besides, as an unsupervised approach, it's not tested in any downstream applications.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper looks at the problem of unsupervised RL where skills must be learned for exploration in an environment without access to rewards. The authors take the approach of considering multiple metrics for guiding this unsupervised learning that have been proposed in the literature while choosing among each of them in an automatic curriculum learning fashion. To guide the curriculum learning process the authors use a discounted version of Pareto UCB in order to handle the non-stationary and multi-objective components of the problem. The authors compare their approach to previous baselines showcasing visualizations and metrics that may indicate their approach has intuitively learned something better or more useful. 
strength_and_weaknesses: Strengths: 
- Based on my subjective view, I do believe the proposed approach has arrived at better skills in some sense. 
- Each component of the system seems logical to me in the context of the goal of the paper.

Weaknesses: 
- While in some sense the solution makes a lot of sense, no individual component of the system is particularly novel in light of the prior literature. Additionally, the motivation to combine all of these objectives in this way seems a bit like an over-engineered solution for a topic that is largely blue sky as currently presented and far from concrete applications. 
- The proposed metrics for evaluating unsupervised RL do seem a bit arbitrary/hacky in the grand scheme of things and I feel like I get the most out of the visualizations, which is counter to the goal of this paper. Why not evaluate the skills with respect to downstream tasks? While I see where the authors are coming from arguing that the skills learned are better than the baselines visualized, it all feels quite subjective to me and really depends on what we would like to use these skills for later. 
- The paper is largely based on a series of intuitions and I didn't really find a clear overarching theory that justifies the contribution. 
- I worry that the proposed solution has a number of moving parts and may be sensitive to hyperparameters and particular implementation details. 
- I wonder if the authors can explain more about the issue with individual objectives for unsupervised RL. It does seem counterintuitive to me that no single metric would be able to result in good skills. I just wonder if the community has not found it yet or if it is obviously something more like a meta-transfer objective. I would be much more excited about this paper if it had more to say from a theoretical perspective about this topic. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written for the most part. However, the contributions are not clearly stated, which may contribute to an impression that the paper makes a number of more minor contributions. The novelty does not seem particularly high to me in comparison to prior work on this topic. The authors were not able to provide their code before the submission and have promised it for afterward. 
summary_of_the_review: I am a bit on the fence about this paper as I think the topic is important, the components of the proposed approach make sense individually, and the results seem pretty good. However, I feel like the novelty is quite low and the approach seems a bit over-engineered given the nature of the domains considered. As a result, I lean towards rejection at the moment, and feel that the paper could be a lot better if it engaged with more of a unified theory and discussed/explored the connection with preparation for downstream tasks more concretely.

Update After Author Feedback: 

While I really appreciate the detailed response from the authors, I honestly did not find it very convincing in addressing my concerns and thus lean towards my original assessment. I will just list some of my feedback on each point listed by the authors in order to be constructive for the authors in forming subsequent revisions:

1. First of all, proving that mutual information is not an optimal single metric is very different from establishing that any such single metric does not exist. Also, the intuition that we would like each skill to be balanced in how informative it is seems purely intuitive to me. 
2. This is just a restatement of the argument from the paper. My personal feeling was that the visualization was what I got the most out of. 
3. While covering all possible downstream tasks may be computationally prohibitive, I don't see why it is not possible to perform a statistical analysis of performance on an unbiased sample. I appreciate the results provided for Mujoco and would suggest this kind of analysis should be further highlighted within the main text. 
4. I disagree that the first point has been established. While past curriculum learning strategies may or may not be applied to unsupervised RL thus far in the literature, they are potentially applicable to unsupervised RL. Additionally, the latter points are as I stated really resulting from a combination of smaller contributions. 
5. I disagree that points 1, 2, and 5 have really been established beyond reasonable doubt in this work. If these things could be formally proven, then I can see the authors perspective. However, I am currently doubtful that it is possible to show this based on the current discourse. 
6. This is not what I was looking for when I asked for a theoretical justification. More concretely, I would suggest formally establishing theories related to points 1, 2, and 5 in response to the question about over-engineering that can be stated as theorems or propositions. Point 3 can be a corollary to these established theoretical statements.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper presents a multi-arm bandit method to choose between to intrinsic rewards for exploration in a synthetic maze, in an Unsupervised Reinforcement Learning (URL) setting.

It also defines some metrics to track the diversity of the policies as well as their coverage of the maze, in order to be able to compare them.

Since there are several methods of interest, the proposed method used a multi-objective bandit method.
strength_and_weaknesses: The paper doesn't provide downstream evaluation of fine-tuned policies on actual task (such as, in their setting, the completion of the maze), as is commonly done in the URL literature. As such, some key underlying assumption of the paper as not empirically verified. For example, the paper proposes to maximize the combination of the proposed metrics (in a pareto optimal way). However, there is no evidence that doing so actually results in better downstream performance on tasks of interest.

One other major weakness of the work is to limit the experiments to a very toy setting, without trying to apply the findings to more challenging environments (such as Habitat, Minecraft3D, Doom, Atari) as is customary in the URL literature.



clarity,_quality,_novelty_and_reproducibility: The experimental section lacks a lot of details on the experiments, to the point that they are not, in my opinion, reproducible. For example, I cannot tell what reinforcement learning algorithm is being used, what the action space is, what the state representation looks like, if any neural networks are being used or if it is a purely tabular model and so on.

I also can't tell at what frequency the bandit algorithm is called? Does the reward change at each time-step, each episode, every couple of episodes?
Is there a different value estimation for each of the rewards or is it a joint estimation?

As a simple baseline, how would the linear combination of APT and ICM (possibly rescaled by their respective magnitude) perform?


summary_of_the_review: Overall, the paper falls short of demonstrating the usefulness of the method, both in terms of downstream evaluation, as well as in non-toy settings. On top of that, the experiments lacks too many details to be replicated. I therefore recommend the rejection of this paper in its current shape.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies unsupervised RL that learns skills for exploration an envorinment without specific rewards. The proposed approach integrates multiple metrics (in a pareto optimal way) and choosing among each of them with automatic curriculum learning. The approach is compared against previous baselines with visualization and other metrics. As the reviewers pointed out, the experiment mainly on 2D maze environment tends to be toyish, and evaluation through visualization is subjective, and the approach seems a combination of existing components that lack strong novelty.
justification_for_why_not_higher_score: The experiment mainly on 2D maze environment tends to be toyish, and evaluation through visualization is subjective, and the approach seems a combination of existing components that lack strong novelty.
justification_for_why_not_lower_score: The problem is interesting and the paper is well written. The approach achieves certain improvement in learning skills.
Decision: Reject