Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors introduced the Unsupervised Reinforcement Learning Benchmark. 12 continuous control tasks from three domains are studied in DeepMind control suite. The eight baselines are compared. The source code is publicly available. 

strengths: I agree with the authors that establishing unified benchmark for unsueprvised RL is necessary. The paper is well-motivated.
The paper is well-written and easy to follow.
I like the part where the authors grouped algorithms and organized the evaluation into state-based and pixel-based scenarios.
The number of comparative methods is sufficient and covers different types of RL approaches.
weaknesses: I have concerns about the evaluation metrics and the way experiments are designed. I would be willing to modify the scores if the authors could explain or justify the following:

1. What is normalized return? normalize with respect to which algorithm? Why not directly show the absolute return over all runs? I looked into table 4 and 5 in the appendix. I am still confused about what the individual results mean. Are those "averaged returns over runs" or anything else?

2. In the paper, the authors evaluated all the algorithms using overall normalized return. What about other evaluation metrics for individual algorithms, e.g. plots of intrinsic rewards for knowledge-based methods or the mutual information plots for competence-based methods or predictive errors for generative approaches?

3. I am curious about why authors only evaluate fine-tuned models at 100k steps. Different methods have different numbers of parameters (due to architecture differences, replay buffer sizes, and so on). I am not sure whether it is a fair comparison to only report performance at fixed 100k steps. And why 100k?

4. When authors refer to unsupervised representation learning, I wonder which environments are all the models trained on for unsupervised representation learning. E.g. are they trained on the same environment where the fine-tuned environments are. This is related to testing the generalization ability of these algorithms. If not, how similar all these environments in terms of representation learning. Similarly, how similar are the behavioral learning tasks at unsupervised and fined-tuned stages? 

5. I like the point that authors disentangle representation and behavioral learning. However, I am not sure why each unsupervised learning RL algorithm should perform consistently well in both behavioral or representation learning, as one of the observations in the Experiment Section verified this point. Why is it necessary to compare these for the same algorithm in the first place?

6.  I found that all the observations in Sec Experiments are interesting. It would be great if authors could provide explanations about why different types of algorithms behave so differently. 

Minor: 
1. line 79: notation for intrinsic reward is wrong: r^{int}_t
2. The generated PDF files for the main paper and supplementary material are only viewable online using browsers. The downloaded version (unzip) causes errors when opening PDF offline. 
correctness: The source code for the implementation of each algorithm is available. I believe that the evaluations are implemented correctly; though I did not run the code by myself.

I have several concerns about the design of the evaluation methods and the way all algorithms are compared (see weakness).
clarity: The paper is well-written and well-structured. The motivation is well-explained.

relation to prior work: There is sufficient coverage of related works.
additional feedback: Please see the weakness section.
review_title: Problems with how all algorithms are evaluated

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper aims to standardize the process of benchmarking unsupervised reinforcement learning (RL) algorithms and sheds light on the limitations of the existing algorithms. To do so, it first proposes 12 benchmark tasks (i.e., 3 domains with 4 downstream tasks per domain) built upon the DeepMind Control Suite. Then, it introduces a methodology for pre-training, fine-tuning, and evaluation of these algorithms on the proposed environments. As baselines, 8 popular unsupervised RL algorithms are selected from the literature. It unifies the training and evaluation procedure of these baselines by keeping all of their components, such as the optimization scheme, the same and only varies the intrinsic rewards across these well-known algorithms. At last, the baselines are trained and evaluated on this set of environments based on the proposed standard procedure, and insights about the current state of the unsupervised RL methods are presented.

Edit: I have read the author response, and they have adequately addressed my concerns.
strengths: Overall, this paper is well-written and it is a valuable contribution to facilitate unsupervised RL experiments, namely those where a representation is first pre-trained and then used for a downstream task. A few choices should be clarified in the author response; given these clarifications, I would be willing to increase the score.
 

weaknesses: As a benchmark paper, completeness and thoroughness in experimental design and description is key. One big omission I see is in discussing how hyperparameters are chosen for algorithms fairly. The main body says per-algorithm hyperparameters are tuned, with details in Appendix B. However, I only see a list of final hyperparameters. Which were swept, and how? Your approach can provide a general strategy for any work that uses this benchmark. 

Similarly, the presentation of some of the results contrasting algorithms needs further explanation. It is important to explain why 10 seeds have been used, and why this is sufficient to make clear claims. Are the differences statistically significant? There are additionally some strong claims from the results here, that could be better qualified. For example, some of the differences could be due to using a base optimization algorithm of DDPG and how hyperparameters are chosen. Is it possible that conclusions about learning from pixel versus state are different if hyperparameters are different? Is this a general statement about these unsupervised losses? In general, quite a lot was run for this work, with some relatively strong conclusions; it could actually be better to run less and be more thorough in testing each question. If you do in fact think you have strong evidence for each claim, then it would be useful to add more justification for why you believe this experiment provides that evidence. 

In this work, only the problem of continuous control is considered as part of the benchmark and only a certain variant of DDPG is considered as the optimization algorithm. How critical to the code-base is it that (a) DDPG be used and (b) the environments be for continuous control? It is understandable that a benchmark be reasonably scoped, but it would be useful to discuss if this choice is central. For example, do the authors think that representation learning for continuous control is a more interesting direction to pursue? Or was it simply a scoping choice that needed to be made?

Finally, some of the choices for the number of steps in learning should be better justified. What was the intuition behind selecting these specific values for pre-training and fine-tuning steps? It is possible that some algorithms might perform quite different for different such choices. Once again, it is not feasible to check all combinations, but an explanation for these choices is important. For example, maybe you choose them based on a the performance of a baseline algorithm. 

correctness: The benchmark seems to be well-designed and the evaluation is mostly reasonable. However, as described under weaknesses, the work is a bit imprecise about certain choices that need clarification.
clarity: The paper is clear, albeit some choices not explicitly explained. 
relation to prior work: There does not appear to be a discussion about other benchmarks. It seems natural to discuss some multi-task RL benchmarks, like Procgen. I understand that here the work is also about providing standardized implementations of the algorithms themselves.
additional feedback: Minor comments:
1. Why is the standard error not visualized in figure 4 of the paper? They are important to visualize because the validity of 1-7 observations depends on them. 
2. In line 79, “with intrinsic rewards $r^{ext}_t$” should be replaced by “with intrinsic rewards $r^{int}_t$.
3. The paper says: ““This suggests that we are still far as a community from efficient generalization in deep RL.” It is a bit strong of a claim. Does it need to be said?
4.It would be interesting to see the learning curve of the unsupervised approaches, especially during the fine-tuning process. It would be great to include these in the paper, even if just in the appendix.
review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper focuses on evaluating unsupervised RL, specifically in the context where pre-training has been applied to agents so that it can help with generalization. Evaluating these methods has been challenging due to the lack of standard benchmarks and metrics. To this end, this paper introduces a benchmark for unsupervised RL that DeepMind's Control Suite. This benchmark is evaluated with existing algorithms and shows that we need more work in this space, which is a good motivation to use the benchmark to help solve this problem. The codebase is open source and publicly accessible. 
strengths: + Good problem statement - making the development and comparison of unsupervised RL algorithms easier. 
+ Open source benchmark with eight different algorithms in the benchmark
+ Baselines for the tasks are provided
weaknesses: - A bit of discussion about how the benchmark may evolve and how will it be kept up to date as the field evolves will be useful. 
- Evaluation section could have been a bit more detailed. 

correctness: Looks good to me. 
clarity: The paper is well written and the appendix provides all the necessary details about the experiments and evaluation. 
relation to prior work: All the relevant related work is fairly well described by the paper. 
additional feedback: I would have appreciated more details on how the agents are learning. The results focus pretty much on the summarized end result but that intermittent training process is quite helpful to understand for readers/reviewers. I would have appreciated that as a reader.

One related topic/work that might be worth discussing is the implications of sim2real gap issues that generally plague RL. This doesn't come up anywhere, and it would be useful to discuss that. 

In the presentation of the benchmark, I would have appreciated calling out the metrics a bit more in detail. Drawing out the attention that would be useful and important because a benchmark is more than just its tasks and algorithms. It is how we measure/assess "performance."
review_title: URLB: Unsupervised Reinforcement Learning Benchmark 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a new benchmark for unsupervised RL, evaluating the effectiveness of pre-training RL agents w varying self-supervised intrinsic rewards to enable improved sample efficiency in following fine-tuned tasks. The benchmark is building on top of the DeepMind Control Suite and contains a selection of twelve continuous control tasks as well as consistent implementations of eight unsupervised RL algorithms implemented on top of DDPG. An evaluation is presented for all tasks and provided algorithms with several observations to guide future research.


strengths: The field of unsupervised RL has become increasingly popular with a wide array of evaluation protocols. Therefore, a unified evaluation protocol and benchmark appears well justified and valuable to the research community. The protocol appears sensible (even though potentially limiting -- see comments below) and open-source URL algorithms are provided which will help researchers to start working on these problems.

Also, the list of observations are appreciated as they can guide future research directions. 
weaknesses: **Benchmark Protocol**

While fully unsupervised pre-training is certainly one interesting approach to leverage innovation in unsupervised RL, some approaches (in particular Knowledge-based baselines) are introduced to train with extrinsic and intrinsic rewards alongside each other. It might be worth considering such an evaluation setup as well (which seems like little additional implementation would be required).



**Evaluation:**

- Figure 4 lacks a point of reference for performance. I'd suggest to include a horizontal line indicating the normalised returns achieved by DDPG (with random initialisation/ without any pre-training) trained for the equivalent 100K steps in the testing task.
- Suggest to add standard deviation for normalised returns at each pre-training setting across all 4 tasks within the respective domain to indicate expected deviation across these runs.
- A precise explanation for the applied normalization scheme would be appreciated. It appears that asymptotic DDPG performance is the point of reference (l. 209) at 100% but this is not explicitely stated. Also, it might be valuable to give an indication how many steps of training are required for DDPG to achieve such asymptotic performance (presumably >> 100K) to get an understanding of the benefits achieved by unsupervised pre-training.



**Evaluation Observations**

- While these observations are certainly helpful, some lack substance. See the following examples
    - O4: do you have any insight into why competence-based approaches underperform? Are these approaches potentially not well suited for the evaluation protocol of unsupervised pre-training and then fine-tuning in separate tasks? Potentially they focus more on learning generalisable skills in contrast to enabling quick fine-tuning?
    - O6: Do you have any insight into why performance does not keep improving as more pre-training is executed? Are intrinsic rewards potentially degrading and thereby leading to a loss of previously learned skills or representations? It might be worth investigating the trend of intrinsic rewards achieved throughout training to investigate such questions.
correctness: The claims made in the paper appear correct and sensible. For any criticism in the design of the benchmark and evaluation, see weaknesses section.


clarity: The paper is written clearly.

One typo I noticed in l. 78f: "In this work, we pre-train agents with **intrinsic rewards $r^{ext}_t$ ** and fine-tune [...]" --> should be $r^{int}_t$
relation to prior work: Related benchmarks and work within the domain of unsupervised RL are clearly stated. As mentioned in "Weaknesses", it might be worth expanding the benchmark to different settings applying unsupervised learning signals in RL outside of pre-training.


additional feedback: No additional comments.
review_title: Relevant Benchmark for increasingly popular domain of unsupervised RL with some restrictions

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Authors present URLB, a benchmark for unsupervised RL, enabling to compare algorithms producing intrinsic rewards.

The test procedure consists in 1) a pre-training phase on a given environment, without extrinsic rewards, then 2) a short fine-tuning phase for a specific task in the same environment (3 environments and 12 tasks are proposed based on DMlab), and finally 3) a post training performance assessment on this task.

Authors add 8 leading intrinsic motivation algorithms in the benchmark, and perform experiments with all of them, showing that URLB is an appropriate testbed for future work.
strengths: This article provides a significant contribution towards simplifying comparative analysis between intrinsic motivation mechanisms, which is a very important step towards the development of robust multi-purpose learning architectures.

Code will be release as an open-source repository, which is good, and it appears to be properly documented.

weaknesses: ## RIDE (potential major issue)

My main concern about this work is that the RIDE algorithm ("RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments" Raileanu et. al, ICLR 2020), is not included in the benchmark. This algorithm was shown to outperform both RND and ICM, which are featured in URLB. I am willing to increase my score if authors manage to include RIDE into URLB during the rolling discussion, or if they provide a clear justification on why it does not make sense to include it.

## scripts for figures (minor)

Does your open source codebase contains scripts to reproduce easily your performance graphics ? Although minor, this is a very helpful addition for future benchmark users.

## aggregated graphics (minor comment)

A minor weakness is also the current status of the main body of the paper, which only reports aggregated results. Benchmark papers are often used by researchers for "shopping", i.e. to select the most efficient algorithm for their problem. I understand that given the breadth of the experimental campaign, aggregated graphics are needed. It is good that per-algorithm results are in the appendix. Besides, algorithm recommandations are made in observation 5 of the experimental section. That being said, I wonder whether adding something more visible in the conclusion might be helpful. Again, this is minor, feel free to keep the manuscript as is.
correctness: Evaluation methods and experiment design are both appropriate and well introduced
clarity: This paper is very well written.

I only noticed two typos:

* typo in figure 3: "extrinisc rewards" --> extrinsic
* l.284 "by a the human task designer" --> remove "a" or "the"
relation to prior work: Related work is properly discussed.

Including "Large-Scale Study of Curiosity-Driven Learning" By Burda et al. might be appropriate.
additional feedback: About observation 3 of the experiments: you say that the performance gap from state-based vs pixel-based tasks is due to the quality of the exploration mechanism. Couldn't it just be a matter of the underlying DDPG-like agent which is less efficient at learning in pixel settings w.r.t. state settings ?
review_title: An appropriate and interesting benchmark

Meta-review: This paper proposes a benchmark for evaluating unsupervised RL agents - i.e. agents that learn from intrinsically generated reward signals or other kinds of pre-training before being exposed to the final task. Developing such endogenous RL agents is important, however progress is hard to measure without a comprehensive benchmark, so this submission is very well placed. All of the reviewers agreed that this is a solid submission, evaluating a large number of algorithms on a good set of tasks. For this reason I recommend acceptance.
Decision: Accept