Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper considers the problem of exploration and skill discovery in unsupervised reinforcement learning. To this end, the paper proposes CIC, an algorithm to explore novel states as well as distill the experience into reusable skills, which can be efficiently adapted to downstream tasks. This work builds upon the ideas of several previous works that maximize the mutual information between (some function of) the visited states and the latent skill vectors. These works differ in the different design choices for estimating the mutual information (generally it is a lower bound), as succinctly summarized in the paper in lines 74-76. 

CIC estimates the entropy term, $\mathcal{H}(\tau)$, using a particle estimate similar to some previous work. The contribution of the paper is the use of a contrastive density estimator, specifically the NCE loss proposed in CPC (Oord et al, 2018), for the variational term in the MI lower bound. The unsupervised pre-training phase involves optimizing the CPC loss for contrastive representation learning between state transitions and skills as well as maximizing the intrinsic reward. The fine-tuning phase involves training the agent (with a few steps) using only the external task reward. 

This approach is shown to outperform similar skill discovery methods on the URLB benchmark using DDPG as the backbone, and extensive experiments investigating some design and hyperparameter choices have been performed.
strengths_and_weaknesses: The problem of exploring novel states and generalizing to multiple downstream tasks are central challenges in RL, and skill discovery algorithms provide an appealing solution to both these problems. There are several works which aim to maximize (some lower bound on) the MI between states and skill vectors. In this regard, the paper has somewhat limited novelty. This work proposes maximizing MI explicitly using CPC loss for representation learning (in addition to the intrinsic reward) and the contrastive density estimator used to estimate the conditional entropy term is also well known. Nevertheless, a combination of well-known techniques can still be valuable, provided it is supported with substantial theoretical and empirical analysis. The paper performs many experiments to showcase and analyze the proposed method, however, the theoretical justification is lacking and unclear in a few places.

The paper contextualizes their work well by providing a succinct and fairly complete overview of similar work (Table 1 is quite helpful). The authors provide intuition-based but clear motivation for their design choices in Section 2, which sets up the proposed method well. However, parts of Section 3 are not clear, particularly the use of CPC loss as the variational distribution, mismatch between the specification of $F_{CIC}$ and its computation in the pseudocode, and the connection to MI maximization. The intrinsic reward mentioned from Section 3 onwards is not defined until Section 6, which might be confusing for the reader. It is also not clear what intrinsic reward is used in the ‘default’ CIC algorithm (used in Fig 4, 5, 6) - I assume it's just the entropy term since it is shown to be the best performing variant in Section 6. If so, the discussion about the variational lower bound on MI and the contribution of using a contrastive density estimator are made redundant. The experimental results, notwithstanding the issues mentioned here, are extensive and the ablation studies for various design choices are insightful.
questions: **The use of CPC loss as variational distribution:** The noise contrastive density estimator used for $q(\tau|z)$ and the representation learning loss is exactly the same, which is the CPC loss from (Oord et al., 2018). The CPC loss is a lower bound on the MI, so it is unclear why this should be a good choice for the variational distribution. Moreover, replacing Eq. (3) in Eq. (2) contradicts the result of (Oord et al., 2018). Since CIC frames itself within the MI maximization family of skill discovery algorithms, it is crucial that the connection between the proposed method and MI maximization is presented as clearly as possible.

**$F_{CIC}$ definition vs. implementation:** The notation in the definition of $F_{CIC}$ is not clear, and there seem to be inconsistencies between this definition and the pseudocode provided for calculating this loss. The index $i$ is not specified, but assuming it is the batch index, it would seem that $F_{CIC}$ is calculated for some $z_i$ with $\tau_i$ as the positive sample and the remaining $\tau_j$ in the batch as negative samples. The pseudocode, however, samples a batch of $\tau_i$ and a batch of $z_i$, similar to how most contrastive learning methods (simCLR, CURL etc.) implement this loss.

**Intrinsic reward specified not specified clearly:** The intrinsic reward is specified in the middle of the experimental results section as a written description, rather than explicit mathematical definitions.  Also, it is not clear which intrinsic reward is used when the authors compare CIC with the baselines or perform ablations (such as in Fig 4, 5, 6). It is appreciable that the paper considers a variety of intrinsic rewards to understand which one works best, however, I believe that specifying them clearly would help improve the readability of the paper.

**CIC used in experiments does not use intrinsic reward to maximize MI:** The paper presents CIC as part of a family of methods that maximize the MI using intrinsic reward (Table 2), however their best performing model (and presumably most of their empirical results) considers only the entropy term as the intrinsic reward. This disconnect between the theory (which describes a novel method of estimating the variational lower bound on MI), and the experiments (which seem to not use this variational lower bound) significantly hurts the quality of the paper. If my understanding is correct, the intrinsic reward closest to the MI maximization objective would be (i) discriminator (from line 253-254), which is also the worst-performing variant of CIC. In addition, the contribution of using a contrastive density estimator in the variational lower bound on MI is made redundant.

**Minor comments:**

- Typo in Line 67, should direct to Table 2 in Appendix A.
- CPC loss, referred in line 165 and numerous other places, is not defined. I assume it refers to $F_{CIC}$, which is same as the CPC loss.
- $N$ is not defined for Eq. (3), assumed to be the batch size.
- The experiment plots, Fig 4, 5, 6, are present on different pages and going back and forth between Section 6 and the plots hurts the readability of the paper. I suggest moving the plots closer to the relevant text.
limitations: The authors acknowledge some limitations of their work, particularly that the method is described for MDPs (not POMDPs), and that they do not consider visual inputs in their experiments. The authors have also given though to the potential negative societal impacts of their work (really unsupervised RL in general), which is appreciable. 

The main limitations of the work as per my review are the somewhat limited novelty in the proposed method, some questions on the theoretical soundness, and the discrepancy between the method described in the theory section and the one used in the experiments. There are also some concerns regarding the notation and writing, which are relatively minor and easily fixable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a contrastive-learning based method that encourages behavioural diversity for unsupervised RL (where unsupervised means without reward). Their method operates by maximizing the mutual information between state-transitions and skill vectors. The authors provide experimental evidence with the Unsupervised Reinforcement Learning Benchmark (URLB) for the efficiency of their proposed method by examining whether pre-training agents with their method can better adapt to downstream tasks.
strengths_and_weaknesses: # Originality
This work is mostly an extension of the work of Liu & Abbeel 2021 ([15] and [27] in paper) where the main contribution (as acknowledged by the authors) is in a novel _discriminator_ (where the discrimination is between latent skills). Their proposed improvement is also based on existing work (e.g. the noise-contrastive estimator of Gutmann & Hyvärinen 2010 ([23] in paper), the parameterization of the function for density estimation of Chen et al. 2020 ([34] in paper)).

However, in my opinion this is an original and novel combination of existing methods towards addressing a problem for which (most of) those methods were not originally designed.

# Quality
Overall I feel the quality of the paper is reasonably good. There are a few issues with clarity that I mention below. I appreciated the ablations and sensitivity analyses done in Figure 5, as it really helped motivate the design choices made in the paper. Finally, there are a few questions related to experiment quality in the Questions section belo.

# Clarity
This is perhaps the weakest point of the paper (although Figure 2 is really nice!). The writing oscillates between proper mathematical notation and a github README file (e.g. "$r,s'\sim$ `env.step(a)`" in line 57). Further, there are some improperly specified mathematical objects in the writing, which I list below:
*  In line 53, you say "$r$" is "sampled" from `env.step(a)`, but a few lines above you stated that $r$ was the reward function.
*  In line 55 it says "$\tau(s)$ ... refers to any function of the states $s$". This is ill-defined and should be more specific. $\tau$ is a function of what to what? In line 57 you're using $\tau$ as a tuple (e.g. $\tau=(s, s')$), so it's not at all clear what it's meant to represent.
*  In line 60,  $\mathcal{Z}$ has not been introduced.
* The paragraph in lines 86-95 is not at all clear because it's not clear what $\tau$ is.
* In footnote 4 in page 4 it says "Note that $\tau$ is not a trajectory but some function of states" which, again, is not at all clear what it's meant to represent.
* It's not clear what is meant in lines 173-174. In particular what are "negatives" and "positives"? Is $N$ the same thing as $N_k$ in the equation?
*  One method of finetuning is presented in lines 195-204, but another one is presented in lines 223-224. It's not clear which one was actually used for the experiments, and if both were used, when each was used.
* The histogram in Figure 5(d) (and accompanying discussion) is not clear. You're evaluating over a pre-selected grid of latents, and the histogram shows the performance of each of these grids? It's not clear how this is measuring the effect of skill fine-tuning?

# Significance
This work is addressing an important problem for a large portion of RL research (lifelong learning, zero-shot transfer, generalizability, etc.), and as such can have an important impact on the community. The empirical evaluations presented in Figure 5 are quite useful, as they help shed light on the most necessary components of the proposed algorithm.
questions: 1. In equation (2), what is the expectation over?
1. Is the second to last line of the pseudocode on page 5 correct? It's not clear that the `cross_entropy` call gives you $F_{CIC}$. In particular, it seems like it's missing the first term?
1.  In line 277 it says "most likely because this reduces the diversity of the skill vector" to justify the projection, but it's not clear why this projection is necessarily reducing the diversity of the skill vector?
1. In the "skill architecture and adaptation ablations", did you try running DIAYN with larger skills?

limitations: Both limitations and potential negative societal impacts were discussed adequately (in my opinion) in section 7.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes CIC, a competence-based unsupervised RL method that maximizes the mutual information between latent skills and state transitions. CIC outperforms related baselines on the Unsupervised RL benchmark (URLB) where the agent only has access to the extrinsic reward for a short adaptation phase in the downstream tasks.
strengths_and_weaknesses: ***Strengths***

This study has significant strengths in the well-established experimental section.

1. Decent experimental setup

The URLB is a well-established benchmark to evaluate the unsupervised adaptation performance of RL methods. I appreciate the comparison between Gym and DMControl.

2. High statistical significance

I appreciate the authors running experiments across as many as 10 seeds, and reporting IQM scores. This provides high statistical significance for the results compared to other methods that report only mean results with 3~5 seeds. Also they compared many relevant prior works.

3. Ablations

Because there are many overlapping contributions with existing studies (APS, APT, CURL,..etc.), it is easy to give a mixed feeling of contribution. But the authors managed to prevent this through ablations in Figure 5.


***Weaknesses***

The weaknesses of this work lie in the originality and motivation. As this is a completely experimental paper, I understand that theoretically, it is difficult to prove the strength of this study. I felt the connection between the motivation and contribution seemed a bit vague.

1. Originality

As the authors are aware, there were previous works that use particle-based entropy or contrastive learning for unsupervised learning (APT, APS, CURL). The novelty of CIC is that it is the first that used contrastive learning between state transition and latent skill. The absence of a related work section may make it feel that novelty is lacking. From this point of view, I appreciate the comparison made in Appendix D. Maybe it would be better if the authors add such content to the main manuscript with a little more detail.

2. Motivation

The two points made in the motivation section ("Competence-based algorithms do not ensure diverse behaviors", "Why it is important to utilize high-dimensional skills") are indeed very important and useful points. But there seems to be a lack of a link between these motivations and the contrastive learning and particle-based entropy. Some questions remain about whether contrastive learning and particle-based entropy are the "only" or the "best" ways to estimate $H(\tau|z)$ and $H(\tau)$, respectively. Some of these concerns seem to be addressed in Appendix L, which I think would be better to be in the main manuscript.

questions:  1. I'm curious how the estimation of decomposed mutual information in Table 3 would distribute across different methods. And how CIC may outperform baselines.

2. The decomposition of mutual information used by CIC encourages diverse behaviors through the entropy term $H(\tau)$. I'm curious whether there is a trade-off for losing some diversity of the skills $H(z)$?

3. I see that previous works employ too small skill dimensions and it's great that CIC employs a larger skill dimension. But the skill dimension should not be too large as well, so there should be some sweet spot that may require hyperparameter tuning. Since the tasks targeted are unsupervised adaptation, does it make sense to tune the dimension without the knowledge of the diversity of the downstream tasks?
limitations: The authors have adequately addressed the limitations.
review_title: null

Meta-review: metareview: This paper presents an algorithm for representation and skill learning in RL, based on a contrastive approach. Experiments combining this method with different intrinsic rewards are presented, along with a number of sensitivity analyses. 

Overall, the reviewers recognize that there are novel components here despite the work being an extension of existing work. There is some disagreement with regards to the value of these novel components, in particular whether the specific discriminator used is sufficiently evaluated, and whether the proposed method is well-motivated (i.e. by theory). While I am sensitive to these concerns, I find that the paper does a reasonable job at detailing the proposed agent architecture and provides some fresh insights into unsupervised RL.
confidence: Less certain
Decision: Accept