Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: To tackle the unsupervised skill discovery problem, the authors attempt to maximize the mutual information between (latent) skills and states $I(\tau; z)$ by using the nonparametric particle-based entropy estimation for $\mathcal{H}(\tau)$ and noise-contrastive estimation for $\mathcal{H}(\tau|z)$. They evaluate their method and the baselines on the Unsupervised Reinforcement Learning Benchmark (URLB), where the agents are pre-trained without extrinsic rewards and fine-tuned for downstream tasks with extrinsic rewards and fewer environment steps.
main_review: **Strengths**

* The experiments are done on URLB, which provides a good evaluation scheme for unsupervised RL. Well-established and common evaluation schemes are important for assessing methods empirically.
* The empirical performance, which is provided with relevant statistics, is good compared to multiple baseline methods.
* The analysis on the effect of different choices of multiple hyperparameters (Fig.6).

**Weaknesses**

* This work's originality is somewhat limited. Particle-based entropy maximization with state representations trained using a contrastive learning scheme has been explored in APT (Liu & Abbeel, 2021a). Using the same entropy maximization form for skill discovery was done by APS (Liu & Abbeel, 2021b).
* The motivation for using noise-contrastive estimation is not entirely clear. While the authors make a comparison with CPC as a lower bound of mutual information, CPC is not very commonly used in skill discovery and the usual variational lower bound can already be tight if the variational approximation $q$ approximates the true distribution $p$ perfectly.
* Theorem 1 is not technically correct because of the particle-based entropy term in $F_{CIC}$, which is also supported by the authors in that they introduced the weighting hyperparameter $\alpha$ due to that it doesn't consider the proportionality constant (Sec.4.2).
* The authors employ $I(\tau; z) = \mathcal{H}(\tau) - \mathcal{H}(\tau|z)$ as the decomposition of the mutual information (the 2nd line in Sec.4.1), but they use $q(z|\tau)$ instead of $q(\tau|z)$ for the rest of Sec.4.1.
* I think the derivation of the noise-contrastive estimator in Sec.4.1 needs more details. For instance, if there are noise samples, where do the correct samples come from?
* Using the notation $\tau$ for something other than trajectories can be misleading.
* Lack of empirical analysis with different values of $\alpha$.
summary_of_the_review: While the empirical results basically show that CIC can outperform multiple baselines on URLB with an appropriately tuned value of $\alpha$, I am mainly concerned about the correctness of the claims and the novelty of the work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper builds upon the DIAYN idea (Eysenback 2018) that an agent could develop skills in an unsupervised environment by finding a set of skills that collectively visits the whole state space but encourages each skill to cover a different subspace and later use one of these skills to simplify the learning of a downstream task. In this paper skills are learned in an unsupervised way using a mutual information based objective. The mutual information between latent states T and skill vector Z, I(T;Z), is decomposed as I(T;Z)=H(T)-H(T|Z) which the paper argues leads to explicit maximization of diversity of latent states T as well as distinct skills with focused effects by penalizing with H(T|Z). More diversity equals better exploration of more distant states leading to more interesting behaviors. The paper explains this decomposition also allows them to user higher-dimensional skill vectors that improve representational capacity and downstream performance. The paper develops a method to calculate the terms of this decomposition and formally shows it is a lower bound for the true mutual information. The latent state entropy H(T) is estimated using an unnormalized k-nearest neighbor method requiring an ad hoc scaling factor \alpha and the conditional entropy H(T|Z) is calculated using an NCE supervised neural net.  The paper illustrates the method by using their loss function to pretrain a DDPG architecture on unsupervised scenarios and then showing a benefit on downstream tasks. The proposed method, CIC, shows a larger IQM stabilized expert normalized score compared to state of the art methods.  Experiments also show that high-dimensio
main_review: The empirical evaluation is rigorous in comparison to common practice (120 runs and IQM stabilization) with strong SOA and baselines. 

Liked the structured review of prior art that organizes work in an interesting thematic way. 

I liked the empirical decomposition of reward into entropy and discriminator terms. It is interesting that latent state entropy is important early in learning and task discrimination is more important later on. 

Useful to know that higher dimensional skills are important (64D performed best). 

Technically I(T;Z)  =  H(T) - H(T|Z)  =  H(Z) - H(Z|T) so theoretically it should not matter which way it is decomposed. The way the terms are calculated, however, could have a significant effect on the practical performance. 

It isn’t clear what a “simple grid sweep of skills over the interval [0,1]” means.  

Early stopping does not “leak information” so much as stop wasted exploration in irrelevant parts of the state space? 

Typo: “we use particule” -> particle 

The text refers to optimality gap, but the figures see to use expert normalized score which I assume is algorithm performance / DDPG baseline performance. 

Did not fully understand the implications of the noise and projection argument, but it seems like an important and worth while design decision to investigate. 
summary_of_the_review: The paper proposes a new algorithm for unsupervised behavior learning that is rigorously shown to be more effective and clearly argues for its design choices through supplementary experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an algorithm (Contrastive Intrinsic Control) for unsupervised skill discovery by maximizing mutual information between skill latents and state transitions. The proposed algorithm is a refinement over existing methods [2]. It uses a contrastive method to estimate conditional entropy and measures entropy on state-transitions as opposed to simply states as done in previous methods [2]. The proposed algorithm shows good performance gains compared to existing competence-based skill discovery algorithms. Further, the paper also contains a rather extensive empirical evaluation of various skill discovery algorithms on the recently proposed URLB benchmark. 

main_review: Strengths:

The empirical evaluation carried out in the paper is extensive with many baseline algorithms for skill-discovery from each class (knowledge-based, data-based, competence-based) on the recently proposed URLB benchmark. Further, evaluation metrics (IQM, Optimality gap etc.) used to measure performance are adopted from the recommendations in [1].  

The discussion and analysis on the reasons for failure of current competence-based skill discovery algorithms. Ablation experiments (in Figure 6) help justify the design choices made by the CIC algorithm. 

Weaknesses:

The proposed algorithm seems like a variation on an existing algorithm [2]. It refines some of the practical design choices used in the general framework of competence-based algorithms for unsupervised skill discovery. However, the authors have shown the differences between the various algorithms (in Table 1) and discussed their pros & cons which is useful to contextualize their contributions.

A few questions I have for the authors on the high-level motivations behind their algorithm design and would like some clarification on:

1] If we intuitively think of the notion of a skill as a form of abstraction of long-term behavior. For example motion primitives like walking, flipping etc. as shown in the paper which occurs over maybe a few tens or hundreds of steps. So, why do several skill discovery algorithms use only highly localized information in the state space (various tau instantiations such as single states or in this case single state-transitions (s, s’)) to infer its corresponding skill latents? Isn’t it more intuitive to infer these latents from more “global” quantities like entire episodes of policy rollouts. Could the authors comment on this? 

2] To maximize the mutual information between state transitions and skills as defined In the CIC, we need to maximize the first term ($H[\tau]$) and minimize the second term ($H[\tau | z]$). This conditional entropy would be minimal when the corresponding distribution $p(\tau | z)$ is sharp/narrow (ideally like a delta-like density function) over the state-transition space. This seems rather counterintuitive to me. If we think of a single skill latent, say walking, shouldn’t the density $p( \tau | Z=z_{walk})$ have a high value for all the possible state-transitions of the walking primitive, which would be a rather wide distribution. Wouldn’t the pressure to keep this distribution as narrow as possible over the state-transition space given a single skill latent lead to several latents codes which essentially cover the same underlying behavior (redundant copies of different walking styles). Wouldn’t this be undesirable for generalization on downstream tasks which would require composing these unsupervised skills?    

3] The authors argue for the need for increasing the dimensionality of the skill latents to ensure skill can be decoded back to a diverse set of behaviors. Couldn’t this be achieved by largely retaining the small latent spaces used for skills in prior work and using a more expressive policy decoder. This could allow for greater representation flexibility when the skill latents are decoded back to the action-space and ensure that skill latents give rise to a diverse set of behaviors.  

Writing/Presentation:

The paper on the whole is well-written and easy to follow.  

I found the phrasing in this sentence rather confusing and ambiguous. “If the set of behaviors outnumbers the set of skills, this will result in degenerate skills -- when one skill maps to multiple different behaviors”. What does “behaviors” refer to in this context -- action trajectories? If so, isn’t it expected that the set of skills would be much smaller (essentially a compressed representation) than the total number of “unique” action sequences. Unless, the authors  

A few minor typos I found are below:

“Why most competence-base algorithms …. ” -> “competence-based algorithms … ”
“Both CIC and CIC use ...” -> do you mean CIC and APS?
In Algorithm 1 -> “Contrastive Intrinisc Control” -> “Intrinsic”

[1] Agrawal et.al, “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, NeurIPS 2021.

[2] “APS: Active Pretraining with Successor Features”, ICML 2021.

summary_of_the_review: Although the proposed algorithm CIC is a variation on an existing algorithm [2], it shows impressive performance gains over several existing algorithms on a large suite of continuous control tasks from the URLB benchmark. These large-scale empirical evaluations and analysis of several algorithms for unsupervised skill discovery methods on a standard benchmark such as URLB would benefit the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of unsupervised pre-training of a (code-conditioned) policy to improve the performance of downstream RL tasks. In line with previous works in unsupervised skills discovery, it proposes a method, called CIC, to maximize a variational lower bound to the mutual information between the code and the visited states. The lower bound is obtained through the combination of non-parametric state entropy estimation and a contrastive predictive coding loss for the conditional entropy. The paper provides an empirical analysis of CIC over a set of continuous control domains.
main_review: I report below some detailed comments and concerns that the authors might address in their author response.

MUTUAL INFORMATION OBJECTIVE

1) There is now a bunch of works targeting the mutual information between a code and the visited states to the purpose of unsupervised RL (some of them are summarized in Table 1 of the paper). Every work is proposing a lower bound to the mutual information. I was wondering if there is a formal way to compare the lower bound of CIC with previous works. Which one is the tightest? Do the authors believe that getting the tightest bound of the MI is really the goal in this setting, or even maximizing the exact MI would not be necessarily better than other methods (i.e., a good inductive bias matters more than the MI approximation)?

2) Since the work motivates the approach as an approximation of the MI, as it is common in previous works as well, I was wondering if the authors also considered a direct non-parametric estimation of the MI (e.g., https://journals.aps.org/pre/pdf/10.1103/PhysRevE.69.066138) instead of independent estimates of the state entropy and the conditional entropy. Especially, such a direct estimation would not require the additional hyper-parameter $\alpha$.

METHODOLOGY

3) Can the authors clarify what is the entropy term $H(\tau)$ denoting? Is it the entropy of state-to-state transitions, or the entropy of the joint probability of two states within a trajectory, or something else? Especially, what is the intuition behind using $\tau$ instead of $s$?

4) The method is built upon a non-specific base algorithm (DDPG) that was originally developed for standard RL, i.e. RL problems where the reward does not change over time but it comes from a consistent reward function. Do the authors experienced any instability working with the non-Markovian intrinsic rewards? Do they believe that the methodology could benefit from an objective-specific algorithm, i.e., an algorithm carefully designed to work with this kind of intrinsic reward?

EXPERIMENTS

5) How can we rule out the possibility that CIC is just a better way to pre-train a DDPG agent in these settings w.r.t. other baselines? This would be significant anyway, but do the authors believe that the same results would generalize to different base algorithms (say TRPO, SAC, A2C...)? 

6) Moreover, DDPG is known to be quite strong on continuous control tasks. Do the authors believe that the combination of DDPG and CIC would be successful in different settings (e.g., visual discrete domains such as Atari games) as well? Or perhaps the base algorithm should be selected to accomodate the specific domain?

7) CIC is quite similar to APS (Liu and Abbeel, 2021), as they both employ non-parametric entropy estimation and a discriminator loss, based on contrastive predictive coding and successor representations respectively. However, in the reported results CIC is way better than APS. Do the authors think it is the different discriminator loss the main cause for this performance gap, or there is some other factor at play? Can they confront the discriminator rewards of APS and CIC (as in Figure 5)?

8) From my understanding, the empirical results are not directly comparable with the URLB (Laskin et al., 2021) despite a very similar setting. I see that the benchmark is very recent, and thus should be considered concurrent to this work, but I believe that reporting a direct comparison with their results would further strengthen the empirical analysis.

9) The results section seems to imply that a key factor under the improvement over previous competence-based methods is the ability of CIC to cope with a larger skill space. Can the authors clarify why previous methods are prevented to work with a comparable skill space? Can they also provide a comparison with previous methods when working with the same (potentially lower-dimensional) skill space?

MINOR
- Adaptation efficiency paragraph: Fig. 3 is reported instead of Fig. 4;
- The normalized score of Fig. 6 does not seem to match the one of Fig. 4;
- It is not easy to track the different baselines in Fig. 4 (top). What is this plot representing?
summary_of_the_review: To my understanding, the main selling points of this paper are:
- It tackles the very relevant problem of unsupervised pre-training for reinforcement learning;
- The methodology is clear, and a quite natural extension of previous works in unsupervised skills discovery literature;
- Strong empirical results, CIC seems to advance significantly the state-of-the-art performance of unsupervised pre-training in continuous control domains.

Instead, potential shortcomings are:
- The novelty seems limited, as CIC is essentially similar to APS (Liu and Abbeel, 2021) with a different discriminator loss (which has been employed for unsupervised skills discovery before);
- It is not completely clear from the paper what are the specific factors that lead to such a performance improvement over previous works.

Whereas the reported empirical progress might be a sufficient reason for acceptance, my current evaluation is just slightly positive in consideration of the mentioned concerns. I do not think the limited novelty is a crucial problem here, if the authors could better clarify in their response how the CIC methodology is so successful, I will consider raising my score to a clear accept.
review_title: null

Meta-review: The paper addresses the question of skill discovery in reinforcement learning: can we (without supervision) discover behaviors so that later (when supervision is available via a reward signal) we can learn faster? The paper proposes a new contrastive loss that an agent can optimize for this purpose, based on a decomposition of mutual information between skills and transitions. The reviewers praised the extensive experimental evaluation and good empirical results, as well as the analysis of failure modes of related algorithms.

Unfortunately, there appeared to be errors in the derivation and implementation. (These include typos in derivations that made them difficult to follow, as well as uploaded code that didn't match the experimental results.) While the authors claim to have fixed all of them, the reviewers were not all completely convinced by the end of the discussion period. In any case, these errors caused confusion during review; so, whether the errors are fixed or not, it seems clear that there hasn't been time for a full evaluation of the corrected derivations and code. For this reason, it seems wise to ask that this paper be reviewed again from scratch before being published.
Decision: Reject