Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents an improved algorithm based on CIC (Laskin et al. 2022), to balance surprise maximization and minimization for RL pertaining. Experiments on a variety of tasks show the effectiveness of the proposed methods.
strengths_and_weaknesses: The motivation of balancing maximizing and minimizing surprises is well explained, but it's unclear how the proposed method resolves the issue and how much it is different from a scaled CIC. I suggest the author make a clear distinction between MOSS objective and a linear interpolation of CIC and negative CIC (or equivalently a scaled CIC), and also show in the experimental section how the prior distribution of M affects MOSS performance.
questions: - It would be better to add MOSS results in Figure 4b) for direct comparison. It makes sense that MOSS_SAME performance is in between CIC and negative CIC, the question is why MOSS can do better. One would argue that in expectation MOSS is still just a linear interpolation of CIC and negative CIC (or a scaled CIC). Why is this not the case?
- How do you sample M? Suppose it's sampled from Bernoulli(p). Does setting p=0 give you CIC and settting p=1 give you negative CIC? What are the optimal p's for different environments? Would that somewhat correspond to environmental stochasticity based on your assumption?
limitations: Limitations and the potential negative impact were discussed at the end of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new method called Mixture of Surprises (MOSS), which tries to maximize and minimize the surprise at the same time. This results in state-of-the-art performance on the URLB benchmark. 
strengths_and_weaknesses: Strengths: 
The paper is well written and covers the related literature quite well. The experiments are well done on variety of different environments. The proposed method also performs well on the URLB benchmark. 


Weakness: 
Already exist methods which try to maximize and minimize surprise at the same time. Thus, it is difficult to find novelty in the proposed method. 
questions: Can this method also be used with demonstrations? 
limitations: The authors should try to motivate the novelty of the paper a bit more.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a method for unsupervised skill discovery via reinforcement learning. Unlike most prior methods which seek to either maximize or minimize surprise, this paper proposes to learn a mixture policy where one component of the mixture is trained to minimize surprise, and another component is trained to maximize surprise. The paper builds very closely on the work of Laskin et al [1] (CIC), and the proposed change to CIC is very small (just a one-line change in Algorithm 1). Essentially, the proposed method partitions the latent skill space into two parts, where one part is trained to maximize surprise, and the other part if trained to minimize surprise. Whether the agent chooses to sample from the surprise-maximizing part of the skill space or the surprise-minimizing part is somewhat arbitrary: the agent uses surprise maximizing skills for the first half of an episode, and surprise-minimizing skills for the second half of the episode. 

The authors evaluate their method on the URLB benchmark and VizDoom tasks, and show that their method slightly outperforms CIC on both unsupervised learning metrics as well as downstream task performance. The main advantage of the method is that, unlike CIC, we do not need to pre-specify whether the agent should minimize or maximize surprise, and the algorithm is able to do well in environments that require either. 

[1] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CIC: Contrastive intrinsic control for unsupervised skill discovery, 2022.
strengths_and_weaknesses: Strengths
- Simple, easy-to-implement method. 
- A somewhat more general technique for unsupervised reinforcement learning as compared to prior methods. 
- Writing quality is good. 

Weaknesses
- The choice of when to switch between a surprise minimizing and maximizing objectives is somewhat ad-hoc. Authors investigate a slightly more sophisticated procedure in the Appendix, but find it to be very sensitive to hyperparameter choices, and omit it from the main paper. 
- Overall writing quality is good, but the authors spend a lot of time (2.5 pages) going over background material and summarizing past work, while the main method section is about 1 page long, including an algorithm box. The main method section does not start until the second half of page 5! Given that the paper builds very closely on CIC, a more detailed exposition on CIC and limited discussion of other techniques would have sufficed. 
- Some of the related work that also proposes to maximize and minimize surprise was only mentioned in the last section. I think this should be moved up to be in Section 3 instead, where other related works are discussed. 

questions: In the Section 5, you say “but performed well on the standing task, which is inherently a low entropy task.” → did you mean to say high-entropy task instead? 

limitations: I don't think the limitations are particularly well-addressed in the paper. The main limitation mentioned was that the proposed method is only applicable to episode RL settings (and not reset-free settings), but this seems slightly orthogonal to the main issue of the paper. I think a more detailed discussion on how to make the objective switching more adaptive would make the limitations section much stronger.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an unsupervised RL algorithm for pretraining a policy. The algorithm is motivated by learning to both maximize and minimize the mutual information between latent skill variables and the state-transition joint distribution. The method approximates this mutual information following prior work, and proposes a learning algorithm that simply splits up the pretraining episodes into two phases: the first phase with intrinsic rewards corresponding to maximization, and the second phase corresponding to minimization. Experiments on the environments and tasks from the benchmark of [28] illustrate that this method achieves superior finetuning results compared to a variety of other methods. Ablations illustrate that this method has less degradation in performance than a maximizing phase alone (corresponding to prior work) when additional stochasticity is added to the transition dynamics, and an ablation that illustrates the importance of having the latent skill priors be different.
strengths_and_weaknesses: # Strengths
- The area of investigation is relevant to the community. Pretraining / unsupervised learning for RL agents is an active area of research interest.
- The proposed method is technically sound.
- The proposed method is technically straightforward.
- The proposed method demonstrates compelling performance compared to prior work.
- The experimental evaluation uses compelling aggregate statistics
- The experimental evaluation compares to a large variety of prior methods

# Weaknesses
- There is some undefined terminology. "Competence"-, "Data"-, and "Knowledge"-based methods were never defined. This makes some parts hard to understand, e.g. L99 "To motivate the potential of competence-based methods over data-based or knowledge-based methods". I recognize that these terms come from CIC, but this paper should either restate the definitions or refer to CIC.
- I think the objective function is mischaracterized. L183 "In particular, we define surprise as the state-transition entropy, which corresponds to maximizing and minimizing $H[S', S]$". Unless I misunderstand the notation, this is the entropy of the *joint* distribution $p(s',s)$, which is *not* the entropy of the state-transition dynamics distribution $p(s'|s,a)$, nor the entropy of the state-transition dynamics induced by the policy $p(s'|s) = E_\pi p(s'|s,a)$ (which are the only two distributions that I would refer to as being "state-transitions"). Therefore, I would not say that the objective is the state-transition entropy. It is *related* to the state transition entropy, since $H(S',S) = H(S'|S) + H(S)$, i.e. it is the entropy of the induced Markov dynamics plus another entropy. The problem with referring to this as only the "state-transition" entropy is that it fails to capture the fact that the value of H(S) affects the resulting value, so I think it is a mischaracterization of the objective. I think it is important to be very precise about this naming, because, in general, optimizing $H(S'|S)$ will lead to very different results than optimizing $H(S'|S)+H(S)$. 
- The connection between the original objective and the resulting optimization procedure is unclear. The objective is $I((S', S) ; Z) = H((S', S)) - H((S', S) | Z)$. The CIC paper, upon which Alg 1 is based, uses both a noise-contrastive estimator (for the latter term) and a KNN estimator (for the former term). Alg 1 states the intrinsic reward is computed using Eq. 9, but this seems to only involve the noise constrastive estimator. Is Alg 1 correct? Where is the KNN estimator?
- The experiments do not adequately illustrate the effect that finetuning has on the pretrained policy. It is possible that the pretrained policy does not benefit from finetuning in some cases, e.g. taking $z \sim Z_{min}$ might lead to a high score on environments where the downstream tasks is very related to entropy minimization. Can you add to the evaluations two baselines for non-finetuned policies constructed from each of the $Z$ distributions (i.e. one for $\pi(a|s, z\sim p(Z_{max}))$, another for $\pi(a|s, z\sim p(Z_{min})$)? These should not be finetuned, instead, simply illustrate how related the learned policy is to the downstream task. I would call these something like $MOSS_{max,frozen}$ and $MOSS_{min,frozen}$ . Note that this doesn't require any additional learning, just an evaluation of these existing policies. Beyond this, you could also include the finetuning learning curves in the appendix, although that's a bit less important.
- The current ablation does not illustrate the importance of the choice of the structure of the Z priors, which includes both the dimensionality $d$, as well as the event space. One thing that seems most natural is simply to take z \in {0,1}, where 0 defines 'minimizing' and 1 defines 'maximizing', i.e. use $P_{min}(Z) \doteq \delta(z=0)$, and $P_{max}(Z)\doteq \delta(z=1)$. It is not clear what the additional dimensionality of $Z$ is useful for (the appendix tells us its 64 dimensional). Some discussion and empirical are needed to justify this choice. It could be the case that very low dimensional Zs are still useful in some (or all) tasks, intuitively, these tasks might be those in which there is only one 'useful' way each to minimize or maximize entropy, rather than environments in which repertoires (>=2 each) of entropy maximizing and minimizing skills is useful. Defining the skill variables to be continuous essentially admits learning an infinite number of skills, but it is not clear that this is the best choice.
- Although this paper pointed out the relevance, it did not include an empirical comparison to [18], which is also motivated by the task of learning to both maximize and minimize rewards. The inclusion of this comparison would significantly strengthen the empirical results.

# Minor weaknesses
- Please add the main method's scores to Figure 4(b).
- By the paper's definition of surprise as the entropy of the joint distribution of state transitions, the method is not optimizing *only* surprise (since it's just one of the two terms of the mutual information). The method name then becomes something of a misnomer...
questions: - Please address the weaknesses:
  - Can you fix the terminology about the three types of methods?
  - Can you fix the characterization of the objective function?
  - Can you fix the ambiguity in the connection between the objective function and the optimization procedure?
  - Can you include the two baseline evaluations I requested?
  - Can you include the ablations I requested?
  - Can you include an empirical comparison to [18]?
- L170: "Intuitively, the NCE loss [ 23 ] pushes p(s′ | s, z) to be a delta-like density function. However, since we define behaviors with n steps, we want a wider distribution" The second sentence does not follow from the first. Even if p(s'|s,z) is deterministic, it can still be applied for N steps, to yield an N-step behavior. Can the paper clarify these statements?
- Can you explain the following inconsistency? L260 claims that setting Z_min = Z_max is equivalent to CIC, yet this results in different empircal performances in Fig 4b. By L260, we should expect their results to be equivalent. Why are they not? Or perhaps it's not equivalent to CIC, because it involves flipping the sign of the intrinsic reward function?
limitations: Yes.
review_title: null

Meta-review: metareview: This paper proposes a method for unsupervised skill discovery, which learns a mixture of policies that simultaneously maximizes and minimizes the surprise. All the reviewers agree that the paper tackles an important active research area. The paper is well written; the motivation is well explained; the proposed method is simple and easy-to-implement; and it performs well on the benchmark. Several concerns were raised by the reviewers, including novelty, delta beyond the interpolated CIC, and the choice of M. The rebuttal and the additional experimental results have addressed some of these concerns. One of the main criticisms, choice on when to switch the objectives, still remains after the rebuttal. Although the current heuristic is simple, it seems ad-hoc, without good justifications. After  discussion with reviewers, we agree that this limitation is compensated by the other contributions of the paper, and helps to set the stage for future work that does optimize M or devises a new method that does not rely on it. Thus, we recommend accepting this paper.
confidence: Certain
Decision: Accept