Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: EDIT: The qualitative results help illustrate what the variational bias entails in practice, and indeed the worse coverage constitutes a problem worth overcoming. The Ant experiment was a good attempt at showing scalability, but the deterministic version isn't terribly informative since then the correction term does nothing. Could add stochasticity by taking some number of random actions between the states the agent sees. I suspect that as you increase stochasticity in this way the uncorrected method would degenerate. Would be a clear accept if you could show that, but as is the paper's contribution is bordering on acceptance. 5-->6  

The authors show that implicit VIC is biased in stochastic environment due to its blindness to the effect of its 'option' on the state transition dynamics. This is addressed by learning a model of these dynamics to allow for the calculation of the missing terms. Toy experiments are then performed that show the boost in mutual information caused by eliminating this bias.

First off, its worth mentioning that this is the first real investigation into what intrinsic VIC actually optimizes. The original VIC paper only really explains things for the explicit case, so even the description of what implicit VIC is trying to do is a novel contribution of this work.

That said, the primary merit of implicit VIC was its scalability, and the new requirement of a generative model of state dynamics can only hurt this. For this paper's extension to be truly significant, it should show a case with significant state cardinality (e.g. the 3D environment from the VIC paper) where the bias hurts more than the reliance on the learned model. The GMM approximation suggests this could be possible, there aren't any experiments that really require its usage.

In addition to the scale of the experiments, the breadth of evaluation could be expanded -- showing a gap in the MI is suggestive of a more more interesting gap in behavior. For example, can you use the reverse predictor q(a | s, s_f) to reach all possible states by forcing s_f to equal an arbitrary state? 'Percent of states empirically achievable' should be an easy metric to evaluate in these toy environments and would strengthen the case for your extension.

I have a few minor complaints about the analysis itself. While generally easy to follow, the notation is a bit cumbersome. Once the options are explained in terms of the underlying trajectories, why stick with the option notation? Eliminating the Omegas from the loss terms would be the implications of your extension much for explicit, though perhaps this would just make everything a bit too ugly. I don't follow the significance of equation 5, and would appreciate it being unpacked a bit more. All of the options considered are fixed-length in practice, so I'd omit all of the bits regarding termination actions. What's the difference between showing an upper-bound on the approximation error versus a lower-bound on the true value? If the extension isn't a lower-bound, then the implications of this should be explained.

Overall, I like this paper and wish more papers would be like this. Illuminating a previously neglected algorithm is worthwhile and I want to reward that. But suggesting a model-based approach for an algorithm which cited its model-free nature as a primary motivation requires a more thorough investigation. I'm convinced that you've found a flaw, but I'm not convinced your solution actually improves things. 




review_title: Technically sound extension to an under-explored algorithm. Additional motivation would help with significance.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary of paper:
The authors study a version of VIC that represents options as partial trajectories. They point out that, in stochastic environments, the implicit VIC formulation in the original paper is missing a term in the mutual information (involving log likelihood ratios of state transitions). They introduce two ways of estimating the missing term: one appropriate for discrete state spaces, and another appropriate for continuous state spaces. They then compare the empowerment of implicit VIC with and without their corrections in a few toy domains, showing that their corrections do not hurt in deterministic environments, and provide a small increase in empowerment in stochastic environments.

Pros:
The missing term highlighted, and the derivations of solutions generally looked correct (at least at the level I followed them). This is a potentially interesting contribution.

Cons:
1) The experiments are a) done only in toy domains and b) even there demonstrate only a 5-10% improvement in empowerment. These experiments are maybe fine as a sanity check, but they are not enough to demonstrate the importance of the authors’ correction term. This is because empowerment is not an important objective in and of itself. Empowerment is used as a unsupervised pre-training step for RL, or as an exploration bonus in conjunction with RL. It is useful to the extent it aids performance in those settings. It is not clear to me whether the correction term is important in those pursuits. It could even be that focussing on the part of empowerment due to stochastic state transitions actually degrades the usefulness of the learnt policy. Additional experiments are needed.

2) In addition to experiments justifying the extra term, it would also be useful for the authors to include more motivation and intuition about why and when it is important. It is not immediately intuitively clear to me why incentivizing an agent to drastically alter its trajectory based on random state transitions is useful.

3) In general, the paper is hard to follow. There are long blocks of equations without enough exposition. New terms are defined without motivating first why they are being introduced. The notation is often too dense (e.g. p^p_p? really?). I mostly felt “in-the-dark” as to where the authors were going while reading the paper. Perhaps they could streamline the derivations, push some of it to the appendix, and spend more time in the main text on motivation and intuition.

4) Given the environments aren’t standard and are very simple, they should definitely be introduced in the main text, and not pushed to the appendix.
review_title: Needs more experiments and motivation

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary

The paper points out a limitation of the implicit option version of the Variational Intrinsic Control (VIC) [Gregor et al., 2016] algorithm in the form of a bias in stochastic environments. Two algorithms are proposed that fix the limitation: the first requiring the size of state space to be known and the second which does not make such assumptions. Experiments on simple discrete state environments demonstrate that the original VIC algorithm works well only on deterministic environments whereas the proposed fix works well on the stochastic environments as well.

## Strengths
- The paper provides a sound theoretical analysis of the limitation of the VIC implicit-option algorithm, the proposed fix and a practical algorithm (*Algorithm 2*).
- A clear distinction is presented with respect to prior work. The differences between the proposed algorithm and VIC shows that the intrinsic reward now has an added term which depends on an approximate model of the transition probability distribution.

## Weaknesses
- The paper focuses on a very specific and narrow topic without providing much stand-alone motivation for the same. It implicitly borrows motivation from prior work (VIC, etc) without providing its own. The rigorous mathematical derivations are simply re-deriving the VIC mutual information bounds with a new added term and with some extra details on how to do it with a gaussian mixture model. Overall, the paper seems like a minor extension of prior work.
- Considering the experiments on partially observed environments presented in the VIC paper, this paper chooses a much simpler set of discrete environments for empirical analysis instead of stepping up to more complicated environments which would have strengthened both the motivation for fixing the bias of VIC and the empirical evidence of the GMM algorithm (*Algorithm 2*).
- The paper's contributions boil down to Eqn 6 (the bias in VIC) and the two proposed algorithms. However, the difficulty in reading the mathematical notations and expressions severely handicaps the reader's ability to carefully understand these contributions. Some suggestions on improving the notation are provided below.

## Feedback to authors
- The paper introduces extremely dense notation. The frequent overloading of symbols or use of similar looking symbols (e.g. $p, p^p, \rho$) makes it quite difficult for the reader to parse each expression. I would recommend usage of longer variable names, e.g.: replace p -> gen, for generative model and replace q -> inf for inference models. Phrases are easier to parse than single character symbols. Also, colorizing certain important symbols can help -- especially for important distinctions such as the true probability distributions vs their estimates.

-------
## Post-rebuttal update

Having read through all reviews and the author's response, I am updating my assessment in light of the responses and new experiments. I agree with the authors that the derivation has theoretical value and is not a simple re-derivation of VIC. The new experiments and visualizations have been helpful (I am happy with the author's responses to R3), but the overall clarity of the paper is still lacking due to the dense mathematical notation. In light of this, I am increasing my score from 4 -> 6, slightly leaning towards acceptance.
review_title: A weakly motivated extension of VIC with implicit options

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the problem of maximizing empowerment in the context of RL, where the aim is to maximize the mutual information between some latent variable and future outcomes (e.g., future states). The paper first observes that a procedure proposed in prior work [Gregor 16] is biased and hence does not recover a (latent-conditioned) policy that maximizes mutual information. The paper then proposes a new method, based on learning the transition dynamics. that does recover the optimal (mutual information maximizing) policy. Experiments on a few simple tasks show that the proposed method outperforms prior methods.

**Significance**: Empowerment remains one of the main methods for autonomous skill discovery. Thus, a better understanding of how to optimize empowerment would be an important contribution to this area. This paper identifies a limitation (a biased objective) in a commmon formulation of empowerment [Gregor 16], and proposes a method to correct for this. I think the significance of this paper hinges on (1) how large this bias is for reasonably complex tasks, and (2) if this type of bias might occur in other RL objectives, besides empowerment. The paper only convincingly shows that removing this bias is useful for maximizing empowerment on small scale problems.

**Novelty**: To the best of my knowledge, this limitation of VIC has not been discussed in prior work.

**Experiments**
* It would be great to include visualizations or ablation experiments to illustrate why implicit VIC has a lower empowerment than the two proposed methods.
* It'd also be good to include *explicit* VIC as a baseline, even though it requires pre-specifying the number of skills.
* The experiments are limited to very simple gridworlds and tree domains.

**Clarity**
* The derivation of the variational bias in S2 is pretty hard to follow. I'd recommend including a bit more discussion of what implicit VIC is and how it differs from explicit VIC, before continuing with the formal derivation.
* S3 and especially S4 are also hard to follow. I'd recommend moving most of the derivation to the appendix and just stating the final objective as an equation. Theoretical guarantees can then be stated as Theorems/Lemmas with proper proofs.

Overall, I give this paper a score of 5 / 10, primarily because of (1) a lack of clarity and (2) the limited experiments. I would increase my score if the clarify of writing were (greatly) improved, if experiments on higher dimensional tasks were added (e.g., see those in [Achaim 18, Eysenbach 18]), and if additional visualizations of the (suboptimal) behavior of implicit VIC were added.

**Questions for discussion:**
* How significant is the bias in implicit VIC [Gregor] in more complex tasks? Is it significant enough to warrant the additional complexity of the proposed approach?
* Is the GMM approach in S4 just a special case of the model learning approach in S3, where the model is taken to be a GMM?
* Does *explicit* VIC have the same bias as *implicit* VIC?


**Minor comments**
* "methods for measuring it" -- This makes it sound like Arimoto + Blahut proposed methods for measuring empowerment. I'd revise to "along with methods for measuring it based on Expectation Maximization [Arimoto + Blahut]"
* "This can severely degrade the empowerment" -- Clarify what this means.
* "This type of option differs..." -- Aren't there two differences? (1) Closed loop options depend on the state at each time step, and (2) these options include a termination condition.
* " which hinders the maximal level of learning" -- Add a citation.
* "implicit VIC which defines the option as the trajectory until the termination" -- This sentence is confusing without knowing about the method apriori.
* "becomes possible to learn the maximum number of options for the given environment" -- Add a citation.
* "achieve the maximal empowerment" -- It'd be good to formally define what the "maximal empowerment" means.
* "environment dynamics modeling incorporating the transitional probability" -- Grammar error.
* "is the inference to be trained" -> "is the inference network/model to be trained"
* Eq 4: Where are \tau_t and s_{g | \Omega} defined?
* In S3, it might be clearer to use q(...) instead of p^q(...).
* In Eq 15, it's unclear how \log p_\rho^p depends on \theta_\pi^q.
* "when the cardinality of the state space is unknown" -- Where does the *cardinality* of the state space show up as a dependency? Perhaps what is meant is that the method is most readily applicable to *discrete* settings, where the distribution over future states can be approximated exactly, without sampling.
* S4: I'd recommend providing some intuition for why this alternative method is being derived. Is it going to address a limitation of the method in S3?
* "Gaussian Mixture Model (GMM) (Reynolds & Rose, 1995)" -- I think GMMs existed before 1995. E.g., see early work by Karl Pearson.
* "extreme gradient" -- What is an extreme gradient?
* "revisited the variational" -> "revisited variational"

----------------------------------
**Update after author response**:  Thanks to the authors for answering my questions during the rebuttal paper and for incorporating the feedback into the paper! My original concerns were about clarity, high-dimensional experiments, and visualizations. Since the paper has been revised to include nice visualizations and improve the clarity, I am increasing my score 5 -> 6. 

I think the experiments on HalfCheetah are a great proof-of-concept of the method! I'd encourage the authors to include some comparisons against baselines for that task.
review_title: Review of "Variational Intrinsic Control Revisited"

Meta-review: This paper revisits the under-explored "implicit" variant of Variational Intrinsic Control introduced by Gregor et al. They identify a flaw that biases the original formulation in stochastic environments and propose a fix.

Reviewers agree that there is a [at least a potential, R4] contribution here: "even the description of what implicit VIC is trying to do is a novel contribution of this work", in the words of R2, and "the derivation has theoretical value and is not a simple re-derivation of VIC", in R4's post-rebuttal remarks. Several reviewers raised significant concerns around clarity, which were addressed in an updated manuscript, which also provided new visualizations and new experiments which reviewers found compelling. All reviewers agreed that the revised manuscript was considerably improved.

R4's score stands at the 5, with the other reviewers all standing at 6. R4's main concerns are around whether the missing term in the mutual information identified by the authors is a problem in practice on non-toy tasks (echoing somewhat R3's concerns re: high-dimensional tasks). While this is a valid concern, the function of a conference paper needn't necessarily be to (even attempt) to provide the final word on a matter. Identifying subtle issues such as the one brought forth in this manuscript and re-examining old ideas is a valuable service to the community, and this paper will serve as a beginning to a conversation rather than an end. The AC also considers themselves rather familiar with the original VIC paper, and found the results herein somewhat surprising and noteworthy.

I recommend acceptance, but encourage the authors to incorporate remaining feedback in the camera-ready.
Decision: Accept (Poster)