Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents an interesting variational bound on the computationally intractable empowerment criterion.
It then gives a simple algorithm for learning options to maximize meta-controller empowerment, in the absence of an external reward signal.
This approach seems novel, insightful and likely impactful, and the results look promising.

The first part of the paper is very clear, principled and promising.
Starting from the right part of Figure 1, through Algorithm 2, to some of the results in Figure 2, the descriptions are vague or missing.

While the lower bound is computationally tractable to update iteratively, the convergence properties are unclear. Algorithm 1 is attempting to estimate an informational quantity, and these require notoriously high sample complexity (Paninski, 2003).

Pros:
- Principled approach
- Computationally tractable lower bound
- Promising results

Cons:
- Unclear convergence properties
- Figure 1 (right part) and Algorithm 2 are inscrutable (with undefined notation)
- Important details are missing of the experiments in Figure 2 (architectures, number of iterations, etc.)
review_title: Great approach, promising results, space too limited for proper presentation

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper tackles the options discovery problem through the notion of 
"empowerment", formalized in information-theoretic terms. The paper builds 
on similar recent ideas, such as Mohamed & Rezende (2015), but proposes 
a tractable objective : a variational lower bound on the 
mutual information between options and the set of reachable states upon termination 
of that option. A policy gradient approach is used to optimize this objective. 

While the idea of using policy gradient methods for options discovery has 
recently been explored by other authors, one of the contribution of this particular
paper is the definition of a new objective, beyond the usual expected sum of 
discounted rewards. In fact, as shown in Bacon & al. (2017), the expected discounted
return is not sufficient to guarantee temporally extension and regularization
might have to be employed. The proposed variational lower bound could provide 
such regularization. 

Another contribution of this paper is the idea of generalizing options 
from discrete entities to "embeddings" aka. families of options.  This is a 
generalization which some other authors have also recently started adopting
(cf. Schaul & al (2015), Borsa & al. (2016), Oh & al. (2017), Vezhnevets & al. (2017))
but I appreciate that this paper develops it more clearly in the text.
review_title: Tractable objective, options embeddings

Decision: Accept