Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a modification of the option-critic algorithm for hierarchical reinforcement learning. The proposed algorithm modifies how the termination conditions of the options are improved by experience. Specifically, the algorithm aims to maximize the mutual information between the options and their termination states. The authors develop an optimization scheme for achieving this objective and provide empirical results in a number of domains

The empirical results help understand the scalability of the approach but they are less useful in evaluating how useful the proposed algorithm is in general. The experimental domains used in the paper are not as diverse as one would wish to see. In addition, across domains, the general structure of the task is the same: there are multiple goal regions offering different amounts of reward; the challenge for the agent is to not get distracted by the lower offerings. So it would be informative to see the algorithm analysed in a more varied set of problems, including problems with a single goal region. Also informative would be to execute the algorithm with varying numbers of options and examine the effect. 

The logic behind using uncertainty-aware option selection is not clear. Further discussion and exploration of it in the paper would be useful. 

Bottom of page 4: “By adding the uncertainty bonus, this strategy encourages selecting lesser-known options.” Given  that $\mu(o, s_t)$ is the probability of choosing option $o$ in state $s_t$, which is not directly related to how well known an option is, it is not clear to me how Equation 12 encourages selecting “lesser-known” options. 

Bottom of page 5: “Thus, an agent is required to switch its behavior around a suboptimal goal (e.g., from “going down” to “going right”), which options can help.” This needs clarification. How do options help with this? And why is that not achievable with primitive actions?

In Figure 2, OurAOC lags behind AOC. This could be noted in the paper and the reasons can be discussed. 

Figures 2 and 6 are very useful.

Section 2.3,  perfoms --> performs
Section 2.3,  terminatio --> termination

Author feedback: The behaviour and performance of the algorithm in single-goal environments should be part of the paper. The paper should show not only where the algorithm succeeds but also where it fails. I appreciated the author's efforts to add diversity to the domains evaluated but they do not go far enough to change my score. 
review_title: The paper proposes a variant of the option critic algorithm for hierarchical reinforcement learning. It would benefit from an analysis of the behaviour of the algorithm in a broader range of problems.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Review for "Diverse Exploration via Infomax Options" 

Thank you for your feedback. I found that the analysis of the method in the revision is informative. However, the comparison with the baselines is still lacking, and the experiments are only performed in simple environments. For these reasons, I keep my rating unchanged after the rebuttal.

=============

Summary:
This paper studies the problem of discovering options for exploration in reinforcement learning. To this end, they propose infomax criteria that maximize the information between options and option-terminating states conditioned on the option-starting states. The paper derives update rules for the infomax criteria and applies the gradient descent algorithm to optimize options. The proposed method is evaluated at Gridworld/Mujoco four rooms tasks and Point Billiard environment, and the authors show that the resulting options are diverse.


Pros:
- The information maximization criteria for diverse option-terminating states is novel.
- The paper does a nice job describing the required background materials in great detail. 
- The paper clearly describes the objective and the corresponding update rule with implementation details. 


Cons:
- The experimental results are too weak to prove the effectiveness of the proposed method. In detail, the paper's experiment tasks are too simple, e.g., all tasks' goals are to reach one of the four points in a simple square room. Moreover, from Figure 3-(b), it doesn't seem that the proposed method (PPIMOC) outperforms the baselines. 
- The performance comparison with important baselines such as termination critic (TC) is absent in the experiments. 
- The paper suggests some techniques such as uncertainty-aware option selection or upgoing option-advantage estimation, but the paper lacks ablation studies.
- It is not clear that maximizing the mutual information in Equation (7) leads to diverse options. The paper should provide theoretical results or more experimental analysis to make the case.


Comments:
- Experiments on more challenging tasks, such as Montezuma's revenge from Atari games, could improve the claim of the paper. 
- I recommend performing an ablation study of the proposed algorithms (in Section 4) to isolate the effect of each technique.
- I am not fully convinced that information maximization leads to diverse options. Aren't there any problems associated with the collapse of options or terminating states? 

(minor)
There are some typos.
- End of Section 2.1: "with and o and not"
- Equation (4) and the following sentence: Q_O(x) -> Q_O(x,o)
- Last paragraph of Section 2.3: "Contrary to the terminatio"


In summary, the paper suggests a new perspective on diverse options via infomax criteria, but the experimental results are too weak, and the analysis of the method is lacking. 
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper seeks to learn diverse options for exploration,
principaling by combining two techniques: the termination gradient
theorem (Harutyunyan et al. 2019) and empowerment (in the spirit of
VIC/DIAYN). This is an appealing idea, and the approaches to
empowerment (such as for DIAYN) are a great fit for the limitations of
the Harutyunyan work.

However, there were a number of issues I found with the paper which
currently leaves me leaning more towards rejecting in the current
form. Again, solid idea, but both the paper itself and the
experimental results (thus the method) need further work.

Comments/Questions:

- Just above Prop 2: The use of the discriminator / classification
model here should be attributed to related work.

- Uncertainty-aware option selection: How is this like UCB? This is
not 'uncertainty', it is the stochasticity of the policy, so this
actually just looks more like entropy regularization as it is
frequently used. Am I missing something?

- Section 5, Gridworld:  Do episodes end as soon as the first non-zero
reward is found? I don't think that this is a particularly good choice
of domain for studying exploration.

- "A2IMOC performed the best and found the optimal goal in most trials."
I don't see any evidence for either part of this statement. Judging by
Figure 1, none of the methods are reliably achieving the 2.0 goal. And
judging by the error bars, it is difficult to say that A2IMOC is
better than AOC. I'd add that looking at Figure2, the policies learned
by the different options for A2IMOC look to terminate in different
states, but appear to have essentially the same behavior policy. This
looks like one of the common failure cases of OC, where the options
either learn the primitive actions (e.g. Figure 2, AOC) or all learn
identical policies (e.g. Figure 2, A2IMOC).

As with Figure 1, the learning curves in Figure 2 do not allow us to
draw any real conclusions about PPIMOC being an improvement over PPO.
Though, Figure 3, does seem to show some significant improvement in
this non-stationary task setting. But this is perhaps the only
positive empirical result, and thus is a lot to hang the paper on.

- Appendix: epsilon-greedy ablation looks to be actually performing
the best, why not use this instead of the UAOS?

Pros:

Interesting idea, very appealing intuitively
Small, but highly relevant, theoretical contributions

Cons:

Experimental results are wholly unconvincing
Uncertainy-aware option selection needs to be reframed
Writing could use some work, especially around putting this in the
proper context of existing work.

review_title: Very nice idea, but unconvincing results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This is an interesting paper that investigates the use of options for improved exploration by encouraging diverse termination functions via a mutual information measure. The paper is well written and mostly clear (with a few points below):

Overall I think the idea has merit, but I think the experiments fall short of demonstrating its performance. What seems to be lacking to me is:
1. Demonstrating performance with stochastic transitions (I believe all experiments are with deterministic transitions).
2. Overcoming the coarse approximation for the infomax objective (7) (see point 5 below).
3. Clarifying the details of the classification model, which seems central to the algorithm (see point 2 below).

Some questions for the authros:
1. What is the law of $X_S$ in equation (7)? In particular, if $X_f|X_s$ is the random variable denoting a state transition, wouldn't this just be the stationary distribution of the current policy? Further, in Proposition 1, the first state $x_s$ isn't sampled (at least not by $\pi^o$ nor $\beta^o$), so it's not clear what the random variable $X_S$ represents.
2. Right before section 4, the authors write "learning a classification model over options $\hat{p}(o|x_s, x_f)$ from sampled option transitions". This seems central to the algorithmic performance, but it's not clear how it gets used.
3. In equation (12), in the condition for $b_t = 1$, it seems it's saying the same option ($o_t$) should be picked again. Is this the case?
4. Above equation (13), the authors write "Supposing that the current option ot terminates at the $t + k$ step". How is this calculated? This seems especially problematic with stochastic transitions.
5. In page 5 the authors write "[$I(A; O | X_S)$] can be interpreted as a local approximation of the infomax objective (7), assuming that each action leads to different terminating regions." This seems like a rather rough approximation; in particular, doesn't this simply encourage diversity of one-step action transitions as opposed to option termination?
6. In Figure 1b the difference between A2IMOC and A2C does not appear to be statistically significant (there is major overlap of the confidence regions). Does this not suggest merely marginal gains?
7. In Figure 2, were all algorithms using 4 options?
8. Were any of the environments evaluated with stochastic transitions?
9. How many options were used in the MuJoCo environments?
10. What do the "sample" indicators mean in your proof of Lemma 1? This suggests the proof is mostly a modification of the proof in Harutyunyan et al. (2019).
11. Do you have a similar plot as Figure 8, but for the MuJoCo experiments?

Minor issues:
1. Given that you're dealing in episodic tasks, below equation (1), shouldn't the $Q^{\pi}$ values be indexed by timestep?
2. In the definition of $V_{\mathcal{O}}(x)$, it should be indexed by $\mu$.
3. Right above section 2.2, remove the extraneous "and".
4. In equation (4), $Q_{\mathcal{O}}$ is missing the second parameter.
5. You should state equation (6) as a proper theorem.
6. Equation (10) is over the margins.
7. In the first sentence of the paragraph above equation (14), remove the extraneous "common"
8. In the section titled **MuJoCo Point Four Rooms** there is a typo, it says "Figure 3b" but it should say "Figure 3a".
9. In Related Work, under **Options for Exploration** you could also cite some related work:
  - "Using bisimulation for policy transfer in MDPs", Castro & Precup, AAAI 2010
  - "Automatic construction of temporally extended actions for mdps using bisimulation metrics", Castro & Precup, EWRL, 2011

review_title: Official Blind Review

Meta-review: The paper introduces a variant to the option-critic framework that encourages options to display a certain level of "diversity" and this is induced by introducing a mutual-information objective between the options and their transitions. The authors conjecture that such criterion makes options more suitable for exploration.

Overall, reviewers agree that the idea behind the proposed method and the general approach is sound and interesting. Nonetheless, there is general consensus that the current submission suffers from a number of shortcomings that make it unsuitable for acceptance.

Following the detailed comments provided by the reviewers, I strongly encourage the authors to focus on the following dimensions to improve the paper:
1- The current experiments indeed provide a first illustration of how the proposed algorithm works, but they need significant improvement in variety and scope: As pointed out by the reviewers, the current experiments do not cover single-reward challenging exploration benchmarks (such as Montezuma). I agree with the authors that the inductive bias implemented in their algorithm is designed with diversity of goals in mind, but if the main point is to improve exploration, it is natural to expect results in that respect. Alternatively, the authors should state more explicitly the type of problems their method is intended to solve from the very beginning of the paper and design experiments accordingly.
2- The initial mutual information objective is simplified across multiple steps and it is unclear how much the approximations impact the original "semantic" of the objective.
3- A more thorough comparison with mutual-information-based methods such as DIAYN or VIC is needed. Also, I wonder what is the connection with more goal-based exploration approaches such as GoExplore or SkewFit.
Decision: Reject