Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an approach for learning diverse temporally extended and reusable options. It is based on the assumption that learning diverse options are generally useful for downstream tasks. Thus, the approach aims to discover options without making any assumptions about the downstream tasks. The main idea behind the approach is to discover options by maximizing the mutual information between the options and the corresponding state transitions and demonstrates the options discovered by this approach. Also, the paper demonstrates that these options are useful for faster learning in a downstream task. 

main_review: The paper presents an extension of the termination critic idea, which is called the InfoMax Termination Critic. Specifically, the approach is different from Termination Critic as it involves computing the entropy over the final states as a function of the option and the start state (termination critic only computes the entropy of the final states as a function of the option).

The overall approach is clear and well-presented.

I do have a number of questions related to this approach and would like some clarifications from the authors: 

1. The Fig 2 presented shows qualitatively how the terminations for options look for IMTC and Termination Critic. Would it be possible to demonstrate a similar visualization on the standard 4 room gridworld, identical to the one presented in Termination Critic’s paper? It would be much easier to make a comparison with the published result as opposed to reimplementing termination critic and demonstrating the differences on a new but smaller gridworld.

2. If I understand the approach correctly, the terminations for the options are discovered through IMTC but the policy for those options are discovered through Variational Intrinsic Control (VIC). Is my understanding correct? 
This seems like a roundabout way of discovering options that are diverse. Why doesn’t the approach learn option-policies where the rewards for those options are obtained through the discovered termination functions?  
Also, this makes it unclear if the performance gains are due to the discovered terminations or due to VIC. Perhaps adding an ablation to address this is necessary to make it clear.

3. Would it be possible to make Fig 3 more prominent? It is hard to understand what each of the option-policies are looking like. Also, how are the options being discovered for OC + VIC? I thought that OC requires extrinsic rewards (from tasks) to discover options and in the experiments considered in this paper, there are no extrinsic rewards (i.e., reward-free RL). 

4. Because the approach is very much related to Termination Critic, I think it is important to include Termination Critic as a baseline in the large scale experiments (the plots in Fig 3 and Fig 5). Without this result, it is difficult to understand how the proposed work is better than its prior work.

5. What are the error bars in Fig 5? 

summary_of_the_review: Overall, I think the paper addresses an important problem in RL: discovering diverse options when there are no external tasks. The paper presents an adaptation of Termination Critic. 
However, some of the design choices made by the approach does not seem to be reasonable. Specifically, why use Variational Intrinsic Control in combination with the proposed approach when there is a direct way of learning option-policies from the discovered termination functions. Also, using VIC in this way makes it unclear as to whether the performance gains are due to the proposed approach or due to VIC.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an algorithm, termed infomax Termination Critic (IMTC) to learn diversified options in RL. This algorithm learns termination conditions of options by maximizing mutual information between options and corresponding state transitions. The experiments demonstrate the IMTC algorithm learns diversified options and can be reused in various tasks.
main_review: Strength: 
1.	the method is well motivated and derived 
2.	Empirical study in various domains show diversity of the learned options and its usefulness in transfer settings. 

Weakness:
1.	The method is incremental and seems trivial. The method is directly derived from the termination gradient theorem from Harutyunyan et al. (2019) with reward signal constructed based on mutual information between terminal state and option. 

2.	Failed to mention and compare some relevant references, E.g., reference [1] also considered learned diversified options using pseudo-reward constructed based on divergence between action distribution of different options. 

[1] Kamat & Precup, Diversity-Enriched Option-Critic, Neurips2020

[2] Ramesh et al., Successor Options: an option discovery framework for reinforcement learning, IJCAI2019

I would like to see some discussions of these references and experimental comparisons.

3. This work is mostly based on empirical study, the paper could be strengthened if some theoretical analysis, such as sample/computation complexity, convergence etc. can be provided. 

Other comments:
In the abstract, “our experiments demonstrate … without rewards”,  you really mean “without environmental/extrinsic rewards”.
Proposition 2 seems simply a trivial extension of proposition 1. 
In the experiment (5.1), why the policy over option $\mu(o|x)$ is fixed to be $1/|O|$, instead of learned?

summary_of_the_review: In all, this paper studied an important problem in reinforcement, learning diversified options that can be reused in various tasks. Their empirical study is able to justify the diversify of the learned options and shows superior performance in transfer learning settings comparing to some baseline.  However, there are still some key references missing and should be compared against. Also, the method seems trivial extension of many existing methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an objective for learning a diverse set of options in which the goal is to maximize the entropy of terminating states from any initial state while at the same time minimize the entropy of terminating states given a specific option. 
Intuitively, this means learning options that tend to be deterministic, while at the same time are able to reach diverse terminating states.
The authors then evaluate against other option learning frameworks in a series of diverse environments.
main_review: Overall I enjoyed the paper, it is well organized and well written, and it touches in a topic that I personally find very interesting which is options learning.

Strengths:
- Well written paper and easy to follow.
- Straight forward objective with a nice intuitive explanation. Should be easy to re-implement for the reader.
- Evaluation done in a set of diverse tasks.

Weakness:
- Maybe a concrete definition of what's meant by diversity in this paper could help. The objective encourages the options to learn different terminating states, but not different trajectories. So, based on the objective two options that "move right" for 38 steps, and one moves "up" on step 39 and the other moves "down" on step 39 would be diverse, and would maximize the objective.
If this is not a behavior that does not happen in practice, it would be nice to have a discussion around it, since nothing strictly prohibits this from happening. 

- Where the number of options pre-defined? If so, how did you pick that number?

- On eq. 9, what is the assumption on the reward function to take the maximum over 0 as optimistic. What happens if all rewards are positive in a task? Taking the max over 0 would probably not help with understimation.

- Looking at the results, it's interesting to see that "reach" tasks with large action spaces is where the proposed method tends to underperform. Does the complexity of the task or the action space play a role in how difficult it is to learn options? I would like to see some move in-depth analysis on the results.

- Following on the results, I would suggest avoiding making non-specific statements like: "IMTC outperforms other methods with a large margin in complex PointBilliard, AntBilliard, and AntPush". In that case, my next question is "what is a large margin?" How significant are these values? How many times did you re-run the experiments to make that claim?
I would like to see some more specific numbers around these results.


Nitpick:
- At the beginning of the paper, there is quite a bit of emphasis on "we hypothesis diverse options are reusable". This is a generally well-accepted hypothesis, but the way its written it makes it sound that no one thought of it before.



summary_of_the_review: Well written paper with an interesting objective that makes intuitive sense.
The results look promising, but the discussion and analysis could have more depth; there are questions left unanswered on this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an HRL algorithm that uses the VIC objective to discover the options, i.e., the termination condition of the options is trained to maximize the mutual information between the set of options and their terminating states. These options are first trained without access to a reward function in an unsupervised manner. Later on there are experiments that show how an RL agent can re-use these options in downstream tasks. 
main_review: The general idea of using the termination gradient in VIC is novel and interesting. That said, I feel that the method which was implemented has some gap from the theory, and that the method that is eventually implemented is too similar to VIC. Since the VIC paper was published there were quite a few papers extending it, like DIAYN and RVIC to give two examples. Thus, the novelty of this work is more limited, as it is only comparing to VIC as a baseline. I also feel that I don't learn enough from the experimental section and the algorithm description about the actual difference proposed here. 

More concretely, the implementation of the option model is not clear to me and is not well justified. The theoretical derivation suggests using a model that predicts the probability of seeing s_f after executing option o from state s. However in practice, this model is not used, and it is replaced with the discriminator as being done in VIC, which is a different kind of a model. Furthermore, using this model instead of the option model makes the methods much more similar empirically to VIC and much less novel in my opinion. I think this should be clarified and discussed in much more detail, e.g., what is exactly the difference from VIC.  

Section 4.2 makes some decisions that are not very clear to me and I would have like to see some experiments supporting them. I don't see it as a major part of the paper though and it might have been better to put this section in the supplementary. 

There seems to be a mistake in the definition of the "undiscounted occupancy measure". First of all, it is unclear which occupancy measure is it; is it the average reward occupancy? is it the finite horizon? being undiscounted doesn't tell much. Secondly, on what is the expectation being taken over? if it is on trajectories executed by the policy, then shouldn't the state occupancy be infinity for all non transient states? the average reward and finite horizon state occupancies do not suffer from this issue
summary_of_the_review: Nice idea, but the connection between the theory and practice is somewhat missing, resulting in an algorithm that might be too similar to VIC, and the differences are not explained clearly enough nor supported by the experiments. In addition, including more baselines would have made the empirical contributions stronger.
review_title: null

Meta-review: This paper proposes InfoMax Termination Critic (IMTC), a new approach for learning option termination conditions with the aim of discovering more diverse options. IMTC relies on a scalable approximation of the gradient of a mutual information objective with respect to the termination function parameters.

Reviewers liked the motivation and the simplicity of the approach. While there were some initial concerns regarding the similarity of IMTC and VIC, the authors did a good job of clarifying the differences and providing additional results in the rebuttal. While two reviewers raised their scores based on the rebuttal, this left reviewers split on whether to accept or reject the paper.

Given that the paper’s main contributions are evaluated empirically I based my decision on the strength of the evaluation. The main claim in the paper is that IMTC significantly improves the diversity of the learned options when combined with intrinsic control methods like VIC and RVIC. The main supporting evidence of this claim is a visualization of the option policies and termination probabilities reached by VIC and RVIC. There are several issues with this comparison:
* This is a poor visualization of the kind of option diversity the paper aims to obtain. Given that mutual information based objectives used by VIC, RVIC and IMTC aim to optimize diversity in the final states reached by the options, visualizing the distribution of final states or the trajectories produced by the options is more meaningful.
* The VIC and RVIC baselines are evaluated with a fixed option termination probability of 0.1 which biases the comparison in favor of IMTC because IMTC is able to choose when and where to terminate while VIC and RVIC with random termination get to control neither. Using fixed option duration with MI-based option discovery methods like VIC, DIAYN and RVIC is more standard and is known to produce options with very clear terminal state clusters which are well-separated for different options. Fixed option duration allows VIC and RVIC precise control of where they will terminate since option duration is fixed, hence it should have been included in the comparison.
* As mentioned in point 2 above, it is well established that VIC tends to learn options with well-clustered end states, especially in simple gridworld domains like in Figure 3 (see VIC, DIAYN and RVIC papers). The authors seem to obtain different qualitative results raising questions.

Overall, I don’t think the qualitative experiments show that IMTC is able to improve the diversity of options discovered by VIC or RVIC due to issues with how the experiments are done (random option duration for VIC and RVIC) and how the results are presented (visualizing action probabilities instead of final states). Given these concerns and the split among the reviewers I recommend rejecting the paper in its current form.
Decision: Reject