Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to solve Semi-Markov decision processes (SMDP) by reformulating the SMDP into an ordinary MDP. In particular, the authors propose a novel MDP view of SMDP that models the options as part of the states and actions, leading to the definition of several option-value functions. The authors propose to utilize an energy-based parameterization in the transition model, which results in an entropy regularizer term in policy optimization. The authors further solve such policy optimization problems by policy gradient and propose the MOPG algorithm for solving SMDPs. Finally, the authors conduct several experiments on the MuJuCo environment and compare MOPG with several SOTA algorithms.
strength_and_weaknesses: $\textbf{Strength:}$
The authors propose a novel MDP view of SMDP that models the options as part of the states and actions, leading to the definition of several option-value functions. The proposed MDP model can potentially extend various existing RL algorithms that solve MDP to the SMDP problem setup.

$\textbf{Weaknesses and Questions:}$

- How does MOPG compare with Attention Option-Critic [1], which also uses attention network to encode option?


- The theory is not sufficient for me. There is no proof or discussion on the convergence or sample efficiency of MOPG. The equivalence between SMDP and MDP could be more significant if authors can provide such type of analyses.

- The contribution is limited, as the proposed MOPG still falls in the well-established PPO framework, and similar architecture was explored before in [1]. 

[1] Chunduru and Precup. Attention Option-Critic. (2022).

clarity,_quality,_novelty_and_reproducibility: See questions above. Implementation details are shared but the code is not released. Typo in p6: dynamix -> dynamics
summary_of_the_review: The idea is interesting, but the contribution is marginal to me.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Mostly survey, not clear what the contribution is. The submission considers option RL. The major challenges the submission tackles is (1) the poor sample efficiency and (2) hard optimization for such a problem. The authors argue that the cause of the two challenges is the semi-MDP framework of existing option RL methods. Therefore, to deal with this issue, the submission reformulates the option RL problem into an HMM. Therefore, RL methods for HMM can be leveraged to deal with the two challenges in option RL.

Both theoretical and empirical results are provided supporting the proposed method. 

strength_and_weaknesses: Hard to learn from the survey. No applications discussed.

Strength 

The submission is well motivated, focusing on two key challenges in option RL. 

A novel formulation of option RL is proposed, which facilitates new option RL methods.

The results provided by the submission are thorough and complete, with both theoretical and empirical support for the methodology. 

Weaknesses

It is not very clear to me why the proposed new formulation and methodology can have a better sample complexity than existing ones. At the beginning, the authors state that the reason for the poor sample complexity of option RL is that one option lasts for multiple steps, which wastes observations. However, this is also true under the proposed HiT-MDP formulation: in each time step, there is a high probability that the option stays the same as the previous step. Both formulations have strong probabilities to maintain the selected option for multiple steps. So, why is the proposed one more sample efficient? 

Would it be possible for authors to provide the option change frequency of each method in experiments? In experiments, the proposed method shows better sample complexity than existing ones. I am wondering whether this is because the proposed method changes options more frequently than the competing ones? If so, can we just manually increase the option switch frequency of existing methods to achieve similar sample complexity gain? 

clarity,_quality,_novelty_and_reproducibility: OK writing.
summary_of_the_review: It's a survey paper with citations permeating through sections 6 and 7.
The submission proposes both novel formulation and novel methodology for option RL. Both theoretical and synthetic results are provided supporting the proposed method. However, I am not quite sure why the proposed method and formulation are able to provide such performance improvement.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel  Markov Decision Process (MDP),  namely, the Hidden Temporal MDP  (HiT-MDP), to solve the semi-Markov
decision process that is unstable, unoptimizable, and inefficient in sampling ,and proves that the option-induced HiT-MDP is homomorphic 
equivalent to the option-induced SMDP. Then an efficient algorithm based on structured mutation reasoning is derived, which leads to a new 
method of finding stable options under the maximum entropy reinforcement learning framework. Finally, HiT-MDP shows excellent performance in a wide range of configurations.
strength_and_weaknesses: Pros:
The paper is very well motivated. The authors precisely present the shortcomings of the current standard option framework based on semi-Markovian decision processes and propose a new HIT-MDP that addresses the shortcomings of the original framework one by 
one. It is also very clearly formulated. The arguments are clear, the derivation of formulas is rigorous. Even the lengthy appendix is fairly easy to parse, with straightforward proofs and relevant details. The most commendable thing is that the author has an innovative spirit. This paper is the first to propose an MDP equivalent to SMDP-Option, which provides reference and guidance for subsequent further research.

Cons:
The related work section is placed after the experimental section and before the conclusion and is somewhat abbreviated.

clarity,_quality,_novelty_and_reproducibility: Quality: The paper presents the first HIT-MDP and fully demonstrates the superiority of the framework over the traditional standard option 
framework with rich and detailed experiments of high quality.
 
Clarity: This work follows the process of posing an existing problem, analyzing the problem, constructing a method to solve it, and performing a formula derivation to prove the effectiveness of the strategy. The exposition is well organized, the proof process is rigorous and scientific, and the derivation of formulas is detailed and convincing. The paper is illustrated with pictures where appropriate, and the premises and details of each formula are explained.
 
Novelty: The author's design is clever, the exposition is rigorous, and the content is original.

summary_of_the_review: I think this is a good paper with solid theoretical analysis and contributions, the problems analyzed are very important, the proposed 
architecture is feasible, the proof process is complete, and needs to be understood as deeply as this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies option learning in RL, and introduces a new framework (HiT-MDP). The authors prove that HiT-MDP is homomorphic equivalent to the standard SMDP formulation, and derive an on-policy policy gradient method with the new HiT-MDP formulation. On Mujuco environments, the proposed method is shown to be effective, exhibit smaller faster convergence and enjoy better interpretability.
strength_and_weaknesses: Strength:
* The writing is in general clear.
* The introduced HiT-MDP formulation is novel.

Questions:
* Can the authors provide more visualizations of the learned options?
* From Figure 6, it seems only 2 out of 4 options are useful. Is it because the environment is relatively simple? Does more complex environments (e.g. Humanoid) result in more options?
* Can the authors conduct similar analysis (as in Figure 5,6) for other environments, such as Humanoid or Hopper?
* How will the proposed method work in video games like Atari or Procgen? Can it learn some interesting options?

Minor issues:
* Please consider increasing the font size in Figure 1, 2, 5 and 6.
clarity,_quality,_novelty_and_reproducibility: * Clarity: Mostly good.
* Quality and Novelty: The quality of this paper is pretty good and the proposed HiT-MDP is novel.
* Reproducibility: The source code is provided. I would assume the results are reproducible.
summary_of_the_review: In summary, this paper introduces a new HiT-MDP framework for option learning in RL. Based on HiT-MDP, the authors derive a policy gradient method and validate its effectiveness and advantages over exiting methods on Mujoco.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I thank the authors for their submission and active engagement during the discussion period. This is a borderline paper. On the positive side, reviewers acknowledged the novel MDP formulation accounting for options [iC2w]. They found the work well motivated and focusing on important problems in option learning [ySTp]. The method is novel [ySTp,uUrN] and the empirical investigation thorough [ySTp]. On the negative side, there are concerns around the method's sample efficiency [ySTp] and comparison to prior work [iC2w]. I believe the authors have addressed reviewer's iC2w concerns appropriately. Likewise, I believe the rebuttal addresses ySTp's concerns regarding the option frequency. Therefore, I recommend acceptance but strongly encourage the authors to take the reviewer feedback into account for the camera ready version of the paper.
justification_for_why_not_higher_score: While this is a solid paper, I believe the topic is somewhat niche to warrant broader dissemination at the conference.
justification_for_why_not_lower_score: Strengths outweigh the weaknesses of this paper. Overall, a solid contribution.
Decision: Accept: poster