Learning Variational Temporal Abstraction Embeddings in Option-Induced MDPs

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:34:31

Abstract: The option framework in hierarchical reinforcement learning has notably advanced the automatic discovery of temporally-extended actions from long-horizon tasks. However, existing methods often struggle with ineffective exploration and unstable updates when learning action and option policies simultaneously. Addressing these challenges, we introduce the Variational Markovian Option Critic (VMOC), an off-policy algorithm with provable convergence that employs variational inference to stabilize updates. VMOC naturally integrates maximum entropy intrinsic rewards to promote the exploration of diverse and effective options. Furthermore, we adopt low-cost option embeddings instead of traditional, computationally expensive option tuples, enhancing scalability and expressiveness. Extensive experiments in challenging Mujoco environments validate VMOC’s superior performance over existing on-policy and off-policy methods, demonstrating its effectiveness in learning coherent and diverse option sets suitable for complex tasks.

Main paper: 
 introduction: 
 Recent advancements in deep reinforcement learning (DRL) have demonstrated significant successes across a variety of complex domains, such as mastering the human level of atari [36] and Go [44] games. These achievements underscore the potential of combining reinforcement learning (RL) with powerful function approximators like neural networks [5] to tackle intricate tasks that require nuanced control over extended periods. Despite these breakthroughs, Deep RL still faces substantial challenges, such as insufficient exploration in dynamic environments [18, 13, 42] , inefficient learning associated with temporally extended actions [6, 9] and long horizon tasks [30, 4] , and vast amounts of samples required for training proficient behaviors [16, 40, 15] .One promising area for addressing these challenges is the utilization of hierarchical reinforcement learning (HRL) [11, 2, 12] , a diverse set of strategies that decompose complex tasks into simpler, hierarchical structures for more manageable learning. Among these strategies, the option framework [47] , developed on the Semi-Markov Decision Process (SMDP), is particularly effective at segmenting non-stationary task stages into temporally-extended actions known as options. Options are typically learned through a maximum likelihood approach that aims to maximize the expected rewards across trajectories. In this framework, options act as temporally abstracted actions executed over variable time steps, controlled by a master policy that decides when each option should execute and terminate. This structuring not only simplifies the management of complex environments but also enables the systematic discovery and execution of temporal abstractions over long-horizon tasks [24, 23] .However, the underlying SMDP framework is frequently undermined by three key challenges: 1) Insufficient exploration and degradation [20, 37, 23] . As options are unevenly updated using conventional maximum likelihood methods [4, 10, 45, 25, 26] , the policy is quickly saturated with early rewarding observations. This typically results in focusing on only low-entropy options that lead Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.to local optima rewards, causing a single option to either dominate the entire policy or switch every timestep. Such premature convergence limits option diversity significantly. 2) Sample Inefficiency.The semi-Markovian nature inherently leads to sample inefficiency [47, 29] : each policy update at the master level extends over multiple time steps, thus consuming a considerable volume of experience samples with relatively low informational gain. This inefficiency is further exacerbated by the prevalence of on-policy option learning algorithms [4, 52] , which require new samples to be collected simultaneously from both high-level master policies and low-level action policies at each gradient step, and thus sample expensive. 3) Computationally expensive. Options are conventionally defined as triples [4] with intra-option policies and termination functions, often modeled using neural networks which are expensive to optimize. These challenges collectively limit the broader adoption and effectiveness of the option framework in real-world scenarios, particularly in complex continuous environments where scalability and stability are critical [14, 34, 26] .To address these challenges, we introduce the Variational Markovian Option Critic (VMOC), a novel off-policy algorithm that integrates the variational inference framework on option-induced MDPs [35] . We first formulate the optimal option-induced SMDP trajectory as a probabilistic inference problem, presenting a theoretical convergence proof of the variational distribution under the soft policy iteration framework [19] . Similar to prior variational methods [31] , policy entropy terms naturally arise as intrinsic rewards during the inference procedure. As a result, VMOC not only seeks high-reward options but also maximizes entropy across the space, promoting extensive exploration and maintaining high diversity. We implements this inference procedure as an off-policy soft actor critic [19] algorithm, which allows reusing samples from replay buffer and enhances sample efficiency. Furthermore, to address the computational inefficiencies associated with conventional option triples, we follow [35] and employ low-cost option embeddings rather than complex neural network models. This not only simplifies the training process but also enhances the expressiveness of the model by allowing the agent to capture a more diverse set of environmental dynamics.Our contributions can be summarized as follows:• We propose a variational inference approach within the maximum entropy framework to enhance diverse and robust exploration of options.• We implement an off-policy algorithm that improves sample efficiency.• We introduce option embeddings into latent variable policies and enhance expressiveness and computational cost-effectiveness of option representations.• We conduct extensive experiments in OpenAI Gym Mujoco [49] environments, demonstrating that VMOC significantly outperforms other option-based variants in terms of exploration capabilities, sample efficiency, and computational efficiency. 
control as structured variational inference: 
 Conventionally, the control as inference framework [19, 31, 19, 53] is derived using the maximum entropy objective. In this section, we present an alternative derivation from the perspective of structured variational inference. We demonstrate that this approach provides a more concise and intuitive pathway to the same theoretical results, where the maximum entropy principle naturally emerges through the direct application of variational inference techniques.Traditional control methods focus on directly maximizing rewards, often resulting in suboptimal tradeoffs between exploration and exploitation. By reinterpreting the control problem as a probabilistic inference problem, the control as inference framework incorporates both the reward structure and environmental uncertainty into decision-making, providing a more robust and flexible approach to policy optimization. In this framework, optimality is represented by a binary random variable E ∈ {0, 1} 1 . The probability of optimality given a state-action pair (s, a) is denoted as P (E = 1 | s, a) = exp(r(s, a)), which is an exponential function of the conventional reward function r(s, a) that measures the desirability of an action in a specific state. Focusing on E = 1 captures the occurrence of optimal events. For simplicity, we will use E instead of E = 1 in the following text to avoid cluttered notations. The joint distribution over trajectories τ = (s 1 , a 1 , . . . , s T , a T ) given optimality is expressed as:P (τ |E 1:T ) ∝ P (τ, E 1:T ) = P (s 1 )T -1 t=1 P (s t+1 |s t , a t )P (E t |s t , a t ) where P (s 1 ) is the initial state distribution, P (s t+1 |s t , a t ) is the dynamics model. As explained in [19, 31] , direct optimization of P (τ | E 1:T ) can result in an optimistic policy that assumes a degree of control over the dynamics. One way to correct this risk-seeking behavior [31] is through structured variational inference. In our case, the goal is to approximate the optimal trajectory P (τ ) with the variational distribution:q(τ ) = P (s 1 ) T -1 t=1 P (s t+1 | s t , a t )q(a t | s t )where the initial distribution P (s 1 ) and transition distribution P (s t+1 | s t , a t ) is set to be the true environment dynamics from P (τ ). The only variational term is the variational policy q(a t | s t ), which is used to approximate the optimal policy P (a t | s t , E 1:T ). Under this setting, the environment dynamics will be canceled out from the optimization objective between P (τ | E) and q(τ ), thus explicitly disallowing the agent to influence its dynamics and correcting the risk-seeking behavior.With the variational distribution at hand, the conventional maximum entropy framework can be recovered through a direct application of standard structural variational inference [28] :log P (E 1:T ) = L(q(τ ), P (τ, E 1:T )) + D KL (q(τ ) ∥ P (τ |E 1:T )) = E τ ∼q(τ ) [ T t r(s t , a t ) + H(q(•|s t ))] maximum entropy objective +D KL (q(a t |s t ) ∥ P (a t |s t , E 1:T ))where L(q, P ) = E q [log P q ] is the Evidence Lower Bound (ELBO) [28] . The maximum entropy objective arises naturally as the environment dynamics in P (τ, E) and q(τ ) cancel out. Under this formulation, the soft policy iteration theorem [19] has an elegant Expectation-Maximization (EM) algorithm [28] interpretation: the E-step corresponds to the policy evaluation of the maximum entropy objective L(q [k] , P ); while the M-step corresponds to the policy improvement of the D KL term q [k+1] = arg max q D KL (q [k] (τ ) ∥ P (τ | E)). Thus, soft policy iteration is an exact inference if both EM steps can be performed exactly.Theorem 1 (Convergence Theorem for Soft Policy Iteration). Let τ be the latent variable and E be the observed variable. Define the variational distribution q(τ ) and the log-likelihood log P (E).Let M : q [k] → q [k+1] represent the mapping defined by the EM steps inference update, so thatq [k+1] = M (q [k]). The likelihood function increases at each iteration of the variational inference algorithm until convergence conditions are satisfied.Proof. See Appendix A.1.In conventional SMDP-based Option Framework [47] , an option is a triple widely used in previous option literatures (e.g., Sutton et al. [47] , Bacon et al. [4] ). A master policy π(o|s) = P (o|s) where o ∈ O is used to sample which option will be executed. Therefore, the dynamics (stochastic process) of the option framework is written as:(I o , π o , β o ) ∈ O,EQUATIONwhere τ = {s 0 , o 0 , a 0 , s 1 , o 1 , a 1 , . . .} denotes the trajectory of the option framework. 1 is an indicator function and is only true when o t = o t-1 (notice that o t-1 is the realization at o t-1 ).Therefore, under this formulation the option framework is defined as a Semi-Markov process since the dependency on an activated option o can cross a variable amount of time [47] . Due to the nature of SMDP assumption, conventional option framework is unstable and computationally expensive to optimize. Li et al. [34, 35] proposed the Hidden Temporal Markovian Decision Process (HiT-MDP): enabling options to be represented as dense latent embeddings rather than traditional option triples.P (τ ) = P (s 0 , o 0 ) ∞ t=1 P (They demonstrated that learning options as embeddings on HiT-MDPs offers significant advantages in performance, scalability, and stability by reducing variance. However, their work only derived an on-policy policy gradient algorithm for learning options on HiT-MDPs. In this work, we extend their approach to an off-policy algorithm under the variational inference framework, enhancing exploration and sample efficiency.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations of the study are discussed in the discussion section. 
methodology: 
 In this section, we introduce the Variational Markovian Option Critic (VMOC) algorithm by extending the variational policy iteration (Theorem 1) to the option framework. In Section 3.1, we reformulate the optimal option trajectory and the variational distribution as probabilistic graphical models (PGMs),propose the corresponding variational objective, and present a provable exact inference procedure for these objectives in tabular settings. Section 3.2 extends this result by introducing VMOC, a practical off-policy option learning algorithm that uses neural networks as function approximators and proves the convergence of VMOC under approximate inference settings. Our approach differs from previous works [19, 33, 34] by leveraging structured variational inference directly, providing a more concise pathway to both theoretical results and practical algorithms.Formulating complex problems as probabilistic graphical models (PGMs) offers a consistent and flexible framework for deriving principled objectives, analyzing convergence, and devising practical algorithms. In this section, we first formulate the optimal trajectory of the conventional SMDP-based option framework (Eq. 1) as a PGM. We then use the HiT-MDPs as the variational distribution to approximate this optimal trajectory. With these PGMs, we can straightforwardly derive the variational objective, where maximum entropy terms arise naturally. This approach allows us to develop a stable algorithm for learning diversified options and preventing degeneracy. Specifically, we follow [31, 28] Figure 1 : PGMs of the option framework.by introducing the concept of "Optimality" [48] into the conventional SMDP-based option framework (Equation equation 1). This allows us to define the probability of an option trajectory being optimal as a probabilistic graphical model (PGM), as illustrated in Figure 1 (a):EQUATIONwhere E ∈ {0, 1} are observable binary "optimal random variables" [31] , τ = {s 0 , o 0 , a 0 , s 1 . . .} denotes the trajectory of the option framework. The agent is optimal at time step t when P (E A t = 1|s t , a t ) andP (E O t = 1|s t , a t , o t , o t-1). We will use E instead of E = 1 in the following text to avoid cluttered notations. To simplify the derivation, priors P (o) and P (a) can be assumed to be uniform distributions without loss of generality [31] . Note that Eq. 3 shares the same environment dynamics with Eq. 1 and Eq. 2. With the optimal random variables E O and E A , the likelihood of a state-action {s t , a t } pair that is optimal is defined as:EQUATIONas this specific design facilitates recovering the value function at the latter structural variational inference stage. Based on the same motivation, the likelihood of an option-state-action {o t , s t , a t , o t-1 } pair that is optimal is defined as,EQUATIONwhere f (•) is an arbitrary non-positive function which measures the preferable of selecting an option given state-action pair [s t , a t ] and the previous executed option o t-1 . In this work, we choose f to be the mutual-informationf = I[o t |s t , a t , o t-1] as a fact that when the uniform prior assumption of P (o) is relaxed the optimization introduces a mutual-information as a regularizer [35] .As explained in Section 2.1, direct optimization of Eq. 3 results in optimistic policies that assumes a degree of control over the dynamics. We correct this risk-seeking behavior [31] through approximating the optimal trajectory P (τ ) with the variational distribution:EQUATIONwhere the initial distribution P (s 0 , o 0 ) and transition distribution P (s t+1 | s t , a t ) is set to be the true environment dynamics from P (τ ). The variational distribution turns out to be the HiT-MDP, where the action policy q(a t | s t ) and the option policy q(o t |s t , o t-1 ) are used to approximate the optimal policy P (a t |s t , o t , E A 1:T ) and P (o t |s t , o t-1 , E O 1:T ). The Evidence Lower Bound (ELBO) [28] of the log-likelihood optimal trajectory (Eq. 3) can be derived as (see Appendix A.3):EQUATIONwhere line 2 is substituting Eq. 3 and Eq. 6 into the ELBO. As a result, the maximum entropy objective naturally arises in Eq. 7. Optimizing the ELBO not only seeks high-reward options but also maximizes entropy across the space, promoting extensive exploration and maintaining high diversity.Given the ELBO, we now define soft value functions of the option framework following the Bellman Backup Functions along the trajectory q(τ ) as bellow:EQUATIONEQUATIONAssuming policies π A , π O ∈ Π where Π is an arbitrary feasible set, under a tabular setting where the inference on L can be done exactly, we have the following theorem holds:Theorem 2 (Soft Option Policy Iteration Theorem). Repeated optimizing L and D KL defined in Eq. 10 from anyπ A 0 , π O 0 ∈ Π converges to optimal policies π A * , π O * such that Q sof t * O [s t , o t ] ≥ Q sof t O [s t , o t ] and Q sof t * A [s t , o t , a t ] ≥ Q sof t A [s t , o t , a t ], for all π A 0 , π O 0 ∈ Π and (s t , a t , o t ) ∈ S × A × O, assuming under tabular settings where |S| < ∞, |O| < ∞, |A| < ∞. Proof. See Appendix A.2.Theorem 2 guarantees finding the optimal solution only when the inference can be done exactly under tabular settings. However, real-world applications often involve large continuous domains and employ neural networks as function approximators. In these cases, inference procedures can only be done approximately. This necessitate a practical approximation algorithm which we present below.Formulating complex problems as probabilistic graphical models (PGMs) allowing us to leverage established methods from PGM literature to address the associated inference and learning challenges in real-world applications. To this end, we utilizes the structured variational inference treatment for optimizing the log-likelihood of optimal trajectory and prove its convergence under approximate inference settings. Specifically, using the variational distribution q(τ ) (Eq. 6) as an approximator, the ELBO can be derived as (see Appendix A.3):EQUATIONwhere D KL is the KL-Divergence between the trajectory following variational policies q(τ ) and optimal policies P (τ|E A 1:T , E O 1:T ).Under the structural variational inference [28] perspective, convergence to the optimal policy can be achieved by optimizing the ELBO with respect to the the variational policy repeatedly:Theorem 3 (Convergence Theorem for Variational Markovian Option Policy Iteration). Let τ be the latent variable and E A , E O be the ground-truth optimality variables. Define the variational distribution q(τ ) and the true log-likelihood of optimality log P (E A , E O ). iterates according to the update rule q k+1 = arg max q L(q(τ ), P (τ, E A 1:T , E O 1:T )) converges to the maximum value bounded by the true log-likelihood of optimality.Proof. See Appendix A.4.We further implements a practical algorithm, the Variational Markovian Option Critic (VMOC) algorithm, which is suitable for complex continuous domains. Specifically, we employ parameterized neural networks as function approximators for both the Q-functions (Q sof t ψ A , Q sof t ψ O ) and the policies (π θ A , π θ O ). Instead of running evaluation and improvement to full convergence using Theorem 2, we can optimize the variational distribution by taking stochastic gradient descent following Theorem 3 with respect to the ELBO (Eq. 7) directly. Share the same motivation with Haarnoja et al. [19] of reducing the variance during the optimization procedure, we derive an option critic framework by optimizing the maximum entropy objectives between the action Eq. 9 and the option Eq. 8alternatively. The Bellman residual for the action critic is:J Q A (ψ A i ) = E (st,ot,at,st+1)∼D min i=1,2 Q ψ A i (s t , o t , a t )- r(s t , a t ) + E ot+1∼π O Q sof t O [s t+1 , o t+1 ] + α O H[π O ] 2where α O is the temperature hyper-parameter and the expectation over option random variable E ot+1∼π O can be evaluated exactly since π O is a discrete distribution. The Bellman residual for the option critic is:J Q O (ψ O i ) = E (st,ot)∼D min i=1,2 Q O ψ O i (s t , o t )- f (•) + E at∼π A Q sof t A [s t , o t , a t ] -α A log q(a t |s t , o t ) 2 αA is the temperature hyper-parameter. Unlike E ot+1∼π O can be trivially evaluated, evaluating E at∼π A is typically intractable. Therefore, in implementation we use a t sampled from the replay buffer to estimate the expectation over π A .Following Theorem 3, the policy gradients can be derived by directly taking gradient with respect to the ELBOs defined for the action Eq. 9 and the option Eq. 8 policies respectively. The action policy objective is given by:J π A (θ A ) = -E (st,ot)∼D min i=1,2 Q ψ A i (s t , o t , ãt ) -α A log q(ã t |s t , o t ) , ãt ∼ q(•|s t , o t )where in practice the action policy is often sampled by using the re-parameterization trick introduced in [19] . The option objective is given by:J π O (θ O ) = -E (st,ot-1)∼D min i=1,2 Q ψ O i (s t , o t ) + α O H[π O ]The variational distribution q(τ ) defined in Eq. 6 allows us to learn options as embeddings [34, 35] with a learnable embedding matrix W ∈ R num_options×embedding_dim . Under this setting, the embedding matrix W can be absorbed into the parameter vector θ O . This integration into VMOC ensures that options are represented as embeddings without any additional complications, thereby enhancing the expressiveness and scalability of the model.The temperature hyper-parameters can also be adjusted by minimizing the following objective:J(α A ) = -E ãt∼π A α A (log π A (ã t | s t , o t ) + H)for the action policy temperature α A , where H is a target entropy. Similarly, the option policy temperature α O can be adjusted by:J(α O ) = -E ot∼π O α O (log π O (o t | s t , o t-1 ) + H)where H is also a target entropy for the option policy. In both cases, the temperatures α A and α O are updated using gradient descent, ensuring that the entropy regularization terms dynamically adapt to maintain a desired level of exploration. This approach aligns with the methodology proposed in SAC [19] . By adjusting the temperature parameters, the VMOC algorithm ensures a balanced trade-off between exploration and exploitation, which is crucial for achieving optimal performance in complex continuous control tasks. We summarize the VMOC algorithm in Appendix B.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we design experiments on the challenging single task OpenAI Gym MuJoCo [7] environments (10 environments) to test Variational Markovian Option Critic (VMOC)'s performance over other option variants and non-option baselines.For VMOC in all environments, we fix the temperature rate for both α O and α A to 0.05; we add an exploration noise N (µ = 0, σ = 0.2) during exploration. For all baselines, we follow DAC [52] 's open source implementations and compare our algorithm with six baselines, five of which are option variants, i.e., MOPG [35] , DAC+PPO, AHP+PPO [32] , IOPG [45] , PPOC [27] , OC [4] and PPO All experiments are run on an Intel® Core™ i9-9900X CPU @ 3.50GHz with a single thread and process. Our implementation details are summarized in Appendix C. For a fair comparison, we follow option literature conventions and use four options in all implementations. Our code is available in supplemental materials. 
experiments: 
 We evaluate the performance of VMOC against six option-based baselines (MOPG [35] ,DAC+PPO [52] , AHP+PPO [32] , IOPG [45] , PPOC [27] , and OC [4] ) as well as the hierarchy-free PPO algorithm [41] . Previous studies [27, 45, 20, 52] have suggested that option-based algorithms do not exhibit significant advantages over hierarchy-free algorithms in single-task environments.Nonetheless, our results demonstrate that VMOC significantly outperforms all baselines in terms of episodic return, convergence speed, step variance, and variance across 10 runs, as illustrated in Figure 2 . The only exception is the relatively simple InvertedDoublePendulum environment, which we suspect is due to hyper-parameter tuning issues and will be addressed in future work. A particularly relevant comparison is with the Markovian Option Policy Gradient (MOPG) [35] ,as both VMOC and MOPG are developed based on HiT-MDPs and employ option embeddings.Despite being derived under the maximum entropy framework, MOPG utilizes an on-policy gradient descent approach. Our experimental results show that VMOC's performance surpasses that of MOPG, highlighting the limitations of on-policy methods, which suffer from shortsighted rollout lengths and quickly saturate to early high-reward observations. In contrast, VMOC's variational off-policy approach effectively utilizes the maximum entropy framework by ensuring better exploration and stability across the learning process. Additionally, the off-policy nature of VMOC allows it to reuse samples from a replay buffer, enhancing sample efficiency and promoting greater diversity in the learned policies. This capability leads to more robust learning, as the algorithm can leverage a broader range of experiences to improve policy optimization. 
related work: 
 The VMOC incorporates three key ingredients: the option framework, a structural variational inference based off-policy algorithm and latent variable policies. We review prior works that draw on some of these ideas in this section. The options framework [47] offers a promising approach for discovering and reusing temporal abstractions, with options representing temporally abstract skills. Conventional option frameworks [39] , typically developed under the maximum likelihood (MLE) framework with few constraints on options behavior, often suffer from the option degradation problem [32, 4] . This problem occurs when options quickly saturate with early rewarding observations, causing a single option to dominate the entire policy, or when options switch every timestep, maximizing policy at the expense of skill reuse across tasks. On-policy option learning algorithms [4, 3, 52, 34, 35] aim to maximize expected return by adjusting policy parameters to increase the likelihood of high-reward option trajectories, which often leads to focusing on low-entropy options. Several techniques [20, 21, 23] have been proposed to enhance on-policy algorithms with entropy-like extrinsic rewards as regularizers, but these often result in biased optimal trajectories. In contrast, the maximum entropy term in VMOC arises naturally within the variational framework and provably converges to the optimal trajectory.Although several off-policy option learning algorithms have been proposed [10, 43, 45, 50] , these typically focus on improving sample efficiency by leveraging the control as inference framework.Recent works [45] aim to enhance sample efficiency by inferring and marginalizing over options, allowing all options to be learned simultaneously. Wulfmeier et al. [50] propose off-policy learning of all options across every experience in hindsight, further boosting sample efficiency. However, these approaches generally lack constraints on options behavior. A closely related work [33] also derives a variational approach under the option framework; however, it is based on probabilistic graphical model that we believe are incorrect, potentially leading to convergence issues. Additionally, our algorithm enables learning options as latent embeddings, a feature not present in their approach.Recently, several studies have extended the maximum entropy reinforcement learning framework to discover skills by incorporating additional latent variables. One class of methods [22, 17] maintains latent variables constant over the duration of an episode, providing a time-correlated exploration signal. Other works [19, 51] focus on discovering multi-level action abstractions that are suitable for repurposing by promoting skill distinguishability, but they do not incorporate temporal abstractions.Studies such as [38, 1, 8] aim to discover temporally abstract skills essential for exploration, but they predefine their temporal resolution. In contrast, VMOC learns temporal abstractions as embeddings in an end-to-end data-driven approach with minimal prior knowledge encoded in the framework.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Justification: All details are provided in the main content and the appendix. 
conclusion: 
 In this paper, we have introduced the Variational Markovian Option Critic (VMOC), a novel off-policy algorithm designed to address the challenges of ineffective exploration, sample inefficiency, and computational complexity inherent in the conventional option framework for hierarchical reinforcement learning. By integrating a variational inference framework, VMOC leverages maximum entropy as intrinsic rewards to promote the discovery of diverse and effective options. Additionally, by employing low-cost option embeddings instead of traditional, computationally expensive option triples, VMOC enhances both scalability and expressiveness. Extensive experiments in challenging Mujoco environments demonstrate that VMOC significantly outperforms existing on-policy and off-policy option variants, validating its effectiveness in learning coherent and diverse option sets suitable for complex tasks. This work advances the field of hierarchical reinforcement learning by providing a robust, scalable, and efficient method for learning temporally extended actions.• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
limitations: 
 Due to limited computing resources, we did not conduct an ablation study of VMOC. Additionally, the temperature parameter was fixed in our experiments, whereas an automatically tuned parameter could potentially enhance performance (see SAC [19] ). While our baselines focus on option variants, a thorough comparison to other off-policy algorithms is also worth investigating. It is particularly important to explore whether VMOC exhibits performance improvements in scalability when the number of option embeddings is significantly increased. These investigations are left for future work.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Computational details are provided in the Appendix. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks. 
 
 Appendix: A.1 Theorem 1Theorem 1 (Convergence Theorem for Structured Variational Policy Iteration). Let τ be the latent variable and E be the observed variable. Define the variational distribution q(τ ) and the log-likelihood log P (E). Let M : q [k] → q [k+1] represent the mapping defined by the EM steps inference update, so that q [k+1] = M (q [k] ). The likelihood function increases at each iteration of the variational inference algorithm until the conditions for equality are satisfied and a fixed point of the iteration is reached:log P (E | q [k+1] ) ≥ log P (E | q [k]), with equality if and only ifL(q [k+1] , P ) = L(q [k] , P ) and D KL (q [k+1] (τ ) ∥ P (τ | E)) = D KL (q [k] (τ ) ∥ P (τ | E)).Proof. Let τ be the latent variable and E be the observed variable. Define the evidence lower bound (ELBO) as L(q, P ) and the Kullback-Leibler divergence as D KL (q ∥ P ), where q(τ ) approximates the posterior distribution and P (E | τ ) is the likelihood.The log-likelihood function log P (E) can be decomposed as:log P (E) = L(q, P ) + D KL (q(τ ) ∥ P (τ | E)),whereL(q, P ) = E q(τ ) [log P (E, τ ) -log q(τ )]andD KL (q(τ ) ∥ P (τ | E)) = E q(τ ) log q(τ ) P (τ | E).Let M : q [k] → q [k+1]represent the mapping defined by the variational inference update, so thatq [k+1] = M (q [k]). If q * is a variational distribution that maximizes the ELBO, so that log P (E | q * ) ≥ log P (E | q) for all q, then log P (E | M (q * )) = log P (E | q * ). In other words, the maximizing distributions are fixed points of the variational inference algorithm. Since the likelihood function is bounded (for distributions of practical interest), the sequence of variational distributions q [0] , q [1] , . . . , q [k] yields a bounded nondecreasing sequence logP (E | q [0] ) ≤ log P (E | q [1] ) ≤ • • • ≤ log P (E | q [k] ) ≤ log P (E | q [k]) which must converge as k → ∞.Theorem 2 (Soft Option Policy Iteration Theorem). Repeated optimizing L and D KL defined in Eq. 10 from anyπ A 0 , π O 0 ∈ Π converges to optimal policies π A * , π O * such that Q sof t * O [s t , o t ] ≥ Q sof t O [s t , o t ] and Q sof t * A [s t , o t , a t ] ≥ Q sof t A [s t , o t , a t ], for all π A 0 , π O 0 ∈ Π and (s t , a t , o t ) ∈ S × A × O, assuming |S| < ∞, |O| < ∞, |A| < ∞.Proof. Define the entropy augmented reward as r sof t (s t , a t ) = r(s t , a t ) + H[π A ] andf sof t (o t , s t , a t , o t-1 ) = f (o t , s t , a t , o t-1 ) + H[π O] and rewrite Bellman Backup functions as,Q O [s t , o t ] = f sof t (•) + E at∼π A [Q A [s t , o t , a t ]] , Q A [s t , o t , a t ] = r sof t (s, a) + E st+1∼P (st+1|st,at) E ot+1∼π O [Q O [s t+1 , o t+1 ]]We start with proving the convergence of soft option policy evaluation. As with the standard Qfunction and value function, we can relate the Q-function at a future state via a Bellman Operator T sof t . The option-action value function satisfies the Bellman Operator T sof tT sof t Q A [s t , o t , a t ] = E[G t |s t , o t , a t ] = r sof t (s, a) + γ st+1 P (s t+1 |s t , a t )Q O [s t+1 , o t ],As with the standard convergence results for policy evaluation [46] , by the definition of T sof t (Eq. 11)the option-action value function Q π A A is a fixed point.To prove the T sof t is a contraction, define a norm on V -values functions V and UEQUATION)where s = {s, o}.By recurssively apply the Hidden Temporal Bellman Operator T sof t , we have: Q O [s t , o t-1 ] = E[G t |s t ,EQUATIONTherefore, by applying Eq. 12 to V and U we have:EQUATIONTherefore, T sof t is a contraction. By the fixed point theorem, assuming that throughout our computa-tion the Q A [•, •] and Q O [•] are bounded and A < ∞, the sequence Q k A defined by Q k+1 A = T sof t Q k Awill converge to the option-action value functionQ π A A as k → ∞.The convergence results of and the Soft Option Policy Improvement Theorem then follows conventional Soft Policy Improvement Theorem Theorem 1. Consequently, the Soft Option Policy Iteration Theorem follows directly from these results.A.3 Derivation of Eq. 10L(q(τ ), P (τ, E A 1:T , E O 1:T )) = E q(τ ) [log P (τ, E A 1:T , E O 1:T ) -log q(τ )] = E q(τ ) [log P (τ |E A 1:T , E O 1:T ) + log P (E A 1:T , E O 1:T ) -log q(τ )] = E q(τ ) [log P (τ |E A 1:T , E O 1:T ) -log q(τ )] + E q(τ ) log P (E A 1:T , E O 1:T ) = E q(τ ) [ log P (τ |E A 1:T , E O 1:T ) log q(τ ) ] + log P (E A 1:T , E O 1:T ) = -D KL (log q(τ ) ∥ log P (τ |E A 1:T , E O 1:T )) + log P (E A 1:T , E O 1:T ) A.4 Theorem 3Theorem 3 (Convergence Theorem for Variational Markovian Option Policy Iteration). Let τ be the latent variable and E A , E O be the ground-truth optimality variables. Define the variational distribution q(τ ) and the true log-likelihood of optimality log P (E A , E O ). iterates according to the update rule q k+1 = arg max q L(q(τ ), P (τ, E A 1:T , E O 1:T )) converges to the maximum value bounded by the data log-likelihood.Proof. The objective is to maximize the ELBO with respect to the policy q. Formally, this can be written as:q k+1 = arg max q L(q, P ).Suppose we q is a neural network function approximator, assuming the continuity and differentiability of q with respect to its parameters. Using stochastic gradient descent (SGD) to optimize the parameters guarantees that the ELBO increases, such that L(q k+1 , P ) ≥ L(q k , P ).Rearranging Eq. 10 we get:D KL (q k+1 (τ )||P (τ |E A 1:T , E O 1:T )) = -L(q k+1 (τ ), P (τ, E A 1:T , E O 1:T )) + log P (E A 1:T , E O 1:T ) ≤ -L(q k (τ ), P (τ, E A 1:T , E O 1:T )) + log P (E A 1:T , E O 1:T ) = D KL (q k (τ )||P (τ |E A 1:T , E O 1:T ))Thus, each SGD update not only potentially increases the ELBO but also decreases the KL divergence, moving q closer to P . Given the properties of SGD and assuming appropriate learning rates and sufficiently expressive neural network architectures, the sequence {q k } converges to a policy q * that minimizes the KL divergence to the true posterior.Algorithm 1 VMOC Algorithm1: Initialize parameter vectors ψ A , ψ O , θ O , θ A 2:for each epoch do Collect trajectories {o t-1 , s t , a t , o t } into the replay buffer 4:for each gradient step do UpdateEQUATIONUpdateQ sof t ψ O i : ψ O i ← ψ O i -η Q O ∇J Q sof t ψ O i for i ∈ {1, 2} 7: Update π O θ O : θ O ← θ O -η π O ∇J π O 8: Update π A θ A : θ A ← θ A -η π A ∇J π A 9:Update target networks:ψA ← σψ A + (1 -σ) ψA , ψO ← σψ O + (1 -σ) ψO 10:Update temperature factors:α O ← α O -η α O ∇J α O , α A ← α A -η α A ∇J α A 11:end for 12: end forIn this section we summarize our implementation details. For a fair comparison, all baselines: MOPG [35] , DAC+PPO [52] , AHP+PPO [32] , PPOC [27] , OC [4] and PPO [41] are from DAC's open source Github repo: https://github.com/ShangtongZhang/DeepRL/tree/DAC. Hyperparameters used in DAC [52] for all these baselines are kept unchanged.VMOC Network Architecture: We use Pytorch to build neural networks. Specifically, for option embeddings, we use an embedding matrix W S ∈ R 4×40 which has 4 options (4 rows) and an embedding size of 40 (40 columns). For layer normalization we use Pytorch's built-in function LayerNorm2 . For Feed Forward Networks (FNN), we use a 2 layer FNN with ReLu function as activation function with input size of state-size, hidden size of [256, 256] , and output size of actiondim neurons. For Linear layer, we use built-in Linear function 3 to map FFN's outputs to 4 dimension. both action policy and critic module, FFNs are of the same size as the one used in the skill policy.Preprocessing: States are normalized by a running estimation of mean and std.Hyperparameters for all on-policy option variants: For a fair comparison, we use exactly the same parameters of PPO as DAC . Specifically:• Optimizer: Adam with ϵ = 10 -5 and an initial learning rate 3 × 10 -4 • Discount ratio γ: 0.99 Guidelines:•• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The only applicable assets are the code which are credited and distributed under a Creative Commons Attribution License.• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.Conventionally, the optimality variable is denoted by O. However, in this context, we use E to avoid conflict with notation used in the option framework.https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.htmlhttps://pytorch.org/docs/stable/generated/torch.nn.Linear.htmlhttps://github.com/pytorch/pytorch/blob/master/torch/distributions/categorical.py 


Decision: The reviewers agree that there's much room for improvement, from the stated contributions to the empirical protocol to better positioning the paper in the literature. The fact that no thorough response was provided to the reviewers, but only an acknowledgement that the paper will be improved in its next version, made the decision quite obvious.

Reviews: review: paper_summary: paper_summary: This paper introduces e Variational Markovian Option Critic (VMOC), which learns actions and options simultatenously. They build upon the Hidden Temporal Markovian Decision Process (HiT-MDP) [1] to build a novel off-policy algorithm that utilizes entropy augmented rewards. Their method learns options’ embedding vectors (rather than conventional option tuples utilized in Semi-MDP [2]). They benchmark the learning performance of their method against several competitors on many classic control benchmark environments. 

*References*
1. Li, C., Song, D., & Tao, D. (2023). Hit-MDP: learning the SMDP option framework on MDPs with hidden temporal embeddings. In The Eleventh International Conference on Learning Representations.
2. Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2), 181-211.
strength_weakness: strength_weakness: The paper does offer a potentially interesting contribution to the wider research community. But it is held back by the lack of clarity and polish in writing. For example, two glaring signs of a hasty submission:
1. Sec 4 and Sec 5 are both titled experiments. Sec 4 is only 1 paragraph, and it essentially repeats the same information from the introductory paragraphs of Sec 5
2. In Sec. 5, line 271 just trails off without completion. I believe the authors moved around the images to correct for vertical space and accidentally hid the text.

While the experimental results focus on learning curves, where VMOC does well, they fail to provide other relevant evaluation metrics:
1. What do the learned options look like? A good evaluation could follow Fig. 5 and Fig. 6 from [1]
2. How many options are learned? Digging through the appendix, it says that they learned 4 option vectors. This leads to another question: how do they choose the number of options to learn?
3. The VMOC algorithm listed in the appendix only describes the gradient update process. No details about action sampling or other hyper-parameter tuning are described here
4. The environments used are challenging for model-free RL algorithms. That said, they may not be satisfactory for showcasing the potential of learned options. 


*References*
1. Li, C., Song, D., & Tao, D. (2023). Hit-MDP: learning the SMDP option framework on MDPs with hidden temporal embeddings. In The Eleventh International Conference on Learning Representations.
questions: questions: - How are actions sampled? Do you use the reparametrization trick  [1] for sampling action? Is the update similar to SAC?
- How does your algorithm compare against SAC? Looking at the gradient update rules, one could argue that it is fairer to compare VMOC against SAC than PPO. My suspicion is that SAC would perform just as well as VMOC
- Why mujoco environments for showcasing options? The true benefit of options would be seen in environments that could benefit from hierarchical policies or composition of policies. Do the authors consider environments from [Meta-World](https://meta-world.github.io/) or the dog fetch or ant  fetch environments in [DM Control Suite](https://github.com/google-deepmind/dm_control/tree/main/dm_control/suite)

*References*
1. Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. ArXiv Preprint ArXiv:1312.6114.
2. Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018, July). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (pp. 1861-1870). PMLR.
limitations: limitations: Much like Soft Actor-Critic, this work develops a novel Soft Option Critic style algorithm. I believe this line of work is very interesting and potentially impactful in the near future. However, their current draft is not well-written and hard to follow. Their experimental evaluation is also insufficient.
paper_summary: paper_summary: The paper introduces the Variational Markovian Option Critic (VMOC) which combines variation policy iteration and the option critic. VMOC also modifies HiT-MDPs, where options are represented as latent embeddings rather than triples of (init states, policy, termination condition), to the off-policy setting. The paper performs comparisons to option-based methods and PPO on 10 Mujoco environments.
strength_weakness: strength_weakness: 1. It is not clear if this gain in sample-efficiency will transfer to discrete environments or is somehow applicable only in continuous envs. Perhaps the authors can perform comparisons on Atari or Procgen to demonstrate the same? It would be great if the authors could also discuss the changes in the algorithm in the discrete and continuous settings (perhaps such as the sampling of a_t from the replay buffer?)

2. It is unclear if all methods use the same number of options (e.g. the value used in VMOC appears to be 4). A clear ablation of various design choices like number of options would help demonstrate that VMOC is thoroughly better than the other option methods and is not brittle to hyperparameter choice. 

The analysis of the actual options learnt is also missing (this is for example seen in the option critic paper). This, alongside an analysis of the number of options, is crucial to understand if the method is actual learning composed actions that are further composable and generalizable or degenerating to something simple like learning the action primitives (although the latter would apply more to a discrete rather than continuous env).

3. Minor comment: The location of Theorem 1 in the preliminaries makes it unclear if it is a contribution of the authors or well-known statement. Perhaps the authors can clarify?

4. Another minor comment: It would be great if the authors could discuss other ways of combining options in the related work such as in [1] and [2].

[1] The Option Keyboard: Combining Skills in Reinforcement Learning, Barreto et al, NeurIPS 2019

[2] Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates, Dutta et al, CoRL 2022
questions: questions: Please see Weaknesses.
limitations: limitations: The authors have addressed limitations.
paper_summary: paper_summary: This paper proposes the Variational Markovian Option Critic (VMOC), an off-policy algorithm for hierarchical reinforcement learning. VMOC aims to address exploration inefficiency and update instability in existing methods. Key contributions include: 1. Use of variational inference for update stabilization 2. Low-cost option embeddings for improved scalability. The authors evaluate VMOC on Mujoco environments, comparing it to other on-policy and off-policy methods. They report improved performance in learning option sets for complex tasks.
strength_weakness: strength_weakness: 1. Very similar ideas of the variational option framework have been proposed in [33] (off-policy) and [35] (on-policy). While [35] proposes an on-policy version, its off-policy version is also straightforward to deduce following [ref1]. The use of option embeddings is following [35].
2. The empirical evaluations are very limited; there is no ablative evaluation reported, which makes it hard to determine the contribution of the proposed method to the overall performance gain over various baselines.

References: 
[ref1] Levine, Sergey. "Reinforcement learning and control as probabilistic inference: Tutorial and review." arXiv preprint arXiv:1805.00909 (2018).
questions: questions: The authors claim that [33] is based on an "incorrect" probabilistic graphical model. I wonder if the authors could elaborate on this claim.
limitations: limitations: The empirical evaluations, especially ablation studies, are somewhat limited in scope.
paper_summary: paper_summary: The paper presents an off-policy hierarchical RL method, based on the HiT-MDP formulation of a Semi-MDP. The HiT-MDP formulation treats the option $o$ as an extension of the original state $s$ (which can be chosen by an extended action), and combines initialization-, termination- and option-policy in a single Markovian master policy $p(o\_{t}|s\_t,o\_{t-1})$. The policy in the extended state-action space, thus, decomposes into the high-level and low-level policies, $p(o\_{t}, a\_t | s\_t, o\_{t-1}) = p(o\_{t} | s\_t, o\_{t-1}) p(a\_t | s\_t, o\_{t})$, which can be trained using standard RL algorihms. 
Compared to the prior work, the paper makes the following contributions:
- Whereas previously PPO was used for reinforcement learning, the paper proposes to use SAC, resulting in improved sample efficiency
- The paper motivates the algorithm from a control-as-inference perspective
strength_weakness: strength_weakness: Originality
-----------
One of the main weaknesses of the submission is the limited novelty. Replacing the PPO agent of MOPG by a SAC agent seems to be straightforward, so this contribution is quite incremental. Indeed, the authors of HiT-MDP stated, that their ELBO "can easily be extended to a SAC-like algorithm" [35]. Furthermore, given that MaxEnt-RL was already derived from a control-as-inference perspective, deriving the special case of an HiT-MDP using this technique does not seem to be significant contribution either. I also don't see the value of this derivation that would justify devoting so much space on it; couldn't we just argue that we apply SAC to such particular form of an MDP?


Quality
---------
The experimental evaluation seems to be another weakness of the submission. While the method is evaluated on a reasonable number of MuJoCo environments, where it outperforms a reasonable number of baselines, the choice of baselines is not convincing because it looks like the method is only compared to on-policy algorithm. The submission claims that there method "significantly outperforms existing on-policy and off-policy option variants", but it is not clear to me to which off-policy baselines this claim refers to. It would be important to focus to flat and hierarchical off-policy methods in the experiments, such as [19], [50], [33] and Hao et. al (2023).  Furthermore, the choice of environments is not convincing, because it does not include more challenging long-horizon tasks that are typically used for evaluating HRL methods, such as Ant-Maze. While the performance on the standard locomotion environments is reasonable, the reported numbers don't seem to improve on the SOTA of flat-RL methods.   

The paper does not discuss the hyperparameter search although it states in the questionnary  that these details are provided in the main content and the appendix.

The paper argues that it did not perform any ablations due to limited computational resources. However, I don't find this argument very convincing, since the experiments are performed on simple vision-free locomotion tasks, that can be run on standard workstation, not even requiring any GPU. Ablations on the number of options would be very useful.



Clarity
---------
I found the background material on control-as-inference a bit confusing. In particular, line 106 which states states policy improvement constitues an M-Step of an EM algorithm that *maximizes* the KL towards $P(\tau|\mathcal{E})$. I don't think any practical algorithm involves such maximization, since the optimum would correspond to a delta distribution on the least-likely trajectory. (

Visually, the presentation is rather bad. Figures are not on the top, and in particular Fig. 2 seems to hide some text, since the sentence in line 271 ends with ", which". Fig. 2 itself could be improved by increasing the plot sizes (there are some unnecessary white spaces) and by making the legend more readable.  


Significance
-----------------
While I think that the proposed combination of the HiT-MDP formulation and SAC is somewhat interesting, the submission does not provide a convincing argument for the method. When should I use it, instead of existing (hierarchical or flat) methods?

References
----------
* Hao, C., Weaver, C., Tang, C., Kawamoto, K., Tomizuka, M., & Zhan, W. (2023). Skill-critic: Refining learned skills for reinforcement learning. arXiv preprint arXiv:2306.08388.
questions: questions: * Line 227 mentions that the targets for the option Q-function are computed using actions from the replay buffer because estimating the expectation with respect to samples from the policy would be intractable. I don't understand this: Why can't we just sample from the policy?

* Which of the baselines where off-policy?
limitations: limitations: The limitations are adequately discussed and I don't have any concerns regarding negative societal impact of the work.