Title: Learning Variational Temporal Abstraction Embeddings in Option-Induced MDPs
Venue: NeurIPS 2024
Accepted: no
Keywords: reinforcement learning

option framework
Abstract: The option framework in hierarchical reinforcement learning has notably advanced the automatic discovery of temporally-extended actions from long-horizon tasks. However, existing methods often struggle with ineffective exploration and unstable updates when learning action and option policies simultaneously. Addressing these challenges, we introduce the Variational Markovian Option Critic (VMOC), an off-policy algorithm with provable convergence that employs variational inference to stabilize updates. VMOC naturally integrates maximum entropy intrinsic rewards to promote the exploration of diverse and effective options. Furthermore, we adopt low-cost option embeddings instead of traditional, computationally expensive option tuples, enhancing scalability and expressiveness. Extensive experiments in challenging Mujoco environments validate VMOC’s superior performance over existing on-policy and off-policy methods, demonstrating its effectiveness in learning coherent and diverse option sets suitable for complex tasks.

Abstract

The option framework in hierarchical reinforcement learning has notably advanced the automatic discovery of temporally-extended actions from long-horizon tasks. However, existing methods often struggle with ineffective exploration and unstable updates when learning action and option policies simultaneously. Addressing these challenges, we introduce the Variational Markovian Option Critic (VMOC), an off-policy algorithm with provable convergence that employs variational inference to stabilize updates. VMOC naturally integrates maximum entropy as intrinsic rewards to promote the exploration of diverse and effective options. Furthermore, we adopt low-cost option embeddings instead of traditional, computationally expensive option triples, enhancing scalability and expressiveness. Extensive experiments in challenging Mujoco environments validate VMOC's superior performance over existing on-policy and off-policy methods, demonstrating its effectiveness in learning coherent and diverse option sets suitable for complex tasks.

1 Introduction

Recent advancements in deep reinforcement learning (DRL) have demonstrated significant successes across a variety of complex domains, such as mastering the human level of atari [36] and Go [44] games. These achievements underscore the potential of combining reinforcement learning (RL) with powerful function approximators like neural networks [5] to tackle intricate tasks that require nuanced control over extended periods. Despite these breakthroughs, Deep RL still faces substantial challenges, such as insufficient exploration in dynamic environments [18, 13, 42], inefficient learning associated with temporally extended actions [6, 9] and long horizon tasks [30, 4], and vast amounts of samples required for training proficient behaviors [16, 40, 15].

One promising area for addressing these challenges is the utilization of hierarchical reinforcement learning (HRL) [11, 2, 12], a diverse set of strategies that decompose complex tasks into simpler, hierarchical structures for more manageable learning. Among these strategies, the option framework [47], developed on the Semi-Markov Decision Process (SMDP), is particularly effective at segmenting non-stationary task stages into temporally-extended actions known as options. Options are typically learned through a maximum likelihood approach that aims to maximize the expected rewards across trajectories. In this framework, options act as temporally abstracted actions executed over variable time steps, controlled by a master policy that decides when each option should execute and terminate. This structuring not only simplifies the management of complex environments but also enables the systematic discovery and execution of temporal abstractions over long-horizon tasks [24, 23].

However, the underlying SMDP framework is frequently undermined by three key challenges:

33 1) Insufficient exploration and degradation [20, 37, 23]. As options are unevenly updated using 34 conventional maximum likelihood methods [4, 10, 45, 25, 26], the policy is quickly saturated with 35 early rewarding observations. This typically results in focusing on only low-entropy options that lead 36

to local optima rewards, causing a single option to either dominate the entire policy or switch every 37 timestep. Such premature convergence limits option diversity significantly. 2) Sample Inefficiency. 38 The semi-Markovian nature inherently leads to sample inefficiency [47, 29]: each policy update 39 at the master level extends over multiple time steps, thus consuming a considerable volume of 40 experience samples with relatively low informational gain. This inefficiency is further exacerbated 41 by the prevalence of on-policy option learning algorithms [4, 52], which require new samples to be 42 collected simultaneously from both high-level master policies and low-level action policies at each 43 gradient step, and thus sample expensive. 3) Computationally expensive. Options are conventionally 44 defined as triples [4] with intra-option policies and termination functions, often modeled using neural 45 networks which are expensive to optimize. These challenges collectively limit the broader adoption 46 and effectiveness of the option framework in real-world scenarios, particularly in complex continuous 47 environments where scalability and stability are critical [14, 34, 26]. 48

49

50

51

52

53

54

55

56

57

58

59

60

61

To address these challenges, we introduce the Variational Markovian Option Critic (VMOC), a novel off-policy algorithm that integrates the variational inference framework on option-induced MDPs [35]. We first formulate the optimal option-induced SMDP trajectory as a probabilistic inference problem, presenting a theoretical convergence proof of the variational distribution under the soft policy iteration framework [19]. Similar to prior variational methods [31], policy entropy terms naturally arise as intrinsic rewards during the inference procedure. As a result, VMOC not only seeks high-reward options but also maximizes entropy across the space, promoting extensive exploration and maintaining high diversity. We implements this inference procedure as an off-policy soft actor critic [19] algorithm, which allows reusing samples from replay buffer and enhances sample efficiency. Furthermore, to address the computational inefficiencies associated with conventional option triples, we follow [35] and employ low-cost option embeddings rather than complex neural network models. This not only simplifies the training process but also enhances the expressiveness of the model by allowing the agent to capture a more diverse set of environmental dynamics.

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

Our contributions can be summarized as follows:

We propose a variational inference approach within the maximum entropy framework to enhance diverse and robust exploration of options.

We implement an off-policy algorithm that improves sample efficiency.

We introduce option embeddings into latent variable policies and enhance expressiveness and computational cost-effectiveness of option representations.

We conduct extensive experiments in OpenAI Gym Mujoco [49] environments, demonstrating that VMOC significantly outperforms other option-based variants in terms of exploration capabilities, sample efficiency, and computational efficiency.

2 Preliminary

2.1 Control as Structured Variational Inference

Conventionally, the control as inference framework [19, 31, 19, 53] is derived using the maximum entropy objective. In this section, we present an alternative derivation from the perspective of structured variational inference. We demonstrate that this approach provides a more concise and intuitive pathway to the same theoretical results, where the maximum entropy principle naturally emerges through the direct application of variational inference techniques.

Traditional control methods focus on directly maximizing rewards, often resulting in suboptimal tradeoffs between exploration and exploitation. By reinterpreting the control problem as a probabilistic inference problem, the control as inference framework incorporates both the reward structure and environmental uncertainty into decision-making, providing a more robust and flexible approach to policy optimization. In this framework, optimality is represented by a binary random variable E ∈ { 0 , 1 } 1 . The probability of optimality given a state-action pair ( s , a ) is denoted as P ( E = 1 | s , a ) = exp( r ( s , a )) , which is an exponential function of the conventional reward function r ( s , a ) that measures the desirability of an action in a specific state. Focusing on E = 1 captures the occurrence of optimal events. For simplicity, we will use E instead of E = 1 in the following text

to avoid cluttered notations. The joint distribution over trajectories τ = ( s 1 , a 1 , . . . , s T , a T ) given 87 optimality is expressed as: 88

where P ( s 1 ) is the initial state distribution, P ( s t +1 | s t , a t ) is the dynamics model. As explained 89 in [19, 31], direct optimization of P ( τ | E 1: T ) can result in an optimistic policy that assumes a degree 90 of control over the dynamics. One way to correct this risk-seeking behavior [31] is through structured 91 variational inference. In our case, the goal is to approximate the optimal trajectory P ( τ ) with the 92 variational distribution: 93

where the initial distribution P ( s 1 ) and transition distribution P ( s t +1 | s t , a t ) is set to be the true 94 environment dynamics from P ( τ ) . The only variational term is the variational policy q ( a t | s t ) , 95 which is used to approximate the optimal policy P ( a t | s t , E 1: T ) . Under this setting, the environment 96 dynamics will be canceled out from the optimization objective between P ( τ | E ) and q ( τ ) , thus 97 explicitly disallowing the agent to influence its dynamics and correcting the risk-seeking behavior. 98

With the variational distribution at hand, the conventional maximum entropy framework can be 99 recovered through a direct application of standard structural variational inference [28]: 100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

where L ( q, P ) = E q [log P q ] is the Evidence Lower Bound (ELBO) [28]. The maximum entropy objective arises naturally as the environment dynamics in P ( τ, E ) and q ( τ ) cancel out. Under this formulation, the soft policy iteration theorem [19] has an elegant Expectation-Maximization (EM) algorithm [28] interpretation: the E-step corresponds to the policy evaluation of the maximum entropy objective L ( q [ k ] , P ) ; while the M-step corresponds to the policy improvement of the D KL term q [ k +1] = arg max q D KL ( q [ k ] ( τ ) ∥ P ( τ | E )) . Thus, soft policy iteration is an exact inference if both EM steps can be performed exactly.

Theorem 1 (Convergence Theorem for Soft Policy Iteration) . Let τ be the latent variable and E be the observed variable. Define the variational distribution q ( τ ) and the log-likelihood log P ( E ) . Let M : q [ k ] → q [ k +1] represent the mapping defined by the EM steps inference update, so that q [ k +1] = M ( q [ k ] ) . The likelihood function increases at each iteration of the variational inference algorithm until convergence conditions are satisfied.

2.2 The Option Framework

In conventional SMDP-based Option Framework [47], an option is a triple ( I o , π o , β o ) ∈ O , where O denotes the option set; o ∈ O = { 1 , 2 , . . . , K } is a positive integer index which denotes the o -th triple where K is the number of options; I o is an initiation set indicating where the option can be initiated; π o = P o ( a | s ) : A × S → [0 , 1] is the action policy of the o th option; β o = P o ( b = 1 | s ) : S → [0 , 1] where b ∈ { 0 , 1 } is a termination function . For clarity, we use P o ( b = 1 | s ) instead of β o which is widely used in previous option literatures (e.g., Sutton et al. [47], Bacon et al. [4]). A master policy π ( o | s ) = P ( o | s ) where o ∈ O is used to sample which option will be executed. Therefore, the dynamics (stochastic process) of the option framework is written as:

where τ = { s 0 , o 0 , a 0 , s 1 , o 1 , a 1 , . . . } denotes the trajectory of the option framework. 1 is an 123 indicator function and is only true when o t = o t -1 (notice that o t -1 is the realization at o t -1 ). 124 Therefore, under this formulation the option framework is defined as a Semi-Markov process since 125 the dependency on an activated option o can cross a variable amount of time [47]. Due to the nature 126 of SMDP assumption, conventional option framework is unstable and computationally expensive to 127 optimize. Li et al. [34, 35] proposed the Hidden Temporal Markovian Decision Process (HiT-MDP): 128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

by introducing the concept of "Optimality" [48] into the conventional SMDP-based option framework 157 (Equation equation 1). This allows us to define the probability of an option trajectory being optimal 158

and theoretically proved that the option-induced HiT-MDP is homomorphically equivalent to the conventional SMDP-based option framework. Following RL conventions, we use π A = P ( a t | s t , o t ) to denote the action policy and π O = P ( o t | s t , o t -1 ) to denote the option policy respectively. In HiT-MDPs, options can be viewed as latent variables with a temporal structure P ( o t | s t , o t -1 ) , enabling options to be represented as dense latent embeddings rather than traditional option triples. They demonstrated that learning options as embeddings on HiT-MDPs offers significant advantages in performance, scalability, and stability by reducing variance. However, their work only derived an on-policy policy gradient algorithm for learning options on HiT-MDPs. In this work, we extend their approach to an off-policy algorithm under the variational inference framework, enhancing exploration and sample efficiency.

3 Methodology

In this section, we introduce the Variational Markovian Option Critic (VMOC) algorithm by extending the variational policy iteration (Theorem 1) to the option framework. In Section 3.1, we reformulate the optimal option trajectory and the variational distribution as probabilistic graphical models (PGMs), propose the corresponding variational objective, and present a provable exact inference procedure for these objectives in tabular settings. Section 3.2 extends this result by introducing VMOC, a practical off-policy option learning algorithm that uses neural networks as function approximators and proves the convergence of VMOC under approximate inference settings. Our approach differs from previous works [19, 33, 34] by leveraging structured variational inference directly, providing a more concise pathway to both theoretical results and practical algorithms.

3.1 PGMFormulations of The Option Framework

Formulating complex problems as probabilistic graphical models (PGMs) offers a consistent and flexible framework for deriving principled objectives, analyzing convergence, and devising practical algorithms. In this section, we first formulate the optimal trajectory of the conventional SMDP-based option framework (Eq. 1) as a PGM. We then use the HiT-MDPs as the variational distribution to approximate this optimal trajectory. With these PGMs, we can straightforwardly derive the variational objective, where maximum entropy terms arise naturally. This approach allows us to develop a stable algorithm for learning diversified options and preventing degeneracy. Specifically, we follow [31, 28]

Ot-1

Ot

Ot-1

0

t

at-1

at

at-1

at

EA

St-1

St

St-1

St

(a) Optimal Option Trajectory

(b) Variational Distribution (HiT-MDP)

as a probabilistic graphical model (PGM), as illustrated in Figure 1 (a): 159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

where E ∈ { 0 , 1 } are observable binary 'optimal random variables' [31], τ = { s 0 , o 0 , a 0 , s 1 . . . } denotes the trajectory of the option framework. The agent is optimal at time step t when P ( E A t = 1 | s t , a t ) and P ( E O t = 1 | s t , a t , o t , o t -1 ) . We will use E instead of E = 1 in the following text to avoid cluttered notations. To simplify the derivation, priors P ( o ) and P ( a ) can be assumed to be uniform distributions without loss of generality [31]. Note that Eq. 3 shares the same environment dynamics with Eq. 1 and Eq. 2. With the optimal random variables E O and E A , the likelihood of a state-action { s t , a t } pair that is optimal is defined as:

as this specific design facilitates recovering the value function at the latter structural variational inference stage. Based on the same motivation, the likelihood of an option-state-action { o t , s t , a t , o t -1 } pair that is optimal is defined as,

where f ( · ) is an arbitrary non-positive function which measures the preferable of selecting an option given state-action pair [ s t , a t ] and the previous executed option o t -1 . In this work, we choose f to be the mutual-information f = I [ o t | s t , a t , o t -1 ] as a fact that when the uniform prior assumption of P ( o ) is relaxed the optimization introduces a mutual-information as a regularizer [35].

As explained in Section 2.1, direct optimization of Eq. 3 results in optimistic policies that assumes a degree of control over the dynamics. We correct this risk-seeking behavior [31] through approximating the optimal trajectory P ( τ ) with the variational distribution:

where the initial distribution P ( s 0 , o 0 ) and transition distribution P ( s t +1 | s t , a t ) is set to be the true environment dynamics from P ( τ ) . The variational distribution turns out to be the HiT-MDP, where the action policy q ( a t | s t ) and the option policy q ( o t | s t , o t -1 ) are used to approximate the optimal policy P ( a t | s t , o t , E A 1: T ) and P ( o t | s t , o t -1 , E O 1: T ) . The Evidence Lower Bound (ELBO) [28] of the log-likelihood optimal trajectory (Eq. 3) can be derived as (see Appendix A.3):

where line 2 is substituting Eq. 3 and Eq. 6 into the ELBO. As a result, the maximum entropy objective naturally arises in Eq. 7. Optimizing the ELBO not only seeks high-reward options but also maximizes entropy across the space, promoting extensive exploration and maintaining high diversity.

Given the ELBO, we now define soft value functions of the option framework following the Bellman Backup Functions along the trajectory q ( τ ) as bellow:

Assuming policies π A , π O ∈ Π where Π is an arbitrary feasible set, under a tabular setting where the 187 inference on L can be done exactly, we have the following theorem holds: 188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

Theorem 2 (Soft Option Policy Iteration Theorem) . Repeated optimizing L and D KL defined in Eq. 10 from any π A 0 , π O 0 ∈ Π converges to optimal policies π A ∗ , π O ∗ such that Q soft ∗ O [ s t , o t ] ≥ Q soft O [ s t , o t ] and Q soft ∗ A [ s t , o t , a t ] ≥ Q soft A [ s t , o t , a t ] , for all π A 0 , π O 0 ∈ Π and ( s t , a t , o t ) ∈ S × A × O , assuming under tabular settings where |S| < ∞ , |O| < ∞ , |A| < ∞ .

Theorem 2 guarantees finding the optimal solution only when the inference can be done exactly under tabular settings. However, real-world applications often involve large continuous domains and employ neural networks as function approximators. In these cases, inference procedures can only be done approximately. This necessitate a practical approximation algorithm which we present below.

3.2 Variational Markovian Option Critic Algorithm

Formulating complex problems as probabilistic graphical models (PGMs) allowing us to leverage established methods from PGM literature to address the associated inference and learning challenges in real-world applications. To this end, we utilizes the structured variational inference treatment for optimizing the log-likelihood of optimal trajectory and prove its convergence under approximate inference settings. Specifically, using the variational distribution q ( τ ) (Eq. 6) as an approximator, the ELBO can be derived as (see Appendix A.3):

where D KL is the KL-Divergence between the trajectory following variational policies q ( τ ) and optimal policies P ( τ |E A 1: T , E O 1: T ) . Under the structural variational inference [28] perspective, convergence to the optimal policy can be achieved by optimizing the ELBO with respect to the the variational policy repeatedly:

Theorem 3 (Convergence Theorem for Variational Markovian Option Policy Iteration) . Let τ be the latent variable and E A , E O be the ground-truth optimality variables. Define the variational distribution q ( τ ) and the true log-likelihood of optimality log P ( E A , E O ) . iterates according to the update rule q k +1 = arg max q L ( q ( τ ) , P ( τ, E A 1: T , E O 1: T )) converges to the maximum value bounded by the true log-likelihood of optimality.

We further implements a practical algorithm, the Variational Markovian Option Critic (VMOC) algorithm, which is suitable for complex continuous domains. Specifically, we employ parameterized neural networks as function approximators for both the Q-functions ( Q soft ψ A , Q soft ψ O ) and the policies ( π θ A , π θ O ). Instead of running evaluation and improvement to full convergence using Theorem 2, we can optimize the variational distribution by taking stochastic gradient descent following Theorem 3 with respect to the ELBO (Eq. 7) directly. Share the same motivation with Haarnoja et al. [19] of reducing the variance during the optimization procedure, we derive an option critic framework by optimizing the maximum entropy objectives between the action Eq. 9 and the option Eq. 8 alternatively. The Bellman residual for the action critic is:

where α O is the temperature hyper-parameter and the expectation over option random variable 224 E o t +1 ∼ π O can be evaluated exactly since π O is a discrete distribution. The Bellman residual for the 225 option critic is: 226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

α A is the temperature hyper-parameter. Unlike E o t +1 ∼ π O can be trivially evaluated, evaluating E a t ∼ π A is typically intractable. Therefore, in implementation we use a t sampled from the replay buffer to estimate the expectation over π A .

Following Theorem 3, the policy gradients can be derived by directly taking gradient with respect to the ELBOs defined for the action Eq. 9 and the option Eq. 8 policies respectively. The action policy objective is given by:

where in practice the action policy is often sampled by using the re-parameterization trick introduced in [19]. The option objective is given by:

The variational distribution q ( τ ) defined in Eq. 6 allows us to learn options as embeddings [34, 35] with a learnable embedding matrix W ∈ R num_options × embedding_dim . Under this setting, the embedding matrix W can be absorbed into the parameter vector θ O . This integration into VMOC ensures that options are represented as embeddings without any additional complications, thereby enhancing the expressiveness and scalability of the model.

The temperature hyper-parameters can also be adjusted by minimizing the following objective:

for the action policy temperature α A , where H is a target entropy. Similarly, the option policy temperature α O can be adjusted by:

where H is also a target entropy for the option policy. In both cases, the temperatures α A and α O are updated using gradient descent, ensuring that the entropy regularization terms dynamically adapt to maintain a desired level of exploration. This approach aligns with the methodology proposed in SAC [19]. By adjusting the temperature parameters, the VMOC algorithm ensures a balanced trade-off between exploration and exploitation, which is crucial for achieving optimal performance in complex continuous control tasks. We summarize the VMOC algorithm in Appendix B.

4 Experiments

In this section, we design experiments on the challenging single task OpenAI Gym MuJoCo [7] environments (10 environments) to test Variational Markovian Option Critic (VMOC)'s performance over other option variants and non-option baselines.

For VMOC in all environments, we fix the temperature rate for both α O and α A to 0 . 05 ; we add an exploration noise N ( µ = 0 , σ = 0 . 2) during exploration. For all baselines, we follow DAC [52]'s open source implementations and compare our algorithm with six baselines, five of which are option variants, i.e. , MOPG [35], DAC+PPO, AHP+PPO [32], IOPG [45], PPOC [27], OC [4] and PPO [41]. All baselines' parameters used by DAC remain unchanged over 1 million environment steps to converge. Figures are plotted following DAC's style: curves are averaged over 10 independent runs and smoothed by a sliding window of size 20. Shaded regions indicate standard deviations. All experiments are run on an Intel® Core™ i9-9900X CPU @ 3.50GHz with a single thread and process. Our implementation details are summarized in Appendix C. For a fair comparison, we follow option literature conventions and use four options in all implementations. Our code is available in supplemental materials.

5 Experiments

We evaluate the performance of VMOC against six option-based baselines (MOPG [35], DAC+PPO [52], AHP+PPO [32], IOPG [45], PPOC [27], and OC [4]) as well as the hierarchy-free

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

PPO algorithm [41]. Previous studies [27, 45, 20, 52] have suggested that option-based algorithms do not exhibit significant advantages over hierarchy-free algorithms in single-task environments. Nonetheless, our results demonstrate that VMOC significantly outperforms all baselines in terms of episodic return, convergence speed, step variance, and variance across 10 runs, as illustrated in Figure 2. The only exception is the relatively simple InvertedDoublePendulum environment, which

we suspect is due to hyper-parameter tuning issues and will be addressed in future work.

7500

DAC+PPO

PO

125000

4000

AHP+PPO

4000

5000

PPOC

100000

3000

2000

2000

75000

2500-

1000

50000

0

0

0

0

1 ×106

0

1 ×106

0

1 ×106

0

1 ×106

Steps

Steps

Steps

Steps

Hopper-v2

Swimmer-v2

Reacher-v2

3000

100

-10

3000

75

2000

2000

-20

50

-30

1000

1000

25

-40

0

0

0

0

1 × 106

0

1 ×106

0

1 ×106

0

1 × 106

Steps

Steps

Steps

Steps

InvertedPendulum-v2

InvertedDoublePendulum-v2

1000

8000

750

6000

500

4000

250

2000

01

0L

0

1 × 106

0

1 × 106

Steps

Steps

A particularly relevant comparison is with the Markovian Option Policy Gradient (MOPG) [35], as both VMOC and MOPG are developed based on HiT-MDPs and employ option embeddings. Despite being derived under the maximum entropy framework, MOPG utilizes an on-policy gradient descent approach. Our experimental results show that VMOC's performance surpasses that of MOPG, highlighting the limitations of on-policy methods, which suffer from shortsighted rollout lengths and quickly saturate to early high-reward observations. In contrast, VMOC's variational off-policy approach effectively utilizes the maximum entropy framework by ensuring better exploration and stability across the learning process. Additionally, the off-policy nature of VMOC allows it to reuse samples from a replay buffer, enhancing sample efficiency and promoting greater diversity in the learned policies. This capability leads to more robust learning, as the algorithm can leverage a broader range of experiences to improve policy optimization.

6 Related Work

The VMOC incorporates three key ingredients: the option framework, a structural variational inference based off-policy algorithm and latent variable policies. We review prior works that draw

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

on some of these ideas in this section. The options framework [47] offers a promising approach for discovering and reusing temporal abstractions, with options representing temporally abstract skills. Conventional option frameworks [39], typically developed under the maximum likelihood (MLE) framework with few constraints on options behavior, often suffer from the option degradation problem [32, 4]. This problem occurs when options quickly saturate with early rewarding observations, causing a single option to dominate the entire policy, or when options switch every timestep, maximizing policy at the expense of skill reuse across tasks. On-policy option learning algorithms [4, 3, 52, 34, 35] aim to maximize expected return by adjusting policy parameters to increase the likelihood of high-reward option trajectories, which often leads to focusing on low-entropy options. Several techniques [20, 21, 23] have been proposed to enhance on-policy algorithms with entropy-like extrinsic rewards as regularizers, but these often result in biased optimal trajectories. In contrast, the maximum entropy term in VMOC arises naturally within the variational framework and provably converges to the optimal trajectory.

Although several off-policy option learning algorithms have been proposed [10, 43, 45, 50], these typically focus on improving sample efficiency by leveraging the control as inference framework. Recent works [45] aim to enhance sample efficiency by inferring and marginalizing over options, allowing all options to be learned simultaneously. Wulfmeier et al. [50] propose off-policy learning of all options across every experience in hindsight, further boosting sample efficiency. However, these approaches generally lack constraints on options behavior. A closely related work [33] also derives a variational approach under the option framework; however, it is based on probabilistic graphical model that we believe are incorrect, potentially leading to convergence issues. Additionally, our algorithm enables learning options as latent embeddings, a feature not present in their approach.

Recently, several studies have extended the maximum entropy reinforcement learning framework to discover skills by incorporating additional latent variables. One class of methods [22, 17] maintains latent variables constant over the duration of an episode, providing a time-correlated exploration signal. Other works [19, 51] focus on discovering multi-level action abstractions that are suitable for repurposing by promoting skill distinguishability, but they do not incorporate temporal abstractions. Studies such as [38, 1, 8] aim to discover temporally abstract skills essential for exploration, but they predefine their temporal resolution. In contrast, VMOC learns temporal abstractions as embeddings in an end-to-end data-driven approach with minimal prior knowledge encoded in the framework.

7 Conclusion

In this paper, we have introduced the Variational Markovian Option Critic (VMOC), a novel off-policy algorithm designed to address the challenges of ineffective exploration, sample inefficiency, and computational complexity inherent in the conventional option framework for hierarchical reinforcement learning. By integrating a variational inference framework, VMOC leverages maximum entropy as intrinsic rewards to promote the discovery of diverse and effective options. Additionally, by employing low-cost option embeddings instead of traditional, computationally expensive option triples, VMOC enhances both scalability and expressiveness. Extensive experiments in challenging Mujoco environments demonstrate that VMOC significantly outperforms existing on-policy and off-policy option variants, validating its effectiveness in learning coherent and diverse option sets suitable for complex tasks. This work advances the field of hierarchical reinforcement learning by providing a robust, scalable, and efficient method for learning temporally extended actions.

8 Limitations

Due to limited computing resources, we did not conduct an ablation study of VMOC. Additionally, 338 the temperature parameter was fixed in our experiments, whereas an automatically tuned parameter 339 could potentially enhance performance (see SAC [19]). While our baselines focus on option variants, 340 a thorough comparison to other off-policy algorithms is also worth investigating. It is particularly 341 important to explore whether VMOC exhibits performance improvements in scalability when the 342 number of option embeddings is significantly increased. These investigations are left for future work. 343

344

345

346

347

348

349

References

Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611 , 2020.

Araujo, E. G. and Grupen, R. A. Learning control composition in a complex environment. In Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior , pp. 333-342, 1996.

Bacon, P.-L. Temporal Representation Learning . PhD thesis, McGill University Libraries, 2018. 350

Bacon, P.-L., Harb, J., and Precup, D. The option-critic architecture. In Thirty-First AAAI 351 Conference on Artificial Intelligence , 2017. 352

Bertsekas, D. and Tsitsiklis, J. N. Neuro-dynamic programming . Athena Scientific, 1996. 353

Brockett, R. W. Hybrid models for motion control systems. In Essays on Control: Perspectives 354 in the Theory and its Applications , pp. 29-53. Springer, 1993. 355

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, 356 W. Openai gym. arXiv preprint arXiv:1606.01540 , 2016. 357

358

359

360

Co-Reyes, J., Liu, Y., Gupta, A., Eysenbach, B., Abbeel, P., and Levine, S. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In International conference on machine learning , pp. 1009-1018. PMLR, 2018.

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

Colombetti, M., Dorigo, M., and Borghi, G. Behavior analysis and training-a methodology for behavior engineering. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 26(3):365-380, 1996.

Daniel, C., Van Hoof, H., Peters, J., and Neumann, G. Probabilistic inference for determining options in reinforcement learning. Machine Learning , 104(2-3):337-357, 2016.

Dayan, P. and Hinton, G. E. Feudal reinforcement learning. Advances in Neural Information Processing Systems , pp. 271-278, 1993.

Dietterich, T. G. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of Artificial Intelligence Research , 13:227-303, 2000.

Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

Fujimoto, S., Van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477 , 2018.

Goyal, A., Islam, R., Strouse, D., Ahmed, Z., Botvinick, M., Larochelle, H., Bengio, Y., and Levine, S. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint arXiv:1901.10902 , 2019.

Guo, Z., Thomas, P. S., and Brunskill, E. Using options and covariance testing for long 377 horizon off-policy policy evaluation. In Advances in Neural Information Processing Systems , 378 pp. 2492-2501, 2017. 379

380

381

382

Gupta, A., Kumar, V., Lynch, C., Levine, S., and Hausman, K. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956 , 2019.

Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy383 based policies. In International Conference on Machine Learning , pp. 1352-1361. PMLR, 384 2017. 385

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum 386 entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 , 387 2018. 388

Harb, J., Bacon, P.-L., Klissarov, M., and Precup, D. When waiting is not an option: Learning 389 options with a deliberation cost. In Thirty-Second AAAI Conference on Artificial Intelligence , 390 2018. 391

Harutyunyan, A., Dabney, W., Borsa, D., Heess, N., Munos, R., and Precup, D. The termination 392 critic. arXiv preprint arXiv:1902.09996 , 2019. 393

Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., and Riedmiller, M. Learning an 394 embedding space for transferable robot skills. In International Conference on Learning Repre395 sentations , 2018. 396

Kamat, A. and Precup, D. Diversity-enriched option-critic. arXiv , 2020. 397

Khetarpal, K. and Precup, D. Learning options with interest functions. In Proceedings of the 398 32nd AAAI Conference on Artificial Intelligence , pp. 1-2, 2019. 399

400

401

402

Khetarpal, K., Klissarov, M., Chevalier-Boisvert, M., Bacon, P.-L., and Precup, D. Options of interest: Temporal abstraction with interest functions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 4,444-4,451, 2020.

403

404

405

406

407

408

409

410

411

Klissarov, M. and Precup, D. Flexible option learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems , volume 34, pp. 4632-4646. Curran Associates, 2021.

Klissarov, M., Bacon, P.-L., Harb, J., and Precup, D. Learnings options end-to-end for continuous action tasks. arXiv preprint arXiv:1712.00004 , 2017.

Koller, D. and Friedman, N. Probabilistic graphical models: principles and techniques . MIT press, 2009.

Kolobov, A., Weld, D. S., et al. Discovering hidden structure in factored mdps. Artificial Intelligence , 189:19-47, 2012.

Konidaris, G. and Barto, A. G. Skill discovery in continuous reinforcement learning domains 412 using skill chaining. In Advances in neural information processing systems , pp. 1015-1023, 413 2009. 414

Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review. 415 arXiv preprint arXiv:1805.00909 , 2018. 416

417

418

Levy, K. Y. and Shimkin, N. Unified inter and intra options learning using policy gradient methods. In European Workshop on Reinforcement Learning , pp. 153-164. Springer, 2011.

419

420

421

422

423

424

425

426

427

428

429

430

431

432

Li, C., Ma, X., Zhang, C., Yang, J., Xia, L., and Zhao, Q. Soac: The soft option actor-critic architecture. arXiv preprint arXiv:2006.14363 , 2020.

Li, C., Song, D., and Tao, D. The skill-action architecture: Learning abstract action embeddings for reinforcement learning. 2020.

Li, C., Song, D., and Tao, D. Hit-mdp: learning the smdp option framework on mdps with hidden temporal embeddings. In The Eleventh International Conference on Learning Representations , 2022.

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

Osa, T., Tangkaratt, V., and Sugiyama, M. Hierarchical reinforcement learning via advantageweighted information maximization. arXiv preprint arXiv:1901.01365 , 2019.

Pertsch, K., Rybkin, O., Ebert, F., Finn, C., Jayaraman, D., and Levine, S. Long-horizon visual planning with goal-conditioned hierarchical predictors. NeurIPS , 2020.

Precup, D. Temporal abstraction in reinforcement learning . University of Massachusetts 433 Amherst, 2000. 434

Schulman, J., Chen, X., and Abbeel, P. Equivalence between policy gradients and soft q-learning. 435 arXiv preprint arXiv:1704.06440 , 2017. 436

437

438

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657 , 2019.

Shiarlis, K., Wulfmeier, M., Salter, S., Whiteson, S., and Posner, I. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning , pp. 4654-4663. PMLR, 2018.

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.

Smith, M., Hoof, H., and Pineau, J. An inference-based policy gradient method for learning options. In International Conference on Machine Learning , pp. 4,703-4,712, 2018.

Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction . MIT press, 2018.

Sutton, R. S., Precup, D., and Singh, S. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence , 112(1-2):181-211, 1999.

Todorov, E. Linearly-solvable markov decision problems. Advances in neural information processing systems , 19, 2006.

Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 5026-5033. IEEE, 2012.

Wulfmeier, M., Rao, D., Hafner, R., Lampe, T., Abdolmaleki, A., Hertweck, T., Neunert, M., Tirumala, D., Siegel, N., Heess, N., et al. Data-efficient hindsight off-policy option learning. arXiv preprint arXiv:2007.15588 , 2020.

Zhang, D., Courville, A., Bengio, Y., Zheng, Q., Zhang, A., and Chen, R. T. Latent state marginalization as a low-cost approach for improving exploration. arXiv preprint arXiv:2210.00999 , 2022.

Zhang, S. and Whiteson, S. DAC: The double actor-critic architecture for learning options. In Advances in Neural Information Processing Systems , pp. 2,012-2,022, 2019.

Ziebart, B. D., Bagnell, J. A., and Dey, A. K. Modeling interaction via the principle of maximum causal entropy. In ICML , 2010.

A Proofs

A.1 Theorem 1

Theorem 1 (Convergence Theorem for Structured Variational Policy Iteration). Let τ be the latent variable and E be the observed variable. Define the variational distribution q ( τ ) and the log-likelihood log P ( E ) . Let M : q [ k ] → q [ k +1] represent the mapping defined by the EM steps inference update, so that q [ k +1] = M ( q [ k ] ) . The likelihood function increases at each iteration of the variational inference algorithm until the conditions for equality are satisfied and a fixed point of the iteration is reached:

Proof. Let τ be the latent variable and E be the observed variable. Define the evidence lower bound 477 (ELBO) as L ( q, P ) and the Kullback-Leibler divergence as DKL ( q ∥ P ) , where q ( τ ) approximates 478 the posterior distribution and P ( E | τ ) is the likelihood. 479

The log-likelihood function log P ( E ) can be decomposed as: 480

Let M : q [ k ] → q [ k +1] represent the mapping defined by the variational inference update, so that q [ k +1] = M ( q [ k ] ) . If q ∗ is a variational distribution that maximizes the ELBO, so that log P ( E | q ∗ ) ≥ log P ( E | q ) for all q , then log P ( E | M ( q ∗ )) = log P ( E | q ∗ ) . In other words, the maximizing distributions are fixed points of the variational inference algorithm. Since the likelihood function is bounded (for distributions of practical interest), the sequence of variational distributions q [0] , q [1] , . . . , q [ k ] yields a bounded nondecreasing sequence log P ( E | q [0] ) ≤ log P ( E | q [1] ) ≤ · · · ≤ log P ( E | q [ k ] ) ≤ log P ( E | q [ k ] ) which must converge as k →∞ .

A.2 Theorem 2

Theorem 2 (Soft Option Policy Iteration Theorem). Repeated optimizing L and D KL defined in Eq. 10 from any π A 0 , π O 0 ∈ Π converges to optimal policies π A ∗ , π O ∗ such that Q soft ∗ O [ s t , o t ] ≥ Q soft O [ s t , o t ] and Q soft ∗ A [ s t , o t , a t ] ≥ Q soft A [ s t , o t , a t ] , for all π A 0 , π O 0 ∈ Π and ( s t , a t , o t ) ∈ S × A × O , assuming |S| < ∞ , |O| < ∞ , |A| < ∞ .

Proof. Define the entropy augmented reward as r soft ( s t , a t ) = r ( s t , a t ) + H [ π A ] and f soft ( o t , s t , a t , o t -1 ) = f ( o t , s t , a t , o t -1 ) + H [ π O ] and rewrite Bellman Backup functions as, where 481

and 482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

We start with proving the convergence of soft option policy evaluation. As with the standard Q498 function and value function, we can relate the Q-function at a future state via a Bellman Operator 499 T soft . The option-action value function satisfies the Bellman Operator T soft 500

As with the standard convergence results for policy evaluation [46], by the definition of T soft (Eq. 11) 501 the option-action value function Q π A A is a fixed point. 502

To prove the T soft is a contraction, define a norm on V -values functions V and U 503

where ¯ s = { s, o } . 504

By recurssively apply the Hidden Temporal Bellman Operator T soft , we have: 505

Therefore, by applying Eq. 12 to V and U we have: 506

507

508

509

510

511

512

513

A.3 Derivation of Eq. 10 514

515

516

517

518

519

520

A.4 Theorem 3

Theorem 3 (Convergence Theorem for Variational Markovian Option Policy Iteration). Let τ be the latent variable and E A , E O be the ground-truth optimality variables. Define the variational distribution q ( τ ) and the true log-likelihood of optimality log P ( E A , E O ) . iterates according to the update rule q k +1 = arg max q L ( q ( τ ) , P ( τ, E A 1: T , E O 1: T )) converges to the maximum value bounded by the data log-likelihood.

Therefore, T soft is a contraction. By the fixed point theorem, assuming that throughout our computation the Q A [ · , · ] and Q O [ · ] are bounded and A < ∞ , the sequence Q k A defined by Q k +1 A = T soft Q k A will converge to the option-action value function Q π A A as k →∞ .

The convergence results of and the Soft Option Policy Improvement Theorem then follows conventional Soft Policy Improvement Theorem Theorem 1. Consequently, the Soft Option Policy Iteration Theorem follows directly from these results.

Proof. The objective is to maximize the ELBO with respect to the policy q . Formally, this can be 521 written as: 522

Suppose we q is a neural network function approximator, assuming the continuity and differentiability 523 of q with respect to its parameters. Using stochastic gradient descent (SGD) to optimize the parameters 524 guarantees that the ELBO increases, such that L ( q k +1 , P ) ≥ L ( q k , P ) . 525

Rearranging Eq. 10 we get: 526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

Thus, each SGD update not only potentially increases the ELBO but also decreases the KL divergence, moving q closer to P . Given the properties of SGD and assuming appropriate learning rates and sufficiently expressive neural network architectures, the sequence { q k } converges to a policy q ∗ that minimizes the KL divergence to the true posterior.

B VMOCAlgorithm

Algorithm 1 VMOC Algorithm

1: Initialize parameter vectors ψ A , ψ O , θ O , θ A

2: for each epoch do

3: Collect trajectories { o t -1 , s t , a t , o t } into the replay buffer

4: for each gradient step do

7: Update π O θ O : θ O ← θ O -η π O ∇ J π O

8: Update π A θ A : θ A ← θ A -η π A ∇ J π A

9:

Update target networks:

¯

ψ

A

σψ

A

+(1

-

σ

) ¯

ψ

A

,

¯

ψ

O

σψ

O

+(1

-

σ

) ¯

ψ

O

10: Update temperature factors: α O ← α O -η α O ∇ J α O , α A ← α A -η α A ∇ J α A

11: end for

12: end for

C Implementation Details

C.1 Hyperparameters

In this section we summarize our implementation details. For a fair comparison, all baselines: MOPG [35], DAC+PPO [52], AHP+PPO [32], PPOC [27], OC [4] and PPO [41] are from DAC's open source Github repo: https://github.com/ShangtongZhang/DeepRL/tree/DAC . Hyperparameters used in DAC [52] for all these baselines are kept unchanged.

VMOCNetwork Architecture: We use Pytorch to build neural networks. Specifically, for option embeddings, we use an embedding matrix W S ∈ R 4 × 40 which has 4 options ( 4 rows) and an embedding size of 40 ( 40 columns). For layer normalization we use Pytorch's built-in function LayerNorm 2 . For Feed Forward Networks (FNN), we use a 2 layer FNN with ReLu function as activation function with input size of state-size, hidden size of [256 , 256] , and output size of actiondim neurons. For Linear layer, we use built-in Linear function 3 to map FFN's outputs to 4 dimension.

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

Each dimension acts like a logit for each skill and is used as density in Categorical distribution 4 . For both action policy and critic module, FFNs are of the same size as the one used in the skill policy. Preprocessing: States are normalized by a running estimation of mean and std. Hyperparameters for all on-policy option variants: For a fair comparison, we use exactly the same parameters of PPO as DAC . Specifically: · Optimizer: Adam with ϵ = 10 -5 and an initial learning rate 3 × 10 -4 · Discount ratio γ : 0 . 99 · GAE coefficient: 0 . 95 · Gradient clip by norm: 0 . 5 · Rollout length: 2048 environment steps · Optimization epochs: 10 · Optimization batch size: 64 · Action probability ratio clip: 0 . 2 We conducted our experiments on an Intel® Core™ i9-9900X CPU @

Computing Infrastructure: 3.50GHz with a single thread and process with PyTorch.

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the claims and findings of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations of the study are discussed in the discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides a full derivation of assumptions and proofs of the theoretical result (convergence of the evidence lower bound)

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes. Our code is provided in supplementary materials. Full details of the experimental setup, model architectures are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides open access to the code and data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Justification: All details are provided in the main content and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All gym env experiments are run with 10 different random seeds. Performance are reported by 1 sigma shaded area over all 10 runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Computational details are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research was conducted in accordance with the NeurIPs Code of Ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work in the paper has no potential for societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The only applicable assets are the code which are credited and distributed under a Creative Commons Attribution License.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: New assets include the code required to run the experiments described in the paper. Documentation is provided along with the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an off-policy hierarchical RL method, based on the HiT-MDP formulation of a Semi-MDP. The HiT-MDP formulation treats the option $o$ as an extension of the original state $s$ (which can be chosen by an extended action), and combines initialization-, termination- and option-policy in a single Markovian master policy $p(o\\_{t}|s\\_t,o\\_{t-1})$. The policy in the extended state-action space, thus, decomposes into the high-level and low-level policies, $p(o\\_{t}, a\\_t | s\\_t, o\\_{t-1}) = p(o\\_{t} | s\\_t, o\\_{t-1}) p(a\\_t | s\\_t, o\\_{t})$, which can be trained using standard RL algorihms. \nCompared to the prior work, the paper makes the following contributions:\n- Whereas previously PPO was used for reinforcement learning, the paper proposes to use SAC, resulting in improved sample efficiency\n- The paper motivates the algorithm from a control-as-inference perspective\n\nSTRENGTHS:\nThe proposed method seems to be technically sound, and using off-policy agents for HiT-MDPs seems sensible. (Quality)\n\nThe provided code clarifies the implementation which helps reproducibility. (Quality)\n\nThe presentation is mostly clear. (Clarity)\n\nApplying an off-policy agents to HiT-MDPs seems to be novel and effective (Origingality, Significance)\n\nWEAKNESSES:\nOriginality\n-----------\nOne of the main weaknesses of the submission is the limited novelty. Replacing the PPO agent of MOPG by a SAC agent seems to be straightforward, so this contribution is quite incremental. Indeed, the authors of HiT-MDP stated, that their ELBO \"can easily be extended to a SAC-like algorithm\" [35]. Furthermore, given that MaxEnt-RL was already derived from a control-as-inference perspective, deriving the special case of an HiT-MDP using this technique does not seem to be significant contribution either. I also don't see the value of this derivation that would justify devoting so much space on it; couldn't we just argue that we apply SAC to such particular form of an MDP?\n\n\nQuality\n---------\nThe experimental evaluation seems to be another weakness of the submission. While the method is evaluated on a reasonable number of MuJoCo environments, where it outperforms a reasonable number of baselines, the choice of baselines is not convincing because it looks like the method is only compared to on-policy algorithm. The submission claims that there method \"significantly outperforms existing on-policy and off-policy option variants\", but it is not clear to me to which off-policy baselines this claim refers to. It would be important to focus to flat and hierarchical off-policy methods in the experiments, such as [19], [50], [33] and Hao et. al (2023).  Furthermore, the choice of environments is not convincing, because it does not include more challenging long-horizon tasks that are typically used for evaluating HRL methods, such as Ant-Maze. While the performance on the standard locomotion environments is reasonable, the reported numbers don't seem to improve on the SOTA of flat-RL methods.   \n\nThe paper does not discuss the hyperparameter search although it states in the questionnary  that these details are provided in the main content and the appendix.\n\nThe paper argues that it did not perform any ablations due to limited computational resources. However, I don't find this argument very convincing, since the experiments are performed on simple vision-free locomotion tasks, that can be run on standard workstation, not even requiring any GPU. Ablations on the number of options would be very useful.\n\n\n\nClarity\n---------\nI found the background material on control-as-inference a bit confusing. In particular, line 106 which states states policy improvement constitues an M-Step of an EM algorithm that *maximizes* the KL towards $P(\\tau|\\mathcal{E})$. I don't think any practical algorithm involves such maximization, since the optimum would correspond to a delta distribution on the least-likely trajectory. (\n\nVisually, the presentation is rather bad. Figures are not on the top, and in particular Fig. 2 seems to hide some text, since the sentence in line 271 ends with \", which\". Fig. 2 itself could be improved by increasing the plot sizes (there are some unnecessary white spaces) and by making the legend more readable.  \n\n\nSignificance\n-----------------\nWhile I think that the proposed combination of the HiT-MDP formulation and SAC is somewhat interesting, the submission does not provide a convincing argument for the method. When should I use it, instead of existing (hierarchical or flat) methods?\n\nReferences\n----------\n* Hao, C., Weaver, C., Tang, C., Kawamoto, K., Tomizuka, M., & Zhan, W. (2023). Skill-critic: Refining learned skills for reinforcement learning. arXiv preprint arXiv:2306.08388.\n\nQUESTIONS:\n* Line 227 mentions that the targets for the option Q-function are computed using actions from the replay buffer because estimating the expectation with respect to samples from the policy would be intractable. I don't understand this: Why can't we just sample from the policy?\n\n* Which of the baselines where off-policy?\n\nLIMITATIONS:\nThe limitations are adequately discussed and I don't have any concerns regarding negative societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

comments: SUMMARY:
This paper proposes the Variational Markovian Option Critic (VMOC), an off-policy algorithm for hierarchical reinforcement learning. VMOC aims to address exploration inefficiency and update instability in existing methods. Key contributions include: 1. Use of variational inference for update stabilization 2. Low-cost option embeddings for improved scalability. The authors evaluate VMOC on Mujoco environments, comparing it to other on-policy and off-policy methods. They report improved performance in learning option sets for complex tasks.

STRENGTHS:
1. The paper is well-written, and the proposed method is theoretically justified.
2. The empirical evaluations show favorable results compared with existing methods.

WEAKNESSES:
1. Very similar ideas of the variational option framework have been proposed in [33] (off-policy) and [35] (on-policy). While [35] proposes an on-policy version, its off-policy version is also straightforward to deduce following [ref1]. The use of option embeddings is following [35].
2. The empirical evaluations are very limited; there is no ablative evaluation reported, which makes it hard to determine the contribution of the proposed method to the overall performance gain over various baselines.

References: 
[ref1] Levine, Sergey. "Reinforcement learning and control as probabilistic inference: Tutorial and review." arXiv preprint arXiv:1805.00909 (2018).

QUESTIONS:
The authors claim that [33] is based on an "incorrect" probabilistic graphical model. I wonder if the authors could elaborate on this claim.

LIMITATIONS:
The empirical evaluations, especially ablation studies, are somewhat limited in scope.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Variational Markovian Option Critic (VMOC) which combines variation policy iteration and the option critic. VMOC also modifies HiT-MDPs, where options are represented as latent embeddings rather than triples of (init states, policy, termination condition), to the off-policy setting. The paper performs comparisons to option-based methods and PPO on 10 Mujoco environments.\n\nSTRENGTHS:\n1. The paper is well-written and easy-to-read. The figures clearly highlight the performance of the method. The translation from theory to the practical algorithm is well detailed.\n\n2. The advantage in sample-efficiency over other option methods and PPO is clearly seen in Figure 2 across Mujoco environments. In fact, this gain looks to be in atleast two orders-of-magnitude (of fewer steps required by VMOC) which is amazing. The underlying MaxEnt objective in VMOC appears to be very useful with exploration in the high-dim mujoco envs.\n\nWEAKNESSES:\n1. It is not clear if this gain in sample-efficiency will transfer to discrete environments or is somehow applicable only in continuous envs. Perhaps the authors can perform comparisons on Atari or Procgen to demonstrate the same? It would be great if the authors could also discuss the changes in the algorithm in the discrete and continuous settings (perhaps such as the sampling of a_t from the replay buffer?)\n\n2. It is unclear if all methods use the same number of options (e.g. the value used in VMOC appears to be 4). A clear ablation of various design choices like number of options would help demonstrate that VMOC is thoroughly better than the other option methods and is not brittle to hyperparameter choice. \n\nThe analysis of the actual options learnt is also missing (this is for example seen in the option critic paper). This, alongside an analysis of the number of options, is crucial to understand if the method is actual learning composed actions that are further composable and generalizable or degenerating to something simple like learning the action primitives (although the latter would apply more to a discrete rather than continuous env).\n\n3. Minor comment: The location of Theorem 1 in the preliminaries makes it unclear if it is a contribution of the authors or well-known statement. Perhaps the authors can clarify?\n\n4. Another minor comment: It would be great if the authors could discuss other ways of combining options in the related work such as in [1] and [2].\n\n[1] The Option Keyboard: Combining Skills in Reinforcement Learning, Barreto et al, NeurIPS 2019\n\n[2] Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates, Dutta et al, CoRL 2022\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nThe authors have addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces e Variational Markovian Option Critic (VMOC), which learns actions and options simultatenously. They build upon the Hidden Temporal Markovian Decision Process (HiT-MDP) [1] to build a novel off-policy algorithm that utilizes entropy augmented rewards. Their method learns options’ embedding vectors (rather than conventional option tuples utilized in Semi-MDP [2]). They benchmark the learning performance of their method against several competitors on many classic control benchmark environments. \n\n*References*\n1. Li, C., Song, D., & Tao, D. (2023). Hit-MDP: learning the SMDP option framework on MDPs with hidden temporal embeddings. In The Eleventh International Conference on Learning Representations.\n2. Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2), 181-211.\n\nSTRENGTHS:\n- Extensive comparison against 8 competitor algorithms on 10 benchmark tasks\n- A novel Soft Option Policy Iteration Theorem\n\nWEAKNESSES:\nThe paper does offer a potentially interesting contribution to the wider research community. But it is held back by the lack of clarity and polish in writing. For example, two glaring signs of a hasty submission:\n1. Sec 4 and Sec 5 are both titled experiments. Sec 4 is only 1 paragraph, and it essentially repeats the same information from the introductory paragraphs of Sec 5\n2. In Sec. 5, line 271 just trails off without completion. I believe the authors moved around the images to correct for vertical space and accidentally hid the text.\n\nWhile the experimental results focus on learning curves, where VMOC does well, they fail to provide other relevant evaluation metrics:\n1. What do the learned options look like? A good evaluation could follow Fig. 5 and Fig. 6 from [1]\n2. How many options are learned? Digging through the appendix, it says that they learned 4 option vectors. This leads to another question: how do they choose the number of options to learn?\n3. The VMOC algorithm listed in the appendix only describes the gradient update process. No details about action sampling or other hyper-parameter tuning are described here\n4. The environments used are challenging for model-free RL algorithms. That said, they may not be satisfactory for showcasing the potential of learned options. \n\n\n*References*\n1. Li, C., Song, D., & Tao, D. (2023). Hit-MDP: learning the SMDP option framework on MDPs with hidden temporal embeddings. In The Eleventh International Conference on Learning Representations.\n\nQUESTIONS:\n- How are actions sampled? Do you use the reparametrization trick  [1] for sampling action? Is the update similar to SAC?\n- How does your algorithm compare against SAC? Looking at the gradient update rules, one could argue that it is fairer to compare VMOC against SAC than PPO. My suspicion is that SAC would perform just as well as VMOC\n- Why mujoco environments for showcasing options? The true benefit of options would be seen in environments that could benefit from hierarchical policies or composition of policies. Do the authors consider environments from [Meta-World](https://meta-world.github.io/) or the dog fetch or ant  fetch environments in [DM Control Suite](https://github.com/google-deepmind/dm_control/tree/main/dm_control/suite)\n\n*References*\n1. Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. ArXiv Preprint ArXiv:1312.6114.\n2. Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018, July). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (pp. 1861-1870). PMLR.\n\nLIMITATIONS:\nMuch like Soft Actor-Critic, this work develops a novel Soft Option Critic style algorithm. I believe this line of work is very interesting and potentially impactful in the near future. However, their current draft is not well-written and hard to follow. Their experimental evaluation is also insufficient.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}