HiT-MDP: Learning the SMDP option framework on MDPs with Hidden Temporal Embeddings
Authors: Chang Li
Dongjin Song
Dacheng Tao
Venue: ICLR 2023 poster (2023)
Keywords: Hiearchical Reinforcement Learning
Reinforcement Learning
Markov Decision Process

Abstract: The standard option framework is developed on the Semi-Markov Decision Process (SMDP) which is unstable to optimize and sample inefficient. To this end, we propose the Hidden Temporal MDP (HiT-MDP) and prove that the option-induced HiT-MDP is homomorphic equivalent to the option-induced SMDP. A novel transformer-based framework is introduced to learn options' embedding vectors (rather than conventional option tuples) on HiT-MDPs. We then derive a stable and sample efficient option discovering method under the maximum-entropy policy gradient framework. Extensive experiments on challenging Mujoco environments demonstrate HiT-MDP's efficiency and effectiveness: under widely used configurations, HiT-MDP achieves competitive, if not better, performance compared to the state-of-the-art baselines on all finite horizon and transfer learning environments. Moreover, HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence, and better interpretability. Our work potentially sheds light on the theoretical ground of extending the option framework into a large-scale foundation model.

Published as a conference paper at ICLR 2023
HIT-MDP: LEARNING THE SMDP OPTION FRAME- WORK ON MDPs WITH HIDDEN TEMPORAL EMBED- DINGS
Chang Li",
Dongjin Song",
Dacheng Tao
'he University of Sydney, spacegoi ng@ com, dongjin fangjin.co wongguconn, .dd,, dacheng. taog:ggooossydddoe hewpadddeyyeuuu.uooeoo edu. au
University of Connecticut
ABSTRACT
The standard option framework is developed on the Semi-Markov Decision Pro- cess (SMDP) which is unstable to optimize and sample inefficient. To this end, we propose the Hidden Temporal MDP (HiT-MDP) and prove that the option- induced HiT-MDP is homomorphic equivalent to the novel transformer- based framework is introduced learn options' embedding vec- tors (rather than conventional option tuples) on HiT-MDDss. We then derive a stable and sample efficient option discovering method under the maximum-entropy policy gradient framework. Extensive experiments challenging Mujoco environments demonstrate HiT-MDP's efficiency and effectiveness: under widely used configura- tions, HiT-MDP achieves competitive, if not better, performance compared to the state-of-the -frt- baselines on finite horizon and transfer learning environments. Moreover, HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence, and better interpretability. Our work potentially sheds light the theoretical ground of extending the option framework into large scale foundation model.
SMDP. A
INTRODUCTION
The option framework (Sutton et al., 1999) is one of the most promising frameworks to enable RL methods to conduct lifelong learning (Mankowitz et al., 2016) and has proven benefits in speeding learning (Bacon, 2018), improving exploration (Harb et al., 2018), and facilitating transfer learning (Zhang & Whiteson, 2019). Standard option framework is developed on Semi-Markov Decision Process, we refer to this as SMDP-Option. SMDP-Option, an option is temporally abstracted action whose execution cross variable amount of time steps. A master policy is employed to compose these options and determine which option should be exccuted and stopped.
The SMDP formulation has two deficiencies that severely impair options' applicability in a broader context (Jong et al., 2008). The first deficiency is sample inefficiency. Since the execution of an option persists over multiple time steps, update of the master policy consumes various steps of samples and thus is sample inefficient (Levy & Shimkin, 201 Daniel et al., 2016; Bacon et al., 2017). The second deficiency is unstable optimizing algorithms. SMDP-based optimization algorithms notoriously sensitive to hyperparameters. Therefore, they often exhibit large variance (Wulfmeier al., 2020) and encounter convergence issues (Klissarov al., 2017).
Extensive research has tried to tackle these issues from aspects such as improving the policy iteration procedure (Sutton et al., 1999; Daniel al., 2016; Bacon al., 2017) and adding extra constraints on option discovering objectives (Khetarpal et al., 2020; Wulfmeier et al., 2020; Hyun et al., 2019). However, rare works (Levy Shimkin, 2011; Smith al., 2018; Zhang & Whiteson. 2019) explore from the perspective of improving the underlying Decision Process. Our work is largely different from literatures above and more details are discussed in Section In this work, we present counterintuitive finding: the SMDP formulated option framework has MDP equivalence which is still able to temporally extending the exccution of abstracted actions.
MDP-based options can address SMDP-based ones' deficiencies from two aspects: (1) sample efficient, i.e., MDPs policies can be optimized at every sampling step (Bacon, 2018); and (2) more stable optimize. i.e., convergence of MDPs algorithms smaller variance (Schulman et al., 2015). In this paper, we propose the Hidden Temporal MDP (HiT-MDP) and theoretically prove the equivalence to the SMDP-Option. We first formulate HiT-
well theoretically justified and have

---

Published as a conference paper at ICLR 2023
MDP as HMM-like PGM and introduce temporally dependent latent variables into the MDP to preserve temporal abstractions. By exploiting conditional independencies in PGM, we prove that the HiT-MDP is homomorphic equivalent (Ravindran, 2003) to SMDP-Option. To the best of our knowledge, this is the first work proposing an MDP equivalence of the standard option framework.
In order to solve optimal values of the HiT-MDPs, we devise Markovian Option- Option-Value Function "[s,,0;_i and prove that it is an unbiased estimation of the standard value function V(se]. We further develop the Hidden Temporal Bellman Equation in order derive the policy evaluation theorem for HiT-MDPs. We also show that the Markovian Option-Vaile Value Function has a variance reduction effect. As result, HiT-MDP is general-purpose MDP that can be updated at every sampling step, and thus naturally address the sample inefficiency issue.
We solve the learning problem by deriving stable on-policy policy gradient method under the maximum entropy reinforcement learning framework. One difficulty of learning standard option frameworks is that they do not have any constraint on qualities of options (Harb et al., 2018). Standard option frameworks have tendencies to learn either degenerate options (Harb et al., 2018) 8) (short execution time) that switching back-and-forth frequently, or dominant options (Zhang & Whiteson, 2019) (long execution time) that executing through the whole episode. We tackle this problem by proposing the Maximum entropy Options Policy Gradient (MOPG) algorithm. MOPG includes cottommaiionthhortttti intrinsic reward to encourage consecutive executions of options and entropy terms to encourage explorations of options. The whole algorithm can be solved in an end-to-end manner under the structional variational inference framework. We theoretically prove that optimizing through MOPG converges to the optimal trajectory.
We conduct experiments on challenging Mujoco (Todorov al., 2012; Brockman et al., 201 Tunyasuvunakool et al., 2020) environments. Thorough empirical results demonstrate that under widely used configurations, HiT-MDP achieves competitive, if not better, performance compared to the state-of-the the-art baselines on all finite horizon and transfer learning environments. Moreover, HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence, and interpretability.
2 BACKGROUND
Markov Decision Process: A Markov Decision Process (Puterman, 1994) M A,,,,,,,,T)) consists of state space S, an action space A. a state transition function P(s [., az) : 6 discount factor y E R, and reward function r(s, a) B[r|s, a] S x A + R which is the expectation of the reward r E R received from the environment after executing action a, at state St. A policy = P(als) A x S [0, 1] is probability distribution defined over actions conditioning on states. A discounted return defined as G. = Σe 7" where 7 E (0, 1) is a discounting factor. The value function V(s.] = Ey>- [G]]s]] is the expected return starting at state st and the trajectory т = {s,, a,, T1+1, S1+1,........... follows policy ग thereafter. The action-valus function is defined as Q/s;, a,] E Gr a,].
Homomorphic Equivalence: Givan et al. (2003) define the equivalence relation between MDPs as symmetric equivalence (bisimulation relation). Ravindran (2003) extends their work to homomorphic equivalence that allows defining symmetries between an MDP and SMDP. Given two processes, an MDP M- = {S, A, R, P.7} with the trajectory T and an SMDP M = (8, A. R, P,A} with the trajectory F. Assume both M and M share the same action space A. An homomorphism B is a tuple of surjection partition functions, M and M is Homomorphic Equivalence if 1) for all state- action pairs (s, a). there exists a many-to-one correspondence equivalent state-actis action pair (s, à} that {s, as, B (š, a)/BB B, or denoted as B( a)) = (s. a). 2) and following conditions hold: P(T/B) = P(7/B), and B is surjection, r(7/B)=r(f/1) rF./B),
1.
2.
The SMDP-based Option Framework: In SMDP-Option (Sutton et al., 1999; Bacon, 2018), an option is triple (Lo, To, 3.) E O, where O denotes the option set; the subscript o E 0 {1,2.... K} is a positive integer index which denotes the o-th triple where K is the number of options: I, is initiation set indicating where the option can be initiated; To (ala)) & S -
[0. is the action policy of the oth option; 8, = Pb = 1|s) : :S + [0, 1] where b E {0, 1) is a termination function. For clarity, we Po Pb (b 1 (s) instead of 8, which is widely used in previous
2

---

Published as a conference paper at ICLR 2023
option literatures (e.g., Sutton et al. (1999); Bacon et al. (2017)). A master policy r(ols) P(o)s) where o E O is used to sample which option will be executed. Therefore, the dynamics (stochastic process) of the option framework is written
P(r) P(so) P Poo (an (o) II
8 1 (agls:)
Po., (b. 1o. Po._, (b, P(o,ss())
(I)
where {su, Ou, ag, $1, 01, a denotes the trajectory of the option framework. 1 is an indicator function and is only true when o, o--1 (notice that o,-1 is the realization o,-1). Therefore, under this formulation the option framework is defined as a Semi-Markov process since the dependency on an activated option o can cross a variable amount of time (Sutton et al., 1999).
AN MDP EQUIVALENCE OF THE SMDP-BASED OPTION FRAMEWORK
In this section, we propose the Hidden Temporal MDP (HiT-MDP). We first reformulate the SMDP- Option HMM-like Probabilistic Graphical Model (PGM). We then propose the HiT-MDP marginalization of the PGM and prove that the HiT-MDP is homomorphic equivalent (Ravindran, 2003) to the SMDP-Option. We also derive the HiT-MDP's Bellman Equation and prove its conver- gence by presenting the policy evaluation theorem. As result, HiT-MDP is general-purpose MDP that can be combined with any policy optimization algorithm off- the-shelf. We propose an efficient learning algorithm and complete the proof of the policy iteration theorem in Section 4.
3.1 AN MDP FORMULATION OF THE OPTION FRAMEWORK
Following Bishop (2006)'s formulation of mixture distributions, we redefine the option random variable o E {1,2, K), which was originally defined as an integer index, but now as K K-dimensional one-hot vector E O = {0, 1} where K is the number of options. We further employ the one-hot vector to reformulate the termination function and action function each option into two mixture distributions by introducing extra dependencies on O:
P(ay|S; 6,) II Po (a.)s)))
O(b_)s),0(-1)+ II Po(by(s.)
(2)
oEO.-1
Since the option random variable ö is one-hot vector, instantiation & k denotes the activation of the option kt and by definition only the k-th entry of og is 1 and all the other entries are 0. Therefore, have Po, (a, (s.) P(a,/s, O, ō,) and 3.._, = Pa.-, (b. 1|s;) P(b, 1|s,, 0;-1 1))
The third reformulation is that propose novel MDP mixture master policy P(o, |st b,, 6.-1). which a mixture distribution containing the SMDP master policy P(or(S)) and a degenerate probability as mixture components by adding two extra dependencies on b, and 0,-1
P(o St b,, 0;-1) P(or(s)) P(o,)
(3)
where the indicator function 10,=0-, used in Eq. is now redefined as a degenerate probability distribution (Puterman, 1994):
if o; 0:-1,
p(o. 1)
if # 0.-1.
and the joint distribution can be written as:
P(7)
so, on)
P (Is: 11
1=1
Σ (( br Or.
(4)
Although the mixture master policy in Eq. MDP-formulated, as a mixture com- ponent within it, the master policy P(o, ss)) is still SMDP-formulated and hence cannot be updated by MDP-based algorithms. nation variable b, in Eq. 4: E., P(b, P(o,) 0,-1), we propose the Markovian master policy P(or|s,, to model this marginal distribution explicitly:
By marginalizing over the termi-
3

---

Published as a conference paper at ICLR 2023
OC
P(f) ((())) 8o, oo)
II (()
P(ar|st, Se, P(o
(5)
where Eq. 5 denotes the joint distribution of the PGM. In this formulation, P(F) actually an HMM with 8t, a, as observable random variables and o as latent variables. We use 7 (S:, 0r- = [s, 0r-1) to denote the master policy and T ( Oz) P(a,/s,, Ôr) to denote the action policy. Figure shows the PGM (Eq. 5).
Figure 1: Probabilistic Graphical Model
(PGM) of HiT-MDP.
3.2 THE HIDDEN TEMPORAL MDPS (HIT-MDPS)
Given the PGM (Eq. 5, Figure 1), the Hidden Temporal MDPs (HiT-MDPs) family can be described by tuple M rr, P. where 8 S O augmented state space, A A x x is an augmented action space, and Φ = P(o, (s) = P(o, 0;-1) the emit function for hidden variables. The joint distribution of HiT-MDPs is factorized Eq. (derivations see Appendix C).
We define partition function B(, = (( at, 8,), (0,-1: ar, 0,, 8:) (proofs in Ap- pendix C), which maps F to T, where f (So, 0o an, $1, 01, a1,.... is the trajectory of the HiT-MDP. Therefore, by following the Partition Function B, the dynamics of the SMDP-Option Eq. under the Surjection B is equivalent to HiT-MDP P(T/B) prove the equivalence between the SMDP-Option and HiT-MDP, share the same expected reward. This is non-trivee non-trivial since compared to the SMDP-Option the MDP formulation introduces extra dependencies on 6. However, in Appendix C, by exploiting conditional independencies we prove that they share the same expected return under the Surjection B. Therefore, SMDP-based option framework has an MDP-based equivalence:
P(F/B). With P(T/B) P(7 (B) in hand,
move on to prove both of them
Theorem 3.1. By the definition of Bisimulation Relation, the SMDP- based option framework, which employs Markovian oplions, has an underlying MDP equivalence because:
1.
P(T/B) P(7/ B) and B is a Surjection.
2.
((r/B)=((/BB
Proof. See Appendix C.
Since the master policy P(o. S., 6;-1) introduces one extra dependency on 6:-1, conventional Bellman equation which is derived by following the conventional value function V(s.] longer applies to HiT-MDP (see Appendix D). In order to derive the Bellman equation of HiT-MDP, we propose the novel Markovian option-value function, value functions with Markov dependencies (such as 1) Specifically, rather than use the conventional value function V [s). let the Markovian option-value function V(st,
0r-1] and the option-acee -action value function Q l (s; Oe, a.] be defined by:
Vss, 6.- E[G, 6--1]= = Σ P(or|se, 01-1) Q (s o,, a.]].
(6)
QAIse, or, a.] = EEG oz, ai] r(s, a) + E, I Σ Yrral I=1
As with the standard Q-function and value function, we can relate the Q-function the Markovian option-value function at future state via a Hidden Temporal Bellman Operator TH.. Theorem 3.2. The option-action value function Eq. 7 satisfies the Bellman Operator T
TH O2, ai} EEG Ōc: an]
= = + Σ P(s a (,)4, B1+1
(8)
where the Markovian option-value function given by Eq. 6. Proof. See Appendix D. We can obtain the option-action value function for any policy by repeatedly applying 7 and the sequence converges to the optimal value function:
4

---

Published as a conference paper at ICLR 2023
Theorem 3.3. (Markovian Option Policy Evaluation Theorem). Assume that throughout our compu- lation the -1 and V are bounded and A. < cG, the sequence Q% defined by Q = T Q4 will converge to the option-action value function Q as k + DC. Proof. As with the standard convergence results for policy evaluation (Suttinn & Barto, 2018), by the definition of T (Eq. 8) the
value function Q is fixed point. In Appendix D we show that T is a contraction, and then Theorem 3.3 follows immediately.
D In Appendix D. we further prove that [is,, 0,-1] is an unbiased estimation of V[sa] and has variance-reduuciio effect compared to the conventional value function. This property is empirically witnessed and further discussed in experiments (Section 5.1).
1]
Proposition 3.4. V (s,, 0.-1] is an unbiased estimation of [(s... Proposition 3.5. The variance of V [Se, 6;-1] is up-bounded by V V[sr].
Proof. See both Proposition 3.4 and 3.5's proof in Appendix D
The theoretical analysis above presents a counterintuitive fact: the SMDP formulated option frame- work has an MDP formulated homomorphic equivalence (HiT-MDPs), and they both converge to the same optimal value function. One natural question to ask is that how could an MDP temporally extend exccutions of options? Note that the Markovian master policy P(o, (s 0;-1) is result of the marginalization over the termination variable b, (See Eq. 5 and Eq. 1) and has an extra dependency on 0,-1). Therefore, the Markovian master policy acts like distance measure. The decision of selecting an option og can be made by simply comparing which ō E ö is closest to the vector (St, 0t-1]. Because a vector is closest itself, this mechanism has a natural tendency to assign O, Or-1+ and thus extends o 1 is execution. On the other hand, a significantly different state S, will pull the distance far enough from 0:-1 and result in other options being assigned. The effectiveness of the Markovian master policy is also empirically addressed in Section 5.4.
4 LEARNING HIT-MDPS UNDER THE MAXIMUM ENTROPY FRAMEWORK
In this section, we propose stable and sample efficient Maximum entropy Option Policy Gradient (MOPG) algo- rithm to learn HiT-MDPs. Learning temporal abstraction like options has been long standing challenge (Sutton et al., 1999; Givan et al., 2003; Kolobov et al., 2012; Ba- con et al., 2017). In theory (Sutton et al., 1999), options not generally necessary for learning optimal policies since MDPs are sufficient enough. Therefore, standard option learning algorithms optimizing Eq. often leads to sub-optimal results: either degenerate options (Harb et al., 2018) (short execution time) switching back-and-forth fre-
quently, or dominant options (Zhang & Whiteson, 2019) (long execution time) executing through the whole episode. Although there are various attempts (Harb et al., 2018; Hyun et al., 2019; Smith et al., 2018) tackling this issue, yet they often introduce bias which accumulates along the length of the trajectory and result in sub-optimal solutions.
Figure 2: PGM of Eq. 5.
Recently maximum entropy reinforcement learning framework (Todorov, 2006; Ziebart et al., 2010; Haarnoja et al., 2017: 2018) has been witnessed success in encouraging exploration and improving sample complexity. With the PGM of the HiT-MDP in hand, we tackle the above issues by deriving the Maximum entropy Option Policy Gradient (MOPG), a stable algorithm for learning diversified options and preventing degeneracy. Following the control as inference framework (Levine, 2018; Haarnoja et al., 2018), we introduce the concept of "Optimality" (Todorov, 2006) into the HiT-MDP and formulate the conventional RL problems Eq. 5 as a probabilistic inference problem (Kappen et al., 2012). Specifically, we follow Levine (2018); Koller & Friedman (2009) and factorize Eq. into another PGM (as shown in Figure 2):
efr, ( = ((5)) II
q(e s.. a. (.)g(e ss, a, o, 1) |s. a (g)
x ((s
II P P(s ad)
II q( s, an) ar or, 0,-1),
(9)
Original Dynamics
Variational Terms
5

---

Published as a conference paper at ICLR 2023
where {0, }} are observable binary "optimal random variables" (Levine, 2018). The agent is optimal at time step when q( = 1 S1, az) and q(e = 1 s a, o,, 0t-1) (to keep notations uncluttered we use e, to denote e, 1. To simplify the derivation, priors (()) and g(a) can be assumed to he uniform distributions without loss of generality (Levine, 2018). Note that Eq. 9 shares the same state-action dynamics with Eq. 5. With the optimal random variables el and eA, the conditional probability of a state-action {St, } pair that is optimal is defined as:
q(e [s, at) = exp(r(s,, exp(r(s,,a))) 2f exp(r(s;,
ac))dal=*
(10)
which is an energy function that follows the Boltzmann distribution (Levine, 2018). This specific design facilitates recovering the value function at the latter structural variational inference stage. Based on the same motivation, the conditional probability of an option-state- {o,, St, a,, 0 pair that is optimal is defined as,
q(e e Bt, ay, o,, ô.-1) exp([[o,)s)) ss, a, ox-l)) / exp(I (s, a,, or-1])do]
(11)
where the -nntuuliiffom [[o st, a,. 01-1] is chosen be the energy function follows the Boltzmann distribution. This design choice arises from fact that when the uniform prior assumption of q(o) relaxed the optimization introduces a mutual-information as a regularizer in the Evi- dence Lower BOund (ELBO) (Proof in Appendix E). When optimal random variables are observed, substituting Eq. 10 & 11 into Eq. 9, the conditional probability of any feasible trajectory F is:
(fFlerr) P(so) II P(s exp( Σ r(s a()) exp Σ t=1
16, Si, a,: O.-))))) (12)
where optimal random variables 61:T are treated as observed random variables (evidences) and the trajectory 7 is treated as latent variables. Therefore, the problem of fitting the optimal trajectory (Proof in Appendix E). This observation immediately gives rise to a structural variational inference solution as keeping the system dynamix P(s)) II P(s 8,, ar) in both Eq. 12 and Eq. fixed while optimizing the action and master policies in the variational distribution of Eq. 5:
is equivalent to maximizing ELBO: log ((E1:T) 2 -KKL (PF [P(
Theorem 4.1. (Markovian Option Policy Improvement Theorem). The problem of learning optimal action and master policies can be simplified as shrinking the KL- KL-Divergence: Dxu (xu(P(F)ll((Flezx)
arg max- - -Drr -Drr, (rn(P(F)ll(FFeerr)) xO,x
π π arg max Σ Ex a (r(s,, ac) + [oo:|s;, ar, 01- H(* + H(m))],
(13)
where H(-) denotes the entropy term. Proof. See Appendix E.
With policy evaluation and improvement theorems, we further prove the convergence of MOPG by proposing the Markovian Option Policy Iteration Theorem (Appendix E). Standard option frameworks (as shown in Eq. 1 and Eq. 5) only consider maximizing the value function and have constraints on qualities of options, e.g., what good options should behave (Harb et al., 2018). As a result, previous researches often lead to suboptimal results. In Eq. 44, the mutual-information is introduced as an intrinsic reward to encourage consecutive executions of options and thus prevent degenerate options, while entropy terms encourage explorations of options thus prevent dominant options.
Following value functions from Section 3.2, we directly optimize ELBO with respect to the variational distribution and derive the Maximum entropy Options Policy Gradients (MOPG) theorems:
Theorem 4.2. Master Policy Gradient Theorem: Given a stochastic master policy which is differentiable w.r.t its parameter vector og. the gradient of the expected discounted return w.E.t 0g is:
OV (Vis, 0 = B(
aP(o'(s',o) Qols', 6'] + T(o''s, a) + H(= st, Or-1]: 00,
(14)
00g
where Qols: or] E[G, ō.] (Appendix D Eq. 28). Proof. See Appendix E
Theorem 4.3. Action Policy Gradient Theorem: Given a stochastic action policy which is
differentiable w.r.t. its parameter vector 0a. the gradient of the expected discounted return w.r.t. 0, is: aQo 8,, 0z]
aP o)
TE.
08, Qa(s, + H(TT))
oz).
(15)
H4,
Proof. See Appendix E. To keep notations uncluttered, we use 0 to denote master policy's parameters A -O and 0, to denote action policy's 's parameters TØ_ The algorithm is summarized in Appendix F. TH

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
In this section, we design experiments to answer five questions: (Q1) Whether MOPG can achieve better performance than other option variants and non-option baselines? (Q2) Does MOPG have performance boost over other option variants in transfer learning settings? (Q3) Is HiT-MDP inter- pretable? (Q4) Whether MOPG can encourage consecutive executions of options while preventing dominant options problem?
For baselines, we follow DAC (Zhang & Whiteson, 2019)'s open source implementations and compare our algorithm with six baselines, five of which are option variants, i.e., DAC+PPO, AHP+PPO (Levy & Shimkin, 201 1), IOPG (Smith al., 2018), PPOC (Klissarov et al., 2017), and OC (Bacon et al., 2017). The non-option baseline is PPO (Schulman et al., 2017). All baselines' parameters used by DAC remain unchanged other than the maximum number of training steps: MOPG only needs 1 million steps to converge rather than the 2 million used in DAC. For single task learning, experiments are conducted on all OpenAI Gym MuJoCo environments (10 environments) (Brockman et al., 201 2016)). For transfer learning, we follow DAC and run 6 pairs of transfer learning tasks based on DeepMind Control Suite (Tassa et al., 2020). Figures are plotted following DAC's protocol: curves are averaged over 10 independent runs and smoothed by a sliding window of size 20. Shaded regions indicate standard deviations. All experiments are run on an Intel@ CMre i9-9900X CPU @ 3,50GHz with single thread and process. Our implementation details are summarized in Appendix G. For fair comparison, we follow DAC and use four options all implementations. Our code available in supplemental materials.
5.1 SINGLE- TASK LEARNING (Q1)
To answer Q1, we compare MOPG against five different option variants (i.e, DAC+PPO (Zhang Whiteson, 2019), AHP+PPO (Levy & Shimkin, 2011), IOPG (Smith al., 2018), PPOC (Klissarov et al., 2017) and OC (Bacon et al., 2017)) and PPO (Schulman et al., 2017). We observe that MOPG exhibits two different behaviors on infinite (Figure 3a) and finite (Figure 3b) horizon environments. Previous literatures (Klissarov et al., 2017; Smith et al., 2018; Harb et al., 2018; Zhang & Whiteson, 2019) find that option-based algorithms do not have advantages over hierarchy-fiee frree algorithms single-task environments. Despite this, MOPG can still achieve comparable performance with state of the arts (Figure 3b). More importantly, on infinite horizon environments (Figure 3a), MOPG's performance significantly outperforms all baselines with respect to episodic return, convergence speed, variance between steps, and variance between 10 runs (Proposition 3.5). Since infinite and finite horizon environments are theoretically identical (Sutton & Barto, 2018), we do not have theoretical explanation for this phenomenon. In Appendix A, we conceptually explain that this might be because conventional value functions are insufficient to approximate environments in which hidden variables only affect rewards but not states. In conclusion, our experiment results show that MOPG is at least as effective as other option variants, but is significantly more sample efficient. Furthermore, it has significant advantages in infinite horizon environments.
5.2 TRANSFER LEARNING (Q2)
We run 6 pairs of transfer learning tasks based on DeepMind Control Suite (Tassa et al., 2020). Each pair contains two different tasks. Following the setting in DAC (Zhang & Whiteson, 2019), we train all models one million steps the first task and then switch to the second task to further train another one million steps. Results are reported in Figure 4. On the transfer learning task, MOPG's
HalfCheetah-v2
HumanoidStandup-22
Walker2d-12
Hopper-v2
2000
Episode Return
Episode Return
PYO
8000
JePP PPO
IOPG
KinG
G0000
1000
1000
x 10f
1x 10
1x 10f
1 10s
Steps
Steps
Steps
Steps
(a) Infinite horizon environments
(b) Finite horizon environments
Figure 3: Single-task episodic returns in 4 different environments (i.e., HalfCheetah-v2, HumanoidStandup-22 W2 and Hopper-v2). Results in all 10 environments are available in Appendix B.

---

Published as a conference paper at ICLR 2023
CartPole
Chestah
903
300
Episode Return
MDNG
000
500
900
230
203
Steps
10%
Steps
10
0
Steps
10
Figure 4: Transfer learning results on three pairs of tasks. All 6 pairs of results are available in Appendix. B.1
performance ranks the first in 5 out of environments. This demonstrate MOPG's advantages in knowledge reuse and its performance is at least comparable with other option variants.
5.3 INTERPRETATION OPTION EMBEDDINGS (Q3)
Interpretability a key property to apply RL agents in real-world applications. Our implementation
follows the Skill-Action (SA) architecture (Li al., 2020) and introduces transformer's decoder distance measure to learn option embeddings.
One unique advantage of HiT-MDP is that hidden variables are interpretable. In the implementation, the master policy is defined as π 2 P(o, St, 0z-1; Wo) where W. = - Óк] is an embedding matrix, E R is the embedding vector for an option with dimension N, and K is the total number of options (N = 40 and K 4 in figure 5(a)). More implementation details are described in Appendix G. Similar word embeddings (Vaswani al., 2017). option embeddings learn semantic space of options with each dimension encodes a particular property and can be interpreted explicitly. As in the capsule network (Sabour al., 2017), we first infer each embedding dimension's semantic by adding perturbations on to it perturb
[r.j '[g + [ where e, E [-0.1, -0.09........ 0.09] is an array ranging from -0.1 to 0.09 with an interval 0.01 and j E N is the jth dimension of the option embedding. We then sample primary actions [[.j] ~ π " o from perturbed options to observe how actions change along with perturbations.
For instance, Figure 5(b) visualizes how perturbations on dimension of the option embedding affect the primary action. The Y-axis denotes actions defined by the HalfCheetah environment. The magnitude of Dim 0 have an opposite effect on the back leg and front leg: a larger value will increase the back leg's ss torque while decrease the front leg's and vice versa. This means Dim 0 has "focus point" property: focuses torque on only one leg. Other dimensions in Figure 5(c) can interpreted in the same manner. Once each dimension is understood, embeddings become straightforward to interpret by simply inspecting on which dimensions each embedding ó (Figure 5(a)) has significant weight. Due space limitations, more details and GIFs are provided in Appendix B.3.
Coden
Grlbet
Gulent
300
Option Embeedinn
Esocation
Mwd
Mikh
18] Parturaations Semandond
10 Perturbatioon ol 15,
Figure
Interpretation option embeddings learned HallCheetahi
Figure Options Composing Patterns
5.4 TEMPORAL EXTENSION (Q4)
One significant advantage of MOPG over other algorithms is that the intrinsic reward and entropy terms (Eq. 44) enable MOPG automatically learns to balance between degenerate and dominant options. In this subsection, we empirically demonstrate this in Figure 6 (experiment results from HalfCheetah, more details in Appendix B.2). At the start of training, all options' durations are short, while Option 3's duration quickly grows. This proves that MOPG can indeed temporally
8

---

Published as a conference paper at ICLR 2023
extend option. More than temporal extension, shown in the video', MOPG learns to compose distinguishable options. Option 3 (green background in the video) is running forward skill thus it is executed most of the time. Option 2 (blue background) is mainly used to recover from falling down thus its duration decreases with training. Therefore, the empirical study also shows that MOPG can compose disentangled options while preventing the dominant option problem.
6 RELATED WORKS
To discover options automatically, Sutton et al. (1999) proposed Intra-option Q-learning to update the master Q value function at every time step. However, all policies under this formulation are approximated implicitly using the Q-learning method. AHP (Levy & Shimkin, 201 1) is proposed to unify the Semi-Markov process into an augmented Markov process and explicitly learn an "overall policy" by applying MDP-based policy gradient algorithmss However, their method for updating the master policy is still SMDP-style thus sample inefficient (Zhang & Whiteson, 2019). OC (Bacon et al., 2017) proposes policy gradient based framework for explicitly learning intra-option policies and termination functions in an intra-option manner. However, for the master policy's ' policy gradients learning, OC remains SMDP-style. One closely related work is DAC (Zhang & Whiteson, 2019) which reformulates the option framework into two augmented MDPs. Under this formulation, all policies can be modeled explicitly and learned in MDP-style. However, DAC is not theoretically proven to be homomorphic equivalent to the SMDP-Option. Moreover, the reward function r(r) used in Lemma 1 & 2 (Zhang & Whiteson, 2019) is undefined and makes the equivalence in proposition 1 (Zhang& & Whiteson, 2019) does not hold. To the best of our knowledge, this the first work proving MDPs' equivalence to option-induced SMDPs under the homomorphic equivalence framework.
Regarding network architectures, Attention Option Critic (AOC) (Chunduru & Precup, 2020) first introduces the attention mechanism into the option framework unique attention mask over state s = W. @s and enables learning interpretable options with unique activation field (e.g.. initiation set) over the observation space. Unlike AOC, Section 5.3 follows the Acll-nctiion architecture (Li et al., 2020) which employs the attention mechanism over the joint space of the observation and option space. Regarding optimizing algorithms, Zhang & Whiteson (2019) pointed out that a large margin of the performance boost of DAC comes from the Proximal Policy Optimization (Schulman et al., 2017) (PPO). However, recent advances in probability inference based RL methods have also shown great success. Haarnoja et al. (2018) combined the idea of soft optimality (Haarnoja et al., 2017) from earlier control as inference frameworks (Todorov, 2006; Ziebart et al., 2010; Kappen et al., 2012) with the actor-critic architecture (Konda & Tsitsiklis, 2000) and proposed the state-of-the-arn off-policy baseline SAC. Recent works show that the option framework trained under soft-optimality off off-policy algorithms outperform on- on-policy methods (Wulfmeier et al., 2020). HiT-MDPs are general MDPs which can be trained by both on-policy and off-policy algorithms: the ELBO proposed in Section 4 can easily be extended to a SAC-like algorithm and remains to the future work.
CONCLUSIONS
In this paper, we propose the Hidden Temporal MDP (HiT-MDP), to the best of our knowledge, the first work proving option-induced MDP' is homomorphic equivalent to the standard SMDP Option framework. As for the learning algorithm, we propose the Maximum entropy Option Policy Gradients (MOPG) algorithm. Unlike conventional option learning algorithms that only maximize the value function without any constraint on the qualities of options, MOPG optimizes the Evidence Lower BOund (ELBO) with an intrinsic reward prevent degenerate options and entropy terms to prevent dominant options. It is also theoretically proven fitting optimal trajectories without adding any bias.
On challenging Mujoco (Todorov et al., 2012; Brockman et al., 2016b: Tunyasuvunakool et al., 2020) environments, thorough empirical results demonstrate that under widely used configurations, HiT- MDP achieves comparable results to all baselines on finite horizon and transfer learning environments, and significantly outperforms all baselines on infinite horizon environments. We also demonstrate hidden temporal embeddings are interpretable, which is a key property for applying RL agents to real-world applications. Embeddings exist in various applied ML domains such as CV and NLP, our work shows that the idea also works for HRL and potentially lays the theoretical ground for building a large scale HRL foundation model.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
CL offers his sincerest gratitude to Shangtong Zhang, Pierre-luc Bacon and Doina Precup for their time and patience answering his questions. We deeply appreciate Shangtong Zhang and Jiayi Weng for their exceptional open source projects DeepRL https: : //ggttgitt //gt//hht hub.. com/ Shangt Shaangtonghhaeeeeee ongžhang/ DeepRL (the original implementation in supplementary material and all experiment results this paper are based on DeepRL) and tianshou https: : ////ttgtbtttt com/ thu-ml/tiansee (the latest github repo based on tianshou).
REFERENCES
Pierre-Luc Bacon. Temporal Representation Learning. PhD thesis, McGill University Libraries, 2018.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-Firsi AAAI Conference on Artificial Intelligence, 2017.
Christopher M Bishop. Puttern recognition and machine learning. springer, 2006.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 0004000000110....0 2016a.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016b.
Raviteja Chunduru and Doina Precup. Attention option-critic. arXiv preprint arXiv:2201.02228 2020.
Christian Daniel, Herke Van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference for determining options reinforcement learning. Machine Learning, 104(2-3)337-357,, 2016.
G David Forney. The viterbi algorithm. Proceedings of the IEEE, 61(3):268-278, 1973.
Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial Intelligence, 147(1-2):163--22) 2003.
Tuomas Haarnoja, Haoran Tang, Picter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor, arXiv preprint 1rXiv:1801 .01290, 2018.
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not op- tion: Learning options with a deliberation cost. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Matthew Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303-1347, 2013.
Minsung Hyun, Junyoung Choi, and Nojun Kwak. Disentangling options with hellinger distance regularizer. arXiv preprint arXiv: (0:::1904 1904006884. 2019.
Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction reinforcement learning, In AAMAS (1), pp. 299-306. Citeseer, 2008
Hilbert J Kappen, Vicenç Gómez, and Manfred Opper. Optimal control a graphical model inference problem. Machine learning, ((2)):59-182. 2012.
Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-BB Chevalier-Boisvert, Boisvert, Pierre-Lue Bacon, and Doina Precup. Options of interest: Temporal abstraction with interest functions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 4,451, 2020.
10

---

Published as a conference paper at ICLR 2023
Martin Klissarov, Pierr-Luuc Bacon, Jean Harb, and Doina Precup. Learnings options end-to-end for continuous action tasks. arXiv preprint arXiv:1712.0000 2017.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Andrey Kolobov, Daniel S Weld, et al. Discovering hidden structure in factored mdps. Artificial Intelligence, 189:19-47, 2012.
Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pp. 1018-004, 2000.
Sergey Levine Reinforcement learning and control as probabilistic inference: Tutorial and review.
arXiv preprint arXiv: wee../005000000 2018.
Kfir Y Levy and Nahum Shimkin. Unified inter and intra options learning using policy gradient methods. In European Workshop Reinforcement Learning, pp. 153-164. Springer, 2011.
Chang Li, Dongjin Song, and Dacheng Tao. The skill-action architecture: Learning abstract action
embeddings for reinforcement learning. 2020.
Daniel Mankowitz, Timothy Mann, and Shie Mannor. Adaptive skills, adaptive partitions (asap).
arXiv preprint arXiv:1602.0333: 2016.
Martin I. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
Balaraman Ravindran. Smdp homomorphisms: algebraic approach to abstraction in semi markov
decision processes. 2003.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems, pp. 3,856-3,866, 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization, In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Matthew Smith, Herke Hoof, and Joelle Pineau. An inference-based policy gradient method for learning options. In International Conference on Machine Learning, pp. 17700344,72. 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):18122 1999.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dmcontrol: Software and tasks for continuous control, 2020.
Emanuel Todorov. Linearly-solvee solvable markov decision problems. Advances in neural information processing systems, 19, 2006.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020. ISSN 2665-9638. doi: https://doi. 0tg/:0...166//smmaa2200.... 100220 URL http:::::/www..eeeeeeeeeeeee.eee wttpee/tt.cooccccccc.e..eeee science/ articleesst///2865538200000
com/
11

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5,998 6,08,, 2017.
Markus Wulfmeierr Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim Hertweck, Michael Neunert, Dhruva Tirumala, Noah Siegel, Nicolas Heess, al. Data-efficient hindsight off- oof-po option learning. arXiv preprint arXiv:2007.15588 2020.
Shangtong Zhang and Shimon Whiteson. DAC:
The double actor-critic architecture for learning options. In Advances in Neural Information Processing Systems, pp. 2,012-2,022, 2019.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In ICMI,,,,000
A LEARNING OPTIONS AT MULTI-LEVELS OF GRANULARITY
Implementations of the option framework share some common limitations. When proposing the option framework, Sutton et al. (1999) expected that learning at multi-level of temporal abstraction should be in favor of faster convergence and better exploration. On the contrary, significant improvements on single task environments have not been witnessed in most option implementations (Klissarov et al., 2017; Smith et al., 2018; Harb et al., 2018; Zhang & Whiteson, 2019). To the best of our knowledge, MOPG is the first option implementation in which these properties are significantly witnessed but only on infinite horizon environments, In this section, we study this problem by explaning an intuition of why the value function is the main reason for this deficiency and how deep wide value functions could solve this problem.
The expectation of improvements of the option framework on single task environment builds on an assumption that, by exploiting hierarchical action and state space, an agent's searching space can be greatly reduced thus accelerates learning and improving exploration. However, reported in section 5.4, most option frameworks suffer from "the dominant option problem" (Zhang & Whiteson, 2019) which prevents option frameworks from effectively learning hierarchy in action and state space as well as coordinating between options.
One intuition behind this problem is that conventional value functions V[S.] and Q|S,, Or, Ar] make values depend on temporal latent variables indistinguishable (i.c. Although different options 0 and 02 results to different values, such V[S,, O.-1 01] 10 and V[Se. 0₂-1 02 = - 10. Because they arrive at the same state St, they have identical values under conventional value function V[S.] 0). This deficiency makes option frameworks can only learn options at very coarse level thus fail to exploit hicrarchical information. The solution might be using deep wide value function: enabling the framework learn fine-grained options at mutli-levels of granularity (deep) and making value functions depend on latent variables with longer (wide) dependencies (e.g. V[S,, O,-1] and Q[S,, O2, A,, Oz-1)).
To have better understanding the importance of the deep wide value function, let us consider a simple environment which can be easily solved by Q[st. ae, ax-1] but not Q[ ar].
Suppose we are training a robot which only has a camera sensor to cook thanksgiving turkey. In this setting there are only two states: S = (Raw Turkey Image, Cooked Turkey Image]. The robot's action space only consists of two actions: A = (Stuff turkey, Roast turkey]. As for reward, if the robot roasted stuffed turkey, then the reward is 10. However, if the robot roasted an un-stuffed turkey, then the reward is 100. The stuff turkey action receives 0 reward.
The difficulty in this environment is, since the robot only has a camera to capture an image of the turkey, it can only observes either (Raw Turkey Image] or (Cooked Turkey Image). There is no way to look inside the turkey and see if the turkey is stuffed. Under this setting, a robot can never leam first stuff turkey and then roast it because Q[Raw Turkey Image, Stuff Turkey Q[Raw Turkey Image, Roast Turkey] = O. Therefore, the robot can only randomly cook a turkey. However, this problem can be easily solved by using deep wide value function QiS,, A,, 1.).
The core problem in this setting is, action has no effect on states, it only affects rewards. At the first glance this is Partially Observed MDP (POMDP) problem since the state of whether the turkey is
12

---

Published as a conference paper at ICLR 2023
stuffed is un-observed. This is true in all reinforcement learning settings without dependencies latent variables. However, it goes much deeper in HRL settings.
In HRL, common formulation is to estimate latent variable O encode hierarchical information and makes the policy depends on P(A, St, O₂). Since is latent variable, is highly likely that state St, different latent variable P(A,|S,, O 0z) and P(A, O₂ oy) emits the same action A, = A₁, and thus makes the conventional value function indistnnguishable between o, and Oy:
This phenomenon is especially common around the switching time step of two options: around switching point, states usually compatible with both old and new options. Conventional value functions will be especially confused at those moments. This is exactly what we observed in Fig- 5.3: overall, option is executed consistently. However, there are some random switches option 2. And the randomization is increased between around switching time steps. To explicitly show this, we visualized "Run4" into video: https: www. yout ube. com wwwrtsstors///ee//ooo /MO GBPqit?
sesture=share The option selection is very random at the beginning of the episode as well as around the switching point (the 16th second). These are exactly
most confusing moments of conventional value functions. Thanks to proposition 3.4 and 3.5, one unbiased solution might be employing the deep wide value function: the higher the order of the MDPs, the smaller the variance will be. We will explore this direction in future work.
B EXPERIMENTS RESULTS
B. PERFORMANCE
In this section we provide results for all ten OpenAl Gym Mujoco Environments (Brockman et al., 2016b; Todorov et al., 2012). Those environments can be classified into two categories: infinite horizon environments (i.e., HalfCheetah, Swimmer, HumanoidStandup and Reacher) and finite horizon environments (the other).
Table 1: Performance of Infinite Horizon Environments
HalfCheetah
Swimmer
HumanoidStandup
Reacher
PPO
2143.6
59.9
62262.2
-7.5
DAC+PPO
1830.1
85.0
38954.9
-8.1
AHP+PPO
1701.7
86.7
38684.9
-7.3
PPOC
1441.2
43.6
39841.7
-9.4
OC
832.3
33.0
52352.7
-15.3
MOPG
3446.7
107.8
91654.5
-4.6
Table 2: Performance of Finite Horizon Environments
Walker2d
Hopper
InvertedPendulum
InvertedDDotheeenndulm
Ant
Humanoid
PPO
1512.5
1489.9
939.9
7112.6
1049.6
562.1
DAC+PPO
1968.0
1702.2
943.7
5804.5
985.8
487.6
AHP4PPO
1520.6
1993.6
940.0
7120.7
1359.3
569.3
PPOC
756.1
1308.1
936.2
7117.6
429.4
483.9
OC
391.9
487.6
207.1
2369.4
433.4
475.1
MOPG
1856.9
1955.3
906.5
6884.1
907.4
$28.7
13

---

Published as a conference paper at ICLR 2023
CaleChet
Swimmer-v2
Huma HumanoidStandup v2
100
80000
3000
Episnile
00200
40000
x 10
x 10
1xx 10
Steps
Steps Walker2d
Steps Happer v2
Reacher
-10
2000
2000
1900
-00
AuspHish
INCD
Steps
Staps
Staps Antiv
Inverted/Penuulleen v2
InvertedDoubleeeendllee
v2
1000
1900
1000
4500
500
2000
x 102
xx 102
x 10
Stops Humanoid-v2
Stops
Stops
600
200
Stops
Figure S: Performance of Ten OpenAl Gym MuloCo Environments.
B.2 TEMPORAL EXTENSION
In Figure 6, we plot the average duration of each option during 430 training episodes (each episode contains a trajectory of 512 time steps) of the HalfCheetah environment. In this environment, the agent learns to run half of Cheetah by controlling 6 joints: back thigh, back shin, back foot, front thigh, front shin, and front foot. The faster the Cheetah runs forward, the higher return gets from the environment, At the start of training, all options' durations are short. After the 100-th episode, Option 3's duration quickly grows. This answers the question Q4 in the main text that MOPG is able to temporally extending the execution of an option in the abscence of the termination function.
500
400
Option 2
300
Option
200
100
0
100
200
300
Figure 6: Duration of options during 430 training episodes HalfCheetah.
To illustrate how MOPG composes options, take the HalfCheetah model trained after million steps and independently run it times (4 episodes. each episode contains 512 time steps). Option activation sequences of 4 runs are then plotted in Figure 7.
14

---

Published as a conference paper at ICLR 2023
PAn3
TAX4
Figure 7: Activated option sequences of 4 independent HalfCheetah runs.
As we can see that there are some common patterns between all independent runs. For example, all runs start with Option and use Option 2 at the early stage. After executing Option 2 for short period, they all switch to Option which has longest durations in all 4 runs. From time to time they will fall back to Option 2 for short periods and quickly switch to Option 3 again. This pattern of coordination indicates that because of the intrinsic reward and entropy terms in Eq. 44, one significant advantages of the MOPG over other algorithms that the ELBO objective function automatically learns to balance between degenerate options and dominant options. As shown in the video?, Option (green background in the video) is running forward option thus is executed most of the time. Option 2 (blue background) is mainly used to recover from falling down thus its duration decreases with training. Option 2 and Option have completely different functionality. Therefore, empircal study also shows that MOPG is able to compose disentangled options while preventing dominant options problem.
AHP
DAC
1PP
OC
PPOC
100
240
256
272
504
320
162
384
400
416
464
Figure 8: All option baselines' option durations for comparison in the HalfChectah environment
One interesting question to ask is whether the performance boost of MOPG comes from increasing decreasing of option frequencies compared to other option variants. In Figure we plot one run of all option variants' option duration.
As shown in Figure it appears to be that there is not a significant difference in terms of option duration between our method and other option variants. All of these methods are able to learn options which execute across various amount time as well as composing those options. It is also worth to mention that comparing with Figure 3, there is not apparent monotonically relation between option changing frequency and performance. Therefore, on simple environments like HalfCheetah, main performance difference comes with sample efficiency of training intra-option policies.
On the contrary, in harder environment like HumanoidStandupV2. there is a significant difference in terms of option composition between our method and other variants. As shown in Figure all option
15

---

Published as a conference paper at ICLR 2023
AHP
DAC
1OP
80
06
112
128
148
178
192
208
200
250
272
788
306
536
557
560
432
411
498
Figure All option baselines failed learn useful options in harder environment (HumanoidSSandupP2)
(b) Swirling around
(d) Balanding
6r1 Lying around
Kneeling around
Figure 10: Illustration of our method's Options composition in the HumanoidStandupp22 Environment.
variants except ours have rather low score in HumanoidStandupP2 As shown in Figure 9, all of these option variants tends to quickly and randomly switching between options. This means that they failed to learn useful options which focus on specific sub-tasks with longer durations. This is because that standard option frameworks have tendencies to learn either degenerate (Harb et al., 2018) options (short execution time) that switching back-and- sock-and-frrth too frequently, dominant options (Zhang & Whiteson, 2019) (long execution time) that executing through the whole episode. Both cases severely impair performance.
Figure 10 shows our method's options composition in HumanoidStandupy2. As shown in the figure that our method employs more options the environment gets harder: it successfully learns 4 useful optons and has a clear options composition schedule between stages of standing up. For option (a), the humanoid is lying around and trying sit up. For option (b) and option (c), these two options are more interchangeably used: option (b) is more sitting and swirling while option (c) is more trying stand up and recover from failed trials. After the humanoid stands up, the agent constantly call option (d) which is running around and trying to balance itself. In conclusion, our method can learn distinguishable options and composition schedule in harder environment. Because of MOPG's maximum entropy framework, our method is able to adapt to environment's complexity in an end-to-end manner and is more robust to degenerate and dominant options.
One last thing worth to mention is that the number of options used is not only related to complexity of the environment, but also the coefficient of entropy terms. Figure 11 shows a histogram of the used in total 20 runs. As
magnitude of entropy weights and how many options that as entropy weight increasing, more options are to be used since there more randomness in option policy: when entropy weight is 0.01, all 20 runs only use 2 options; when entropy weight
see
16

---

Published as a conference paper at ICLR 2023
Frequency
No. Options 2 4
20
15
Entropy Weight:
Figure 11: Histogram of entropy weights number of options used HalfCheetah environment. The larger the entropy weight, more options will be used.
0.2, 9 of 20 runs 4 options while only 7 2 options. Therefore, having an annealing training schedule of entropy weights might significantly boost performance of MOPG. However, to demonstrate the robustness our method we fixed the entropy weight to 0.01 in all experiments.
B.3 INTERPRETATION OF OPTION CONTEXT VECTORS
In this section continue with the HalfCheetah model used in Section B.2 and demonstrate how to interpret option context vectors as well as option activation sequences (Figure 7). In HalfCheetah, the agent learns to run half of Cheetah by controlling joints: back thigh, back shin, back foot, front thigh, front shin, and front foot. The faster the Cheetah runs forward, the higher return it gets from the environment. We interpret option context vectors and activation patterns by first inspecting what property each dimension of the option context vector encodes (Figure 13). Once each dimension is understood, options (Figure 12) become straight forward to interpret by simply inspecting on which dimension (property) they have the most significant weights (Figure 14). These interpretations can further taken to explain option activation patterns Figure 7.
2:0 611
EO0E LOS 6:81
LA1 3:1 CN4
A31 5:31 KCR SATE 039 K.BEUU SU 531 AISMCHI AROEOOO
eos 4x2 Hai
943 S/O/ 6.06
2018 4:00 Aa
K.Hi 4.1
GET SoN MI2
Figure 12: Heatmap of all 4 option context vectors
As the first step, we follow Sabour et al. (2017) to interpret what property each dimension of the option context vector in Figure 12 encodes by perturbing each dimension and decode perturbed option context vectors into primary actions. Specifically, we perturb one dimension by adding a range of perturbations [-0.1, 0.09] by intervals of 0.01 onto while keep the other dimensions fixed. After perturbation, each option context vector dimension has 20 perturbed vectors. We then use the action policy decoder decode all those vectors into primary actions and see how the perturbation affects the primary action. As an illustration, we plot Dimension 4's all 20 perturbed results in Figure 13.
With visualization of perturbation results in hand, can interpret what property each dimension encode by inspecting relationships between perturbations and primary actions. In Figure 13, as an example, is clear that changes on Dim has the same direction: as the magnitude of Dim 4 increase, all actuators move towards the same direction. This Dim can be seen as having acceleration of running forward effect.
Once we know how interpret one dimension, we move on interpret the whole option context vector. Since Option and Option 2 are two main options employed in Figure 7, here we provide an example of how to interpret them. Figure 12 shows that Option 1 has significant values dimension 11, 15 and 22. Option 2 significant on dimension 2, and 36. demonstrate these dimensions in the manner Figure 13 below:
17

---

Published as a conference paper at ICLR 2023
bthigh-
bshin
bfoot
fthigh
fshin
ffoot-
Figure 13: Perturbation on the Dim 4
DimS: Back Leg Jump
Dim2: Back Leg Run
Dim36: Front Run
Dim11: Back Landing
Dim15: Landing
Dim22: Balance
Figure 14: Interpretation of Option I and Option 2
Subtigures in Figure 14 can be interpreted in the same manner as Figure 13. As an example, from Figure we can see that Option has significant small value on Dim 11. In Figure 14, shows that smaller Dim 11 will twist the front leg forward and back foot forward while twist back thigh, back shin backward. Composition we can interpret that Dim 15 is front leg landing property and Dim 22 is balancing property. Therefore, Option is focusing on landing from all positions.
these movements a back leg landing property. Similarly,
Unlike other option context vectors which have apparent focusing dimensions, Option 2 has rather balanced option context vector. It has no apparently dominant dimension. only has slightly more significant values Dim 2, 5. 36, which focusing jumping and running properties. Therefore, Option 2 is more like an "all-weather" option: it is option having very balanced properties with slightly demonstration running and jumping.
Interpretations of Option and above can then be taken to understand option activation patterns in Figure 7: as all-weather option, Option 2 is the most frequently executed one and has the
18

---

Published as a conference paper at ICLR 2023
longest duration. From time to time, when the Cheetah needs land and balance itself, Option 1 will be executed. However, since landing option does not provide power of moving forward and thus has lower returns to continue, once the body is balanced the Cheetah will quickly stop Option 1's execution and keep running with Option 2.
B.4 TRANSFER LEARNING RESULTS
Below are statistics for Deepmind Control Suite (Tunyasuvunakool et al., 2020) transfer learning environments.
Table 3: Performance of Deepmind Control Suite Transfer Learning Enviromments
CartPole
Reacher
Cheetah
Fish
Walkerl
Walker2
PPO
829.7
327.6
73.0
287.9
231.8
72.2
DAC+PPO
970.8
517.2
211.2
505.4
590.3
360.5
AHP+PPO
966.5
395.2
167.4
357.9
362.1
143.2
PPOC
942.1
400.1
72.7
336.7
236.6
80.9
OC
106.1
19.4
100.6
286.6
356.3
238.7
MOPG
974.1
675.3
233.8
562.1
473.8
403.0
C AN MDP EQUIVALENCE TO THE SMDP OPTION FRAMEWORK
In this section, show that the the conventional Semi-Markoy Decision Problem (SMDP) option framework which employs Markovian options has an option-induced MDP equivalence. We first introduce the conventional SMDP formulated option framework (SMDP-Option). In Section C.2 we follow Bishop (2006)'s 's method and formulate the dynamics of the SMDP- framework as an Hidden Markov Model (HMM) style Probability Graphical Model (PGM) (Koller Friedman, 2009). With the PGM and its conditional independence relationships (Chapter 8.2.1 (Bishop, 2006)) in hand, in Section C.3 we move on to propose the Hidden Temporal MDPs (HiT-MDP) and prove its equivalence to the SMDP-Option under the definition of homomorphic equivalence (Ravindran, 2003). To the best of our knowledge, this is the first work discovering the option framework's MDP equivalence.
Following Bishop (2006)'s notation, we use A, B and C' denote three non-overlapping sets of arbitrarily many random variables. Sets A and B are conditional independent C if P(A, = P(BIC) denoted as A JL B C. We mainly use head-to-tail conditional independence properties (Chapter 8.2.1 (Bishop, 2006)) in this section.
C.I THE SMDP FORMULATED OPTION FRAMEWORK
Sutton al. (1999) proposed the option framework demonstrate the temporal abstraction problem. Following Bishop (2006)'s notation, we use bolded letter E S to denote a random variable and normal letter denote its realization. Without special clarification, random vector can have either a vector of continuous or discrete entries. A scalar E ž denotes the index of an option where O {1,2 2........ M) and M the number of options. An Markovian option is triple (Io. Pa (a(as)) (bls)) in which I, C is an initiation set where the option can be initiated. A is the intra-option policy which maps environment states E A. P.(bls) S Z2 is a termination function where b is binary random variable. It is used determine whether to terminate (b 1) the policy P, Pa (s) or not (b 0). Conventionally, variable period of time. set of
(ala))
S to an action vector
3, P. 1). Since option's execution may persist options' execution together with its value functions constitutes a Semi-Markov Decision Problem (SMDP) (Puterman, 1994). When old option is terminated, master policy
option will be sampled from the o P(0 [St+1) S O.
A master policy T(o)s) P(o|s) where E O is used to sample which option will be executed. Note that we use the bold-case o to denote unrealized random variables and the light-italic-cant o to denote realized instantiation. Conventionally, the execution of option employs the aall-and-return
19

---

Published as a conference paper at ICLR 2023
Figure 15: Illustration the SMDP Option Framework. An option o--1 is selected by master policy P(o:-1|ss.- 1) at time step - 1. At time step t. termination function B.,_, (sx) determines to continue option So that there is no random variable o, at time step / compared to there are random variables at every time step in MDP formulation (figure 16).
model (Sutton et al., 1999): at time step L, an agent either continues the previously executed option 0,-1 = o with probability Po (b = 0|s) and sets o, = 0,-1 = o, or terminates o with probability option from the master policy P(o, Therefore, the dynamics (stochastic process) of the option framework written as:
P. 1[s) and samples
20
P(r) (P(s) (ag
II ((s (a.ss: [s) Bo) + SoP [s:)].
=((s)))(oo))Pa a2 (s)
II [=1
(a)
Po. (b, = where {su, Ou, ag. $1, 01.01.111.... a1.... denotes the trajectory of the option framework. 1 is an indicator function and is only true when o, 0.-1 (notice that o the realization at 0.-1). For clarity reasons, we use P.(b- = 1[s) instead of 99, which is widely used in previous option literatures (c.g. (Sutton et al., 1999; Bacon et al., 2017)). Therefore, under this formulation the option framework is defined a Semi-Markov process since the dependency variable amount of time (Sutton et al., 1999).
[Po, (b. = 0s:
(16)
activated option cross
C.2 AN MDP FORMULATION OF THE OPTION FRAMEWORK
We follow Bishop (2006)'s formulation of mixture distribution and Probabilistic Graphical Models (PGMs). By introducing option variables as latent variables and adding extra dependencies into the termination function and master policy, we show that the conventional SMDP version of the option framework (Bacon et al., 2017: Sutton & Barto, 2018; Sutton et al., 1999; Harb et al., 2018; Zhang & Whiteson, 2019) can be re-formulated into an MDP formulation. We first follow Bishop (2006)'s formulation of mixture distributions and redefine the option random variable E O = (1.2.....K) which was originally defined as an integer index, but now as K -idimensionaal one-hot vector ō 0 = {0. ]]K where K is the number of options, and each entry (0, 1} is binary random variable. P(o, (s:) denotes the probability distribution over one-hot vector ó at time step conditioned on state s,. P(o, oz/sc) denotes probability entry (a scalar value) of the random variable with a realization at time step where o, and o E o./0t
In figure 16, s E o OK b BKK and A. denote the state, option, termination and action random variable respectively. ō is an K-dimensional one-hot vector and b is an K-dimensional binary vector where each entry b {0, 13. K is the number of options. Re+1 is the actual reward received from the environment after executing action a, in state s, G, = R4+1 + R + 2 R,43 ... is the discounted expected return where IR is discount factor.
20

---

Published as a conference paper at ICLR 2023
Figure 16: PGM of the MDP Option Framework. Notice the difference Figure 15. Figure SMDP formulated, suppose the agent decides to continue the execution of o--1, then the random variable does not exist, and thus o,41 depends the random variable directly. However, here the PGM is MDP formulated, and the random variable exists at every time slep.
The termination policy distribution P(b, (s : + B can be formulated as mixture distribution conditioned on option vector (the one-hot vector) 0:-1 and state St.
((b.)s;,,.-1>>>
II Pi(b,(s,)'.
(17)
iEO.-1
Because each option has its own termination policy Pb(b)s) with slightly abuse of notation, in
equation (17) we P(b,|s;, 6;-1) to denote the termination policy activated time step by previous chosen option 0,-1- To keep notation uncluttered, we use 3r = P(b, = Sr o,- 1) to denote the probability of option o,-1 terminates time step i and (1 (1-B3 (.-3,) P(b, 0|s,, 0.-1 denote the probability of continuation.
Conventionally, master policy (Zhang & Whiteson, 2019) (also called "policy-over-opptions"" (Sutton et al., 1999; Bacon et al., 2017))) is defined as:
P(o,(s.).
(18)
Similarly, we propose a novel mixture master policy as mixture distribution":
P sr b,, 6;-1) P(o.)ss))) ((o, where P(ō:(o..1) is degenerated probability distribution (Puterman, 1994)
(19)
f
if o; ō if + ō
P(o, o.
(20)
As shown in equation (19), the master policy only exists when b, 1 the option terminates. Therefore, PPOC (Klissarov et al., 2017) uses inaccurate gradients for updating the master policy during an option's 's execution.
According to the conditional dependency relationships in PGM (figure 16), the joint probability distribution of o, and b, can be written
P br|sc, ōz-1) P(b, |s,, o (s,, b;, ō,-1),
(21)
Different from conventional formulation which only depends st, termination function has extra dependence on 6,-1
'Different from conventional formulation which only depends state our mixture master policy has dependencies o and b,
21

---

Published as a conference paper at ICLR 2023
and the marginal probability distribution can be written as:
P(o: S1, 0z-1) = Σ o(- 01-1
(22)
bi
P(b, 0/s, 01-) + P(b, 1 1|s P(o1)))
(1 8,) 3,P(o.)s:))
The intra-option (action) policy distribution can also be formulated as mixture distribution
P(a St, o,) = II
P(aa))s))
(23)
iE0:
Therefore, the dynamics of the PGM in figure 16 can be written as:
P(f)
6o)
00
II
P(s s a: P( ot) Σ P(o ōz-1)
(24)
where P(F) Pss., do, ag, $1, b 01, a denotes the joint distribution of the PGM. Notice that under this formulation, P(T) is actually an HMM with St, a as observable random variables and bj, Og latent variables.
is worth to mention that equation (20) is essentially the indicator function =6.= used in conventional SMDP option framework papers and the last line in equation (22) is identical to transitional probability distribution in their formulation. However, as we show in this section, by adding latent variables 0,-1 and introducing the dependency between o, and b;. our formulation is essentially an HMM. It opens the door to introduce many well developed PGM algorithms such as message passing (Forney, 1973) and variational inference (Hoffiman et al., 2013) to the reinforcement learning framework. As we show below, the nice conditional independence relationships enjoyed by this model also enable us prove the equivalence between the option framework's 's SMDP and MDP formulation.
C.3 HIDDEN TEMPORAL MDPs (HIT-MDPS)
In this section we propose the Hidden Temporal MDPs (HiT-MDPs) and prove its equivalence under the definition of homomorphic equivalence. Following notations from Section C.2, the Hidden Temporal MDPs (HiT-MDPs) family can be described by a tuple M {S, A. p. where 0 = $ is an augmented state space, A - x O an augmented action space, and s P(o ( P(o, 0.-1) is the emit function for hidden variables. The joint distribution of HiT-MDPs is:
2
P(F) (s)) II
(( 5: = P(ov. $1)
II [=1
(( as or|s,, o.-1),
(25)
(=1
P((s)
II P (' 1/5,, a: at) o, P o,-1),
(26)
Notice that the Markovian master policy P(o, ō:-1) is actually the marginalization over the termination variable b, in Eq. 24: P(ō, Ss, o Σb. P(b, 6, b,, st, 01-1). The joint distribution is factorized from Eq. 25 to Eq. 26 by following conditional independences in the PGM (Figure 17):
We define a linear function f(6) o- :='''"''
which maps ö to o, where 21,
K -ddimensional constant integer vector, and hence f(o) o. Note that f is Bijection since linear function defined on a finite integer space. We define a tuple of partition function B <<< F.I. I, > where Í is the identical function and is also Bijection. Therefore, the partition function B is a tuple of Bijection functions BSs a) B (({0. (f a, o,}) ( S,, a, o,). which maps where
(So, 0g, ag, $1, 01, a1, is the trajectory of the HiT-MDP Therefore,
22

---

Published as a conference paper at ICLR 2023
Figure 17: PGM of HiT-MDP
by following the Partition Function B, the dynamics P(T) of the SMDP-Option in Eq. 24 under the Bijection is equivalent to the dynamics P(F) of the HiT-MDP
As for the reward function notice that from the PGM have the conditional independence and the HiT-MDP also share the same expected reward
T(+1 IL o Therefore. function T++1(1,, a, o₂) E'Ir+11 ( a, 0₂] E're+1 [S,, ay] (s, Therefore, the SMDP-based option framework has an MDP-based equivalence:
Theorem C.1. By the definition of Bisimulation Relation, the SMDP-based option framework, which employs Markovian options, has an underlying MDP equivalence because:
P(r/B) = P(T/B) and Bijection.
2.
Although the reward function r is equivalent, in Section we identify a problem that under the configuration HiT-MDPs, the conventional value function V[s, does not yield Bellman equation. We have to propose a Markovian Option-Valas Function Vlst. 6,-1] to solve this problem and prove that it equivalent V[s.]. This is non-trivial since compared to the SMDP-Option the MDP formulation introduces extra dependencies on ö. In Section D.2, by exploiting conditional independencies we prove that they do have the same expected return under the Bijection
D PROOF OF THE OPTIMAL VALUE EQUIVALENCE
In this Section we derive value functions of HiT-MDPs and prove optimal value equivalence to the option framework. Following the structure of Section C, we first reformulate value functions of SMDP-Option into an MDP formulation by following the dynamics in Section C.2. then following the HiT-MDPs' dynamics proposed in Section C.3 and derive Markovian Option- Option-Value Value function of HiT-MDPs, and prove converges to the optimal value. the end we prove that MDP-Option is optimal value equivalent to the SMDP-Option.
D.1 THE MDP FORMULATED OPTION FRAMEWORK'S VALUE FUNCTIONS
With dynamics proposed in C.2, now derive the MDP formulated value functions for the option framework (Bacon et al., 2017; Sutton et al., 1999). We follow Sutton & Barto (2018)'s notation in this section and write value functions for MDP below:
23

---

Published as a conference paper at ICLR 2023
Vis.] = [[ = Σ G:
Σ o.
G.
Oz
Σ P(oulsi)
G.P(G](s,.o))
Gr
Σ P(ools))B(GGlor,,
Σ P(oo)s:)Qoloo..s
(27)
where [[sr] is the state value function(Sutton & Barto, 2018) and Qolo,, S,] is the option value function(Bacon et al., 2017; Sutton et al., 1999), G; = E. 7++1 is the return. Note that in deriv- ing equation (27) only summation rule and production rule, the conditional dependency relationships in PGM (figure 16) are not used.
The option value function Qolo:, 8i] can be further expanded as:
Qo so] = :
Σ
a
= Σ P(ads st,
(28)
a.
where Qu (tr St, ar] is the option-action value function.
Proposition D.1.1. MDP formulation has identical state value function V[se] and option value function Qolo,, si] SMDP formulations
Proof. Note that in derivations above we only use summation and production rules. Both equation (27)
and (28) are identical to the conventional SMDP option framework.
From now on, we will continue derivations with conditonal independence relationships encoded in PGM (Chapter 8.2.1 (Bishop, 2006)). We have following conditional independence relationships from PGM (figure 16):
(Re+2, G+t1) Il (b41)
{oit))
(29)
(Rez,, G441] IL {s}}
fs1+1,01t
(30)
(Re42, G++1] Il {ad]
{81+1),
(31)
(Re+2, G++1) Il. {or}
0+1),
(32)
{ [R41, Gr, IL. {or}
{a.}.
(33)
With above conditional independence relationships in hand, we now show that the MDP formulation has identical value functions to the conventional SMDP formulation(uttonn et al., 1999; Bacon et al., 2017).
Proposition D.1.2. MDP formulation has identical option-action value function Qu (o 81, ag) to SMDP formulatioss
Quior, an] r(St, a1) + +7 Σ
P(s (,)
(34)
S1t1
24

---

Published as a conference paper at ICLR 2023
Proof.
Quior, Sc, ar]
or S1, ai]
Re( + 7r St, ar]
by definition of G2 use eq (33)
10 Re+ |sc. ar]+
Σ G++1 Σ
P(s Se, Oc, at) P:) 0z, St, ar)
7
Ge+1
8112
-r(St, a+)+
Σ
Go+1
Σ
use eq 30 31 and 33
G+t1
86+1
=r(s,, ax)
+ Σ
P(s a |[ 15,1,,
81-1
=r(st, a1) +
U o,].
However, unlike conventional value functions V[s,), Qolor, st). Qu st, of the option frame- work derived Section D.I, the value function U [ or] has an extra dependency on the hidden variable o, Therefore, conventional value function V(s.] does not yield Bellman equation under the configuration of HiT-MDPs. To tackle this issue, we propose the Markovian Option Value Function V (derivations of Eq. (6) in the main text):
V St, 0 EEG 6--1]
P(o,|ss: Or or, ō.-1)
Σ B(G
P(o. Qo-)) st),
(35)
where from line 2 to line 3 we the conditional independence property in PGM that G, JL 6 (s,, 6.).
Proposition D.1.3. MDP formulation has identical option-value function upon arrival U [s oi] SMDP formulations
U|8:41, =(1 Br+ 10o000+011 = Or, St+1] 1] + 3 V [[(++]]
(36)
=Qolor+1 O,, Sc+1] B.41A oz, S141]
(37)
'ott equations (36) and (37) is largely used in the conventional SMDP papers(Sutton al., 1999; Bacon al., 2017).
25

---

Published as a conference paper at ICLR 2023
Proof.
St41, orl
G++1
G++1
Pob. 0r,
11
Σ P S(11)
GGeL G++1
Σ
0+11 bs+1
Σ [11 B441) + B2+1 P( 1
=(1 Be+1 1 = 01. St+1 + Bc+ [(s+tt]
=Qolo++1 = 1c+1 - Br+1 A
1
from line 3 to line 4 use equation (29) and (32). From line 4 to line 5 use equation (22) and definition of Qo. The second last line use equation (27). The last line use the definition of advantage function A.
Under our MDP formulation, we also propose proposition D.1.4. We derive our gradient theorems based on equation (38) in section E. This important relationship largely simplify derivations than the original paper (Bacon et al., 2017) as well as give rise to the the HiT-MDP.
Proposition D.1.4. The option-value function upon arrival U[ o.] is expectation over option value function Qo (0 St+1] conditioned on previous option O:
U[$++1.01]= = Σ 00 500
(38)
0ct1
Proof. Following proof of proposition D.1.3,
U[$t41,00]
Σ
Σ Pbb::lo:::::))foo::bbbbb (bet 0,, S++1)
Σ
Got1 P(e:110HH1
Oc+1 b
Ge+1
Σ P(o 9o,41o0, 9010
D.2 THE OPTIMAL VALUE EQUIVALENCE BETWEEN HIT-MDP AND OPTION FRAMEWORKS
Theorem D.I. The option-action value function Eg. 7 satisfies the Bellman Operator TH THQAlse, or, ar] [E|Ge|Se, oe, ar]
= = +7 7 Σ P(s 21
(39)
where the Markovian option-value function given by Eq. 6.
Proof. Following dynamics derived in Section C.3, we can define value functions on HiT-MDPs as (derivations of Eq. (8) in the main text):
QAlse o,, ar] |s oG ar) E[R GGe+1/s, o, a.
=r(s,0,a) a) + Σ P( o, a; 1 o, ar]
=r(s,a) + Σ P(s a, 6.]
8111
=r(s, + Σ
P(s a:) ([541,1400
8441
26

---

Published as a conference paper at ICLR 2023
where from line to line the conditional independence property in PGM that Re+1 IL o G IL se|[s141. ) and G++1 TL a S1411 " E is a discounting factor. The last line uses the definition of the Markovian option value function (Eq. 35).
Theorem D.2. (Markovian Option Policy Evaluation Theorem). Assume that throughout our compu- tation the Qals-l and [--] are bounded and A < oc, the sequence Q% defined by Q g++1 THQU will converge the option-actio action value function Q as k + 3C.
Proof. As with the standard convergence results for policy evaluation (Sutton & Barto, 2018), by the definition of 7 (Eq. 8) the option-action value function Q is a fixed point.
To prove the T is contraction, define norm on V-values functions V and U
|V Ulo - max [V(3) - U(3)-
(40)
RES
where 08
By recurssively apply the Hidden Temporal Bellman Operator TH we have:
Vis,, = E =
P(o; St, δz- o1-1))) Σ
6,
Σ P(or|s), 01-1) Σ Play σt)) r(s,a)+ +7 Σ P(s.+ (s., az) ((s
8141
= =r(s,a) + Σ P(orls )-11)
Σ P(ay 0,)
Σ
P(s (
S1+1
= =r(8,a) +7 Σ
0158111
=r(s, a) + 7E 0e
(41)
Therefore, by applying Eq. to and we have:
1 - TU
max RES
V [[
FE U
7 max E a..
V ((s++1, o,] -Ü(s++1 oz.
SES
S max F 6.
max BES
V(st41, o.] 0 ((
SES
S max V - U[]]
> V Ul
(42)
Therefore, H. is contraction and by the fixed point theorem, Theorem 3.3 follows immediately. O Proposition D.2.1. The option-induced Markovian Option-Value Function V equivalent to the conventional value function V
Proof. for Proposition 3.4: By law of total expectation:
Eo, [sr, i1] = Eo. [E[r. -.] E[G[|s:] = V[se.]
thus V(ss, 01-1] is an unbiased estimator V[st). with conditional independences defined in PGM 17.
D
Option-Value Proposition D.2.2. The induced Markovian Option- Value Function V has smaller variance than the conventional value function V
27

---

Published as a conference paper at ICLR 2023
Proof. for Proposition 3.5: By law of total conditional variance:
Var(V[s:)) = Var( = B/VarrEi |ar o sx-
| [a V [s,, o;- [.] 0
2 Var(EE 01-1)](s)):
with conditional independences defined PGM 17
E MAX-ENTROPY OPTION POLICY GRADIENTS
In the main text Section we solve the HiT-MDPs under the maximum entropy framework (Levine, 2018). Here proof the Evidence Lower BOund (ELBO) proposed in the main text and derive the Maximum entropy Option Policy Gradient (MOPG) theorems.
E.1 THE MAXIMUM ENTROPY ORJECTIVE FUNCTION
Following notations defined in the main text Section 4, optimality variables 61:7 (e denotes the realization of e when e = 1) are treated as observed variables while random variables from the trajectory == (s, a, o...) are treated as latent variables. The variational Evidence Lower BOund (ELBO) is given by:
Theorem E.1. The problem of learning optimal action and master policies can be simplified as shrinking the KL-Divergence: Dxx(P(F)la(((eaxr)))
Proof.
/ q(e1:r S1., airr, 01:1 (1:r) do
log (ogrr) = log
/ / q(e.r Sir, air 01:T)
P(
log
ds ds
P(
log Ei g(s a177 (1
(43)
P(F)
10g((5i7..117,0077ll. log P(F)
=-DxuP(FFllatteeiry
where third last line the second last line follows from the Jensen's inequality and P(f) is the dynamics of the HiT-MDPs defined in Eq. 26
Theorem E.2. (Markovian Option Policy Iteration Theorem). Repeated application of Marko- vian Option evaluation and improvement to any TO,A E II converges to a policy TO,A such that
Q (St, āt) 2 Q at).oor for all TO,A E [[ and 8;, à, E $ x &, assuming | < co.
Proof. In this proof we use n for illustration, the proof of go follows the same derivation. From Theorem 4.1 we have that:
". arg max DKL. DKI, ((F((((((((Fle)))))
arg max
Ex [r(se, ar) +[o.] H(TO) + H(TA)),
(44)
arg max
Ex (r(s)) ar) /(o.] + H(TA)),
Σ Fxx [r(s), ad) + /[o.)] + H(Tr))
= arg max
A
Therefore, it must be the that E.. - (r(s a+) I(o:) + H( > E Eag (r(e, ar) Too:] + H(m)(d)) V=2'd [s, since always choose = oold II. Substituting this
28

---

Published as a conference paper at ICLR 2023
inequality into Theorem 3.2 leads to:
8. a) + [[]] 7 Fx [V*X [s,ō]
< a) + [[o] + y F. F Fawraru ( + [o++1] H(=x(1))
(45)
$ Q O,
where the convergence to Q" follows from the Markovian Option Policy Evaluation Theorem D.2. Therefore, the iteration of the sequence 2 monotonically increasing and we get Q% all A # T and (s, o, a) E $ x 0 A.
Q for
E.2 PROOF FOR THE MASTER POLICY GRADIENT THEOREM
Proof.
[Qo(sr,
ozl
0V [s oz
((0a - +7
P(ar|sr
P(sc a.)
00,
00,
a1
8141
VV [V(s++,,0
Σ P σt)
00.
0V St, 0
(P (sc, O 1)
Qo1s, ozl
ools2, oz] [))
+H(TT)) +7 Σ P(o,|se, δc-1)
00,
09,
OP(o St. 01-1)
0V [V|s o,]
Q018; 10.,0,00+000000 T(o'ss, + H(T +
Σ
P(
09.
00.
6
81+1.0,
Σ
k=
OP @P 0
p(k) (St+ 014k-1 0,-1)
Qo Qo(setk,004t Ō144] + T(o'ss (,oo|s,a a. H(mm))
0A,
OctN
OP(o'ls'. .0)
EE
H(mo)) St, 0.-
00,
E.3 PROOF FOR THE ACTION POLICY GRADIENT THEOREM
Proof. Similar to the first equation above, continue expanding gradients of 8Qo 09,
by equations (6) (28)
and (8):
aQo|s,, 6,]
ÖP(ay (st oz)
OV S141, o,
+ H(TTT.) + Σ QA(se, Or, arl
P(S St, oz)
00,
a0,
00,
at
8e11
OP OP(ay a St. ō.)
P(s 0411, 0Q0
Q QA(s,, 0,, ad] + +H(Tg) +7
Σ
00,
00,
8111.0:11
00
Σ
k=0 BLik.
(P(a4+l$/4kk.00 Q (8(SE o arta] H(m)))
00,
ai
OP OP(a S14b. or+h)
EE|
QA B1 o acta] + H(TT)) |s öz)
00,
29

---

Published as a conference paper at ICLR 2023
F LEARNING ALGORITHM FOR MOPG
Algorithm 1:
The MOPG Algorithm
Initialize the option embedding matrix Ws
Assign Initial State: S, So
Assign Initial Option: 0z-1 <- 0g
while Converge do
# Rollout trajectories and store in replay buffer
repeat
Retrieve the option context vector ô = W -6-1
Sample 6, p(ó (8t, Óạ-1)
Retrieve the option context vector o; W ó,
11
Sample a, ~ P( [S:, 6ς)
Compute @o(s;. 6r] and Vist, 0
12
13
Take action a, in St, observe new state 5,+1 and reward R(+1
14
until Rollout Length Reached
15
16
# Compute Advantages for option & action policies
17
Assign reversely, from RelgtutLength 1 repeat
18
19
Compute option Advantage
A = R&+1 + 7(V [$++1, ô:] V|s 0,-1] + Al + AHH) + 7AAI+1
20
Compute action Advantage
A, RL+1 7QQ (st+1, 0++1] Qo (s, o.] + AHd) + 7AA
21
until Rollout Length Reached
22
23
# A the GAE coefficient used in PPO.
# Optimize PPO Obj
25
while < PPO Optimization Epochs do
0. PPO @P (P(o'(s'.o) .0)
AA)
0. + PPO @Plass.))
AA)
end
end
G IMPLEMENTATION DETAILS
G.1 NEURAL NETWORK ARCHITECTURE
Interpretability is a key property to apply RL agents in real-world applications. Attention Option Critic (AOC) (Chunduru Precup, 2020) first introduces the attention mechanism into options. In AOC, each option has a unique attention mask over state W. O Therefore, each option has unique activation field (e.g., initiation set) over the observation space. Options learned by AOC are also interpretable since each option attends different context in the state vector.
Option
Our methods contributes to the option framework from a different aspect. AOC, an option is still a tuple of (I, B3 AA) while in our work an option is an embedding vector ó. The attention mechanism used as a distance measure to compare which option is closer to the concate- nated state-option pair (as shown in Eq. 48). As explained in the main text Section 5.3, options in HiT-MDP can be learned as option embeddings. Inspired by the Transformer (Vaswani al., 2017), in this section implement the
Categorienl
Dutibution
nutus,
Unioor
Critic
0.05.00
Canterin
Ads & Norm
Forward
Action Policy
Saussian Eleritutt
Fool Forward
Fand Forward
Cancatty, 0...l
30
Concaris,
W
D,
10...

---

Published as a conference paper at ICLR 2023
HiT-MDP simple yet effective Multi-Head Attention (MHA) (Vaswani et al., 2017) based Encoder-Decoder architecture as shown in Figure 18. Specifically, an atten- tion mechanism is described as the mapping from a q &R and a set of key-value pairs, i.e., K I M and V E IR MM (M and E' are total number of options and embedding dimensions), an output:
9KT
Attention(q. K, V) softmax
(46)
VE
A Multi-Head Attention MHA(q, K, V) is a linear projection of h (number of heads) concatenated
linearly projected Attention outputs:
q. K, V) = Concat[head,, heads] [
(47)
where head, = Attention (qW,, KW.K, VW.)
where projections are parameter matrices W." E REXE, W.K E REXE, W.Y e REXE, W.O R In this paper we use MHA as one building block as illustrated in Figure 18.
We implement the option policy P(ör|S+, 0 W.) the encoder and treat the option embedding matrix Wo as encoder's parameters. Specifically, we define the option encoder as:
o₂ ~
(Clustering(s;, 1,-1, W.)
(48)
where Categorical()) is K-dimensionall categorical distribution, K is the number of options, and distances between the pair [s:, ô,-1] (where -,-1 = W -1o,:1))Under this configuration, options can be seen as clustering centroids. The problem of selecting the next option o, is equivalent to caleulate
which clustering centroid ô in embedding matrix W. = [ō]..... δк] is closest to the projected state-option pair ([N d,-1)) by an efficient MHA-based clustering module:
Clustering(s;, 0,-1, W.) = FFN(MHA Query FFN((s, [s,, 6;-1)), Key=Value = W)) (49)
The action policy can be simply implemented as one decoder, which learns to decode ô, and s; into primary actions ag-
a, ~ Gaussian 6.]))
(50)
Because of the Markovian option-value function V( o.] is an expectation the option-value func- tion ( 0z++1] I] in Eq. (6), we only need to model only one critic function: Qo FFN(s,, oz), where Qo is also decoder of s; and o₁. We summarize the detailed algorithm in Appendix F and upload our code in supplementary materials.
G.2 HYPERPARAMETERS
In this section we summarize our implementation details. For fair comparison, all baselines: DAC+PPO (Zhang Whiteson, 2019), AHP+PPO (Levy Shimkin, 20. PPOC (Klissarov et al., 2017), OC (Bacon et al., 2017) and PPO (Schulman et al., 2017) are from DAC's open source Github tuu........ com/ Shangt ong waagg/hhnnn/////eeeee wtt///eeepPeeeeeeee DAC. Hyper-parameters used in DAC (Zhang& & Whiteson, 2019) for all these baselines are kept unchanged.
repo: https: / ////tittttt
MOPG Architecture: For all experiments, our implementation of MOPG is exactly the same as Figure 18 (b). We use Pytorch to build neural networks. Specifically, for skill policy module, we use a skill context matrix Ws E Rx4 which has skills (4 rows) and an embedding size of 40 (40 columns). For Multi-Head Attention, we use Pytorch's 's built-in MultiheadAttentin function with num heads = 1 and embed _dm = 40. For layer normalization we use Pytorch's 's built-in function LayerNorm For Feed Forward Networks (FNN), activation function with input size of 40, hidden size of 64, and output size of 64 neurons. For Linear layer, built-in Linear function map FFN's outputs dimension. Each dimension acts
use 2 layer FNN with ReLu function as
http:///wwccccoooooooo//sstteeee http:///eetttotttoettaiitaaaatta MultihhaddAttentinn...m
http:://yyoo.aaaaaaaaaaaaaaaeeee .n....naar..mll
31

---

Published as a conference paper at ICLR 2023
like logit for each skill and is used as density in Categorical distribution'. For both action policy and critic module, FFNs are of the same size as the one used in the skill policy.
Preprocessing: States are normalized by running estimation of mean and std.
Hyperparameters PPO: For a fair comparison, we use exactly the same parameters of PPO as DAC. Specifically:
Optimizer: Adam with e 10-5 and an initial learning rate 3 x 10
Discount ratio "y: 0.99
GAE coefficient: 0.95
Gradient clip by norm: 0.5
Rollout length: 2048 environment steps
Optimization epochs: 10
Optimization batch size: 64
Action probability ratio clip: 0.2
Computing Infrastructure: We conducted our experiments on an Intel@ CM 19-9900X CPU @ 3.50GHz with single thread and process with PyTorch.
32


Reviews:
[{"review_id": "w16vNf1PNL7", "reviewer": "Reviewer_ySTp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Mostly survey, not clear what the contribution is. The submission considers option RL. The major challenges the submission tackles is (1) the poor sample efficiency and (2) hard optimization for such a problem. The authors argue that the cause of the two challenges is the semi-MDP framework of existing option RL methods. Therefore, to deal with this issue, the submission reformulates the option RL problem into an HMM. Therefore, RL methods for HMM can be leveraged to deal with the two challenges in option RL.\n\nBoth theoretical and empirical results are provided supporting the proposed method. \n", "summary_of_the_review": "It's a survey paper with citations permeating through sections 6 and 7.\nThe submission proposes both novel formulation and novel methodology for option RL. Both theoretical and synthetic results are provided supporting the proposed method. However, I am not quite sure why the proposed method and formulation are able to provide such performance improvement. ", "strength_and_weaknesses": "Hard to learn from the survey. No applications discussed.\n\nStrength \n\nThe submission is well motivated, focusing on two key challenges in option RL. \n\nA novel formulation of option RL is proposed, which facilitates new option RL methods.\n\nThe results provided by the submission are thorough and complete, with both theoretical and empirical support for the methodology. \n\nWeaknesses\n\nIt is not very clear to me why the proposed new formulation and methodology can have a better sample complexity than existing ones. At the beginning, the authors state that the reason for the poor sample complexity of option RL is that one option lasts for multiple steps, which wastes observations. However, this is also true under the proposed HiT-MDP formulation: in each time step, there is a high probability that the option stays the same as the previous step. Both formulations have strong probabilities to maintain the selected option for multiple steps. So, why is the proposed one more sample efficient? \n\nWould it be possible for authors to provide the option change frequency of each method in experiments? In experiments, the proposed method shows better sample complexity than existing ones. I am wondering whether this is because the proposed method changes options more frequently than the competing ones? If so, can we just manually increase the option switch frequency of existing methods to achieve similar sample complexity gain? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "rLwa0JEOW_t", "reviewer": "Reviewer_uUrN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel  Markov Decision Process (MDP),  namely, the Hidden Temporal MDP  (HiT-MDP), to solve the semi-Markov\ndecision process that is unstable, unoptimizable, and inefficient in sampling ,and proves that the option-induced HiT-MDP is homomorphic \nequivalent to the option-induced SMDP. Then an efficient algorithm based on structured mutation reasoning is derived, which leads to a new \nmethod of finding stable options under the maximum entropy reinforcement learning framework. Finally, HiT-MDP shows excellent performance in a wide range of configurations.", "summary_of_the_review": "I think this is a good paper with solid theoretical analysis and contributions, the problems analyzed are very important, the proposed \narchitecture is feasible, the proof process is complete, and needs to be understood as deeply as this work.", "strength_and_weaknesses": "Pros:\nThe paper is very well motivated. The authors precisely present the shortcomings of the current standard option framework based on semi-Markovian decision processes and propose a new HIT-MDP that addresses the shortcomings of the original framework one by \none. It is also very clearly formulated. The arguments are clear, the derivation of formulas is rigorous. Even the lengthy appendix is fairly easy to parse, with straightforward proofs and relevant details. The most commendable thing is that the author has an innovative spirit. This paper is the first to propose an MDP equivalent to SMDP-Option, which provides reference and guidance for subsequent further research.\n\nCons:\nThe related work section is placed after the experimental section and before the conclusion and is somewhat abbreviated.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LM5RLZNE9ue", "reviewer": "Reviewer_iC2w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to solve Semi-Markov decision processes (SMDP) by reformulating the SMDP into an ordinary MDP. In particular, the authors propose a novel MDP view of SMDP that models the options as part of the states and actions, leading to the definition of several option-value functions. The authors propose to utilize an energy-based parameterization in the transition model, which results in an entropy regularizer term in policy optimization. The authors further solve such policy optimization problems by policy gradient and propose the MOPG algorithm for solving SMDPs. Finally, the authors conduct several experiments on the MuJuCo environment and compare MOPG with several SOTA algorithms.", "summary_of_the_review": "The idea is interesting, but the contribution is marginal to me.", "strength_and_weaknesses": "$\\textbf{Strength:}$\nThe authors propose a novel MDP view of SMDP that models the options as part of the states and actions, leading to the definition of several option-value functions. The proposed MDP model can potentially extend various existing RL algorithms that solve MDP to the SMDP problem setup.\n\n$\\textbf{Weaknesses and Questions:}$\n\n- How does MOPG compare with Attention Option-Critic [1], which also uses attention network to encode option?\n\n\n- The theory is not sufficient for me. There is no proof or discussion on the convergence or sample efficiency of MOPG. The equivalence between SMDP and MDP could be more significant if authors can provide such type of analyses.\n\n- The contribution is limited, as the proposed MOPG still falls in the well-established PPO framework, and similar architecture was explored before in [1]. \n\n[1] Chunduru and Precup. Attention Option-Critic. (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7N2-3yAIeof", "reviewer": "Reviewer_CFgS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies option learning in RL, and introduces a new framework (HiT-MDP). The authors prove that HiT-MDP is homomorphic equivalent to the standard SMDP formulation, and derive an on-policy policy gradient method with the new HiT-MDP formulation. On Mujuco environments, the proposed method is shown to be effective, exhibit smaller faster convergence and enjoy better interpretability.", "summary_of_the_review": "In summary, this paper introduces a new HiT-MDP framework for option learning in RL. Based on HiT-MDP, the authors derive a policy gradient method and validate its effectiveness and advantages over exiting methods on Mujoco.", "strength_and_weaknesses": "Strength:\n* The writing is in general clear.\n* The introduced HiT-MDP formulation is novel.\n\nQuestions:\n* Can the authors provide more visualizations of the learned options?\n* From Figure 6, it seems only 2 out of 4 options are useful. Is it because the environment is relatively simple? Does more complex environments (e.g. Humanoid) result in more options?\n* Can the authors conduct similar analysis (as in Figure 5,6) for other environments, such as Humanoid or Hopper?\n* How will the proposed method work in video games like Atari or Procgen? Can it learn some interesting options?\n\nMinor issues:\n* Please consider increasing the font size in Figure 1, 2, 5 and 6.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster