Diverse Exploration via InfoMax Options
Authors: Yuji Kanagawa
Tomoyuki Kaneko
Venue:  (2021)
Keywords: Reinforcement Learning
Hierachical Reinforcement Learning
Exploration

Abstract: In this paper, we study the problem of autonomously discovering temporally abstracted actions, or options, for exploration in reinforcement learning. For learning diverse options suitable for exploration, we introduce the infomax termination objective defined as the mutual information between options and their corresponding state transitions. We derive a scalable optimization scheme for maximizing this objective via the termination condition of options, yielding the InfoMax Option Critic (IMOC) algorithm. Through illustrative experiments, we empirically show that IMOC learns diverse options and utilizes them for exploration. Moreover, we show that IMOC scales well to continuous control tasks.


Under review as a conference paper at ICLR 2021
DIVERSE EXPLORATION VIA INFOMAX OPTIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we study the problem of autonomously discovering temporally ab- stracted actions, options, for exploration in reinforcement learning. For learning diverse options suitable for exploration, we introduce the infomax termination ob- jective defined the mutual information between options and their corresponding state transitions. We derive scalable optimization scheme for maximizing this objective via the termination condition of options, yielding the InfoMax Option Critic (IMOC) algorithm. Through illustrative experiments, we empirically show that IMOC learns diverse options and utilizes them for exploration. Moreover, we show that IMOC scales well to continuous control tasks.
INTRODUCTION
Abstracting a course of action as a higher-level action, or an option (Sutton et al. 1999). a key ability for reinforcement learning (RL) agents in several aspects, including exploration. In RL problems, an agent learns to approximate an optimal policy only from experience, given no prior knowledge. This leads to the necessity of exploration: environmental information, sometimes sacrificing immediate rewards. For statistical efficiency, important to explore the state space in a deep and directed manner, rather than taking uniformly random actions (Osband et al. 2019). Options can represent such directed behaviors by capturing long state jumps from their starting regions to terminating regions. It has been shown that well-defined options can facilitate exploration by exploiting an environmental structure (Barto et al.) 2013) or, more generally, by reducing decision steps (Fruit and Lazaric 2017).
agent needs explore the poorly known states for collecting
A key requirement for such explorative options is diversity. If all options have the same terminating region, they will never encourage exploration. Instead, options should lead to a variety of regions for encouraging exploration. However, automatically discovering diverse options in scalable, online manner is challenging due to two difficulties: generalization and data limitation. Generalization with function approximation (Sutton 1995) is important for scaling up RL methods to large continuous domains. However, many existing option discovery methods for exploration are graph- based (e.g., Machado et al. (2017) and incompatible with function approximation, except for that by Jinnai et al.) (2020). Discovering options online in parallel with polices requires us to work with limited data sampled from the environment and train the model for evaluating the diversity in data-efficient manner.
To address these difficulties, we introduce the infomax termination objective defined as the mutual information (MI) between options and their corresponding state transitions. This formulation reflects simple inductive bias: for encouraging exploration, options should terminate in variety of regions per starting regions. Thanks to the information ceammation-thhorttica formulation, this objective is compatible with function approximation and scales up to continuous domains. A key technical contribution of this paper is the optimization scheme for maximizing this objective. Specifically, we employ simple classification model over options as critic for termination conditions, which makes our method data-cfficient and tractable in many domains.
The paper is organized as follows. After introducing background and notations, we present the infomax termination objective and derive practical optimization scheme using the termination gradient theorem (Harutyunyan et al., 2019). We then implement the infomax objective on the option-critic architecture (OC) (Bacon et al. 2017) with algorithmic modifications, yielding the InfoMax Option Critic (IMOC) algorithm. Empirically, we show that (i) IMOC improves exploration in structured environments, (ii) IMOC improves exporation in lifelong learning, (iii) IMOC is scalable

---

Under review as a conference paper at ICLR 2021
to MuloCo continuous control tasks, and (iv) the options learned by IMOC are diverse and meaningful. We then relate our method to other option-learning methods and the empowerment concept (Klyubin et al. 2005), and finally give concluding remarks.
2 BACKGROUND AND NOTATION
We assume the standard RL setting in the Markov decision process (MDP), following Sutton and Barto (2018). An MDP M consists of a tuple A. p. where X' is the set of states, A is the set of actions, p X' x A X X' + [0, 1] is the state transition function, r A' x A + [Tmin. T'max] is the reward function, and - 7 S is the discount factor. policy is a probability distribution over actions conditioned on state 2, π. X' x A + [O, 1]. For simplicity, we consider the episodic setting where each episode ends when a terminal state zT is reached. In this setting, the goal of an RL agent is to approximate policy that maximizes the expected discounted cumulative reward per episode:
(RL(
R 1=0
(1)
where R2 = r( ax) is the reward received at time 4, and Do is the initial state of the episode. Relatedly, we define the action- value function Q (T(( def Ex,,. Etzl -t Ry and the
state-value function V** E.., E. (a)))) (x),,).
Assuming that æ is differentiable by the policy parameters 0z simple way to maximize the objective ① is the policy gradient method (Williams) 1992) that estimates the gradient by:
Vo, JRL
Ex,
Va, log=(ar (o an)
where A(x2, a.) is the estimation of the advantage function A (x,, a1) del a)) common choice of Á(x,, ae) is N-step TD error Σ Reti + V(+++N) V(x,), where N is a fixed rollout length (Mnih et al. 2016).
2.1 OPTIONS FRAMEWORK
Options (Sutton et al. 1999 provide a framework for representating temporally abstracted actions in RL. An option E O consists of tuple (J%, ;,3",²)) .T",, where Jo C X' is the initiation set, 3o X' > [0,
a termination function with (xr) denoting the probability that option o terminates in state xr and nT is intra-option policy. Following related studies (Bacon et al. 2017 Harutyunyan cct al. 2019). we assume that J = X and learn only 8 and T". Letting x, denote an option-starting state and z denote aption-terminating state,
write the option transition function as:
(=(x)1z)) 8 ("(
(x.x)) Σ pT "p""<"/aa (x1x1)))(zz)z))
(3)
where II is the indicator function and p²" is the policy-induced transition function pd (a'1x) der a). We assume that all options eventually terminate that po is valid probability distribution over æy, following Harutyunyan et al. (2019).
EeEA <a (r))pp/((//))) (x)
To present option-learning methods, we define two option-value functions: Qo and Uo. where Qo is the option-value function denoting the value of selecting an option o at a state I, defined def E.,,9.1 [ - Rs Analogously to Q and V we let Vo denote the
by Q)
Mool)))) o), where p(o|zz) x [0. 1] is the policy over options. Function Uo o) (1 (r)) (α,o) o) + (z)Voo is called the option-value function upon arrival (Sutton et al. 1999) and denotes the value reaching a state x
marginalized option-value function Vo(x) def
with and not having selected the new option.
2.2 OPTION CRITIC ARCHITECTURE
OC (Bacon et al. 2017) provides an end-to-end algorithm for learning T" and 3 in parallel. To optimize nọ, OC uses the intra-option policy gradient method that is the option-conditional version
2

---

Under review as a conference paper at ICLR 2021
of the gradient estimator Vo (R. VV 8. log 1og="(a(1z=)) ael] where A is an estimation of the option-condtiional advantage A
For optimizing 30, OC directly maximizes Qo using the estimated gradient:
Qola, o) >B [ V (xx) (Qo(z, o) -Vo(o))
(4)
Intuitively, this decreases the termination probability (z) when holding an is advantageous, i.c., Qo(x)) Vo(x) is positive, and vice versa. Our method basically follows OC but has different objective for learning 8".
2.3 TERMINATION CRITIC
Recently proposed termination critic (TC) (Harutyunyan et al. 2019) optimizes 330 by maximizing the information-theorett objective called predictabilit::
(P") -X
(5)
where H denotes entropy and X is the random variable denoting the option-terminating states. Maximizing - ( [o) makes the terminating region of an option smaller and more predictable. In other words, can compress terminating regions by optimizing the objective (5). To derivate this objective by the beta parameters 0ze. Harutyunyan et al. (201)) introduced the termination gradient theorem:
Theorem 1. Let 80 be parameterized with sigmoid function and Eg. denote the logit of 8o. We have
Vo, Pa(g(lr)))
Σ
(6)
Leveraging the theorem |
TC performs gradient ascent using the estimated gradient:
((1gg Pa log Pi(z)) + Pa(xr)
Vo T((O) -Eg,,z,y (x)
where (x) is the marginalized distribution of option-terminating states.
Contrary the termination objective of OC 4 this objective does not depend on state values, making learned options robust against the reward structure of the environment. Our method is inspired by TC and optimizes a similar cooommatinntthoreeee objective, our infomax objective requires an estimation of p(o)x.. zy) instead of the option transition model po (a (x,), which makes our method tractable more environments.
for predictability but for diversity. Also,
INFOMAX OPTION CRITIC
We now present the key idea behind the InfoMax Option Critic (IMOC) algorithm. We first formulate the infomax termination objective based on the MI maximization, then derive a practical gradient estimation for maximizing this objective on 3 utilizing the termination gradient theorem |
To evaluate the diversity of options, we use the MI between options and option--erminatinng states conditioned by option-starting states:
JIMOC I(xx; = -..XXX.--XXXXX.))) -
(7)
where 7 denotes conditional MI T(A: B|Z) H(A[Z) - H(ABB, 2), X, is the random variable denoting option-starting state, and O is the random variable denoting an option. We call this objective the infomax termination objective. Let us interpret X, |X,, as the random variable denoting state transition induced by an option. Then maximizing the MI 7 (i) diversifies a state transition XX and (ii) makes an option-conditiona state transition | s. o more deterministic. Note that the marginalized MI I I(Xy: O) also makes sense in that it prevents the terminating region of each option from being too broad, as predictability (5) does. However, in this study, we focus on the conditional objective since it is easier to optimize.
3

---

Under review as a conference paper at ICLR 2021
Option
D-2.0
Option
9e0.0
Option
1-1.0
Option
1-1.
1000
p-2.
Two insances of InfoMax options in the four state deterministic chain. Left: Options are diverse but all state transitions per option are one-step. Right: Options enable relatively long state transitions but option-policies are the same at some states.
Figure 1:
To illustrate the limitation of infomax options, we conducted an analysis in toy four-state determin- istic chain environment, which has four states and two deterministic actions (go left and go right) per each state. Since deriving the exact solution is computationally difficult, we searched options that maximize H (X.) from deterministic options that has deterministic option-policie, and termina- tion functions (thus has the minimum H O)). Among multiple solutions, Figure] shows two interesting instances of deterministic infomax options when |O] 2. The left options enable diverse behaviors per state, although they fail to capture long-term behaviors generally favorable in the literature (e.g.. Mann et al. (2015)) On the other hand, the right options enable relatively long, two step state transitions, but they are the same and the rightmost state and the next one. Furthermore, an agent can be caught in small loop that consists of the leftmost state and the next one. This example shows that (i) we can obtain short and diverse options with only a few options, (ii) to obtain long and diverse options,
need sufficiently many options, and (iii) an agent small loop with only a few options, failing to visit diverse states. As we show in Appendix [A] this "small loop' problem cannot happen with four options. Thus, the number of options important when we are to maximize the MI (7) and a limitation of this method. However, in experiments, we show that we can practically learn diverse options with relatively small number of options.
be caught in
For maximizing the MI by gradient ascent, we now derive the gradient of the infomax termination objective (7). First, we estimate the gradient of the objective using the option transition model p and marginalized aption-transition model P( (r.)
Proposition 1. Let So be parameterized with a sigmoid function. Given trajectory I...... F, ..... Fy sampled by x" and 89, we can obtain unbiased estimations of Vo HX, and Vo, () (X,, O) by
V 0 (X,, = E,,,,, - Vo (log P(r(x,) - log P(z
(8)
Vo, HX, = Ex (o log P (z|z,) log P (2 (x2)l11)))
(9)
where (x) denotes the logit of (xr).
Note that the additional term 8 is necessary because æ is not actually a terminating state. The proof follows section in Harutyunyan et al. (2019) and is given in Appendix B.1
The estimated gradient of the infomax termination objective
can now be written as:
Vog Vo, (X,) H(,l( O)
E V e (x) (log P(x[xx)) log ((x - (log Po (x((x.) - log (e ((xs((x))))
(10)
optimize this objective by estimating po and P. However, estimating the probability over the state space can be difficult, especially when the state space is large, as common the deep RL setting. Hence, we reformulate the gradient using Bayes' rule in similar way as Gregor et al. (2017). The resulting term consists of the reverse option transition p(o|z., that denotes the probability of having an given state transition F,, 2.
which means that
4

---

Under review as a conference paper at ICLR 2021
Proposition 2.
We now have
Vo, O H(Xg
-E Voglz-(z)
(2)))()))))10p)a))))) logp(o/=z;:)) logp(o/2,,
(11)
The proof is given in Appendix B.2 In the following sections, we estimate the gradient
(1] by
learning classification model over options x.) from sampled option transitions.
4 ALGORITHM
In this section, we introduce modifications for adjusting the OC (Bacon et al.] 2017) to our infomax termination objective. Specifically, we implement IMOC on top Advantage-Option Critic (AOC), synchronous variant of A2OC (Harb et al. 2018), yielding the Advantage-Actte InfoMax Option Critic (A2IMOC) algorithm. To stably estimates p(o]z,, 2y) for updating 0g, we sample recent option state transitions o, 2g 2y from We follow AOC for optimizing option-policies except the following modifications and give a full description of A2IMOC in Appendix [C.] In continuous control experiments, also used Proximal Policy InfoMax Option Critic (PPIMOC) that is implementation of IMOC based on PPO (Schulman et al.) 2017). We give details of PPIMOC in Appendix C.2
Upgoing Option-Advantaan Estimation Previous studies (e.g.. Harb et al. (2018) estimated the advantage Å (a+e) ignoring the future rewards after the current option or terminates. Since longer rollout length often helps speed up learning (Sutton and Barto 2018). is preferable to extend this estimation to use all available future rewards. However, future rewards after option termination heavily depends the selected option, often leading to underestimation of Åo. Thus, to effectively use future rewards, we introduce an upgoing option-ad advantage estimation (UOAE). Let t + & denote the time step where the current option or terminates in sampled trajectory. Then, UOAE estimates the advantage by:
Σ RRe+ + max
Σ j=k
Y Retj 7 Vo((et))
(k< <<<N) N)
ALOAE = -Qo(z,,0))
upgoing estimation
ΣN Rott + N Uo(EHNN
(otherwise)
(12)
Similar to upgoing policy update (Vinyals et al. 2019), the idea is to be optimistic about the future rewards after option termination by taking the maximum with Vo.
Policy Regularization Based Mutual Information To perform MI maximization not only on termination functions but also pettoon-ooliies, we introduce a policy regularization based on the maximization of the conditional MI. (A:OO)x, where A is the random variable denoting an action. local approximation of the infomax objective
This MI can be interpreted each action leads to different terminating regions. Although optimizing the infomax termination objective diversifies option- policies implicitly, we found that this regularization helps learn diverse reliably. Letting TH denote the marginalized policy nH (ala) we write T(A; O(x,) as:
7 assuming that
def E. (olor)
O(A = H(AXX.) (AAAXX.)H(AAOA,)) H(AOO,
= E,, (x.(( Ex,, (+TT(()))))
We use this regularization with the entropy bonus (maximization of H(m))) common in policy gradient methods (Williams and Peng 1991 Mnih et al. 2016) and write the overall regularization
term
ch, H (z)) en (
(13)
where CHr, and Cиr are weights of each regularization term. Note that we add this regularization term on not only option-starting states but all sampled states. This introduces bias, which did
5

---

Under review as a conference paper at ICLR 2021
La
Cump Rewarts
1.0
0.5
Methed
ACC
LGD
0.0
CU/ADC
100020
202000
300200 400000 500020
(a) Gridworld environment.
Total Envirorment Steps
Blue grid is the start and green grids are goals.
(b) Performance progression.
Figure 2: Single Task learning in Gridworld Four Rooms.
not find to be harmful when CH2,, is reasonably small. To approximate H( we employ ji that is the empirical estimation of μ.. Using jt, H (Tx) is computed by n (a|z) ~ E. A(olz)T for descrete action spaces and approximated by Monte Carlo method for continuous action spaces. We show the details in Appendix C.T
EXPERIMENTS
We conducted a series of experiments to show two use cases of IMOC: exploration in structured environments and exploration for lifelong learning (Brunskill and Li 2014). In this section, we used four options for all option-lcarning methods and compared the number of options in Appendix D.7
5.1 SINGLE TASK LEARNING IN STRUCTURED ENVIRONMENTS
We consider two 'Four Rooms' domains, where diverse options are beneficial for utilizing environ- mental structures.
Gridworld Four Rooms with Suboptimal Goals First, tested IMOC in variant of the classi- cal Four Rooms Gridworld (Sutton et al. 1999) with suboptimal goals. An agent is initially placed the upper left room and receives positive reward only at goal states: two closer goals with +1 reward and the farthest goal with +22 reward, as shown in Figure 2a The episode ends when an agent reaches one of the goals. The optimal policy is aiming the farthest goal in the lower right room without converging to suboptimal goals. Thus, an agent is required to learn multimodal behaviors leading to multiple goals, which options can help. this environment, we compared A2IMOC with A2C (Mnih et al.) 2016: Wu et al. (017). AOC, and our tuned version of AOC (our AOC) with all enhancements presented in section to highlight the effectiveness the termination objective among all of our improvements/ []] We show the progress of average cumulative rewards over ten trials in Figure 2b A2IMOC performed the best and found the optimal goal most trials. AOC and our AOC also occasionally found the optimal goal, while A2C overfitted to either of the suboptimal goals through all trials.
Figure illustrates learned option-polices and termination functions of each compared method Terminating regions learned with A2IMOC is diverse. For example, option O mainly terminates in the right rooms while option 3 terminates in the left rooms. We that termination regions learned by A2IMOC are diverse and clearly separated per each option. Although all option policies converged the same near optimal one, we show that A2IMOC diversifies option-policies learning in Appendix D.4 On the other hand, terminating regions learned with AOC overlap each other, and notably, option 3 has no terminating region. We assume this is because the loss function 44) decreases the terminating probability when the advantage is positive. We can see the same tendency our AOC, although we cannot see the vanishment of the terminating regions.
the beginning of
Note that we did not include ACTC (Harutyunyan et al. 2019 comparison since we failed to reliably
reproduce the reported results with our implementation of ACTC.
Note that choose the best model from multiple trials for visualization throughout the paper.

---

Under review as a conference paper at ICLR 2021
A2IMOC
9.6
8
Optio
Option 1
Option
Option 3
0.0
AOC
10
B
-04
Optin
Oppon
Option 2
Option 3
OurAOc
04
Option
Option
Option
Opton
Figure Learned option-policies (I²") and termination probabilities (3") each option in Gridworld Four Rooms after 6 x 10 steps. Arrows show the probablities of each action and heatmaps show probabilities of each 8 First row: A2IMOC. Terminating regions are clearly different each other and there are few overlapped regions. Second row: AOC. Almost all states has high termination probablity with option 0, 1,3 and option has no terminating region. Third row: Our AOC. Termination regions are not clearly separated per each option.
1.0
Method
PPINDC
Cumulativa Rowards
0.8
PPOC
OurPPOC
0.6
0.4
0.2
0.5
lu5
(a) MuJoCo Point Four Rooms. The agent an orange ball and there are three goals: green goals are suboptimal (+0.5) and the red one in optimal (+1.0).
(b) Performance progression.
Figure 4: Single Task learning in MuJoCo Point Four Rooms.
MuJoCo Point Four Rooms To show the scalability of IMOC in continuous domains, we con- ducted experiments in similar four rooms environment, based the MuJoCo (Todorov al. 2012) physics simulator and "PointMaze" environment in rllab (Duan et al. 2016). This environment follows the Gridworld Four Rooms and has three goals as shown Figure 44 two green goals with +0.5 reward, and red one with +1.0. An agent controls the rotation and velocity of the orange ball and receives a positive reward only at the goals. this environment, we compared PPIMOC with PPOC (Klissarov et al.) 2017). PPO, and our tuned version of PPOC (our PPOC) that is the same as PPIMOC except for the termination objective. An important difference is that PPOC uses a parameterized μ trained by policy gradient, while PPIMOC and our PPOC use e-greedy for option selection. Figure 4b show the progress of average cumulative rewards over five trials. PPIMOC found the optimal goal four times in five trials and performed slightly bettern than PPO. In qualitative
7

---

Under review as a conference paper at ICLR 2021
100
Method
PRIMOC
Cumulatve keweros
50
Cumulative Rewerds
PPDC
CurPPOC
200
-25
Method
1.00
-50
PPINOC
PPCC
-75
OutPPC
PPG
Total Environment Steps
Totall Environment Steps
(a) Performance progression Mountain Car.
(b) Performance progression in Cartpole swing
Figure 5: Single Task learning in MuJoCo Point Four Rooms.
1.0
Cumulative Rewards
0.4
Methad
PRIMOC
PPOC
0.0
OurPPOC
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Total Environment Steps
leb
(b) Performance progression.
(a) MuJoCo Point Billiard. Four goals periodically clockwise.
Figure 6: Lifelong learning in MuloCo Point Billiard.
analysis, we observed the same tendency in learned options as the Gridworld experiment, where the details are given in Appendix D.5
5.2 SINGLE TASK LEARNING IN CLASSICAL CONTINUOUS CONTROL
Additionally, we test PPIMOC on two classical, hard- explorrxpoontoonn control problems: Mountain Car (Barto al. 1983) and Cartpole swingup (Moore 1990). Mountain Car, PPIMOC and our PPOC successfully ly learns to reache the goal, while PPO and PPOC converged to run around the start posisition. In Cartpole swing up, PPIMOC and our PPOC performed better than PPO and PPOC, but still failed to learn stable behaviors.
5.3 EXPLORATION FOR LIFELONG LEARNING
As another interesting application of IMOC, we consider the lifelong learning setting, Specifically, we tested IMOC in 'Poit Billiard' environment. In this environment, an agent receives positive reward only when the blue objective ball reaches the goal, pushed by the agent (orange ball). Figure 6a shows all four configurations of Point Billiard that used. There are four goals: green goals with +0.5 reward and a red one with +1.0 reward. The positions of four goals move clockwise after IM environmental steps and agents need adapt the new positions goals. We compared PPIMOC with PPO, PPOC, and our PPOC in this environment. Figure 6b) shows the progress of average cumulative rewards over five trials. Both PPIMOC and our PPOC performed the best and adapted to all reward transitions. On the other hand, PPO and PPOC struggle to adapt to the second transition, where the optimal goal moves behind the agent. The ablation study given in Appendix [.6 shows that UOAE 12 works effectively in this task. However, without UOAE, PPIMOC still outperformed PPO. Thus, we argue that having diverse terminating regions itself is beneficial for adapting to new reward functions in environments with subgoals.
8

---

Under review as a conference paper at ICLR 2021
6 RELATED WORK
Options for Exploration Options (Sutton et al. 1999 in RL are widely studied for many applica- tions, including speeding up planning (Mann and Mannor 2014) and transferring skills (Konidaris and Barto 2007 Castro and Precup 2010). However, as discussed by [Barto et al. (2013), their benefits for exploration are less well recognized. Many existing methods focused on discovering subgoals that effectively decompose the problem then use such subgoals for encouraging exploration. Subgoals are discovered based on various properties, including graphical features of state transitions (Simsek) and Barto, 2008; Machado et al. 2017;
Vigorito and Barto 2010). In contrast, Jinnai et al. 2019) and causality (Jonsson and Barto 2006] method directly optimizes termination functions instead of discovering subgoals, capturing environmental structures implicitly. From a theoretical perspective, Fruit and Lazaric (011 analyzed that good options can improve the exploration combined with state-action visitation bonuses. Using infomax options with visitation bonuses would be an interesting future direction.
2017:iinnai
End-to-end learning of Options While many studies attempted to learn options and option-policies separately, |acon et al. (2017) proposed OC to train option-policies and termination functions in parallel. OC has been extended with various types of inductive biases, including deliberation cost (Harb et al. 2018), interest (Khetarpal et al. 2020), and safety (Jain et al., 2018). Our study is directly inspired by information-theoreet approach presented by Harutyunyan et al. (019))
noted in section
Mutual Information and Skill Learning MI often appears in the literature of intrinsically moti- vated (Singh et al. (004)) reinforcement learning, as driver of goal- coal-directed behaviors, A well-known example the empowerment (Klyubin et al. 2005 Salge et al.) 2013). which is obtained by maxi- mizing the MI between sequential h actions and the resulting state Tat,.... ..... Gt+k; Te++ (zz).. Some works (Mohamed and Rezende 2015 Zhao et al. 2020) implemented lower bound maximization of empowerment intrinsic rewards for RL agents, encouraging goal-directed behaviors in the absence of extrinsic rewards We can interpret our objective I (XX,) as empowerment between limited action sequences and states corresponding to options. Gregor et al. (2017) employed this interpretation and introduced a method for maximizing the variational lower bound of this MI via option-policies, using the same model our P, while aim to maximize the MI via termination functions. MI is also used for intrinsically motivated discovery of skills, assuming that diversity is important to acquire useful skills. Eysenbach al. (2019) proposed to maximize MI between skills and states (O; extended to the conditional one I(O; X") (X) by Sharma et al. (2020). Although our study shares the same motivation for using MI as these methods, i.e. diversifying sub-policies, the process of MI maximization is significantly different: our method optimizes termination functions, while their methods optimize conditional policies by using MI as intrinsic rewards.
7 CONCLUSION
We presented a novel end-to-end option learning algorithm InfoMax Option Critic (IMOC) that uses the infomax termination objective to diversify options. Empirically, we showed that IMOC improves exploration in structured environments and for lifelong learning, even in continuous control tasks. We also quantitatively showed the diversity of learned options. An interesting future direction would be combining our method for learning termination conditions with other methods for learning option-policiest e.g., by using MI as intrinsic rewards. A limitation of the infomax objective presented in this study is that it requires on-policy data for training. Hence, another interesting line of future work is extending IMOC to use for off- off-po option discovery.
REFERENCES
P. Bacon, J. Harb, and D. Precup. The option-critt architecture. In Proceedings of the Thirty- First AAAI Conference Artificial Intelligence. February 4-9, 2017, San Francisco. California, USA, pages 1724-1734, 2017. URL http:://aaaaaiggooee wwtpppppppp...aa.....
paper wtteer/viwwie 14858

---

Under review as a conference paper at ICLR 2021
A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern 13(5):834 846, 1983. doi: 10.1109/TSMC..193..333 13077.
A. G. Barto, G. D. Konidaris, and M. Vigorito. Behavioral hierarchy: Exploration and representa- tion. In G. Baldassarre and M. Mirolli, editors, Compulational and Robotic Models of the Hierarchi- cal Organization of Behavior, pages 13-46. Springer, 2013, doi: 10. 1007/978-33-42239877777 URL wttp://///o.ooo/11110000000
642 39875-9887922
E. Brunskill and L. Li. Pac-inspired option discovery in lifelong reinforcement learning. In Pro- ceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 324. JMLR.org. 2014. URL http: /tttt::/proot http://roceeedeee ing.mmmmimn.mmmmmm..m ngs.ml r.press v32 1t552/bunskii. www.mmskk111.hh.... 14.html
P. S. Castro and D. Precup. Using bisimulation for policy transfer in mdps. In M. Fox and D. Poole, editors, Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 1 11-15, 2010. AAAI Press, 2010. URL ht.t::t::// www./aawwwaaaaaaowaooao/aaaaaaaa i.org/
ocs/ hess/idxx.ppp///
Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In M. Balcan and K. Q. Weinberger, editors, Proceedings of the 33nd International Conference Machine Learning, ICML. 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1329-1338. JogR. 2016.
/ duan16.htm.
B. Eysenbach, A. Gupta, J. Ibarz, and Levine. Diversity is all you need: Learning skills without a reward function. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL SJx63 (RqFm
for
R. Fruit and Lazaric. Exploration-explottaiii in mdps with options. In Proceedings of the 20th International Conference on Artificial Intelligence and Statisties, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, pages 576-584, 2017. #54//ruittin (aa.uutt1a..hmml
K. Gregor, D.
Rezende, and D. Wierstra. Variational intrinsic control. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL httpps::/ppenreview...ee.. forum? ?id=
Skc-Fo4Yg
J. Harb, P. Bacon, M. Klissarov, and D. Precup. When waiting is not option: Learning options with a deliberation cost. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the Sth AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18). New Orleans, Louisiana, USA, February 2-7, 2018, pages 3165-3172. 2018. URL https 7/ //www.aaai.org// /ww.////////////oooo///ooo/o.oo/ ocs index. AAAI/ // AAAIIAAAI 1I88/111///// wttpe/er//ieww/110
A. Harutyunyan, W. Dabney, D. Borsa, N. Heess, R. Munos. and D. Precup. The termination critic. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, pages 2231-2240, 2019. URL http:// proceedi ngs.ml wwwpess/vv//aatttyyooo
9a.html
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. 7. Leibo, D. Silver, and K. Kavukcuoglu. Rein- forcement learning with unsupervised auxiliary tasks. Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https////ppeeeeeee..eeeeeoooeees
A. Jain, K. Khetarpal, and D. Precup. Safe option-critic: Learning safety in the option-critic architecture. CoRR, abs/180) 807. 807.08060, 2018.
Y. Jinnai, J. Park, D. Abel, and G. D. Konidaris. Discovering options for exploration by minimizing cover time. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International
10

---

Under review as a conference paper at ICLR 2021
Conference on Machine Learning, ICML 2019, 9-15 June 2019. Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages $130-3139 PMLR, 2019. URL
vv7/ (eei///nnnnn99... 19..tm11
Y. Jinnai, w. Park, M. C. Machado, and D. Konidaris. Exploration in reinforcement learning with deep covering options. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020. URI. https://
A. Jonsson and A. G. Barto. Causal graph based decomposition of factored mdps. J. Mach. Learn. Res., 7:2259-2333 2006. URL
K. Khetarpal, M. Klissarov, M. Chevalier-Boisvett, P. Bacon, and D. Precup. Options of interest: Temporal abstraction with interest functions. In The Thirty- SourtthFourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 4444 4451. AAAI Press, 2020. URL https: http:::/aaaa.org//// Ballgg//jsiinnn index.php... hwtp:ppp//AAAAaaiaaeeeieeee /tt.///aaiicee//eee/////
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
abs/
1412.6988 6980
M. Klissarov, P. Bacon, Harb, and D. Precup. Learnings options end-to-end for continuous action tasks. CoRR,
2017. URL
A. S. Klyubin, D. Polani, and C. L.. Nehaniv. All else being equal be empowered. In M. S. Capcarrère, A. A. Freitas, P. J. Bentley, C. G. Johnson, and J. Timmis, editors, Advances in Artificial Life, 8th European Conference, ECAL 2005, Canterbury, UK, September 5-9, 2005, Proceedings, volume 3630 of Lecture Notes Computer Science, pages /ttp://:////...oo//0000010111/11
Springer, 2005.
doi: 10.1007/1 1553090' 75.
G. D. Konidaris and A. G. Barto. Building portable options: Skill transfer in reinforcement learning. In M. M. Veloso, editor, IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6- 2007, pages 895-900, 2007. URL
14f.
M. C. Machado, M. G. Bellemare, and M. H. Bowling, A laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 2295-2304, 2017. URL http://proceedi.ng www.prsss..pppeee/ompp.mm..s.e http://///////1////// a.html
T. A. Mann and S. Mannor. Scaling up approximate value iteration with options: Better policies with fewer iterations. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 127-135. JMLR.org, 2014. URL http:::ppppr..eed.g...nn..nppppp v32 2022mann14.htm.
A. Mann, Mannor, and D. Precup. Approximate value iteration with temporally extended actions. J. Artif. Intell. Res., $3:375-434 2015. doi: 1/663//air.4666.. URL https: http::://ddoioog /
10.1 613/ jaa./jarr.46666
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd Interna- tional Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1928-1937, 2016. URL http:
mn mnihal m
S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated rein- forcement learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gareett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information
11

---

Under review as a conference paper at ICLR 2021
Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2125 2015. URL http:////aacccacccca//////aaaaa.
maximisation-foos htt.aaaaaaaaaaaaaaaaaaaaaaaaaaaa
A. W. Moore. Efficient memory-based learning for robot control. Technical Report UCAM- CL-TR-209, University of Cambridge, Computer Laboratory, Nov. 1990. URL https://
www.cl.cam.accuk/feccrreroooor CCA
I. Osband, B. V. Roy, J. Russo, and Z. Wen. Deep exploration via randomized value functions. Mach. Learn. Res.,
2019.
339.html
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf. E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chil- amkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. M. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Bue, E. B. Fox, and R. Garnett, editors, Advances Neural Informa- tion Processing Systems 32: Anmual Conference on Neural Information Processing Sys- tems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 8024-8035, 2019. URL htttp://paapee http://ppaaaaaapppppppp//ppp///s anttsorccha--impeeee anntsorrch----mmratiiiv style- whw-hiiihh-erffmaacce wtwp:/ootaaeeeeeeeeeeeeeeeeeeeee 1ibrary
C. Salge, C. Glackin, and D. Polani. Approximation of empowerment in the continuous domain. Advances in Complex Systems. 16(2-3), 2013. doi: 10.114220011955511550000 URL https
A. M. Saxe, L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. hRLhtt:: http:://atii..otttt////
ftos/131..66
J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In F. R. Bach and D. M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference r..peess/
Proceedings, pages 1889-1897. JMLR.org. 2015a. URL man 5.
J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. CoRR, abs/1506.024388, 2015b. URL http: //axiivorg//ass/155 abt.g//aaa/1550023...3
Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
algorithms. CoRR, abs/1707.063477, 2017. URL
A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics unsupervised discovery of skills. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ner, 2020. URL. httts::/ ew.ne// forum? (immmidd-ii H
Ö. Simsek and A. G. Barto. Skill characterization based on betweenness. In D. Koller, D. Schu- urmans, Y. Bengio, and L.. Bottou, editors, Advances in Neural Information Processing Sys- tems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Process- ing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008, pages 1597-1504. Curran Associates, Inc., 2008 URL
skill
S. P. Singh, A. G. Barto, and N. Chentanez. Intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems 17 (Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada), pages 1281-1288. 2004. URL httpp::/eaperr.nps..ccpppeeeee httpp//c/pppeeee........aaaaaaaa
reinforcement learning
R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
12

---

Under review as a conference paper at ICLR 2021
R. S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse cod- ing. In D. S. Touretzky, M. Mozer, and M. E. Hasselmo, editors, Advances in Neural Information Processing Systems 8, NIPS, Denver, CO. USA, November 27-30, 1995, pages 1030-10444 MIT Press, 1995. URL
rei htt.://wwn..t..taaa.....oooaeeaa
coding
R. S. Sutton, D. Precup, and S. P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 1111111):811211, 1999. doi: 10. 10100000004-- 3702(99)0005--1. URL http://///l...or//110..000 0000000000000000000 (99) 00052-1
T. Tieleman and G. Hinton. Lecture 3-5-RmsProp: recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Divide the gradient by running average of its
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2012, Vil- amoura, Algarve, Portugal. October 7-12, 2012, pages 5026 5033, IEEE, 2012. doi: 10.1109/ IROS 201...1861.6.,
C. M. Vigorito and A. Barto. Intrinsically motivated hierarchical skill learning in structured environ- ments. IEEE Trans. Auton. Ment. Dev. 2(2): 2010. doi: 10.1109TAMM.2R 200.00/AAD..20022502222222 2010.205025... URL
O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, Georgiev, Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden,
Y. Sulsky, Molloy, L. Paine. C. Gulcehre, Z. Wang. T. Pfaff, Wu, Ring. D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782) 354, 2019, //dot.org/ http:///111111111111111111111111
doi: 10. 10.100/41188--0199 1724-z. |RLhttps
R. Williams and J. Peng. Function optimization using connectionist reinforcement Iearning algorithms. Connection Science, 3:241-, 09 1991. doi: 10.1080009540009909996656
R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement
learning. Mach. Learn., 8:229-256, 1992. doi: 10. 10.1007//BF009922666 URL tttp::t:s////oi.orr
10.1007
Y. Wu, E. Mansimov, R. B. Grosse, Liao, and J. Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, Fergus, V. N. Vishwanathan, and Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5279-5288, 2017. URL o
approximation.
R. Zhao, Abbeel, and S. Tiomkin. Efficient online estimation of empowerment for reinforcement learning, CoRR, abs/aabs..o000 00356,, 2020. UKt.:ttp:////xxxvrrr//////00/000
13

---

Under review as a conference paper at ICLR 2021
Option
$-2.0 pw0.0
Option
Option
p-1.0 p-0.0
Option
1-1.
p-2.
Option
p-1.0
Option
1-12
g-0.0
p-0.
Option
Figure 7: InfoMax options the four state deterministic chain. Left: With three options. With four options.
Right:
A MORE ANALYSIS ON DETERMINISTIC CHAIN EXAMPLE
Figure 7 shows infomax options with three options and four options in the four state deterministic chain example. Among multiple solutions, we selected options with an absorbing state per option (i.e., (x) 1.0 for only one r),, which are partially the same as the right options in Figure 1 With four options, Pr:
0.25 for all z and xs, thus (X)))X, is the maximum. This example shows that we need sufficiently many options for maximizing the MI, otherwise an agent can be
caught in a small loop as we described in Section
B OMITTED PROOFS
B.1 PROOF OF PROPOSITION 1
First, we repeat the assumption 2 in Harutyunyan et al. (2119).
Assumption 1. The distribution d' (-lo) over the starting states of an option o under policy ş is independent of its termination condition 8
Note that this assumption does not strictly hold since c-Greedy option selection depends on So via Qo. However, since this dependency experiments.
so strong, we found that 8o reliably converged in our
Lemma 1. Assume that the distribution d' (-lo) over the starting state of an option o under policy independent with 3 Then the following equations hold.
Vo, (Xx = dz, ('oaala.) (x) log P(x(2.)+
+1 +1 (e) ((xy
22.0
25
(14)
Vo.H(X,(X..) = Σ (Ie. 0)
Σ (x) log (x(xx) + Σ
po (z (x)
Po po (x + +1]].
20.0
(15)
Sampling Zs, I Ty o from d and po
Vo, (X, Ex Vo, log P(x(x.) log P(x (ma))]]
(16)
V No H( 0) = E -Er
(2) log Po - log p (2 (20)(1r))
(17)
14

---

Under review as a conference paper at ICLR 2021
Proof of Lemmal 1
Proof. First, prove Equation (4). Let (x.) denote the probability distribution over x. under the policy or the marginal distribution of dl (z,)o)). and d' (I.: o) denote the joint distribution of Is and o. Then, we have:
Vag (X = V8 Σ (x Σ Plrg
21
0og Plzy (x,) log P(ry(as)) + P(xr)æx,) Vo, P(rg(ee))
Σ dl (x,)
P(æslæz)
Σ (xs)
Σ
Vag Pix (20) log ((x +1)
3
Σ
d (xs)
u(olo.))
Vo, (ar
(log ((xyyloa)) +1)
Apply theorem
Σ dld (x.)
ΣΣ
Σ
po - -P(cy (og +1)
35
Σ (x:s) Σ ((
) Vo (x) Σ (Tzz=x P (()g + +1)
Σ d' (I., Σ
po (z(2z) ( x log P(x)ox)) + +1 Σ
p (2)l log P(aylzz) + +))
2x,0
24
sample
sample
sample
Sampling 2s, I, Iy, o, we get (66).
Then prove Equation [5
Vo, H(x 0) = - 8 Σ d' (x 0) Σ
Pd (x) (x
Vog (x (xa)
Σ (i 0)
Σ Vog po (Ta log po (x)) + (x)
(x)
2010
Σ
d (xs, 0)
Σ
VopP" (((y(())))
(Iog
z5
Apply theorem
Σ
d (x.,
po
log (x +11
2,.0
21
Σ
d' (I,, 0)
Σ
Pd
(x x.
V (x)
Σ
log 1o (x +1
Σ (I,, 0)
Σ (x(xx) Vog (x) x
|og Pa (x(xx)) + Σ pa (xgltt (log ( +1)]
Z0.0
sample
sample
sample
Sampling 2,, Wy o. get 17
15

---

Under review as a conference paper at ICLR 2021
B.2 PROOF OF PROPOSITION 2
Proof. First, we have that:
Pa (2) (x)))
log Po (x) log P(rrylas) loog
P(æylas)
Pr(zz [I)
loog
P(r (188
Pr(Es, 2 Pr(x,)
=log
TT) (r(
Pr(zz, (.)0)
=log Pr(wa, (5) Pr(o)(t))
Pr(o| (5,) Pr(
log Pr( Pr(olr.))
p(o)
log u(olxa))
Using this equation, we can rewrite the equation
10 as:
Oo) Vo, (X, Xx) - Ho, O)
E E0
Vo, (x) log P(r|x,)) log P(æylar)) log Px( log P (2) (a
Vo, C (x) P'((((r)))) log P(x) (2o)))))) llog (x)((x log P(x: (z,
log p(opras))) log p(olors5
E2,,,,
Vo (x)
u(olz:))
u(ooor,)
Ex,,,,
Vo, (x)
(ogg(ol/rs (1o
-log
C IMPLEMENTATION DETAILS
C.I THE WHOLE ALGORITHM OF A2IMOC
Algorithm 1 shows a full description of A2IMOC. It follows the architecture of A2C (Mnih et al. 2016, Wu et al. 2017) and has multiple synchronous actors and a single learner. At each optimization step, we update म.. Qo. and 8o from online trajectories collected by actors. We update p(olæ.,, xr) for estimating the gradient (TT) and f for entropy regularization (13). To learn z and je stably, we maintain a replay buffer Bo that stores option- transitions, implemented by a LIFO queue. Note that using older o, Fs, f sampled from the buffer can introduce some bias in the learned p and ft, since they depend on the current π" and f However, we found that this is not harmful when capacity of the replay buffer is reasonably small.
We also add maximization the entropy of 3o to the loss function for preventing the termination probability saturating zero one. Then the full objective of 33 is written as:
log p(ll,, .) log p(olx, 2z) + en, (z)())))
where CH2 is weight of entropy bonus.
C.2 IMPLEMENTATION OF PPIMOC
For continuous control tasks, we introduce PPIMOC on top of PPO, with the following modifications A2IMOC.
16

---

Under review as a conference paper at ICLR 2021
Algorithm I Advantage-Actor InfoMax Option Critic (A2IMOC)
1: Given: Initial option-value Qo. option-policy лº, and termination function 3".
2: Let Bo be a replay buffer for storing option-transitions
3: for k 1.... do
4: for == 1, .V do
Collect experiences from environment
5: Sample termination variable b, from (x;)
6:
by then
7:
Store option transition F; Bo
8:
end
9:
Choose next option 0:+11 by e-Greedy Receive reward R; and state Ti+1, taking Tol (x;)
10:
end for
12:
for all z; in the trajectory do Train option-policy, option-values, and termination function
Update T7" (a:(a;; with PG via the UOAE advantage (12) and policy regularization (13)
14:
Update Qo(,,, o) via the optimistic TD error 12
15:
if o, has already terminated then
16:
Update 3 (x;;) via (T) and the maximization of cH₂ (I:())
17:
end end for
18:
19:
Train ç and fú by option-transitions sampled from B
20:
end for
Upgoing Option-Advantage Estimation for GAE To use upgoing option-advantage estima- tion (2) with Generalized Advantage Estimator (GAE) (Schulman et al. 2015b) common with PPO, introduce upgoing general option advantage estimation (UGOAE). Letting denote the TD error corresponding to the marginalized option-state--vlles. & = R. + (Vo Vo(x,),
write the GAE for marginalized policy T A o (TA)'8 8 where À is coefficient. Supposing that or terminates at the t + step and letting 8 denote the TD error corresponding to an option-state value 89 R2 + Q0((2411 Qo o), we formulate UGOAE by:
(7)) + max
(k < N)
A AGGAE
upgoing estimation
- + (rA) (RetN + *Uo((4N++) -2((24N00))
(otherwise).
(18)
The idea is the same as UOAE 12) and is optimistic about the advantage after option termination.
Clipped 3 Loss In our preliminary experiments, we found that performing multiple steps of optimization on the gradient (TT) led to destructively large updates and resulted in the saturation of 3 to zero or one. Hence, to perform PPO-style multiple updates on 30, we introduce a clipped loss for 3o:
V clip((3-(x) Co (I), -Eg, 8 (x) logp(o)=,, -lg)
(19)
where Eg a small coefficient, 8 old max(- - min(c, 2)). Clipping makes the gradient zero when 8o is sufficiently different than S and inhibits too large updates.
80 before the update, and clip(x,- -,,6)
D EXPERIMENTAL DETAILS
D.1 NETWORK ARCHITECTURE
Figure iillstratss the neural network architecture used in experiments. In Gridworld experiments, we used the same state encoder for all networks and we found that effective for diversifying x" auxiliary loss (Jaderberg al. 2017) However, in MuloCo experiments, found that sharing
17

---

Under review as a conference paper at ICLR 2021
8
π 0
Si-Sigood
p(o
x.)
p( zf)
IT'-Surtaux
TC-Sulma
f
sandard devistions
π
( xss)
IT:
Redignai
Stared PC
Figure 8: Neural Network architecture used for the Gridworld experiments (top) and the MuJoCo
tasks (bottom).
the state encoder can hurt the policy update because the magnitude of 9 loss is larger even if clipped loss (19) used. As a remedy for this, we used two encoders in MuJoCo experiments: one for л and Qo and the other is for 8 and P.
Gridworld experiments, we represented a state as an image and encoded by a convolutional layer with 16 filters of size 4 x with stride 1, followed by convolutional layer with 16 filters of size 2 x with stride 1, followed by a fully connected layer with 128 units. In MuJoCo experiments, we encode the state by two fully connected layers with 64 units. Tº parameterized with separated networks for standard derivations per option, similar to Schullman et al. (2015a). We used ReLU as an activator for all hidden layers and initialized networks by the orthogonal (Saxe et al.) 2014) initialization in all experiments. Unless otherwise noted, we used the default parameters in PyTorch (Paszke et al. 2019) 1.5.0.
Gaussian distribution
D.2 HYPERPARAMETERS
When evaluating agents, we used e-Greedy for selecting options with Copt and did not use deterministic evaluation (i.e., an agent samples actions from x²) all experiments. We show the algorithm-specific hyperparameters of A2IMOC in Table In Gridworld experiments, we used Our AOC implementation
0.1 for AOC. based on the released code and uses truncated N-step advantage. Other parameters of AOC and A2C are the same A2IMOC. We also show the hyperparamcters of PPIMOC Table 2 For PPOC, we used Cuem = 0.001 for the weight of the entropy H Hu(x)). Our PPOC implementation is based on the released code and uses N-step (not truncated) GAE for computing advantage. PPOC and PPO shares all other parameters with PPIMOC.
18

---

Under review as a conference paper at ICLR 2021
Description
Value
0.99
Optimizer
RmsProp (Tieleman and Hinton 2012)
RmsProp Learning Rate
2 10
Copt
0.8 - 0.2
Number of timesteps per rollout Number of actors
20
12
CH2
0.04
CH
0.01
CH3 (3" (z)
0.01
Gradient clipping
1.0
Capacity of Bo
480
Batch size for training P and j
240
Table 1: Hyperparameters of A2IMOC in Gridworld experiments.
Description
Value
0.99
Optimizer
Adam (Kingma and Ba 2015)
Adam Learning Rate
3 x 10
Adam
1 x 10-4
Eopt
0.4 -> 0.1 (Cartpole swingup), 0.2 - 0.1 (Otherwise)
Eg
0.05 (Lifelong Billiard).O.I (otherwise)
Number of timesteps per rollout
256
Number of actors
16
GAE parameter (Q)
0.95
Number of epochs for PPO
10
Minibatch size for PPO
1024
CH2
0.004
0.001
CH (3O(()
0.01
Gradient clipping
0.5
Number of monte carlo rollout to estimate H(T,)
20
Capacity of Bo
4096
Batch size for training p and jı
2048
Table 2: Hyperparameters of PPIMOC in MuJoCo and Classical Control experiments.
19

---

Under review as a conference paper at ICLR 2021
A2IMOC
9.6
8
Opton D
Option 1
Option 2
Option 3
0.0
AOC
100
B
-04
Optin
Oppon
Option 2
Option 3
OurAOc
04
Option
Option
Figure 9: Learned option-policies (T²) and termination functions (89) for each option in Gridworld Four Rooms after 5 x 10 steps. Arrows show the probablities of each action and heatmaps show probabilities each 3 First row: A2IMOC. Option tends to go down, option and 2 tend to go right, and option 3 tends to go up. Second row: AOC. All options tend to go down. Third row: Our AOC. Option 1, and 3 tend to go down and option tends to right.
D.3 ENVIRONMENTAL DETAILS
In the Gridworld experiment, an agent can select four actions: go up. go down, go left and go right. With the probability 0.1, the agent takes uniformly random action. If the agent reaches one of goals, receives +1.0 ...00l or +200 reward. Otherwise, an action penalty -0.002 given. The maximum episode length is 100.
MuJoCo Point environments are implemented based on "PointMaze" in rllab (Duan et al. 2016 with some modifications, mainly around collision detection. The maximum episode length is 1000. In Four Rooms task, an agent receives +0.5 or +1.0 reward when it reaches a goal. Otherwise, an action penalty -0.0001 is given. This reward structure the same in Billiard Task: agent receives a goal reward when the object ball reaches a goal, otherwise receives penalty.
D.4 EARLY
LEARNED GRIDWORLD FOUR ROOMS
Figure 9 shows early option-polices and termination probabilities in Gridworld Four Rooms experi- ment. We can see that A2IMOC learned the most diverse option-policies
D.5 QUALITATIVE ANALYSIS OF POINT FOUR ROOMS EXPERIMENT
Figure I0 shows the visualizations of learned option- polinns-occessseo and termination functions in MuJoCo Point Four Rooms, averaged over 100 uniformly sampled states per each position. Arrows show the expected moving directions, computed from rotations and option-policies. Terminating regions and option-policiaa learned with PPIMOC are diverse. For example, option tends to go down while option 2 tends to go right. In the sampled trajectory of PPIMOC, we can see that it mainly used option but occasionally switched to option 0 and option for reaching the goal, and switched to option around the goal. Contrary, for reaching the goal PPOC only used option that does not terminate in any region. Options learned by Our PPOC is almost the same: termination probability is high around the upper left corner and opticnspolicies direct downward.
20

---

Under review as a conference paper at ICLR 2021
Paicy
Optin:
Optin:
Opriss
b8
[(0)
64
Option
Option
PPOC
Expected valuee Pulcy
1.
Option:
Option:
6a
Option:
[(X)
Opsand
Optan
Optinn
OurPPOC
Expected Policy
Optun
Optin::
Oytion:
B(x)
044
0.
Option
Figure 10: Left: Learned aption-policits (T²" and termination functions (39) MuJoCo Point Four Rooms experiment. Arrows show the expected moving direction of the agent and heatmaps show probabilities of each f Right: Sampled trajectories of each method. First row: PPIMOC. Terminattion regions are clearly separated and option-polices Option 0 terminates at almost everywhere, while option2 and 3 does not terminate anywhere. Third row: Our PPOC. All options almost the same.
diverse. Second row: PPOC.
D.6 ABLATION STUDIES
We conducted ablation studies with three variants of A2IMOC/PPIMOC:
CH2 0: Do not use the policy regularization based MI (13).
N-step Advantange: Use N-step advantage N-step GAE instead of UOAE 12 UGOAE (18).
Truncated N-step Advantange: Compute advantage ignoring future rewards instead using UOAE or UGOAE.
Figure 11 shows all results in three tasks. We can see that UOAE is effective in all tasks, since both step advantage and truncated N-step advantage performed worse than UOAE. The policy regularization based on MI (13) is effective only in the Point Billiard lifelong learning task.
D.7 NUMBER OF OPTIONS
Figure 12 shows the performance IMOC with varying the number of options. Two options perfomed worse in all experiments and we need four more options to make of IMOC. However, when we increase the number options to six and eight, we don't see any peroformance improvement from four options, despite of our analysis that need sufficiently many options cover the state space Appendix A This would be an interesting problem for future works.
21

---

Under review as a conference paper at ICLR 2021
2,0
Cumullttive Rewards
Medification
0.5
dlu=00
0.0
Adv.
Truncated N-step Adv.
100000
200000
300000
400000
500000
Environment Steps
Cumulative Rewards
Cumulative Rewards
0.6
0.6
Modification
Modification
0.2
cHμmO
CHi=0 H-step
0.0
0.0
Truncated stop GAE
Reward Transition
Truncated N: ststep GAE
3.5
0.5
2.0
3.5
Steps
Steps
Figure 11: Ablation studies. Top: Peformance progression of A2IMOC in Gridworld Four Rooms.
Bottom: Peformance progression of PPIMOC in MuloCo Point Four Rooms (left) and MuloCo
Point Billiard (right).
attve Rewards
0.5
Num, Options
0.0
100000
200000 300000
400000 500000
Environment Steps
Num, Options
Cumulalt Curulaative Rewards
ative Rewards
0.6
0.6
Num, Options
0.2
0.0
0.0
Reward Transition
1.5
3.5
0.5
2.0
3.5
Steps
Steps
Figure 12: Ablation studies. Top: Peformance progression of A2IMOC in Gridworld Four Rooms.
Bottom: Peformance progression of PPIMOC in MuloCo Point Four Rooms (left) and MuloCo Point Billiard (right).
22


Reviews:
[{"review_id": "qx7MAr_G5Rb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a modification of the option-critic algorithm for hierarchical reinforcement learning. The proposed algorithm modifies how the termination conditions of the options are improved by experience. Specifically, the algorithm aims to maximize the mutual information between the options and their termination states. The authors develop an optimization scheme for achieving this objective and provide empirical results in a number of domains\n\nThe empirical results help understand the scalability of the approach but they are less useful in evaluating how useful the proposed algorithm is in general. The experimental domains used in the paper are not as diverse as one would wish to see. In addition, across domains, the general structure of the task is the same: there are multiple goal regions offering different amounts of reward; the challenge for the agent is to not get distracted by the lower offerings. So it would be informative to see the algorithm analysed in a more varied set of problems, including problems with a single goal region. Also informative would be to execute the algorithm with varying numbers of options and examine the effect. \n\nThe logic behind using uncertainty-aware option selection is not clear. Further discussion and exploration of it in the paper would be useful. \n\nBottom of page 4: “By adding the uncertainty bonus, this strategy encourages selecting lesser-known options.” Given  that $\\mu(o, s_t)$ is the probability of choosing option $o$ in state $s_t$, which is not directly related to how well known an option is, it is not clear to me how Equation 12 encourages selecting “lesser-known” options. \n\nBottom of page 5: “Thus, an agent is required to switch its behavior around a suboptimal goal (e.g., from “going down” to “going right”), which options can help.” This needs clarification. How do options help with this? And why is that not achievable with primitive actions?\n\nIn Figure 2, OurAOC lags behind AOC. This could be noted in the paper and the reasons can be discussed. \n\nFigures 2 and 6 are very useful.\n\nSection 2.3,  perfoms --> performs\nSection 2.3,  terminatio --> termination\n\nAuthor feedback: The behaviour and performance of the algorithm in single-goal environments should be part of the paper. The paper should show not only where the algorithm succeeds but also where it fails. I appreciated the author's efforts to add diversity to the domains evaluated but they do not go far enough to change my score. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "eOwPNTqTjB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper seeks to learn diverse options for exploration,\nprincipaling by combining two techniques: the termination gradient\ntheorem (Harutyunyan et al. 2019) and empowerment (in the spirit of\nVIC/DIAYN). This is an appealing idea, and the approaches to\nempowerment (such as for DIAYN) are a great fit for the limitations of\nthe Harutyunyan work.\n\nHowever, there were a number of issues I found with the paper which\ncurrently leaves me leaning more towards rejecting in the current\nform. Again, solid idea, but both the paper itself and the\nexperimental results (thus the method) need further work.\n\nComments/Questions:\n\n- Just above Prop 2: The use of the discriminator / classification\nmodel here should be attributed to related work.\n\n- Uncertainty-aware option selection: How is this like UCB? This is\nnot 'uncertainty', it is the stochasticity of the policy, so this\nactually just looks more like entropy regularization as it is\nfrequently used. Am I missing something?\n\n- Section 5, Gridworld:  Do episodes end as soon as the first non-zero\nreward is found? I don't think that this is a particularly good choice\nof domain for studying exploration.\n\n- \"A2IMOC performed the best and found the optimal goal in most trials.\"\nI don't see any evidence for either part of this statement. Judging by\nFigure 1, none of the methods are reliably achieving the 2.0 goal. And\njudging by the error bars, it is difficult to say that A2IMOC is\nbetter than AOC. I'd add that looking at Figure2, the policies learned\nby the different options for A2IMOC look to terminate in different\nstates, but appear to have essentially the same behavior policy. This\nlooks like one of the common failure cases of OC, where the options\neither learn the primitive actions (e.g. Figure 2, AOC) or all learn\nidentical policies (e.g. Figure 2, A2IMOC).\n\nAs with Figure 1, the learning curves in Figure 2 do not allow us to\ndraw any real conclusions about PPIMOC being an improvement over PPO.\nThough, Figure 3, does seem to show some significant improvement in\nthis non-stationary task setting. But this is perhaps the only\npositive empirical result, and thus is a lot to hang the paper on.\n\n- Appendix: epsilon-greedy ablation looks to be actually performing\nthe best, why not use this instead of the UAOS?\n\nPros:\n\nInteresting idea, very appealing intuitively\nSmall, but highly relevant, theoretical contributions\n\nCons:\n\nExperimental results are wholly unconvincing\nUncertainy-aware option selection needs to be reframed\nWriting could use some work, especially around putting this in the\nproper context of existing work.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OvFyNpef_y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review for \"Diverse Exploration via Infomax Options\" \n\nThank you for your feedback. I found that the analysis of the method in the revision is informative. However, the comparison with the baselines is still lacking, and the experiments are only performed in simple environments. For these reasons, I keep my rating unchanged after the rebuttal.\n\n=============\n\nSummary:\nThis paper studies the problem of discovering options for exploration in reinforcement learning. To this end, they propose infomax criteria that maximize the information between options and option-terminating states conditioned on the option-starting states. The paper derives update rules for the infomax criteria and applies the gradient descent algorithm to optimize options. The proposed method is evaluated at Gridworld/Mujoco four rooms tasks and Point Billiard environment, and the authors show that the resulting options are diverse.\n\n\nPros:\n- The information maximization criteria for diverse option-terminating states is novel.\n- The paper does a nice job describing the required background materials in great detail. \n- The paper clearly describes the objective and the corresponding update rule with implementation details. \n\n\nCons:\n- The experimental results are too weak to prove the effectiveness of the proposed method. In detail, the paper's experiment tasks are too simple, e.g., all tasks' goals are to reach one of the four points in a simple square room. Moreover, from Figure 3-(b), it doesn't seem that the proposed method (PPIMOC) outperforms the baselines. \n- The performance comparison with important baselines such as termination critic (TC) is absent in the experiments. \n- The paper suggests some techniques such as uncertainty-aware option selection or upgoing option-advantage estimation, but the paper lacks ablation studies.\n- It is not clear that maximizing the mutual information in Equation (7) leads to diverse options. The paper should provide theoretical results or more experimental analysis to make the case.\n\n\nComments:\n- Experiments on more challenging tasks, such as Montezuma's revenge from Atari games, could improve the claim of the paper. \n- I recommend performing an ablation study of the proposed algorithms (in Section 4) to isolate the effect of each technique.\n- I am not fully convinced that information maximization leads to diverse options. Aren't there any problems associated with the collapse of options or terminating states? \n\n(minor)\nThere are some typos.\n- End of Section 2.1: \"with and o and not\"\n- Equation (4) and the following sentence: Q_O(x) -> Q_O(x,o)\n- Last paragraph of Section 2.3: \"Contrary to the terminatio\"\n\n\nIn summary, the paper suggests a new perspective on diverse options via infomax criteria, but the experimental results are too weak, and the analysis of the method is lacking. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "L2N6h-kBP1G", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper that investigates the use of options for improved exploration by encouraging diverse termination functions via a mutual information measure. The paper is well written and mostly clear (with a few points below):\n\nOverall I think the idea has merit, but I think the experiments fall short of demonstrating its performance. What seems to be lacking to me is:\n1. Demonstrating performance with stochastic transitions (I believe all experiments are with deterministic transitions).\n2. Overcoming the coarse approximation for the infomax objective (7) (see point 5 below).\n3. Clarifying the details of the classification model, which seems central to the algorithm (see point 2 below).\n\nSome questions for the authros:\n1. What is the law of $X_S$ in equation (7)? In particular, if $X_f|X_s$ is the random variable denoting a state transition, wouldn't this just be the stationary distribution of the current policy? Further, in Proposition 1, the first state $x_s$ isn't sampled (at least not by $\\pi^o$ nor $\\beta^o$), so it's not clear what the random variable $X_S$ represents.\n2. Right before section 4, the authors write \"learning a classification model over options $\\hat{p}(o|x_s, x_f)$ from sampled option transitions\". This seems central to the algorithmic performance, but it's not clear how it gets used.\n3. In equation (12), in the condition for $b_t = 1$, it seems it's saying the same option ($o_t$) should be picked again. Is this the case?\n4. Above equation (13), the authors write \"Supposing that the current option ot terminates at the $t + k$ step\". How is this calculated? This seems especially problematic with stochastic transitions.\n5. In page 5 the authors write \"[$I(A; O | X_S)$] can be interpreted as a local approximation of the infomax objective (7), assuming that each action leads to different terminating regions.\" This seems like a rather rough approximation; in particular, doesn't this simply encourage diversity of one-step action transitions as opposed to option termination?\n6. In Figure 1b the difference between A2IMOC and A2C does not appear to be statistically significant (there is major overlap of the confidence regions). Does this not suggest merely marginal gains?\n7. In Figure 2, were all algorithms using 4 options?\n8. Were any of the environments evaluated with stochastic transitions?\n9. How many options were used in the MuJoCo environments?\n10. What do the \"sample\" indicators mean in your proof of Lemma 1? This suggests the proof is mostly a modification of the proof in Harutyunyan et al. (2019).\n11. Do you have a similar plot as Figure 8, but for the MuJoCo experiments?\n\nMinor issues:\n1. Given that you're dealing in episodic tasks, below equation (1), shouldn't the $Q^{\\pi}$ values be indexed by timestep?\n2. In the definition of $V_{\\mathcal{O}}(x)$, it should be indexed by $\\mu$.\n3. Right above section 2.2, remove the extraneous \"and\".\n4. In equation (4), $Q_{\\mathcal{O}}$ is missing the second parameter.\n5. You should state equation (6) as a proper theorem.\n6. Equation (10) is over the margins.\n7. In the first sentence of the paragraph above equation (14), remove the extraneous \"common\"\n8. In the section titled **MuJoCo Point Four Rooms** there is a typo, it says \"Figure 3b\" but it should say \"Figure 3a\".\n9. In Related Work, under **Options for Exploration** you could also cite some related work:\n  - \"Using bisimulation for policy transfer in MDPs\", Castro & Precup, AAAI 2010\n  - \"Automatic construction of temporally extended actions for mdps using bisimulation metrics\", Castro & Precup, EWRL, 2011\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a variant to the option-critic framework that encourages options to display a certain level of "diversity" and this is induced by introducing a mutual-information objective between the options and their transitions. The authors conjecture that such criterion makes options more suitable for exploration.

Overall, reviewers agree that the idea behind the proposed method and the general approach is sound and interesting. Nonetheless, there is general consensus that the current submission suffers from a number of shortcomings that make it unsuitable for acceptance.

Following the detailed comments provided by the reviewers, I strongly encourage the authors to focus on the following dimensions to improve the paper:
1- The current experiments indeed provide a first illustration of how the proposed algorithm works, but they need significant improvement in variety and scope: As pointed out by the reviewers, the current experiments do not cover single-reward challenging exploration benchmarks (such as Montezuma). I agree with the authors that the inductive bias implemented in their algorithm is designed with diversity of goals in mind, but if the main point is to improve exploration, it is natural to expect results in that respect. Alternatively, the authors should state more explicitly the type of problems their method is intended to solve from the very beginning of the paper and design experiments accordingly.
2- The initial mutual information objective is simplified across multiple steps and it is unclear how much the approximations impact the original "semantic" of the objective.
3- A more thorough comparison with mutual-information-based methods such as DIAYN or VIC is needed. Also, I wonder what is the connection with more goal-based exploration approaches such as GoExplore or SkewFit.
Decision: Reject