Learning Diverse Options via InfoMax Termination Critic
Authors: Yuji Kanagawa
Tomoyuki Kaneko
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
hierachical reinforcement learning
options
life long reinforcement learning
skill transfer

Abstract: We consider the problem of autonomously learning reusable temporally extended actions, or options, in reinforcement learning. While options can speed up transfer learning by serving as reusable building blocks, learning reusable options for unknown task distribution remains challenging. Motivated by the recent success of mutual information (MI) based skill learning, we hypothesize that more diverse options are more reusable. To this end, we propose a method for learning termination conditions of options by maximizing MI between options and corresponding state transitions. We derive a scalable approximation of this MI maximization via gradient ascent, yielding the InfoMax Termination Critic (IMTC) algorithm. Our experiments demonstrate that IMTC significantly improves the diversity of learned options without rewards, combined with an intrinsic option learning method. Moreover, we test the reusability of learned options by transferring options into various tasks, confirming that IMTC helps quick adaptation, especially in complex domains where an agent needs to manipulate objects.

Under review as a conference paper at ICLR 2022
LEARNING DIVERSE OPTIONS VIA INFOMAX TERMINATION CRITIC
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of autonomously learning reusable temporally extended actions, or options, in reinforcement learning. While options can speed up transfer learning by serving as reusable building blocks, learning reusable options for un- known task distribution remains challenging. Motivated by the recent success of mutual information (MI) based skill learning, we hypothesize that more diverse options are more reusable. To this end, we propose a method for learning termina- tion conditions of options by maximizing MI between options and corresponding state transitions. We derive scalable approximation of this MI maximization via gradient ascent, yielding the InfoMax Termination Critic (IMTC) algorithm. Our experiments demonstrate that IMTC significantly improves the diversity of learned options without extrinsic rewards, combined with intrinsic rewards. Moreover, we test the reusability of learned options by transferring options into various tasks, confirming that IMTC helps quick adaptation, especially in complex domains where an agent needs to manipulate objects.
INTRODUCTION
Behavior learning from environmental interaction is a fundamental problem in artificial intelligence and roboties. Recently, combined with deep neural networks (DNN), reinforcement learning (RL) has successfully learned complex behaviors guided by human human-designe designed reward functions (Heess et al., 2017: OpenAI et al., 2019). To reuse trained agents across multiple reward functions, or shortly tasks, abstracting course of action as a higher-level building block (Barto and Mahadevan, 2003; Barto et al., 2013) would be useful. representative formulation of action abstraction in RI. the options framework (Sutton et al., 1999), where each option consists of a sub policy and its termination condition. Serving as reusable behavioral blocks, options have the potential of accelerating learning in new tasks (Brunskill and Li, 2014). For example, if wheeled robot already had learned to lean and turn left and right, it can learn to drive on unfamiliar roads faster than learning from scratch. However, discovering reusable options remains challenging due to the difficulty of defining and measuring reusability a priori. Although we can measure the reusability of options when we know the task distribution (e.g., in Bayesian sense (Solway et al., 2014)), it is difficult measure it without prior knowledge. Hence, in such cases, we need a reasonable heuristic assumption about future tasks.
In this paper, we follow common assumption that diversifying options improves the reusability of options (e.g., Barto et al. (2004)). That is, we assume that tasks are uniformly distributed over the state space. Diverse options are expected work to some extent for any task, thus we argue that this is reasonable heuristics. Under this assumption, the next problem is to measure the diversity of options. The most intuitive strategy would be visiting all states as equally as possible. However, this can be computationally too expensive when the environment has continuous or large state spaces. As an alternative, in the context of learning skills without termination conditions, mutual information (MI) maximization has been widely applied for learning diverse policies (Eysenbach et al., 2019; Baumli et al., 2021; Choi et al., 2021). An advantage of MI maximization is the existence tractable approximations with DNN, scaling it up to complex domains.
For learning diverse options, we maximize MI between options and their terminating states con- ditioned by starting state. This MI maximization diversifies the destinations of agent when choosing different options at a state while keeping the transition as deterministic as possible, leading to diverse and meaningful options. Specifically, propose to maximize this MI by optimizing

---

Under review as a conference paper at ICLR 2022
IMTC VIC
Expected volue Pulcy
60
66
OC VIC
B00]
Orten
Figure 1: Left: PointBillia aettiiilllrg environment. Right: Learned options by IMTC (upper row) and OC (lower row). Arrows show policies, and heatmaps show termination conditions of options.
the termination conditions of options. We derive an unbiased gradient estimator this MI w.r.t. termination conditions using the termination gradient theorem (Harutyunyan et al., 2019). As key contribution, we reformulate the estimated gradient using Bayes' rule and derive a tractable approximation with leaned classifier, yielding the InfoMax Termination Critic (IMTC) algorithm. The easiness of approximation is advantageous compared to the Termination Critic (TC, Harutyunyan al. (2019)). The same objective is used by Variational Intrinsic Control (VIC, Gregor al. (2016)) for diversifying sub-policies, but the original VIC used a constant termination probability for all states and options. However, since VIC depends on relating starting and terminating states to an option, we argue that diversifying termination conditions is also important. We show that our method helps VIC learn clearer options.
In this paper, we first introduce backgrounds and notations, followed by the derivation of IMTC. We implement IMTC on the Option Critic Architecture (OC, Bacon al. (2017)) and Proximal Policy Optimization (PPO, Schulman et al. (2017)). As a minor contribution, our implementation includes newly proposed optimistic advantage estimation that speeds up option learning for single task. In experiments, we qualitatively demonstrate that IMTC successfully learns diverse and meaningful options in a reward-free -ree RL (Jin et al., 2020) setting. Figure shows the learned options by IMTC and OC combined with VIC rewards in PointBillia domain. We can that options learned by IMTC are clearly separated and different. Then we test the reusability of learned options in task adaptation experiments:
first pre-train options without rewards and then transfer them various tasks. We show that IMTC helps quick adaptation to specific tasks especially in complex tasks where object manipulation is required. Since IMTC only for learning terminating conditions of options, we can combine it with other methods for learning diverse sub-policies. Our experiments observed that IMTC improves the performance of VIC and RVIC (Baumli al., 2021).
2 BACKGROUND AND NOTATION
We assume the standard RL setting in the Markov decision process (MDP). following Sutton and Barto (2018). MDP M consists of tuple (X, p, p)r), where X' is the finite set of states, A is the finite set of actions, p X' x A x X - [0, 1] is the state transition function, A + ['min: F'ax] is the reward function, where Tmin: Tmax are minimum and maximum. We use X, and A, for denoting random variables of the state and action experienced time t. We define Ghdd del Ee RR, as discounted cumulative return, where R2 = r(x,, A,) is the reward received at time and 0 < <> < is the discount factor. We consider maximizing expected discounted return E [ T). As an agent, we consider a memoryless policy π : X' x A - [0, 1]. क induces two value functions: action-value function Q a) def E, [G x, Ao a, m and state-value function (=(x)) E. (2a)))Q"(z,a) a). We use d' for denoting an agent's 's average occupancy measure d= (x: lim [x where I the indicator function.
X' x
n-000
Assuming that is differentiable by the policy parameters 0,, the policy gradient (PG) method (Williams, 1992) maximizes G by updating Qx via gradient ascent. common formulation of PG estimates the gradient by Vo_ T Ex,,, Voo log log ((ag(()axAA, a)). where Á(x, a) is an estimation of the advantage function A (2:, a) def Qa (xx a) V (x)) Among many variants of PG
2

---

Under review as a conference paper at ICLR 2022
methods, we implemented our method PPO (Schulman al., 2017). At each gradient step, PPO ((aa))
updates 0x to maximize clip( [ A, e), where clip(x,- -c, e) max(-e, min(e, and Told the policy before being updated. This clipping heuristics prevents я from updating too rapidly.
Options Framework Options (Sutton et al., 1999) provide a framework for formulating temporally
abstracted actions in RI. An option E O consists of tuple ,0,,,')), =²), where Jo C X' is the initiation 3 + [0, 1] is a termination function, and π is an intra-option policy. Following related studies (Bacon et al., 2017; Harutyunyan et al., 2019), we assume that J A' focuses on learning 3 and σ., We let μ X O - [0, 1] denote a policy over options. A typical RL agent sample A, from x--]_x). Analogously, at time t, an RL agent with options first samples termination T. from go. (X,) T; = 1, O++1 is sampled from (u(((x,) and if not, the current option remains the same. The next action A, is sampled from 70, = (-|Xx)). For option learning methods, we to denote the resulting policy induced by jr, UsEOT" and U.
Option value functions With options, we have three option-value functions Qo. Vo, and Uo. Qo is the option-value function denoting the value of selecting option at state defined by Qo(2,
E X x, Os = o]. Similar to the relationship between Q" and V", we let Vo denote the marginalized option-value function (o(x) def u(olz)Qoo (2, o). Uo(a, o) der
(1 31-38( Qo)) o) + 3 (x) (x)) is called the option-value function upon arrival (Sutton et al., 1999) and denotes the value of reaching a state æ with o and not having selected the new option. We use these notations in Section 4.2.
Termination gradient theorem state transition probability induced by options. When an agent is at xs and having an option o, the probability that ends at f is given by:
Analogously to p. we let P X' x + [0, 1] denote the
(T(x)(=s) (Iy + (0 Σ (x/o::)
(1)
where p"" is the policy-induced transition function p" (x' (o'(a) def Enes T (alz)) a). Here
we assume that all options eventually terminate, such that pe is a valid probability distribution over Ty. Interestingly, po is differentiable with respect the parameter of 30. Harutyunyan et al. (2019)
introduced the termination gradient theorem:
Theorem 1. Let 8 be parameterized by 0z. and let Eg- denote the logit of 80, iee Ego =
(x) We have log( (x)
=
(())))))
(2)
We use this theorem derive an unbiased estimator of our target gradient.
INFOMAX TERMINATION CRITIC
We now present the InfoMax Termination Critic (IMTC) algorithm. To lear diverse options, propose to maximize the following MI at each state Fg:
T(X = ((x) H (.,, O) = H x.),
(3)
where I denotes the MI Blc) = (( - H(A c). H denotes entropy, and O is a random variable denoting options experienced by an agent. Precisely, let v) denote the probability of having an option o when leaving a state x: 77(0)z)) = ((())((olr) + (1 (x) Exex p" (ar(²)) (n(o)=") and let 17" denote 77 marginalized over dz: (0) ErEx de (z))((((zz Then, we define O as a ran- dom variable with r7a By decomposing this MI as 1(x; = H())(x,) - (x,, O). we can interprete this MI maximization (XX, O). Maximizing (x)) diversifies possible destinations of an agent. Thus, the resulting options expected to be diverse in the sense that they likely lead to different destinations.
maximization of ((x)l=,) and minimization of
On the other hand, minimizing H H(X,lea 12,, O) makes option state transitions more deterministic, leading to more meaningful options.
3

---

Under review as a conference paper at ICLR 2022
Options with high (xT: (IMTC)
Options with high -H(X]Jo) (TC)
B(x)
B(x)
Option 0
Option
Option
Option
Option
Option
Figure 2: IMTC options (left) and TC options (right).
To build intuition, we searched options that maximize the MI (3) by a simple brute-force search from limited of TO and 8 in a Gridworld environment. Specifically, we considered only deterministic intra-option polciies and 0.1 or 0.8 as the value of 3 (z). The number of options fixed |O|
In this environment, an agent can take four actions: Up, Down, Left, and Right. The taken action fails with probability 0.1, and it takes uniformly random action. Figure 2 shows the result. Arrows represent intra-option policies and colors of the room represent termination probabilities. Here, can see the tendency that IMTC prefers options that have almost separated termination regions and different policies per option. We also visualized TC (Harutyunyan et al., 2019) options that maximize - H (Xy (o). For TC options, options, but options O and of TC share the intra-option policies on the left side of the room. Thanks the diversity term H (X3 f a. IMTC successfully avoids such a policy overlap. In addition, see that IMTC has smaller termination regions than TC, which can enhance long-range abstraction.
see similar tendency as IMTC
We propose maximize the MI (3) by updating 3 via gradient ascent w.r.t. 0g. For this pur- pose, we now derive an unbiased estimator of the gradient. First, we write the gradient of the MI using the option transition model po and marginalized aption-transition model p(zylz,) E. (o) (x{](zx))
Proposition 1. Let 8 be parameterized using a sigmoid function. Given a trajectory T Ta:..., L, 2y sampled by To and 80, we can obtain unbiased estimations of V (X (x.)
and Vo H (x 0) by
Va, HAHHH(,laa = Eatr. Vo, Togeao (x) (x) log P(alats) - log P(arl(ts))
(4)
Vo, HogH((X 0) Eg
Vos (x) 3x) log P (x|xs) log P (z (x2)(xx))) (
(5)
where Ege (x) denotes the logit of (x)).
Note that the additional term 8 is necessary because æ is not actually a terminating state. The proof is based on Section 4 in Harutyunyan et al. (2019) and is given in Appendix A.1.
Now the targetting gradient can be written as:
Vozi(X O2z,) Voz H (x,(=))) Ho O)
= E. [- ( (x) (log P(x(x=) log P(z (xa) - (log (x|xx) - log Pa Pa [)))
which means that we can approximate MI maximization by estimating p and P. However, in RL, learning density model over the state space in RL is often difficult, especially with large or continuous state spaces. For example, it has been tackled with data compression methods (Bellemare al., 2016) and generative models (Ostrovski al., 2017). Hence, reformulate the gradient using Bayes' rule to avoid estimating pa and P. The resulting term consists of the inverse option transition function (o(x,.,11)) which denotes the probability of having an given state transition I... 2/.
Lemma 1. We now have
1o,1(XX; Vo, (o.l(()) Vo, (X
=E mn (o logpo x) log
The proof is provided in Appendix A.2. Equation (7) requires estimating of po per updating x² and 88 which is computational quite expensive. Thus, approximate approximate the gradient (7) by regressing a classification model over options po(o) x on sampled option transition.
4

---

Under review as a conference paper at ICLR 2022
4 IMPLEMENTATION
Since IMTC can be combined with any on-policy RL methods, we choose PPO (Schulman et al., 2017) as base algorithm because of its stability and ease of implementation. As notable implementation details, this section explains the estimation of po and advantage estimation. We provide further details and the full description of the algorithm in Appendix B.
4.1 ESTIMATING po
To estimate Po. employ classification model over options po(o and regress it on sampled option transitions, as per Gregor et al. (2016). However, our preliminary experiments observed that this online regression could be unstable because the supply of transition data depends on the termination probability and can drastically increase or decrease during training. To address this problem, we maintain a replay buffer Bo. which stores option state transitions {(o, (f)) to stabilize the regression of Po. Note that using older option state transitions can introduce bias to po because depends on the current policy. However, we found that this is not harmful when the capacity of the replay buffer is reasonably small.
4.2 ADVANTAGE ESTIMATION
The original PPO implementation employs GAE (Schulman et al., 2015b) for estimating the advantage. which is important for learning performance (Andrychowicz et al., 2021). Therefore, we employed two variants of GAE in experiments for option learning according to the experimental setup. In the following paragraphs, we let N denote the rollout length used for advantage estimation and let t + & denote the time step at which the current option o, terminates. Thus, need to consider the effect of sption-switching in advantage estimation When < N. Furthermore, we use two variants of the option-specific TD errors o(.)) Rg (Q( Qo( oz) and dv(o)) R, + (1++1, o.) - Qo ( 0t).
Independent GAE for Reward-Free RI. For no-reward experiments with VIC, we used the following variant of GAE:
min(k.N)
A Qu 01) +
Σ (7A)
(8)
Here, we ignore the future rewards produced by other options after the current option terminates. This formulation enhances learning diverse intra-option policies per option.
Upgoing GAE for task adaptation For single-task learning, increasing the rollout step N often speeds up learning (Sutton and Barto, 2018). However, future rewards after option termination heavily depend on the selected option and have high variance, especially when learning diverse options. This high variance of future rewards slows advantage learning and causes underestimation of Á Thus, to prevent underestimation, we introduce an upgoing GAE (UGAE) for estimating advantage with options:
(T)(())) + max
Σ
(1880(0444))))) (k < N)
Agg Qo( 00) +
(72)&881 + upgoing estimation
(otherwise).
Like the upgoing policy update (Vinyals et al., 2019), the idea optimistic regarding future rewards after option termination by taking the maximum over 0. We use A for task adaptation experiments in Section 5, and confirmed its effectivity in the ablation study in Appendix C.6.
EXPERIMENTS
As presented in this section, conducted two series of experiments analyze the property of IMTC. First, we qualitatively evaluated the diversity of options learned by IMTC with intrinsic rewards, without any extrinsic rewards. Second, quantitatively test the reusability of learned options by
5

---

Under review as a conference paper at ICLR 2022
22
A
pixt
IMTC + RVIC
Ceseced whk VIC 18-0.11
(gran IMTC. (R-00.01)
Tohoo (S-0.11
Mxl
Figure 3: Learned intra-option policies (T*) and termination probabilities (39) for each option in PointMaze after training 4 x 10 steps. Arrows show the expected value of action, and heatmaps show probabilities of each 3 Left: Options learning by IMTC with three different intrinsic rewards. We can see that intra-option policies have various directions, and termination regions clearly separated across options. To see the difference between intrinsic rewards, RVIC and constant rewards have an overlapped pair of options (option 1 and 3). Also, policies tends larger with constant rewards. Right: Options learned by other methods. OC produces dead option 3 that terminates everywhere and never-ending options 0 and 1. About intra-option policies, all methods successfully avoided learning the same policy, but they only have two directions.
that the magnitude of intra-option
task adaptation on a specific task. As a baseline of termination learning method, we compared our method with OC (Bacon et al., 2017). OC is trained with VIC (Gregor et al., 2016) rewards during pre-training. We did not compare IMTC with TC (Harutyunyan et al., 2019) because our TC implementation failed to learn options with relatively small termination regions as reported in the paper, and there is no official public code for TC'. During pre-training without extrinsic rewards, IMTC receives intrinsic rewards when the current option terminates. We compare three IMTC variants with different intrinsic rewards: (i) VIC (Gregor et al., 2016), (ii) RVIC (Baumli et al., 2021), and (iii) constant value (RIMre = 0.01). Note that RMTC = 0.01 is chosen from 10.1, 0.05, 0.01] based on the task adaptation results. We also compare IMTC with vanilla VIC and RVIC with fixed termination probabilities. We used Vz 8z = 0.1 since it performed the best in task adaptation experiments, while 0.05 was used in Gregor et al. (2016). Note that RVIC's objective I (X,; Olzy) is different from ours, while IMTC and VIC share almost the same objective. Thus, the use of VIC is more natural, and the combination with RVIC is tested to show the applicability of IMTC. Further details of our VIC and RVIC implementation are found in Appendix B. In order to check only the effect of the different methods for learning beta 80, the rest of the implementation is the same for all these methods. That is, OC, vanilla VIC, and vanilla RVIC are also based on PPO and advantage estimation methods in Section 4.2. In this section, we fix the number of options as |O| = 4 for all option-learning methods. We further investigated the effect of the number of options Appendix C, where we confirmed that |O] = 4 is sufficient for most domains. All environments that we used for experiments are implemented on the MuJoCo (Todorov et al., 2012) physics simulator. We further describe the details in Appendix C.
Option Learning From Intrinsic Rewards We now qualitatively compare the options learned by IMTC with options of other methods. Learned options depend on the reward structure in the environment, which enables manually designing good reward functions for learning diverse options. Thus, we employed reward-free RL setting where no reward is given to agents. Instead, each compared method uses some intrinsic rewards, as explained. We fix μ as p(olz)) = jo| in this experiment, since we assume that the future tasks are uniformly distributed. Intra-option policies are trained by PPO (Schulman et al., 2017 107) and independent GAE (8). We show network architectures and hyperparameters in Appendix C. We set the episode length to 1 x 104, i.c. an agent is reset to its starting position after 1 x 10s steps. For all visualizations, chose the best one from five independent runs with different random seeds.
Note that we also could not find any unofficial open source implementation.

---

Under review as a conference paper at ICLR 2022
+
(a) PointReach
(b) SwimmerReach rReach
(c) AntReach
(d) PointBilliard
Goal marker
Other possible goal positions
(e) AntBilliard Figure 4: Domains used task adaptation experiments.
(f) PointPush
(g) AntPush
We visualized learned options in PointReach environment shown in Figure 4a. In this environment, an agent controls the ball initially placed at the center of the room. The state space consists of positions yx, and velocities (Ax, Ay) of an agent, and the action space consists of acceralations ( At At ). Figure 3 shows the options learned in this environment after 4 x 10t steps. Each arrow represents the mean value of intra-option policies, and the heatmaps represent 33, In this experiment, we observed the effect of IMTC clearly, for both termination regions and intra-option policies. Interestingly, don't see clear differences between options Icarned with VIC and RVIC rewards, while constant rewards tend make options peaker. OC failed to learn meaningful termination regions: option 0 and never terminate, and option 3 terminates almost everywhere. This result confirms that IMTC can certainly diversify options. Moreover, compared to vanilla VIC and RVIC, intra-option policies learned by IMTC with VIC or RVIC rewards are clearer, in terms of both the magnitude and directions of policies. We believe that this because diversifying termination regions gives more biased samples to the option classifiers employed by VIC and RVIC. Figure also show options learned by IMTC and OC in PointBil lliard domain, where we can see the same tendency.
[Ax 4y
Transferring skills via task adaptation Now we quantitatively test the reusability of learned options by task adaptation with specific reward functions. Specifically, we first trained agents with intrinsic rewards as per the previous section. Then we transferred agents to environment with the same state and action space but with external rewards. We prepared multiple reward functions, which call tasks, for each domain and evaluated the averaged performance with OC, vamilla VIC, vanilla RVIC, and PPO without pre-training. Also, we compare three variants of IMTC with different intrinsic rewards during pre-training. For fair comparison, UGAE (9) and PPO are used for all options learning methods. Note that we found UGAE is very effective this experiments, we show the ablation study in Appendix C.6. For vanilla VIC and vanilla RVIC, termination probability is fixed to 0.1 through pre-training and task adaptation. e-greedy based on Qo with e 0.1 is used the option selection policy μr. We hypothesize that diverse options learned by IMTC can help quickly adapt to given tasks, supposing the diversity of tasks.
tasks. We compare IMTC
Figure shows all domains used for task adaptation experiments. For simplicity, all tasks have goal- based sparse reward functions. Le.,, an agent receives R, = 1.0 when it satisfies goal condition, and otherwise the control cost -0.0001 is given. Red circles show possible goal locations for each task. When the agent fails to reach the goal after 1000 steps, is reset to starting position. Point Ent.eaacc ch, Swimmer CemmerReache ch, and AntReach are simple navigation tasks where an agent aim to just navigate itself to the goal. We also prepared tasks with object manipulation: in PointBil lilllridd and AntBilliard an agent aims to kick the blue ball to the goal position, and in PointPush and AntPush, it has to push the block out of the way the goal. We pre-traine options learning agents for x 10 environmental steps and additionally trained them for x 10" steps for each task. Figure 5 shows learning curves and scatter plots drawn from five independent with
7

---

Under review as a conference paper at ICLR 2022
Partheach
Sultmmerhaach
spphaadh
Additt Bars
64 C4
Arrwic
0.5
Machod
IMCC
03
(t=00.11
oc a Vic
VIc (p-0
RVIC (5-0.))
Cheroweer 100)
Dnvisiners tif
PPO
Figure 5: Learning curves for transfer learning experiments.
different random seeds per domain. Here, we observed that IMTC with VIC or RVIC rewards performed the best was compatible with baselines. IMTC with VIC performed better than OC with VIC except for Ant rch, which backs up the effectiveness of diversifying termination regions for learning reusable options. Also, IMTC with VIC and IMTC with RCIC respectively performed better in most of the tasks than VIC and RVIC with fixed termination probabilities. This result suggests that IMTC can boost the performance of option learning methods based on option classifiers, even when the objective is different as with RVIC. On the other hand, IMTC with constant rewards (RIMC = 0.01) performed worse than IMTC with VIC or RVIC rewards, although it also learned diverse options as we show in Figure 3, suggesting the importance of adjusting rewards. We further analyzed the evolution of intrinsic rewards of VIC and RVIC in Appendix C.5. In addition, we can observe that IMTC's performance is especially better than other methods in relatively complex Point Bi iiliard, AntBilliazin and Ant Push, where object manipulation is required. Considering that manipulated balls and boxes move faster than agents in these domains, choice of options can lead to larger differences in the future state. IMTC suitable to these domains since it maximizes the diversity of the resulting states, while PPO struggles to learn. Contrary, IMTC's performance is close to other methods in Reach tasks, where the goal states are relatively close to the starting states terms of euclidian distances.
Gridworld experiments and limitation of the method Although IMTC successfully learned diverse options in MuJoCo experiments, our analysis in Figure 2 shows the possibility learning options that are not interesting but have large MI. We further investigated this possibility by visualizing options in a classical four rooms gridworld in Appendix C.8. Interestingly, we observed that IMTC could fall into diverse but unmeaningful options in that environment. We believe that IMTC is often sufficient in a large environment where a randomly-inittaaized agent rarely gets the same trajectory. However, when the number of possible trajectories is small, diversifying the destinations could be insufficient. In such cases, it can be necessary to extend IMTC to diversify trajectories as done in Sharma et al. (2020).
RELATED WORK
Discovering diverse options Since Sutton et al. (1999) first formulated the options, discovering good options has been challenging. classic concept for the goodness of options is a bottleneck region (MeGovern and Barto, 2001), which refers to important states for reaching diverse areas, such as a passage between two rooms. Once an important region is discovered, we can construct an option that guides agent to that area. Various approaches have been proposed define bottleneck regions concisely and compute bottleneck options efficiently, including betweenness (Simsek and Barto, 2008), Eigen options (Machado al., 2017), successor options (Ramesh al., 2019), and covering options (Jinnai et al., 2019). Barto et al. (2013) discussed the importance of diverse options for exploration, referring to an option construction method with a graph- decomposition of an MDP Vigorito and Barto (2010). We share the same motivation with these methods but have
We used seaborn (Waskom, 2021)'s 1 1p1 at. with order-3 is used draw learning curves.
8

---

Under review as a conference paper at ICLR 2022
different approach. While these methods construct point options that bridge two states, we capture set of states by learning 3 directly, making it easy to scale up with function approximation. To our knowledge, (Jinnai al., 2020) only succeeded in scaling up point options to continuous options using approximated computation of the Laplacian.
End-to-end learning of options As described in the previous paragraph, many studies have at- tempted to construct options and then train intra-option policies. However, motivated by the recent success of RL with DNN, Bacon et al. (2017) proposed OC train intra-option policies and termi- nation functions in parallel using a neural network as a function approximator. OC updates 3 by gradient ascent, so that maximize Qo. Thus, the resulting terminating regions heavily depend on the reward function. OC also proposed a PG-style method for learning intra-option policies. We used a similar method to OC for learning policies and values, but we employed a different method that does not depend on rewards for learning termination functions. Also, we proposed UGAE (9) for enhancing OC-style policy learning. While OC maximizes the option-value function directly, many heuristic objectives have been proposed with similar architectures, including deliberation cost (Harb et al., 2018), interest (Khetarpal et al., 2020), and safety (Jain et al., 2018). Kamat and Precup (2020) extended OC so that resulting options are diverse by maximizing the divergence between intra-option policies, while IMTC considers the diversity of destinations. Notably, Harutyunyan et al. (2019) proposed the termination critic (TC) that maximizes an informatiionthhoreee objective referred to as predictability - -H(x)lo) Our method is inspired by TC and maximizes a similar information-theoortie objective for diversity rather than predictability. In addition, TC requires estimating ("(zz (x,)) and marginal distribution of Pa, which can be quite difficult in environments with large or continuous state spaces. We avoid such difficult approximations using Bayes' rule, making IMTC more scalable.
Mutual Information, Empowerment, and Skill Acquisition MI also appears in the literature regarding intrinsically motivated RL (Singh et al., 2004), as a driver of goal-directed behavior. well-known example is the empowerment (Klyubin et al., 2005; Salge et al., 2014), obtained by maximizing MI between sequential k actions and the resulting state / [at. .... a++s: #(++//z)) = (21+4//z)) - -I (Z de. ..., ac+ks xt). Empowerment represents both large degree of freedom and good preparedness: i.e., larger ( (24)) implies that there can be more diverse future states, while smaller H (Z+++ (0t ..., ...,, a++k: 2x) indicates that an agent can realize its intention with greater certainty. In RL literature, empowerment is often implemented by maximizing the variational lower bounds as intrinsic rewards ( (Mohamed and Rezende, 2015; Zhao et al., 2020). We can interpret our objective (X;; O st.)) as a variant of empowerment where fixed number of options represents action sequences. Gregor et al. (2016) also employed this interpretation and introduced VIC for training intra-option policies with fixed termination probabilities in a no reward RL setting. Our experiments observed that IMTC helps VIC to learn meaningful intra-option policies. As a variant of VIC, Baumli et al. (2021) proposed to use the reversed MI I [X,;; (zz)). MI has been used for discovering diverse skills without termination functions. Eysenbach et al. (2019) proposed maximizing MI between skills and states I(O; X), and Sharma et al. (2020) extended the objective to a conditional MI I(O; X'(x). These methods also maximize variational lower bounds of MI as rewards.
7 CONCLUSION
In this paper, we considered the problem of learning diverse options in RL. To learn diverse ter- mination regions of options in a scalable way, we proposed to maximize MI between options and terminating states per starting state. We derived an unbiased gradient estimator to approximately maximize this MI, yielding the InfoMax Termination Critic (IMTC) algorithm. Also, we proposed a practical implementation of IMTC with enhanced advantage estimation in Section 4.2. In reward- free experiments, we visualized that IMTC helped learn diverse and clear options combined with intrinsic rewards. We also showed that options learned by IMTC help an agent quickly adapt to a specific reward function by transferring learned options. Although our experiments observed that IMTC can learn clear and meaningful options, a potential problem that learning of classification model po heavily depends on exploration. For example, an agent cannot explore a room well, it would not be able to learn sufficiently diverse options. The relationship between diversity and exploration would be interesting. Also, our analysis in Figure 2 and Gridworld experiments in Appendix C.8 suggest that IMTC options can fall into small loops, forming uninteresting options. To prevent this, considering distances between states, e.g., by using bisimulation metric (Castro and Precup, 2010), is plausible research direction.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We publish anonymized source code used for all our experiments on https:// -odeeee-doode-
REFERENCES
M. Andrychowiez, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and Bachem. What matters for on-policy deep actor- critic methods? large-scale study. In 9th International Conference on Learning Representa- tions, ICLR 2021. OpenReview.net, 2021. URL https: httpptss//geeeeeeee...eeet forum?id nIAxjsniDzge.
P. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 1726-1734, 2017. URL httt:://aaaa.opp////// haper/vieww14BB1
ocs htt.://///.s///d....x.p..../ CAt.a//AAAIIAAAAAT///
A. G. Barto and S. Mahadevan. Recent advances in bicrarchheal Dyn. Syst. 13(1-2))41-77, 2003, doi: 10. 00.1023/A:102224449199777 URL https: /ttp:///ooo//////oo 10.1023 A: /ttp/////1112222199990993
reinforcement leamiin Discret. Event
A. G. Barto, S. Singh, N. Chentanez, et al. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning, pages 112-19. Piscataway, NJ, 2004.
A. G. Barto, G. D. Konidaris, and C. M. Vigorito. Behavioral hierarchy: Exploration and representa- tion. In G. Baldassarre and M. Mirolli, editors, Computational and Robotic Models the Hierarchi- cal Organization of Behavior, pages 13-46. Springer, 2013, doi: 10. 1000/998--3-442-9875-55 URL https:tt/////c... htt///..oog/..10007..0 /tt.1.1..110/11111
/ettps://doi.org///
K. Baumli, D. Warde-Farley, S. Hansen, and p Volodymyr Mnih. Relative variational intrinsic control. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, February 2-9, 2021, 2021.
M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count- based exploration and intrinsic motivation. In D. D. Lee, M. Sugiyama, von Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1471-1479, 2016. URL http:://ppppr...npp..oooo
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. CoRR, Abs//100.01540, 2016. URL http: http//axxvvvorrgraaaa 606.01540.
E. Brunskill and L. Li. Pac-inspired option discovery lifelong reinforcement learning. In Pro- ceedings of the 31th International Conference on Machine Learning, ICMI. 2014, Beijing, China, 21-26 June 2014, volume 32 JMLR Workshop and Conference Proceedings, pages 316-324. JMLR.org, 2014. URL http: http::::/ppccceeepeeemmmmm ml www.:ssswm..p......mmmmm /viesss/vv///e ht33///////s/lll.ee 14.html.
P. S. Castro and D. Precup. Using bisimulation for policy transfer in mdps. In M. Fox and D. Poole, editors, Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010. AAAI Press, 2010. URL http: http://wwwwaa.rr////////// ocs/ wttpx.ddexpppppppp AAAI AAAIIO /AAA////A110ppaar//taaaaaaaaaa http///pppp//eeeeeeeeeeepeeeeeee
J. Choi, A. Sharma, H. Lee, S. Levine, and S. S. Gu. Variational empowerment representation learning for cond-connd reinforcement learning. In M. Meila and T. Zhang, editors, Pro- ceedings of the 38th International Conference Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1953-1963. PMLR, 2021. URL http: prwceede wtw.eedii..smsm......s..mmm r.press/vviii vv939/ cho 1b.tml.
10

---

Under review as a conference paper at ICLR 2022
R. Coulom. Reinforcement Learning Using Neural Networks, with Applications to Motor Control. (Apprentissage par renforcement utilisant des réseaux de neurones, avec des applications au contrôle moteur). PhD thesis, Grenoble Institute of Technology. France, 2002. URL https: //tel.a www.vvve-vvveeee
00000985.
Y. Duan, X. Chen, R. Houthooft, Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In M. Balcan and K. Q Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1329-1333 Jrg, 2016. URL http: / / http://oooennnnll..ccccm. www.am....rrs///eemmmmmmmmseeeee v488/ 8/ /uann 6.html.
B. Eysenbach, A. Gupta, J. Ibarz, and Levine. Diversity is you need: Learning skills without a reward function. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: http:////ppeeeeeeeeeeefrooe RJ
A. Geramifard, C. Dann, R. H. Klein, W. Dabney, and J. P. How. Ripy: value-function--aase reinforcement learning framework for education and research, J. Mach. Learn. Res., 16:1573-1578, 2015. URL http: http://dd............ http:/iiiaiiiicctnn./c/ (ttp:/cffmmm222269999
K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. CoRR, abs/1611 1.07577 2016. URL htttt:tp///rreee /ttp:///ax/////////////////./oo/ /16.///16...
Harb, P. Bacon, M. Klissarov, and Precup. When waiting is not an option: Learning options with a deliberation cost. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (FAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3165-3172, 2018. URL https: // //wwwaaaai.org//eeoa /ow.a//ooroo/o/o/o///ooooooooooe wnwei.nnd//iiiiiitt AAAI AAAII 8 wwtpa/i//paaeee/e/e MAAIIIppp/AAAIII.
A. Harutyunyan, W. Dabney, D. Borsa, N. Heess, R. Munos, and D. Precup. The termination critic. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, pages 2231-2240, 2019. URL http:// proceedime www.inggmmmm....... ml wer.smll.pprr.. r.press v h/aaaauuuuuaallll 9a.html.
N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. A. Riedmiller, and D. Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1 77028.... 2017. URL http::/arxiv..or///eee
A. Jain, K. Khetarpal, and D. Precup. Safe option-critic: Learning safety in the option-critic architecture. CoRR, abs/l807 0000600 2018. URL http::t//// http::/aaxii..rrg//////o//// http://///11/000///
Jin, A. Krishnamurthy, M. Simchowitz, and Yu. Reward-free exploration for reinforcement learn- ing. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4870- 4879. PMLR, 2020. URL / http://ppprecccc.......pppppppep www.mm....e./ll111111
jin2 htmmm...
Y. Jinnai, J. W. Park, D. Abel, and G. D. Konidaris. Discovering options for exploration by minimizing cover time. K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3130-3139, PMLR, 2019, URL / vv7 jiinaiil 9b.html.
http: http::/:ppoceeeeeeeeeeeeee www.oeeddings...r....
Y. Jinnai, J. W. Park, M. C. Machado, and G. D. Konidaris. Exploration in reinforcement learning with deep covering options. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://
openreview.ng openreviewnet/Eozure www.eee.eeeeoemmiieiieee Dor.umiid-SSeIyaattt..
A. Kamat and D. Precup. Diversity-enriched option-critic. CoRR, abs/2011.02565, 2020. URL
https: httpss////rrdi.otrg//////111....
11

---

Under review as a conference paper at ICLR 2022
K. Khetarpal, M. Klissarov, M. Chevalier-Boiswe Boisvert, P. Bacon, and D. Precup. Options of interest: Temporal abstraction with interest functions. In The Thirty-Four Fourth AAAI Conference on Artificial Intelligence, AAAI 2020. The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 4444-445 AAAI Press, 2020. URL. attpss:/aatt http::://aaaa.rrgg// /tjo////rro////oooo//oooooo//ssm index.php/ AAAI //A.A//artiiiel/viieee//////////
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Reprexentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL httt::::.. http://axivvvooo abs/ 1412.6980.
A. S. Klyubin, D. Polani, and C. L. Nehaniv. All else being equal be empowered. In M. S. Capcarrère, A. A. Freitas, P. J. Bentley, C. G. Johnson, and J. Timmis, editors, Advances in Artificial Life, 8th European Conference, ECAL 2005, Canterbury, UK, September 5-9, 2005, Proceedings, volume 3630 of Lecture Notes in Computer Science, pages 744-753. Springer, 2005. URL https:
doi: 10. 10000111550090
M. C. Machado, M. G. Bellemare, and M. H. Bowling. A laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages htttp://rocceediin www.:eeddding..m.......... v 1/000//aaaaoooo a.html.
2017. URL
A. McGovern and A. G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In C. E. Brodley and A. P. Danyluk, editors, Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001). Williams College, Willianstown, MA, USA, June 28 July 1, 2001, pages 361-368. Morgan Kaufmann, 2001.
V. Mnih, A. Badia, M. Mirza. A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd Interna- tional Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1928-1937, 2016. URL http:/ wtw.mmoor/promee/eeeeeo mnihal 6.html.
papers/v48/
S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated rein- forcement learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1/25-1133, innooriatiioooooo
2015. URL http: maximi sation- for www.orriiiiiiiiiallllleeeeeee.ee www.::iiattteeeeeeeeeeeeeeeeeeee -ew.errrcreeeeeeeeeeneaneee
O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett, editors, Advances Neural Information Processing Systems 31: Annual Conference on Neurul Information P Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréul, Canada, pages 3307-3317, 2018 URL https: wtwp:roeeedd.nn..ori...........e wttp:errrppp.cppprrr/./pppp paper/ 2018/
OpenAI, I. Akkaya, M. Andrychowizz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's cube with robot hand. CoRR, abs/1910.071 13,00,11,, 2019. URL http: / aaa/aiiivvii www.v.ogg//o....o.o..... abs/ 91 0.07113.
G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with neural density models. In Precup and W. Teh, editors, Proceedings of the 34th International Conference on Machine Leurning, ICML 2017, Sydney, NSW, Australia, 6-1 August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pages 2721-2730. PMLR, 2017. URL. http:// / /ppppppeeiigg..../// .m1 www.mmrr...... w.w.epp.ee//............oo.....e wwt0///trooss....ot
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf. E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chil- amkurthy, B. Steiner, L.. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style,
12

---

Under review as a conference paper at ICLR 2022
high-performance deep learning library. In H. M. Wallach, H. Larochelle, A. Beygelz- imer, F. d' Alché-Burs Buc. E. B. Fox, and R. Gamett, editors, Advances in Neural Informa- Neural Information Processing Sys- tems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 8024-8035, 2019. URL http: /pppe r.. wwtpppccpppppp//111aa../// ----yyorrcha---mmrr-aa library.
tion Processing Systems 32: Anmual Conference
peeapers.nips.com style Seyyle-high-ae noigh-performanne wewpemaaanedddeeeeeeeeeeeeeee
R. Ramesh, M. Tomar, and B. Ravindran. Successor options: An option discovery framework for reinforcement learning. In S. Kraus, editor, Proceedings of the Twenty-Eighuh International Joint
Conference Artificial Intelligence, ICAI 2019, Macao, China, August 10-16, 2019, pages 3304- 3310. ijcai.org, 2019. doi: 10.24463/ijcai 2019//58. 4jcai.2019/458.
URL https: https:://dbi.org/0..2555
C. Salge, C. Glackin, and D. Polani, Empowerment-AN Introduction, pages 67-114. Springer Berlin Heidelberg, Berlin, Heidelberg, 2014. ISBN 978-3--42--53733499 doi: 10. 00.10077978-3-662-53774 9_4. URL https: http://ttoooooo////ooooooooo http:/...1000/ 97889978--3 3- 03--442-33334 53734-333-3 9_4.
A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http: / arriv.orgi abs/ 1312.6122..1
J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In F. R. Bach and D. M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1889-1897. JMLR.org. 2015a. URL http: BtR.nttp:p/prcceedinnge http:///ppoeeeee.........o.meeee v37/schulman 5.hmml.
J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. CoRR, abs/1506.02438, 2015b. URL http: /aber/g////////// abs/ 1506.02438.
Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707 2017. URL http::/aaxxvv.or//aatet httpa//wwww..o/////////....../.. aawpaab///777...7....
A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: http:s://ppeeeevieeeess http///penneeeeeetetteeee forum? htt.iidhHHgggg24//
Ö. Simsek and A. G. Barto. Skill characterization based on betweenness. In D. Koller, D. Schu- urmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Sys- tems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Process- ing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008, pages 1497-1504. Curran Associates, Inc., 2008. URL http: http://paapera.ns.cceeaeeeeeeeep characterization www.eerrriziiioobbbeeeee..e based- on-betweenness
skill-
S. P. Singh, A. G. Barto, and N. Chentanez. Intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems 17 (Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada), pages 1281-1288. 2004. URL
1et.:552--nnnnnnnnnnn ca calyicllllllmmlllllll 1y-mot vated-
reinforcemente coinnentt-eerinnng learning.
A. Solway, C. Diuk, Córdova, D. Yee, A. G. Barto, Niv, and M. Botvinick. Optimal behavioral hierarchy, PLoS Computational Biology, 10(8), 2014. doi: 10. Io.umal/o 100.m/077.ppiii0007777... https:// http://00001////oaaaaoo..o.00110
1003779. URL
R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
R. S. Sutton, D. Precup, and S. P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2):111-111, 1999. doi: 10. 1016/00/0 3702(19))0002--1. URL https: htttt:::///oi.oogg////00 00000000000000000000000000000000 (99) 00052-1.
13

---

Under review as a conference paper at ICLR 2022
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012. Vil- amoura, Algarve, Portugal, October 7-12, 2012, pages 0026-50333 IEEE, 2012. doi: 10.1109/ URL https: http://tt.oooggo//ooooooooooo wtt.:10000//00000022222
C. M. Vigorito and A. G. Barto. Intrinsically motivated hierarchical skill learning in structured environ- ments. IEEE Trans. Auton, Ment. Dev. 2(2):132-143, 2010. doi: 1L..19/AAAD.2210025500.000 URL https: http://t.oooooo//////oo 1tt./10.1111 0.1109 TAM.:2110.20002222222222
O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, Georgiev, Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai,
J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine. C. Gulcehre, Z. Wang, T. Pfaff, Wu, R. Ring. D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, 'T. Lillicrap. K. Kavukeuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 375(7782)350-354, 2019, doi: 10.10338441566--1-1722-- URL https: //dbi..rgg/1.103//// 0000338/4418880 -19---111-1--------11-111------1
M. L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60))3021, 2021. doi: 10.21105/jsss.00021 URL
R. J. Williams. Simple statistical gradient- followenn algorithmss for connectionist reinforcemeettt learning. Mach. Learn., 8:229-256, 1992. doi: 10. 10.1007/BF00999666 URL 10.1 (..107//F00992696
Y. Wu, E. Mansimov, R. B. Grosse, S. Liao, and J. Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In Guyon, von Luxburg. S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Gameet, in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 2289-5288. 2017. URL http:
https//nwpp...ppppp/////eeeeeaae method- for-deep reinforcement- learning using kwwneceer- - factored- approximation.
R. Zhao, Abbeel, and S. Tiomkin. Efficient online estimation of empowerment for reinforcement learning. CoRR, hats/2007.073555 2020. URL httttpp://aaa.i..rt///////101110
14

---

Under review as a conference paper at ICLR 2022
A OMITTED PROOFS
A.1 PROOF OF PROPOSITION
First, we make an assumption on the dependence of 1j from B.
Assumption 1. The empirical distribution of options 7/ x,) is independent of termination conditions 0
This assumption does not strictly hold, and we can consider using, c.g., two-time scale optimization to suppress the distribution shift of by updating 3 We employed a replay buffer to mitigate this issue in Section 4.1.
Lemma 2. The following equations hold.
Vo H()) = Σ 7(( Σ (o
log P(x(2x)) + Σ
(x) (1) p(xy)aa)))
21
(10)
Vo, (x 0)= - En(oZz, Σ Vo,
log Po +1 Σ po(zylz) (og (1 Po
+ ]
(11)
Then, sampling 2, y from d= and from 1/.
0/ H (Xy (x,) Ed
Vog Voglo- log P(x|x,) - log P(x/ (x=a))
(12)
V (x,, = 1Edd, Vo 23,
log P log 1o (xz 12x)
(13)
Proof of Lemma
Proof.
First, prove Equation (10). We have:
Va, (((x(xx)))) = -os Σ
P(x xx)) log ((z
24
Vo, P(xgla))
Vos (( log ((,y(o,)) + p(
P(z)(ø,)
I
Vog P(æylax) log ((2)(())) +1)
n(lk:.)) Vo, Pd log P(zy((a)+
m
Apply eq. (2)
Pd (o (o -Poc (1g
n(lo..))
log (og P(x,(=x)) + +1)
21
Σ n(op..))
(" E[e, log P(ar)lz)) +1)
21
Σ n(ol.s)) Σ (x((x) (((rll..)Voooooo (x) x P(r( +1 Σ pa (xrlax) log P(xylax)) ((rgla))))))) +11 1)]
sample
sample
sample
15

---

Under review as a conference paper at ICLR 2022
Sampling z, 2 o, get (12). Then prove Equation (11).
Vo, (o,H(X 0) vo, Σ ((lo)) Σ Pd (xy((x,)) P (x
Vag Vag (2 (2
Σ n(ol23))
Σ Vag Po (x log (2 (2 + (x)
pa pa(x (z))
Σ m(oktts)
Σ Vgg P Pa (1)) (1xg
Apply eq. (2)
Σ m(ok)r))
p Voll
log (((x)(())))
x
Σ m(l
Σ [o, E - PP
(3)
21
Σ n(llrs)) Σ (x(xx) Vo, L0 (x)
log (2 +1 Σ pd (x)lzz log P (x) +1]
sample
sample
sample
Sampling 2; fy o, we get eq. (13).
A.2 PROOF OF LEMMA
Proof. First, using Bayes' rule, we have:
p=(x)<=))= Pr(olæ,, x. Pr(x)(æs)
Pr(o]æs)
17(0(1))
Then, we have:
po (x
log (x))((x)) - log P(xy|æs) log
P(xy (28)
polo|x.. ((xyle))
log
P(ay|æx)
po(o)(s::1))
log
n)(oxx))
Applying this equation to the eq. (6), we have:
Vo, Oxxx Vos (X ((,)) V H(X
Ed m
Vo.l.0(z) log P(xlo.) - log P(x)(xx)) log (x|xx.) + log P (zx (2:))(
E
V Vg (xr [oog (x - log P(ol(ar)) - (log (2 - log P(x ((x)((0a))
Veg(aa(z) log po(O)x,11)) log po(o)x,.
Eda,m
n(loo.))
n(lkza)
Edamm
Vgs Lo (x)) (ogpo(ol2s) - log (o
B IMPLEMENTATION DETAILS
Clipped 3 loss Common PPO implementation updates no multiple times. However, our preliminary experiments observed that performing multiple updates for 3o led to destructively large updates and
16

---

Under review as a conference paper at ICLR 2022
Algorithm 1 InfoMax Termination Critic with VIC, PPO Style
1: Given: Initial option-value Qo. option-policy π², and termination function 3".
2: Let Bo be a replay buffer for storing option-transitinn
3: for k 1.... do
4: for == 2, N do
Collect experiences from environment
5: Sample termination variable b, from (x;)
6:
by = then
7:
Store (1, 0z), (1s+1, 0z), (F 02) to the replay buffer Bo
8:
end
9:
Choose next option 0:+1 by e-Greedy
10:
Receive reward R; and state T4+1, taking ~ Tol (2:)
end for
12:
for == 1.,.... Num. of PPO epochs do for all zi the trajectory do
> Optimize x², Qo. and 3
14:
Compute Rvie from the target network Compute Ag by (8) using Rvic Update (a,(2)) via PPO using A Update Qo o) to regress to Åo if of has already terminated then
15:
17:
18:
19:
Update (x;) via (7) end if end for end for
20:
21:
22:
23:
Train ç and üi by option-transitions sampled from Bo
24: if mod Kvic then
25:
Update the target network for VIC
end if
27:
end for
resulted in the saturation of 3 to zero or one. Hence, to perform PPO-style multiple updates, we introduce a clipped objective of eq. (7):
I CLIP (0z) clip(E.- (x) Ex- as (x), Ca. 8o ona (z) (oggoo(lo,, log (o(lo....)))))
(14)
where eg is small coefficient, and 8 old is an old 30 before updating. We also add maximization of the entropy 3o for preventing the termination probability saturating on zero or one. To this end, we maximize CCIP((s) + CHa H (x)) via gradient ascent w.r.t. 0g. where CHş is a weight of the entropy bonus.
Full description of the algorithm Algorithm shows a full description of our implementation of IMTC on PPO when combined with VIC rewards. As of the original PPO, it is built on the A2C-style (Mnih et al., 2016; Wu et al., 2017) architecture with multiple synchronous actors and a single learner. First, we collect N-step experiences interacting with environments. At line 7, we append tuples corresponding to option transitions (f, 0z): (Isth; not use all transitions and store first 1max options to prevent memory shortage. We used hmax = 10 or hmax 20. Then we update T², Qo, and | Line 13 is done via minibatch sampling in the actual implementation. We also update Po for estimating the gradient (7), sampling from the replay buffer Bo. We empirically found that rapidly changing Rvic leads to unstable learning, especially when IMTC is used in parallel. Thus, we employ target network to compute RvIC and periodically update at line 26. We used KviC 20 in the experiments.
Bo. Here,
do
Our implementation of VIC and RVIC In experiments, show that IMTC helps agent learn diverse options without reward signals. For this purpose, we employ VIC (Gregor et al., 2016) and RVIC (Baumli et al., 2021) for providing intrinsic rewards. Here explain VIC and RVIC implementation. VIC updates intra-option policies to maximize the lower bound of MI (3) (OOxs.) H(O)a x., X,) 2 H(O)z..) + Eo,z [log 2(0)x.,,))) as rewards,
17

---

Under review as a conference paper at ICLR 2022
8
π 0
Si-Sigood
23)
p zT))
IT'-Surtaux
TC-Sulma
f
sandard devistions
π
( xss)
IT:
Redignai
Stared PC
Figure 6: Neural Network architecture used for the Gridworld experiments (top) and continuous control tasks (bottom).
where can be any distribution and called an option inference model. We use Po We also learn fj(olr)) from sampled option transitions and approximate H(O)r,) by H(O)x,) Eooo n(l[2z,) ((lr() -E [log n(o]x,))] 2 log (folxx). To this end, we giving agent Rvic = evic (log po (o)xx,, log rj(o)or,) as an immediate reward when an option terminates, where evic scaling coefficient. This is different from the original VIC implementation where constant. However, empirically found this formulation helps diversity options in our preliminary experiments. In case of RVIC, we add a another classifier ql ((olay) y ). and constructed rewards by Rgvic cvic (log po 2 log ((o)xy)) is learned in the same fashion as po and f using the replay buffer Bo.
H (Olzz, is treated
C EXPERIMENTAL DETAILS
Our anonymized code
used for all our experiments E5D1/.
on https://
C.I NETWORK ARCHITECTURE
Figure 6 illustrates the neural network architecture used in our experiments. We used a shared convolutional encoder for Gridworld experiments, and two separated fully connected layers with 64 units for continuous control experiments. x² is parameterized as a Gaussian distribution with separated networks for standard derivations per option, similar to Schulman et (2015a). We used ReLU as activator for all hidden layers and initialized networks by the orthogonal (Saxe et al., 2014) initialization in all experiments. Note that the last layer for intra-option policy T" was initialized with small values, following the standard practice (Andrychowicz et al., 2021). We used the Adam (Kingma and Ba, 2015) optimizer in all experiments. Unless otherwise noted, we used the default parameters in Py (Paszke et al., 2019) 1.8.1.
18

---

Under review as a conference paper at ICLR 2022
Description
Value
0.99
Adam Learning Rate
10
Adam
10
Clip parameter for Eg- (6.)
0.05
Num. timesteps per rollout
256
Num. actors
16
GAE À
0.95
Num. epochs for PPO
10
Minibatch size for PPO
1024
Weight of H (r)) (CH)
0.001
Weight of (33) (CH,)
0.01
Gradient clipping
0.5
Capacity of Bo
8T92
Max num. option transitions to store (/ima))
20
Num. epochs for training and
Minibatch size for training and
2048
Scaling of Rvvic (evIc)
0.005
Synchronizing interval of the target network for VIC (Kvic)
20
Table Used hyperparameters
C.2 HYPERPARAMETERS
Table shows all hyperparameters used in IMTC VIC experiments on MuJoCo continuous control tasks.
C.3 ENVIRONMENT IMPLEMENTATION
Gridworld is based on RLPy (Geramifard et al. (2015), BSD3 License). We constructed continuous control environments on the MuloCo (commercial license, Todorov et al. (2012)), using OpenAl Gym (MIT license, Brockman et al. (2016)). Especially, point environments are implemented based on "PointMaze" in rllab (MIT license, Duan al. (2016)) with some modifications, mainly around collision detection. We also refered to the modified PointMaze code (Apache 2.0 license) relased by Nachum et al. (2018). The swimmer robot is originally used in Coulom (2002).
C.4 COMPUTATIONAL RESOURCES
All experiments conducted private cluster with NVIDIA P100 GPUs. On the cluster, training IMTC with VIC on MuloCo PointMaze domain for 4 x 10 steps takes about 27 minutes.
C.5 COMPARISON OF VIC AND RVIC REWARDS
Figure shows the
C.6 EFFECTIVITY OF UGAE
To investigate the effect of UGAB (9), we compared IMTC and OC with and without UGAE in task adaptation experiments. IMTC (no UGAE) and OC (no UGAE) estimate the adavantage ignoring the future rewards after switching options, similar to advantage estimation proposed by Bacon et al. (2017). Figure 8 shows the result. We can see that UGAE improves the performance in all domains for both IMTC and OC.
19

---

Under review as a conference paper at ICLR 2022
Comparison of Intrinsic Rewards
Comparison of log p(o X.
log ptols,
0.015
IMIE
IMIE VIC
INTC RVC
IMTC + RVIC (2-10:1 RVC (p=0
-0.020
IVIC (5-0.1)
VCC (3-0 13
3.0 1.5
4.0
0.
1.0
3.0
3.5
4.0 Comparison of
Environmental Steas (10f)
Environmental Steas (10%)
Figure 7: Left: Comparison of intrinsic rewards during training. log pio (,, x) during training.
Right:
Fal-tflanch
Satnmerfeasch
Antfleach
PorstBilland
03
60
16
130P)
Aardillard
Ponthish
ArtPuss
Method
IMTC
IMTC (no UGAE)
OC
OC Ino UGAE)
Eisiommnte Stppe OP?
Eravenmental Strpo 132%
Envionmenta Steps 2001)
Figure 8: Learning curves for transfer learning experiments with
C.7 NUMBER OF OPTIONS
Figure 9 shows learning curves of IMTC agents with variable number of options (2, 4, 6, 8, 10, 12) in task adaptation experiments. Increasing the number options sometimes improves the performance (e.g., in Swimme rRech)) but the relationship is not obvious. However, we can confirm that increasing the number of options does not have bad effect on the performance, while reduces training steps per each intra-option policy, thanks to network sharing Appendix C.I.
C.8 OPTIONS LEARNED FOUR ROOMS
Figure 10 show the options learned in the classifical Four Rooms Gridworld (Sutton et al., 1999).
20

---

Under review as a conference paper at ICLR 2022
03
42
Ensicemerral Sings D07
Oniceemrca Siaps
Arc Biard
Poinsuuh
Anthysh
Options
12
02
34
16
Figure 9: Learning curves for transfer learning experiments.
Uptin
Option
Option
Opti13
INTC RVIC
Uption
Opton
Opt on
Optin
Option
RVIC (9-.1)
Figure 10: Options learned in four rooms environment
21


Reviews:
[{"review_id": "X4FfHup_yfx", "reviewer": "Reviewer_SWkr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents an extension of the termination critic idea, which is called the InfoMax Termination Critic. Specifically, the approach is different from Termination Critic as it involves computing the entropy over the final states as a function of the option and the start state (termination critic only computes the entropy of the final states as a function of the option).\n\nThe overall approach is clear and well-presented.\n\nI do have a number of questions related to this approach and would like some clarifications from the authors: \n\n1. The Fig 2 presented shows qualitatively how the terminations for options look for IMTC and Termination Critic. Would it be possible to demonstrate a similar visualization on the standard 4 room gridworld, identical to the one presented in Termination Critic’s paper? It would be much easier to make a comparison with the published result as opposed to reimplementing termination critic and demonstrating the differences on a new but smaller gridworld.\n\n2. If I understand the approach correctly, the terminations for the options are discovered through IMTC but the policy for those options are discovered through Variational Intrinsic Control (VIC). Is my understanding correct? \nThis seems like a roundabout way of discovering options that are diverse. Why doesn’t the approach learn option-policies where the rewards for those options are obtained through the discovered termination functions?  \nAlso, this makes it unclear if the performance gains are due to the discovered terminations or due to VIC. Perhaps adding an ablation to address this is necessary to make it clear.\n\n3. Would it be possible to make Fig 3 more prominent? It is hard to understand what each of the option-policies are looking like. Also, how are the options being discovered for OC + VIC? I thought that OC requires extrinsic rewards (from tasks) to discover options and in the experiments considered in this paper, there are no extrinsic rewards (i.e., reward-free RL). \n\n4. Because the approach is very much related to Termination Critic, I think it is important to include Termination Critic as a baseline in the large scale experiments (the plots in Fig 3 and Fig 5). Without this result, it is difficult to understand how the proposed work is better than its prior work.\n\n5. What are the error bars in Fig 5? \n", "summary_of_the_paper": "The paper presents an approach for learning diverse temporally extended and reusable options. It is based on the assumption that learning diverse options are generally useful for downstream tasks. Thus, the approach aims to discover options without making any assumptions about the downstream tasks. The main idea behind the approach is to discover options by maximizing the mutual information between the options and the corresponding state transitions and demonstrates the options discovered by this approach. Also, the paper demonstrates that these options are useful for faster learning in a downstream task. \n", "summary_of_the_review": "Overall, I think the paper addresses an important problem in RL: discovering diverse options when there are no external tasks. The paper presents an adaptation of Termination Critic. \nHowever, some of the design choices made by the approach does not seem to be reasonable. Specifically, why use Variational Intrinsic Control in combination with the proposed approach when there is a direct way of learning option-policies from the discovered termination functions. Also, using VIC in this way makes it unclear as to whether the performance gains are due to the proposed approach or due to VIC.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9-z2u4j5Riv", "reviewer": "Reviewer_XKs3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n1.\tthe method is well motivated and derived \n2.\tEmpirical study in various domains show diversity of the learned options and its usefulness in transfer settings. \n\nWeakness:\n1.\tThe method is incremental and seems trivial. The method is directly derived from the termination gradient theorem from Harutyunyan et al. (2019) with reward signal constructed based on mutual information between terminal state and option. \n\n2.\tFailed to mention and compare some relevant references, E.g., reference [1] also considered learned diversified options using pseudo-reward constructed based on divergence between action distribution of different options. \n\n[1] Kamat & Precup, Diversity-Enriched Option-Critic, Neurips2020\n\n[2] Ramesh et al., Successor Options: an option discovery framework for reinforcement learning, IJCAI2019\n\nI would like to see some discussions of these references and experimental comparisons.\n\n3. This work is mostly based on empirical study, the paper could be strengthened if some theoretical analysis, such as sample/computation complexity, convergence etc. can be provided. \n\nOther comments:\nIn the abstract, “our experiments demonstrate … without rewards”,  you really mean “without environmental/extrinsic rewards”.\nProposition 2 seems simply a trivial extension of proposition 1. \nIn the experiment (5.1), why the policy over option $\\mu(o|x)$ is fixed to be $1/|O|$, instead of learned?\n", "summary_of_the_paper": "This paper proposed an algorithm, termed infomax Termination Critic (IMTC) to learn diversified options in RL. This algorithm learns termination conditions of options by maximizing mutual information between options and corresponding state transitions. The experiments demonstrate the IMTC algorithm learns diversified options and can be reused in various tasks.", "summary_of_the_review": "In all, this paper studied an important problem in reinforcement, learning diversified options that can be reused in various tasks. Their empirical study is able to justify the diversify of the learned options and shows superior performance in transfer learning settings comparing to some baseline.  However, there are still some key references missing and should be compared against. Also, the method seems trivial extension of many existing methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4xekqPTiiQf", "reviewer": "Reviewer_rEv8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I enjoyed the paper, it is well organized and well written, and it touches in a topic that I personally find very interesting which is options learning.\n\nStrengths:\n- Well written paper and easy to follow.\n- Straight forward objective with a nice intuitive explanation. Should be easy to re-implement for the reader.\n- Evaluation done in a set of diverse tasks.\n\nWeakness:\n- Maybe a concrete definition of what's meant by diversity in this paper could help. The objective encourages the options to learn different terminating states, but not different trajectories. So, based on the objective two options that \"move right\" for 38 steps, and one moves \"up\" on step 39 and the other moves \"down\" on step 39 would be diverse, and would maximize the objective.\nIf this is not a behavior that does not happen in practice, it would be nice to have a discussion around it, since nothing strictly prohibits this from happening. \n\n- Where the number of options pre-defined? If so, how did you pick that number?\n\n- On eq. 9, what is the assumption on the reward function to take the maximum over 0 as optimistic. What happens if all rewards are positive in a task? Taking the max over 0 would probably not help with understimation.\n\n- Looking at the results, it's interesting to see that \"reach\" tasks with large action spaces is where the proposed method tends to underperform. Does the complexity of the task or the action space play a role in how difficult it is to learn options? I would like to see some move in-depth analysis on the results.\n\n- Following on the results, I would suggest avoiding making non-specific statements like: \"IMTC outperforms other methods with a large margin in complex PointBilliard, AntBilliard, and AntPush\". In that case, my next question is \"what is a large margin?\" How significant are these values? How many times did you re-run the experiments to make that claim?\nI would like to see some more specific numbers around these results.\n\n\nNitpick:\n- At the beginning of the paper, there is quite a bit of emphasis on \"we hypothesis diverse options are reusable\". This is a generally well-accepted hypothesis, but the way its written it makes it sound that no one thought of it before.\n\n\n", "summary_of_the_paper": "The authors propose an objective for learning a diverse set of options in which the goal is to maximize the entropy of terminating states from any initial state while at the same time minimize the entropy of terminating states given a specific option. \nIntuitively, this means learning options that tend to be deterministic, while at the same time are able to reach diverse terminating states.\nThe authors then evaluate against other option learning frameworks in a series of diverse environments.", "summary_of_the_review": "Well written paper with an interesting objective that makes intuitive sense.\nThe results look promising, but the discussion and analysis could have more depth; there are questions left unanswered on this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4OmkJxFFVJw", "reviewer": "Reviewer_YG8c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The general idea of using the termination gradient in VIC is novel and interesting. That said, I feel that the method which was implemented has some gap from the theory, and that the method that is eventually implemented is too similar to VIC. Since the VIC paper was published there were quite a few papers extending it, like DIAYN and RVIC to give two examples. Thus, the novelty of this work is more limited, as it is only comparing to VIC as a baseline. I also feel that I don't learn enough from the experimental section and the algorithm description about the actual difference proposed here. \n\nMore concretely, the implementation of the option model is not clear to me and is not well justified. The theoretical derivation suggests using a model that predicts the probability of seeing s_f after executing option o from state s. However in practice, this model is not used, and it is replaced with the discriminator as being done in VIC, which is a different kind of a model. Furthermore, using this model instead of the option model makes the methods much more similar empirically to VIC and much less novel in my opinion. I think this should be clarified and discussed in much more detail, e.g., what is exactly the difference from VIC.  \n\nSection 4.2 makes some decisions that are not very clear to me and I would have like to see some experiments supporting them. I don't see it as a major part of the paper though and it might have been better to put this section in the supplementary. \n\nThere seems to be a mistake in the definition of the \"undiscounted occupancy measure\". First of all, it is unclear which occupancy measure is it; is it the average reward occupancy? is it the finite horizon? being undiscounted doesn't tell much. Secondly, on what is the expectation being taken over? if it is on trajectories executed by the policy, then shouldn't the state occupancy be infinity for all non transient states? the average reward and finite horizon state occupancies do not suffer from this issue", "summary_of_the_paper": "The authors propose an HRL algorithm that uses the VIC objective to discover the options, i.e., the termination condition of the options is trained to maximize the mutual information between the set of options and their terminating states. These options are first trained without access to a reward function in an unsupervised manner. Later on there are experiments that show how an RL agent can re-use these options in downstream tasks. ", "summary_of_the_review": "Nice idea, but the connection between the theory and practice is somewhat missing, resulting in an algorithm that might be too similar to VIC, and the differences are not explained clearly enough nor supported by the experiments. In addition, including more baselines would have made the empirical contributions stronger. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes InfoMax Termination Critic (IMTC), a new approach for learning option termination conditions with the aim of discovering more diverse options. IMTC relies on a scalable approximation of the gradient of a mutual information objective with respect to the termination function parameters.

Reviewers liked the motivation and the simplicity of the approach. While there were some initial concerns regarding the similarity of IMTC and VIC, the authors did a good job of clarifying the differences and providing additional results in the rebuttal. While two reviewers raised their scores based on the rebuttal, this left reviewers split on whether to accept or reject the paper.

Given that the paper’s main contributions are evaluated empirically I based my decision on the strength of the evaluation. The main claim in the paper is that IMTC significantly improves the diversity of the learned options when combined with intrinsic control methods like VIC and RVIC. The main supporting evidence of this claim is a visualization of the option policies and termination probabilities reached by VIC and RVIC. There are several issues with this comparison:
* This is a poor visualization of the kind of option diversity the paper aims to obtain. Given that mutual information based objectives used by VIC, RVIC and IMTC aim to optimize diversity in the final states reached by the options, visualizing the distribution of final states or the trajectories produced by the options is more meaningful.
* The VIC and RVIC baselines are evaluated with a fixed option termination probability of 0.1 which biases the comparison in favor of IMTC because IMTC is able to choose when and where to terminate while VIC and RVIC with random termination get to control neither. Using fixed option duration with MI-based option discovery methods like VIC, DIAYN and RVIC is more standard and is known to produce options with very clear terminal state clusters which are well-separated for different options. Fixed option duration allows VIC and RVIC precise control of where they will terminate since option duration is fixed, hence it should have been included in the comparison.
* As mentioned in point 2 above, it is well established that VIC tends to learn options with well-clustered end states, especially in simple gridworld domains like in Figure 3 (see VIC, DIAYN and RVIC papers). The authors seem to obtain different qualitative results raising questions.

Overall, I don’t think the qualitative experiments show that IMTC is able to improve the diversity of options discovered by VIC or RVIC due to issues with how the experiments are done (random option duration for VIC and RVIC) and how the results are presented (visualizing action probabilities instead of final states). Given these concerns and the split among the reviewers I recommend rejecting the paper in its current form.
Decision: Reject