Effectively Learning Initiation Sets in Hierarchical Reinforcement Learning
Authors: Akhil Bagaria
Ben M Abbatematteo
Omer Gottesman
Matt Corsaro
Sreehari Rammohan
George Konidaris
Venue: NeurIPS 2023 poster (2023)
Keywords: hierarchical reinforcment learning
TL;DR: Jointly learning option initiation sets and policies in online reinforcement learning

Abstract: An agent learning an option in hierarchical reinforcement learning must solve three problems: identify the option's subgoal (termination condition), learn a policy, and learn where that policy will succeed (initiation set). The termination condition is typically identified first, but the option policy and initiation set must be learned simultaneously, which is challenging because the initiation set depends on the option policy, which changes as the agent learns. Consequently, data obtained from option execution becomes invalid over time, leading to an inaccurate initiation set that subsequently harms downstream task performance. We highlight three issues---data non-stationarity, temporal credit assignment, and pessimism---specific to learning initiation sets, and propose to address them using tools from off-policy value estimation and classification. We show that our method learns higher-quality initiation sets faster than existing methods (in MiniGrid and Montezuma's Revenge), can automatically discover promising grasps for robot manipulation (in Robosuite), and improves the performance of a state-of-the-art option discovery method in a challenging maze navigation task in MuJoCo.

Effectively Learning Initiation Sets in Hierarchical Reinforcement Learning
Akhil Bagaria" Brown University Providence, RI, USA.
Ben Abbatematteo
Omer Gottesman Amazon, New York, NY, USA. omergott@gmail... com
Brown University, Providence, RI, USA. abba@brown.ede
akhil_ www.baaaabbroon....o.o
Matt Corsaro
Sreehari Rammohan
George Konidaris Brown University, Providence, RI, USA.
Brown University, Providence, RI, USA.
Brown University, Providence, RI, USA.
matthew..corsaroBrromm
Bachhari_rammohan@bbooooo edu
gdkłcs. www.ss.boonn... edu
Abstract
An agent learning an option in hicrarchical reinforcement learning must solve three problems: identify the option's 's subgoal (termination condition), learn policy, and learn where that policy will succeed (initiation set). The termination condition is typically identified first, but the option policy and initiation set must be learned simultaneously, which is challenging because the initiation set depends on the option policy, which changes the agent learns. Consequently, data obtained from option execution becomes invalid over time, leading to an inaccurate initiation set that subsequently harms downstream task performance. We highlight three issues-data non-stationarity, temporal credit assignment, and pessimism-sppecine learning initiation sets, and propose to address them using tools from off off policy value estimation and classification. We show that our method learns higher-qualice initiation sets faster than existing methods (in MINIGRID and MONTEZUMA'S REVENGE), can automatically discover promising grasps for robot manipulation (in ROBOSUITE), and improves the performance of state-of-the- art option discovery method in challenging maze navigation task in MuJoCo.
Introduction
Temporal abstraction, which is erucial for scaling RL to long-horizon problems [Konidaris 2019
Sutton et al. [2022). is elegantly captured using the options framework (Sutton et al.) 1999] Unlike primitive actions that only last for single timestep, options execute a policy until some termination condition is met. General purpose agents must possess many such options to solve wide variety of tasks [White] 2017). but increasing the size of the action raises both the complexity of learning and the branching factor of search [Littman et al.) 1995]. This necessitates a mechanism for locally pruning irrelevant options: most options do not apply effective branching factor by ignoring them [Precup et al. 1998 Wen et al. 2020). Fortunately, the options framework includes the concept initiation sets which captures the idea of local applicability: an option can only be executed from states inside the initiation set.
given state and the agent can lower its
is natural view initiation set as the set of states from which option execution is likely to succeed; in that case, learning it is straightforward when the option's 's policy is fixed: run the
"Equal Contribution
Work done while Brown University.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

option from different states, record the outcome, and learn a classifier that predicts whether option execution will succeed at given state [Konidaris and Barto, 2009, Kaelbling and Lozano-Péree Pérez 2017]. However, this is complicated by non-stationarity during learning: as the option policy changes, previously recorded outcome labels become invalid. How should the initiation classifier change in response? Another challenge is that initiation set learning suffers from a pessimistic bias: the option policy mostly improves for states inside the initiation set because option execution only occurs from such states. Therefore, if state is excluded from the initiation set early during learning. it will likely remain excluded even as the policy improves over time. This pessimistic bias causes initiation sets to mostly shrink and rarely expand (Bagaria et al. 2021a], even in the face of policy improvement; this is potentially catastrophic for learning since the agent prematurely gives up on options in favor of primitive actions (Harb et al. 2018].
To address the issue of non-stationarity, we introduce the Initiation Value Function (IVF), a general value function (GVF) [Sutton et al.] 2011, White] 2015) that predicts the probability that option execution will succeed from a given state. Since the IVF purely predictive, can be learned using tools from off-policy policy evaluation (OPE) [Voloshin et al.) [021] and does not interfere with the option's ss main task of maximizing its internal reward function [Sutton et al. 2023 Unlike the classification approach described above, the IVF adapts to a changing policy: as the policy improves over time, so does the IVF's estimates of its probability of success. We show how the IVF can be used direct estimate of the option's initiation set imput to a weighted classifier that accounts for changes in the option policy.
To address the pessimistic bias of learning initiation sets, we expand the criterion for including state in the option's initiation set: in addition to states from which option execution is likely to succeed, we also include states for which the option policy is most likely to improve. By identifying and adding these states to the initiation set, we mitigate the pessimistic bias and prevent initiation sets from collapsing as the option is learned.
For evaluation, we first measure how accurately we can deduce an option's initiation set in MINIGRID- FOURROOMS and the first screen of MONTEZUMA's REVENGE. Then, we demonstrate that our proposed methods can effectively identify promising grasps in challenging robot manipulation problems in ROBOSUITE. Finally, by integrating our method for learning initiation sets into an existing option discovery algorithm, we solve a maze navigation problem in MuJoCo that baseline agents are unable to.
2 Background and Related Work
As is standard in HRL [Barto and Mahadevan 2003 we consider problems that can be modeled as Semi Markov Decision Processes (SMDPs) M- (S, A, R. T, 7), where S is the state space, .A is the action space (which contains options and primitive actions), R is the reward function, T is the transition function and y is the discount factor. An option o E O C A models temporally extended behaviors; o (I., To, 9o), where I., the initiation set of the option, describes the states from which the option can be executed, 3, is the termination or subgoal region and n, is the policy [Sutton et al. 1999]. When using RI. to learn the option policy, an internal option subgoal reward function R. and timescale % completes the subtask description of the option and is used to train x, [Precup] 2000
Barto and Mahadevan 2003 White 2017].
An important problem in HRL is that of option discovery, where algorithms focus on identifying option termination conditions (subgoals) and usually assume that all options apply everywhere (Dayan and Hinton| 1993 Bacon et al.) 2017 Eysenbach et al.) 2019 Machado et al., [017]: as previously discussed, this is not scalable strategy for developing general purpose agents. We hypothesize that well-learned initiation sets will improve the quality of options, regardless of how they are discovered, and will eventually lead to more performant hierarchical RL agents.
General Value Functions. Value-based RL typically predicts and maximizes single scalar reward, but a value function can be generalized to predict (and sometimes control [Jaderberg et al., (017)) the discounted sum of any real-valued "cumulant" that can be computed from the agent's observations [Sutton et al. 2011 White (2055. Such Generalized Value Functions (GVFs) efficiently solve the temporal prediction tasks and can represent rich knowledge about the world (Schaul and Ring 2013

---

Initiation Sets in Robotics and Planning, Initiation sets are crucial in classical planning Fikes
al. 1972 as well as task and motion planning (where they are referred to as preconditions) Garrett et al. 2021 but they are usually designed by domain expert [Kaelbling and Lozano- Lozano-Pérez, 2017
In robotics (affordances), they are usually programmed to avoid collisions [Xu et al., 2021 or to correspond to relevant object attributes [Sahin et al., 2007 Huang et al. 2023] In control theory (regions of attraction). they denote states where policies are provably stable Tedrake Ames and Konidaris 2019]. While our proposed methods do not come with finite-time convergence guarantees, they are arguably more general and scalable because they do not assume pre-programmed policies or additional structure such as objects and robot dynamics.
Ames
Techniques for learning initiation sets fall into categories: classification, value-based and end-to-end
Classification approach. When initiation sets are learned, it is most often framed as binary
classification [Konidaris and Barto 2009 Bagaria and Konidaris) 2020 Khetarpal et al. 2020 Bagaria et al.] 2021b]. This approach is sound when the option policy is lixed [Konidaris et al. 2018] or when the affordances correspond to primitive actions [Khetarpal et al. 2020]. But, this
approach does not scale to the continual learning setting [Ring] 1995 Mcgovern 2002] where new,
temporally extended options keep getting discovered and their policies and initiation sets must be learned simultaneously.
Value function approach. SayCan grounds large language models using value functions of skills trained using offline RL [Brohan et al. 2023). Since the policies pre-trained, they do not confront the challenges of jointly learning policies and their initiation sets discussed in this paper. Nica et al.
[022] learn affordances via Monte Carlo estimation (which is equivalent to binary classification under a restrictive 0/1 reward function), but do not account for the pessimistic bias or the temporal structure in the initiation set learning problem. Furthermore, both SayCan and Nica al. [2022] design the option reward function so that the resulting value function can be interpreted as an affordance/initiation probability: this is strong restriction because RL is famously sensitive to reward design [Randløv and Alstrom 1998). Our method allows the designer to pick the reward function appropriate for their task and the initiation function is learned using the appropriate cumulant using policy evaluation. Relay Networks [Kumar et al. 2018 learn initiation regions by thresholding the critic, but have difficulty picking threshold because it is not possible to interpret arbitrary value functions as initiation probabilities.
End-to-end approaches. loC [Khetarpal and Precup 2019] learns initiation sets in the Option- Critic (OC) framework: GrASP [Veriiah et al., [2022] learns affordances that are useful for Monte Carlo Tree Search (MCTS). These are promising gradient- based approaches to learning affordances, but are specific to OC and MCTS respectively. Furthermore, they both learn affordances that maximize task reward, we instead focus on learning options that each represent their own subtask White 2017 and could later be composed to solve many downstream tasks [Barreto et al. 2019
3 Effectively Learning Initiation Sets
The problem of learning initiation sets can be naturally framed as training classifier where states on trajectories that achieve the subgoal are labeled as positive examples, and states on trajectories that fail to achieve the subgoal are labeled as negative examples. A probabilistic classifier trained on this data will predict the probability that an option will succeed from given state, which is exactly the desired semantics of an initiation set. However, when initiation sets and policies have to be learned together, such a classifier is no longer good estimator, for three reasons:
Data non-stationarity. The first reason is data non-stationarity, which refers to the phenomenon that previously collected training examples become invalid during the course of learning, but continue to impact classifier training. Consider trajectory Ty obtained by rolling out an option policy x at time to. If the policy is in the early stages of training, T will likely fail to reach the subgoal region 3, and To will be labeled as a negative example (8)))= = O, Vss E m)) At some future time 4, the policy might improve, but the states along To will continue to have negative label, which means that they will continue to be outside the initiation set. As a result, we will fail to capture the growing
competenccrrecchabiliy of nt>to, A naive solution to this problem would be to discard old training
3

---

W4
W1
Po
B
$2
W2
~ 5
50
SO
Figure I: Learning initiation set for option targeting subgoal 3o (star). (left) Temporal structure in initiation set learning: even though Tı (black) fails to reach the goal, states along it should get credit because of the portion shared with the successful trajectory T2 (red); this does not happen using a simple classification approach. (middle) the initiation value function (IVF) V. (See (.T] is shown in green: darker the shade, higher the value; classification examples should be re-weighted to account for a changing policy. (right) Pessimistic bias causes initiation sets to shrink (blue - red) over time.
examples, but not only would that be sample-inefficien it is also unclear at what point training example is "old enough" to be discarded.
Pessimistic bias. The second reason, pessimism, refers to the fact that once states are deemed to be outside the initiation set, they are unlikely to switch labels because the option has low probability of being executed from them, and so policy improvement from these states is unlikely (Figure I (right)).
Temporal Structure. The third reason is that classification (or equivalently, Monte Carlo learning [Sutton and Barto 2018)) does not exploit the temporal structure in the initiation set learning problem. Figure I (left) shows two trajectories sampled from the same option's 's policy: TI reaches the goal and 72 does not. Any form of Monte Carlo estimation using this data would assign an initiation probability of O to most states in trajectory TI. However, this does not assign enough credit to most of the good decisions made along Ti tte only way to achieve that would be via temporal bootstrapping. This form of generalization not possible when using classification.
To address non-stee stationarity and exploit temporal structure, we formulate new general value function, the Initiation Value Function (IVF; Section 3.1). learned using temporal difference (TD) methods [Sutton 1988]. To address pessimism, we augment the initiation set with states from which the option policy is most likely to improve (Section 3.3
3.1 Initiation Value Function (IVF)
An option o's initiation set I, is the of states from which the option policy n. can succeed with high probability: Z, = {s : IP(s' E B.|S- = 8, A = =o) > T} (T' is a predefined threshold). Treating the success condition of the option as a cumulant c, : s + {0, 1) and setting the timescale Ye. := the corresponding GVF
H. Σ
(T.( = F.T,
co(8441)11441 ~ T( No (+1))) = P(c, = 1|S 8,, 8, = o)
represents the initiation probability at state 8, (Ho is the option horizon). Note that y is different from Vo, the value function used by the option policy for control; this is because T, maximizes an arbitrary reward function R. and so the resulting value function Vo, which approximates the value of the optimal option policy V=s, cannot be interpreted as an initiation probability.
Using the IVF as an initiation set. To directly the IVF the initiation set, need pick threshold T above which a state is deemed to be in the option's 's initiation set. Previous work (Kumar et al. 2018 reused the option value function V, the IVF, but had to develop heuristic schemes to pick threshold. But since DT outputs an interpretable number between and (we use a sigmoid layer at the end of é to enforce this), it is easy to threshold, regardless of R..
Learning the IVF Computing the probability of success of given policy n. is equivalent to the problem of off off-policy policy evaluation. Our approach to policy evaluation must sample-efficient if it is not, the option policy will change before are able to sufficiently evaluate it (Sutton et al.)

---

2007]. Furthermore, the initiation cumulant is a sparse binary function, which makes learning with the Monte Carlo estimator high variance and sample inefficient. This once again underscores the importance of exploiting temporal structure of the problem using TT>-larniig (we use TD(O)) to estimate V which unlike a classifier, is able to propagate value from partial trajectories (as illustrated in Figure I (left)). In this view, using a classifier is equivalent to using a Monte Carlo estimate of the IVF: high-variance, sample inefficient, and unable to bootstrap.
3.2 Combining Classification and the Initiation Value Function (IVF)
In Section 3.1 connected the initiation set to GVF, which allows us to use all the tools of value-based RL (Sutton and Barto 2018) to learn the initiation set. However, the classification approach mentioned earlier has some attractive qualities: specifically, classification is easier than regression [Bishop and Nasrabadi 2006) and the supervised learning community has developed powerful tools for learning classifiers using neural networks and cross-entropy loss (Goodfellow et al. 2016). To get the best of both worlds, we propose an additional method that combines value estimation and classification.
Recall that binary classification can be described as the process of finding parameters that minimize
the cross-entropy loss L(0) in the training data D {(s,, m) i= where are the inputs (states in our case) and Y; are the classification labels (whether option execution was successful from s). To deal with non-stationarity. we instead use the weighted binary cross-entropy loss L.. (0) which minimizes loss over a dataset desired contribution of the training example to the classification loss [Natarajan et al. 2013).
[P]
{si, Y,, we (x:) where ue (x:): a state-dependent weight, represents the
How should we set the weights wr(s) of training example (st, Y,) a way that reflects the evolving competence of the option policy? Since the IVF evaluates the success probability of the current option policy n. we can heuristically use reflection of our certainty in the training example
(s; positivliy (negatively) labeled states contribute more to L when the IVF prediction is high (low). Specifically, the weighting function defined via the following simple equation:
wr(s) = (+)( + (1 - Y.) - V(s)).
(1)
As the IVF estimates that the option policy is becoming more competent at state s, the classifier's s uncertainty about a negative example at will increase, thereby causing that negative example to contribute less to the classifier's decision boundary. Similarly, if the policy degrades at 8, then the contribution of positive label at to L will go down. By weighing training examples in this way, binary classifier is able to adapt to data non-stationarity that results from a changing option policy.
At the end of every option execution, we therefore recompute weights using Equation ! and re-train option initiation classifiers using a weighted cross-entropy loss Ly
We can use the fact that policies in RL usually improve over time to guide classifier training. To do this, we the weight of positive examples ws)(s) as 1 throughout training. The underlying idea is that a state which leads to success once is likely to do so again with an even better policy. The weights of negative examples are allowed to vary with the IVF as in Eq
Policy Improvement Prior.
3.3 Overcoming Pessimistic Bias
Unsuccessful option trajectories cause the initiation set to shrink. When a state is outside the initiation set, the option can no longer be executed from there. So even if the option could succeed from that state in the future, it remains outside the initiation set and the option ends up with a smaller region of competence than it needs to. This issue, which call the pessimistic bias of learning initiation sets, can prevent the learning of useful options and lowers the effectiveness of hicrarchical RL agent
To mitigate this pessimistic bias, we expand the initiation set to also include states from which policy improvement most likely:
I I- (s: Vo(s) + B()) T, S}.
Effectively identifying such states in high-dimensional spaces is the topic of bonus- exploration in deep RL (Taiga et al. 2019]. We propose two simple approaches:
Thii issue is analogous non-stationary dynamics in flat MDPs; Section in the Appendix.
5

---

1. Competence progress attempts to capture regions where a policy is either improving or
regressing (Şimşek and Barto, 2006, Stouu and Barto 2010 Baranes and Oudeyer 2013].
This can be computed changes in the IVF over time: (s)
(*)() V/= K (s) where
V/ is the current IVF and V is the IVF estimate K timesteps ago (obtained using the target network).
2. Count-based bonus approach keeps track of the number of times N(s, o) option o has been executed from state This is then converted into an uncertainty measure: (s) c/ / (V, o), where is scalar hyperparameter [Strehl and Littman 2008
We use count-based bonuses in problems where tabular counts are readily available; otherwise, we use competence progress.
4 Experiments
We aim to evaluate whether our methods can improve hierarchical RL. First, evaluate whether they result in better, more efficiently learned initiation sets. Second, we test if hetter initiation set learning improves option learning whole. Finally, we incorporate our changes into state-of-the-ar skill discovery method and check if resulting agent is able to outperform the baseline in a sparse-reward continuous control problem.
Implementation Details. Option policies are learned using Rainbow [Hessel et al. 2018 when the
action-space is discrete and TD3 (Fujimoto et al.] 2018 when it is continuous. Following Bagaria let al. [2 all options share the same UVFA [Schaul et al. 2015) but condition using their own subgoals. The IVF is learned using Fitted Q-Evaluation [Le et al. 2019), prioritized experience replay (Schaul et al.] 2016) and target networks (Mnih et al.) 2015] The IVF Q-function and initiation classifier are parameterized using neural networks that have the same architecture as the Rainbow/TD3, Each option has "gestation period" of 5 [Konidaris and Barto 2009), which means that before the option sees successful trajectories. its initiation set optimistically initialized be true everywhere. Since the training data for classification can be severely imbalanced, we use the standard technique of upweighting the minority class (Japkowiez and Stephen 2002].
4.1 Measuring the Quality of Initiation Sets
Before comparing different methods based on task performance, we specifically test the quality of initiation sets learned in SANGGRID-FUURROMMS [Chevalier-Boisseett et al. 2018) and the first screen of MONTEZUMA'S 's REVENGE [Bellemare et al.] 2013). In both domains, observations are 84 84 images and the action-space is discrete. For this experiment, we design start states Sp in each domain-in MINIGRID-FOURROOME and in MONTEZUMA'S REVENGE, So is a series of 100 states scattered across the first screen (more details in the appendix). For the purpose of evaluating our initiation set learning algorithms, at every episode, we sweep through states So and reset the simulator s. Option termination conditions are also hand-defined: in FourRooms, we create options that target the center of cach room: in Montezuma's 's Revenge, we define five options: those that navigate the player to each of the two doors, one that navigates the player to the bottom-right and one to the bottom-left of the first screen and finally an option that attempts to get the key.
So is the set of all states (since this is small tabular domain)
Initiation Set Accuracy. At each state E Sn. we record the initiation decision made by the learning algorithm as (s; 0) E {0, We then execute the option policy To from that state and record whether or not the agent reached the option's subgoal as Y, (0, 1}. Accuracy at state s, for (x; 0) = Y.). This process repeated several times for all options option is then given by 1 (1;
E O and start states 8 E Sg. To be faithful to how options are learned in the online RL setting, we only use the trajectory obtained by running the option for updating the initiation classifier and the
option policy if the learned classifier returned true at the start state, i.e, [o f) = 1 (in which case T, is updated using Rainbow (Hessel et al. 2018:; pseudocode in Appendix C.I
Initiation Set Size. Although accuracy is a natural evaluation metric, it does not fully capture the quality of the learned initiation set. For example, if the predicted initiation function returns false

---

MinGGric-FourRooms-0
MonntzumaRReeeggeloTaaeeeke
0.9
Accuracy
Normalized Size Set
Accuracy
Normalized Size True Init Set
1.0
0.0
0.8
0.9
0.5
0.4
0.0
0.5
0.5
0.3
0.4
0.3
Weighted
Weighted
0.2
0.2
Beseline
0.1
Boseline Binary
100
50
100
150
200
50
100
150
200
Episode
Eplsode
Episode
Episade
Figure 2: Measuring the quality of initiation sets in ManiGrid-FourRoomss (left) and the first screen of Montezuma's 's Revenge (right). Solid lines denote mean accuracy and initiation set size, shaded regions denote standard error. All curves are averaged over all state-option pairs and 5 random seeds.
everywhere (i.e, if (s 0) = 0, Vs E So), then the option policy would not get new data to improve. the policy never improves, its true initiation set could also collapse; while such an initiation learner would register high accuracy, the option would not be useful. As a result, we additionally measure the normalized size of the "true" initiation set Y,|: this is the fraction of start states So from which Monte Carlo rollouts of the policy succeeds. A well-learned initiation set is not only accurate, it is large region of option policy competence.
also as large as possible possibleereflecting reflecting
Competing methods. We compare the accuracy and size of the initiation sets learned by:
Baseline binary. Binary classifier used to learn the initiation set. This is what is used in essentially all prior work and the only baseline method this experiment.
IVF Threshold the Initiation Value Function discussed in Section 3.1
Weighted, This is weighted binary classifier discussed in Section 3.2
Both the IVF and Weighted approaches use the competence progress exploration bonus described in Section 3.3 its impact is ablated in Appendix H
Discussion of results. Figure 2 shows that our proposed methods significantly outperform the baseline binary classification approach, which learns classifiers that are less accurate and smaller than those learned using other techniques. Furthermore, the accuracy of the baseline binary classifier falls over time; this is because even though the policy improves (and hence the size of the true initiation set increases), the learned initiation remains small and unable adapt By modifying the binary classifier using the weighting scheme described in Section 3.2 we are able to learn more accurate initiation sets that reflect highly competent option policies. Interestingly, the optimistic version of the IVF is necessary to learn good initiation sets Montezuma's 's Revenge, but it underperforms compared the plain IVF in FourRooms; we hypothesize that this is because FourRooms is a small and simple enough domain that an exploration bonus is not strictly necessary.
improving policy.
4.2 Robot Manipulation: Identifying Promising Grasps
In the previous section, we showed that our methods can lead higher quality initiation sets; now we evaluate whether that can in turn improve option learning whole. We adopt robot manipulation as a testbed because is natural problem setting for evaluating initiation set learning algorithms for example, consider robot manipulating hammer: not only should it choose a feasible, stable grasp on the hammer, but should also choose one that would allow to subsequently drive nail. This problem, referred to as task-specific grasping in robotics, and has been studied extensively for fixed policy [Kokic et al. 22000] Fang et al,, 2020 Zhao al., 202T Wen et al. 2022 (chhavi et al. 2022]; here use RL to deduce which grasps are most likely to afford success while simultancously learning the skill policies. Furthermore, dense reward functions are often necessary to train RL policies in
This is similar to the considerations of precision/recall in supervised learning.

---

Door
Lever
Success Rale
0.6
0.4
0.2
0.0
0
1000
2000
3000
4000
500D
0
1000
2000
300D
4000
5000
2002
4000
8000
80DO
10000
Episode
Episode
Episode
Baseline Random
Baseline Binary
IV
Weighted
Figure 3: Task success rate for manipulation domains aggregated over 4 random seeds.
robot manipulation domains Gu ct al. 2017]; this section highlights that initiation probabilities can be learned even when the reward function used to train To is not sparse like the initiation cumulant.
We use three constrained manipulation tasks in ROBOSUITE [Zhu et al. 2020 opening a door, flipping a lever, and manipulating sliding mechanism. Each task is modeled the policy and initiation set are learned simultaneously. Each environment has 250 possible start states Sp. At the start of each episode, a configuration sampled from the initiation set learner:
single option and
In(s) EvESO (s') Vs E So:
30
the manipulator is then reset to The continuous observation space is 52-dimensional, the continu- ous action space 13-dimensional; we use TD3 for policy learning (Fujimoto et al., 2018).
Our algorithms are compared against two baselines: Random, which selects an arm configuration random the start of each episode and Binary which treats initiation learning vanilla binary classification. Our algorithms, IVF and Weighted, employ the optimistic bias discussed in Section 3.3 by adding count-based bonus the predicted initiation probability.
Task success rates are shown in Figure 3 The DOOR task is the simplest: most of the grasp candidates afford task success. As result, the Random and Binary baselines The LEVER and SLIDE tasks are more challenging as the number of promising grasps is significantly lower. In these domains, only the Weighted variant is able to consistently recover policy with success rate above 50%.
competitive with our algorithms.
TigureFste shows that our methods can identify good grasp poses without task-specific engineering or human demonstrations. This is impressive given that human data [Mandikal and Grauman 2021 2022 heuristics like image segmentation [Kokic et al.) 2017 Rosen et al.] [022] are typically required to learn these affordances efficiently, even in the case of fixed policy. Additional visualizations and ablations can be found in Appendix D and
4.3
Improving Option Discovery
Finally, evaluate whether better option learning can improve option discovery. we integrate our approach into an existing state-of-the of-the-art algorithm: deep skill chaining (DSC) [Konidaris and Barto 2009 [Bagaria and Konidaris 2020]. DSC learns collection of options so that the subgoal region 3o, of option into each other's initiation sets, DSC learns how to sequentially compose options to reliably achieve goal. It does so by first learning an option that reaches the task goal, then another option that targets the first option's 's initiation set and so on, until the start-state is inside the initiation set of some option; for more details, please refer to Appendix C.3 and the original papers [Konidaris and Barto 2009 Bagaria and Konidaris) 2020]. We chose to integrate our techniques with DSC because of its focus on learning initiation sets jointly with options in online and incremental reinforcement learning setting and because the quality of initiation sets is crucial to its performance and stability [Bagaria et al. 2021a].
the initiation region Io, of another option oj: by learning options that funnel
TThis baseline equivalent initiation that uniformly everywhere.

---

Figure 4: Examples of promising grasp poses (top row) DOOR (left), LEVER (middle) and SLIDE (right): these are high probability samples from our initiation function; contrasted with bad grasp poses (bottom row), which are low probability samples and are ruled out by the initiation function.
Ant Maze Medium
Ant Medium Maze
1.0
Weighted Cif
S2
DSC Initiation GVF
0.8
Baseline DSC
Success Rate
0.6
0.4
0.2
0.0
50
100
150
175
200
Episode (x10)
Figure 5: (left) Comparing the performance of baseline DSC |Baaaiia et al. 2021a to versions of DSC with our initiation learning techniques. Solid lines denote average success rate over 5 random seeds, shaded regions denote standard eгror. (right) A visualization of initiation sets learned in ANT MEDIUM-M
where the ant robot has to navigate from the bottom-right to the top-left. Each color denotes initiation set a different option; although the plot only shows the location ant in the maze, the initiation set is learned using the full 30-dimensional state.

---

Experimental setup. We compare baseline DSC to versions that modify the way in which option initiation sets are learned. We use the ANT MEDIUM MAZE environment where the agent gets a sparse terminating reward of 0 for reaching the goal and -1 every other step; each episode lasts maximum of 1000 steps [Fu et al. 2020 Todorov et al., 2012]. This a challenging problem that cannot be solved with flat RL algorithms in reasonable amount of time. Baseline DSC was able to solve this problem by learning initiation sets defined over a subset of the state (I, y location of the ant) Baaariia et al. 2021a], but here, we do not assume access to such privileged information- all methods learn initiation sets using a dense neural network that maps the full 30-dimensional continuous state to initiation probability.
We used the author implementation [Bagaria et al. 2021al where option policies are parameterized using goal-conditioned value functions [Schaul et al. [215]] When using the IVF directly as the initiation set, we mimic the structure of main agent and use a goal-conditioned value function to represent the IVF; the final initiation set is defined as
I. s max Vo(s.g) T.
gER,
More details about the task and implementation details can found Appendix A and
Figure 5 (lett) shows that baseline DSC is unable to solve Medium Ant-Maze; but by changing the way options learn their initiation sets, we are able solve the problem quickly and reliably. The weighted classifier slightly underperforms the pure IVF approach in terms of mean return, but it has lower variance across runs. Figure [ (right) visualizes the initiation sets learned by the weighted classification approach; the figure makes it clear that different learned options specialize in different parts of the maze.
5 Conclusion
Learning initiation sets is a critical component of skill discovery, but treating it as binary classification misses key characteristies of the problem. Specifically, does not address the non-stationarity that results from a continually changing policy, the pessimistic bias of learning initiation sets online, or the temporal structure in the problem. To address these challenges. we proposed the Initiation Value Function (IVF), a general value function tailored specifically to learning initiation sets. We used the IVF directly via thresholding and also via weighted binary classifier which adapts to changing option policies. Through our experiments, we showed that our proposals lead to higher quality initiation sets, can lead to faster learning of single option and boost the performance of an existing skill discovery algorithm.
limitation of the IVF cumulant proposed in Section 3.1 is that only applies for goal-reaching options, which although is quite general, it is not universal. For example, if the option's 's task is to maximize the velocity of robot and there no specific target velocity, then we could not write down a 0/1 cumulant that faithfully describes that subtask. Designing cumulants that result in initiation probabilities for general option subtasks is an important avenue for future work.
Acknowledgements
We thank Mark Rowland and Rafael Rodriguez- Sancheu for useful discussions. We also thank our anonymous reviewers for making thoughtful suggestions that improved the quality of our paper. This research was supported in part by NSF 1955361, NSF CAREER 1844960 to Konidaris and ONR N00014-21-1-22000 The U.S. Government authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The content is solely the responsibility of the authors and does not necessarily represent the official views of NSF or the ONR.
References
Barrett Ames and George Konidaris. Bounded-error LQR-Trees. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 154-150. IEEE, 2019.
Barrett Ames, Jeremy Morgan, and George Konidaris. Ikflow: Generating diverse inverse kinematics solutions. IEEE Robotics and Automation Letters, 73):717777184, 2022.
10

---

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critte architecture. In Thirty-Fint First AAAI Conference on Artificial Intelligence, 2017.
Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. In International Conference on Learning Representations, 2020. URL httppp://oppnreveee.eeeee//e//
BigqipNYwH,
Akhil Bagaria, Jason Senthil, Matthew Slivinski, and George Konidaris. Robustly learning compos- able options in deep reinforcement learning. In 30th International Joint Conference on Artificial Intelligence, 2021a.
Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. International Conference on Machine Learning, pages 521-531. PMLR, 2021b.
Adrien Baranes and Pierre- Yves Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.
André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, et al. The option keyboard: Combining skills in reinforcement learning. In Advances in Neural Information Processing Systems, pages 13031-13041, 2019.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, (3(12)):41-77 2003.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as I can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
Maxime Chevalier- for gymnasium, 2018. /ttps:t/t:/// com Paraaaa-aaouuu Porraaaaoounaaatoonniiiiiaa
Lucas Willems, and Suman Pal. Minimalistic gridworld environment
Peter Dayan and Geoffrey Hinton. Feudal reinforcement Iearnin In Advances in neural information processing systems, pages 27 271-278, 1993.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, and Silvio Savarese. Learning task- aoientrie grasping for tool manipulation from simulated self-supervision. The International Journal of Robotics Research, 39(2-3):202-22 216, 2020.
Richard Fikes, Peter Hart, and Nils J Nilsson. Learning and executing generalized robot plans. Artificial intelligence, 3:25 1-88,, 1972.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In Proceedings of the 35th International Conference on Machine Learning, pages 1582-1591, 2018
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.

---

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- aff-po updates. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
Jean Harb, Pierre-Lue Bacon, Martin Klissarov, and Doina Precup. When waiting is not an op- tion: Learning options with a deliberation cost. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence. volume 32, 2018.
Zixuan Huang, Xingyu Lin, and David Held. Self-supervised cloth reconstruction via action- conditioned cloth tracking. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK. May 29 June 2, 2023, pages 7111-7118. IEEE, 2023. doi: 10.1109/ ICRA48891 2222..1011065..
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. 5th International Conference on Learning Representations, ICLR, 2017.
Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data analysis, 6(5):429 449, 2002.
Leslie Pack Kaelbling and Tomás Lozano-Pérez. Learning composable models of parameterized skills. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 886-893.
IEEE, 2017.
Khimya Khetarpal and Doina Precup. Learning options with interest functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9955-9956, 2019.
Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, David Abel, and Doina Precup. What can do here? A theory of affordances in reinforcement learning. International Conference on Machine Learning, pages 5243-5253. PMLR, 2020.
Mia Kokic, Johannes A Stork, Joshua A Haustein, and Danica Kragic. Affordance detection for task-specific grasping using deep learning. In 2017 JEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids). pages 91-98. IEEE, 2017.
Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning task-triented grasping from human activity datasets. IEEE Robotics and Automation Letters, 5(2):3352- 2020.
George Konidaris. On the necessity of abstraction. Current Opinion in Behavioral Sciences, 29:1-7, 2019.
George Konidaris and Andrew Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. Advances Neural Information Processing Systems, pages 1015-1023, 2009.
George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learn- ing symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215-289, 2018.
Visak CV Kumar, Sehoon Ha, and C Karen Liu. Expanding motor skills using relay networks. In Conference on Robot Learning, pages 744-756. PMLR, 2018.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Pro- ceedings of the 36th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pages 3703-3712, PMLR, 09-15 Jun 2019.
12

---

Michael L. Littman, Thomas L. Dean, and Leslie Pack Kaelbling. On the complexity of solving markov decision problems. UAI '95: Proceedings of the Eleventh Annual Conference on Uncertainty in Artificial Intelligence, Montreal, Quebec, Canada, August 18-20, 1995, pages 394 402. Morgan Kaufmann, 1995.
Marlos C. Machado, Marc G. Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference Machine Learning Volume 70, page 2295-2304 2017.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523- 562,, 2018.
Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE international conference on robotics and automation (ICRA), pages 6169-6176. IEEE, 2021.
Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, pages 651-661. PMLR, 2022.
Roberto Martin-Martin Michelle A Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, and Ani- mesh Garg. Variable impedance control in end-effector space: An action space for reinforcement learning contact-rich tasks. In 2019 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 1010-1017. IEEE, 2019.
Elizabeth Amy Megovem. Autonomous discovery of temporal abstractions from interaction with an environment. PhD thesis, University of Massachusetts at Amherst, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540)5529, 2015.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
Andrei Nica, Khimya Khetarpal, and Doina Precup. The paradox of choice: Using attention in hierarchical reinforcement learning. arXiv preprint arXiv:2201 220.06230.,,6653, 2022.
Doina Precup. Temporal abstraction in reinforcement learming. University of Massachusetts Amherst, 2000.
Doina Precup, Richard Sutton, and Satinder Singh. Theoretical results on reinforcement learning with temporally abstract options. In Machine Learning: ECML-98: 10th European Conference on Muchine Learning Chemnitz, Germany, April 21-23, 1998 Proceedings 10, pages 382-393. Springer, 1998.
Jette Randløy and Preben Alstrøm. Learning to drive bicycle using reinforcement learning and shaping. In Proceedings of the 15th International Conference on Machine Learning, volume 98, pages 463-471, 1998.
Mark B. Ring. Continual learning in reinforcement environments. PhD thesis, University of Texas at Austin, TX, USA, 1995.
Eric Rosen, Ben M Abbatematteo, Skye Thompson, Tuluhan Akbulut, and George Konidaris. On the role of structure in manipulation skill learning. In CoRL 2022 Workshop on Learning, Perception, and Abstraction for Long- Long-Horiizn Planning, 2022.
Erol Şahin, Maya Cakmak, Mehmet R Doğar, Emre Uğur, and Göktürk Üçoluk. To afford or not to afford: A new formalization of affordances toward affordance-based robot control. Adaptive Behavior, 15(4):447-477 2007.
Tom Schaul and Mark B. Ring. Better generalization with forecasts. In Francesca Rossi, editor, IJCAI 2013, Proceedings of the 23rd International Joint Conference on Artificial Intelligence, Beijing, China, August 3-9, 2013, pages 1656-1662, 2013.
13

---

Tom Schaul. Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320, 2015.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2016.
Giulio Schiavi, Paula Wulkop, Giuseppe Rizzi, Lionel Ott, Roland Siegwart, and Jen Jen Chung. Learning agent-aware affordances for closed-loop interaction with articulated objects. arXiv preprint arXiv:2209.05802, 2022.
Öngür Şimşek and Andrew G Barto. An intrinsic reward mechanism for efficient exploration. In Proceedings of the 23rd international conference Machine learning, pages 833 833-840, 2006.
Andrew Stout and Andrew Barto. Competence progress intrinsic motivation. In 2010 FEEE 9th International Conference Development and Learning, pages 257-262. IEEE, 2010.
Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1331 2008.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning. 3: 9-44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Anna Koop, and David Silver. On the role of tracking in stationary environments. In International Conference on Machine Learning, 2007.
Richard Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Volume 2, pages 761-768. International Foundation for Autonomous Agents and Multiagent Systems, 2011.
Richard S. Sutton, Michael H. Bowling, and Patrick M. Pilarski. The Alberta plan for AI research. arXiv preprint ar.iv:2208.11111111 2022.
Richard S. Sutton, Marlos C. Machado, G. Zacharias Holland, David Szepesvari, Finbarr Timbers, Brian Tanner, and Adam White. Reward-respecting subtasks for model-based reinforcement learning. Artificial Intelligence, 324: 324:104001, 2023. ISSN 0004-3702.
R.S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 12(()):11-211, 1999.
Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, and Mare G Bellemare. Benchmarking bonus- exploration methods on the arcade learning environment. arXiv preprint ar arXiv:1908 023388 2019.
Russ Tedrake. LQR-trees: Feedback motion planning on sparse randomized trees. 2009.
Andreas Ten Pas, Mareus Gualtieri, Kate Saenko, and Robert Platr. Grasp pose detection in point clouds. The International Journal of Robotics Research, 36( 1355-44)::455-147) 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, pages 5026-5000 2012.
Vivek Veeriah, Zeyu Zheng, Richard Lewis, and Satinder Singh. Grasp: Gradient Gradient-based affordance selection for planning, arXiv preprint arXiv:2202.04772 2022.
Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021
14

---

Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. Catgrasp: Learning category-level task-relevant grasping in clutter from simulation. In 2022 International Conference Robotics and Automation (ICRA), pages 6401-6408. IEFF, 2022.
Zheng Wen, Doina Precup, Morteza Ibrahimi, Andre Barreto, Benjamin Van Roy, and Satinder Singh. On efficiency in hierarchical reinforcement learning. In Advances Neural Information Processing Systems, volume 33, pages 6708 67 18, 2020. URL httpps://proceeccingg neurips cc/ -Papee pdf
paper_
Adam White. Developing a predictive approach to knowledge. PhD thesis, University of Alberta, 2015.
Martha White. Unifying task specification in reinforcement learning. In International Conference on PMLR, 2017.
Machine Learning, pages
Danfei Xu, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Deep affordance foresight: Planning through what can be done the future. In 202/ IEEE International Conference Robotics and Automation (ICRA), pages 0206-6213. IEEE, 2021.
Jialiang Zhao, Daniel Troniak, and Oliver Kroemer. Towards robotic assembly by predicting robust, precise and lask-oriented grasps. In Conference on Robot Learning, pages 1184-1194, PMLR, 2021.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Thu. Robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXhv:2009.12299) 2020.
15

---

Figure 6: Domains used for our experiments in Section 4 Enumerating left to right from the top-left: first screen of Montezuma's Revenge, MiniGrid FourKooms, Ant Medium Maze, Robosuite Door, Robosuite Lever, and Robosuite Slide.
Task Descriptions
Four Rooms. This task, which is part of the MINIGRID suite [Chevalier Boisvert al. 2018), is an adaptation of the four rooms problem presented in the paper introducing the Options framework [Sutton et al. 1999]. Observations are 84 84 images the folly-observable underlying state-space, which contains approximately 19 x 19 grid locations and possible orientations of the player. We defined 4 options whose termination conditions (subgoals) were to navigate to the center of the 4 rooms. So was the set of all empty grid locations.
CChevalierBooisvertt
Montezuma's Revenge. As standard in ALE [Bllemare et al. 2013]] observations are 84 x 84 images, action space set of discrete actions |Machado al. 2018]. We defined start states scattered across the first room where the player was on the ground (not jumping or falling), was not too close to the skull and did not already have the key in possession. We also defined 5 options whose termination conditions are reaching the left door, right door, bottom-right of the first screen, bottom-left of the first screen and getting the key.
Robosuite manipulation tasks. Three constrained manipulation tasks were used study the task- oriented grasping performance of our initiation set learning algorithms: opening a door, flipping lever, and sliding knob. The door task was originally implemented in ROBOSUITE /Zhu et al. 2020): the others were implemented and described in the work Rosen et al. [022]] All three are 1-DoF articulated objects which require making and sustaining contact manipulate The 52-dimensional observation space consists of the robot's proprioceptive state (joint position, joint velocity, end- effector position, gripper state, tactile data) as well as the object state (object pose, joint position, handle position). The action space employed is operational space control with variable impedance MMartin-Martinn Martún et al.) al.] 2019]] the agent controls the 6-DoF change in position and orientation of the end-effector. the 6-DoF change in stiffness, and 1-DoF gripper state. Episodes have maximum length of 250 steps. each task, So was a set of arm configurations establishing contact with the object; see Section
Ant Medium Maze The goal location is small region around (20, 20). A state is considered to satisfy goal if the two have euclidean distance of 0.5 units less R(s,g) ||s gll2 < 0.5. The agent is evaluated by rolling out the learned policy once every 10 episodes; during evaluation,
16

---

10
+1
a
2
a
$3
Figure 7: Illustration of the pessimistic bias in a tabular MDP. Action or has non-stationary effects, just like option in HRL. As or executed more, the probability of landing in Sy increases, otherwise it leaves the agent $1.
the agent starts from a small region around (0, 0), during training it starts at a location randomly sampled from the open locations in the maze. The task reward function is -1 for non-goal transitions for reaching the goal. Episodes last a maximum of 1000 steps. The
and a terminating reward training and evaluation protocol is identical to Bagaria et al. 2021a]. except for the fact that learn initiation sets over the full state.
B Pessimistic Bias: Connections to Non-Hieeaarhical MDPs
MDPs with actions that have non-stationary effects mirror hierarchical MDPs in which the option policies being learned online. Figure shows such MDP-there three states: $1, 82, $3. two actions a and Oz, and discount factor < 1. Action a takes the agent one state to the right, state 83 yields terminating reward of 1, reseting the state 81. Action O, has non-stationary effect: it either leaves the agent at state 81, or takes the agent to 83. The probability with which o takes the agent 83 evolves over time-the more it executed, the more consistently takes the agent to 53, similar to an option in HRL whose policy improves with execution.
A Q-learning agent quickly learns Q(#1,
and ((1,,
leading to greedy policy x(81)
a. But the agent executes or enough times, it will get higher return Q(s1, or) = 1 > Q(s1. a) y leading to the greedy policy (T$1) o₁, Vanilla Q-learning alone would not discover the higher value solution because does not account for non-stationary effect of o,; some form of exploration is needed for that.
C Details about Learning Algorithm
C.1 Accuracy Experiment
Algorithm D
is the pseudocode used for the experiments described in Section 4.1
Every episode,
every option is executed from every start state in Sg. The result of that execution is recorded as
ground-truth (t) and stored to later compute the size of the true initiation set |Y| = E. V.,((t)
[e)
If the start was predicted be inside the initiation set by the learning algorithm, then the trajectory generated by rolling out the option policy is used to update the policy and the initiation learner (e.g,
IVF, classifier). We report the agreement between the predicted initiations and the ground-truth accuracy measurement for that state-option pair. The fraction of start states in So that lead to success reported as a measurement of the size of the true initiation set.
C.2 Robot Manipulation
The task-specific grasping problem is typically phrased as identifying grasp poses y SE(33 that afford task success. practice, the difficulty of this problem is compounded by the fact that, for redundant manipulators, each grasp pose yields an infinite number of corresponding arm
17

---

Algorithm Accuracy/Size Experiment Procedure
Inputs:
Option termination conditions Soy Vo E O, start states So, number of episodes n_ episodes.
Outputs: Accuracy table A and Initiation Size table S; both map state-option pairs to a list of booleans.
Initialize yoal-conditioned policy we 8 - A.
Initialize Initiation Value Function (IVF) V. S x - Initialize binary classifier for each option I (s; 1b) - {0,1}. Initialize replay buffers for Rainbow Bд and IVF B₁.
Initialize buffers to store positive and negative examples for each option's initiation classifier. Initialize tables A and S as mapping each state-option pair to empty list. for episode E range in_episodess do for start state 80 E Sn do for option o O do
Reset simulator 80. Record option o's initiation decision X = (s0; 2)
Rollout option policy (o 9 3,) to get trajectory and next state s'. Record whether the option policy reached the goal Y = s' E 3o-
Record accuracy A[so][o] append (1 Y)).
Update ground -tunt size table S[so][o] append(())))) if predicted initiation X then
Add trajectory r to policy's replay BR-
Relabel trajectory т with initiation cumulant cp + {0, 1}.
Add relabeled trajectory to IVF's replay B₁.
Add trajectory 7 to o's positive/negative example buffer.
end if end for end for Sample minibatch and update To using Rainbow. Sample minibatch and update V using TD(0).
for option
Compute weights w(s) for all training examples using Equation] [
Update o's initiation classifier by minimizing weighted cross-entropy loss.
end for end for
configurations (solutions the inverse kinematics problem). Explicitly, this relation is governed by the manipulator's forward kinematics f C - SE(3) which maps (typically) 7-DoF configurations of the arm q E C to poses in Cartesian space. In practice, only subset of these configurations for given grasp pose enable successful manipulation (Schiavi et al., 2022]. As result, we task the initiation set learning algorithm with choosing start states directly from the space of arm configurations C.
We generate collision free grasp poses on each object using off-the- off-the-shelf grasp generation method GPG [Ten Pas et al.) [017]] and corresponding arm poses using FK ([mmes al.) [0022]. We chose to generate 50 grasp poses with 5 random inverse kinematics solutions each yielding a total of 250 starting configurations for each task.
The reward functions are implemented as progress toward 1-DoF object joint position goals. The agent receives reward when the current joint position exceeds its previous maximum in a given episode.
Reward function.
Parameterization. described Section A observations and controls the manipulator's end-effector pose and impedance. The learning algorithm employed is TD3 (Fujimoto al. 2018). Goal-conditioning is omitted these experiments as they have a single goal and single option.
the agent receives proprioceptive and object-state
18

---

C.3
Deep Skill Chaining
Deep skill chaining (DSC) [Konidaris and Barto, 2009 Bagaria and Konidaris 2020 proceeds recursively backward from the goal: the termination region of the first option is the task goal (Line 2 Algorithm 3) the agent first learns an option policy and initiation set for that option. Then, it learns another option whose termination region is the initiation set of the first option; this process continues until there is some option whose initiation set covers the start state. The result is a collection of options that can be sequenced to reliably reach the goal.
Since the initiation the way in which the initiation sets are learned: poorly estimated initiation sets can lead to subgoals that do not improve the agent's abilty reach the task goal.
of options are the subgoals for other options, the entire process is sensitive
Details about line of of A using the pure IVF approach, perform as many minibatch gradient updates the length of the option rollout; when using weighted classification, we recompute weights using Eq [ for all training examples and then proceed minimize weighted cross-entropy loss (3 epochs, batch size 128). When using classification (weighted or unweighted), we boost the contribution the minority class by the ratio of the size of the majority class to that of the minority class.
Saaimm@@iiiii differ based on the method used to learn the initiation set. When
Algorithm 2 Robust DSC Rollout
Inputs. Skill Chain O
Hyperparameters. Option horizon H
1: Initialize empty trajectory buffer B
2: for each timestep do
3: Select option using policy over options (s:))
4: Sample a goal for selected option: ~ 3o
5: Execute option policy (-(g)) in the environment
6: Add trajectory
- - (8i, O, 02 8;+1; g)
7: if final state ви reached goal g then
8: Add o's list of positive examples
9:
else
10:
Add to o's list of negative examples
11: end if
12: Refit option o's initiation classifier
13: Add to replay buffer and update To using TD3
14: end for
15: return B = Ug (St, Ot, ae; S1+1, 9t)
Algorithm 3 Robust DSC Algorithm
Inputs. Start state Su. Goal region g.
Initialize global option og such that Ioa (.)
2: Initialize goal option o, such that Bo,
3: Initialize skill chain with {o,}
4: for each episode do
5: transitions = ROLLOUT(O)
6: if so & I,, Yo O then
7: Create new option
8: Add to skill chain
9: end if'
end for
Picking a goal for option execution. Line of Algorithm 2 samples a goal from the option' termination region. To implement this sampling procedure, we consider the option's parent w in the chain (the parent option is the one whose initiation set is being targeted by the current option o). We enumerate the positive examples used train IT and pick the goal with the highest initiation probability under the current option. This process is done iteratively backward from the goal: the first
19

---

goal is the task goal, the next one is the positive example closest (in terms of highest IVF value) to the task goal and on.
D Additional Manipulation Experiments
Initiation set accuracy and true size are computed during training by performing an analogous procedure Algorithm [ Periodically, the manipulator was reset to each candidate start state and the initiation prediction was compared with the outcome of a policy rollout. Initiation set accuracy is visualized in Figure Sa The methods generally converge to similar accuracy. True initiation set size
is plotted in Figure 8b size increases with optimism and correlates with success rates.
Door
Lever
Slide
1.00
Accuracy
0.75
0.50
0.25
0.00
2000
4000 Episode
2000
4000
5000
10000
Episode
Episode
Optimistic Weighted
Optimistic Binary Weighted
GVF
Optimistic GVF
Baseline Binary
(a) Initiation set accuracy for manipulation domains.
Door
Lever
Slide
200
150
Size
100
50
0
2000
4000
2000
4000
0
5000
10000
Episode
Episode
Episode
Optimistic Weighted
Optimistic Binary
GVF
Optimistic GVF
Weighted
Baseline Binary
(b) Initiation set size for manipulation domains (out of 250 states).
Figure 8: (a) Accuracy of the learned initiation sets in the robot manipulation domains. (b) The size of the "true" initiation sets measured by performing Monte Carlo rollouts of the option policy.
Ablation: Optimism
We ablate role of optimism in initiation set learning in the manipulation experiments of Section 4.2 the results are visualized in Figure To Count-based bonuses are computed as described in Section 3.3 The optimistic bias is critical for the success of the Weighted algorithm in the SLIDE task. in which only a small fraction of potential grasps are amenable to task success. For the Binary classification baseline, optimism dramatically deteriorates performance in the DOOR task, but yields large improvement the LEVER domain.
Figure shows that the optimistic bonus slightly hurts in FOURROOMS but is beneficial in MON- TEZUMA'S REVENGE. This likely because FourRooms is much simpler problem with smaller state-space, thereby not demanding exploration bonus.
20

---

Nontecumaffevenehhrrramssaaa v4
Acoracy
Norms Lned Sie of Tree 212 Set
ACCLENC
Noraalzed Size of TuE
0.6
3.5
0.5
0.3
2.5
126
150
50
10g
153
200
50
150
200
rood Optanoro
Wenghttd
tue
Weighted No Optimart
Werghted
Figure 9: Optimism ablation in MINIGRID- FOURRRID-FFURRRR ROOMS and MONTEZUMASREVENGGE All curves are averaged over all state-option pairs and 5 random seeds.
Optimism
False|
Door
Optimism False Lever
Optimism False]
Slide
0.8
Successs Rale
0.6
0.4
0
1000
2000
300D
4000
5000
1000
2000
3000
4000
5000
2000
4000
6000
8000
10000
Oplimism True Door
Optimism True Lever
Oplimism True Slide
0.8
Suczess Rate
0.6
0.4
1300
2000
300D
4030
0000
1000
2000
3000
4000
5000
2000
4000
6030
8000
10000
Episode
Episode
Episode
Binary
Weighted
Figure 10: Optimism ablation ROBOSUITE, Solid line represents mean success rate aggregated over four seeds; shaded region shows standard error.
Initiation Sets In Four Rooms
Figure [ the initiation set and IVF of the option targeting the center the top-right room in SANIGRID-FOURROOO
These figures show that the initiation of the option expands over time to reflect the growing competence of the option policy.
G Hyperparameters
Rainbow was used for policy learning in Section 4.1 TD3 was used in the other experiments. Their hyperparameters (Tables 2 and (5) were not tuned and are either identical to the original paper implementation borrowed from Bagaria et al. [22 The bonus scale (described in Sec 3.3
21

---

Figure 11: Initiation set (left) and Initiation Value Function (IVF) (right) of the option targeting the center of the top-right room (subgoal position is (14, 14)). Plots in the top row visualize the initiation after the 1st episode of training, those in the bottom row are after 200 episodes of training.
Env
Horizon H
FourRooms
50
Montezumas Revenge
200
Robosuite
250
Ant-Maze
200
Table 1: Option Horizon Ho for each environment.
tuned over the set {0.05, 0.1, 0.25, 0.5, 1.0], the best performing hyperparameters are listed in Table 3
G.1
Option Horizon
Options operate over a fixed time horizon H. (see Section 3.1). These values (Table ! were picked based on overall time limit (max_steps per episode) for each domain and we did not tune them.
22

---

Parameter
Value
Replay buffer size
10c
Critic Learning rate
3 10 10-4
Actor Learning rate Optimizer
3 10
Adam
Target Update Rate
5 10
Batch size
256
Iterations per time step
1
Discount Factor
0.99
Output Normalization
False
Policy Improvement Prior
False in ROBOSUITE; True otherwise
Table 2: TD3 Hyperparameters for ROBOSUITE and DSC Experiments
Method
Bonus Scale
Optimistic Binary
Optimistic GVF
0.5
Optimistic Weighted
0.5
Table 3: Exploration Hyperparameters for Robosuite Experiments.
Parameter
Value
Replay buffer size
10
Replay start size
1024
Learning rate
10-4
Table 4: Rainbow Hyperparameters for Accuracy Experiments
Parameter
Value
Learning rate
10 4
Optimizer
Adam
Replay buffer size
10a
Batch size
32
Threshold
0.5
Target network update rate
5
10-3
Table 5: IVF Hyperparamters
23


Reviews:
[{"review_id": "WpCaYKJV8n", "reviewer": "Reviewer_RLJn", "summary": "The paper discusses three difficulties in effectively learning option initiation sets in reinforcement learning --- these are non-stationarity, temporal credit assignment, and pessimism --- and proposes a method that addressed these difficulties. The proposed method is evaluated empirically in a variety of environments. The results are positive.", "questions": "Have you considered alternatives to the weighting function in Equation 1?\n\nNote: I have read the rebuttal. Thanks to the authors for answering my question and their additional comments.", "limitations": "I do not see a discussion of limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper makes a good contribution on a topic of interest. It is clear and well structured. The algorithmic approach is well motivated and sound. Experimental analysis is conducted on a range of environments and, importantly, includes the use of the proposed method within a specific skill discovery algorithm, with positive results. Overall, I believe the paper is a useful addition to the literature.", "weaknesses": "It would be useful to see a discussion on the computational cost of the proposed approach. \n\nThe experimental evaluation can be broadened to include a detailed analysis of the utility of the weighting function (equation 1) and the discussion could include whether any other alternatives would be sensible. In addition, it would be useful to see experimental results with the proposed approach in the context of skill discovery methods other than deep skill chaining.\n\nAdditional Comments:\n\nIn the abstract, and later in the paper, the authors state that they identifiy and address three difficulties in effectively learning option initiation sets: non-stationarity, pessimism, and temporal credit assignment. In the introduction, I do not see any reference to temporal credit assignment. \n\nIn line 201, the bonus reward is the change in the value function. This was explored by Simsek & Barto (ICML, 2006, An Intrinsic Reward Mechanism for Efficient Exploration) earlier than the papers cited here.\n\nIn figure 2, the second plot from the right: either the plot title or the label of the vertical axis is wrong. The title is \"size of the true initiation set\" but the axis labels range from 0.2 to 0.9. The same holds for the last plot in the figure.\n\nThe writing is generally well structured and clear but at times repetitive. Reducing the repetition would allow the authors to include in the main paper (rather than the supplementary material) the results on initiation set accuracy and size in the robot manipulation environments, which are informative. If space cannot be found in the main paper for these plots, the results can still be summarised and discussed in the main paper. These plots show key performance variables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review. We are glad that you found our paper to be clear and our approach and experiments to be sound. \n\n> It would be useful to see a discussion on the computational cost of the proposed approach.\n\nThe computational cost of the IVF approach is very similar to that of the baseline approach of training a binary classifier. The weighted classifier is more expensive since the IVF is used to weigh each of the training examples for the classifier. However, it is worth noting that our new approaches lead to faster experiments in Section 4.3: when the agent reaches the goal in a given episode, that episode is shorter than 1000 steps, so the overall experiment finishes in a lot fewer environment steps than the baseline. In other words, although we add some compute, we drop sample complexity dramatically.\n\n> Have you considered alternatives to the weighting function in Equation 1?\n\nEq 1 was the simplest way to incorporate the IVF into a weighting function. Experimentally, we verified that the weighting function adapted to a changing policy over time. Finally the weighted classifier was accurate and performed well in the skill-discovery setting, so we did not feel the need to try more complex weighting functions. \n\n> it would be useful to see experimental results with the proposed approach in the context of skill discovery methods other than deep skill chaining (DSC).\n\nWe chose DSC because (a) it shows strong performance in sparse-reward continuous problems and (b) its performance is sensitive to initiation set learning. It is possible that other skill-discovery algorithms would also benefit from better initiation set learning, but we leave that analysis to future work.\n\n> In figure 2, the second plot from the right: either the plot title or the label of the vertical axis is wrong.\n\nWe apologize for the confusing axis label: the y-axis is the *normalized* size of the initiation set---it is the fraction of states inside the ground-truth initiation set (described in Section 4.1), so it is a real number between 0 and 1. We will change the axis labels to make this more clear.\n\n> This was explored by Simsek & Barto (ICML, 2006, An Intrinsic Reward Mechanism for Efficient Exploration) earlier than the papers cited here.\n\nYou are right, thank you, we will certainly cite this paper in the camera-ready.\n\nThank you again for the positive review and thoughtful suggestions.\n"}, {"review_id": "s8evIDjQJZ", "reviewer": "Reviewer_zsHA", "summary": "This paper studies the problem of learning initiation sets, which are important in hierarchical reinforcement learning for indicating where a policy will succeed, and identifies three main challenges that arise with existing methods to learn these sets. These three challenges are data non-stationarity, temporal credit assignment, and pessimism. The authors introduce the Initiation Value Function (IVF), which predicts the probability that an option will succeed from that state, which is learned using off-policy policy evaluation and adapts as the policy improves. IVF is effective when used as a input to a weighted classifier and additionally includes states for which the option policy is likely to improve in order to address pessimistic bias. Experiments in Minigrid, Montezuma's Revenge, Robosuite, and a maze navigation problem are shown, and in these environments, the method improves upon the baseline of using a General Value Function. ", "questions": "See weaknesses. Additionally:\n-In the Figure captions (or elsewhere), can you clearly define what the each method is? Figure 2, for example, does not have a method called \"IVF\" so it's unclear which is yours. Also, it'd be helpful to add y-axis labels for all the plots.  ", "limitations": "The authors have not addressed limitations. It would be helpful to include discussions of limitations. ", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well-written and has empirical evaluations on a wide range of domains. \n- The paper brings up some good discussion on three challenges to address for learning initiation sets in hierarchical RL.", "weaknesses": "-The paper involves adding a lot of complicated tricks into a hierarchical method, and it's unclear if they're worth it for the often small improvements. I'm not sure if people will want to adopt this method. It's also unclear which of the challenges is most important, and which of the tricks added is most important. Can you add some more analysis/discussion on this?\n- It's a bit strange how the authors emphasize the important of exploiting temporal structure to learn IVF, in order to be more effective than a classifier, but then just uses IVF to weight a classifier. Seems like a lot of effort to learn a value function just to weight the classifier. It's not clear if the temporal aspect is actually meaningful in the classifier. For example, how does this compare to just using binary classifier that has mixup regularization or other types of heavier regularization along with the optimism? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you found our paper to be well written and our empirical evaluations to be diverse. We hope that we can change your mind about the potential value of our paper to the HRL community.\n\n> In the Figure 2, can you clearly define what each method is? \n\nWe accidentally labeled the IVF approach as “GVF”, we will fix that for the camera-ready if the paper is accepted. \n\nHere is a brief description of the different lines in Figure 2:\n1. **Baseline Binary.** Binary classifier used to learn the initiation set. This is what is used in essentially all prior work and is the only baseline method in this figure.\n2. **GVF.** Threshold the Initiation Value Function (IVF). \n3. **Optimistic GVF.** Threshold the sum of IVF and an exploration bonus.\n4. **Weighted.** This is a weighted binary classifier where the weights are assigned by the IVF.\n\nAgain, (1) is a baseline; (2), (3), and (4) are our contributions. \n\n> The paper involves complicated tricks with small improvements. It is unclear which of the tricks added is most important.\n\nWe are not sure what the reviewer means here: our proposed methods are simple and they lead to substantial performance increase. Figure 2 aims to ablate the different contributions, are there specific “tricks” whose contribution you are unsure about?\n\nFurthermore, we did not randomly add tricks to see if they increase accuracy/reward. Instead, we are introduced modifications specifically designed to address an important, known issue in HRL: the collapse of initiation sets [1, 2].\n\n> It's a bit strange how the authors emphasize the importance of exploiting temporal structure to learn IVF, but then just uses IVF to weight a classifier.\n\nFirst, we show that thresholding the IVF learns better initiation sets than the baseline binary classification approach, so we do not *just* use the IVF to learn a classifier. Our second contribution is the IVF-weighted classifier, which adds a temporal dynamics-aware target to combine the strengths of the value function and the classification approaches.\n\nIn *Montezuma’s Revenge*, the pure IVF and weighted classifier approaches increase accuracy by $50$% and $40$% respectively over binary classification; in *AntMediumMaze*, they are the difference between solving the task reliably ($>80$% success rate) and not solving the task at all ($0$% success rate), so our approaches are certainly worth the effort.\n\nMixup regularization (or anything similar) is a good suggestion, but it cannot learn temporal structure, TD learning (or temporal bootstrapping in general) is required for that. To show the benefit of using TD for learning the IVF, **we have included a new experiment in the accompanying rebuttal PDF**: Figure B shows that even when a trajectory does not reach the goal, TD unsurprisingly assigns positive credit to states that are near the goal, thereby aiding more sample efficient learning.\n\n## References:\n[1] Harb, Jean et al. When waiting is not an option: Learning options with a deliberation cost. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.\n\n[2] Bagaria, Akhil, et al. \"Robustly learning composable options in deep reinforcement learning.\" Proceedings of the 30th International Joint Conference on Artificial Intelligence. 2021."}, {"review_id": "3qsKMzRJ18", "reviewer": "Reviewer_pGKy", "summary": "This paper addresses the problem of learning initiation sets in hierarchical reinforcement learning. Initiation sets define the states from which an option can be executed successfully. However, learning initiation sets is challenging because they depend on the option policy, which changes as the agent learns. Previous approaches suffer from data non-stationarity, pessimistic bias, and a lack of exploitation of temporal structure. To address these issues, the authors propose the use of the Initiation Value Function (IVF) as a predictive value function that estimates the probability of option success. The proposed method is evaluated in various domains, including grid worlds, robot manipulation, and maze navigation, and shows improved performance compared to existing methods.", "questions": "1) Is it possible to use TD learning with cross-entropy loss from the logits instead of regression? Would this approach outperform TD learning with regression?\n\n2) Regarding the \"Initiation Set Accuracy\" plot (Fig 2), it would be valuable to have a similar plot that compares the different methods while choosing a single option policy learning shared by all methods, focusing solely on the variation in the initiation set learning. The current plot, although informative, compounds the evaluation of initiation set learning with the resulting option policy learning.\n\n3) The abstract mentions that the termination condition is typically identified first. However, the algorithm could potentially work even when the termination policy changes. Could you provide insights into the performance of the method if the termination state is learned in parallel? Alternatively, how restrictive is the assumption?", "limitations": "see weaknesses", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) Novelty: the paper presents a novel method for learning initiation sets from off-policy trajectories by incorporating the temporal structure of the MDP. It also introduces an original technique to mix leverage the strengh of classification togfether with TD learning.\n\n2) The evaluation of the proposed method is comprehensive, with separate experiments conducted to systematically test and validate each claim. \n\n3) The performance of the proposed method, particularly when incorporating the bonus correction, show significant improvements compared to existing methods.", "weaknesses": "1) There are certain parts of the paper that lack clarity, such as the algorithm for selecting the goal for the DSC method. Additionally, Figure 3 is challenging to interpret due to the excessive use of similar colors.\n\n2) The paper primarily focuses on the DSC algorithm and does not explore the application of the proposed method to other option frameworks where termination states are also learned simultaneously. Expanding the evaluation to different option frameworks would have strengthened the paper's overall contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and suggestions. We are glad that you found our work to be novel and our experimental approach to be comprehensive and systematic! \n\n> There are certain parts of the paper that lack clarity, such as the algorithm for selecting the goal for the DSC method.\n\nThe algorithm for selecting goals for DSC is discussed in Section B.3 of the Appendix. Additionally, [1] discuss the nuances of goal-selection in DSC. Briefly, the subgoal state for the current option is the highest probability (IVF prediction from the current state $\\mathcal{V}_o(s_t)$) positive example used to train the parent option’s initiation set.\n\n> Figure 3 is challenging to interpret due to the excessive use of similar colors.\n\nYou are right, thank you, we will improve the presentation for the camera-ready if the paper is accepted.\n\n> Expanding the evaluation to different option frameworks would have strengthened the paper's overall contribution.\n\nWe chose DSC because of its explicit dependence on initiation set learning. Having said that, the primary focus of our experiments is evaluating how accurate the initiation sets are, and we added a secondary experiment to show the effect of that on downstream learning. There are many option discovery methods and we expect this approach will help all of them similarly.\n\n> Is it possible to use TD learning with cross-entropy loss from the logits instead of regression? Would this approach outperform TD learning with regression?\n\nYes, that is a good suggestion: using distributional RL might further improve initiation set learning. We used the traditional TD error in this paper because of its simplicity and because distributional RL is not the focus of our paper.\n\n> The current initiation set accuracy plot (fig 2), although informative, compounds the evaluation of initiation set learning with the resulting option policy learning.\n\nInitiation set learning is extensively studied and well understood when the option policy is fixed; in that setting, it is just binary classification. While we agree that learning initiation sets in tandem with policies complicates evaluation, that is exactly the focus of our paper. We introduced the \"initiation set size\" metric alongside accuracy specifically to understand the interplay between initiation set learning and policy improvement, which naturally arises during continual/online reinforcement learning. \n\n> Could you provide insights into the performance of the method if the termination state is learned in parallel? Alternatively, how restrictive is the assumption?\n\nDSC uses goal-conditioned policies to deal with non-stationary termination/subgoal regions, and our proposed methods seem to work very well with DSC, so in that sense we already have an experiment that does what you are asking for (Figures 5 and 7). Other than that, if the termination condition of the option changes, its policy will likely degrade; since our initiation value function (IVF) is learning using off-policy policy evaluation, it will adaptively lower the policy’s probability of success. Over time, as the policy becomes more competent at the new subgoal, the IVF’s predictions will also increase. In other words, as long as the policy learning adapts to the non-stationarity, we fully expect our initiation set learning to also adapt.\n\nThanks again for your thoughtful reviews! If you have any questions/concerns, please let us know.\n\n[1] Bagaria, Akhil, et al. \"Robustly learning composable options in deep reinforcement learning.\" Proceedings of the 30th International Joint Conference on Artificial Intelligence. 2021."}, {"review_id": "z9ug2NB2wN", "reviewer": "Reviewer_y3Ji", "summary": "The paper presents an approach for efficiently learning initiation sets of options for approaches that automatically learns temporal options for hierarchical planning. The paper introduces the concept of initiation value function (a probability measure for successful option executions for a given imitation set) and uses IVF to learn initiation sets of options while option policies are also changing. Lastly, the approach learns a neural network for predicting the probability of successful option execution using a classifier. The authors evaluate their approach in variety of settings including discrete spaces (minigrid settings) as well as continuous robotics settings. ", "questions": "Please refer to the weakness mentioned above.", "limitations": "The authors do not explicitly address the limitations of the paper. \n\nIn my opinion the rest of the paper (apart from not explicitly highlighting the limitations and the weakness mentioned about) is written nicely.  ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is well written. Details are highlighted nicely. \n\n- Authors have thoroughly presented related work and mentioned differences with existing works. \n\n- The experiments are thorough. Sufficient details are provided for experimental setup. \n\n- The paper successfully motivates the readers about the technical challenges in learning initiation sets of the option using the traditional binary classifier based approach and provides clear references to what part of their approach tackles each problem. This makes understanding the paper and following the approach lot easier. ", "weaknesses": "The only weakness I see of the paper is lack of clarity on how the approach is used with an option discovery approach. The given approach requires a policy to compute IVF and learn a neural network. However, for problems that require option discovery, do not have these options along with a learned policy. It is unclear from the paper how was this made possible without having a fixed policy. \n\nIt would be helpful to highlight how the approach was configured to be used with DSC. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted that you thought that our paper was well written, thorough and easy to follow! \n\nWe understand your desire for more details about how our methods are incorporated into the option discovery algorithm, DSC. Section B.3 (including Algorithms 2 and 3) in the Appendix provides more details about the DSC algorithm and the experimental setup. A lot of these details are part of the DSC algorithm, which is not our contribution, so we deferred the discussion to the Appendix. However, we are happy to add some more details to the main paper so that Section 4.3 is easier to follow.\n\nBased on your suggestion, we will also add a discussion about the limitations of our method to the camera-ready.\n\nThank you once again for your thoughtful review! "}, {"review_id": "1mHcBGzffG", "reviewer": "Reviewer_uZYk", "summary": "The paper considers an HRL setting with options (including sub-goal reward functions) in which termination conditions for options are provided, and the goal is to learn the option policies and initiation sets simultaneously. The main focus is on learning of initiation sets in an online scenario where option policies are changing. The authors highlight three possible issues with learning the initiation set with classifiers in such an online setting. Firstly, the data is non-stationary because the policy keeps changing during learning. Secondly, classifiers ignore the temporal structure of the RL task. Thirdly, the difficulty of expanding the initiation set arises due to the agent not being initialized (and consequently not visiting) states outside of the initiation set.\n\nTo alleviate the first two issues, they propose combining a classifier with initiation value function (IVF) weights (expected probability of successfully completing the task from a state). They additionally propose addressing the third issue by adding an exploration bonus to IVF. The experiments compare the performance with these proposed changes against ablations without (some of) these changes. The last experiment includes a comparison against a prior method where the proposed method is used in an existing HRL algorithm.", "questions": "- In the abstract you mention that option policy and initiation set must be learned simultaneously? Could you elaborate on why this is the case?\n- In line 92 you mention that \"initiation function is used in the context of primitive actions\". What does context of primitive actions mean here?\n- Regarding data non-stationarity part in section 3. Wouldn't it be possible to discard old training samples based on the probability of current policy executing old trajectories?\n- Is it possible to use IVF with thresholding directly instead of learning a weighted classifier? If yes, why haven't you included this baseline in the paper?\n- Why does the option use internal reward function instead of a global one?\n- Shouldn't reaching \"success\" (termination states) be something that the option policy is rewarded for?\n- If the option maximizes internal reward function, is the problem of finding policy $\\pi_0$ pretty much an MDP $<S,A,R_0, T, \\gamma>$?\n\nComments:\n- line 70 - you cite Feudal RL with \"usually assume that all options apply everywhere\" afaik, this approach does not use options and in fact was proposed before options\n- line 109 - It is probably better to cite the full paper Options of Interest: Temporal Abstraction with Interest Functions (Khetarpal et al. 2020) from AAAI 2020 as a reference to IoC\n\n", "limitations": "Authors do not sufficiently discuss the limitations/weaknesses of their method", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Originality: \n  - Submission proposes an original approach for learning of initiation functions in the option setting that is considered. However, since inferring success probability is a problem that should span other parts of RL, I am not certain whether such approach isn't related to other works from different branches of RL.  \n\n- Clarity: \n  - Submission is well organized (proper separation into sections and subsections etc.)\n  - Data non-stationarity, pessimistic bias and temporal structure parts in section 3 are well explained \n \n- Significance: \n  - Submission tackles a task of learning initiation sets. This task is an important problem that is being researched by the HRL community", "weaknesses": "- Quality: \n  - I am not fully convinced that the submission is technically sound. In particular:\n    - I think that the problem setting described in the paper is somewhat confusing. The background section mentions SMDP and the options framework but adds internal option reward function and discount (lines 66-67). As far as I am aware, these are not standard part of the options framework. From the lines 147-150 I understand that the option policy is trained to maximize this internal reward function. This makes the problem different from the one that is considered in the original options paper [1], which relies uses a single global reward function but the paper does not elaborate on this. It is also unclear to me, what the relationship between the success condition cumulant $c_0$ and the option reward function is in section 3.1. The former is not used to learn option policy but shouldn't reaching \"success\" (termination states) be something that the policy is rewarded for?\n    - When the IVF is introduced the timescale (discount) is set to 1 and the horizon is capped at $t=H_0$. However, there is no discussion about the effect of the $H_0$ on learning. As far as I can tell this seems to be a very important hyperparameter especially when discount is set to 1, since based on this value the size of initiation set should increase (for bigger $H_0$) or decrease (for smaller $H_0$)\n    - The paper does not include qualitative visualization of the learned initiation sets and IVF in the easy environment (four rooms). I think that such visualization should be in the paper to evaluate and justify proposed changes and help to understand their effect on learned initiation sets and IVF.  \n    - It is unclear from the experiments whether (and in which cases) proposed adjustments help. The plots (in particular Figure 2), seem inconclusive to me. \n    - Authors do not discuss the limitations/weaknesses of their method\n    - In Figure 5., the prior work baseline seems to perform very poorly. While I understand that the setting was slightly adjusted, I am not sure whether the baseline was properly tuned.\n\n- Clarity:\n  - The comparisons are sometimes not well explained (what are GVF baseline in Fig. 2?)\n\n[1] Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that you found our method to be original and the problem setting to be important. We hope that we can clarify some misunderstandings and convince you about the value our proposals to the HRL community.\n\n## Weaknesses\n\n> Inferring success probability is a problem that should span other parts of RL\n\nWe agree the problem is very general and we hope that our solution is also adopted by other parts of RL. However, we are not sure how that is a weakness of our work.\n\n> The background section mentions SMDP and the options framework but adds internal option reward function and discount, which makes the problem different from the one in the original options paper, which relies on a single global reward function.\n\nIt is very common in HRL for options to have their own internal reward functions [1]; this was discussed in the HRL survey paper [3] (whose formalism we adopt in our Background section) and has been the case in option discovery research for several decades [2]. We can add more formalism around this in the background section if you want, but we hope that this is not a reason for rejection.\n\n> What is the relationship between the success cumulant and the option reward function in section 3.1?\n\nThe option reward function is used to learn the option policy--it depends on the environment and is designed to enable sample efficient policy learning. For e.g, in Robosuite, the reward function is dense and depends on the configuration of objects in the scene. However, the initiation cumulant is fixed and environment agnostic---it is always $+1$ for reaching the goal and $0$ otherwise. However, this cumulant should not always be used for policy learning; for e.g, you might want negative rewards for states to avoid or shaped rewards for sample efficiency.\n\n> When the IVF is introduced the discount is set to 1 and the horizon is capped at H0. However, there is no discussion about the effect of the horizon on learning.\n\nIn our experiments, we treated the horizon as a hyperparameter but forgot to include them in the appendix. Here are the values: MiniGrid $50$, MontezumaRevenge $200$, RoboSuite $250$, AntMaze $200$ (all methods used the same horizon). These numbers were picked based on the overall time limit (max_steps/episode) for each domain and we did not tune them. We apologize for the omission and will include this information in the camera-ready if the paper is accepted.\n\nBased on your suggestion, **we ran an experiment in which we swept over the option horizon $H_o$** in MiniGrid-FourRooms and recorded the accuracy and size of the learned initiation sets; the results are included in the accompanying PDF. Option horizon $H_o$ has a predictable effect on accuracy: as long as $H_o$ is not too small ($\\geq 40$ timesteps), accuracy is high. Your intuition about the relationship between horizon and initiation set size is *mostly* correct: $H_o$ sets an upper bound on the size of the initiation set, the final size is also dependent on the competence of the learned policy and the connectivity of the underlying MDP.\n\n> The paper does not include qualitative visualization of the learned initiation sets and IVF in four rooms\n\nThank you for your suggestion, **we visualized the learned classifier and IVF** for MiniGrid-FourRooms; see Fig A in the accompanying PDF.\n\n> The plots (in particular Figure 2), seem inconclusive to me.\n\nFigure 2 shows that our method achieves \\~$25$-$30$% increase in accuracy in MiniGrid and \\~$50$% increase in accuracy in Montezuma’s Revenge; these results suggest that our techniques **conclusively** and significantly outperform the baseline approach to learning initiation sets. To be clear, the line labeled “Baseline Binary” is the only one that is not our contribution and corresponds to the method used in almost all prior work; the other three lines are novel to our paper.\n\n> (Fig 5) I am not sure whether the baseline was properly tuned.\n\nWe used the author implementation of baseline DSC. We reproduced their reported results when the initiation was learned over the $(x, y)$ location of the ant, but the performance drops dramatically when the full state is used for initiation learning. We tried different values of gestation period ${5, 10}$ and learning rates {1e-3, 1e-4, 1e-5}; none of these settings could solve AntMediumMaze. Our algorithms (Weighted Clf and Initiation GVF) were also implemented in the author codebase for an apples-to-apples comparison. The DSC codebase is open-source, so if you want to try the experiment yourself, we are happy to share the run commands and configs we used.\n\n## Questions\n\n> Why should the option policy and initiation set be simultaneously learned?\n\nThe initiation set evaluates the reachability of the current policy, so as the policy changes in the continual learning setting, so should the initiation set, i.e, they depend on each other.\n\n> Is it possible to use IVF with thresholding directly instead of learning a weighted classifier? If yes, why haven't you included this baseline in the paper?\n\nWe *do* use the IVF by itself – since it always outputs values between 0 and 1, it is much easier to threshold than the value function used for action-selection (see Section 3.1). We should clarify that the thresholded IVF is *not* a baseline, it is one of the two algorithms we propose. That line was mistakenly labeled as “GVF” in Fig 2, but it should read “IVF”.\n\n> If the option maximizes internal reward function, is the problem of finding policy pretty much an MDP?\n\nYes, that is correct! In fact, M White et al 2017 formalized options as RL subtasks.\n\n## References\n\n[1] Precup, Temporal abstraction in RL, PhD Thesis (2000).\n\n[2] McGovern & Barto. Automatic discovery of subgoals in reinforcement learning using diverse density (2001).\n\n[3] Barto & Mahadevan. Recent Advances in Hierarchical Reinforcement Learning (2003).\n\n[4] Sutton et al. Reward-respecting subtasks for model-based reinforcement learning (2022).\n"}]

Author rebuttal:
We are grateful to all the reviewers for their thoughtful reviews and constructive feedback. We are happy to see that all reviewers found our work to be novel and our paper to be well written and clear in its presentation. We hope that our rebuttal helps resolve misconceptions and remaining reservations about our work.

## Misconceptions

Some reviewers seemed to think that the GVF approach in Fig 2 was distinct from the IVF approach in Section 3.1; they are the same – this approach thresholds our Initiation Value Function (IVF). Furthermore, thresholding the IVF is not a baseline, it is one of our two contributions (in addition to the weighted classifier). This is because the prevalent way to use value functions as initiation sets is to threshold the option value function that is also used for action-selection. But as we discussed in Sections 2 and 3.1, that is not a good idea because it either leads to difficulties in policy learning or involves complicated and fragile normalization tricks.

## Common Concern

A common cause for concern was our use of DSC as a skill-discovery algorithm in Section 4.3. We used DSC for two reasons:
- It is a recent option discovery algorithm that performs well in sparse reward control problems and DSC learns options whose subgoals are initiation sets of other options, i.e, its performance is heavily dependent on learning good initiation sets. Because learning high quality initiation sets is central to DSC, it makes for a good test-case for our proposed ideas.
- Furthermore, the experiments preceding that section are meant to evaluate initiation set accuracy directly. This one is to demonstrate initiation set accuracy's effect on downstream learning. Of course that depends on the skill discovery algorithm, of which there are many, so we picked a representative and recent one.

## Limitations of our Work

Reviewers suggested that we discuss limitations of our method: our proposed cumulant for the IVF in Section 3.1 only works for goal-reaching options, which although is quite general, it is not universal. For example, if the option’s task is to maximize the velocity of a robot and there is no specific target velocity, then we could not write down a 0/1 cumulant that faithfully describes that subtask. We will add this discussion to the camera-ready if our paper is accepted.

## New Experiments 

- Reviewer uZYk asked how option horizon impacts initiation sets, Figure C in the PDF aims to answer this question.
- Reviewer zsHA was unclear about the contribution of TD to the IVF learning problem: Figure B in the PDF shows how TD is exploiting temporal structure for more sample efficient learning.
- Reviewer uZYk asked for visualizations of the initiation sets and the Initiation Value Function (IVF) in FourRooms; this is now included in Figure A of the PDF.





Meta-review: This paper proposes a method for learning an initiation set for options framework. After identifying challenges in learning initiation sets, the paper introduces Initiation Value Function (IVF) which approximates the success probability of a given option in a given state and proposes to use it as a weight for training a binary option initiation classifier. The paper also proposes two types of exploration bonus when determining the initiation set to mitigate the pessimistic bias while learning options. The proposed method was evaluated in terms of the classifier's accuracy, option learning efficiency, and option discovery efficiency in various domains and tasks.  

Most of the reviewers appreciated the structure of the paper (e.g., identifying/addressing challenges) and using a variety of domains for evaluation. However, several reviewers were confused by what exactly is the baseline or the proposed method (e.g., GVF), how exactly the evaluation was conducted (e.g., true initiation set, how to select horizon), why DSC is used for option discovery, and how the proposed method is combined with DSC, and so on. Although a few of them were clarified during the rebuttal period, two reviewers remained unconvinced even after the rebuttal, while one of the reviewers was willing to defend the paper for acceptance. 

Since the reviewers have not come to an agreement, I read through the paper. Although I also found that the presentation of the empirical result is not great as pointed out by the reviewers, I believe that this paper has quite interesting ideas and findings that are worth to be presented. Thus, I recommend accepting the paper. 

I strongly encourage the authors to improve the clarity of the empirical result section, by reflecting most of the reviewers' comments. Here is my additional suggestion:
- There are unnecessarily many combinations of the proposed method in each plot (Weighted, GVF, Optimistic in Fig 2, 3, and Weighted, GVF in Fig 5), and the explanation of each term is somewhat spread over the text. Instead, I would rather pick one variation and compare it against the baseline in the main figure and make a separate figure for ablation where many variations are compared. 
- I do not see why it is necessary to show two variations of the exploration bonus instead one. They are not only distracting but also never compared with each other unless I missed it. I would suggest showing only one throughout the paper for clarity (and showing the other in the appendix if needed).
Decision: Accept (poster)