NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
Authors: Myunsoo Kim
Hayeong Lee
Seong-Woong Shim
JunHo Seo
Byung-Jun Lee
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Unsupervised Learning
Skill Learning
Primary area: reinforcement_learning
TL;DR: Enhance policy learning in downstream tasks by learning terminated skills through state-action novelties.

Abstract: Intelligent agents are able to make decisions based on different levels of granularity and duration. Recent advances in skill learning enabled the agent to solve complex, long-horizon tasks by effectively guiding the agent in choosing appropriate skills. However, the practice of using fixed-length skills can easily result in skipping valuable decision points, which ultimately limits the potential for further exploration and faster policy learning.
In this work, we propose to learn a simple and effective termination condition that identifies decision points through a state-action novelty module that leverages agent experience data.
Our approach, Novelty-based Decision Point Identification (NBDI), outperforms previous baselines in complex, long-horizon tasks, and remains effective even in the presence of significant variations in the environment configurations of downstream tasks, highlighting the importance of decision point identification in skill learning.

NBDI:
A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
Myunsoo Kim 11
Hayeong Lee" ! Seong- Woong Shim
JunHo Seo
Byung-Jum Lee! 12
Abstract
the agent has to make, thereby alleviating the challenges that RL faces in long-horizon sparse reward tasks.
Intelligent agents are able to make decisions based different levels of granularity and duration. Re- cent advances in skill learning enabled the agent to solve complex, long-horizon tasks by effec- tively guiding the agent in choosing appropriate skills. However, the practice of using fixed-length skills can easily result in skipping valuable de- cision points, which ultimately limits the poten- tial for further exploration and faster policy learn- ing. In this work, we propose to learn a simple and effective termination condition that identi- fies decision points through state-action novelty module that leverages agent experience data. Our approach, Novelty-based Decision Point Identifi- cation (NBDI), outperforms previous baselines in complex, long-horizon tasks, and remains effec- tive even in the presence of significant variations in the environment configurations of downstream tasks, highlighting the importance of decision point identification in skill learning.
Due to the advantages of temporal abstraction, there has been active research on developing hierarchical RL algo- rithms, which structure the agent's policy into a hierarchy of two policies: a high-level policy and low level policy. The option framework (Sutton, 1998) was proposed achieve temporal abstraction by learning options, which are high- level actions that contain inner low level policy, initiation set and termination conditions. Termination conditions are used to figure out when to switch from one option to another, enabling the agent to flexibly respond to changes in envi- ronment or task requirements. While the option framework can achieve temporal abstraction without any loss of perfor- mance when the options are optimally learned, it is usually computationally challenging to optimize for the ideal set of options within complex domains.
In this case, the skill discovery framework, which aims to discover meaningful skills (fixed-length executions of low-level policy) from the dataset through unsupervised learning techniques, has been used cently, notable progress has been made in skill-based deep RL models, showing promising results in complex envi- ronments and robot manipulations (Pertsch et al., 202 la; Hakhamaneshi et al., 2021; Park et al., 2023). However, the use of fixed-length skills and the absence of appropriate termination conditions often restrict them from making de- cisions at critical decision points (e.g., crossroads), which can result in significant loss in performance. While there have been some studies incorporating the option framework into deep RL as is, the algorithmic complexity and unsta- ble performance in large environments limit its widespread adoption (Kulkarni et al., 2016; Hutsebaut-Buysee et al., 2022). In this paper, we present NBDI (Novelty-based Decision Point Identification),, a simple state-action novelty-based decision point identification method that allows the agent to learn terminated skills from task-agnostic demonstrations. In this context, the term task-agnostic refers to the collec- tion of trajectories from diverse set of tasks, excluding the
alternative. Re-
1. Introduction
The ability to make decisions based on different levels of granularity and duration is one of the key attributes of intelli- gence. In reinforcement learning (RL), temporal abstraction refers to the concept of an agent reasoning over a long horizon, planning, and taking high-level actions. Each high- level action corresponds to sequence of primitive actions, or low-level actions. For example, in order to accomplish task with robot arm, it would be easier to utilize high-level actions such as grasping and lifting, instead of controlling every single joint of robot arm. Temporal abstraction sim- plifies complex tasks by reducing the number of decisions
'Equal contribution 'Department of Artificial Intelligence, Korea University, Scoul, Republic of Korea *Gauss Labs Inc., Scoul, Republic of Korea. Correspondence to: Byung-Jun Lee <byungjunlee @korea.ac.k>..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Code: https: / /ttpttgtttttt/////////////t/bhht com/ www/ku-ooeeeeeeeeoooooeeeee. ku-dml ab/ /NDDI

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
one we are specifically interested in (see Appendix E for visualizations). Identifying critical decision points promote knowledge transfer between different tasks and stimulate exploration by closely connecting different areas in the state space (McGovern & Barto, 2001; Menache et al., 2002; Simşek & Barto, 2004). For example, detecting doorways between rooms useful regardless of the specific task at hand. We demonstrate the straightforward applicability of our method to the bkill-based deep RL framework, and illus- trate how it can lead to improvements in decision-making To summarize, our three main contributions are as follows: (i) For the first time, we propose a skill termination condi- tion for aagoosic demonstrations that remains effective even when the environment configuration of complex, long- horizon downstream tasks undergo significant changes. (ii) We present a novel method in reinforcement learning, which is the identification of state-action eovelty-based critical decision points. Furthermore, we demonstrate that execut- ing terminated skills, based on state-action novelty, can substantially enhance policy learning in both robot manip- ulation and navigation tasks. (iii We conduct extensive experiments, especially with other possible termination con- ditions, to provide insights for future research in the field of skill termination learning.
to achieve temporal abstraction. For example, skill dis- covery (Gregor al., 2016; Achiam et al., 2018; Mavor- Parker et al., 2022; Park et al., 2023) and skill extraction (Yang et al., 2021; Singh et al., 2020: Pertsch et al., 2021b; Hakhamaneshi et al., 2021) frameworks have proven to be successful in acquiring meaningful sets of skills. Especially, Pertsch et al. (2021a) showed promising results in complex, Long-horizon tasks with sparse rewards by extracting skills with data-driven behavior priors. The learned prior enables the agent to explore the environment in a more structured manner, which leads to better performance in downstream tasks. However, we believe that their performances are greatly constrained by the use of fixed-length skills, which restricts them from making decisions at critical decision points. There are prior works introducing variable-length skill extraction methods. Salter et al. (2022) focus on learn- ing an option-level transition model leveraging predictabil- ity to compress offline behaviors into options that terminate at bottleneck states. However, due to its model-based de- sign, the extracted skills are not suitable for transferring to downstream tasks with significantly different environment configuration. Jiang et al. (2022) is an option framework based model that learns both options and termination con- dition from the task-agnostic demonstrations in terms of minimum description length (Rissanen, 1978). As it serves as suitable benchmark for evaluating our approach, we later compare our method to Jiang et al. (2022) in Section 6.4.
2. Related Works
Option Framework One major approach of discovering good options is to focus on identifying good terminal states, or sub-goal states. For example, landmark states (Kaelbling, 1993), reinforcement learning signals (Digney, 1998), graph partitioning (Menache et al., 2002; Şimşek et al., 2005; Machado et al., 201 7a;b), and state clustering (Srinivas et al., 2016) have been used to identify meaningful sub-goal states. (Digney, 1998; Simsek et al., 2005) and (Kulkarni et al., 2016) focused on detecting bottleneck states, which are states that appear frequently within successful trajecto- ries, but are less common in unsuccessful trajectories (e.g., state with access door). (Şimşek & Barto, 2004) tried to identify access states, which are similar to bottleneck states, but determined based on the relative novelty score of predecessor states and successor states. Access states are found based on the intuition that sub-goals will exhibit a relative novelty score distribution with scores that are fre- quently higher than those of non sub-goals. These studies motivated us to search for states with meaningful properties to terminate skills. However, these methods frequently face challenges in scaling to large or continuous state spaces.
Novelty- RL Novelty has been utilized in reinforce- ment learning for various purposes. Depending on its design, novelty can be used for curiosity- exploration (Burda et al., 2018; Pathak al., 2019; Sekar et al., 2020), or data coverage maximization (Bellemare et al., 2016; Hazan et al., 2019; Seo et al., 2021). It has been also used to identify sub-goals in discrete environments (Goel, 2003; Şimşek & Barto, 2004). However, to the best of our knowledge, there has been no research that has utilized state-action novelty for identifying decision points in the context of deep RL.
3. Background
Markov Decision Process (MDP) MDP is mathemat- ical framework to model decision making problems with discrete-time control processes. It is defined by a tuple (S, A, P, R, where S denotes a state space, A denotes set of actions the agent can execute, P(s'|s, a) denotes a transition probability, R(s, a) is reward function and y is discount factor. In MDP, the probability of transitioning to a future state depends solely on the current state, which is known as the Markov property. Given MDP, we aim to find an optimal policy x² that maximizes the expected discounted sum of reward E, 1E R(s, a)). The state value function V* (s) and the action value function Q" (8, (s.a)
Skill-based deep RL As extending the classic option framework to high-dimensional state spaces through the adoption of function approximation is not straightforward a number of practitioners have proposed acquiring skills, which are fixed-length executions of low-level policies,
2

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
( 4g.1) 0.05
Open the Slide Cabinet
15 = 0.
Open the Saide Cabinet
x(5 4n) 0.01
X 4 0.43
X($ 0.05
( ag) 0.33
Move the Cottle
Flip Light Switch
Turn the oven Knob
Open the Hinge
Cabinet
x( (g+1)) 0. 0.
( (R-+1) 0. 04
Figure I. Visualization of an example of critical decision points in the kitchen environment. High state-action novelty can be found in states where subtask has been completed, and multiple subsequent subtasks accessible. If termination occurs a high state-action novelty point, the agent retains multiple plausible options. For example, after moving the kettle. it may choose to open the sliding cabinet turn the oven knob (Leftt). Similarly, after flipping the light switch, the agent may proceed to open either the sliding cabinet or the left-hinged cabinet (Right).
denote the conditional expectation of discounted sum of re- ward following policy и.
skills. In particular, we use intrinsic curiosity module (ICM) (Pathak et al., 2017) as our state-action novelty estimator (more details in Section 6.1). However, any state-action nov- elty estimation mechanism that measures the joint novelty of state-action pairs can be used for our approach.
The option framework (Sutton, 1998) is one of the first studies to achieve temporal abstraction in RL. The option framework is composed of two major elements: a meta-control policy and a set of options O. An option is defined as (I, n, B), where I C S defines an initiation set, π. S x A + [0, 1] defines a policy, and 3 [0, 1] defines termination condition. The policy chooses the next action, until the option is terminated by the stochastic termination condition B. Once the option terminates, the agent has opportunity to switch to another available option at the termination state. Options usually refer to low-level polices that are promised to be good only for a subset of the state space. Thus, the presence of an appropriate initiation set I and termination condition 8 is crucial for the agent's overall performance.
Option Framework
4.1. State-action Novelty-based Decision Point Identification
Our proposed method classifies a state-action pair with high joint state-action novelty as decision point. more insight- ful perspective on this choice can be obtained by breaking down the novelty estimator (Equation 1). By interpreting joint novelty ((s, a) as the reciprocal of joint visitation count N(s, a)², we can decompose state-action joint nov- clty into the product of a state novelty and a conditional action novelty. The proposed method combines the strength of both novelty estimates.
Any MDP with fixed set of options can be classified as Semi-Markov Decision Process (SMDP) (Sutton, 1998). SMDP (Bradtke & Duff, 1994) is an extended version of MDP for the situations where actions have different exe-
x(s,a) N(s,a)
1
1
N(s)
N(ass) x(s)
(I)
X(als)
state novelty conditional action novelty
cution lengths. It serves as the foundational mathematical framework for many hierarchical RL algorithms, including the option framework.
The state novelty x(5) will seek for novel state, which refers to a state that is either challenging to reach or rare in the dataset of agent experiences. As the skills are derived from the same pool of experiences that we use to estimate novelty, a high state novelty implies a potential lack of di- verse skills to explore neighboring states effectively. Thus, increasing the frequency of decision makinn in such unfa- miliar states will lead improved exploration and broader coverage of the state space when solving downstream tasks.
4. Simple and Effective Identification of Decision Points
The option framework aims to achieve temporal abstrac- tion by learning good options, and good options be learned through the identification of meaningful sub-goal
states (Menache et al., 2002; Şimşck & Barto, 2004), i.c., the critical decision points. In this work, we propose to use state- action novelty to identify critical decision points for skill termination, which leads to the execution of variable-length
A conditional action novelty x(a]s) will seek for novel
Note that although the motivation is based pseudo counts, any novelty estimator based on 8, a can be used for our method.
3

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
12%
[A]
8D-
(A]
100-
Prodiction Error
600
(C)
Prediction
(A]
(B)
40
(A)
2000 4000
8000 trajectories
1000
2000
3000
State-Action
(D)
(R)
(F)
State Peste-Action 25 trajectories
(0)
[4) IF
(a) Prediction error Maze
(b) Prediction error Block Stacking
Figure 2. Visualization of prediction error of ICM in maze and block stacking environment. Note the same offline data that is used to train ICM was used to compute this prediction error. (A). (B) and (C) are the state-action pairs with the highest prediction error. while (D), (E) and (F) are the ones with the lowest. Critical decision points such as crossroads or states involving block manipulation are typically associated with high prediction error. contrast, low prediction errors are observed in states that are less important for decision-making.
4.2. Termination Improvement from State-action Novelty-based Terminations
We provide an alternative interpretation on the potential benefits of identifying decision points based state-action novelty. While maximizing skill length is advantageous in terms of temporal abstraction, extended skills can result in suboptimal behavior, especially when the skills are de- rived from task-agnostic trajectories. Such suboptimality of extended skills (or options) can be theoretically quantified using the termination improvement theorem (Sutton, 1998).
Figure 3. The relative frequency of termination improvement oc- currences (left), conditional action novelty (middle), and state novelty (right) in small grid with three different goals. Higher percentile colors indicate relatively greater number of termina- tion improvement occurrences, higher conditional action novelty, and higher state novelty. Further details on the visualization proce- dure are provided Appendix I.1.
Theorem 4.1. [Termination Improvement, (Sutton, 1998), informal] For any meta-control policy p. on set of options O, define new set of options O', which is a set of options that we can additionally choose to terminate whenever the value of state V"(s) is larger than the value of state given that keep the current option o, Q" o)- With pl, which has the same option selection probability as μ but over new set of options O', we have VP (s) 2 VP()).
action. With the state conditioning, action novelty will be high state where a multitude of actions have been frequently executed. For example, unlike straight roads, crossroads provide the agent with options move in multi- ple directions. In such states, the agent may need to perform different actions to accomplish the current goal, rather than solely depending on the current skill. This necessity arises because the current skill may have been originally designed for different goals, making it potentially less than ideal for the current goal. Guiding the agent to make more decisions in such states increase the likelihood of solving the task at hand, ultimately accelerating the policy learning.
The termination improvement theorem implies that should terminate option when there are much better alter- natives available from the current state.
To identify the states where termination improvement oc- curs, we plotted the relative frequency of termination im- provement occurrences in small 8 8 grid maze with three different goal settings (Figure 3 (left)). It shows that termination improvement frequently occurs in states where diverse plausible actions exist. In states with single avail- able option, V"(s) would be equal to (s,ss.) On the other hand, as more actions/options are plausible, (" o) would exhibit a broader range of values, thereby increasing the likelihood satisfying Q" o) < V*()). When the skills (or options) are discovered from diverse trajectories (e.g., trajectories gathered from diverse set of goals), termi- nation improvement is typically observed in states where multitude of actions have been executed, such as crossroads.
In the kitchen environment, shown in Figure 1, high state-action novelty x(s, a) tends to occur in states where a subtask has been completed. For example, after moving the kettle, the agent may choose to either open the sliding cabinet or turn the oven knob. Similarly, after completing the subtask of flipping the light switch, the agent has the option to open either the left hinge cabinet the right slide cabinet.
In sequential manipulation tasks, such critical points are valuable because completing one subtask grants access to multiple other subtasks.
However, terminating skills based on the termination im- provement theorem can be challenging when the down- stream task unknown, it requires (*( o) and V*(s)

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
Algorithm Downstream RL with NBDI
of two major steps. First, we train the state-action novelty module and then the low-level policy using task-agnostic demonstrations for skill extraction. Next, we perform on- line reinforcement learning with the learned variable-length skills to solve an unseen task.
trained low-level policy π (al2, s), trained novelty module p(3)s, s, a), the maximum skill length
Input:
Output: the high-level policy /40
1: Initialize replay buffer D, parameters of high-level pol-
icy 0
2: +
Problem Formulation For training the state-action nov- elty module, we assume access to task-agnostic, expert-level demonstrations of states and actions the form of N tra- jectories D = {T' {(st: )=)) i=o These trajecto- ries are collected across diverse set of tasks except for the one we are specifically interested in (see Appendix E for more details). Since we do not make any assumptions about rewards or task labels, our model can leverage real- world datasets that can be collected at lower cost (e.g.. autonomous driving and drones).
3: for each iteration do
4: 21 ~ (x)
5: for k = 0, 0, do
6:
de+* TT(++k Re, S(tk)
7:
Be+k= ~ -((e+x/(stte 0t-k)
8: Take action ar+k, observe Tetks S1+k+1
9: BL+ = or = H then
10:
Break
11:
end if
end for
it t+*
13:
5.1. Unsupervised Learning of State-action Novelty Module
14:
D (st, Ze, Fz, 11
15: Update (z(2)) using RL with samples from D
16:
end for
In the process of unsupervised learning, our goal pre- train the low-level policy π 2, s) and the state-action nov- city module p(8)s,)) a). We define skill
Z em- bedding of state-action pairs = {(s;, a:) [=)][==+++ and ter- mination conditions , = (8.) The termination conditions / are Bernoulli random variables that decide when to stop the current skill. Through the classification of actennctionn pairs demonstrating significant novelty X(s, a), 3 are trained predict the critical decision points. The point at which novelty is considered significant varies de- pending the environment. In downstream tasks, the skill being executed will be terminated either when 3 = 1 is sampled or when the maximum skill length H is reached. In all experiments, we set H = 30. The low-level pol- icy is trained following standard practices (Pertsch et al., 2021a; Hakhamaneshi et al., 2021). During the training of the low-level policy, the deep latent variable model receives randomly sampled experience from the training dataset, along with a termination condition vector provided by the state-action novelty module. The model is trained to recon- struct the corresponding action sequence and its length (i.e., the termination point) by maximizing the evidence lower bound (ELBO) (see Appendix B for details).
to be computed in advance with the skills extracted from the downstream task trajectories. Thus, by leveraging the data collected across a diverse set of tasks, we can use con- ditional action novelty as tool for pinpointing the states where multitude of plausible actions can be taken (Fig- ure 3 (middle)). Through experiments, we also found state novelty to be useful in terminating skills, as it encourages the agent to sufficiently explore unfamiliar parts of the state space (Figure (right)). As a result, we propose to use state-action novelty, which combines the strength of both conditional action novelty and state novelty as in Equation as our skill termination condition. In Section 6, we also demonstrate how these different novelty measures, utilized as termination conditions, lead to different performance outcomes.
5. Learning Termination Conditions through State-action Novelty Module
Our goal is to improve the learning of new complex and long-horizon task by identifying critical decision points through a state-action novelty module. While fixed-length skills have been mostly considered for temporal abstractions in recent studies (Pertsch et al.. 2021a; Hakhamaneshi al., 2021), utilizing fixed-length skills can easily skip valuable decision points, ultimately reducing the opportunities for further exploration.
5.2. Reinforcement Learning with Skill Termination Conditions
In downstream learning. our objective is to learn a skill pol- icy (2(2))) that maximizes the expected sum of discounted rewards, parameterized by 0. The pre-trained low-level pol- icy <[]]2,)) decodes skill embedding z into series of actions, which persists until the skill is terminated by the predicted termination condition B.
In this work, we propose to use state-action novelty as a termination condition to effectively capture critical decision points and execute terminated skills. Our approach consists
The downstream learning can be formulated as a SMDP
5

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
NBDI (Ours)
BC+SAC
IQL+
SPRRL
SAC SAC+Novylty
1.5
Completed Subtasks
0.3
0.3
Success Rate
Success Rate
0.2
0.2
0.1
0.1
0.5
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.5
1.0
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Environment steps (1
(a) Maze 40 x 40 Navigation
(b) Maze 30 30 Navigation
(c) Sparse Block Stacking
(d) Kitchen Environment
ENVIRONMENT
SAC
SPIRL
NBDI
IMPROVEMENT OVER SPIRL(
MAZE 30x30 (Success rate)
0.04
0.2410.01 0.13
84.62
MAZE 40x40 (Success rate)
0.01=0.01
0.09=0.02
0.25=
177.78
SPARSE BLOCK STACKING (Stacked Blocks)
0.14
0.67
1.12:10.16
67.16
KITCHEN (Completed Subtasks)
0.0
3.0.1..0
3.67:10.43
22.33
Figure 5. Performances of our method and baselines in solving downstream tasks. The shaded region represents 95% confidence interval across five different seeds. The last column of the table below illustrates the percentage improvement of method over SPiRL.
which is an extended version of MDP that supports ac- tions of different execution lengths. We aim to maxi- mize discounted sum of rewards ExeT i(s 21) where T is set of time steps where we execute skills, i.e., T = {0, ko, ko + k ko + ki + ... and k; is the variable skill length of i-th executed skill. The RL learning loop is described in Algorithm 1. In downstream RL tasks, we use SAC (Haarnoja et al., 2018) to update the high-level policy. More details of the learning procedure are in Appendix B.
for training ICM. We visualized the states of the state-action pairs with high prediction error ((A), (B), (C) and low pre- diction error ((D), (E). (F)) in maze environment (Figure 2a). It can be seen that high prediction error can be typi- cally seen in states where we have multitude of plausible actions ((A), (C)) or rare state configuration (B). Note these characteristics correspond the conditional action novelty and state novelty illustrated in Figure 3, and leads to high state-action novelty as in Equation 1. On the other hand, low prediction error can be seen in states where we do not have any of these properties (DD), (E), (F)). When the agent encounters a maze with an unseen goal, it would have no way of knowing which direction would lead to the goal. Therefore, encouraging the agent make more decisions at such crossroads would effectively connect different areas within the maze, ultimately promoting exploration.
6. Experiments
We design the experiments to address the following ques- tions: (i) Does learning state-action novelty-based termina- tion condition improve policy learning in unseen tasks? (ii) How does each component of state-action novelty contribute to the identification of critical decision points? (iii) Have we successfully identified the decision points that match our intuition? Additional studies are in Appendix A.
Figure 2b shows prediction error of state-action pairs in sparse block stacking environment, which is a complex robotic simulation environment that has no clearly defined subtasks. In this environment, the agent needs to stack blocks on top of each other. We can see that high predic- tion error occurs in states where the agent has multitude of plausible actions ((A), (B), (C)). When the robotic arm is po- sitioned above a block, it must choose between descending to lift that block or moving towards other blocks. Likewise, when the robotic arm is holding a block, it needs to deter- mine which block to stack it onto. This characteristic also corresponds to the conditional action novelty as depicted in Figure 3, and results in high state-action novelty as in Equation 1. Similar to the maze environment, low predic- tion error occurs in states where we do not have multitude of plausible actions, or rare configuration ((D), (E), (F)).
1.1. State-action Novelty Module
We utilize ICM (Pathak et al., 2017) to calculate state-action novelty for both image-based and non- con-image-baat based obser- vations. While ICM is typically recognized for providing intrinsic motivation signals to drive exploration in online RL, we found it to be an effective state-action novelty esti- mator when it is pre-trained with offline trajectory datasets. Since ICM takes in state-action pair to predict next state representation, it would have high prediction error for sparse state-action pairs in the offline dataset. Figure 2 illustrates the prediction error of state-action pairs from 25 randomly selected trajectories within the offline trajectory dataset used
6

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
NBDI (Ours)
SPIRI
Relative Novelty
LOVE
(Ourst
NBDI-(5)
NBDi-yialst
0.30
Completed Subtasks
Completed Subtasks
4
4-
Stacked Blocks
Success Rate
0.25
1.0
0.20
2
0.15
2
0.10
0.0
0
1-
0.05
0.0
0.5
1.0
1.5
2,0
Enviranment steps (1M)
Environment steps (1M)
0.00
0
(a) Comparison with other terminating criteria in sparse block stacking (left) and kitchen (right) environments
(b) Comparison with other variable skill length methods in 40 maze (left) and kitchen (right) environments
Figure 6. Performances of our method and baselines in solving downstream tasks. The shaded region and error bar represents 95%
confidence interval across five different seeds.
Thus, our findings validate that ICM can serve as an effec- tive state-action novelty estimator when pre-trained with offline trajectory datasets.
ing. Flat Offline Learning w/ Finetuning (BC+SAC, IQL+SAC): The supervised behavioral cloning (BC) pol- icy and Implicit Q-Learning (IQL) (Kostrikov al., 2021) policy that are trained on offline data and subsequently fine- tuned for the downstream task using SAC. Fixed-length Skill Policy (SPiRL): The agent that learns a fixed-length skill policy (Pertsch et al., 2021a) by leveraging prior expe- rience. This comparison demonstrates the benefit of critical decision points identification through actionn novelty. NBDI (Ours): The agent that learns terminated skill policy through state-action novelty X(s, a)). It learns state-action novelty based termination distribution p(3(2, s) to predict skill termination at current step. State Novelty Decision Point Identification (NBDI-x(s)): The agent that learns a terminated skill policy through state novelty. To exclusively assess the influence of the novelty type, we distilled the state-action novelty module used in NBDI into a separate network, x()), which solely depends the current state. Conditional Action Novelty Decision Point Identification (NBDI-x(als)) The agent that learns a terminated skill pol- icy through conditional action novelty x(s.a)
6.2. Environments
Two navigation tasks (Mazes sized 30 30 and 40 x 40) and two simulated robot manipulation tasks (Kitchen, Sparse block stacking) are used to evaluate the performance of NBDI. All of these environments are challenging sparse reward tasks, with continuous state space and continuous action space.
large set of task-agnostic agent experiences is collected from each environment to pre-train the low-level policy and the state-action novelty module. For downstream tasks with significantly different environment configurations (maze, sparse block stacking), we used agent-centered cropped im- ages to extract consistent structural information from task- agnostic demonstrations. We introduce the train/transfer domain similarity for the environments used in our experi- ments in Appendix E. To demonstrate the effectiveness of our method on complex downstream tasks involving signif- icant configuration changes, evaluate the models on maze environment with an unseen layout and large-scale block stacking environment with a greater number of blocks in randomized positions. Both settings differ significantly from the task-agnostic offline datasets. Further details of the environments and the task-agnostic data collection pro- cedure are in Appendix H.
X(s) where x(s)
is the distilled state novelty module used for NBDI-x(s).
Implementation details of the baselines are in Appendix
In both the robot manipulation tasks and the navigation tasks, executing terminated skills through state-action nov- city (NBDI-x(s, a)) facilitates convergence toward more effective policy (Figure 5). While NBDI manages to accom- plish all four subtasks in the kitchen environment, others never achieves the maximum return. Furthermore, as shown the table in Figure 5, NBDI surpasses SPiRL even within a challenging robotic simulation environment where there are clearly defined subtasks (Sparse block stacking). How- ever, SAC, NAC+Novelty BC+SAC and IQL+SAC show poor performance due to their lack of temporal abstraction, which limits their ability to explore unseen tasks effectively.
6.3. Results
We use the following models for comparison: Flat RL (SAC): Soft Actor-Critic (Haarnoja al., 2018) agent that does not leverage prior experience. This compari- son illustrates the effectiveness of temporal abstraction. SAC+Novelty: Flat RL that uses state-action novelty as intrinsic rewards. This comparison demonstrates the impor- tance of incorporating statc-action novelty in skill learn-
In alignment with our motivation for acteonction novelty, conditional action novelty (NBDI-x(aja s)) appears to play crucial role in identifying decision points (Figure 6a).

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
100
Percentile
(a) VSsi)))ion(PPiRL)
(b) Visitation(BBDD)
(c) Skill (SPRLL)
(d) Skill (NrDi)a,((BDI)
Figure 7. Visualization of decision points made by SPiRL and NBDI the maze environment. We sampled 100 trajectories for cach trained policy to observe the points at which they make decisions. Higher percentile colors suggest relatively greater number of visitation frequencies and termination frequencies.
Table 1. Success rate of NBDI and SPiRL with offline trajecto- ries generated by mediocre-level policy with weighted Gaussian random noise in maze environment.
DATASET QUALITY
NBDI
SPIRL
92 B3 Dataset Usage
STOCHASTIC BC 0.5)
28%
0%
STOCHASTIC BC (o 0.75)
22%
0%
Hioden Layer Contiguration
Figure 8. Illustration of the impact of varying the width and depth configuration of hidden layer and dataset usage state-action novelty estimation. (Left) shows how varying hidden layer con- figurations (depth and width) influences the estimated state-action novelty (box plots) and the total number of parameters (blue line). The x-axis lists different hidden layer setups. (Right) illustrates how varying the proportion of dataset usage affects novelty esti- mation.
While it appears that terminating skills solely based on state novelty doesn't 't lead to better performance, combining it with conditional action novelty (resulting in state-action novelty) leads to better exploration and better convergence.
Figure 7 compares decision points made by SPiRL and NBDI in the maze environment. This result provides the answer to our third question. While the SPiRL agent makes decisions in random states, our model tends to make deci- sions in crossroad states states that are unfamiliar. For instance, in the lower-right area of the maze, SPiRL shows periodic skill terminations due to its fixed-length of skills, whereas our approach tends to make decisions in states char- acterized by high conditional action novelty or state novelty.
cantly different environment configuration (Maze 40 x 40). Implementation details of baselines are in Appendix I.4.
As shown in Figure 6b, LOVE performs comparably to SPiRL the maze environment, demonstrating its applica- bility continuous action spaces. However, its underperfor- mance compared
method suggests that the variable skills learned by LOVE do not effectively generalize to downstream tasks with significantly different environment configuration. Moreover, we observed that LOVE's per- formance significantly declines as the complexity of the action space increases (kitchen). NBDI's high performance compared these baseline methods illustrates the need for more effective termination condition for skill models in solving such challenging tasks.
6.4. Comparison to Other Variable-length Skill Extraction Methods
We compare the performance of NBDI, relative novelty (Şimşek & Barto, 2004) and LOVE (Jiang al., 2022) in the maze and kitchen environment (Figure 6b). Relative novelty identifies termination condition based on the assumption that sub-goals typically show a relative novelty score dis- tribution with higher scores than those of non-sub-goal -ooauu.-goals. LOVE an option framework based model that learns both options and termination condition from the task-agnostic demonstrations in terms of minimum description length. Since LOVE is designed for discrete action space, used an MLP layer as the action decoder to handle continuous actions. Furthermore, since LOVE was only evaluated in downstream tasks that has the same MDP as the data collec- tion environment (except the reward function), we evaluated whether it can transfer to downstream tasks with signifi-
6.5. Critical Decision Points with Suboptimal Data
We investigate how the quality of offline data affects the performance of our approach. We trained a behavior cloning (BC) policy on expert-level trajectories generate mediocre quality demonstrations. We additionally added weighted Gaussian random noise (a) to actions of BC policy to add stochasticity to the generated dataset. Table shows that even in less challenging goal setting compared to Figure
8

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
1.0
SMPPL SNNLLNBCI
op20
ep100
cp300
ep500
SIMPL
0.143=00.03
0,593 191
0.667-0.199 193
0.990.10.006
SIMPL+NBDI
0.560=0.121
0.900=00221
0.980=0.011
0,993-100003
0.8
Table 2. Comparison of success rates in the maze environment during the target task learning phase of SiMPL with standard errors over three random seeds.
0.4
0.2
Figure 9, the result shows that the extracted variable-length skills allows the meta-policy to better promote knowledge transfer between different tasks, helping the meta-policy in combining the skills to complete complex tasks. We report mean success rates on the 10 meta-training tasks across 3 different seeds with standard errors. Furthermore, during the target task learning phase, the meta-policy learned through our approach leads to significantly better sample efficiency on the unseen target task. Results Table 2 indicate that our approach can be effectively integrated with a broader class of skill-based methods that leverage task-agnostic demon- strations.
Eptsodes
Figure 9. Comparison of success rates in the maze environment during the skill-hased meta-training phase of SiMPI. across 10 meta-training tasks (10 random goal locations). The shaded regions indicate standard errors over three random seeds.
7, SPiRL fails to reach the goal, while NBDI achieves success rate of 28% and 22%. However, as the policy gen- erating the trajectory becomes more stochastic (Table 1), gathers data primarily around the initial state, leading to an overall reduction in the scale of prediction errors. Thus, we can see that the level of stochasticity in the dataset in- fluences critical decision point detection, which remains a limitation of our work. Visualizations of decision points and prediction error with suboptimal dataset are in Appendix C.
8. Conclusion
We present NBDI, an approach for learning terminated skills through a state-action novelty module that leverages of- fline, task-agnostic datasets. Our approach significantly outperforms previous baselines in solving complex, long- horizon tasks and shows effectiveness even under signifi- cant changes in environment configuration of downstream tasks. A promising direction for future work is to use novelty-based decision point identification to learn variable- length skills in offline skill execution (Ajay et al., 2020; Hakhamaneshi et al., 2021).
6.6. Model Capacity and Critical Decision Point Detection
To assess how the model capacity and dataset utilization influence critical point detection, we varied the width and depth of the neural network across different settings. Figure 8 (left) shows that the estimated state-action novelty by ICM not affected by the number of parameters used to train the model. Figure 8 (right) demonstrates that the scale of the estimated state-action novelty remains consistent even as the dataset size decreases. We can see that our proposed approach for detecting critical decision points is robust to various number of parameters or size of datasets.
Acknowledgments
This work was partly supported by Institute of Informa- tion & communications Technology Planning & Evaluation (IIP) grant funded by the Korea government (MSIT) (No. RS-2022-I1220311 Development of Goal-Oriented Rein- forcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects, No. RS-2024-00457882, Al Research Hub Project, and No. DS-2001-11190077 Arti- ficial Intelligence Graduate School Program (Korea Univer- sity)). the InTtttne of Information & Coummunications Technology Planning & Technology Research Center) grant funded by the Korea govermenn(Minnstrr of Science and ICT>LITP-2025-RS. 2024-00436857), the NRF AAS-202-00045 162) funded by the Ministry of Science and ICT, Korea, BK21 Four project of the National Research Foundation of Korea, and the Na- tional Research Foundation of Korea (NRF) grant funded by the Korea govermment (MSIT) (2TT))R--222--0- 00560367,,,, and the IITP under the Artificial Intelligence Star Fellowship
7. Applying NBDI to Skill-based Meta Reinforcement Learning
To explore the broader applicability of our approach, we investigated its compatibility with a baiel-based approach, SiMPL (Nam et al., 2022) that leverages task-agnostic demonstrations solve challenging long-horizoo sparse- reward meta-RL tasks. Specifically, we apply our method during the skill extraction phase to learn variable-length skills in place of fixed-length skills. In the maze environ- ment, we randomly sampled 10 goal locations for meta- training. The table below compares the success rate of meta- policies trained with SiMPL (fixed-length skills) and our method during the meta-tran veta-training phase across episodes. In

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
support program to nurture the best talents (IITP-2025-RS- 2025-02304828) grant funded by the Korea government (MSIT).
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Impact Statement
This paper presents work whose goal to advance the field of Reinforcement Learning. There are many potential soci- etal consequences of our work, none which we feel must be specifically highlighted here.
Hakhamaneshi, K., Zhao, R., Zhan, A., Abbeel, P., and Laskin, M. Hierarchical few-shot imitation with skill tran- sition models. arXiv preprint arXiv: 107.0881,, 2021.
Hazan, E., Kakade, S., Singh, K., and Van Soest, A. Prov- ably efficient maximum entropy exploration. In Interna- tional Conference on Machine Learning, pp. 2681-2691, PMLR, 2019.
References
Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. Variational option discovery algorithms. arXiv preprinl arXiv:180.11099, 2018.
Higgins, I., Matthey, L.,, Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta- vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2016.
Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for acceler- ating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 98)):1335-1780 1997.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based ex- ploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Hutsebaut-Buysse M., Mets, K., and Latré, S. Hierarchical reinforcement learning: A survey and open research chal- lenges. Machine Learning and Knowledge Extraction, 4 11):172-221, 2022
Bradtke, S. and Duff, M. Reinforcement learning methods for continuous- time markov decision problems. Advances in neural information processing systems, 7, 1994.
Jiang, Y., Liu, E., Eysenbach, B., Kolter, J. Z., and Finn, C. Learning options via compression. Advances in Neural Information Processing Systems, 35:21 14-11194-1199, 2022.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Ex- ploration by random network distillation. arXiv preprint arXiv: 12894, 2018
Kaelbling, L. P. Hicrarchical learning in stochastic domains: Preliminary results. In Proceedings of the tenth interna- tional conference on machine learning, volume 951, pp. 167-173, 1993.
Digney, B. Learning hierarchical control structures for mul- tiple tasks and changing environments. In Proceedings of the fifth conference on the simulation of adaptive behav- ior: SAB, volume 98, pp. 295. Citeseer, 1998.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint 1rXhv:2110.06166 2021.
Fu. J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, 2021.
Kulkarni, T. D., Saeedi, A., Gautam, S., and Gershman, S. J. Deep successor reinforcement learning. arXiv preprint arXiv: 6006.02366666... 2016.
Gallé, M. Investigating the effectiveness of bpe: The power of shorter sequences. In Proceedings the 2019 confer- ence on empirical methods in natural language process- ing and the 9th international joint conference on natural language processing JEMNNP--JCNNIP), pp. 1375-1381, 2019.
Machado, M. C., Bellemare, M. G., and Bowling, M. laplacian framework for option discovery in reinforce- ment learning. In International Conference on Machine Learning, pp. 2295-2304, PMLR, 2017a.
Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., and Campbell, M. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv: /710. 1089, 2017b.
Goel, S. K. Subgoal discovery for hierarchical reinforce- ment learning using learned policies. The University of Texas at Arlington, 2003.
Gregor, K., Rezende, D. J., and Wierstra, D. Variational intrinsic control. arXiv preprint 161 2016.
Mavor-Parker, A., Young, K., Barry, C., and Griffin, L. How to stay curious while avoiding noisy tvs using aleatoric
10

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
uncertainty estimation. In International Conference on Machine Learning, pp. 15220-15240. PMLR, 2022.
Simsek, Ö., Wolfe, A. P., and Barto, A. G. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Raedt, L. D. and Wrobel, S. (eds.), Ma- chine Learning. Proceedings of the Twenty-Second In- ternational Conference (ICMI. 2005), Bonn, Germany, August 7-11, 2005, volume 119 of ACM International Conference Proceeding Series, pp. 816-823, ACM, 2005. URL https: /doi.
McGovem, A. and Barto, A. G. Automatic discovery of subgoals in reinforcement learning using diverse density. 2001.
Menache, I., Mannor, S., and Shimkin, N. Q-cut-dynamic discovery of sub- sub-goals reinforcement learning. In Ma- chine Learning: ECML 2002: 13th European Conference Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13, pp. 295-306. Springer, 2002.
doi: 10. 1 org 10 1tt.1111/1111250 1102444
Şimgek, Ö., Wolfe, A. P., and Barto, A. G. Identifying useful subgoals in reinforcement learning by local graph partitioning. Proceedings of the 22nd international conference Machine learning, pp. 816-823, 2005.
Nam, T., Sun, S.-H., Pertsch, K., Hwang. S. and Lim, J. Skill-based meta-reinforcement learning, arXiv preprint arXiv:2204.11828 2022.
Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., and
Park, S., Rybkin, O., and Levine, Metra: Scalable unsu- pervised with metric-aware abstraction, 2023.
Levine, S. Parrot: Data-driven behavioral priors for re- inforcement learning. arXiv preprint arXiv:2011.100244 2020.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-drine exploration by self-supervised predic- tion. International conference on machine learning, pp. 2778-277777 -7777. PMLR, 2017.
Srinivas, A., Krishnamurthy, R., Kumar, P., and Ravin- dran, B. Option discovery in hierarchical reinforcement learning using spatio-tempont clustering. arXiv preprint arXiv:1605.0533) 2016.
Pathak, D., Gandhi, D., and Gupta, A. Self-supervised exploration via disagreement. In International conference on machine learning, pp. 5062-5071, PMLR, 2019.
Sutton, R. S. Between mdps and semi-mdps: Learning, plan- ning, and representing knowledge at multiple temporal scales. 1998.
Pertsch, K., Lee, Y., and Lim, J. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188 204. PMLR, 2021a.
Todorov, E.. Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ inter- national conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Pertsch, K., Lee, Y., Wu, Y., and Lim, J. J. Guided rein- forcement learning with learned skills. arXiv preprint arXiv:2 10253, 2021b.
Yang. M., Levine, S., and Nachum, O. Trail: Near-optimal imitation learning with suboptimal data, arXiv preprint 10.1::21 2021.
Rissanen, J. Modeling by shortest data description. Auto- matica, 14(5):465-47) 471, 1978.
Salter, S., Wulfmeeer, M., Tirumala, D., Heess, N.. Ried- miller, M., Hadsell, R., and Rao, D. Mo2: Model-based offline options, 2022. URL https::///// https::/ttattttee /ttp:v///axvvvvvo abs 2209. 01947.
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pp. 8583-8592, PMLR, 2020.
Seo, Y., Chen, L., Shin, J., Lee, H., Abbeel, P., and Lee, K. State entropy maximization with random encoders for efficient exploration. International Conference on Machine Learning, pp. 9443-9454. PMLR, 2021.
Şimşek, Ö. and Barto, A. G. Using relative novelty to identify useful temporal abstractions in reinforcement learning. Proceedings of the twenty-fist international conference on Machine learning, pp. 95, 2004.

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
A. Ablation
0.3
Completed Subtasks
Completed Subtasks
Success Rate
NBDI NBDI-Cumul
NBDI-tho.3 3
NBDI-tho.1
NBDI
NBDI-th0.5
NBDI-NoTermDistr
0.00
2,00
3.00
4.00
5,00
0.00
1.00
3,00
4.00
1.0 Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
(a) Kitchen Environment
(b) Kitchen Environment
(c) Maze 40 40
Figure 10. Ablation in variable-length skills (a), termination distribution (b). and criteria to determine decision points (c). The shaded region represents 95% confidence interval across five different seeds.
A.1. Ablation in NBDI
Figure 10a compares the performance of our model (NBDIIth0.33 in the kitchen environment with different state-action novelty threshold values. We can see that there is no significant improvement in performance compared to SPiRL when the threshold value is not appropriately chosen. For example, as illustrated in Figure 11, termination distributions learned with low threshold values can disturb the policy learning by terminating skills in states that lack significance. illustrates that threshold value needs be appropriately chosen to capture meaningful decision points.
A.2. Ablation in No Termination Distribution
Figure 10b shows the performance drop when we do not learn the termination distribution advance (see Appendix for more details).
directly uses state-action novelty module the downstream learning phase to terminate skills. The performance gap indicates that the skill embedding space offline skill extraction learning needs be learned with terminated skills to effectively guide the agent in choosing variable-length skills. Thus, is necessary jointly optimize the termination distribution, skill embedding space, and skill prior using the deep latent variable model in offline skill extraction.
A.3. Ablation in Criteria to Determine Decision Points
Figure 10c shows the performance difference when we use cumulative sum of state-action novelties to learn decision points. SubllCCumulaiveeSum terminates skills once the cumulative sum of state-action novelty reaches or surpasses predefined threshold. This comparison implies that accumulating novelties does not lead to the identification of significant termination points.
Figure 11. example of decision point the maze environment.
12

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
(Ours)
NBDI (Ours)
0.3
SPIRL
Stacked Blocks
SPRLL lavg-fixed)
SPRL
Completed Subtasks
(avg-fixed)
SAC
0.2
Success
2
NBDI (Ours)
0.1
0.5
SPIRL
SPIRL (avg-fixed)
SAC
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
1
2
4
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
(a) Maze 40 40 Navigation
(b) Sparse Block Stacking
(c) Kitchen
Figure 12. Performances method and baselines in solving downstream tasks. The shaded region represents 95% confidence interval across five different seeds.
A.4. Comparison to SPiRL with Fixed Average Skill Length NBDI
Figure 12 shows whether NBDI still outperforms SPiRL when SPiRL uses average skill length NBDI (SPiRL (ivg)fixed)). The average lengths of skills of NBDI was 26 in the maze environment, 22 in the sparse block stacking environment, and 25 in the kitchen environment. We found NBDI still outperforms SPiRL with those fixed average skill lengths. We also observed that SPiRL, when set to those average skill lengths, performs worse than SPiRL configured with fixed skill length of 10 in the block stacking environment, and better in the maze environment. However, across any fixed skill length ranging from 10 to 30, there was no instance where SPiRL outperformed NBDI. This demonstrates that our model can effectively leverage critical decision points in the environment compared to fixed length approaches.
To investigate how frequently the maximum skill length gets reached, we tracked the skill lengths of high-level policy for each environment during the downstream reinforcement learning process. The percentages of executed skill lengths shorter than H follows: 19.8% in the maze environment, 30.8% in the sparse block stacking environment, and 17.4% in the kitchen environment. When categorizing the skill lengths into intervals of 1-10, 11 11-20, and 21-30, the distributions are as follows: 13.2%, 2.8%, and 84% for the maze environment; 28.6%, 1.6%, and 69.8% for the sparse block stacking; and 16.8%, 0%, and 83.2% for the kitchen.
can be observed that, for the majority of the time, our high-level policy maximizes temporal abstraction by executing longer skills (ranging from 21 to 30). However, our method also allows the high-level policy to capture important decision points through shorter skills (ranging from 1 to 10), promoting more efficient exploration of the state space and enhancing the transfer of knowledge across various tasks.
Table 3. Truncated skill lengths generated by our method across various tasks with a maximum skill length of H 30.
SKILL LENGTH
MAZE
SPARSE BLOCK STACKING
KITCHEN
h <
19.8%
30.8%
17.4%
1 1<h 10
13.2%
28.6%
16.8%
11 h 20
2.8% 2
1.6%
0%
21
84%
69.8%
83.2%
13

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
B. Skill Extraction from Demonstration in NBDI
(ii) Skill Extraction
(i) Novelty Learning
Data
Data
State-Action
State-Action
Novelty Module
Novelty Module
x(s,0)
a(s,a)
Pet
PeoR
Rea2t-1
(iii) Skill Execution
Tee1
Teee
Skill Encoder 4o (4]v, /3
Skill Prior
Skill Policy (#lsg)
Skill Prior Pr (2)s,)
Py (els,)
Skill Embedding
Skill Embedding
Regularization
St
Skill Decoder Py s,)
Skill Decoder Pylas.Bale,s))
Environment
Environment
ag
B.
A
Peos
Beva
PeoKix
Besit-t
P:
Pes
Bena 1
PesKEz ... Bees--1
&
Reet
Bessss
Rco1
Beok
Figure 13. Novelty-based Decision Point Identification (NBDI), has two main procedures: (i) novelty learning and skill extraction: training the statc-action novelty model and lcarning the skill prior, skill embedding space and termination distribution with the pre-trained novelty model. (ii) skill execution: performing reinforcement learning with termianted skills to solve an unseen task.
B.1. Learning the Skill Prior, Skill Embedding Space and Termination Distribution
To learn the skill embedding space Z, we train a latent variable model consisting of a Long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) encoder (=(=|r,3) 33 and decoder Py (at, -,|2, 8t). To learn model parameters Φ and th, the latent variable model receives a randomly sampled experience т from the training dataset along with a termination condition vector /3 from the state-action novelty module, and tries to reconstruct the corresponding action sequence and its length (i.e., point of termination) by maximizing the evidence lower bound (ELBO):
log p(at, Be(ss) (g log Ps (ar, 9.12. st) t (log((2) log go (2
Lux (d).0)
where is used as the weight of the regularization term (Higgins al., 2016). The Kullback-Leibler (KL) divergence between the unit Gaussian prior p(z) = (0, I) and the posterior log (=|T,, 33) makes smoother representation of skills.
To offer effective guidance in selecting skills for the current state, the skill prior (z(ss)), parameterized by is trained by minimizing its KL divergence from the predicted posterior 4s (2), (). In the context of the option framework, it can also be viewed as the process of obtaining an appropriate initiation set for options/skills. This will lead to the minimization of the prior loss:
Cpror( (T) EE-D
The basic architecture for skill extraction and skill prior follows prior works (Pertsch et al., 2021a; Hakhamaneshi et al., 2021), which have proven be successful. In summary, termination distribution, skill embedding space, and skill prior jointly optimized with the following loss:
Lixat Lrec(d,, 16) ofhreg (0) Leprior
(4)
14

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
Algorithm 2 Reinforcement learning with NBDI
Input: trained skill decoder (a, 2| s), discount factor 7,, target divergence 6, learning rates Xs, Ao, Au target update rate e
1: Initialize replay buffer D, high-level policy mo(=|s), critic Qg(s, 2), target network &
2: for each iteration do
3: for each environment step do
4: 24 ~ (2))))8))
5: for k = 1......0
6:
ar-k, Bttk ~ Po Be+kl2t, 81tk)
7:
St4k+1 ~ ((5+++(++11000 br as+k)
8:
if BL++ = or k = H then
9:
Break
10:
end if
11:
end for
Fe + E 7f (R 0z)
12:
D DU (8t, 2t, Fe, 11+
end for
15:
for each gradient step do
16:
2++++1 ~ no 2 (1+++11 Q = f: + 1 (g -occtaaaaD
17:
(sc+8+1)))
18:
0 - - Aa V Qe
19:
o + { ^y Ve
1 (QE 2)) Q)²
g m V. w. (DxL (mg (1)p -5]]
f+ + (1 c)
22:
end for end for
23:
24: return trained policy (z())ss:)
Table 4. Skill Prior Hyperparameterss
HYPERAPARAMETER
VALUE
BATCH SIZE
16
OPTIMIZER
RADAM( = 0.9, = 0.999, Ir 16 3)
REGULARIZATION WEIGHT a
1.0
SKILL. ENCODER
DIM-Z IN VAE
32
HIDDEN DIM
128
L.STM LAYERS
SKILL PRIOR (KITCHEN)
HIDDEN DIM
128
FC LAYERS
6
SKILI. PRIOR (MAZE, BLOCK STACKING)
KERNEL SIZE
(4, 4)
CHANNELS
8, 16, 32
CONVOLUTION LAYERS
3
SKILL DECODER
HIDDEN DIM
128
# HIDDEN LAYERS
6
15

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
B.2. Reinforcement Learning with NBDI
In downstream learning, our objective learn a skill policy (z(2)))) that maximizes the expected sum of discounted rewards, parameterized by 8. The pre-trained decoder Pwl (o,, Be|2, st) decodes skill embedding into series of actions, which persists until the skill is terminated by the predicted termination condition 3.. The downstream learning can be formulated as SMDP which is an extended version of MDP that supports actions of different execution lengths.
Adapted from Soft Actor-Critic (SAC) (Haarnoja et al., 2018), we aim to maximize discounted sum of rewards while minimizing its KL divergence from the pre-trained skill prior on SMDP. The regularization weighted by w effectively reduces the size of the skill latent space the agent needs to explore.
J(0) Ex
Σ F(se, w WDL (=(2)()))) (n) (n(((()))) EET
where T set of time steps where we execute skills, i.e., T {0, ko: No + k1, ko + k + k2.... where KG the variable skill length of i-th executed skill.
To handle actions of different execution lengths, the following Q-function objective is used:
( 2x) Q)2
(5) E
where Q (Q (or
WDKz(10(2488)) s++*+1) 1p (1((2+++1//44444))) (s4+*+1))]
represents the temperature for KL-regularization, denotes the number of time steps elapsed from the start state 8 the termination state St4A-1, and represents the cumulative discounted reward the k time steps. The detailed RL learning loop is described in Algorithm and Figure 13.
B.3. Threshold for environments
In practice, the thresholds tuned through experiments (see Appendix A.I) approximately correspond the 97th percentile of the novelty values computed over task-agnostic demonstrations e.g....ooonnn.............. (96.86th percentile), and block stacking 40 12th percentile). We hope the percentile-based guidance would help the readers more easily apply our method across new environments.
e.g.. kitchen 0.3 (97.47th percentile), maze 50
Table 5. Downstream RL Hyperparameters
HYPERAPARAMETER
VALUE
DOWNSTREAM REINFORCEMENT LEARNING
BATCH SIZE
256
OPTIMIZER
ADAM(3; = 0.9, B2 = 0.999, = -) 4)
REPLAY BUFFER SIZE
1E6
DISCOUNT FACTOR
0.99
TARGET NETWORK UPDATE RATE
5e
TARGET DIVERGENCE 8
(KITCHEN), (MAZE, SPARSE BLOCK STACKING),
VARIABLE LENGTH SKILL
THRESHOLD NOVELTY
0.3 (KItcHen), 50 (MAZE), 40 (SPARSe BLOCK STACKING)
MAXIMUM SKILL LENGTH H
30
16

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
C. Visualization of Critical Decision Points with Suboptimal Data
100
775
Percentile
start
Start
start Target
start
Target
(a) (SPRL)
(b) Visitation(NBDI)
(c) Skill Term.
(d) Skill TemD(NBDD)
100
Percentile
start
0 Start
O
Staft
staff
Target
target
target
Target
(e) Visi))tion(PPiRL)
(I) Visitation(NBDI)
(g) Skill (SPRLL)
(h) Skill TerD.
Figure 14. Visualization of visitations and decision points made by SPiRL and NBDI in the maze environment (top: trained with stochastic BC 0.5) dataset, bottom: trained with stochastic BC 0.75) dataset). We sampled 100 trajectories for each trained policy observe the points at which they make decisions. Higher percentile colors suggest relatively greater number of visitation frequencies and termination frequencies. Note that the termination frequencies are normalized by the overall visitation frequencies for better visualization.
Figure 14 and Figure 15 (top, middle) show that with suboptimal dataset, NBDI is still able to learn termination points characterized by high conditional action novelty or state novelty. Figure 14a and Figure 14e shows that SPiRL can only navigate around the initial state using fixed-length skills extracted from suboptimal dataset, whereas NBDI can successfully reach the goal efficiently (Figure 14b and Figure 14f).
However, with dataset generated by random walk (Figure 15 (bottom))) it becomes challenging to learn meaningful decision points. As the policy generating the trajectory becomes more stochastic, it gathers data primarily around the initial state, leading to an overall reduction in the scale of prediction errors. Thus, we can see that the level of stochasticity in the dataset influences critical decision point detection.
17

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
160
[A]
(C)
(8)
120
Prediction
80
(A)
(B)
(C)
40
20
(F)
State-Action in trajectories
50
(B)
40
(C)
Prediction Error
30
(0)
(C)
2000
4000
6000
8000
10000
(D)
(6)
IF]
State-Action in 25 trajectories
1.2
(B)
1.0
Prediction Error
0.8
(A)
(B)
(C)
0.2
0.0
2000
4000
6000
8000
10000
12000
(D)
[E]
(F)
State-Action trajectories
Figure 15.. Visualization of prediction error of ICM in maze environment (top: trained with stochastic BC (a = 0.5) dataset, middle: trained with stochastic BC (o 0.75) dataset, bottom: trained with random walk dataset). Note the offline data that used train ICM was used to compute this prediction error. (A), (B) and (C) are the statc-action pairs with the highest prediction error, while (D), (E) and (F) the ones with the lowest.
18

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
D. Visualization of Critical Decision Points in Complex Physics Simulation Tasks
((k+1 (x+1) 1.52
X ($ (k+1) = 3.60 3.66
(5. aq)
2.80
x(5k. ag) 39.43
( an) = 24
X (2) 18.
Hallchettah-mediu-eeyyy
medium-replay
1(5k:1 4.14
($ (R+1)) x 4.72
(5k++ ax+1) = 9.90
X($211.
( 6o) = 1.00
2(2.. ag) 65. 12
X(5. ag) 64
X(5,. a²) 95.22
Ant nat-mmedummexppee um-expert
Ant undiiuduum-rppla
( 4x+1) z 4, 14
((5x: ag11) 10.38
Figure 16. Visualization of critical decision points in MuloCo (Todorov et al., 2012) environment (top-left: malfchhetah-medium-eeee top-right: halfcheetah-mmediun--rpppy bottom-left: ant-medium-expert, bottom-right: ant-mediumm-rella)
We investigated whether meaningful decision points can be found in complex physics simulation tasks. We trained ICM using different offline datasets provided by D4RL (Fu et al., 2021) hhafchhetah-mediun-eepree halfchherah-mediun--elllle ant-medium-cxpee exppet, ant- ant-medium- replay) to assess its ability to detect critical decision points. Figure 16 illustrates the presence of critical decision points in complex physics simulation tasks. For instance, the cheetah has the option of spreading its hind legs lowering them the ground, and the ant has the choice of flipping the right lowering vhe to the ground. However in completely random datasets (halfcheetah-rrnndon) ant-random), meaningful decision points. Similar to Appendix C, shows that the degree of stochasticity present in the offline dataset can influence critical decision point detection.
mediummreeplay))
not able to find any
19

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
E. Train/transfer Domain Similarity
To apply our method to downstream environments with significantly different overall layouts (e.g., a maze with a completely new structure or larger-scale block stacking environment with more blocks in random positions), it necessary to collect cropped image centered around the agent from task-agnostic demonstrations (visualizations of cropped images are available in Figure 2). This way, trajectories, which then can be used to detect state-action novelty based decision points even in downstream tasks with significantly different environment configurations. On the other hand, for environments where the overall layout remains consistent (e.g. the positions of manipulatable objects in the kitchen environment do not change during downstream task), we can use any state information that sufficiently describes the environment to apply our method (see Appendix for environment details).
extract consistent structural information that can be extracted across multiple task-agnostic
Training Data
Downstream Tasks
Figure 17. Examples of maze layouts used for training (left) and downstream task (right). The agent starts at the white point and aims to reach the red point.
F. -aso-Ag Dataset Size and Coverage
The number of task-agnostic trajectories collected for each environment follows: maze (85,000 trajectories), block stacking (37,000 trajectories), and kitchen (400 trajectories). Since used image-based observations in the maze and block stacking environments, and the downstream tasks involve significant configuration changes (e.g., entirely new maze layouts or larger-scale block stacking environment with more blocks in random positions), we require a larger set of task-agnostic demonstrations compared to the kitchen environment, where the overall layout remains consistent across tasks.
Since our goal maze and block stacking environment is to solve downstream tasks with significantly different environment configuration, we do not assume that task-agnostic demonstrations fully cover the state space of the downstream tasks. However, it is important that they provide good coverage of the observation space. Note are using cropped image centered around the agent from task-agnostic demonstrations as observations. This way, we can extract consistent structural information that can be extracted across multiple task-agnostic trajectories, which then can be used to detect state state-action novelty based decision points even in downstream tasks with significantly different environment configurations.
20

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
G. Comparison to baseline methods on discrete state/action space
To provide an intuitive comparison between NBDI, Relative Novelty (Şimşek & Barto, 2004), and LOVE (Jiang et al., 2022) in discrete settings, we conducted case study using a grid-based maze environment (Figure 18). This allowed to directly visualize and compare termination points across methods in a controlled, discrete domain. To align with our task-agnostic setup, we collected diverse expert-level demonstrations from random start and goal positions, and used these datasets to extract skills and termination points.
Relative Novelty defines novelty at given state as the ratio between the average novelty of future and past states, measured within a fixed-size sliding window (n ln_). A high value indicates that the agent transitions from a familiar region to a less familiar one. Following the original formulation, we only evaluated states with sufficient trajectory context on both sides of the window. As shown in the visualization, Relative Novelty is highly dependent on transition history, often identifying only subset of bottleneck states.
LOVE uses variational inference framework to extract skills based on the Minimum Description Length (MDL) principle in that its objective is to "effectively compress a sequence of data by factoring out common structure". While LOVE employs a variational inference framework to implement the MDL principle, we used Byte Pair Encoding (BPE) to extract skills in discrete setting as BPE is a specific formulation of MDL (Gallé, 2019) and it offers a more intuitive and interpretable formulation of compression. Termination points were visualized based on where the segmented skills terminated during 100 goal-reaching tasks. The results show that terminations vary significantly with the number of trajectories used extract skills, as LOVE focuses capturing common structure rather than consistent bottlenecks.
NBDI terminates skills based on both conditional action novelty and state novelty (Section 4.1). The visualization demonstrates that NBDI consistently identifies key bottleneck states and exhibits robustness to the number of trajectories collected in contrast to other methods. Moreover, states with high state novelty often correspond to regions that are rare or hard to reach within the task- agnostic dataset. By increasing the decision frequency in such unfamiliar states, NBDI promotes more effective exploration of the state space.
21

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
Termination Campa rison with 100 trajectories
Conditional letionn
+
Rebriiv Kooely (u)g)-2)
Ninirumn Descr (00rges00))
H
-11
21
-92
12
-2
Nowetiy
Relarive Kountiy (age?))
Vtrimum Descrge Langth Imerges-108:
21
31
Tomination Comparison trajectories
Condiconall Action Novelty
Rolarive Kovelty to (ogg))2))
Mivimu Desor plen Length (merges=20)
3.
31
11
State Nowelty
Rebeiiv Kouelty (eg:7)
M4nimum Deecrie
32
34
24
24
-2
Figure 18. Visualization of skill termination points in grid-based maze environment using NBDI (Conditional action novelty, State novelty), Relative Novelty, and LOVE (Minimum Description Length). Darker shades represent states with relatively higher frequency of termination occurrences. Black cells are impassable wall regions in the environment. Top: termination points computed from 100 trajectorics. Bottom: results from 10,000 trajectories.
22

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
H. Data and Environment Details
We evaluate NBDI on three environments: one simulated navigation task (Maze navigation) and two simulated robotic manipulation tasks (Kitchen and Sparse block stacking). We employ the environment configuration and dataset provided by (Pertsch et al., 2021a) (Maze navigation and Sparse block stacking) and (Fu al., 2021) (Kitchen) (CC BY 4.0). Note that task and environment setup differ between the training data and the downstream task, demonstrating the model's capacity to handle waseen downstream tasks.
The kitchen environment is provided by the D4RL benchmark (Fu et al., 2021), featuring seven manipulable objects. The training trajectories consist of sequences of object manipulations. The downstream task of the agent involves performing an unseen sequence four object manipulations.
Kitchen Environment
The agent is tested by its ability to reassemble skills learned from the training dataset to solve the downstream task.
State space: 30-dimensional vector of the agent's joint velocities and the positions of the manipulatable objects
Action space: 7-dimensional set for controlling robot joint velocities and a 2-dimensional set for gripper opening/closing degree
Reward:
one-time reward upon successfully completing any of the subtasks
Maze Navigation The maze navigation environment is derived from the D4RL benchmark (Fu et al., 2021). During the collection of training data, maze is generated randomly, and both the starting and goal positions are selected at random as well. The agent successfully reaches its goal in all of the collected trajectories.
In the downstream task, the maze layout is four times bigger than the one employed during training.
State space: (x, y)-loocciss and an image of local top-down view centered around the agent
Action space: (I, y)-directions
Reward: binary reward when the agent's position is close to the goal (computed using Euclidean distance)
The sparse block stacking environment is created using the Mujoco physics engine. To gather training data, hand-coded data collection policy interacts with a smaller environment with five blocks to stack as many blocks as possible.
Sparse Block Stacking
In the downstream task, the agent's objective to stack as many blocks as possible in a larger version of the environment with eleven blocks.
State space: (x, disilllaassssnss for the robot and an image of local view centered around the agent
Action space: 10-dimensional continuous symmetric gripper movements
Reward: only rewarded for the height of the highest stacked blocks
Differences to (Pertsch et al., 2021a). While (Pertsch et al., 2021a) employed block stacking environment with dense rewards (the agent is rewarded based on the height of the stacked tower and for actions like picking up or lifting blocks), we evaluated our model and the baselines in sparse block stacking environment, leading to different performance outcomes. In this setting, the agent is rewarded solely for the height of the tower constructs, which increases the task complexity. Furthermore, there have been consistent reports indicating that performance in large maze environments displays a high level of sensitivity to random seeds, primarily due to the high stochasticity of the task. For the five different seeds that we used compare the algorithms, we found the performance be generally lower than what was previously reported, mainly due to the sensitivity to seeds.
23

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
I. Implementation Details
Our codebase builds upon the released code of SPiRL (Pertsch et al., 2021a). Our code is available at: attps://gitte com/ku-cmlab, Manllab/Ma /NBDI.
L.I. Termination Improvement and Novelty
We present termination improvement and conditional action novelty in a simple x 8 grid maze domain. As we mentioned in Appendix H. training data for the state-action novelty module is collected with diverse tasks. Thus, we set up the environment as follows: We randomly select one starting location and three goal locations generate three trajectories for each goal location. The goal-reaching data collection policy randomly executes discrete action, moving towards four different directions (left, right, forward, backward) while avoiding moving toward walls. The agent receives binary reward when reaching the goal state.
We define an option o = (I, π, (3) where a deterministic policy æ follows the given trajectory, an initiation set I C S defines all states that the policy visits, and a termination condition 3 defines states where the option terminates (every option has a length of three). Each set of options, denoted as Og for each goal g = 2, 3, contains distinguishable options for each goal location. The frequencies of termination improvement occurrences in each state for each goal setting have been aggregated to generate Figure 3.
Using the trajectories collected from different goals, state novelty and conditional action novelty are simply computed as N(.) and (() respectively. (s)) represents the number of times discrete state has been visited and N(ss, a) represents the number of times a discrete state-action pair has been used.
(s.a)
1.2. State-Action Novelty Module
We use Intrinsic Curiosity Module (ICM) to calculate state-action novelty for both image-based and non-image-based observations. The feature encoder Φ. responsible for encoding a state 8; into its corresponding features ((s)), is implemented differently for each environment. In the kitchen environment, it consists of single fully-connected layer with a hidden dimension of 120. The both maze and sparse block stacking environments have three convolution layers with (4, 4) kernel sizes and (8, 16, 32) channels.
The forward dynamic model f takes a, and Φ($t) as inputs to predict the feature encoding of the state at time step & + 1. In the kitchen environment, the structure the dynamic model is the same as its feature encoder. In the maze and sparse block stacking environment, single fully-connected layer with hidden dimension 52 and 70 have been used, respectively.
The state-action novelty x(s, a) is computed as the squared L2 distance between ø ((((s;), at) and d(s representing the prediction error in the feature space. We employed the Adam optimizer with 31 0.9, 32 0.999 and a learning rate of 1e 3 to train the ICM. We found that state-actions within the top 1 % prediction error percentile serve well as critical decision points, and used the corresponding threshold to learn the termination distribution.
I.3. BC+SAC and IQLASAC
We trained flat offline learning methods, including Behavior Cloning (BC) and Implicit Q-Learning (IQL) (Kostrikov et al., 2021). using an offline dataset. In the downstream RL step, the trained policy p(a(s) imposes a KL-divergence penalty on the SAC agent. In IQL, we set the expectile hyperparameter r to 0.7 and the inverse temperature hyperparameter 3 to 0.5 across all tasks.
(()) = E,
Σ r((, aDKI (T(alls)) LET
(6)
1.4. LOVE and Relative novelty
Since LOVE (Jiang et al., 2022) requires a discrete action space for option learning, we implemented some modifications. In maze environment, we discretized the continous action space into discrete actions. In the kitchen environment, we modified the action decoder handle continuous distribution.
24

---

A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
Table State-Action Novelty Module Hyperparameters
HYPERAPARAMETER
VALUE
BATCH SIZE
150
OPTIMIZER
ADAM(3 = 0.9, 32 = 0.999, Ir le -
LOSS WEIGHT B
0.2
SCALING FACTOR 7)
1.0
KITCHEN
FEATURE ENCODER
HIDDEN DIM
120
FORWARD DYNAMIC MODEL
HIDDEN DIM
120
MAZE
FEATURE ENCODER
KERNEL SIZE
(4, 4)
CHANNELS
8, 16, 32
FORWARD DYNAMIC MODEL
HIDDEN DIM
52
SPARSE BLOCK STACKING
FEATURE ENCODER
KERNEL SIZE
(4, 4)
CHANNELS
8, 16, 32
FORWARD DYNAMIC MODEL
MIDDEN DIM
70
Since relative novelty (Şimşek & Barto, 2004) used pseudo counts to measure state novelty in discrete environments, we could not apply this method to continuous environments straightforwardly. Furthermore, since this technique requires environments that support backward transitions for computing the relative novelty score, its direct application in general environments seems challenging. Thus, to align with the author's intuition that target states should exhibit higher novelty scores compared to non-target states, we utilized our state novelty module x(s) measure the state novelty, and trained neural network with offline datasets estimate the relative novelty score of states.
I.S. Experiments Compute Resources
Each experiment was conducted on single CPU (Intel Xeon Gold 6330) with 256GB of RAM and single GPU (NVIDIA RTX 3090). Each training session took about 36 hours (12 hours for prior learning, 24 hours for downstream learning), utilizing approximately 30% of the RAM and 25% of the GPU memory. We implemented all RL algorithms using PyTorch v1.3 and executed them on Ubuntu 22.04.04 LTS.
25


Reviews:
[{"review_id": "ekqh5F7ag5", "reviewer": "Reviewer_A7Qu", "summary": "The paper is on the topic of skill learning in reinforcement learning. Its contribution is on the subject of when skills should be terminated. The authors build on the existing work on the literature on skill learning, where the learned skills are executed for a fixed number of time steps. Here the authors add the flexibility of terminating the skills at other times. Specifically, they focus on identifying states that would serve as particularly useful points of termination based on the novelty of the state and the stat-action pair. The authors present empirical results in several domains.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough and constructive comments. We hope we can address your concerns below.\n\n**Q1: Description of the train/transfer environments**\n\nIn Appendix E, we introduced the train/transfer domain similarity metrics for the environments used in our experiments. To demonstrate the effectiveness of our method in challenging downstream tasks involving significant configuration changes, we tested on a maze with an unseen layout and a larger-scale block stacking environment with more blocks in random positions —both of which differ significantly from the task-agnostic offline datasets. The significant performance improvements reported in Section 6 clearly demonstrate the effectiveness of our method.\n\n**Q2: Confusing expression**\n\nWe apologize for the confusion caused by our wording. We agree that the term \"finding\" may have been misleading in this context. We will revise the word to “a novel method”.\n\n**Q3: Applying NBDI to other skill-based approaches**\n\nThank you for your constructive suggestion regarding the experiments. Our work focuses on extracting meaningful termination points from task-agnostic demonstrations. To explore the broader applicability of our approach, we investigated its compatibility with a skill-based approach that leverages task-agnostic demonstrations to solve challenging long-horizon, sparse-reward meta-RL tasks [1]. Specifically, we applied our method during the skill extraction phase of [1] to learn variable-length skills in place of fixed-length skills.\n\nIn the maze environment, we randomly sampled 10 goal locations for meta-training. The table below compares the success rate of meta-policies trained with [1] (fixed-length skills) and our method during the meta-training phase across episodes. The result shows that the extracted variable-length skills allows the meta-policy to better promote knowledge transfer between different tasks, helping the meta-policy in combining the skills to complete complex tasks. We report mean success rates on the 10 meta-training tasks across 3 different seeds with standard errors.\n| |ep200|ep400|ep600|ep800|ep1000|\n|---|---|---|---|---|---|\n|[1]|0.193$\\pm$0.030|0.329$\\pm$0.037|0.501$\\pm$0.049|0.436$\\pm$0.048|0.582$\\pm$0.037|\n|[1]+NBDI|**0.341**$\\pm$0.035|**0.554**$\\pm$0.037|**0.737**$\\pm$0.028|**0.828**$\\pm$0.028|**0.896**$\\pm$0.025|\n\nFurthermore, during the target task learning phase, the meta-policy learned through our approach leads to **significantly better sample efficiency** on the unseen target task. These results indicate that our approach can be effectively integrated with a broader class of skill-based methods that leverage task-agnostic demonstrations (3 seeds).\n| |ep20|ep100|ep300|ep500|\n|---|---|---|---|---|\n|[1]|0.143$\\pm$0.063|0.593$\\pm$0.191|0.667$\\pm$0.193|0.990$\\pm$0.006|\n|[1]+NBDI|**0.560**$\\pm$0.121|**0.960**$\\pm$0.021|**0.980**$\\pm$0.011|**0.993**$\\pm$0.003|\n\n**Q4: The trajectory collecting policy**\n\nAs our work primarily focuses on learning termination conditions from task-agnostic demonstrations, we will revise Line 264–268 and 318 to clarify that we assume access to **task-agnostic, expert-level demonstrations**. In Section 6.5, we analyzed the impact of dataset quality on our method’s performance. The results indicate that the level of stochasticity and the suboptimality of the dataset influence the effectiveness of our approach, which remains a limitation of our work.\n\n**Q5: Skill length impact on baselines**\n\nDue to space limitations, we kindly refer you to our response to **Reviewer bZqs’s Q2**, where we provide a detailed discussion.\n\n**Q6: Details in environment settings**\n\nWe apologize for not including information about the environments (Appendix E, F) in the main paper. In the kitchen environment, the agent completes an unseen sequence of object manipulation subtasks, like opening the cabinet or turning on the oven. In the maze environment, it receives a binary reward for reaching the goal. In the sparse block stacking environment, the agent is rewarded based on the height of successfully stacked blocks at the end of the episode. We will include a brief introduction to the environments in the experiment section.\n\n**Q7: Other comments and suggestions**\n\n**(Line 185-190)** We apologize for the confusion. Since the statement in Line 185–190 is based on Figure 3, we will move it to the paragraph where Figure 3 is discussed in detail.\n\n**(Figure 5)** Figure 5(a) and 5(b) illustrate the performance of the baselines and our method on a challenging downstream maze task, which involves reaching the goal in a maze with an unseen layout (Appendix E). Due to the complexity of this task, performance is highly sensitive to random seeds, leading to generally lower success rates compared to previously reported results (Appendix F). In the sparse block stacking environment, the maximum number of blocks that can be stacked is five.\n\n[1] N., Taewook, et al. \"Skill-based meta-reinforcement learning.\" ICLR, 2022.", "claims_and_evidence": "The main claims are as follows:\n\n(1) The proposed skill termination condition for task-agnostic demonstrations remains effective even when the environment configuration of complex, long-horizon downstream tasks undergo significant changes. \n\n(2) The paper presents \"a novel finding in reinforcement learning, which is the identification of state-action novelty-based critical decision points.\"\n\n(3)  Executing terminated skills, based on state-action novelty, can substantially enhance policy learning in both robot manipulation and navigation tasks.\n\nIt is not clear how well the paper supports claim (1). There is not a clear description of the test environments and how they change in future tasks. \n\nThe second claim is confusing because \"identification\" of critical decision points is not a \"finding\".\n\nFor the third claim, there is some supporting evidence in the paper. However, it must be noted that the experimental evaluation does not consider a broad range of skill-based approaches to policy learning.", "methods_and_evaluation_criteria": "The environments and the tests that are carried out make sense for the problem at hand. However, ideally a broader set of skill-based methods would be used in the evaluation. Currently, only a single approach is tested (SPiRL)", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "No particular problems I have noticed.", "relation_to_broader_scientific_literature": "The paper implements an intuitive idea for skill termination in reinforcement learning. This proposed approach to skill termination is an alternative to executing skills for a pre-determined fixed period of time. \n\nThere are a large number of approaches in the literature to skill discovery -- here I am using the term \"skill\" broadly to denote a behaviours that can take multiple time steps to execute. Many of these approaches identify termination conditions explicitly, which means that these skills terminate in accordance with their termination conditions rather than after executing for a fixed period of time steps. So the ideas in this paper will not necessarily be relevant for those approaches. \n\nHowever, there are also some existing methods that learn skills with no explicit termination conditions; with these methods, the skills are executed for a fixed amount of time. The proposed approach can be useful for such skills.", "essential_references_not_discussed": "The paper proposes a termination condition for skills, which could in potential be useful with many existing approaches to skill discovery that do not identify termination conditions for the discovered skills and instead executes them for fixed time lengths. But the paper examines only one such approach (SPiRL). A few other approaches are mentioned in the paper but not included in the experimental evaluation. I suggest that the authors include a broader discussion of existing approaches to skill learning for which their approach may be useful and that they include some of these methods in their experimental evaluation.", "other_strengths_and_weaknesses": "The main idea in the paper -- that skills should be terminated at points where renewed decision making may be useful -- is I think a good one. It may be useful in combination with many different ways of discovering skill policies. \n\nThe writing can be more organised and clear. I have given some specific comments in the next section, which I hope the authors will find useful.", "other_comments_or_suggestions": "Figure 1 -- The figure shows a number of images from the kitchen environment but without sufficient context. The reader needs a better understanding of the domain in order to get meaningful information from these images. \n\nFigure 2 -- The last figure here has low resolution. More importantly, in both figures showing states from the environment, it is not fully clear what these figures are showing. The domains, and their visual depiction, needs to be explained to the reader. \n\nLine 198 -- \"where a subtask has been completed\" -- What is a \"subtask\"? How is it defined? Additionally, this domain needs to be explained in more detail. \n\nFigure 3 -- Needs a larger font size on the side bar showing the color scale. \n\nLine 185-190, second column: \"When the skills (or options) are discovered from diverse trajectories (e.g., trajectories gathered from a diverse set of goals), termination improvement is typically observed in states where a multitude of actions have been executed, such as crossroads.\" It is not clear what the basis is for this observation. \n\nLine 264 - 268, column 1. Here a discussion on the agent policy would be useful. Does the policy matter? Are we making any assumptions about the agent policy when trajectories are collected. Would there be a difference if the agent is acting optimally within the current task versus acting randomly?\n\nFigure 5. Why is the success rate so low in the two mazes? In sparse block stacking, what is the maximum number of stacked blocks possible? \n\nLine 325 -- The problem should be specified by using the reward function not in terms of what the agent needs to do. How specially is the agent being rewarded? \n\nLine 318: \"A large set of task-agnostic agent experiences is collected from each environment\" Please state what the agent policy was in these environments. Was it the optimal policy for the task? Something else?\n\nThere is very little description of the environments in the main paper. A large part of the evidence provided against the claims in the paper comes from empirical evaluation. Therefore the structure of the environments is important in understanding the strengths and weaknesses of the proposed approach. Ideally the main paper would provide sufficient high-level description of the environments, and this is currently not the case. These include, for example, the states, actions, reward function, and some explanation of the domain dynamics, for example, the level of stochasticity. \n\nFor baseline skill-based algorithms with fixed execution length, it would be useful to explore the impact of the skill trajectory length on the results.", "questions_for_authors": "Do you think your proposed approach would be useful in conjunction with other existing methods of determining skill policies? Which ones? Have you done any analysis of the use of the proposed skill termination with different ways of setting skill policies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "v6ekvW1fbk", "reviewer": "Reviewer_M4gG", "summary": "This paper introduces NBDI (Novelty-based Decision Point Identification), a novel approach for learning termination conditions to extract skills from task-agnostic demonstrations. The method consists of using state-action novelty to identify critical decision points where skills should terminate, allowing for variable-length skill execution. The method leverages an intrinsic curiosity module (ICM) to estimate novelty, considering state-action pairs with high novelty scores as potential decision points. NBDI is evaluated across multiple environments (maze navigation and robotic manipulation tasks) and consistently outperform fixed-length skill extraction methods like SPiRL. The authors demonstrate that by executing variable-length skills that terminate at meaningful decision points, agents can explore more efficiently and better transfer knowledge across different tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. Please find the responses to your questions below.\n\n**Q1: Details in implementation of termination conditions**\n\nThank you for the suggestion. To improve clarity, we will revise Line 232 (second column) to include a more detailed explanation of the termination condition implementation, which is currently described in Appendix B.1. The revised sentence will read:  “During the training of the low-level policy, the deep latent variable model receives a randomly sampled experience from the training dataset, along with a termination condition vector $\\beta$ provided by the state-action novelty module. The model is trained to reconstruct the corresponding action sequence and its length (i.e., the termination point) by maximizing the evidence lower bound (ELBO).”\n\n**Q2: Adaptive thresholds for environments**\n\nThank you for your insightful suggestion. In practice, the thresholds we tuned through experiments (Appendix A.1) approximately correspond to the 97th percentile of the novelty values computed over task-agnostic demonstrations—e.g., kitchen = 0.3 (97.47th percentile), maze = 50 (96.86th percentile), and block stacking = 40 (96.12th percentile). We will include this percentile-based guidance in the paper to help readers more easily apply our method across new environments.\n\n**Q3: Comparison NBDI to other methods on discrete state/action space**\n\nThank you for your valuable suggestion. To provide an intuitive comparison between NBDI, Relative Novelty[1], and LOVE[2] in discrete settings, we conducted a case study using a grid-based maze environment. This allowed us to directly visualize and compare termination points across methods in a controlled, discrete domain. To align with our task-agnostic setup, we collected diverse expert-level demonstrations from random start and goal positions, and used these datasets to extract skills and termination points. The figure is available at: https://imgur.com/a/aRFrZ30.\n\n[1] defines novelty at a given state as the ratio between the average novelty of future and past states, measured within a fixed-size sliding window (n_lag). A high value indicates that the agent transitions from a familiar region to a less familiar one. Following the original formulation, we only evaluated states with sufficient trajectory context on both sides of the window. As shown in the visualization, Relative Novelty is highly dependent on transition history, often identifying only a subset of bottleneck states.\n\n[2] uses a variational inference framework to extract skills based on the Minimum Description Length (MDL) principle in that its objective is to “effectively compress a sequence of data by factoring out common structure”. While [2] employs a variational inference framework to implement the MDL principle, we used Byte Pair Encoding (BPE) to extract skills in discrete setting as BPE is a specific formulation of MDL [3] and it offers a more intuitive and interpretable formulation of compression. Termination points were visualized based on where the segmented skills terminated during 100 goal-reaching tasks. The results show that terminations vary significantly with the number of trajectories used to extract skills, as [2] focuses on capturing common structure rather than consistent bottlenecks.\n\nNBDI terminates skills based on both conditional action novelty and state novelty (Section 4.1). The visualization demonstrates that NBDI consistently identifies key bottleneck states and exhibits robustness to the number of trajectories collected in contrast to other methods. Moreover, states with high state novelty often correspond to regions that are rare or hard to reach within the task-agnostic dataset. By increasing the decision frequency in such unfamiliar states, NBDI promotes more effective exploration of the state space.\n\n\n[1] Ş., Özgür, and A. Barto. \"Using relative novelty to identify useful temporal abstractions in reinforcement learning.\" Proceedings of the twenty-first international conference on Machine learning. 2004.\n\n[2] J., Yiding, et al. \"Learning options via compression.\" Advances in Neural Information Processing Systems 35 (2022): 21184-21199.\n\n[3] G., Matthias. \"Investigating the effectiveness of BPE: The power of shorter sequences.\" Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019.", "claims_and_evidence": "The claims are generally well-supported by evidence. The authors demonstrate NBDI's greater performance compared to fixed-length skill approaches (like SPiRL) across multiple environments. The claim that state-action novelty can identify meaningful decision points is supported by visualizations showing that high novelty points correspond to intuitive decision points (e.g., crossroads in mazes, and completed subtasks in manipulation tasks).\n\nThe authors are transparent about limitations regarding sensitivity to dataset quality. They provide experiments (Section 6.5) and visualizations (Appendix C) to demonstrate how varying levels of stochasticity in demonstrations affect decision points detection.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate. The authors evaluate NBDI on two navigation tasks and two robotic manipulation tasks, providing diverse testing environments with continuous state and action spaces. They compare against multiple relevant baselines (SAC, BC+SAC, IQL+SAC, SPiRL, and other variable-length skill methods). \n\nThe metrics used (success rate for navigation, number of stacked blocks for manipulation, and number of completed subtasks for kitchen environment) are suitable for the respective tasks. The ablation studies (with \"NBDI-x\" baselines, or in Appendix A) isolate contributions from different novelty measures and design choices.", "theoretical_claims": "Not applicable (as no new theoretical claims)", "experimental_designs_or_analyses": "The analyses appear sound. The authors conduct experiments across multiple seeds (five different seeds per experiment) and report confidence intervals. They compare against ablations, fixed-length skill methods (SPiRL), other variable-length skill approaches (LOVE, relative novelty), and even algorithms that do not leverage temporal abstractions (flat RL methods). The analyses of (1) how dataset quality affects performance and (2) where the decision points are placed in maze environments both add valuable insights.", "relation_to_broader_scientific_literature": "The paper positions itself well within the literature on skill-based RL, particularly regarding skill extraction and termination condition learning. It builds upon the option framework (Sutton, 1998), and SPiRL (Pertsch et al., 2021), a recent fixed-length skill extraction method.\n\nThe authors acknowledge prior work on identifying sub-goals through novelty (Şimşek & Barto, 2004), but extend this to continuous state-action spaces. The work also relates to research on curiosity-driven exploration (Pathak et al., 2019) by repurposing novelty estimation for termination conditions rather than exploration bonuses.", "essential_references_not_discussed": "I am not aware of any essential reference not discussed in this work.", "other_strengths_and_weaknesses": "Strengths:\n\n- The approach is simple and clear. The theoretical motivation linking state-action novelty to termination improvement is well-justified.\n- Visualizations nicely illustrate how the detected decision points align with intuitive critical points in the environment.\n\nWeaknesses:\n\n- The novelty threshold is a critical hyperparameter requiring environment-specific tuning, which may limit generalizability.\n- The effectiveness appears sensitive to the quality of demonstration data, potentially limiting applicability with highly stochastic or suboptimal demonstrations.", "other_comments_or_suggestions": "The paper would benefit from clearer explanation of the termination condition implementation in Section 5.1.", "questions_for_authors": "1. Given the significant variation in novelty thresholds across environments (0.3 for kitchen vs. 50 for maze tasks), have you explored adaptive threshold setting approaches? For instance, would it be effective to set a coefficient $\\alpha$ as a hyperparameter, track the maximum novelty score $m$ observed during training, and consider all state-actions whose novelty is higher than $\\alpha m$ as critical decision points?\n2. How would you expect NBDI to compare against LOVE on discrete action environments, and against \"relative novelty\" approaches on discrete state spaces? This could help clarify the method's advantages across different types of environments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EDsmEoXOFt", "reviewer": "Reviewer_bZqs", "summary": "This paper presents  NBDI (Novelty-based Decision Point Identification), a state-action novelty-based decision point identification method that allows an agent to learn terminated skills from task-agnostic demonstrations. The key advancement presented in this work includes the mechanism for determining critical decision points, allowing variable-length skills to be learned. The authors present a cohesive methodology section followed by an abundance of results. Overall, the work concludes with several key advancements and interesting insights, displaying the importance of the formulation of the state-action novelty-based termination module.\n\n\n### Post-Rebuttal\nThank you for your responses regarding the diversity of the task demonstration set and the average length of skills learned. Please include this in the updated manuscript.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Please find our responses to your concerns below.\n\n**Q1: Diversity of the task demonstration set**\n\nThank you for your insightful suggestion. In our experiments, we found that the initial task-agnostic demonstration set needs to be diverse enough to cover the **observation space** (i.e., cropped images centered around the agent (Figure 2)), rather than fully covering the **state space** of downstream tasks. This diversity is especially important in environments like maze and block stacking, where downstream tasks involve significant changes in configuration (e.g., unseen maze layouts or random block arrangements in a larger-scale block stacking environment). To ensure sufficient diversity, we used 85,000 trajectories in the maze environment and 37,000 trajectories in the block stacking environment. In contrast, the kitchen environment, where the scene layout is consistent across tasks, required only 400 trajectories.\n\n**Q2: Details in the average length of skills learned**\n\nThank you for your feedback. In Appendix A.4, we explored the impact of fixed skill trajectory lengths by evaluating whether NBDI still outperforms SPiRL [1] when SPiRL is configured to use the average skill length observed in NBDI (referred to as SPiRL (avg-fixed)). We found that NBDI consistently outperforms SPiRL even when SPiRL uses these average fixed skill lengths. Interestingly, SPiRL (avg-fixed) performed worse than SPiRL with a fixed skill length of 10 in the block stacking environment, but better in the maze environment. However, across all fixed skill lengths tested (ranging from 10 to 30), there was no instance where SPiRL outperformed NBDI. These findings support our claim that NBDI can effectively leverage critical decision points in the environment compared to fixed length approaches.\n\nFurthermore, in Table 2 of Appendix A.4, we report the skill lengths of high-level policy for each environment during downstream reinforcement learning. The results show that our method maximizes temporal abstraction by frequently executing longer skills (lengths 21–30). At the same time, it retains the flexibility to terminate earlier (lengths 1–10) when critical decision points are encountered. This adaptability enables more efficient exploration of the state space and enhancing the transfer of knowledge across various tasks.\n\n**Q3: More informative figure captions**\n\nThank you for your suggestion. We will revise the captions for Figure 1 and Figure 2 to provide additional context about the tasks and domains depicted. Additionally, we will expand the caption for Figure 3 to briefly explain the visualization procedure, thereby helping readers better interpret the figures and their significance.\n\n[1] P., Karl, et al. \"Accelerating reinforcement learning with learned skill priors.\" Conference on robot learning. PMLR, 2021.", "claims_and_evidence": "Yes, the claims are supported by an abundance of evidence.", "methods_and_evaluation_criteria": "Yes, the paper has sufficient evaluation for the problem at hand.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the validity of the experimental design was checked thoroughly.", "relation_to_broader_scientific_literature": "Yes, the work is compared to several related works in Section 2 and compared against several works in Section 6.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n+ The paper contains abundant results and support for the proposed method.\n+ There are several intuitive explanations that help with understanding the paper.\n\n\nWeaknesses:\n- Due to the amount of material within the paper, many important details are found in the Appendix. The paper could be improved with longer figure captions.", "other_comments_or_suggestions": "- Could you comment on how diverse the initial task demonstration set can be (or needs to be)?", "questions_for_authors": "- Could you provide further details regarding the average length of skills learned?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nr8WFQZn09", "reviewer": "Reviewer_Qf8T", "summary": "The paper considers the problem of learning the termination conditions of the Option framework from task-agnostic demonstrations (demonstrations that are either exploratory or for other tasks). The key idea of the paper is to use state-action novelty to find states where termination will likely be reasonable to choose alternative skills: when a skill is finished or a decision-making should be made at an important point, the error of predicting the next state is lower. The paper's approach enables skills to be variable length based on the learned termination condition. The experiments show that the proposed approach improves performance on learning in unseen tasks by utilizing the termination condition to make essential decisions at appropriate states. The authors also conduct comprehensive benchmark tests against other termination condition learning approaches as well as ablations of its components. The results strongly support the claims made by the paper. In general, the paper presentation is very clear and easy to follow. Although the novelty of the paper is somewhat limited (by utilizing a previously known method of curating novelty to identify termination conditions), I believe the paper presents some meaningful contribution in the novel idea of applying novelty measure in the option framework, and the results show the benefit of doing so.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive suggestions. We address your questions below.\n\n**Q1: Details in skill termination and execution**\n\nThank you for your feedback. To clarify, the variable-length skill embedding space $z$ is learned **offline** during the skill extraction phase using a deep latent variable model. During this phase, the low-level policy (i.e., the skill decoder in Figure 12) receives a randomly sampled experience from the task-agnostic demonstration, along with a termination condition vector $\\beta$ provided by the pre-trained state-action novelty module. The latent model is trained to reconstruct the corresponding action sequence and its length (i.e., the termination point) by maximizing the evidence lower bound (ELBO) (Appendix B.1).\n\nDuring the execution phase (or when solving downstream tasks), the pre-trained low-level policy (i.e., the skill decoder) takes as input a latent skill vector $z$ from the high-level policy and reconstructs action and termination signal $\\beta$ at each step, without needing to observe or predict future states or actions (Appendix B.2). Thus, while future trajectory segments are used during skill extraction to learn variable-length skill embeddings and terminations, they are **not required at execution time**, ensuring our approach remains practical and deployable.\n\n**Q2: Task-Agnostic Dataset Size and Coverage**\n\nThank you for your feedback. The number of task-agnostic trajectories collected for each environment were as follows: maze (85,000 trajectories), block stacking (37,000 trajectories), and kitchen (400 trajectories). Since we used **image-based observations** in the maze and block stacking environments, and the downstream tasks involve significant configuration changes (e.g., entirely new maze layouts or a larger-scale block stacking environment with more blocks in random positions), we require a larger set of task-agnostic demonstrations compared to the kitchen environment, where the overall layout remains consistent across tasks (Appendix F).\n\nSince our goal in maze and block stacking environment is to solve downstream tasks with significantly different environment configuration, we do not assume that task-agnostic demonstrations fully cover the **state space** of the downstream tasks. However, it is important that they provide a good coverage of the **observation space**. Note we are using cropped image centered around the agent from task-agnostic demonstrations as observations. This way, we can extract consistent structural information that can be extracted across multiple task-agnostic trajectories, which then can be used to detect state-action novelty based decision points even in downstream tasks with significantly different environment configurations (Appendix F).\n\n**Q3: Impact of model capacity and dataset usage**\n\nThank you for the insightful feedback regarding the experiments. We agree with the reviewer that simply reporting the mean and standard deviation of state-action novelty across 25 trajectories may not sufficiently demonstrate the robustness of our method with respect to model capacity or dataset size. The concern that machine learning models can trivially capture mean values under i.i.d. sampling is valid, and we appreciate the opportunity to clarify our intent.\n\nTo address this, we revised the analysis in Figure 8 to present box plots that visualize the full distribution of state-action novelty scores across 25 sampled trajectories. The revised version of Figure 8 is available at: https://imgur.com/a/9gQ8IrU. This allows us to assess the variance, range, and consistency of state-action novelty estimates across different network sizes (left) and dataset usage levels (right). From the revised plots, it can be observed that the spread (variance, median and interquartile range) of state-action novelty remains relatively stable across different model sizes and dataset usage.\n\n**Q4: Paper structure rearrangement**\n\nThank you for your thoughtful suggestions regarding the structure of the paper. We agree that the current presentation of Sections 5.1 and 5.2 may limit accessibility for readers who are less familiar with skill extraction from task-agnostic demonstrations. To address this, we will incorporate concise summaries of Appendix B.1 and B.2 into Sections 5.1 and 5.2. Additionally, we will expand the caption of Figure 3 to briefly explain the visualization procedure used to generate the figure. This added context will help readers better interpret the figure and understand its significance. \n\nTo accommodate these changes within the page limit, we will move Figure 2(b) and its accompanying explanation (currently in Section 6.1) to the appendix.  Furthermore, we will relocate Figure 2 to Section 6, where it is first referenced, to improve the flow of the paper.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The idea of using novelty to detect termination conditions for options framework makes sense. The evaluation is done to see if the better termination conditions help options framework to adapt to unseen tasks faster and achieve better performance. \n1. One concern I do have is for Section 6.6 and Figure 8. Why is \"mean value of state-action novelty for 25 trajs\" a good metric to state \"proposed approach is robust to various number of parameters or size of datasets\"? To me, learning the mean value is the easiest thing to do for ML, and of course with an i.i.d. sampled sub dataset or a smaller neural network it can still learn it. The key of ML is to learn the variances among data (some data corresponds to a high predicted value vs. some data corresponds to a low predicted value). Please clarify if I misunderstood, otherwise I believe the experiment for Section 6.6 should be redone with a different metric.", "theoretical_claims": "The paper does not have theoretical claims.", "experimental_designs_or_analyses": "Yes, the experimental design is sound and through in comparing with benchmarks and ablations of the proposed approach.", "relation_to_broader_scientific_literature": "The contribution of the paper is to relax previous option framework's fixed-length skills to be variant-length skills according to the learned termination function.", "essential_references_not_discussed": "Not that I am aware of", "other_strengths_and_weaknesses": "Although I find the paper overall well-written, I do have a few suggestions for the structure of the paper. Currently, the technical details of Section 5.1 and 5.2 are largely in supplementary, which I believe hinders the ability for readers that are less familiar with the topic to appreciate the work. The learning of the novelty function \\Chi(s,a), and the significant novelty criteria to determine termination, \\beta, are important details to understand the reproduce the approach. As such, I believe moving them into the main paper would be beneficial, and for the page limit's sake, some contents of Section 6.1 could be moved to supplementary, as the illustration of Figure 2a and 2b seem very similar. Similarly, Figure 3 seems hard to interpret without reading Appendix G.1, and I recommend at least providing a brief explanation of how the figure is generated.", "other_comments_or_suggestions": "The Figure 2 should be placed later in the paper as it is no introduced until Section 6.", "questions_for_authors": "1. According to Section 5.1, z is the embedding of the s-a pairs and \\beta from time t to t+H-1. It is unclear to me how this information is available during execution time - how can one predict the future trajectory in order to calculate z? \n2. According to line 318 (right column), \"a large set of task-agnostic agent experiences is collected\" - how large is this dataset? Is it assumed that the demonstration set covers the state space well?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission received 3 accepts and 1 weak reject, with all reviewers recognizing that the paper introduces a well-motivated and practical approach to skill termination in RL through Novelty-Based Decision Point Identification (NBDI). Strengths highlighted include strong empirical performance across diverse environments, clear methodology, insightful visualizations, and the effective application of state-action novelty for variable-length skill learning. While initial concerns were raised regarding the breadth of evaluated baselines, sensitivity to dataset quality, hyperparameter tuning (e.g., novelty thresholds), the authors provided detailed rebuttals, additional analyses, and clarifications, for example adaptive thresholding strategies, broader applicability to other skill-based methods, and improved experimental explanations. Overall, most reviewers agreed that this paper offers meaningful contributions to skill-based RL, particularly in enhancing generalization and efficiency through learned termination conditions. The AC would have given a higher recommendation if the submission provided more thorough evaluations on more recent benchmarks compared to the ones in SPiRL.
Decision: Accept (poster)