Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning
Authors: Seungyul Han
Sanghyeon Lee
Sangjun Bae
Yisak Park
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Meta Reinforcement Learning
Skill-based Reinforcement Learning
Hierarchical
Noisy Demonstration
Skill Refinement
Generalization
Primary area: reinforcement learning

Abstract: Meta-reinforcement learning (Meta-RL) facilitates rapid adaptation to unseen tasks but faces challenges in long-horizon environments. Skill-based approaches tackle this by decomposing state-action sequences into reusable skills and employing hierarchical decision-making. However, these methods are highly susceptible to noisy offline demonstrations, leading to unstable skill learning and degraded performance. To address this, we propose Self-Improving Skill Learning (SISL), which performs self-guided skill refinement using decoupled high-level and skill improvement policies, while applying skill prioritization via maximum return relabeling to focus updates on task-relevant trajectories, resulting in robust and stable adaptation even under noisy and suboptimal data. By mitigating the effect of noise, SISL achieves reliable skill learning and consistently outperforms other skill-based meta-RL methods on diverse long-horizon tasks. Our code is available at https://github.com/epsilog/SISL.

Published as a conference paper at ICLR 2026
SELF-IMPROVING SKILL LEARNING FOR ROBUST SKILL-BASED REITA-REINFOR LEARNING
Sanghyeon Lee, Sangjun Bae, Yisak Park, Seungyul Han"
Graduate School of Artificial Intelligence
Ulsan National Institute of Science and Technology (UNIST)
Ulsan, South Korea 44919 {s (sanghyeon, bajuntujutti iger, isaac1018, syhan] ac.kr
ABSTRACT
Meta-reinforcemeen learning (Meta-RL) facilitates rapid adaptation to unseen tasks but faces challenges long-horizon environments. Skill-based approaches tackle this by decomposing state-action sequences into reusable skills and employ- ing hierarchical decision-making. However, these methods are highly susceptible noisy offline demonstrations, leading to unstable skill learning and degraded performance. To address this, we propose Improiin Skill Learning (SISL), which performs self-guided skill refinement using decoupled high-level and skill improvement policies, while applying skill prioritization via maximum return re- labeling focus updates on task-relevant trajectories, resulting in robust and ble adaptation even under noisy and suboptimal data. By mitigating the effect of noise, SISL achieves reliable skill learning and consistently outperforms other skill-based meta-RL methods on diverse long-horizon tasks. Our code is available at https://gitubb.com//sslll/////
INTRODUCTION
Reinforcement Learning (RL) has achieved significant success in various domains (nni et al. 2013 Andrychowicz et al. 2020). While advanced exploration frameworks have been proposed to
improve sample efficiency (Pathkk al. 201 et al.) 2018 Han & Sung 202 Jo et al.
2024), traditional RI struggles to adapt quickly to new tasks. Meta-RI. addresses this limitation by enabling rapid adaptation to unseen tasks through meta-learning how policies solve problems (Duan) jet al., 2016;; Finn et al.) (017). Among various approaches, context-based meta-RL stands out for its ability to represent similar tasks with analogous contexts and leverage this information in the policy, facilitating quick adaptation to new tasks (Rakelly et al. 2019; Zintgrat' et al. 2019) Kim al. 2023). Notably. PEARL (Rakelly et al. 2019) has been widely studied for its high sample efficiency, achieved through off off-policy learning, which allows for the reuse of previous samples. Despite these strengths, existing meta-RL. methods face challenges in long-horizon environments, where extracting meaningful context information becomes difficult, hindering effective learning.
Skill-based approaches address these challenges by breaking down long state-action sequences into reusable skills, facilitating hierarchical decision-making and enhancing efficiency in complex tasks (Pertsch et al. 2021 2022 Shi et al. 2023). Among these, SPiRL (Pertsch et al.) 2021) defines skills temporal abstractions of actions, employing them low-level policies within hierarchi- cal framework to achieve success in long-horizon tasks. SiMPL (Nam et al., (022) builds on this by extending skill learning meta-RI,, using offline expert data to train skills and a context-based high-level policy for task-specific skill selection. Despite these advancements, such methods are highly susceptible to noisy offline demonstrations, which can destabilize skill learning and reduce reliability. In real-world settings, noise often stems from factors like infrastructure aging or environ- mental perturbations, making it crucial design methods that remain robust under such conditions
(Brys et al. 2015 Chae et al. 2022 Yu et al. 2024a
Lee et al. 2025)
While noisy demonstration handling has been explored in other RL settings (Sasaki & Yamashina 2020 Mandlekar et al. 2022), skill- basled meta-RL has largely overlooked this challenge. We iden-
indicates the corresponding author: Scungyul Han.

---

Published as a conference paper at ICLR 2026
Train Starting point End
Path
Skil
Explonation
Task
Starting point End point
(c)
Figure I:
Sample trajectories in the Maze2D environment: (a) Noisy demonstrations from the offline dataset, (b) Trajectories explored by the exploration policy near the noisy dataset to uncover useful skills, and (c) Trajectories utilizing refined skills to solve unseen test tasks
tify a critical failure mode: when offline data are suboptimal, the skill library becomes corrupted, and this degradation propagates to the high-level policy, ultimately harming adaptation performance. To address this, we propose Self-Improving Skill Learning (SISL), a robust skill-based meta-RL frame- work with two key contributions: (1) Decoupled skill self-improvement, using an improvement pol- icy that perturbs trajectories near the offline data distribution, discovers higher-quality rollouts, and self-supervises updates via a prioritized online buffer. This process progressively denoises the skill library while preserving stability. (2) Skill prioritization via maximum return relabeling, which eval- uates offline trajectories with a learned reward model, assigns rask-relevant hypothetical returns, and reweights them through a softmax prioritization scheme. This suppresses noisy or irrelevant samples and focuses skill updates on the most beneficial trajectories for adaptation. These components dy- namically balance offline and online data contributions, yielding progressively cleaner skill library and accelerating meta-RL RL convergence. To our knowledge, SISL is the first framework to explicilly address suboptimal demonstrations in skill-based meta-RL, significantly improving robustness and generalization of skill-learning in real-world noisy scenarios.
Fig. [ illustrates how the proposed algorithm learns effective skills from noisy demonstrations in the Maze2l) environment, where the agent starts at a designated point and must reach an endpoint for each task. Fig. aa) shows noisy offline trajectories, which fail produce effective skills when used directly. In contrast, Fig. (Tb) demonstrates how the prioritized refinement framework uses the improvement policy navigate near noisy trajectories, identifying paths critical for solving long-horizon tasks and refining useful skills through prioritization. Finally, Fig. [c) shows how the high-level policy applies these refined skills to successfully solve unseen tasks. These results highlight the method's 's ability to refine and prioritize skills from noisy datasets, ensuring stable learning and enabling the resolution of long-horizon tasks in unseen environments. This paper is organized as follows: Section 3 an overview of meta-RL and skill learning, Section the proposed framework, and Section [ presents experimental results showcasing the framework's s robustness and effectiveness, along with ablation study of key components.
2 RELATED WORKS
Skill-based Reinforcement Learning: Skill-based RL has gained traction for tackling complex tasks by leveraging temporally extended actions. Researchers have proposed information-theorret approaches to discover diverse and predictable skills (Gregor et al. 2016 Eysenbach et al.) 2018
Achiam et al., 2018 Sharma et al., 2201)) with recent work improving skill quality through addi- tional constraints and objectives (Strouse et al.) 2022 Park et al.) 2022 2023 Hu et al. 2024). In
offline scenarios, approaches focus on learning transferable behavior priors and hierarchical skills
from demonstration data (Pertsch et al. 2021, 2022) Shi et al. 2023 Xu et al. [022] Kipf et al.)
2019 Rana et al.) 2023). Building upon these foundations, various basel meta-RL. approaches have been developed, from hierarchical and sabbedding-basedd methods (Nam et al. 2022 Chien &
Lai 2023 Cho & Sun| 2024) to task decomposition strategies (Yoo et al. 2022 He et al. 20244 and
unsupervised frameworks (Gupta et al. 2018 Jabri et al. 2019 Shin et al. 2024). For clarity,
skill in the canonical skill-basch RL sense: a latent over fixed-lengte action sequences. Therefore, studies YYu et al. 2024b Wu et al. 2019) that lack explicit skill learning are not categorized here.
2

---

Published as a conference paper at ICLR 2026
Relabeling Techniques for Meta-RL: Recent developments in meta-RL have introduced various relabeling techniques to enhance sample efficiency and task generalization (Pong et al. 2022 Jiang et 2023). Goal relabeling approaches have extended hindsight experience replay to meta-learning contexts (Packer et al., 2021 Wan et al., 2021), enabling agents to learn from failed attempts. For re- ward relabeling, model based approaches have been proposed relabel experiences across different tasks (Mendonca et al. 2020), improving adaptation to out-of-distributin scenarios. Beyond these categories, some methods have introduced innovative relabeling strategies using contrastive learning |Yuan & Lu 2022 Zhoou et al. 2024) and metric-based approaches (Li et al., 2020 to create robust task representations in offline settings.
Hierarchical Frameworks: Hierarchical approaches in RL have been pivotal for solving long-
horizon tasks, including goal-conditioned learning (I (Lvv et al. 2019 Li et al.
2019 2019 Gehring al. Barreto et al. 2019
2021), and option-based frameworks (Bacon et al. (017eRiemmer et al. 2018
Araki et al. 2021). The integration of hierarchical frameworks with meta- RL
has shown potential for rapid task adaptation and complexity handling (Frans et al. 2018 Fu et al. 2020a 2023) Recent work has demonstrated that hierarchical architectures in meta- RL can provide theoretical guarantees for learning optimal policies (Chua et al. 2023) and achieve efficient learning through transformer-
based architectures (Shala et 20.44. Recent advances in goal- coul-condiiioeed RL have focused on improving sample efficiency (Robert al.,
2024 Hwang et al.) 2025), state representation (Yin) [024] (Schmidt et al. (025). Offline-to-online
eet al. 2024), and offline-to-online RI. (Park et al.)
RL assumes reward-annotated offline datasets to pretrain the policy based on RL before online fine-
tuning. In contrast, our setting provides only reward-frre offline data for skill learning, making
direct application of offline-toonlline RL infcasible and clearly distinguishing our approach.
BACKGROUND
Rect-RReinfrremment Learning Setup: In meta-RL, each task T is sampled from a distribution P(T) and defined as an MDP environment M = (S, RT, RRT ,T,,), where S x A represents the state-action space, Rd is the reward function, pT denotes the state transition probability, and is the discount factor. At each step t, the agent selects an action a via the policy π, receives a reward rz := Rd (st, dt), and transitions to St+1 ~ =P (-[st, at). The goal of meta-RL is to train to maximize the return G = EE,'r, on the training task set Merain while enabling rapid adaptation to unseen test tasks in Mtest, where Merain 0 Miest 0.
To address bong-horizon tasks, skill learning from an offline dataset Bon == (f0.1} is considered, which comprises sample trajectories fru+* (8t, "r.,.., .8+*) without reward information, where H is the episode length. The dataset Bar is typically collected through human interactions pretrained policies. Among various skill learn- ing methods, SPiRL (Pertsch et al. (02T]) focuses on learning a reusable low-level policy m, using q( ((.17 as a skill encoder to extract the skill latent 2 by minimizing the following loss function:
Offline Dataset and Skill Learning:
E Fru H, BBa, (C(mL,9.p,2) zngg (Fuettr))
where [(T,, (...) - log (ag (8,, + X DKL (O) + DKL (Ialll)) H, is the skill length, 2 the coefficient for KL. divergence (KLD) D L-] the stop gradient operator, and (H,, Σ) represents a Normal distribution with mean μ and covariance matrix Σ. Here, p(= [()) is the skill prior to obtain the skill distribution 2 for a given state 8; directly. Using the learned skill policy ne the high-level policy na is trained within a hierarchical framework using RL methods. In our paper, the skill refinement procedure builds on Eq. T and is applied during both initial skill learning phase and subsequent refinement. A more detailed description is provided in Appendix B.
Sksll-based Reet--Reinforeementt Learning: SiMPL (Nam et al., 2022) integrates skill learning into meta-RA RL by utilizing offline dataset of expert demonstrations across various tasks. The skill policy my is trained via SPiRI, while a task encoder de extracts the task latent eT ~ qe using the PEARL (Rakelly et al.
2019 framework, a widely-used meta-RI. method. During meta-training. the high-level policy Th [2) ss ee') selects skill latent and executes the skill policy ii(als, 2) for H. time steps, optimizing TT to maximize the return for each task T as:
min E ~BT Te ma [ (T()) + X (T 11p) Th.
(2)
3

---

Published as a conference paper at ICLR 2026
where A is the KL divergence coefficient, c² represents the contexts of high-level trajectories TH = (50, 20, _Haa1 Vr, HH., Lt=H, - T.....) for task T, CRL denotes the RL loss for The and T {Th is the high-level buffer that stores ThK for each T E Merain. Here, the reward sums Σi=kH1, Tt are obtained via environment interactions of a, ~ xT(-]st, 2kH.) for HH ..... (k 1)H, with k = ..... During meta-test, the high-level policy adapted using limited number of samples, showing good performance on long-horize tasks.
4 METHODOL.GGY
4.1 MOTIVATION: TOWARD ROBUST SKILL LEARNING UNDER NOISY DEMONSTRATIONS
Noisy Skill
Expert Skill
task success
Initial State
Expert
Noley
c 0
15
15
0.1K 0:K Daк DAK 0.5K Iteration
(b)
Figure 2: Comparison of prior skill learning method in microwave-openigg task: (a) Learned skills with expert and noisy demonstrations. (b) Meta-RI. performance with learned skills. Low-level skills are learned via SPiRI., while the meta-learning component follows the structure of SiMPL.
Most existing skill-based meta-RL. approaches discussed in Section assume clean offline demon- strations, but real-world datasets are often corrupted by noise from aging hardware, disturbances, or sensor drift. Unlike online training that can adapt through continuous re-training, static offline datasets particularly susceptible to such noise. This issue becomes critical in long-horizon tasks, where errors accumulate, and in precise manipulation tasks that require reliable execution. Fig. 2(a) illustrates this problem: in the Kitchen microwave-opening task, skills learned from expert demon- strations complete the task successfully. whereas skills learned from noisy data fail even to grasp the handle. This results significant downstream performance drop, as shown in Fig. bb), where noisy skills lead to poor task success rates and unstable training curves, with each iteration denoting a training loop consisting of policy rollout and update. The root cause that existing methods treat all trajectories equally, allowing low-quality samples to dominate skill learning.
To address this, we propose the Self-Improving Skill Learning (SISL) framework, which enhances meta-RI. by introducing a decoupled skill improvement policy. The high-level policy maximizes returns using the current skill library, while the improvement policy independently perturbs trajec- tories near the offline data distribution to discover higher-quality variants. The resulting trajectories are selectively stored and prioritized, then used to refine the skill encoder and low-level policy. Through this iterative refinement, SISL progressively denoises the skill library and improves gener- alization. We further incorporate prioritized buffering and maximum return relabeling as auxiliary mechanisms that enhance sample efficiency and accelerate convergence.
4.2 SEL.--MPROVING SKILI. REFINEMENT WITH DECOUPLED POLICIES
We now describe the proposed Self-Improving Skill Learning (SISL) framework, which formal- izes the iterative refinement process motivated in Section 4.1 We adopt the standard saied- meta-RL. setup where the skill encoder (t,fftt extracts latent skill from trajectory seg- ments, the high-level policy Th (z|s, e² selects z every H, steps, and the low-level skill policy mi(als, z) executes the chosen skill for H, steps. Building on this setup, SISL decouples training into two complementary components: a high-level policy тr that exploits the current skill library to maximize return, and skill-impir skill-improvement policy (a)s(a))s,,)) i), defined for each training task T, (i 1,....Ny ..r.ain) that deliberately perturbs trajectories near the offline data distribution to discover improved behaviors. This decoupling enables simultaneous exploitation and targeted skill
4

---

Published as a conference paper at ICLR 2026
Decoupled Policy Learning
Self-Improvement Skill Learning
Maximum Return Relabeling
G
Taxc Excoder
qe
p
Bon
1.2
Skil Ensodar
h
imp
Tt
q
Talk Tast2 Taak3
Skiill
Prtorhtzed
Fq
(
Boff
(F22)
B
h
imp
B on
Boff
G(tg)
PB
Rollout 4 Prionitaation Filterigg
Figure 3: Proposed SISL framework. The de- coupled policy learning phase (Th, Timp) to solve tasks and discover improved behav- iors. The skill learning phase periodically up- dates the skill components (TI, q).
Figure 4: Maximum return relabeling. After re- labeling the online buffers 8% and the offline buffer Boff using the learned reward model, SISL performs prioritized skill refinement by mixing both buffers via coefficient 3.
improvement. Perturbations are guided by intrinsic motivation signals (Burda et al. 2018) to encour- age coverage of diverse trajectories, although any eevelty-driven exploration mechanism could be used. Importantly, unlike generic novelty-driven exploration, Timp restricts perturbations to remain close to the offline data manifold, producing realistic trajectories that can be effectively used to refine However, when the offline dataset contains noise, training Timp near this distribution in the early stages may be hindered by low-quality samples, reducing the effectiveness of skill improvement.
To overcome this issue, SISL employs a self-supervised guidance mechanism using two additional buffers: The improvement buffer Bimp [Timpp stores all trajectories generated by Timp- where each trajectory is imp = (sb, ag. s s3H) with af ~ (- i), and is directly used to update Timp itself, and the prioritized online buffer B. = high selectively retains the highest-return
trajectories, where each Tigh is chosen based on its return G(r*) = Etoo > This buffer is initialized with the offline dataset Bof and gradually becomes dominated by improved trajectories generated by both Timp and πи This buffer serves two purposes: (1) it provides cleaner, progres- sively improving dataset that supervises Timp in a self-supervised manner, guiding it toward regions that have empirically led to success, and (2) it supplies high-quality samples for refining the skill encoder skill prior p, and low-level policy πi
The skill-improvement policy is then trained that its trajectory distribution increasingly matches the successful samples in Bi while still exploring variations through controlled perturbations:
Σ Ey By By UB (imp (Tmp)) +
A Amp E,t.B DKL ( Timp);
where Akldd is the KLD coefficient and #t denotes the empirical action distribution derived from B Chup consists of the standard RL loss combined with intrinsic reward terms for perturbation, with details provided in Appendix B This loss encourages Timp to iteratively focus on more promising state-action regions discovered so far, effectively turning B into a self-improving curriculum that steers exploration away from noisy or uninformative samples.
Building on the improved trajectory distribution obtained from this update, SISL performs skill refinement in dedicated phases every Kitt iterations rather than after every update step. At the end of each phase, the skill encoder q. skill prior and low-level policy are re- re-trained on both Bar and B using the SPiRL objective in Eq. (1.. and the high-level policy m, reinitialized to fully benefit from the updated skill library. This periodic refinement mitigates bias from outdated skill embeddings and accelerates adaptation, as shown in our ablation study in Appendix G.4] These phases yield progressively denoised skills and a stable training signal, enabling Tr to solve tasks more efficiently as training progresses. The overall SISL structure is illustrated in Fig. 3 However, the large number of noisy trajectories in Bod can still reduce skill learning efficiency. The next section introduces a maximum return relabeling mechanism that prioritizes the most relevant offline trajectories so that only high-value samples significantly influence skill refinement.
5

---

Published as a conference paper at ICLR 2026
4.3 SKILL PRIORITIZATION VIA MAXIMUM RETURN RELABELING
The proposed SISL refines the low-level skills by leveraging both the offline dataset Bor and the prioritized online buffers 8 for each training task T. While B provides high-quality trajectories collected from the skill-improvement policy Timp and successful rollouts from тy. relying solely on risks overfitting diverse trajectories that are beneficial for generalization but also contains many noisy subopti- mal rollouts, which can degrade skill quality if sampled uniformly. To address this trade-off, we introduce skill prioritization via a maximum return relabeling mechanism that assigns hypothetical returns to offline trajectories and reweights samples in both Bot and Bi according to their estimated task relevance. Specifically, maximum return relabeling assigns each trajectory E Bor hypothet- ical return that reflects its potential contribution to task success. To compute this return. SISL trains a reward model R(2,, az, i) for each task Tú, optimized with the regression loss
narrow distribution and limiting generalization. Conversely, Ber provides
E (st.at B amp us:
of i) - r.
(4)
where the targets rf come from improved trajectories generated by Timp and stored in B Since this regression is performed online using actual environment rewards, it remains stable throughout training. Using the trained reward model, SISL computes for each F the maximum return
G(F) = max Σ Ex )
or. (rlss,
(5)
which represents the highest predicted cumulative reward across all training tasks. The offline tra- jectories are then sampled according softmax distribution PB. (f) Softmax(((F)/TT where
T > 0 is a temperature parameter controlling prioritization sharpness. This procedure biases sam- pling toward promising trajectories while suppressing noisy irrelevant ones, resulting in cleaner training signal for skill learning. The resulting skill learning objective becomes
Catai (.P) (1 B) (C(m,, x-)((15)
Nr,arain Σ Bian ((p,)) smg(113")
(6)
where C(mi,q.,,3 is the SPiRL skill loss defined in Eq. Since B.n already contains high-return trajectories, samples are drawn uniformly from this buffer. In addition, the mixing coefficient 3 is computed dynamically from the offline and online datasets based on their average returns, adaptively balancing their contributions during training as
exp(Gon/T)
3 Exp((Gan/T + GT
(7)
where Gor is the mean & across Bar and Gon is the mean return in B% Fig- illustrates the prioritization process, showing how 3 dynamically balances contributions from offinne and online datasets. This mechanism ensures the selection of task-relevant trajectories from both datasets, facilitating efficient training of the low-level policy. As result, meta-training yields a refined low- level policy Ni and high-level policy Ta optimized over progressively cleaner data. During meta- test, Ti is frozen and is adapted to unseen tasks using a small number of interaction trajectories, following the other skill-based meta-RL methods. Additional implementation details for the meta- train and meta-test phases, along with the algorithm table, are provided in Appendix B
5 EXPERIMENT
In this section, we evaluate the robustness of the SISL framework to noisy demonstrations in long- horizon environments and analyze how self-improving skill learning enhances performance.
5.1 EXPERIMENTAL SETUP
We compare the proposed SISL with 3 non-meta RL baselines: SAC. which trains test tasks di- rectly without using the offline dataset; SAC+RND, which incorporates RND-based intrinsic noise for enhanced exploration; and SPiRL, which learns skills from the offline dataset using Eq. ①

---

Published as a conference paper at ICLR 2026
(a) Kitchen
(b) Office
(c) Maze2D
(d) AntMaze
Figure 5: Considered long-horizon environments
and trains high-level policies for individual tasks. Also, we include 4 meta-RI. baselines: PEARL, widely used context-based meta-RL RL algorithm without skill learning: PEARL+RND, which in- tegrates RND-based exploration into PEARL; SiMPL, which applies skill-based meta-RL using Eq. and our SISL. SISL's hyperparameters primarily follow Nam et al. (2022). with additional parameters (e.g., temperature T tuned via hyperparameter search, while other baselines use author- provided code. Although SISL introduces additional improvement policy Timp- ensured fair comparison by keeping the total amount of training and interaction the same. Specifically, in each iteration only half of the sampled meta-train tasks T, and the other half use Timp: the total number of samples and policy updates does not exceed that of SiMPL. Results are averaged over 5 random seeds, with standard deviations represented as shaded areas in graphs and + values in tables.
5.2 ENVIRONMENTAL SETUP
We evaluate SISL across four long-horizon, multi-task environments: Kitchen and Maze2D from Nam et al. (2022), and Office and AntMaze, newly introduced in this work, as illustrated in Fig. 5 Offline datasets Boff are generated by perturbing expert policies with varying levels of Gaussian action noise, tailored to each environment. In the Kitchen environment, based on the Franka Kitchen from the D4RL benchmark (Fu et al.) 2020b) and proposed by Gupta et al. 2020). a robotic arm 0.1, 0.2, and 0.3.
completes sequence of subtasks, with noise levels ranging from expert to The Office environment, adapted from Pertsch et al. (2022). involves picking and placing randomly selected objects into containers. with Gaussian noise applied the same levels Kitchen. Maze2D, based on D4RL. (Fu et al.] 2020b), requires a point-mass agent to navigate a large 20x20 maze, while AntMaze features a more complex ant agent maneuvering through 10x10 maze. In both Maze2D and AntMaze, Gaussian noise is introduced at higher levels of a = 0.5, 1.0, and 1.5. Each environment is structured with distinct meta-trait train and meta-test tasks, ensuring that test tasks involve different goals from those seen during training. Further experimental details, including descriptions of other baselines, environment configurations (number of tasks, state representations, and reward setups), initial data collection processes, and hyperparameter settings, are provided in Appendix
5.3 PERFORMANCE COMPARISON
We compare SISL with various baseline algorithms. Non-met RL algorithms trained directly on each test task for 0.5K iterations due to the absence of a meta-train phase. Meta-RL algorithms undergo meta-training for 10K iterations in Kitchen and Office, and 4K in Maze2D and AntMaze, followed by fine-tuning on test tasks for an additional 0.5K iterations. For SISL, the skill refinement interval Kiter is set 2K for Kitchen, Office, and AntMaze: IK for Maze2D. To ensure a fair com- parison, SISL counts each update process from its skill-improvement policy and high-level policy as one iteration. Table shows final average return across test tasks after the specified test iterations. In these environments. Kitchen and Office assign a reward of 1 for each successfully completed subtask, while Maze2D and AntMaze give reward of upon reaching the goal, meaning the return directly corresponds to the success rate. The corresponding learning curves are in Appendix [D] for a more detailed comparison. From the result, SAC and PEARL baselines, which do not utilize skills or offline datasets, perform poorly on long-horizon tasks, yielding single result across all noise levels. In contrast, SPiRL, SiMPL, and SISL, which leverage skills, achieve better performance.
SPiRL and SiMPL, however, show sharp performance declines as dataset noise increases. While both perform well with expert data, SiMPL struggles under noisy conditions due to instability in its task encoder de. sometimes performing worse than SPiRL. Here, the baseline results for Maze2D (Expert) are somewhat lower than those reported in the SiMPL paper. This discrepancy likely arises because, in constructing the offline dataset, considered fewer tasks compared to SiMPL,, result-

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison: Final test average return for all considered environments
Envirommennt/Noss)
SAC
SAC+RND
PEARL
PEARLARND
SPIRL
SiMPL
SISL
Kitchen(Expert)
3.11+0.38
3.40=0.18
3.97+0000
Kitchenia = 0.1)
Kitchenía 0.2)
0.01
0.02 #0.05
0.23 #0.14
0.4240.16
3.37=0.31
2.06+0.48
2.18-0.38
3.76=0.14
3.73++0.1
20.12
Kitchenta 0.3)
0.83+0.17
0.81
3.48200.07
Office(Expert
0.65.10.24 24
2.50
2 1.8610.35
Officeto 0.1)
0.00:10.00
0.00
0.01
0.0110.01
0.91
3.33=0.39
8.40220.38
Officeto 0.2)
0.49110,22
1.20
2.01
Officeto 0.3)
0.4220.1 14
0.11=0.04
1. 1.6810.115
Maxe2D(Expert)
0.77.10.00
0.80
0. 1.0.5
Maze2Dia = 0.5)
0.20:10.06
0.3610.07
0.1010.01
0.1110.08
0.8920.03
0.87=0.05
0.89110003
Maac2Die = 1.0)
0.80.10.01
0,87
0.93.10.05
Maze2Dio 1.5)
0.81
0.68=0.06
0.99100.02
AntMaze(Expertt
0.64
0,07
0.80100.8
AmMazzlo = 0.5)
AntMazeee 1.0)
0.00=t0.00
0.00=0.00
0.00=0.00
0.00+6.00
0.50100.6
0.76++0.10
0.33-0009
0.77=0.00
0.60 0.6010.02
0.82++0.05
AntMazel = 1.5)
0.30++0.01
0.27=0.05
0.41 4.1100001
Meta-Train
Meta-Test
Meta-Train
Meta-Test
1.0
1.0
0.8
0.8
Average
0.4
0.4
0
2k
4K
6K
8K
10K
0
0.1K D.2K 0.3K 0.4K 0.5k
0
1K
2K
3K
4K
D.1K 0.2K D.k... D.5K Iteration
Iteration
Iteration
Iteration
Kitchen(a==
(b) Maze2D(a
S1SL//hrp)
Si5Lin))
SIMPL
SPIRL
PEARL RND
PEARL
SAC RND
SAC
Figure 6: Learning curves of the meta-train and meta-test phases on Kitchen (a= = 0.3) and Maze2D (a 1.5). SISL (Tump) and SISL (T₁) denote the performance of the sail-imprrveemett policy Tamp and high-level policy Th during meta-training.
ing in trajectories that do not fully cover the map. Interestingly, minor noise occasionally boosts performance by introducing diverse trajectories that improve skill learning, detailed analysis of changes in state coverage is provided in Appendix D.I In contrast, SISI. demonstrates superior robustness across all evaluated environments, consistently outperforming baselines at varying noise levels. For example, in the Kitchen environment, SISL maintains strong performance under signif- icant noise by effectively refining useful skills, while in Maze2D, higher noise levels lead to the improvement of diverse skills, achieving perfect task completion when a = 1.5. These results high- light SISL's ability to discover improved behavior and refine robust skills, significantly enhancing meta-RL performance. Moreover, SISL excels with both noisy and expert data, achieving superior test performance by learning more effective skills.
Fig. shows the learning progress during the meta-train/test phases for Kitchen (a 0.3) and Maze2D (a = L.5), highlighting the performance gap between SISL and other methods. The peri- odic drops in SISL's high-level performance correspond to the reinitialization of TT every Kiter- Non- meta RL algorithms, including those with RND-based exploration, struggle with long- long-horizon tasks, while SPiRL and SiMPL show limited improvement due to their reliance noisy offline datasets. In contrast, SISL's self-improving skill refinement supports continuous skill improvement, resulting in superior meta-test performance. To further assess SISL's robustness, we perform experiments with limited offline data, random noise injection, and diverse sub-optimal datasets in Appendix D reflecting real- world challenges such as costly data collection and unstructured anomalies. Even un- der these conditions, SISL consistently outperforms the basclines, demonstrating strong robustness. Furthermore, we provide a computational complexity comparison of SISL,, its ablation variants, and baseline methods in Appendix [E] The results show that SISL requires only about 16% more compu- tation time per iteration during meta-training, while the meta-test cost remains unchanged compared to SiMPL. Although SISL introduces the improvement policy Timp. the overall training cost remains similar because the total number of samples and updates matches SiMPL by splitting training tasks evenly between Timp and n for fair comparison, as described in Section 5.1 The remaining 16% overhead comes primarily from the skill refinement and reward model trainig. Notably, even with extended training, SiMPI. fails to achieve further performance gains, highlighting SISL's advantage.
8

---

Published as a conference paper at ICLR 2026
Mixing Coeff. (B)
Kitchen
1k < Kieer
5k
10k
Kitchen
0.2
Starting
1-3
c- 15
30
2 15
[ 30
Maze2D
0.5k K
1.
0.8
Troectory
lleration
Figure 7: Visualization of buffer mixing coefficient, 8 dynamics and refined skill evolution in Kitchen
(a = 0.3) and Maze2D (a = 1.5). In Kitchen, the refined skills at ( 15 and L = 30 during the
microwave--ppenng task are depicted, while Maze2D, trajectories using refined skills illustrate the process of progressively expanding to broader areas to solve tasks.
5.4 IN-DEPTH ANALYSIS OF THE PROPOSED SKILL REFINEMENT PROCESS
To analyze skill refinement and prioritization in more detail, Fig. illustrates the evolution of the buffer mixing coefficient and skill refinement in Kitchen (a 0.3) and Maze2D (a 1.5). For Kitchen, the microwave-opening subtask is evaluated, while Maze2D focuses on navigation improvements.
the early stages (IK iterations for Kitchen, 0.5K for Maze2D), pretrained skills from the offline dataset are used without updates, resulting in poor performance, with the agent failing to grasp the handle Kitchen and producing noisy trajectories in Maze2D. As training progresses, 3 increases to shift contribution from offline data to newly collected high-quality data, providing task-relevant refinement. By design, the increase of B tracks task-return improvement and serves soft curriculum that prevents abrupt distribution shift. At the same time, prioritized offline samples keep & below 1, thereby ensuring generalization.
As result, by iteration 5K in Kitchen, the agent learns open the microwave, refining this skill to complete the task more efficiently by iteration 10K. In Maze2D, the agent explores more diverse tra- jectories over iterations, ultimately solving all training tasks by iteration 4K. These results highlight how SISL refines skills iteratively by leveraging prioritized data from offline and online buffers. As shown in Table [[] the learned skills generalize effectively to unseen test tasks, demonstrating SISL's robustness and efficacy. While variations in return scales across tasks can introduce bias in reward model training, our experimental environments adopt simple reward structure based on subtask completion, under which the reward model remains stable, as shown in Appendix F.4 For envi- ronments with more complex reward functions, per-task reward standardization can be considered. In skill-based settings, both the offline trajectories and the online training tasks share similar un- derlying subtasks, such as region-to-region transitions in Ant/Point environments or object-centric subtasks in Kitchen/Office domains. Because these subtasks define reward semantics consistently across training tasks and offline dataset, this relabeling process does not introduce additional in- stability. In addition, softmax-base prioritization forms distribution rather than relying single trajectory, which further mitigates the impact of any minor estimation error. We provide / trends, compare refined skills via skill trajectory visualization, taskrrepresentation improvements, and policy skill composition in Appendix F These analyses provide insights into SISL's effective- ness in optimizing skill execution and enhancing task representation.
5.5 ABLATION STUDIES
We evaluate the impact of SISL's components and key hyperparameters in Kitchen (a 0.3) and Maze2D (a 1.5), focusing on the effect the prioritization temperature T.
Component Evaluation: To evaluate the importance of SISL's components, compare the meta- test performance of SISL with all components included against the following variations: (1) Without BaR, relying solely B3 assessing the influence of offline data in skill refinement; (2) Without

---

Published as a conference paper at ICLR 2026
Test Average Return
1.0
Test Test Average Return
3.5
10
0.8
3.0
0.6
0.7
0.4
0.6
0.1K 0,3K 0.4K
0.2
0.2K 0.3K 0.4K 0.5k
(8) Kitchen(===
0.3)
(b) (b)Maze2D(o=1.5)
Iteration
Iteration
(a) (a)Kthhen(==0.33
(b) Maze2D(o=115)
Alre
Ours wio the
T=0.1
T.0.5
T-2.0
Figure Component evaluation
Figure Impact of prioritization temperature T
PT.. applying uniform sampling in Bor instead of maximum return relabeling, to test the effect of prioritization; (3) Without B using only Bon. examining the contribution of high-quality samples obtained by Timp and иhi and (4) Without Timp- removing the still-mmpooeement policy, verifying whether exploration near the skill distribution discovers improved behaviors. Fig. X presents the comparison results, demonstrating significant performance drops when either buffer is removed, highlighting their critical role in effective skill discovery. Uniform sampling in Bor also reduces performance, underlining the importance of maximum return relabeling. Lastly, excluding Tip notably degrades performance, emphasizing the critical role of discovering improved behavior.
Prioritization Temperature T:: The prioritization temperature adjusts the prioritization between online and offline buffers. Specifically, lower T' biases sampling toward high-return buffers, while higher results uniform sampling. Fig. iilustrates the performance variations with different pri- oritization temperatures T. When T 0.1,, performance degrades due excessive focus single buffer, aligning with the trends observed in the component evaluation. Conversely, high T = 2.0 also degrades performance by eliminating prioritization. These results highlight the importance of proper tuning: T 1.0 for Kitchen and T = 0.5 for Maze2D achieve the best performance. Based on the result, we T approximately proportional the high-return range of each environment, which consistently yielded the best performance while avoiding extensive tuning.
To provide clearer understanding of SISL's components, we include several additional ablations in the Appendix. First, we analyze the temperature T for all environments and noise levels in Ap- pendix G..) showing that its optimal value depends mainly on the return scale of each environment. Second, we study the KLD coefficient Xile in Appendix G.3 and find that performance is stable across a wide range of values, but drops sharply when the coefficient is zero because the guiding ef- fect of the KI. term disappears. Third, we examine the reinitialization interval Kitr Appendix G.5) confirming that the high-level policy needs minimum amount of adaptation time after each refine- ment step, while larger intervals produce similar outcomes as long as skills are updated a few times during training. Together, these results highlight the role and influence of each component in SISL.
6 LIMITATION
Although SISL demonstrates strong performance, it has several limitations. First, although SISL achieves notable performance gains, computation time per iteration increases by 16% over the baseline. This overhead mainly comes from training the skill model without freezing it during meta- training, but the performance table and ablation study confirm that this component is essential. (See Appendix
E for details) Second, SISL requires fine-tuning during the meta-test phase for optimal performance, which introduces additional computational overhcad. Addressing this through zero- shot skill adaptation could enhance its practicality, enabling transfer new tasks without retraining. Future work in this direction could significantly improve SISL's applicability in real-world scenarios.
7 CONCLUSION
In this paper. we propose SISL, a robust skill-based meta-RL framework designed to address noisy offline demonstrations in long-horizon tasks. Through self-improving skill refinement and prioriti- zation via maximum return relabeling, SISL effectively prioritizes task-relevant trajectories for skill learning and enables efficient exploration and targeted skill optimization. Experimental results high- light its robustness to noise and superior performance across various environments, demonstrating its potential for scalable meta-RI RL. in real-world applications where data quality is critical.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported partly by the Institute of Information & Communications Technology Plan- ning & Evaluation (ITTP) grant funded by the Korea govermment (MSIT) (No. RS-2022-11220469, Development of Core Technologies for Task-oriented Reinforcement Learning for Commercializa- tion of Autonomous Drones), (No. 04025-25442824, AI Star-Fellowshh Program (UNIST)), and (No. RS-2020-11201 Artificial Intelligence Graduate School Support (UNIST)), and partly by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) L.M-aasedd Multi-Agent Reinforcement Learning for End-to-Bi fad-to-EEd Large Autonomous Swarm Control).
(No.
ETHICS STATEMENT
By introducing self-improving skill refinement and skill prioritization via maximum return rela- beling. SISL improves the stability and generalizability of skill learning, allowing agents to adapt rapidly to new tasks even when data quality is imperfect. This advancement has the potential to make reinforcement learning more practical and reliable in real-world settings where collecting high-quality data is difficult expensive. While this framework may have potential societal im- plications by enhancing reinforcement learning's 's real-world applicability, we believe it is primarily foundational in nature and does not introduce any new risks of malicious use.
REPRODUCIBILITY STATEMENT
To reproduce SISL, we provide the loss function redefined as neural network parameters, along with the meta-train and meta-test algorithm tables in Appendix B For the algorithm implementation, we provide the system specifications used for experiments, the source of the bascline algorithm, envi- ronment details, the offline dataset construction method. and the hyperparameter setup in Appendix C Additionally, provide the anonymized code for SISL in the supplementary material, enabling the reproduction of the proposed algorithm and experiment results.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv:1807.10290 2018.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39:3-20, 2020.
Brandon Araki, Xiao Li, Kiran Vodrahalli, Jonathan DeCastro, Micah Fry, and Danicla Rus. The logical options framework. In International Conference Machine Learning, pp. 307-317. PMLR, 2021.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, et The option keyboard: Combining skills in reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In Twenty-fourth interna- tional joint conference on artificial intelligence, 2015.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.
Exploration by random network distillation. In International Conference Learning Representations, 2018.
11

---

Published as a conference paper at ICLR 2026
Jongseong Chae, Seungyul Han, Whiyoung Jung, Myungsik Cho, Sungho Choi, and Youngchul Sung. Robust imitation learning against variations in environment dynamics. In International Conference Machine Learning, pp. 1828-2853 PMLR, 2022.
Jen-Tzun Tzung Chien and Weiwei Lai. Variational skill embeddings for meta reinforcement learning. In 2023 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2023.
Minjae Cho and Chuangchuang Sun. Hierarchical meta-reinforcement learning via automated macro-action discovery. arXiv preprint arXiv:2412. 2024.
Kurtland Chua, Qi Lei, and Jason Lee. Provable hierarchy-basel meta-reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 10918-10967. PMLR, 2023.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RIP: Fast reinforcement learning via slow reinforcement learning. arXiv preprint drXiv:161..0277,.2 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. International Conference on Learning Representa- lions, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp- 126-1135. PMLR, 2017.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learn- ing Representations, 2018 URL ht..hhtts:/// http:ss//ppeeeeeiie net/ forum Fyd-CyWCCKKAN
Kevin Frans, Jonathan Ho, Xi Chen, Picter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations, 2018.
Haotian Fu, Hongyao Tang, Jianye Hao, Wulong Liu, and Chen Chen. Mghrl: Meta goal-generation for hierarchical reinforcement learning. In Distributed Artificial Intelligence: Second Interna- tional Conference, DAI 2020, Nanjing, China, October 24-27, 2020, Proceedings 2, pp. 29-39. Springer, 2020a.
Haotian Fu, Shangqun Yu, Saket Tiwari, Michael Littman, and George Konidaris. Meta-learning parameterized skills. In Proceedings of the 40th International Conference on Machine Learning, pp. 10461-10481, 2023.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020b.
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for ef- ficient exploration. Advances in Neural Information Processing Systems, 34:11553-11566 2021.
Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recur- rent neural networks, pp. 37-45, 2012.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 161iv:161 1.07507, 2016.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta- learning for reinforcement learning, arXiv preprint S.div:8806.0466 2018.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- long-horizon tasks via imitation and reinforcement learning. In Conference Robot Learning, pp. 1025-1037. PMLR, 2020.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. PMLR, 2018.
12

---

Published as a conference paper at ICLR 2026
Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-awar entropy regularization for
sample-eficiint exploration. In International conference on machine learning. pp. 4018-4029. PMLR. 2021a.
Seungyul Han and Youngchul Sung. A max-min entropy framework for reinforcement learning. Advances in Neural Information Processing Systems, 11-25732--27455 2021b.
Hongcai He, Anjie Zhu, Shuang Liang, Feiyu Chen, and Jie Shao. Decoupling mete-reinforremmnt learning with gaussian task contexts and skills. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 38, pp. 22358-12366 2024.
Jiaheng Hu, Zizhao Wang, Peter Stone, and Roberto Martín-Marthn Disentangled unsupervised skill discovery for efficient hierarchical reinforcement learning. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
Jaebak Hwang. Sanghyeon Lee. Jeongmo Kim, and Seungyul Han. Strict subgoal exeeu- tion: Reliable long-horizon planning in hierarchical reinforcement learning. arXiv preprint arXiv:2506. 20039, 2025.
Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Un- supervised curricula for visual meta-reinforcement lcarning. Advances in Neural Information Processing Systems, 32, 2019.
Yuankun Jiang, Nuowen Kan, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Dou- bly robust augmented transfer for menffreenforeement learning. Advances in Neural Information Processing Systems, 36:77002-77012, 2023.
Yonghyeon Jo, Sunwoo Lee. Junghyuk Yeom, and Seungyul Han. Fox: formation-awant explo- ration in multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12985-12994, 2024.
Jeongmo Kim, Yisak Park, Minung Kim, and Scungyul Han. Task-aware virtual training: Enhanc- ing generalization in meta-reinnorceement learning for out-of-distributio tasks. In Forty-second International Conference on Machine Learning, 2025.
Woojun Kim, Yongjae Shin, Jongeui Park, and Youngchul Sung. Sample-efficient and safe deep inforcement learning via reset deep ensemble agents. Advances in neural information processing 2023.
systems,
Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pp. 1-3488. PMLR, 2019.
Rui Kong, Chenyang Wu, Chen-Xiao Gao, Zongzhang Zhang, and Ming Li. Efficient and stable offline-to-online reinforcement learning via continual policy revitalization. In Proceedings of the Thirty-Thine International Joint Conference Artificial Intelligence, IJCAI-24, pp. 4317-4325, 2024.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invarian neural networks. In Interna- fional conference on machine learning, pp. 3744-3753. PMLR, 2019.
Sunwoo Lee, Jaebak Hwang, Yonghyeon Jo, and Seungyul Han. Wolfpack adversarial attack for robust multi-agent reinforcement learning. In Forty-second learning Learning, 2025.
International Conference on Machine
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In Proceedings of International Conference on Learning Representations, 2019.
Jiachen Li. Quan Vuong. Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. Advances in neural information processing systems, 33:6197-6210, 2020.
13

---

Published as a conference paper at ICLR 2026
Siyuan Li, Rui Wang, Minxue Tang, and Chongjie Zhang. Hierarchical reinforcement learning with advantage-bast auxiliary rewards. Advances in Neural Information Processing Systems, 32, 2019.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. Conference on Robot Learning, pp. 1678-1690. PMLR, 2022.
Russell Mendonca, Xinyang Geng, Chelsea Finn, and Sergey Levine. Meta-reinforcement learning robust to distributional shift via model identification and experience relabeling. arXiv preprint arXiv:2006.00 077788 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 318:529-533 2015.
Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J. Lim. Skill-based meta- reinforcement learning. International Conference on Learning Representations (ICLR), 2022.
Charles Packer, Pieter Abbeel, and Joseph E Gonzalez. Hindsight task relabelling: Experience replay for sparse reward meta-rl. Advances in neural information processing systems, 34:2466- 2021.
Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. International Conference on Learning Represen- tations, 2022.
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability unsupervised skill discovery. In International Conference on Machine Learning. pp. 27225-27245. PMLR, 2023.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine Hiql: Offline goal- conditioned rl with latent states actions. Advances in Neural Information Processing Systems, 36, 2024.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188-204, PMLR, 2021.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J Lim. Guided reinforcement learning with learned skills. In Conference on Robot Learning, pp. 729-739. PMLR, 2022.
Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. In International Conference on Machine PMLR, 2022.
Learning. pp.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinffrrcmmett learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Krishan Rana, Ming Xu, Brendan Tidd, Michael Milford, and Niko Sünderhauf. Residual skill policies: Learning adaptable skill-based action space for reinforcement learning for robotics. In Conference on Robot Learning, pp. 2094-2104. PMLR, 2023.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. Advances in neural information processing systems, 31, 2018
Arnaud Robert, Ciara Fike-Burke and Aldo A Faisal. Sample complexity of goal-conditioned hier- archical reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2026
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- tional Conference on Learning Representations, 2020.
Carolin Schmidt, Daniele Gammelli, James Harrison, Marco Pavone, and Filipe Rodrigues. Offline hierarchical reinforcement learning via inverse optimization. In The Thirteenth International Conference on Learning Representations, 2025.
Gresa Shala, André Biedenkapp. and Josif Grabocka. Hierarchical transformers are efficient meta- reinforcement learners. arXiv preprint arXiv:2402.0644 064002 2024.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference Learning Representations, 2019.
Lucy Xiaoyang Shi, Joseph Lim, and Youngwoon Lee. Skill-based model- reinforcement learning. In Conference on Robot Learning, pp. 2262-2272. PMLR, 2023.
Sangwoo Shin, Minjong Yoo, Jeongwoo Lee, and Honguk Woo. Semtra: A semantic skill translator cross-domain zero-shot policy adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15000-15008, 2024.
DJ Strouse, Kate Baumli, David Warde-Farley, Volodymyr Mnih, and Steven Stenberg Hansen. Learning more skills through optimistic exploration. In International Conference on Learning Representations, 2022.
Michael Wan, Jian Peng, and Tanmay Gangwani. Hindsight foresight relabeling for meta-
reinforcement learning. In International Conference on Learning Representations, 2021.
Te-Lin Wu, Jaedong Hwang, Jingyun Yang, Shaofan Lai, Carl Vondrick, and Joseph Lim. Learning from noisy demonstration sets via meta-learned suitability assessor. 2019.
Mengda Xu, Manuela Veloso, and Shuran Song. Aspire: Adaptive skill priors for reinforeement
learning. Advances in Neural Information Processing Systems, 35:38600-38613, 2022.
Xiangyu Yin, Sihao Wu, Jiaxu Liu, Meng Fang, Xingyu Zhao, Xiaowei Huang, and Wenjie Ruan. Representation-basee robustness in goal-conditioned reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21761-21769, 2024.
Minjong Yoo, Sangwoo Cho, and Honguk Woo. Skills regularized task decomposition for multi- task offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 37432-37444, 2022.
Xingrui Yu, Bo Han, and Ivor W Tsang. Usn: robust imitation learning method against diverse action noise. Journal of Artificial Intelligence Research, 79: 19:1237-1280 2024a.
Xuchui Yu, Mhairi Dunion, Xin Li, and Stefano V Albrecht. Skill-aware mutual information opti- misation for zero-shot generalisation in reinforcement learning. Advances in Neural Information Processing Systems, 37:1 1PP:110573-101111 2024b.
Haoqi Yuan and Zongqing Lu. Robust task representations for offline meta-reinforcement learning via contrastive learning. In International Conference on Machine Learning, pp. 25747-25759. PMLR, 2022.
Renzhe Zhou, Chen-Xiao Gao, Zongzhang Zhang, and Yang Yu. Generalizable task representation learning for offline meta-reinforcement learning with data limitations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 17132-17144 140, 2024.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep via meta- learning. In International Conference on Learning Representations, 2019.
15

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We wrote the entire manuscript ourselves, including the main text and the appendix. We used large language models only for copy editing improve spelling and readability, and we verified all suggested revisions before incorporation. LLMs were not used to generate ideas, methods, analyses, results, code, figures, citations beyond minor edits. All technical content and experiments were conceived, implemented, and validated by the authors. We manually audited every citation and numerical claim and accept full responsibility for the manuscript.
B IMPLEMENTATION DETAILS ON SISL
This section provides detailed implementation of the proposed SISL framework. As outlined in Section 4 SISI. begins with an initial skill learning phase (pre-train) to train the low-level skill policy, skill encoder, and skill prior. then progresses to the meta-train phase, where decoupled policy learning is performed using high-level policy, task encoder, and skill-improvement policy. Also, self-improvement skill learning executed via maximum return relabeling using reward model, low-level skill policy, skill encoder, and skill prior. Finally, in the meta-test phase, rapid adaptation to the target task is achieved via fine-tuning based on the trained high-level policy and task encoder. Section B.1 details the initial skill learning phase, Section B.2 elaborates on the meta- train phase, and Section expplain the meta-test phase. All loss functions in SISL are redefined in terms of the neural network parameters of its policies and models. Additionally, the overall structure for the meta-train and meta-test phases is provided in Algorithms and
B. INITIAL SKILL LEARNING PHASE
Following SPiRL (Pertsch al. 2021), introduced in Section 3 train initial skills using the offline dataset Bort The low-level skill policy Ti, skill encoder qo. and skill prior ps are parame- terized by & and trained using the following loss function (modified from Eq. (T))
(1)
(TTL, Gor Po: Po: $)
:= (sus) He deee H, J-Bat snigo-1leees (- (a
H4H1-1 1
=B (seet Reic B-B zmag (118 i Paa
log m (axx(skk)) + DKL
Dex((sst (aa(+ (9o(-1 Suat,, amctH.) ||/(O.D)
k=
+ DKI. (Lo0(1 am.4 po(-(ss))
(B.1)
where |.| represents the stop gradient operator, which prevents the KL term for skill prior learning from influencing the skill encoder. Using the pre-trained TT, Jo, and Ps. SISL refines skills during the meta-train phase to further enhance task-solving capabilities.
B.2 META-TRAIN PIASE
As described in Section SISL comprises two main processes: Decoupled Policy Learning. which explores task-relevant behavior near skill distribution using skill saillimmprovement policy, and trains the high-level policy, task encoder to effectively utilize learned skills for solving tasks; and Seff-Improvement Skill Learning, which improves skills using prioritization via maximum return relabeling. Detailed explanations for each process are provided below.
Decoupled Policy Learning
As described Section 4.2 the skill-improvement policy Timp.sh, parameterized by vh is designed to expand the skill distriuttin and discover task-relevant behaviors near trajectories stored in the prioritized on-policy buffer Bi, for each training task T'. This buffer prioritizes trajectories that best solve the tasks. Additionally, the state-action value function Qimp. ss also parameterized by th is defined to train the ialllimmpoveement policy using soft actor-critic (SAC). To enhance exploration, both extrinsic reward rf and intrinsic reward rin, employed. The intrinsic reward, based
16

---

Published as a conference paper at ICLR 2026
random network distillation (RND), is computed the L2 loss between randomly initialized target network f and prediction network fip parameterized by yi and f), respectively, and is expressed as:
1imt, -
(B.2)
where i is the task index, and fo is updated to minimize this loss. A dropout layer is applied to fa to prevent over-sensitivity to state The RL loss functions of SAC for training the skill-improvement policy Timp,o and the state-action value function Qimp.o using the intrinsic reward Fint,t are defined as follows:
Liip (v)
2
E te )- aapt oo (a..))
Qmepy(ss (Bwrtt Suatins Timyl (Qimp, (8141, (1+1.
Ammp log Simp (Re+ (ess.
Cimp (1h) = Σ
T aoiit 18! u
Ammp log Timpro (pelse)))) Qumpuolss
Ammp Σ B
log Fimp.o (nelse.))
(B.3)
Here, dext and Oint are extrinsic and intrinsic reward ratios, Timp is the discount factor, Ximp is the exploration entropy coefficient adjusted automatically by SAC, and Ahnn is the KLD coefficient. Also, note that Eq. (B.3) provides a parameterized and detailed reformulation of Eq. (3) from
Section 4.2 explicitly incorporating parameterization and loss scaling details.
To mutually update skill selection based on the refined skills, the updated and fixed low-level skill policy # and skill prior 4a utilized to train the high-leve policy following the SiMPL frame- work introduced in Section 3 The objective is to select skill representations = that maximize task returns while ensuring the high-level policy remains close the skill prior for stable and efficient learning. The high-level policy Th.0 and value function Qh.e are parameterized by 0 and trained using the soft actor-critic (SAC) framework, with the RL loss functions defined as:
Ly (0) = E (enaeet aot - og 2 Qn 2e, e,,)) (r² + at -iee)
C2 = B 288 se ma ol Ahaa DKa, (Tnollss Bollss)) - Qno(sn)
zemaana al
(B.4)
where 4e,0 is the parameterized task encoder with parameter 0, 7/h is the high-level discount factor, and A/. is the high-level KLD coefficient. The term ri 1 rkk represents the cumulative rewards, with states and rewards obtained by executing the low-level skill policy गं. using 2 ~
(-1a() over H, timesteps. The context cT (Sk, Rk: rh 1++H Npow where Nprior is the number of context transitions, denotes the high-level transition set of task T. This context is used select the task representation eT from the task encoder ge.0. Also, note that Eq. (B.4)
parameterized modification of Eq. (2) from Section
Self-Improvement Skill Learning
To extract better trajectories and learn skills that effectively solve tasks, the online buffer Bi selec- tively stores high-return trajectories collected during the trainin phase through the execution of the low-level policy Ti. and the skill-improvement policy Timp.y- A trajectory r' is added to Bi, if its return G(r*) exceeds the minimum return in the buffer, min,/B5 G(r'). To refine skills,
maximum return relabeling is applied using the parameterized reward model RG with parameter Ç. The reward model is trained by minimizing the following MSE loss:
[ (Re(sh -ri))
Lreward (C) = E (sshat))) uB
(B.5)
17

---

Published as a conference paper at ICLR 2026
This assigns priorities to offline trajectories 7 Bonr (Eq. b updated for NNpiority samples per iteration.
For skill learning, the low-level skill policy Mi,d, skill encoder 4o, and skill prior Pa optimized using the following loss function. This incorporates both high-return trajectories from the online buffer 8 and trajectories from the offline buffer Ban, weighted by their importance:
Coil(o) (1 BB) (suet st, aeett 11, Pret soogo erooolleesee (-Res4 0.44HH)
((TL, 1o, Po.
(B.6)
B
ΣB (eea Ha oua n. me zmga se14 (aneetH))
[( 9o. Po.
NT. train
where is the mixing coefficient defined in Eq. 77 and C(Ti, qd. Pa. 2) the skill learning objective defined in Eq. (B.1) for optimizing Ti Ta. and Pe- During training, update the low- level policy Ti skill encoder Tø. and skill prior Pa used for the skill-based meta-RL. every Kaer iterations. Specifically, the updates are performed as follows: क < T1 + 9á. and Pe t Ps-
After the meta-train phase is completed, the final meta-train phase parameter stored as Orinal <- and is subsequently used during the meta-test phase.
B.3 META-TEST PIIASE
After completing the meta-train phase of SISL, the meta-test phase performed on the test task set Must. In this phase, previously learned components, including the low-level skill policy Ti. skill prior Pa. and task encoder Ge.0ma are kept fixed and are no longer updated. Only the high-level policy Th and high-level value function Qh.e are trained for each test task using the soft actor-critic (SAC) framework.
During meta-testing, for each test task T, the task representation e is inferred from the fixed task encoder 9e, The SAC algorithm is then applied to optimize the high-level policy and value function for the specific test task, following the same loss functions as defined Eq. (B.4) from the meta-training phase. This approach ensures efficient adaptation to unseen tasks by leveraging the fixed, pre-trained low-level skills and task representations.
18

---

Published as a conference paper at ICLR 2026
Algorithm 1: SISL: Meta-Train Phase
Require: Training tasks Musin. offline dataset Ber, low-level policy mi,d, skill encoder 9p. and skill prior Pa-
Initialize: High-level policy 7,,0, skill-improvement policy Timp.ch, task encoder fe,0, reward model R2, and value functions hi.0. Qimp.v-
(Initial Skill Learning)
Update 11, ạ6- Pe using Eq.
B.J with @ c X VoL
Fix #t. f- T. 9o <- 9s. and ps
Ps-
for iteration = 1, 2,
for task i to NT do
Collect high-level trajectories TH and low-level trajectories of using The with o1,6 2c.0-
Collect kkll-immpovemmett trajectories Timp using Timp.47-
Filter high-return trajectories Thigh from Ti and mm s.t. G min G(r"). Store T "mmp' and Tiih into Bi Bmpp' and Bi o0'
Compute prioritization factors: PBar and 3.
for gradient step do (Decoupled Policy Learning)
10
Update Timp.e. Qipp.v using Eq. B.3
with do + 1/2 AIr
imp
V V (Carip
(1/b + L (v:)).
Update Th.0. Qr.0. Qc.0 using Eq. (B.4
with 0 0 N/. Vo (Caritio
L (0))
(Self-Improvement Skill Learning)
Update reward model RC using Eq.
B.5
with & 5- >
V V ((Leward
Update Ti. 4d- Pá using Eq. (B.6) with d Φ A/. V (oLssin()))
14
if h: mod Kiter = 0 then
15
Update Ni, + T1 Ts < qos and Ps + Po. Reinitialize Th,a-
Save the final meta-train phase parameter Onnal 0.
Algorithm 2: SISL: Meta-test phase
Require: Target task T, high-level policy Th,0. value function Qi,0, task encoder Oe.0 low-level policy Fi and skill prior Pa-
Collect context cT using Th,9 with TT. I).
Compute task representation eT -e,0nal (.e))
for iteration & 1, 2.. do
Collect high-level trajectory Ti using The with oo eT.
Store 7/. into BTT
for gradient step do Update hh Qr using Eq. B.4 with <- - A/. (o + +x)tto(())
19

---

Published as a conference paper at ICLR 2026
C DETAILED EXPERIMENTAL SETUP
In this section, we provide detailed description of our experimental setup. The implementation is built on PyTorch with CUDA 11.7, running on AMD EPYC 7313 CPU with an NVIDIA GeForce RTX 3090 GPU. SISL is implemented based on the official open-source code of SiMPL, available at SiMPL's code for the Kitchen and Maze2D environments, SkiLD's open-source code for the Of- fice environment at https:/// https://gituub.com. com/cl wtt.c/ooo///cciiiill.... and D4RL's open-source code for AntMaze at https:/ggithu..com wwt.c//oo//aaaaaee Coainaa--ounnttiontoom wwwpindaattoo/DL/ttreeee/eee /eetttre/mmatttrr
com/naassan96/SiMP,, For the environment implementations,
used
The hyperparameters for low-level policy training were referenced from SPiRL (Pertsch et al. 2021 Additional details about the baseline algorithms are provided Section C.1 while Section C.2 elaborates the environments used for evaluation. Section C.3 expllinns the construction of offline datasets for varying noise levels, and Section C.4] the network architectures and hyperparam- eter configurations for policies, value functions, and other models.
C.I
OTHER BASELINES
Here are the detailed descriptions and implementation details of the algorithms used for performance comparison:
SAC
SAC (Haarnoja al. 2018 is reinforcement learning algorithm that incorporates entropy to improve exploration. Instead of a standard value function, SAC uses a soft value function that combines entropy, with the entropy coefficient adjusted automatically tropy. To enhance value function estimation, SAC employs double Q learning, using two inde- pendent value functions. SAC learns tasks from scratch without utilizing meta-train tasks or of- fline datasets. For the Kitchen and Office environments, the discount factor is set to 0.95, while y 0.99 is used for Maze2D and AntMaze environments. We utilize the open-source code of SAC at
maintain the target
SAC+RND
SAC+RND combines SAC with random network distillation (RND) (Burda et al., 2018), an intrin- sic motivation technique, enhance exploration. Like SAC, it learns tasks from scratch without meta-train tasks or offline datasets. RL hyperparameters are shared with SAC, and RND-specife hyperparameters are set to match those SISL. Additionally, for fair comparison, the ratio of extrinsic to intrinsic rewards is aligned with SISL. We utilize the open-source code of RND at network-distillaaie distillation
https:/gittub.cco/ppps////aaattt
PEARL
PEARL (Rakelly et al., 2019) is a context-based meta-RL algorithm that leverages a task encoder le to derive task representations, which are then used to train meta-policy. PEARL adapts its learned policy quickly to unseen tasks without utilizing skills or offline datasets. Unlike the original PEARL, which does not fine-tune during the meta-test phase, modified it include fine-tuning on target tasks for a fair comparison. We utilize the open-source code of PEARL at https//github.coo/atttaalllllllt
PEARL+RND
PEARL+RND extends PEARL by incorporating RND to enhance exploration. Like SAC+RND, the ratio of extrinsic to intrinsic rewards is set to match SISL for fair comparison.
SPiRL
SPiRL (Pertsch et al. 2021 is a skill-based RL algorithm that first learns fixed low-level policy from an offline dataset and then trains high-level policy for specific tasks. SPiRL's loss function is detailed in Section 3 and for fair comparison, loss scaling is aligned with SISL. We utilize the open-source code of SPiRI. at httpps//gihuub.om////aa///////
SiMPL
skill-based meta-RL algorithm that uses both offline datasets and meta-train tasks. While it shares SISL's approach of extracting reusable skills and performing meta- train and meta-test phases, SiMPL fixes the skill model without further updates during treinngai SiMPL's loss function is also detailed in Section and SiMPL's implementation uses the same hyperparameters as SISL to ensure a fair comparison. We utilize the open-source code of SiMPL at https///gthub.co//mmmm666666
SiMPL (Nam et al. 2022)
20

---

Published as a conference paper at ICLR 2026
C.2 ENVIRONMENTAL DETAILS
Kitchen
Figure C.I: Kitchen: An example of task (microwave-kketle-bbttte burner-ssiie cabinet)
The Franka Kitchen environment is robotic manipulation setup based the 7-DoF Franka robot. introduced by Gupta et al.] (2020) and later adapted by Nam et al.] (2022) to exclude task in- formation from the observation space, making it more suitable for meta-learning. The environment features seven manipulatable objects: bottom burner, top burner, light switch, slide cabinet, hinge cabinet, microwave, and kettle.
Each subtask involves manipulating one object to its target state, while a full task requires sequen- tially completing four subtasks. The agent earns reward of for each completed subtask, with a maximum score of achievable within a 280-timestep horizon. For instance, an example task, illustrated in Fig. C.T requires the agent to complete the sequence: microwave - kettle - bottom burner - slide cabinet. The observation space is a 60-dimensional continuous vector representing object positions and robot state information, while action space a 9-dimensional continuous vector. Based the task setup from Nam al. (2022). expanded the meta-train task set by adding two additional tasks, resulting in total of 25 meta-train tasks and 10 meta-test tasks. De- tailed task configurations provided in Table C.I
Table C.I: List of meta-train tasks and meta-test tasks in Kitchen environment
Meta-tran train task
Meta-tel test-tsst task
Submokk
Subcrske
Subies14
microwave
kettle
bottom hurner
siide cabinet
micnoave
bottom burner
light switch
top hamer
bottom burner
top burner
slide cabinet
botlom burner
burner
light switch
light
light
cabinut
ketile
bottom burner
light switch
hinge cabinet
microwave
kettle
top burner
hingo cabinet
microwave
buttom burner
hinge cabinet
top bumner
kettle
buttom burner
slide cabinet
top bumner
kotile
burner
light switch
slide cabinel
ketlle
light switch
slide cabinet
hinge cabinet
burner
burner
ketile
light switch
slide cabinet
botlum burner
microwave
buttom burner
slide cabinet
hinge cabinet
microwave
kettle
bettom burner
bumer
boctom bumner
top burner
slide cabinet
hinge cabinet
kettle
slide cabinet
hinge cabinet
micrewave
kettle
bottom bumer
hings cabinet
top bumer
ketile
bottom burner
light switch
top bumer
13
microwave
kettle
top burnee
light switch
kettle
switch
cabinet
microwave
light
15
kertle
battom burner
top burner
light switch
17
micromave
light switch
slide cabinet
hinge cabinet
burner
cabinet
hinge
bottom humer
top hurner
slide cabinet
light switch
microsave
kettle
light switch
siide cabinet
kottle
battom
cabinet
botiom humer
top burner
light switch
slide cabinet
24
top hamer
hingo cabinet
microwave
slide cabinet
botcom humer
hingy cubinet
light switch
keitle
21

---

Published as a conference paper at ICLR 2026
Office
Figure C.2: Office: An example task ((shed2, drawer)-><craerr) container) tray))
The Office environment
robotic manipulation setup featuring a 5-DoF robotic arm. Originally proposed by Pertsch et al. (2022). it has been modified to accommodate meta-learning tasks. The en- vironment simulates an offie cleaning scenario with seven objects (eraserl, shedl, pepsil, gatorade, eraser2, shed2, pepsi2) and three organizers (tray, container, drawer).
The goal is move objects to their designated organizers, with each object-to-organizer transfer constituting subtask. A full task involves completing three sequential subtasks, where subtask is defined as an (object, organizer) pair, For tasks involving a tray or container, the agent earns a reward of I for both picking and placing the object. For tasks involving the drawer, the agent receives reward point for each of the following actions: opening the drawer, picking, placing, and closing the drawer. This scoring setup allows for maximum
of 8 within 300-timestep horizon.
An example task, depicted Fig. C.2 requires the agent to sequentially complete: (shed2 - drawer), (eraserl container), and (pepsi2 - tray). The observation space is 76-dimensional continuous vector, including object positions and robot state information, while the action space an 8-dimensional continuous vector. The meta-train and meta-test include 25 and 10 tasks, respectively, similar to the configuration in the Kitchen environment. A detailed task list is provided in Table C
Table C.2: List of meta-train tasks and meta-test tasks in Office environment
Mcta-train lask
Mcta-Test task
Subtaski
Subcask2
Salcaska
Subtsski
Subrssk2
Subaakk
(sheel, drawer)
teraseri, contanner)
(pepai,, tray)
Igotorade; drawer)
(eraserl, tray) (erwser2, containet)
pppsi2, contaner)
(shod2, container)
(enaserl, drawer)
(pepsil, tnay)
2
(eraser), drawer) (eraser2, drawer) (sheel, drawer) cuntainer)
(pepal, tny)
(eraserl, tmay)
(shed2, drawer)
(gatoade, contniner)
3
(pepsil, tray)
(pitordde container)
(pepsil, tray)
container)
(eraser2. drawer)
(pepsi2, tray)
(pepsil, container)
Imay)
drawer)
(pepsi2. container)
(gatordd,, tray)
(craser
(pepsi), container)
(shel), Iny)
(craser2, drawer)
(gatorade, container) (gatorade, tray) (popsi2. drawer)
(craser2. drawer)
(pepsi2, mny)
(gatorade, tray)
(eraser2, container)
(craserl, drawer)
7
(shed), container)
(craserl, drawer)
(pepsi2, cuntainet)
(shed2, drawer)
(crsserl,
(sbed), tray)
(popsil, container)
(shod2, (emser)), contniner)
(gatorade, container)
(pepsi2. Ima)
(popsit, trag)
(popsi2, container)
(shed2. drawer)
(pepsi2, drswer)
(shed), tnay
(gatorade, drower)
(pepsi), container)
(emser2, tray)
11
feraser2, drawer)
(shed2, Iray)
(popsi2, cuntainer)
12
(pepsiZ, contuiner)
(shed2, drawer)
(shed), tray)
(shed2, tray)
(popsil,,
(eraserl, drawer)
(gatorale, tras)
(craserl, drawer)
(popsil, container)
15
(eraverl, tray)
(shed1, drawer)
(gatorade, contriner)
16
(craser2, drawer)
(gatorade, contniner)
(shed2, tray)
(pepsi2, drawer)
(shell,
(pepsi), contuiner)
(pepeiZ, tray)
(craserl, drawer)
19
(shed2, tray)
(gotorads, drawer)
(shed1. container)
(goiorade, tray)
(popsil, comainer)
(xepsi2, drawer)
tray)
(shed2, drawer)
(pepsi2,
(enaserl, may
(gotordd, drawer)
(shef2, container)
23
(pepsil, container)
(shed2, drawer)
(enser2,, tray)
24
(gttorade, drower)
(shed), tay)
(popsi2, container)
container)
(pepsil, drawer)
(enaser tray)
22

---

Published as a conference paper at ICLR 2026
Maze2D
Start ng Paint
Trin Thsk
Test Tisk
(a) Simulation
(b) Training and test tasks
Figure C.3: Maze2D: Visualization of simulation and meta-train/test tasks Maze2D
The Maze2D environment is navigation setup where 2-DoF ball agent moves toward goal point. Initially introduced by FFu al. (2020b) and later adapted by Nam et al. (2022) for meta-learning tasks, the environment is defined on 20x20 grid. The agent receives reward of upon reaching the goal point within horizon of 2000 timesteps.
Fig. C.3 (a) provides visualization of the Maze2D environment, while Fig. C.3 (b) illustrates the meta-train and meta-test tasks. In Fig. C.3 (b), green squares indicate the goal points for meta- train tasks, and red squares represent the goal points for 10 meta-test tasks. All tasks share the same starting point at (10, 10), marked by a blue cross. The observation space 4-dimensional con- tinuous vector containing the ball's 's position and velocity, while the action space is 2-dimensional continuous vector.
AntMaze
ng Paint
Simulation
(b) Training and test tasks
Figure C.4: AntMaze: Visualization of simulation and meta-train/test tasks in AntMaze
The AntMaze environment combines navigation and locomotion, replacing the 2-DoF ball from the Maze2D environment with a more complex 8-DoF quadruped Ant robot. Initially proposed by Fu al. (2020b) and later adapted meta-learning setups, the environment defined on 10x grid. The agent receives a reward of upon reaching the goal point within horizon of 1000 timesteps.
Fig. C.4 (a) shows simulation image of the AntMaze environment, and Fig. C.4 (b) depicts the meta-train and meta-test tasks. In Fig. C.4 (b), green squares mark the goal points for 20 meta-train tasks, while red squares denote the goal points for 10 meta-test tasks. All tasks share
common starting point at (5, 5), indicated by blue cross. The observation space a 29-dimensional continuous vector that includes the Ant's state and its (x, y) coordinates, while the action space is an 8-dimensional continuous vector.
23

---

Published as a conference paper at ICLR 2026
C.3 CONSTRUCTION OFFLINE DATASET
In this section, we detail the offline datasets used in our experiments. For the Office, Maze2D, and AntMaze environments, employ rule-based oracle controllers provided by each environment. The Office oracle controller is available at https//ggthub.com/lwaaaaaa while the Maze2D and AntMaze oracle controllers can be found in https://gihubb.cmm ctttluub.co///aaaaaaa For the Kitchen environment, which only provides human demon- strations, we train a policy using behavior cloning to serve as the oracle controller.
For the Kitchen environment, 1M transitions are collected using 25 tasks that are not part of the training or test task sets Musin Mess Similarly, the Office environment collects 1M transitions using 80 tasks. The Maze2D and AntMaze environments follow the same approach, collecting 0.5M transitions using 40 and tasks respectively, with randomly sampled initial and goal points. Unlike SiMPL, which randomly samples initial and goal points for each trajectory in the Maze2D environment, we limit our data collection to 40 distinct tasks, resulting in trajectories that do not fully cover the map. To introduce noise in the demonstrations, Gaussian noise with various standard deviations is added to the oracle controller's actions. For the Kitchen and Office environments, noise levels of a 0.1, 0.2, and 0.3 are used, while for Maze2D and AntMaze, 0.5, 1.0, and 1.5 are applied.
24

---

Published as a conference paper at ICLR 2026
C.4 HYPERPARAMETER SETUP
In this section, we outline the hyperparameter setup for the proposed SISL framework. For high- level policy training, we adopt the hyperparameters from SiMPL for the Kitchen and Maze2D envi- ronments. For the Office and AntMaze environments, we conduct hyperparameter sweeps using the Kitchen and Maze2D configurations as baselines.
To ensure fair comparison, we inherit from SiMPL all hyperparameters that are shared with SISL, given its multiple loss functions, and we perform parameter sweeps only over SISL specific com- ponents, namely self relf-inmroving skill refinement and skill prioritization via maximum return rela- beling. We explore prioritization temperature values T [0.1, 0.5. 1.0, 2.0] and KLD coefficients Amp E [0, 0.001, 0.002, 0.005) for skill exploration, selecting the best-performing configuration as defaults. Additionally, the ratio of intrinsic to extrinsic rewards is fixed at levels that show optimal performance in single-task SAC experiments.
For implementing the skill models (Tt 9, P). we follow SPiRL by utilizing LSTM (Graves & Graves 2012) for the skill encoder and MLP structures for the low-level skill policy and skill prior. For im- plementing the high-level models (Th, ge), we follow SiMPL by utilizing Set Transformer (Lee) jtt al., 2019) for the task encoder and MLP structures for the high-level policy and value function. Additionally, for implementing the SISL, we utilize MLP structures for Timp Qimp= and R. The de- tailed hidden network sizes are presented in Table C.3 and Table C.4 Table C.3 presents the network architectures (the number of nodes in fully connected layers) and the hyperparameters shared across all environments, while Table details the environment-speee -specfff hyperparameter setups.
Table C.3: Network Architecture and Shared Hyperparameters
Group
Name
Environments
Kitchen
Offic
Maze2D
AntMace
Discount Factor
0.99
High-level
Learning rate X/
0.0003
Network size uA. Qa
[128]x6
[256]x4 x
Buffer size Bi
TDK
KLD coefficient XM
0,0005
Skill longth H.
10
Skill dimension dim(a)
10
# of priority update trajectory Npsiiit
200
Shared Hyperparancters
Low-level
Learning rate Xim
0.001
Lerrning rate Agread
Network
[|28|
Network size
[128]
Network size
[128]x7
Network size
LSTM[
RND state drapout ratio
0.7
output dimension
Skill-Improvettent
Learning Xap
0,0003
Network size Sivp: Qiey
[256] x4
Network size f.f
[128]
Table C.4: Environmental Hyperparameters
Environments
Group
Name
Kitchen
Office
Maze2D
AntMuze
B)
cocfficient Aldd
0.001
leve-
Task lutent dimension dim(()
Batch size (RI., per task)
256
256
1024
512
Batch size (context, per task)
1024
1024
8192
4096
Skill refinement Rive
2000
2000
1000
2000
Low-level
Prioritization temperature
1.0
1.0
Environmental
Buffer size Exp
200K
Hyperommeetess
Discount factor "ue
0.95
RND extrinsic ratio des
5
10
10
RND intriusic ratio dur
0.1
0.1
0.01
0.01
Skill Improveeet
Entropy coefficient X
0.2
0.2
0.1
0.1
0.005 (Expert)
0.005
KLD coofficient ARM
0.001
0.001
0.001
0.002
0.001 (or 0.5)
25

---

Published as a conference paper at ICLR 2026
D ADDITIONAL COMPARISON RESULTS
In this section, we provide additional comparison results against baseline algorithms. Following Fig. 6 additional performance comparisons across all environments and noise levels are presented in Section D.I limited offline dataset size in Section D.2. random noise injection in Section D.3 and diverse sub-optimal offline datasets in Section D.4
D.1 PERFORMANCE COMPARISON
Fig. D.1 presents the learning curves of average returns for the algorithms on test tasks, corre- sponding to the experiments summarized in Table [ Rows represent evaluation environments, and columns denote noise levels. SISL consistently demonstrated superior robustness, outper- forming all baselines across various environments and noise levels. At higher noise levels such as Noise(o 0.2). Noise(a 0.3) for Kitchen and Office, and Noise(a 1.0). Noise(o 1.5) for Maze2D and AntMaze, significant performance improvements highlight the effectiveness of skill refinement in addressing noisy demonstrations. Even with high-quality offline datasets like Expert and Noise(o 0.1) for Kitchen and Office, and Noise(o 0.5) for Maze2D and AntMaze, SISL further improved performance by learning task-relevant skills. These learning curves align with the trends observed in the main experiments, confirming that skill refinement enhances performance across various dataset qualities and effectively adapts to the task distribution.
Interestingly, SPiRL and SiMPL sometimes perform better with mild Gaussian noise than with pert data, especially in Maze2D and AntMaze. Our analysis suggests that mild noise increases the diversity of behaviors in the dataset without significantly reducing success rates, allowing agents reach goals via more diverse paths. This expands state-action coverage (by about 7.3% in Maze2I) and 5.2% in AntMaze), helping skill-based methods learn more flexible and reusable skills. How- ever, as the noise level increases further, a significant portion of trajectories fail to reach the goal, leading to low quality skill learning and degraded downstream performance.
Expert
Noise(a=0.1)
Naise(a=0.2)
Noise(o=0.3)
Kitchen Test Average Return
0
0
100 200 300 400
500
100 200
300 400
500
100
200
300
400
500
100 200
300
400 500
Office Test Average Return
1.5
1.5
1.0
1.0
100 700 300 400 5o0 Iteration
%
100 200 300 400
8.0
100
200
300
400
508
100
200 300
400
500
Iteration
Iteration
Iteration
Expert
Nolse(a=0.5)
Noise(a=1.0)
Noise(o=1.5)
Test Average Return
0.8
0.0
Maze2D
0.6
0.2
0.2
0.0
100
200
300
0.0
100
200
300
400
500
0.0
100
200
300
400
500
0.0
100
200
300
400
500
Average Return
0.8
0.6
AntMaze
0.6
0.3
0.4
0.4
0.2
0.1
0.0
100
200 300 400 508
100 200
300 400 508
100
200 300 Iteration
400
508
100
200 300 400 500 Iteration
Iteration
Iteration
Figure D.I: Learning curves across considered environments and noise levels
26

---

Published as a conference paper at ICLR 2026
D.2 LIMITED OFFLINE DATASET SIZE
To further investigate the robustness of SISL, we conducted additional experiments to assess the im- pact of dataset size on skill learning and downstream performance. While SISL primarily addresses the challenge of refining corrupted skills from noisy demonstrations through online interaction, the size of the offline dataset remains an important factor, especially in practical scenarios. To assess the impact of dataset size, we conducted additional experiments on the Kitchen environment using the same expert dataset but reduced to 50% (0.5M). 25% (0.25M), and 10% (0.1M) of its origi- nal dataset size (IM). As shown in Table D.I SISL consistently outperforms baselines even under limited expert data, and its performance degrades more gracefully compared to baselines. These results confirm SISL's ability refine skills even from limited data, which is particularly valuable in roaldworld scenarios where collecting high-quality demonstrations is costly or infeasible.
Table D.I: Final performance on the Kitchen environment with varying sizes of expert datasets.
Dataset Size
SPiRL
SiMPL
SISL
Expert(100%)
3.11
3.4010.18
1.909
EXpert(50%)
3.11
3.29
3. 3.65++0.6
Expert(25%)
2.91
2.99
3.60++.14
Expert(10%)
2.37
2.56+0.09
3.28:t0.15
D.3 RANDOM NOISE INJECTION
While Gaussian noise widely used approach for degrading demonstration quality in offline RL studies, it does not fully capture the diverse and unstructured nature of real-world noise such as sensor failures, occlusions, or actuator malfunctions. In our main experiments, we adopted multi- level Gaussian noise for two reasons: (1) it is standard and accepted method for systematically degrading demonstration quality, and (2) it enables controlled analysis of robustness under varying degrees of skill degradation. Notably, in domains such as Kitchen and Office, sufficiently high levels of Gaussian noise render learned skills nearly unusable, mimicking weal-world failure scenarios in precision control tasks and providing highly challenging regime for evaluating robustness.
To further assess the generality of our robustness claims and address concerns about the limitations of Gaussian noise, we conducted additional experiment using random action injection in the Kitchen environment. This approach better simulates real-world anomalies such as actuator faults each timestep. the oracle action replaced with randomly sampled action with a probability of 25%, 50%, or 100% (resulting in a uniformly random dataset the highest level). This method introduces severe, unstructured corruption into the offline dataset, representing worst-case real-world failures beyond the smooth degradations induced by Gaussian noise. As shown in Table D.2] SISL consistently outperformed baseline methods, confirming its robustness even under extreme, non-gaussian corruption. These results demonstrate SISL's ability to refine useful behaviors and maintain strong performance in the presence of severe dataset corruption, further validating the generality of our robustness claims.
sensor failures. Specifically,
Table D.2: Final performance on the Kitchen environment with the random noise injection.
Noise Type
SPiRL
SiMPL
SISL
Expert
3.1 +0.33
3.40
3.97=1.09
Injection(255
0.80+0.15
0.77+0.12
3.42+0.11
Injection(509g
1.14
0.04
3.26
Injection( 100%)
0.07
0.02-t0.05
1.68
27

---

Published as a conference paper at ICLR 2026
D.4 DIVERSE
DATASET
Beyond injecting noise into expert demonstrations, we also considered datasets of diverse qual- ity sub-optimal demonstrations. Our decision focus on noisy expert trajectories stems from the specific challenge we aim to address. Unlike offline-RL, which typically assumes access to reward- labeled trajectories and aims improve performance from sub-optimal data, our setup considers reward-free offfine data where noise degrades originally ncar-optimal demonstrations. This setting better reflects our goal of studying how exploration can help enhance skill learning when clean supervision is unavailable. To test whether robustness holds across different data qualities, we con- ducted additional experiments in the Kitchen domain using three dataset types: expert (return = 4), medium (return s 2), and random (collected from uniformly random policy). As summarized in Table D.3] SISL consistently outperforms other methods across all dataset types, further validating its robustness to data sub-optimality
Table D.3: Final performance the Kitchen environment with diverse quality of offline datasets.
Dataset
SPiRL
SiMPL
SISL
Expert
3.11 20.33
3.40=0.18
3.97
Medium 2.62
3.17=0.26
3.77
Random
0.07
0.02
1.68
28

---

Published as a conference paper at ICLR 2026
E COMPUTATIONAL COMPLEXITY
Table E.1 summarizes SISL's computational complexity, showing that the meta-train time increases by about 16% relative SiMPL, which we view modest overhead given the performance gains. For fair comparison, we keep the amount of training and interaction identical to SiMPL as de- scribed in Section 5.1 so the number of samples and policy updates does not exceed that of SiMPL.
Table E.2 summarizze the complexity of SISL ablations. "SISL w/o Pa... and "SISL w/o Bett" do perform reward model training and only add skill refinement compared SiMPL, increasing computation time by about 10%. In contrast, "SISL. w/o Timp", "SISL w/o Ban". and full SISL addi- tionally perform reward model training, incurring an additional increase of about 3% in computation time. In particular, "SISL w/o Timp" differs from full SISL by only 2.3% in training time, indicating that introducing Timp does not lead to a substantial increase computation cost.
Thus, the 16% overhead mainly arises from skill refinement and reward model training rather than additional policy learning. This overhead modest light of the performance gains: as shown in Fig. 6 SiMPL does not improve under high-noise settings even with extended training, whereas SISL continues to improve, justifying this computational overhead. Importantly considering that the main goal of meta-RL is adaptation unseen tasks, SISL does not require additional computational cost during the fine-tuning phase and maintains similar computation time as SiMPL in the meta-test phase while still achieving superior performance compared baseline algorithms.
Table E. Comparison of total training time for SiMPL and SISL in Kitchen(Expert).
Algorithm
Training time (h)
A time vs. SiMPL (%)
SiMPL SISL
42, 16
48.96
+ +16.1
Table E.2: Comparison of total training time for SISL ablation variants in Kitchen(Expert).
Algorithm
Training time (h)
A time vs. SiMPL (%)
SISL w/o Ban
46.23
+9.6
SISL w/o Paur
46.71
+10.7
SISL w/o Timp
47.98
+13.8
SISL w/o Bon
48.81
+15.7
29

---

Published as a conference paper at ICLR 2026
F FURTHER ANALYSIS RESULTS FOR THE SISL FRAMEWORK
In this section, we provide a more detailed analysis and visualization results of the skill refinement process in the SISL. It includes the evolution of the mixing coefficient 3 in Section visualization of the refined skills and the corresponding skill sequence visualization Section F.2] task represen- tation in Section
F3 learning stability of the reward model during the meta-train phase in Section F.4 and comparison of skill trajectory with baseline algorithms in Section E.5
F.1 EVOLUTION OF THE MIXING CORFFICIENT B
Fig. F.1 illustrates the evolution of the mixing coefficient during the meta-train phase across all environments and noise levels. Initially, low 3 values reflect reliance on offline datasets for skill learning particularly
high-return environments like Kitchen and Office, where training starts with 5 values close to zero. This approach prevents performance degradation by avoiding early dependence on lower-quality online samples, while ensuring gradual and stable changes in the skill distribution. As training progresses, the quality of online samples improves, leading to a gradual increase in B, which leverages online data for skill refinement. For offline datasets with higher noise levels, converges to higher values. Consequently, SISL learns increasingly effective skills as training proceeds, achieving superior performance on unseen tasks, underscoring the importance of SISL's ability dynamically balance the use of offline and online data based dataset quality.
Mixing C
1.0
1.0
1.0
1.0
0.0
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
2K
SK
7K
10K
2k
5K
7K
10K
1K
2K
3K
4K
2K
3K
Iteration
Iteration
Iteration
Iteration (d) AntMaze
(a) Kitchen
(b)Office
(c) Maze2D
Expert
Noisele- 0.21
Expert
Noiselg 1.0)
Noisela= 0.11
Noise(c=0.3)
Noiselo= 0.5)
Noisefa 15:
Figure F.1: The evolution of the mixing coefficient during the meta-train phase
F.2 ADDITIONAL VISUALIZATIONS OF REFINED SKILLS
Here, we present additional visualization results for SISL the Kitchen and Maze2D environments. Fig. F22 illustrates the results in the Kitchen environment (a = 0.3) after training is completed. On the left, the t-SNE visualization shows the skill representation 2 ~ Th,a, while the right side high- lights the distribution of skills corresponding to each subtask in the t-SNE map and how these skills solve subtasks over time in the Kitchen environment. In the t-SNE map, clusters of markers with the same shape but different colors indicate that identical subtasks share skills across different tasks. From the results, is evident that the skills learned using the proposed SISL framework are well- structured, with representations accurately divided according to subtasks. This enables the high-level policy to select appropriate skills for each subtask, effectively solving the tasks. Furthermore, while SiMPL trained on noisy data often succeeds in only one or two subtasks, SISL progressively refines skills even in noisy environments, successfully solving most given subtasks.
Fig. F3 illustrates how the high-level policy utilizes refined skills obtained at different meta-train iterations (0.5K, 2K, and 4K) during the meta-test phase to solve task in Maze2D (a = 1.5). When using skills trained solely on the offline dataset (meta-train iteration 0.5K), the agent failed to perform adequate exploration at meta-test iteration OK. Even at meta-test iteration 0.5K, the noise within the skills hindered the agent's ' ability to converge the target task. In contrast, after refining the skills at meta-train iteration 2K, the agent successfully explored most of the maze during explo- ration, except for certain tasks the upper-left region, and achieved all meta-test tasks by iteration 0.5K. Finally, using skills refined at meta-train iteration 4K, the agent not only explored almost the entire maze at meta-test iteration OK but also completed all meta-test tasks by iteration 0.5K. Additionally, trajectories generated with refined skills showed significantly reduced deviations and shorter paths compared to those using noisy skills. Overall, the results in Fig. F3 highlight the importance of SISL's skill refinement process in ensuring robust and efficient performance.
30

---

Published as a conference paper at ICLR 2026
Hinge Cabliet
Skill Representation
Slide Cablnet
Bottom Burner
Rothom Burner
Cantnet Hirge
Miroowo
Micoowav
Switch
Figure F.2: t-SNE visualization of skill representations (left) and refined skill trajectories for various subtasks (right) in Kitchen (a = 0.3). In the skill representation, marker colors denote tasks, while marker shapes indicate subtasks.
Mesa-Test Iteration
0.5K
0.5K < Kizrr
Meta- Meta-Train Train Iteration
2K
4K
Trajectory
Figure F3: Illustration of trajectories of refined skills during the meta-test phase in Maze2D (a 1.5), across various training and test iterations.
31

---

Published as a conference paper at ICLR 2026
F.3 IMPROVEMENT IN TASK REPRESENTATION THROUGH SKILL REFINEMENT
Fig. illustrates the effect of skill refinement on task representation through t-SNE visualizations 0.3), with different tasks represented by distinct colors. In Fig. F.4 (a), the task encoder is trained using fixed skills directly derived from noisy demonstrations. The noisy skills obstruct the encoder's ability form clear task representa- tions, making task differentiation challenging. This limitation highlights why, in SiMPL, relying on skills learned from noisy datasets can sometimes result poorer performance compared to SPiRI., which focuses on task-specific skill learning.
of task embeddings e? qu. in the Kitchen environment
Conversely, Fig. F.4 (b) presents the t-SNE visualization when the task encoder is trained during the meta-train phase with refined skills. The improved skills enable the encoder to form more distinet and task-specific representations, facilitating better task discrimination. This improvement allows the high-level policy to differentiate tasks more effectively and select optimal skills for each, thereby enhancing meta-RL performance. These findings demonstrate that the proposed skill refinement not only improves the low-level policy but also significantly enhances the task encoder's ability to represent and distinguish tasks, contributing to overall performance improvements.
(a)
(h)
(microwave bottom burner - cabinet)
bottom burner
burner)
(microwave - bottom bumnr top burner siide cabinet)
(micrewave - kettle burner light switch)
(kettle - bottomm burner light switch hinge cabinet)
[microwave
burne- light switch siide cabinet)
(microwave kekklllleee-- slide cabinet bottom burner)
(kettle bottom burner burner light switch)
(microwave kettle e bottom burner - top burner)
(micreweve keetle - light switch slide cabinet)
Figure F4: t-SNE visualization of task representations in the Kitchen environment (a 0.3): (a) Using only the noisy offline dataset, (b) Proposed SISL trained with refined skills
F.4 STABILITY REWARD MODEL LEARNING
Unlike offline-RL. RL, which typically relies reward-labeled datasets, skill-based RL operates reward-free offline data. In SISL, only the skills are learned from the noisy offline trajectories, while the reward model and the high-level policy trained online using clean, noise-free training tasks, as in SiMPL. As shown in Eq. 4.. the reward model is trained using transitions sampled from Bi and B which contain only online trajectories collected by interacting with training task Since these buffers contain offline data, the reward model is unaffected by offline noise. To further support this point, we present Table F.I which shows the MSE of the reward model remains consistently low across different offline noise settings in the Kitchen environment. These results indicate that the reward model maintains high accuracy and continues to support effective relabeling even when offline skill pretraining noisy.
Table F.1: MSE of reward model under different noise levels in the Kitchen environment.
Kitchen(Expert)
Kitchen(o=0.1)
Kitchen(G=00.2)
Kitchen(o=0.3)
MSE
0.005:10.00
0.005:40.00
0.004
0.005-10.001
32

---

Published as a conference paper at ICLR 2026
F.5 COMPARISON OF SKILL TRAJECTORY VISUALIZATIONS
To visually demonstrate the skill difference between the baseline algorithm and SISL, we have added visualization comparison of the skill trajectories for SPiRL, SiMPL, and SISL. Specifically, Fig. and [.6 present the skill trajectory visualizations of the microwave-opening and bottom-burner control subtasks in the Kitchen(o 0.3). The SPiRL and SiMPL fail complete these subtasks due to noise in the learned skills, either do not succeed in opening the microwave door or fail to properly reach the bottom- burner switch. contrast, SISL successfully grasps and opens the microwave door These results highlight the substantial impact of noise
F.5
and correctly manipulates the offline demonstrations on subtask performance, and demonstrate that SISL progressively refines skills even in noisy environments, successfully solving most given subtasks.
Timestep (t)
SPIRN
SiMPL
SISL
Figure F:5: Comparison of skill trajectory visualizations for the microwave-opening subtask in Kitchen(a = 0.3) across algorithms.
Timestep
t=5
SPiRL
SiMPL
SISL
Figure F.6: Comparison of skill trajectory visualizations for the bottom-burner control subtask in Kitchen(a = 0.3) across algorithms.
33

---

Published as a conference paper at ICLR 2026
G ADDITIONAL ABLATION STUDIES
In this section, we conduct additional ablation studies for Kitchen and Maze2D environments across all noise levels. These studies include component evaluation and SISL's skill refinement-related hyperparameters discussed in Section [.I] prioritization temperature T in Section [G.2] the KLD coefficient Amp for the skill-improvement policy in Section G.3 additional component evaluation on RND and re-initialization in Section G.4 skill refinement interval Kier in Section G.3 and com-
G.5
parison with Coal-Conditioned RI. Section G.6
G.1 COMPONENT EVALUATION
Fig. G. presents comprehensive results across all noise levels from the component evaluation in Section 5.5 While improvements are modest under conditions with high-quality offline datasets, such as Expert and Noise (o 0.1) for Kitchen, and Expert and Noise (a 0.5) for Maze2D, notable performance gains are still observed. The significant degradation observed in the absence of the skill-improvement policy Timp highlights its crucial role mitigating minor noise and dis- covering improved paths. For higher noise levels, such Noise (o = 0.2), Noise (a 0.3) for Kitchen, and Noise (o 1.0), Noise (o = 1.5) for Maze2D, excluding the online buffer or skill prioritization via maximum return relabeling (B.n PB.)) caused significant performance drops, emphasizing the importance of maximum return relabeling in SISL. Additionally, relying solely on the online buffer without utilizing the offline dataset led performance deterioration, demonstrat- ing the offline dataset's value in addressing meta-test tasks involving behaviors not available during meta-train. Beyond these specific findings, most trends align with the results discussed in the main text, further validating the effectiveness of SISL's components across different noise levels.
Expert
Noiselo 0.1)
Noiseld 0.2)
Noiseld 0.3)
4.0
4.0
4.0
Test Kitchen
3.6
3.6
3.2
3.2
2.5
Test Average Return
0.9
Maze20
0.8
0.8
0.7
0.7
0.7
SISL
SISL w/b PaL.
SISL w/o Aut
SISL wio Ran
SISL. w/o Rag
Figure G.1: Component evaluation across all noise levels Kitchen and Maze2D
G.2 PRIORITIZATION TEMPERATURE T
The prioritization temperature T' regulates the balance between sampling from online and offline buffers. Fig. G.2 shows performance ronments as varies. Low T values lead to excessive sampling from high-return buffers, while high T' approximates uniform sampling, diminishing the prioritization effect. Both environments experienced degraded performance at T = 0.1 and T = 2.0. highlighting the importance of proper tuning. In the Kitchen environment (maximum return 4), across all noise levels, whereas in the Maze2D environment (maximum return 1), 7 = 0.5 was optimal. This difference occurs because environments with lower max returns exhibit smaller gaps between low- and high-return buffers. reducing the effect of prioritization. Based these findings, we suggest a practical guideline: select T roughly in proportion to the environment's ''s maximum achievable return. This approach offers principled starting point for tuning T without requiring extensive hyperparameter sweep, thereby improving usability and reproducibility. Accordingly, we set the best-performing hyperparameter values defaults for each environment.
different noise levels in Kitchen and Maze2D envi-
1.0 achieved the best performance
34

---

Published as a conference paper at ICLR 2026
G.3 KLD COEFFICIENT Amppp
The KLD coefficient Ampp regulates the strength of the KLD term between the skill-improvement policy and the action distribution induced by the prioritized online buffer 8: for each task T Fig.
G.3 illustrates performance variations with Amp E [0, 0,000,11,,0 0.002, 0.005).
In the Kitchen environment, Aigg 0.005 performed best for Expert and Noise (a = 0.1), while Amp 0.002 and Amp 0.001 were optimal for Noise (o 0.2) and Noise (o 0.3), respectively. At lower noise levels, the high-level policy benefits from quickly following high-return samples, whereas at higher noise levels, focusing on exploration to discover shorter paths becomes more ad- vantageous. For the Maze2D environment, performance
consistent across Ampp 0.001, 0.002,
and 0.005, with only minor variations observed. However, when Amp = O, removing the KLD term resulted in significant performance degradation across all noise levels in both Kitchen and Maze2D environments. This highlights the necessity of guidance from high-return samples for effectively solving Long-horizon tasks. Based on these results, we selected the best-performing hyperparameter values defaults for each environment.
Expert
Noiselo 0.1)
Noise(a 0.2)
Noiselo - 0.3)
Test Average Retum
4.0
4.0
3.5
Kitchen
3.5
3.5
3.0
3,0
3.0
3.0
2,5
200
500
200
400 500
500
200
Iteration
Iteration
Iteration
Iteration
Expert
Noiselo maiselo-0.33 0.5)
Noiselo - -1.0)
Noisela -155)
Test Average Return
0.8
0.75
0.0
Maze2D
0.6
0.6
0.6
0.4
0.00
0.2
100 200 300 Iteration
400
500
100
200
300
400
500
100 200 500
400
500
100
200
300
400
500
Iteration
Iteration
Iteration
7=0.1
T=5
7= 1.0
T= 2.0
Figure G.2: Impact of the prioritization temperature 7' across all noise levels in Kitchen and Maze2D
Expert
Noise(a - 0.1)
Noise(a - 0.2)
Noise(a - 0.3)
Average Return
4.00
3.5
3.75
3.75
Kitchen
3.50
3.50
3.0
3.25
3.0-
3.00
2.5
2.75
2.0
100
200 300 Iteration
400
500
100
200 300 400 500
100 200
300 Iteration
400 500
100 200 300 400 500
Iteration
Iteration
Expert
Noise(a - 0.5)
Noise(a 1.0)
Noise(a - 1.5)
0.8
0.8
0.9
Maze2D
Awerage
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
200 300 400 Iteration
500
100
200
300
400 500
100 200
300
400 500
100 200
300 400 500 Iteration
Iteration
Iteration
AW% 0.001
AM
0.002
ANY 005
Figure G.3; Impact of the KLD coefficient Akld across all noise levels in Kitchen and Maze2D
35

---

Published as a conference paper at ICLR 2026
G.4 ADDITIONAL COMPONENT E EVALUATION
we performed additional ablations on the remaining components that could affect performance. These evaluations conducted in the Kitchen environment using both a Expert and a high-noise condition Noise(a = =0.3,, and the results
Following the ablation of SISL's core components in Section G.1
are shown Table G.1
The SISL w/o RND is a variant that excludes RND from skillimmprovement policy Timp training and relies solely on SAC. The results show performance degradation compared full SISL, with especially large drops under high-noise conditions where exploration capability is critical. This suggests that RND encourages exploration toward rare or less frequently visited states, enabling the discovery of more diverse and useful trajectories.
The SISI w/o Re-Initialize refers to variant that continues training the high-level policy m, without re-initializing it at each Kier during the meta-training process. In SISI,, the skill model is period- ically updated every Kiter steps, which effectively changes the environment dynamics observed by the high-level policy. Continuing to train the same high-level policy across different skill sets intro- duces non-stationarity. leading to instability. To address this, we reset both the policy parameters and the buffer at each skill update that the high-level policy can re-lear from scratch under a new, stable MDP defined by the updated skills. This technique is consistent with practices in continual and safe reinforcement learning, where re-initialization is often used manage sudden changes in task dynamics (Kim et al., 2023 Kong et al., (2024). To empirically validate this decision, we con- ducted ablation experiments on high-level policy re-initialization. The results show that removing re-in re-initialization degrades performance, supporting the necessity of this design choice.
Table G.I: Comparison of SISL with/without RND and re-initialization.
Dataset
SISI.
SISL. w/o RND SISL w/o Re-Initialize
Kixehen(Expert)
3.97+0.0
3,90
3. +
Kitchen(o 0.3)
3.48
3.14
0.41
G.5 SKILL REFINEMENT INTERVAL Kaer
As shown in Fig. 6 the high-level policy Ta in the Kitchen environment typically requires at least 1K iterations within each interval to improve and converge, implying that if Kisr is too small, skill refinement is triggered before Ti has sufficiently adapted to the current skill library. Based on this observation, we used Kiter = 2000 as the default setting in the Kitchen envi- ronment. To validate this intuition more rigorously, we conducted hyperparameter search over Kier E {100, 500, 1000,00000 5000} in the Kitchen (a = 0.3) setting. The results confirm our hypothesis: with Kite- 100, mi updated before it can meaningfully exploit the skills, leading to a substantial performance drop. Conversely, when Kier = 5000, тh converges well but skill refinement happens infrequently, reducing the overall performance gain. These findings support the reasoning behind our chosen interval.
Meta-Train
Average Retum
2K
6K
8K
10K
Iteration
Kew =
100
Ken
500
Kiner
1000
Knne 2000
Ko.r 5000
Figure G.4: Impact of the skill refinement interval Kaer in Kitchen(a = 0.3)
36

---

Published as a conference paper at ICLR 2026
G.6 COMPARISON WITH GOAL- CONDITINDT RL
We first note that standard goal-conditioned RL (GCRL) assumes explicit, environment-provided goals and single-task learning, whereas in meta-RL. setting goals given and the agent must infer task while learning a policy that generalizes across many tasks. Despite these differences, to provide an empirical reference point, we modify our environments to explicitly provide goals and evaluate a representative GCRL method, HER (Andrychowicz et al. 2017) with SAC, trained directly on the test tasks without offline data. As presented in Table G.2 HER achieves notably lower performance than SISL in Maze2D and fails to make progress in AntMaze, where the success rate remains at zero. Even with explicit goals, these sparse-reward long-horizon tasks appear difficult to solve without pretrained skills, which helps contextualize the advantages of our framework.
Table G.2: Final performance of goal-conditioned RL (HER) given target goal.
Environment
SAC+HER
SISL
Maze2D(EEpprt) Maze2D(o = 0.5)
0.87 10.05
0.89110.0
0.37+0.09
Maze2D(o = 1.0)
0.00
Maze2D(o = 1.5)
0.99
Antpaze(Expert)
0..1
AntMaze(a = 0.5)
0.82
0.00
AntMaze(o 1.0)
0.60=+000
AntMaze(o = 1.5)
0.41#0001
37


Reviews:
[{"review_id": "K3WwTlOBds", "reviewer": "Reviewer_brpU", "summary": "The authors propose SISL (Self-Improving Skill Learning), a skill-based meta-RL framework designed to handle noisy offline demonstrations in long-horizon tasks. SISL introduces two key mechanisms: (1) decoupled skill self-improvement, where a dedicated improvement policy explores near the offline data distribution to discover higher-quality rollouts that progressively refine the skill library, and (2) skill prioritization via maximum return relabeling, which uses a learned reward model to assign estimated returns to offline trajectories and reweight samples through softmax prioritization.", "questions": "Shouldn't $\\beta$ in Eq. 7 be task-dependent? I guess that sometimes, different tasks may benefit from different balances between $B_{\\text{off}}$ and $B_{\\text{on}}$. How sensitive is performance to using a global $\\beta$ versus task-specific $\\beta^i$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-motivated and very clearly presented, with very useful illustrations.\n- Separating exploitation ($\\pi_h$) and skill improvement ($\\pi_{imp}$) with self-supervised guidance via prioritized buffers appears to be a novel and elegant contribution.\n- The evaluation is sound and detailed, with 4 diverse environments with multiple noise levels, thorough baseline comparisons, and extensive ablations.", "weaknesses": "I believe the paper would benefit from comparing against a GCRL baseline with Hindsight Experience Replay (HER) or similar relabeling techniques. This would help demonstrate that SISL's approach to leveraging the offline dataset is superior to existing relabeling methods in both sample efficiency and final performance. \n\nMinor typo: \n- Line 279: \"addtion\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "quEREY58zt", "reviewer": "Reviewer_jnLn", "summary": "This paper targets the sensitivity of skill-based meta-RL methods to noisy offline demonstrations. The proposed method, Self-Improving Skill Learning (SISL), addresses this by dynamically refining the skill library during meta-training. Its core contributions are:\n\n- Decoupled Skill Self-Improvement: A dedicated skill-improvement policy $\\pi_{imp}$ is introduced alongside the standard high-level policy $\\pi_h$. $\\pi_{imp}$ explores near the offline data distribution to find higher-quality trajectories.\n- Skill Prioritization via Maximum Return Relabeling: To fuse the noisy $B_{off}$ with the clean $B_{on}$ for skill refinement, SISL trains a reward model using only online data. This model assigns a hypothetical maximum to each offline trajectory. The skill update then samples from $B_{off}$ using a softmax prioritization based on $\\hat{G}$, effectively filtering out low-quality data.\n\nExperiments on long-horizon tasks show that SISL substantially outperforms baselines in noisy data regimes.", "questions": "- Figure F.1 shows that the mixing coefficient $\\beta$ quickly converges to high values, implying that the online-generated data quality significantly surpasses the offline data. This raises the question: could the offline data $B_{off}$ be discarded altogether? For instance, what if one first trained an expert policy (e.g.,$\\pi_{imp}$ trained without the KL-divergence term, acting as a standard RL agent) to collect a new, clean dataset, and then used this dataset for learning $\\pi_l$ and $\\pi_h$? Would this not be a simpler and potentially lower-cost alternative to the complex relabeling and mixing process?\n- Does the framework assume that only optimal trajectories are useful for training the low-level policy $\\pi_l$? It is possible that certain suboptimal trajectories, while not useful for the current set of training tasks, might contain skills that are highly beneficial for generalizing to unseen test tasks. The \"Maximum Return Relabeling\" mechanism prioritizes trajectories based on their estimated maximum return on *training* tasks. Does this design inadvertently suppress these \"suboptimal but generalizable\" trajectories, thereby weakening the utilization of data that could be crucial for meta-generalization?\n- See Weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This work focuses on a key practical problem in meta-RL, where offline demonstrations may be noisy.\n- The paper provides strong empirical evidence demonstrating significant performance improvements over all baselines.\n- The paper is well-structured. The method is presented logically.", "weaknesses": "- The paper claims only \"16% more time per iteration\", which seems surprisingly low. A SISL iteration appears to involve: (1) rollout with $\\pi_h+\\pi_l$, (2) rollout with $\\pi_{imp}$, (3) $\\pi_h$ update, (4) $\\pi_{imp}$ update, (5) $\\pi_l$ update, and (6) $\\hat{R}$ update. In contrast, the baseline presumably only includes steps (1) and (3). It is unclear how the 16% figure was calculated. A more detailed breakdown and a comparison of total training time, not just per-iteration cost, would be more informative.\n- The skill refinement interval, $K_{iter}$, is a critical new hyperparameter that balances the stability of the high-level policy $\\pi_h$ against the speed of skill refinement for $\\pi_l$. However, the paper provides no ablation study for $K_{iter}$, making it difficult to assess the algorithm's sensitivity to this important design choice.\n- The \"maximum return relabeling\" mechanism hinges on a reward model $\\hat{R}$ trained on online data. This implies a potential failure mode: in tasks with sparse or complex rewards, training $\\hat{R}$ could become unstable, compromising the data prioritization mechanism and overall performance.\n- Lack of Clarity in Pseudocode. For example:\n    - In Algorithm 1 (Meta-Train), line 14 updates the skill parameters $\\phi$ (including $q_{\\phi}$). However, the algorithm does not show where the skill encoder $q_{\\phi}$ is used for inference on trajectories.\n    - In Algorithm 2 (Meta-Test), line 7 updates parameters $\\theta$. It is ambiguous whether$\\theta$also includes the task encoder $q_{e,\\theta}$. If the task encoder is frozen during the meta-test phase (as is typical), it would be clearer to use different notation to distinguish its parameters from the policy/value function parameters being adapted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lhanaLncGr", "reviewer": "Reviewer_SPY1", "summary": "The paper addresses the topic of skill-based meta-learning in reinforcement learning. It points out existing skill learning methods relying on offline demonstrations often suffer from suboptimal data quality. The proposed Self-Improving Skill Learning (SISL) framework shows robustness to noisy demonstrations by introducing a skill improvement policy and performing skill refinement through trajectory prioritization based on returns.", "questions": "(1) How does the method avoid overfitting to the source tasks when evaluating trajectories based on returns?\n\n(2) Please compare the skills extracted by SISL with those obtained by other baseline methods.\n\n(3) Consider including an additional experiment where Equation (3) is evaluated without the KLD term.\n\n(4) Please report the success rate on the maze environments.\n\n(5) Compare the computational cost across different ablation settings.\n\n(6) Could you clarify why performance increases as $\\sigma$ increases in some cases in Table 1?\n\n**Minor**\n\nIn Figure 2, which algorithm is being illustrated?\n\nHow is the KLD term in Equation (3) computed?\n\nHow many offline trajectories generated by the noisy behavior policy are used to solve each task?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper adopts perturbations, a representative technique in meta-learning, where target tasks are unseen by the skill-based reinforcement learning agent. This approach constrains the learning process to remain close to the demonstration manifold, thereby facilitating effective skill acquisition.", "weaknesses": "The SISL framework appears to be quite complex and computationally expensive. For example, it involves maintaining multiple buffers that serve similar functions. The proposed method also does not seem to be specifically tailored to address the meta-learning problem. Baselines such as SPiRL and SiMPL learn skills solely from offline demonstrations, whereas SISL additionally collects online data, making the comparison unfair.\n\n**Minor**\n\nAppendix B should be moved to the main body of the paper. The figures and their captions are positioned too close to the main text, which reduces readability. Figure captions are brief and could be improved to provide clearer explanations of the content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h2a64T6KJZ", "reviewer": "Reviewer_LUR5", "summary": "This paper addresses a critical challenge in skill-based meta-RL: performance degradation when learning skills from noisy or suboptimal offline demonstration datasets.\n\nThe authors propose a new framework, SISL, designed to be robust to such data imperfections. SISL's novelty lies in two key contributions:\n- Decoupled Skill Self-Improvement\n- Skill Prioritization via Maximum Return Relabeling", "questions": "- **Sensitivity to $K_{iter}$**: The periodic re-training of the skill model and re-initialization of $\\pi_h$ every $K_{iter}$ steps is a critical design choice. The ablation in G.4 proves re-initialization is necessary, but how sensitive is the algorithm to the value of $K_{iter}$? What happens if re-training is too frequent (e.g., $K_{iter}=100$) or too infrequent (e.g., $K_{iter}=5000$)?\n\n- **Necessity of Decoupled Policies**: Why is the decoupling of $\\pi_h$ and $\\pi_{imp}$ strictly necessary? Could a single high-level policy not perform both roles?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Significance**: The paper tackles a highly significant and practical problem: the sensitivity of skill-based (meta-)RL algorithms to noisy offline demonstration data. As the field moves toward using large, imperfect, real-world datasets, methods that can \"denoise\" and \"self-improve\" upon this data are critical. This work provides a strong solution to this problem.\n\n- **Clarity and Presentation**: The paper is a clear. It is well-written, logically structured, and supported by high-quality figures that provide both conceptual intuition (Fig. 1, 3) and compelling qualitative evidence (Fig. 7, F.2).\n\n- **Empirical Rigor and Performance**: The experimental evaluation is good. The authors demonstrate SISL's superiority across four challenging long-horizon environments against a comprehensive set of relevant baselines.", "weaknesses": "- **Complexity and Computational Cost**: The SISL framework introduces several new components that add to the computational load: training the skill-improvement policy, the RND networks, and the reward model, plus periodically re-training the entire skill library.\n\n- **Periodic Re-initialization**: The design choice to re-train the skill library and re-initialize the high-level policy is a key part of the algorithm. The ablation in G.4 shows that not re-initializing is catastrophic, which makes sense due to non-stationarity. However, this periodic \"reset\" might be disruptive, and the paper does not explore the sensitivity to the frequency itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)