Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning
Authors: Jinmin He
Kai Li
Yifan Zang
Haobo Fu
QIANG FU
Junliang Xing
Jian Cheng
Venue: ICML 2025 poster (2025)
Keywords: Offline Multi-Task Reinforcement Learning
Skill Abstraction
Primary area: reinforcement_learning->batchoffline

Abstract: Offline multi-task reinforcement learning aims to learn a unified policy capable of solving multiple tasks using only pre-collected task-mixed datasets, without requiring any online interaction with the environment. However, it faces significant challenges in effectively sharing knowledge across tasks. Inspired by the efficient knowledge abstraction observed in human learning, we propose Goal-Oriented Skill Abstraction (GO-Skill), a novel approach designed to extract and utilize reusable skills to enhance knowledge transfer and task performance. Our approach uncovers reusable skills through a goal-oriented skill extraction process and leverages vector quantization to construct a discrete skill library. To mitigate class imbalances between broadly applicable and task-specific skills, we introduce a skill enhancement phase to refine the extracted skills. Furthermore, we integrate these skills using hierarchical policy learning, enabling the construction of a high-level policy that dynamically orchestrates discrete skills to accomplish specific tasks. Extensive experiments on diverse robotic manipulation tasks within the MetaWorld benchmark demonstrate the effectiveness and versatility of GO-Skill.

Goal-Oriented Skill Abstraction for Offline Multi-Tash Reinforcement Learning
Jinmin He
Kai Li Yifan Zang" Haobo Fu
Qiang Fu
Junliang Xing
Jian Cheng
Abstract
door-open
drawer-upen
Task Set
Offline multi-task reinforcement learning aims to learn unified policy capable of solving mul- tiple tasks using only pre-collected task-mixen datasets, without requiring any online interaction with the environment. However, it faces signifi- cant challenges in effectively sharing knowledge across tasks. Inspired by the efficient knowledge abstraction observed in human learning, we pro- pose Goal-Oriented Skill Abstraction (GO-Skill), novel approach designed to extract and utilize reusable skills to enhance knowledge transfer and task performance. Our approach uncovers reusable skills through goal-oriented skill extrac- tion process and leverages vector quantization to construct a discrete skill library. To mitigate class imbalances between broadly applicable and task- specific skills, we introduce skill enhancement phase refine the extracted skills. Furthermore, we integrate these skills using hierarchical pol- icy learning, enabling the construction of high- level policy that dynamically orchestrates discrete skills to accomplish specific tasks. Extensive ex- periments on diverse robotic manipulation tasks within the Meta World benchmark demonstrate the effectiveness and versatility of GO-Skill.
Unicycle
Motorcycle
Grab Handle
button- press
coffee button
Bicycle
Automobile
Balance
Press Button
Podal
bin pickingking
pick of hole
Ignition
Pick Up
(a) Skill Sct in Vehicles
(b) Skill Set Manipulation
Figure 1. Skill shared across different domains. (a) Skill set for human learning to operate four types of vehicles: bicycles and motorcycles share the skill of balancing and bicycles share the pedal-driven skill; motorcycles and auto- mobiles share the ignition-to-movv skill. (b) Skill in robotic manipulation tasks: grabbing various handles, pressing buttons different heights, and picking up objects from diverse locations.
wheels; unicycles
address these challenges, offline RL (Levine et al., 2020) has emerged as a data-driven approach, enabling policy learn- ing entirely from pre-collected datasets, thereby eliminating the need for real-time interaction. Recent advancements in offline RL have expanded its applicability to single-task domains, such robotics (Kalashnikov et al., 2018; 2021) 1) and healthcare (Guez et al., 2008; Killian et al., 2020). In contrast, offline multi-task reinforcement learning (MTRL) seeks to effectively master set offline RL tasks using the pre-collected task-mixed dataset. Joint multi-task learning generally enhances adaptability and performance compared to training each task individually.
1. Introduction
Deep reinforcement learning (RL) has made significant progress in recent decades, demonstrating its effectiveness in diverse domains such game playing (Mnih al., 2015; Ye et al., 2020) and robotic control (Lillicrap et al., 2016; Levine et al., 2016). However, its deployment in real-world scenarios is often limited by the high costs and risks as- sociated with directly interacting with the environment. To
DL' Institute of Automation, Chinese Academy of Sciences "School of Artificial Intelligence, University of Chinese Academy of Sciences 'Tencent AI Lab "Tsinghua University AiiRiA Cor- respondence Kai <kai. ia..aaaaa.n...... Junliang Xing <jlx- ing @ig@iinnuua..du.c.>>
A critical challenge in offline MTRL lies in effectively shar- ing knowledge between tasks, Recent research has proposed some approaches by leveraging network parameter shar- ing, incorporating carefully designed architectures (Yang et al., 2020; He al., 2024b; Hu al., 2024), task-specific
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Goal-Oriented Skill Abstraction for Offline Multi- Task Reinforcement Learning
representations (Sodhani et al., 2021; Lee et al., 2022; He et al., 2023), and tailored optimization procedures (Chen et al., 2018; Yu et al., 2020a; Liu et al., 2021). However, these methods predominantly operate at a lower level, fo- cusing on action-based imitation learning, which contrasts with how humans typically approach learning. Humans tend to abstract knowledge more effectively by identify- ing and summarizing common strategies or patterns from past experiences into set of reusable skills. Instead of learning solely at the action level, they dynamically com- bine these high-level skills to address specific tasks. For instance, when learning to operate four different types of vehicles vehicles-unicycle,, bicycle. motorcycle, and automo- bilenhumans distill shared skills as illustrated in Fig- ure I(a). This human-inspired mechanism of skill abstrac- tion offers a fresh perspective for offline MTRL, suggest- ing that agents could similarly distill and leverage skills to enhance learning efficiency, as demonstrated in robotic manipulation scenarios in Figure 1(b).
tasks. Our method demonstrates significant improvements over existing offline MTRL algorithms. Additionally, we conduct comprehensive ablation studies to assess each GO- Skill component's 's contribution.
2. Preliminaries
Offline Multi-Task Reinforcement Learning. We aim to learn N tasks simultaneously, each represented as Markov Decision Process (MDP) (Bellman, 1966; Puter- man, 2014). Specifically, each task i E T is defined by the tuple A, R,, de), where S denotes the state space, A the action space, P: S x A - S the environment transi- tion dynamics, R, : S x A - the reward function, and d, the initial state distribution. In line with common MTRL setting (He et al., 2024b; Hu et al., 2024), all tasks share the same state and action spaces but differ in transition dynam- ics, reward functions, and initial state distributions. The goal of the MTRL agent is to maximize the average expected re- turn across all tasks, expressed as ExgrE i-i= (s,,,, at)]. In the offline setting (Levine et al., 2020), instead of collect- ing data through environment interactions, each task has a static dataset D. = {{(s,, a, 8 r)), consisting of trajectories from the environment.
Motivated by this, we propose Goal-Oriented Skill Abstrac- tion (GO-Skill), a novel method for discovering skill li- brary from offline task-mixed datasets and learning skill- based policy capable of addressing multiple tasks through skill combination. First, we introduce a skill model that leverages goal-oriented representations to extract reusable skills, then uses vector quantization to construct a discrete skill library. These skills act as low-level policies, enabling interaction with the environment. To address the potential class imbalance dilemma, where widely applicable skills have abundant data while task-specific skill data is limited, we incorporate skill enhancement phase. Finally, we uti- lize the hierarchical policy learning approach, in which ball- policy is learned using the discrete skill space as the action space. This enables the effective combination of these skills to solve specific tasks.
Prompt Decision Transformer. The Transformer architec- ture (Vaswani et al., 2017), widely used in natural language processing (Devlin et al., 2019) and computer vision (Car- ion et al., 2020), has demonstrated superior performance over traditional RNN-based models. Recently, it has been applied solve RL problems, leveraging its efficiency and scalability when dealing with long sequential data. The De- cision Transformer (DT) (Chen et al., 2021) for offline RL forms the policy learning as a sequential modeling prob- lem. Prompt-DT (Xu et al., 2022) extends the DT approach by incorporating prompt-based techniques, enabling few- shot generalization to new tasks. It introduces trajectory prompts, which consist of return-to-go, state and action tuples (f*,s" s", s²"a") from task-specific trajectory demonstra- tions. During training with offline collected data, Prompt- DT utilizes Tn = (Tf, Ti,e) as input for each task i, where of represent -tes trajectory prompt and Ti, denotes the most recent K K-step history. This is formally represented as:
GO-Skill not only mirrors the way humans learn but also has the potential to improve sample efficiency in offline MTRL significantly. Specifically, offline trajectory data is not exclusively composed of expert demonstrations but may include sub-optimal or random trajectories. Although sub- optimal trajectories may provide limited direct benefits for a specific task, they often contain transferable skill sub- trajectories that can be valuable for other tasks. GO-Skill excels at extracting these reusable skill fragments from less- than-optimal data, enabling their application across diverse tasks. Furthermore, skill abstraction shortens the decision- making horizon for the skill-based policy to accomplish tasks, simplifying the policy learning process. This approach improves sample efficiency and contributes to a more robust and effective
(f.,,, ai...... F... KKK, input T,t
(1)
K, ait- -+.............. Pra, 8 a.,))
The model's prediction head, connected to a state token 8j. is designed to predict the corresponding action ay. For con- tinuous action spaces, the training objective is to minimize the mean-squared loss, defined as:
learning process.
We evaluate GO-Skill in the MetaWorld benchmark (Yu et al., 2020b), which consists of 50 robotic manipulation
CDT = B,iuu
Σ K+ K+1
l - (Tf (0i1 (2)
2

---

Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforvement Learning
a
II-2
Skill Decader Transformer
VQ
Prompt 2
I-1
0
Goal Encoder
Goal Encoder
Goal Encoder
3. 8g
Skill Codebook
#g 80
8 By
Figure 2. Illustration of the skill model in GO-Skill, which comprises three key components: (1) Goal Encoder maps the trajectory-level difference between states a latent embedding space: (2) Skill Codebook distills the -horizon goal embedding into discrete skill set via vector quantization; (3) Skill Decoder reconstructs actions with skill prompt and history sequence using a Transformer architecture.
Vector Quantization. The vector quantization (VQ) mod- ule (Van Den Oord et al., 2017) has been used for many applications by learning discrete latent representations. This VQ module FF maintains a latent embedding space (code- book) M E RCXL, composed of C' vectors e; E IR Given an encoder output = E RR,,, it selects the discrete latent vari- able as arg min; ||= 2l... Based on the discrete code j. get the query vector F(z) ej from the codebook. During training, the codebook and encoder output are opti- mized with the following loss,
consists of three parts: goal encoder, skill quantization, and skill decoder.
Goal Encoder. The skill achieves a dynamic transfer from the initial state 8 to the target state $1+4- To capture this process a task-agnostic manner, we introduce the goal as representation of this dynamic transfer. For the H-horizon sub- tubjttrjectory, we define the goal encoder 9 S - Z, which maps the trajectory- the initial state 8, and the target state 8t-11 to latent embed- ding space Z. Specifically, the goal embedding is expressed as,
dynamic transfer between
Lvo(z,ej) ez) = ||s][=] -e,l2 + sgle,]11
H1, = G( - -8)--
(4)
where sg[-] is stop-gradient operator, and a adjusts the bal- ance of minimizing the discrepancies between and F(z).
Here, we represent the H-horizon dynamic transfer directly in terms of state variation. For more complex state situation, goal-oriented representation can be defined. For example, in navigation tasks, the goal may primarily depend on the agent's 's position. cases where the state is represented as image, we can leverage the pre-trained vision-language models (Nair et al., 2022; Ma et al., 2023) convert the state into a relevant embedding representation. Notably, we do not use action sequences to represent dynamic transfer, as different action sequences can result in the same transfer outcome. This distinction is analyzed in our experiments Section 5.3.
3. Method
Motivated by knowledge abstraction learning illustrated in Figure 1, we propose Goal-Oriented Skill Abstraction (GO- Skill). This approach addresses two key challenges: 1) Ex- tracting reusable skills from the offline task-mixed dataset. 2) Leveraging these skills effectively to tackle diverse tasks.
3.1. Goal-Oriented Skill Extraction
GO-Skill extracts discrete library of reusable skills from the offline dataset by learning a skill model, which leverages trajectories across diverse tasks to build a robust foundation. We define skill as a sub-sequence of trajectories with a fixed horizon H. Distinguishing from common sequence modeling approaches (Chen et al., 2021; Xu et al., 2022; Lee al., 2022) that predict action based on past (return- to-go, state, action) experience, our skill model excludes reward signals (reward and return-to-ge from the skill defi- nition, as the reward function varies across tasks, even for identical trajectories. While skill trajectory may not yield high rewards in the context of a particular task, it can be invaluable for mastering essential skills required in other tasks. As shown in Figure 2, the skill model in GO-Skill
Skill Quantization. sequences into discrete library of skills, we use vector quantization (Van Den Oord et al., 2017) to learn discrete la- tent skill representations with goal embeddings Ht Specif- ically, the acquired collection of skill embeddings (code- book) is denoted as Z = (2))) (2)) ,...,, (MM) }. where M is the predefined number of skills. The skill embedding is
To distill the large number of sub-
generated through the VQ module F : Z + with the
H horizoo goal embedding from the Goal Encoder:
2, = F(z..)) = arg min
21, 20) 2'
(5)
and the skill codebook Z is optimized using the VQ loss as described in Equation 3.
3

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Reinforcement Learning
Skill Embedding
Skill- Baill-Based Decision Transformrr
Prompt
Skill Codebook
H
AIR
Skill Index
Demonstration
F
K
K
Figure 3.
Illustration of the skill-based policy in GO-Skill. The policy, employing the Transformer architecture, uses the discrete skill space the action space. Its inputs include (return-tooo state, skill) prompts, and H Htspep interval historical trajectory. At each decision point, the policy predicts a skill index, which is subsequently mapped to its corresponding skill embedding via the skill codehook.
Skill Decoder. The skill model serves as low-level policy for interacting with the environment. We run the history sub-trajectory along with the skill embedding through the skill decoder to obtain an approximate reconstructed action. Beyond utilizing historical states and actions, we extend the goal include historical sequences. The goal encoder is adapted to within horizon of H, extending Equation 4 as,
We summarize the training procedure for skill extraction and enhancement in Algorithm 1.
3.2. Goal-Oriented Skill Policy Learning
The skill model can be treated as a frozen low-level policy to interact with the environment at the action level. To ac- complish tasks, we employ a hierarchical policy learning scheme. In this scheme, a high-level skill-based policy is learned within discrete skill space to select the appropriate skills.
rr, = (8++4 - st), 0 S h S H.
(6)
We refer to the goal embeddings (h (h- H), represent- ing the difference between the current state Sith and the initial state s, in the history sequence, as the reached-gon goal, which facilitates the agent in assessing the progress toward completing the current skill.
Dataset Preprocessing. We partition the trajectory data into segments of H time steps and annotate each segment with a skill index z' and skill embedding 2, which are gener- ated by the frozen goal encoder and skill codebook. The resulting
In summary, the skill decoder is formalized as P x $s - AA, where $s.2,, denotes the (state, reached- goal, action) trajectory space, and AA stands for the proba- bility distributions space over A. In our implementation, we use the Transformer architecture (Vaswani et al., 2017),
policy trajectory data is represented as T {(fr, ST, &r, 24-} where T denotes decision points at intervals of H time steps. The same preprocessing is applied to prompt demonstration trajectories.
GO-Skill Policy Learning. As shown in Figure 3, we train a skill-based decision transformer using Prompt-DT (Xu et al., 2022) as the policy to select a skill at intervals of H time steps.
av ~ P(ze, sgv' 2 zgu, acv), t <t H.
where sg1 = St. St+1....
is the state history, zge 2r,0 21,1 2t,1....., "1,1' is the reached-goal history, acr' ag d.t1s..., ar- is the action history, and the skill em- bedding constitutes the prompt for this decoder transformer.
T--1"", s" AST, SCT, RcT),
(8)
where f fi, f2.......... fk. represents the return-to-go prompt, 8" = st 8 s...... 8k. represents the state prompt, 2 21, 25, k- represents the skill prompt. Addition- ally, fer is the return-to-go history, 8&T is the state history, & is the skill history. The skill-based policy outputs a skill index 8/., which is then used to query the codebook for the corresponding skill embedding žy. To address the imbalance in skill classes, incorporate focal loss (Lin et al., 2017) into training,
Skill Enhancement. As the skill model undergoes train- ing, the set of skills gradually stabilizes. However, no- table challenge, skill class imbalance, arises. The decoder tends to favor mastering broadly applicable skills with abun- dant data while neglecting task-specific skills often con- strained by limited data availability. Since every skill is essential, including task-specific ones, we propose a skill enhancement phase after the skill set has been fixed. Once the skill set converges, skill codebook. Next, we partition the dataset by skill class D() = {-(() = {( H, aca+ -1)}. To miti- gate the imbalance, we apply a resampling strategy that uniformly samples across skill classes, thereby enhancing the training of the skill decoder.
freeze the goal encoder and the
(FI(T) (11 - ()T()))) log (T()))
(9)
where 2 0 is the tunable focusing parameter. The GO- Skill policy learning process is outlined in Algorithm 2. Since the skill decoder and the skill-based policy operate in- dependently, they are trained in parallel in implementation.

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Reinforcement Learning
Algorithm 1 GO-Skill Extraction and Enhancement
Algorithm 2 GO-Skill Policy Learning
Input: skill horizon H, skill set size M, goal encoder g, skill VQ module F, skill decoder P. learning rate 1) Initialize: the parameters of the network Fg
Input: skill-based policy т, learning rate 1 Initialize: the parameters of the policy 0z for each training iteration do for each task E T do
// Skill Extraction for each extraction training iteration do for each task i E T do
get minibatch Ti = (fi,T.TH SI,T.T- KK 2.TT+K) get prompt T," = (ff,.K- I. K.I.K)) end for
get minibatch Ti = (S1, (1.+++-11 end for
get batch data T = , TheeTT skill-based policy predict (-17))
get batch data r = [n.er
get goal embeddings (Z = (s 81) + build batch goal-oriented trajectory training data 7'
CF = (11 (2)x11))) log (T (1)+)))
fx s- 0x 1VVFF
(Sta+ 11 Ft -,,...++1111
end for
get skill embedding &, = H(
predict action 0.f ~ P(2,, $5u, zge; acv)
1
Lvq ||sg[zr.11] - 2/1/2 + a |||,,11 sgl 2,
Skill Codebook
Skill-based Policy
LMSE - 17 2x= a.. - arl 112
$T
Gg.F.P + - + Lvq)
end for
Goal Encoder
8,
// Skill Enhancement build skill class dataset D( = (7)
Environment
for each enhancement training iteration do for each skill index = 1.... M do
Skill Decoder
a2
get minibatch (1) b = 1.. ( end for get batch data 7' = { MM get skill embedding 21 = F (2)..1)
(
1
Figure 4. Deploying GO-Skill as a control policy to interact with the environment. T represents the decision points of the skill-based policy at intervals of H time steps, while f indicates the time steps where the skill model predicts actions within the H7-horizon.
Jim1
predict action a. ~ P( 85u' zst' azr)
LMSE H -t | a l|2
0p + 0p VLVSE
icy are fine-tuned in parallel, enabling skill enhancement and policy adaptation to new tasks.
end for
4. Related Work
3.3. Deployment
Offline Reinforcement Learning, Offline RL focuses on learning policies from static offline datasets without requir- ing direct interaction with the environment. Unlike tradi- tional RL, offline RL faces the challenges of distribution shift between the collected data and the learned policy (Fu- jimoto al., 2019). To address this, prior research has ex- plored constrained and regularized policy updates that limit deviations from the behavior policy (Kumar et al., 2020; Kostrikov et al., 2021; Ghosh et al., 2022). A notable di- rection in offline RL is conditional sequence modeling, in which actions are predicted from sequences of past expe- riences. This approach, which is grounded in supervised learning, inherently constrains the learned policy within the distribution of the behavior policy while optimizing for fu- ture trajectory metrics (Chen et al., 2021; Xu et al., 2022; Yamagata et al., 2023). Additionally, recent studies have incorporated diffusion models into offline RL (Janner et al., 2022; Chen et al., 2023; Wang et al., 2023), leveraging the strengths of generative modeling to represent policies or
Deploying GO-Skill a policy is illustrated in Figure 4. every H-step interval, the skill-based policy selects the skill index E'z for the next H time steps, based on the given prompt and historical trajectories. The predicted skill index 2/, is then mapped to its corresponding skill embedding 2r by querying the skill codebook. During the H-step pe- riod, the initial state Sto (where sto = ST) and the current state 8, are processed by the goal encoder to generate the reached-gan goal 2tg. Finally, using the skill embedding ēr and historical information within the current skill hori- (8)
(+) zon (gg:: ttt a1):t-), .). the skill decoder generates actions a, to interact with the environment.
3.4. Fine-Tuning for New Tasks
GO-Skill is also well-suited for transfer scenarios. First, we construct the based policy trajectory dataset de- scribed in Section 3.2 using the frozen goal encoder and skill codebook. Next, the skill decoder and skill-based pol-
5

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Reinforcement Learning
MTDT
Episode Return (x110)
3.0
MH-MTDT
PromptDT
MH-PrompeDT
MTDIFF-P
HarmoDT
2.0
GO-Skill
MH-GO-Skill
1.5
MT30 Near-Optimal
MT30 Sub-Optimal
MT50 Near-Optimal
MT50 Sub-Optimal
Figure 5. Average episode return across 5 random seeds on the MetaWorld benchmark with four different setups. The error bars represent the standard deviation across the 5 seeds. Each method is trained for 1c5 iterations, and each task is evaluated over 50 episodes.
dynamics, yielding competitive or superior performance.
to acquire a new one. Moreover, He al. (2024a) employs a guide policy that leverages the control policies of other tasks to facilitate exploration and guide the target task learning. In work, also utilize skill embedding technique, but GO-Skill adapts to conditional sequence modeling with an innovative goal-oriented skill abstraction in offline MTRL.
Multi-Task Reinforcement Learning. Multi-task learning is training paradigm that enhances generalization by lever- aging domain knowledge shared among related tasks (Caru- ana, 1997). MTRL builds on this concept to discover shared knowledge across tasks by concurrently learning multiple reinforcement learning tasks. A straightforward approach to MTRL involves task-conditional modeling, using tech- niques like conditional representation (Sodhani et al., 2021), conditional sequence modeling (Xu et al., 2022; Lee al., 2022), or conditional diffusion model (He et al., 2023). Due to gradient conflicts arising from simultaneously optimizing a shared model across diverse tasks, some methods (Chen et al., 2018; Yu et al., 2020a; Liu et al., 2021) reformulate MTRL as multi-objective optimization problem, modi- fying the update process mitigate these conflicts. Alter- natively, other approaches avoid fully sharing the network across tasks, instead dynamically selecting sub-networks through mechanisms like distinct heads (D'Erar 'D'Ermmo et al., 2020), specialized modules (Yang et al., 2020; He et al., 2024b), or task-specific masks (Hu et al., 2024), enabling the construction of tailored task policies. While most ap- proaches implicitly incorporate knowledge sharing within network parameters, GO-Skill leverages explicit knowledge abstraction to enhance generalization.
5. Experiments
5.1. Environments and Baselines
Environments. Our experiments are evaluated on the Meta- World benchmark (Yu et al., 2020b), an MTRL benchmark consisting of 50 robotic manipulation tasks using the sawyer arm in the MuJoCo environment (Todorov et al., 2012). We consider two setups: MT30, which includes suite of 30 tasks, and MT50, which comprising all 50 tasks. In line with recent studies (Yang et al., 2020; He al., 2024b), we extend all tasks to a random-goa goaalllooollloooo setting. Following He et al. (2023), the dataset are sourced from SAC-Replay (Haarnoja et al., 2018) ranging from random to expert experiences, di- vided into two compositions: Near- Optimal containing the complete experiences and Sub-Optimal consisting of the ini- tial 50% experiences. Details on environment specifications are available in the Appendix A.
Baselines. We compare GO-Skill against six baselines. (i) MTDT: Extension of DT (Chen et al., 2021) multi- task settings. (ii) MH-MTDT: Extension of MTDT apart from independent heads for each task to predict action. (iii PromptDT (Xu et al., 2022): Builds upon DT,, leveraging trajectory prompts for generalization to multi-task learn- ing. (iv) MH-PromptDTT: Extension of PromptDT apart from independent heads for each task predict action. (v) MTDIFF-P (He et al., 2023): A diffusion-based method combining Transformer architectures and prompt learning for generative planning in offline MTRL settings. (vi) Har- moDT (Hu et al., 2024): A DT-based approach that learns tpec-s masks for each task, mitigating the adverse effects of conflicting gradients during parameter sharing.
Skill in Reinforcement Learning. To utilize offline datasets for long-horizon tasks, recent research has explored hierar- chical skill learning approaches within the realms of online reinforcement learning (Pertsch et al., 2021; 2022) and imi- tation learning (Hakhamaneshi al., 2021; Nasiriany al., 2023; Du et al., 2023). Pertsch al. (2021) introduced hierarchical skill framework designed to enhance the adapta- tion to downstream tasks by incorporating pre-learned skill priors that inform a high-level policy. At the same time, Hakhamaneshi et al. (2021) developed a semi- parametric method within a hierarchical framework, emphasizing its application shot imitation learning. In online MTRL, Shu et al. (2018) uses hierarchical policy that decides when to directly previously learned policy and when
6

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Reinforcement Learning
Episode Return (x10")
3.2
3.0
8
10000 20000
30000
40000
Skill Horizon (H))
Codebook Size (M)
Goal Embedding Size (12))
Extraction Iteration (t.)
Figure Ablation study on the hyper-pane pperr-rareette of GO-Skill under the Near- Sear-Optimal setting in MT50 setup. Default parameter values are set skill horizon H = 10, codehook size M = 16, code emhedding size 2 64, and extraction iteration fe = 3ed (30% of total iteration). Each ablation varies single parameter while keeping others fixed at their default values to isolate its effect on performance.
Table 1. Ablation study on the skill model of GO-Skill under Near- Optimal and Sub- Sut-Optimal settings in the MT50 setup. RG: The skill decoder utilizes the reached-goal history. VQ: The skill set discretized using the VQ module. AE: The skill embedding is encoded based on the action sub-sequence rather than the goal.
Table 2. Ablation study on the skill imbalance challenge under Near-Optimal and Sub-Optimal settings in the MT50 setup.
Focal Loss
Resampling
Near-Optimal
Sub-Optimal
3159.1 =710
2123.8 2123.8_m8.. 78.1
V
3048.4 3048.4499.9
2092.7 _7100
RG
VQ
AE
Near-Optimal
Sub-Optimal
X
2962.3180.
2012.9_109.2
x
117.0
2123.8+78.1
X
/
X
2952. 19923
2041.5
X
X
2905. 2905.0492.7
1999.0
5.3. Ablation Study
X
/
/
1441.7
1947.92126.2
Ablation on Skill Model. Table I delineates our ablation study three key components of the skill model. First, incorporating the skill decoder with reached-goal history enables the agent to better assess its progress toward com- pleting the current skill. Second, we explore defining the skill space as a continuous space, where the skill-based policy directly outputs the skill embedding, rather than se- lecting skill index from the discrete space. The result shows that, similar to how humans summarize skills, using discrete sets of actions facilitates faster and more intuitive learning of skill utilization. Finally, to highlight the benefits of goal-oriented skills, we use continuous actions as input to the skill encoder and apply the skill quantization. The result demonstrates that goal-oriented skill significantly enhances the agent's 's ability to extract and master effective skills.
5.2. Main Results
Similar to MH-MTDT and MH-PromptDT, we also imple- ment a multi-headed variant GO-Skill. However, instead of the head division at the task level, MH-GO-Skill uses the multi-head architecture in the skill model, and the skill- based policy model still fully shares parameters.
As shown in Figure 5, GO-Skill demonstrates its effective- ness in offline MTRL settings, achieving superior perfor- mance compared to existing statc-of-the-art approaches. excels in handling both sub-optimal and near-optimal datasets. Specifically, it effectively extracts valuable infor- mation from sub-optimal trajectories and accurately emu- lates optimal behaviors when higher-quality data is avail- able. Notably, the advantages of GO-Skill become increas- ingly pronounced as the number of tasks increases. It is also worth noting that baselines typically perform worse on MT50 than on MT30 when comparing the different setups. In contrast, GO-Skill maintains comparable performance across both setups, particularly in the Near-Optimal case, where its performance improves with more tasks rather than fewer. This suggests that GO-Skill effectively leverages the richer shared knowledge to develop more robust skills as the task count grows. In addition, MH-GO-Skill using a multi-head skill model, mitigates gradient conflicts between different skills, further improving performance.
Ablation Skill Imbalance. Some skills are broadly ap- plicable across most tasks, while some are specific smaller subset of tasks, resulting in a skill class imbalance challenge. To address this, we employ a resampling strategy during the skill enhancement phase for the skill model. For the sbill-based policy, we utilize focal loss during training. Table presents the results of our ablation study on these two methods. The findings indicate that the skill model's learning is more significantly impacted by class imbalance, and the resampling strategy effectively enhances the agent's ability to master each skill.
Ablation on Syper-Parameters We conduct comprehen- sive ablation studies to establish an empirical strategy for

---

Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning
Skill in Dataset
Skill in Evaluation
MI.45 Near-Optimal
ML.5 Sub-Optimal
1.0
Episodo Return (xC10)
10
0.8
10
Task Index
0.6
10
25
Fine-TTmiie Tanengnnnng Iterations (x11)) Fine- Fine-Tuning Iterations (x100)
0.4
MTDT
PromptDT
GO-Skill
60
Figure 8, Fine-tuning on five new tasks under Near-Optimal and Sub-Optimal settings in the ML setup demonstrates that GO- Skill is highly effective for transfer scenarios.
0.0
Skill Index
Skill Index
this, the data associated with these unused skills can sup- port other tasks in leveraging those skills effectively. For instance, while the dataset for Task 2 (bin-picking-v2) con- tains limited data for Skill O, the agent successfully utilizes Skill to complete the task during evaluation. Further visu- alization of the learned skill set is provided in Appendix C.
Figure? 7. Visualization of skill distribution the dataset and eval- uation under the Near- Optrmmltiimal setting in the MT50 setup. This result reveals the imbalance in skill classes, where some skills are small subset.
widely used across tasks, while others are unique Notably, GO-Skill demonstrates the ability to learn optimal skill combinations from data containing sub- sub-optimal trajectorics.
5.5. Fine- Fine-Tuning on New Tasks
selecting four critical the ablation results under the Near-Optimal setting in the MT50 setup. Across a wide range of hyper-parameter se- lections, our approach consistently outperforms baseline methods. Among these, the skill horizon exerts the most significant impact performance. Short horizons fail to provide adequate temporal abstraction, while excessively long horizons increase the informational demands on the skill space, leading to slower convergence. The extraction training iterations also play a pivotal role. Too few iterations are insufficient to extract a valid set of skills, whereas too many iterations extend the total training time. Based on our findings, we recommend halting skill extraction once the skill divisions have largely converged and then proceeding to skill enhancement and skill-based policy learning. Our insights suggest the following recommended settings: skill horizon = 10, codebook size M = 16, code embedding size ||| = 64, and extraction iteration l, 3e4 (30% of total iteration). These bypee-parameter settings collectively contribute the superior performance.
Figure 6 illustrates
The MTRL, model is typically employed as a base model that requires minimal fine-tuning to adapt to new tasks, We utilize the ML45 benchmark in MetaWorld for evaluation. We first pre-train the model for 100,000 iterations on the original 45 tasks and then fine-tune it for 3,000 on 5 new tasks. Near-Optimal and Sub-Optimal represent the qual- ity of the pre- pre-training dataset, while the fine-tuning data consists exclusively of Near-Optimal expert trajectories. As shown in Figure 8, the results demonstrate that regardless of the quality of the pre-training dataset, GO-Skill effectively extracts valid set of skills that can be efficiently adapted to the new task set with minimal fine-tuning. Moreover, we find that MTDT outperforms PromptDT in fine-tuning scenarios, as PromptDT generally requires more training iterations to converge on new tasks.
6. Conclusion and Discussion
In this work, we propose Goal-Oriented Skill Abstraction (GO-Skill), novel approach inspired by the efficient knowl- edge abstraction observed in human learning. Leveraging the goal-oriented skill representation, we learn a skill model to extract reusable skills and develop sbill-based policy to utilize these skills effectively. To address the challenge of skill class imbalance, we introduce skill enhancement phase and incorporate focal loss into policy learning. Rigor- ous empirical evaluations on the offline MTRL. benchmark demonstrate that GO-Skill achieves superior performance.
5.4. Skill Abstraction Visualization
As shown in Figure 7, the left panel visualizes the distribu- tion of skills corresponding various tasks in the dataset. In contrast, the right panel visualizes the distribution em- ployed by the agent across different tasks during evaluation. Notably, the skill class imbalance exists within the dataset and during evaluation. In addition, each task in the dataset contains fragments of trajectories for multiple skills, but specific skills are not employed during evaluation. Despite
Limitations and Future Works. We present an innovative approach in offline MTRL, achieving state-of-the-ar per-
8

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Task Reinforcement Learning
formance across various tasks. However, its effectiveness relies the predefined skill horizon and the size of the skill set. Future research could explore methods for dynami- cally extending the skill set and adaptively determining the skill horizon to enhance flexibility and scalability. In addi- tion, investigating more robust goal-oriented representation emerges as another important direction for future endeavors.
forcement learning. In Proceedings of International Con-
ference on Learning Representations, pp- 1-11, 2020.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-trainie Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 4171 1111144166.00 2019.
Acknowledgements
Du, M., Nair, S., Sadigh, D., and Finn, C. Behavior re- trieval: Few-shot imitation learning by querying unla- beled datasets. CoRR, abs/2304.08742, 2023.
This work is supported in part by the Strategic Priority Re- search Program of the Chinese Academy of Sciences (Grant No. XDA0480200), the National Science and Technology Major Project (2022ZD01 16400)) the Natural Science Foun- dation of China (Grant Nos. 62222606 and 61902402), and the Key Research and Development Program of Jiangsu Province (Grant No. BE2023016).
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Proceed- ings of International Conference on Machine Learning, pp. 2052-2062, 2019.
Impact Statement
Ghosh, D., Ajay, A., Agrawal, P., and Levine, S. Offline rl policies should be trained to be adaptive. In Proceedings of International Conference on Machine Learning, pp. 7513-7530, 2022.
This paper presents work that aims to advance the field of Offline Multi-Task Reinforcement Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Guez, A., Vincent, R. D., Avoli, M., and Pineau, J. Adaptive treatment of epilepsy via batch-mode reinforcement learn- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1671-1678, 2008
References
Bellman, R. Dynamic programming. Science, 153(3731): 34-37, 1966.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- Off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In Proceedings of International Conference Machine Learning, pp. 1856-1865, 2018
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In Proceedings of European Conference on Computer Vision, pp. 213-229, 2020.
Caruana, R. Multitask learning. Machine Learning, 28: 41-75, 1997.
Hakhamaneshi, K., Zhao, R., Zhan, A., Abbeel, P., and Laskin, M. Hierarchical few-shot imitation with skill transition models. In Proceedings of International Con- ference on Learning Representations, pp. 1-12, 2021.
Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline rein- forcement learning via high- fidelity generative behavior modeling. In Proceedings of International Conference on Learning Representations, pp. 1-12, 2023.
He, H., Bai, C., Xu, K., Yang, 7.,, Zhang, W., Wang, D., Zhao, B., and Li, X. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. In Advances in Neural Information Processing Systems, pp. 64896 14896-64917, 2023.
Chen, I.,, Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P.. Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, pp. 1 33084-1509,, 2021.
He, Li, K., Zang. Y., Fu, H., Fu, Q. Xing, J., and Cheng, J. Efficient multi-task reinforcement learning with cross- task policy guidance. In Advances in Neural Information Processing Systems, pp. 117997 118024, 2024a.
Chen, 7., Badrinarayanan, V., Lee, C.-Y., and Rabinovich, A. Gradnorm:
Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of International Conference on Machine Learning, pp- 794 803, 2018
He, J., Li, K., Zang, Y., Fu, H., Fu, Q., Xing J., and Cheng, J. Not all tasks are equally difficult: Multi-task deep reinforcement learning with dynamic depth routing. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pp. 12376 12384, 2024b.
D'E Eramoo C., Tateo, D., Bonarini, A., Restelli, M.. and Peters, Sharing knowledge in multi-task deep rein-

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Task Reinforcement Learning
Hu, S., Fan, Z., Shen, L.,, Zhang. Y., Wang, Y., and Tao, D. Harmodt: Harmony multi-task decision transformer for offline reinforcement learning. In Proceedings of Inter- national Conference on Machine Learning, pp. 19182- 19197, 2024.
Lin, T.-Y,, Goyal, P., Girshick, R., He, K., and Dollar, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2999-3990200 3007, 2017.
Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict- averse gradient descent for multi-task learning. In Ad- vances in Neural Information Processing Systems, pp. 18878-18890, 2021.
Janner, M., Du, Y., Tenenbaum, B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. In Proceedings of International Conference Machine Learning, pp. 9902-9915, 2022.
Ma, Y. J., Kumar, V., Zhang, A., Bastani, O., and Jayaraman, D. Liv: language-image representations and rewards for robotic control. In Proceedings of International Confer- ence on Machine Learning, pp. 23301-23322 2023.
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Van- houcke, V., and Levine, S. Scalable deep reinforcement learning for vision-based robotic manipulation. In Pro- ceedings of Conference on Robot Learning, pp. 651-673, 2018.
Mnih, V., Kavukeuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fid- jeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 51897540):529-533. 2015.
Kalashnikov, D., Varley, J., Chebotar, Y., Swanson, B., Jon- schkowski, R., Finn, C., Levine, S., and Hausman, K. Mt- opt: Continuous multi-task robotic reinforcement learnin at scale. CoRR, abs/2104.08212, 2021.
Killian, T. W., Zhang, H., Subramanian, J., Fatemi, M., and Ghassemi, M. An empirical study of representation learning for reinforcement learning in healthcare. CoRR, abs/2011 1.12001..11200 1235, 2020.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: universal visual representation for robot manipu- lation. In Proceedings of Conference on Robot Learning, pp. 892-909, 2022.
Nasiriany, S., Gao, T., Mandlekar, A., and Zhu, Y. Learn- ing and retrieval from prior data for skill-based imitation learning. In Proceedings of Conference on Robot Learn- ing. pp. 2181-111221 2204, 2023.
Kostrikov, I,, Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. In Proceedings of In- ternational Conference on Learning Representations, pp. 1-11, 2021.
Pertsch, K., Lee, Y., and Lim, J. Accelerating reinforcement learning with learned skill priors. In Proceedings of Conference on Robot Learning, pp. 188-204, 2021.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative q-larnnin for offline reinforcement learning. In Advances in Neural Information Processing Systems. pp- 1179-1191, 2020.
Pertsch, K., Lee, Y., Wu, Y., and Lim, J. Demonstration- guided reinforcement learning with learned skills. In Proceedings of Conference on Robot Learning, pp. 729- 739, 2022.
Lee, K.-H., Nachum, O., Yang, M. S., Lee, L.,, Free- man, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., et al. Multi-game decision transformers. In Advances in Neural Information Processing Systems, pp. 27921-27932 2022.
Puterman, M. L. Markov Decision Processes: Discrete
Stochastic Dynamic Programming. John Wiley & Sonn, 2014.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to- end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-133, 2016.
Shu, T., Xiong, C., and Socher, R. Hierarchical and in- terpretable skill acquisition in multi-task reinforcement learning. In Proceedings of International Conference on Learning Representations, pp. 1-11, 2018.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. CoRR, (iis/2005.01444 2020.
Sodhani, S., Zhang, A., and Pineau, J. Multi-task rein- forcement learning with context-based representations. In Proceedings of International Conference on Machine Learning, pp. 9767-9779 2021.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In Proceedings of International Conference on Learning Representations, pp. 1-10, 2016.
Todorov, E., Erez, T., and Tassa, Y. MuJoCo: A physics engine for model-based control. In Proceedings of
10

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Reinforcement Learning
IEEE/RSJ International Conference Intelligent Robots and Systems, pp- 5026-5033, 2012.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, Atten- tion all you need. In Advances Neural Information Processing Systems, pp. 5998-6005 2017.
Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. In Proceedings of International Conference on Learning Representations, pp. 1-12, 2023.
Xu, M., Shen, Y., Zhang, S., Lu, Y., Zhao, D., Tenenbaum, J., and Gan, C. Prompting decision transformer for few-shot policy generalization. In Proceedings of In- ternational Conference on Machine Learning, pp. 24631- 24645, 2022.
Yamagata, T., Khalil, A., and Santos-Rodriguez, R. Q- learning decision transformer: Leveraging dynamic pro- gramming for conditional sequence modelling in offline rl. In Proceedings of International Conference on Machine Learning, pp. 38989 -90097, 2023.
Yang, R., Xu, H., Wu, Y., and Wang, X. Multi-task reinforce- ment learning with soft modularization. In Advances in Neural Information Processing Systems, pp. 4767-4777, 2020.
Ye. D., Liu, Z., Sun, M., Shi, B., Zhao, P., Wu, H., Yu, H., Yang, S., Wu, X., Guo, Q., Chen, Q., Yin, Y., Zhang, H., Shi, T., Wang, L., Fu, Q. Yang. W., and Huang, L. Mastering complex control in MOBA games with deep reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pp. 6672-6679, 2020.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, pp. 3824-5836, 2020a.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-Worlds A benchmark and evalu- ation for multi-task and meta reinforcement learning. In Proceedings of the Conference on Robot Learning. pp. 1094-1100, 2020b.

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Task Reinforcement Learning
A. Environment Details
The MetaWorld benchmark (Yu et al., 2020b) consists of a diverse set of 50 distinct manipulation tasks, all unified by shared dynamics. These tasks involve Sawyer robot manipulating a variety of objects, each with unique shapes, joints, and connectivity properties. In its original configuration, the source tasks are defined with fixed goals, which restrict the policy's ability to generalize across tasks of the same type but with varying goal conditions. To address this, extend these tasks to random-goal setting, similar to the approach taken in prior work (Yang et al., 2020; He et al., 2023; Hu et al., 2024), where both objects and goals are reset randomly at the start of each episode. We experiment with three different setups: (I) MT50, which includes the full set of 50 tasks, (2) MT30, a subset containing 30 operational tasks from MT50, and (3) ML45, where 45 tasks are used for pre-training and the remaining tasks are used for fine-tuning evaluation. A detailed breakdown of the skill sets provided Table As MetaWorld is an evolving benchmark, our experiments are conducted on the stable version, MetaWorld-V2
For the offline dataset, we follow the approach outlined by He al. (2023), using the Soft Actor-Critic (SAC) algo- rithm (Haarnoja et al., 2018) 8) to train a separate policy for each task until convergence. This generates a dataset consisting of 2 million transitions per task, drawn from the SAC replay buffer. These transitions represent the entire set of samples observed during training until the policy's performance stabilizes. We define two distinct dataset settings: (1) Near-Optimal, which includes the complete experience (100M transitions) from random to expert-level performance in SAC-Replay, and (2) Sub-Optimal, which contains the first 50% (50M transitions) of the near-optimal dataset for each task, where the proportion of expert-level data is significantly lower.
Table Detailed task set different MetaWorkd setups.
MetaWorld MT30
""basketballv2"" "bin "bin-picking-- v2", "button-press prattoo-press-tooowwrr topdown-v"", putton-prres--222
-uttoo--butt "coffee-pull" puileee2,lll-"",, "coffce-push-v2 "dial-turn-v2,, "disassemble-v2", "door-close-v2",
"door-lock-v2" sooor-ppe--v2,, "door-unlock" (noor-unlock-v2", "drawer-close-v2" "drawer-open-v2,, "faucet-close-v2",
Training tasks
"fauuet-open-v22, "hand-insed -aand-issert-v2", hhandle-reess-sdee
"handle-press-v2" phandle-pull-sieex2"""
"handlc-pull-v2", "lever-pull-v2", "peg-insert-side-222 "pick-outtof-holc-22" "pick-placc-wall--2" "push-back-v2" "push-v2", "reach-v2")
Meta World MT50
Training tasks
All tasks in MetaWorld ML45 including pre-training and fine-tuning.
Mera ML45
"assembly-v2" "basketball-v22,, "button--rese press-topdoon-v22 "button-press
pratton-pees-oopooowwwallll www.owowwwwl--222222
cuntonn-press-v2 v2" "button prep---p-sssss-s--- "cofffe- "coffcc-pull--2"" "coffec-push-v2",,
"dial-tum-v2", "disassemble-v2". "door-close-v2", "door-open-v2", "drawer-close-v2,, "drawer-open-x2",
"faucet-close-v2", "faucet-open-v2", "hammer-v2", prandseerees-siieevv2.,, "handle-press-v2".
Pre-training tasks
"handd--pull-side-2",,, "handle-pull-v2", "lever-pull-v2", "peg-insert-ssde-v2"" "rop
ppick-out-of-holev22 "pick"""pppppeeeeeee "pick-place-v2", "fick-place-wall-v2", "plate-slid-bbacc-siid---
siaie-slide-bac--222 polate-lidde-side-22 "wltt"""pl-eesideaaae "plate-slide-v2", "push-bbak-v22", "push-v2", wpllh-wall-v2",
"reach-v2", "reach-wall-v2", "shelf-place-v2", "soccer-v2", "stick-pull-v2". "stiek-push-v2", "sweep-into-v2",
"sweep-v2", "window-close-v2., "window-open-v2"]
Fine-tuning tasks
'"bin-picking-v2"" "box-close-v2", "door-lock-v2", "door-unlock-v2", "hand-insert-v2")
B. Implementation Details
We implement all experiments using the Prompt-DT (Xu et al., 2022) codebase and access the MotaWorld environment to this framework.
B.1. Details Computational Resources
We use NVIDIA Geforee RTX 3090 GPU for training and AMD EPYC 7742 64- 44-Core Processor for evaluation with the environments. The training duration for each model is typically takes around 9 hours in the MT50 setup and approximately 8 hours in the MT30 setup. Since each environment is trained five times with different seeds, the total training time is generally multiplied by five.
http://ttth...eoeeemmmmmmaaaaaee hhups://githu..co http://ggghuuccmm////uuupppppppp
12

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Task Reinforvement Learning
B.2. Details Hyper-Parameters
We present the common hyper-parameters in Table 4 and the additional hyper-parameters for GO-Skill in Table 5. Notably, the total number of iterations for the baselines is 1e5. GO-Skill first performs skill extraction using 3e4 iterations, followed by parallel iterations of skill enhancement and policy learning for 7c4.
Table Common hyper parametess configuration of GO-Skill and baselines.
Table 5. Additional hyper-parameters configuration of GO-Skill.
Hyper-parameter
Value
Value
Number of layers
6
Number attention heads
Skill horizon
Embedding dimension
256
Skill set (codebook) size
16
Nonlinearity function
ReLU
Goal/skill embedding dimension
64
Batch size
8 per task
Skill extraction iterations
3e4
Context length
20
Skill enhancement iterations
7c4
Prompt Length
10
Skill-based policy learning iterations
7e4
Dropout
0.1
Focusing parameter
2
learning rate
3e-4
Total iterations
le5
C. Additional Visualization
As shown in Figure 9, we visualize the set of skills learned by GO-Skill under the Near-Optimal setting in the MT50 setup. Across various tasks, GO-Skill effectively learns shared skill abstraction. For instance, in tasks requiring the grasping of items with unique shapes. joints, and connectivity properties, GO-Skill applies the same grabbing skill across all variations.
Skill: Upwards
acoembly-22
prese toplow-yy
baskethall-2
beskaball--22
sheli-paac--2
bin-picking-22
baadle-peos222
leva-pull-e2
box-close-v2
stick-pull-22
Gloe-slie-ksck--iiee
rog-itsel-iidde-d
swoep-12
2212
H/2
H/2
Skill: Insert Into Hole
Skill: Left
Figure 9, Visualization of the five skills in GO-Skill. The three images represent the states within the IT-horizon.
D. Single- Task Task Performance
We present the detailed performance evaluation of GO-Skill the MT50 setup in Section 5.2, with results for each task provided in Table 6.
13

---

Goal-Oriented Skill Abstraction for Offline Multi-Tass Reinforvement Learning
Table 6. Evaluated mean episode return of GO-Skill for each task in the MT50 setup. The model is trained for le5 iterations, with each run evaluated over 50 episodes. Reported values represent the mean and standard deviation across 5 different seeds.
Task Name
Near-Optimal
Sub-Optimal
assembly-v2
2366.7 1082.3
630.8420.1
basketball-v2
3634.54628.1
2774.4
bin-picking-v2
4137.9 2122.9
324. 2205.6
box-close-v2
3387.7+22u.
2177. 1177.1+436..
2787.5
1733.5 1733.5#2644.
putton-preesstooddoo-wllllll
2773.34
1728. 4:277.1
button-press-Ve
1338.51130.6
2112.9 2112.9 23346
button-press-wall--22
2358.31239.7
1962. =591.
coffee-button-v2
2939.4
795.0
coffce-pull-v2
1421.645100
113. 113.4436.6
coffee-push-v2
569. 119.2
220. 1
dial-turn-v2
2872.34202.7
1814. 1814.8g297.77
disassemble-v2
923 223.2::45.4
376. ::210
door-close-v2
4322.7
4218.9 4218.9
door-lock-v2
3428.441944.
2819.3
door-open-v2
1183.2
2177. 2177.4-214.1
door-unlock-v2
3722.3
3028. 3028.02398.
drawer-close-v2
4717.6164.2
4751.3
drawer-open-v2
2918.8+
2028.3
faucet-close-v2
1481.44124.7
4168.0 42203.
faucct-open-v2
4527 64.0
4570. 4570.8
hammer-v2
3188.71446..1
2610.9
hand-insert-y2
2515.94192.0
857.84281.
pansse--ress-sidee-22
1630.1
4103.8
handle-press-v2
4281.8+149.9
3533. 4
handle- pulld-eeuull-ssde-222
3715.4-447.5
1731.2=733.0
handle-pulle pulll-22
3451.34480.
2549.2 4883.
lever- lever-pull-v2
2819.01 2819.0+447.6
1424.6
peg- neg-insert-side-22
4096.64217.9
2391. 7791177 6442.8
peg-unplug-side-ve
3197.6 $197.644451.0
170.444-3..8
pock-out-of-holee
3044.04162..
2047. 14447.9
pick-place-v2
358.62400.1
19.3
pick-place-wall-vy
957. 957.6-767.7
5-1..44506
plate-slide-back-siin
4333.44210.4
3210.5
plate-slide slide-back-v?
4
3518. G
sate-slide-side--ee
2925. 179.994
1816. 6.6.4.422...
plate-slide-ve
4414.04977
3393.5 #340.6
push-back-v2
2186.2.4258.5
409.2
push-v2
1114.9
546.22 2:26..6
push-wall-v2
4093.3+193.0
2280. 6 2280.6:223.4
reach-v2
3510.04250.7
2404. 2
reach-wall-v2
4103.34202.5
3714.4+164.
shelf-place-v2
3532.00 3532.0+305.0 30s.0
1365. 83567.2
soccer-v2
1013.8+2114
967. 1077.6+:00 6
stick-pull-v2
3408. 4
3591. +309.9
stick-push-v2
3785.7
2265.6 6 464.5
sweep-into-v2
4030.7 7
3031. 3031.6-546..
sweep-v2
1100.44227..
3760.5 3760.8+208..
window-close-v2
2774.66.-496.
2174. 8
window-open-v2
2557. 2442.0
1206. 1206.641s0..
14


Reviews:
[{"review_id": "V28RctHlW2", "reviewer": "Reviewer_dNvS", "summary": "This paper proposes a method for offline multi-task reinforcement learning. The main idea is to an approach based on goal-oriented skill abstraction to better learn to extract and reuse skills to transfer to new tasks. Technically, the method utilizes vector quantization to form a discrete skill library. The idea is reasonable, and the paper shows improved performance.\n\n# update after rebuttal\n\nThe rebuttal helps address some of the concerns and I remain broadly positive with the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.\n\n> Technical components of the paper are quite standard, so better justification would be beneficial.\n\nWe appreciate your comment and acknowledge that certain technical components of our framework, such as vector quantization and transformer-based policy modeling, are standard techniques. However, these choices are made to **validate the effectiveness of our framework**, rather than serving as core contributions. Alternative implementations, such as diffusion-based policies or different discretization methods, could also be applied without altering the fundamental ideas of our approach.\n\nThe key contribution of GO-Skill lies in proposing **a novel skill-based MTRL framework** that enables **explicit knowledge sharing** through skill learning and composition, inspired by how humans generalize through modular skills. Furthermore, we introduce **Goal-Oriented Skill Abstraction**, a new method for defining and extracting reusable skills based on goal transitions rather than action sequences or rewards. This task-agnostic representation enhances skill transferability and enables effective skill discovery even from sub-optimal data.\n\nIn Section 5, we justify different components through an ablation study. The results show that while standard techniques like VQ contribute to performance improvements, Goal-Oriented Skill provides the most significant boost, highlighting its critical role. In addition, all ablated variants of GO-Skill outperform the baselines, demonstrating the effectiveness of our skill-based framework.\n\nWe hope this clarifies that our contributions are primarily conceptual and architectural, and that our use of standard components supports, rather than limits, the novelty of our approach.\n\n> The main comparison results shown in Fig. 5 are presented without actual numbers. The y-axis starting from 1.5 makes it difficult to judge the difference of different settings.\n\nWe appreciate your valuable feedback on Fig.5. In the revised manuscript, we will include a table presenting the exact numerical results corresponding to Fig.5 for clarity. Below is the table summarizing the results:\n\n| Method      | MT30 Near-Optimal | MT30 Sub-Optimal | MT50 Near-Optimal | MT50 Sub-Optimal |\n| ----------- | ----------------- | ---------------- | ----------------- | ---------------- |\n| MTDT        | \\$2342 \\pm 149\\$  | \\$1690 \\pm 64\\$  | \\$2290 \\pm 121\\$  | \\$1682 \\pm 116\\$ |\n| MH-MTDT     | \\$2434 \\pm 123\\$  | \\$1799 \\pm 69\\$  | \\$2356 \\pm 158\\$  | \\$1786 \\pm 84\\$  |\n| PromptDT    | \\$2497 \\pm 138\\$  | \\$1780 \\pm 86\\$  | \\$2315 \\pm 101\\$  | \\$1787 \\pm 105\\$ |\n| MH-PromptDT | \\$2587 \\pm 105\\$  | \\$1843 \\pm 103\\$ | \\$2478 \\pm 57\\$   | \\$1841 \\pm 35\\$  |\n| MTDIFF-P    | \\$2458 \\pm 109\\$  | \\$1756 \\pm 96\\$  | \\$2387 \\pm 91\\$   | \\$1781 \\pm 89\\$  |\n| HarmoDT     | \\$2874 \\pm 84\\$   | \\$2136 \\pm 68\\$ | \\$2745 \\pm 91\\$   | \\$1921 \\pm 104\\$ |\n| GO-Skill    | \\$3059 \\pm 70\\$   | \\$2168 \\pm 73\\$  | \\$3159 \\pm 71\\$   | \\$2124 \\pm 78\\$  |\n| MH-GO-Skill | \\$3096 \\pm 76\\$   | \\$2175 \\pm 67\\$  | \\$3181 \\pm 68\\$   | \\$2166 \\pm 57\\$  |\n\nThe reason for setting the y-axis starting from 1.5k is to better highlight the performance differences among different methods while maintaining a consistent y-scale across all four settings for easy cross-comparison. In addition, we would greatly appreciate any suggestions you may have for an improved visualization standard, and we will gladly incorporate the necessary adjustments in the revised manuscript accordingly.\n\n**Thank you once again for your valuable feedback. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.**", "claims_and_evidence": "The claims are supported with experimental evidence.", "methods_and_evaluation_criteria": "The methods are plausible and and the evaluation criteria follow the standard practice.", "theoretical_claims": "The paper does not make significant theoretical claims.", "experimental_designs_or_analyses": "The experimental and analyses are reasonable and appear correct.", "relation_to_broader_scientific_literature": "The proposed method shows improved performance compared with existing multi-task reinforcement learning methods on standard benchmark datasets.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\nThe method is plausible and well justified.\nThe paper shows improved performance on standard benchmark datasets.\n\nWeaknesses:\n\nTechnical components of the paper are quite standard, so better justification would be beneficial.", "other_comments_or_suggestions": "The main comparison results shown in Fig. 5 are presented without actual numbers. The y-axis starting from 1.5 makes it difficult to judge the difference of different settings.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JmA6zrF2vP", "reviewer": "Reviewer_BwoC", "summary": "This paper presents Goal-Oriented Skill Abstraction (GO-Skill), an approach aimed at enhancing knowledge transfer in offline multi-task reinforcement learning (MTRL). GO-Skill extracts reusable skills from task-mixed offline datasets through goal-oriented representations combined with vector quantization, creating a discrete skill library. To handle imbalances among skill categories, the method includes a refinement phase. These skills are then utilized via a hierarchical policy, demonstrating performance improvements on MetaWorld benchmark tasks compared to several baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.\n\n>Overlook offline MTRL baseline CDS.\n\nThank you for pointing out the omission of the CDS, which is indeed a significant contribution to offline MTRL. We will cite and discuss CDS more thoroughly in the revised manuscript.\n\nCDS focuses on a conservative data-sharing strategy, selectively utilizing data from other tasks to benefit learning on a given target task. In contrast, GO-Skill approaches **knowledge sharing at a higher level of abstraction** by discovering reusable skills that can be naturally composed across tasks. Moreover, these two methods could be integrated, where the high-level skill-based policy in GO-Skill can be enhanced by applying conservative filtering to determine which skill-based transitions to share across tasks as proposed in CDS, leading to more robust policy learning. We believe this hybrid direction holds promise and represents an exciting avenue for future work.\n\nUnfortunately, CDS’s codebase and experimental data are not publicly available, making a direct comparison difficult. Moreover, reimplementing such a method without official reference results for validation could risk an unfair or inaccurate evaluation. Nonetheless, we are currently emailing the authors to request the necessary resources and guidance for reproduction. If successful, we will experimentally explore the potential complementarity between CDS and GO-Skill, and we are excited to investigate the results in the final version.\n\n>Only a single environment setup is used — Metaword.\n\nBased on suggestions from other reviewers, we have conducted experiments on two additional benchmarks beyond MetaWorld. One is the locomotion task suite (*Cheetah-Vel* and *Ant-Dir*) used in PromptDT, and the other is *LIBERO*, a multi-task imitation learning benchmark.\n\nDue to space constraints, we kindly refer you to our response to **Reviewer FkAC** for detailed information. For convenience, we recommend using **Ctrl+F** to search for \"**LIBERO benchmark**\" to quickly locate the relevant response.\n\n>The method is quite complex and it is not clear whether all the components are necessary in general. I'd suspect for different MTRL task suites, not all the proposed contributions would be necessary.\n\nAs noted above, we also evaluate GO-Skill on additional benchmarks. The results demonstrate the general applicability of GO-Skill, supporting that it is not tailored to a specific environment.\n\nIn addition, our method is not conceptually complex. It consists of two main components: the skill model, which is shared across all tasks and generates actions to interact with the environment, and the skill-based policy, which completes specific tasks by composing different skills. In our ablation study, we systematically analyze both the skill model architecture and the methods for handling skill imbalance, demonstrating that each component contributes to overall performance.\n\n> Lacks success rate, which limits the comparability of the results.\n\nWe chose to report return as the primary evaluation measure in MetaWorld because it provides a more informative metric of task performance. For example, in the 'drawer-close' task, the environment considers the task successful once the drawer is closed past a certain threshold — even if it is not fully closed. However, the agent can continue to receive additional rewards for closing the drawer further, indicating a higher degree of task completion. In such cases, success rate acts as a binary threshold, while return reflects the quality and completeness of the agent’s behavior.\n\nIn addition, we conducted additional experiments and report the success rate of GO-Skill as follows:\n\n|MT30 Near-Optimal|MT30 Sub-Optimal|MT50 Near-Optimal|MT50 Sub-Optimal|\n|-|-|-|-|\n|$85.5\\pm1.0$|$62.8\\pm1.4$|$82.8\\pm1.0$|$58.2\\pm1.4$|\n\n> How is the hierarchical skill abstraction fundamentally different from previously proposed methods?\n\nOur **skill-based MTRL framework** offers a fundamentally different perspective from prior methods. Specifically, most prior multi-task RL methods achieve generalization through **implicit knowledge sharing** through various parameter-sharing strategies. In contrast, GO-Skill adopts an **explicit knowledge sharing** approach by directly extracting reusable skills from offline data. A key advantage of this explicit skill abstraction is its ability to better leverage suboptimal data. In offline settings, a suboptimal skill in one task may serve as an optimal skill for another, allowing GO-Skill to effectively master useful behaviors across tasks. This leads to more efficient skill acquisition and enhances generalization across diverse multi-task scenarios.\n\n**Thank you once again for your valuable feedback. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.**", "claims_and_evidence": "All the claims and architectural choices are well-motivated and validated by ablations.", "methods_and_evaluation_criteria": "The methods and evaluation metrics used are appropriate for offline multi-task reinforcement learning problems. Using the widely accepted MetaWorld benchmark provides valid grounds for comparison with baselines. However, results are reported using average return rather than success rate, which is the standard metric in this domain. I'd recommend including the success rate as the primary comparison metric.", "theoretical_claims": "The paper does not include explicit theoretical claims or proofs.", "experimental_designs_or_analyses": "Overall, the experimental setup is thorough, clearly comparing GO-Skill to several relevant baseline methods across multiple scenarios (MT30, MT50, near-optimal and sub-optimal datasets).\n\nHowever, an important baseline from existing literature, such as data-sharing methods described in \"Conservative Data Sharing for Multi-Task Offline Reinforcement Learning\" by Yu et al., is notably missing.\n\nFurthermore, the paper contains only one category of MTRL benchmark, Metaworld, which is not sufficient to justify all the components introduced in this paper.", "relation_to_broader_scientific_literature": "The authors effectively place their work within the existing offline RL and MTRL literature, building on prior approaches such as Decision Transformers, Prompt-DT, and vector quantization, clearly situating their contribution.", "essential_references_not_discussed": "The authors overlooked important related offline MTRL baselines, particularly the conservative data-sharing approach proposed by Yu et al., which is essential for the proper contextualization of their results.", "other_strengths_and_weaknesses": "## Strengths\n- Intuitive and well-motivated skill abstraction inspired by human learning.\n- Strong empirical evidence of improved performance against baselines and ablations.\n\n## Weaknesses:\n- The method is quite complex and it is not clear whether all the components are necessary in general. I'd suspect for different MTRL task suites, not all the proposed contributions would be necessary.\n- Lacks success rate, which limits the comparability of the results.\n- Only a single environment setup is used — Metaword.", "other_comments_or_suggestions": "More environments should be introduced, at least for the baseline comparison.", "questions_for_authors": "1. How is the hierarchical skill abstraction fundamentally different from previously proposed methods in offline RL and multi-task learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mn9SwNd0Qr", "reviewer": "Reviewer_FMhW", "summary": "This paper introduces GO-Skill, a novel hierarchical framework for offline multi-task reinforcement learning. The approach decouples learning into two components:\n- A low-level action predictor (\"skill-decoder transformer\") that generates actions based on a given skill prompt\n- A high-level skill predictor (\"skill-based decision transformer\") built upon the PromptDT architecture that selects appropriate skills to serve as prompts for the skill-decoder\n\nThe main innovation lies in the skill representation: embeddings that capture state variations over a fixed horizon $H$. The authors evaluate their approach on the MetaWorld benchmark, demonstrating consistent improvements over relevant baseline methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.\n\n> Perform on locomotion tasks \"Cheetah-vel\" and \"Ant-dir\"\n\nWe followed the experimental setup of PromptDT and conducted few-shot experiments on the *Cheetah-Vel* and *Ant-Dir* environments. The detailed results are as follows:\n\n| Method   | Cheetah-Vel        | Ant-Dir            |\n| -------- | ------------------ | ------------------ |\n| MTDT     | \\$-158.6 \\pm 9.8\\$ | \\$168.3 \\pm 20.7\\$ |\n| PromptDT | \\$-39.5 \\pm 4.2 \\$ | \\$398.9 \\pm 39.1\\$ |\n| GO-Skill | \\$-40.8 \\pm 3.6 \\$ | \\$411.2 \\pm 32.5\\$ |\n\nAs shown in the table, GO-Skill achieves comparable performance to PromptDT. This is expected because these environments are relatively simple, and the high similarity between different tasks causes both GO-Skill and PromptDT to reach the performance ceiling of these benchmarks.\n\nFurthermore, based on suggestions from other reviewers, we also conducted experiments on another multi-task imitation learning benchmark *LIBERO*. Due to space constraints, we kindly refer you to our response to **Reviewer FkAC** for detailed information. For convenience, we recommend using **Ctrl+F** to search for \"**LIBERO benchmark**\" to quickly locate the relevant response.\n\n> The trajectory-level state difference representation of skills may not capture all relevant aspects of useful skills.\n\nWe appreciate your thoughtful observation. Indeed, no single representation can capture all aspects of useful skills. GO-Skill adopts a goal-based representation to define skills in a task-agnostic manner. As shown in our ablation study in Section 5.3, this representation outperforms action-based alternatives. We believe this is because multiple action sequences can lead to the same state transition, and abstracting skills based on goals allows the agent to generalize across these variations, leading to more transferable and reusable skills.\n\nWe agree that goal-based representation may not always be optimal, but our framework is flexible and can accommodate alternative skill representations. We thank you for highlighting this direction, and we plan to further explore more expressive representations as part of our future work.\n\n> How GO-Skill could be extended to learn skills with different temporal horizons?\n\nThank you for raising this important point. While the current GO-Skill implementation adopts a fixed temporal horizon for simplicity and consistency, the framework itself is **fully compatible with variable-length skills**.\n\nTo support variable-length skills, the primary change would be in the **skill decoder**, which can be extended to learn a terminate signal to determine when to exit a skill. The key challenge lies in **how to extract and define variable-length skills** during the offline skill discovery phase. As mentioned in our Future Work Section, this is a central focus of our ongoing research. We are currently exploring approach by applying thresholds on state changes to identify skill. When the state change exceeds a certain threshold, it indicates the need to switch to a new skill.\n\nWe believe enabling dynamic skill lengths will further enhance the expressiveness and adaptability of GO-Skill, and we look forward to sharing these results in future work.\n\n> Add a reference to Appendix C in results section.\n\nThank you for your suggestion and for recognizing the value of the visualization results. We will add a reference to Appendix C in the results section in the revised manuscript.\n\n> \"get prompt\" appears outside the \"for each task\" loop in Alg.2.\n\nThank you for pointing out this mistake. This is indeed a writing error, and we will correct it in the revised manuscript.\n\nAdditionally, to clarify the prompt selection process: similar to PromptDT, we construct a Prompt Set for each task with the top-N highest return trajectories (N=4 in our experiments). Each time \"get prompt\" is called, a trajectory is randomly sampled from the corresponding task’s Prompt Set.\n\n> Goal encoder should be $\\mathcal{G}: \\mathcal{S} \\rightarrow \\mathcal{Z}$\n\nThank you very much for pointing out the error. We will correct it in the revised manuscript.\n\n**Thank you once again for your valuable feedback. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.**", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. The authors demonstrate GO-Skill's greater performance across multiple MetaWorld benchmarks (MT30, MT50) under different data conditions through relevant comparisons with previous methods. The ablation studies validate the contribution of each component (goal-oriented representation, vector quantization, skill enhancement). The paper provides interesting qualitative analysis, with visualizations of skill distributions proving method's ability to learn transferable skills, and examples of skills depicted in Appendix. The experimental methodology is sound: the paper uses multiple random seeds for statistical significance and provides detailed per-task results in the supplementary material.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem. The MetaWorld benchmark provides a diverse set of robotic manipulation tasks suitable for evaluating multi-task reinforcement learning. The authors evaluate their approach under different conditions (including varying numbers of tasks and dataset quality) and compare against relevant state-of-the-art approaches. The ablation studies effectively isolate the contributions of different components of the GO-Skill architecture.", "theoretical_claims": "Not Applicable (there are no theoretical claims).", "experimental_designs_or_analyses": "The experimental design is solid and well-organized. The authors test GO-Skill by comparing it with six different baseline approaches across four test conditions (MT30/MT50 combined with Near-Optimal/Sub-Optimal data). They also include several ablation studies that examine the importance of different parts of their system, such as how the skill model is designed, how they handle the problem of skill imbalance, and how different parameter settings affect performance. To help understand how their method works, they include helpful visualizations showing how skills are distributed and how well their system adapts to new tasks during fine-tuning. I also found the skill visualizations from Appendix C particularly interesting.", "relation_to_broader_scientific_literature": "The paper differentiates itself from previous methods that mainly work on low-level action imitation. Instead, this work puts emphasis on higher-level skill abstraction, which takes inspiration from how humans learn. This approach allows the system to identify reusable skills across different tasks.", "essential_references_not_discussed": "To the best of my knowledge, all the releva related works are cited and discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is well-structured and clearly written, making complex technical concepts accessible.\n- The approach has a strong conceptual motivation inspired by human learning.\n- The skill visualization provides intuitive understanding of what the model is learning.\n- The method appears to effectively leverage sub-optimal data by extracting useful skill fragments.\n\nWeaknesses:\n\n- The approach has only been evaluated on a single benchmark (MetaWorld), which limits understanding of its generalizability to other domains.\n- The trajectory-level state difference representation of skills may not capture all relevant aspects of useful skills.\n- The fixed skill horizon is a limitation that the authors acknowledge, as different skills naturally operate at different timescales.\n- The evaluation doesn't include locomotion tasks like \"Cheetah-vel\" and \"Ant-dir\" that were studied in related work like PromptDT.", "other_comments_or_suggestions": "Minor issues:\n\n- Line 132, 2nd column: Technically, the goal encoder should be a function from $\\mathcal{S}$ to $\\mathcal{Z}$ ($\\mathcal{S} \\rightarrow \\mathcal{Z}$).\n- In Algorithm 2: The \"get prompt\" line appears outside the \"for each task\" loop, which seems counterintuitive since prompts would typically be task-dependent.", "questions_for_authors": "1. In Algorithm 2, why is the \"get prompt\" line not included in the \"for each task\" loop? Since prompts are typically task-specific, this seems counterintuitive. Could you clarify how prompts are handled across different tasks?\n2. Have you considered how your approach could be extended to learn skills with different temporal horizons? Some manipulations (like grabbing) might occur over shorter horizons than others (like transporting).\n3. How might your approach perform on locomotion tasks like those studied in the PromptDT paper (e.g., \"Cheetah-vel\" and \"Ant-dir\")? These environments have different dynamics from manipulation tasks, which could test the generality of your skill abstraction method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6YAYIQNLAf", "reviewer": "Reviewer_FkAC", "summary": "The paper proposes Goal-Oriented Skill Abstraction (GO-Skill), a novel method for offline multi-task reinforcement learning (MTRL) that learns a unified policy to solve multiple tasks using pre-collected, task-mixed datasets. It introduces a skill extraction process using a goal encoder, vector quantization (VQ), and a skill decoder to build a discrete skill library from offline trajectories, followed by a skill enhancement phase to address class imbalance. A high-level skill-based policy, implemented as a Prompt Decision Transformer, dynamically combines these skills to tackle specific tasks. Main findings include superior performance on the MetaWorld benchmark (MT30 and MT50 setups), with average episode returns of 3159.1 (Near-Optimal) and 2123.8 (Sub-Optimal) on MT50, outperforming baselines like MTDT and PromptDT.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.\n\n> Skill Extraction need addtional training cost to mitigate the skill imbalance problems.\n\nAs detailed in Appendix B.2, we ensure that the total number of training iterations remains consistent across all methods. Specifically, the sum of stage 1 iterations (Skill Extraction) and stage 2 iterations (Skill Enhancement and Skill-Based Policy Learning) in GO-Skill is equal to the total iterations of the baselines. Importantly, since Skill Enhancement and Skill-Based Policy Learning are entirely independent learning processes, they are trained in parallel during the second stage of GO-Skill. As a result, no extra training time or iterations are introduced compared to the baselines. This design allows us to incorporate skill balancing mechanisms without increasing overall computational cost or compromising fairness in comparison.\n\n> Over-reliance on MetaWorld limits domain diversity, such as LIBERO.\n\nThank you for suggesting the LIBERO benchmark. However, we would like to clarify that LIBERO is fundamentally a multi-task imitation learning benchmark rather than a MTRL benchmark. The dataset only provides a limited number of expert demonstrations (50 trajectories per task) and lacks reward information (only a binary success signal). This setup does not fully capture one of the key advantages of GO-Skill: suboptimal trajectories from certain tasks can still facilitate skill learning and potentially benefit other tasks leveraging the same skill. We have analyzed this phenomenon in Section 5.4 of the main paper.\n\nNevertheless, we adapted our approach to the LIBERO setting by employing an imitation learning paradigm, removing the return-conditioning in DT-based models, and comparing GO-Skill, PromptDT, and MTDT under this framework. Additionally, since LIBERO is a vision-based environment, we utilized R3m\\[1] to encode image observations into vector representations. We use the LIBERO90 setting, and the experimental results are as follows:\n\n| Method   | LIBERO           |\n| -------- | ---------------- |\n| MTDT     | \\$9.9 \\pm 3.5\\$  |\n| PromptDT | \\$17.6 \\pm 3.7\\$ |\n| GO-Skill | \\$29.4 \\pm 4.0\\$ |\n\nFrom the experimental results, it can be seen that the explicit knowledge sharing provided by skill abstraction can also help agent to learn multiple tasks quickly in a multi-task imitation learning scenario.\n\nFurthermore, based on suggestions from other reviewers, we also conducted few-shot experiments on *Cheetah-Vel* and *Ant-Dir*, following the PromptDT setup. The results are shown below:\n\n| Method   | Cheetah-Vel        | Ant-Dir            |\n| -------- | ------------------ | ------------------ |\n| MTDT     | \\$-158.6 \\pm 9.8\\$ | \\$168.3 \\pm 20.7\\$ |\n| PromptDT | \\$-39.5 \\pm 4.2 \\$ | \\$398.9 \\pm 39.1\\$ |\n| GO-Skill | \\$-40.8 \\pm 3.6 \\$ | \\$411.2 \\pm 32.5\\$ |\n\nAs shown in the table, GO-Skill achieves comparable performance to PromptDT. This is expected because these environments are relatively simple, and the high similarity between different tasks causes both GO-Skill and PromptDT to reach the performance ceiling of these benchmarks.\n\n\\[1] Nair, Suraj, et al. R3m: a universal visual representation for robot manipulation. CoRL 2022.\n\n> What does \"dynamic transfer\" mean in Section 3.1.\n\n\"Dynamic transfer\" refers to the MDP’s transition dynamics, i.e., how the state evolves. The key distinction here is that this dynamic transition describes the **state change after H steps of decision-making**, rather than a single-step transition. If this phrasing has caused any misunderstanding, we will revise it to \"H-step dynamic transfer\" in the revised manuscript to clarify the meaning.\n\n**Thank you once again for your valuable feedback. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.**", "claims_and_evidence": "The claim that GO-Skill enhances knowledge transfer and task performance via reusable skills is well supported by experiments on MetaWorld MT30 and MT50, showing higher average episode returns.", "methods_and_evaluation_criteria": "The GO-Skill method, involving goal-oriented skill extraction and hierarchical policy learning, is appropriate for offline MTRL, leveraging offline data to abstract reusable skills.", "theoretical_claims": "The paper does not present formal theoretical claims or proofs. It includes serveral equations defining the loss objective to optimize the proposed model’s performance and basic formulation of model framework. As such, there are no proofs to verify.", "experimental_designs_or_analyses": "The experiment compare serveral Decision Transformer-style RL models in MTRL and demonstrates its effectiveness with statistical significant improvement. The qualitative result of ablation study is well-designed and clear.", "relation_to_broader_scientific_literature": "The paper omits discussion of \"LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning\" by Liu et al. (NeurIPS 2023), which is highly relevant to its contributions in knowledge-sharing for multi-task RL. LIBERO provides a benchmark for evaluating knowledge transfer—both declarative and procedural—in lifelong RL settings. \n\nGiven the paper’s focus on sharing knowledge across RL tasks, referencing LIBERO could contextualize its approach against a standardized benchmark. This omission counld limits the ability to assess how the proposed method compares to or builds upon established knowledge transfer frameworks in RL.", "essential_references_not_discussed": "To my knowledge, no essential references are missing. See the section \"Relation To Broader Scientific Literature\".", "other_strengths_and_weaknesses": "Strengths: \n1. the paper is well written and clear.\n2. Originality in goal-oriented skill abstraction, strong empirical results on MetaWorld, and clear visualizations (e.g., Figure 7). The skill enhancement phase addressing imbalance is a practical innovation.\n\nWeaknesses: \n1. Skill Extraction need addtional training cost to mitigate the skill imbalance problems.\n2. Over-reliance on MetaWorld limits domain diversity, such as LIBERO.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "1. What does “dynamic transfer” mean in Section 3.1.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission received 2 accepts and 2 weak-accepts, and all reviewers agreed that this paper presents a well-motivated and novel approach to offline multi-task reinforcement learning through Goal-Oriented Skill Abstraction. Strengths highlighted include strong empirical results, clear writing, and effective use of skill-based frameworks for knowledge transfer. While initial concerns were raised about reliance on standard techniques, limited benchmarks, missing baselines (CDS), and evaluation metrics, the authors provided thorough rebuttals, additional experiments (LIBERO, Cheetah-Vel, Ant-Dir), and clarifications that addressed most issues. All agreed that this paper demonstrates clear contributions and has been improved through the review process.
Decision: Accept (poster)