Skill Expansion and Composition in Parameter Space
Authors: Tenglong Liu
Jianxiong Li
Yinan Zheng
Haoyi Niu
Yixing Lan
Xin Xu
Xianyuan Zhan
Venue: ICLR 2025 Poster (2025)
Keywords: Skill expansion
skill composition
parameter-efficient finetuning
decision making
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A flexible framework that can efficiently reuse piror knoweledge to learn new tasks and progressively evolve during training.

Abstract: Humans excel at reusing prior knowledge to address new challenges and developing skills while solving problems. This paradigm becomes increasingly popular in the development of autonomous agents, as it develops systems that can self-evolve in response to new challenges like human beings. However, previous methods suffer from limited training efficiency when expanding new skills and fail to fully leverage prior knowledge to facilitate new task learning. We propose Parametric Skill Expansion and Composition (PSEC), a new framework designed to iteratively evolve the agents' capabilities and efficiently address new challenges by maintaining a manageable skill library. This library can progressively integrate skill primitives as plug-and-play Low-Rank Adaptation (LoRA) modules in parameter-efficient finetuning, facilitating efficient and flexible skill expansion. This structure also enables the direct skill compositions in parameter space by merging LoRA modules that encode different skills, leveraging shared information across skills to effectively program new skills. Based on this, we propose a context-aware modular to dynamically activate different skills to collaboratively handle new tasks. Empowering diverse applications including multi-objective composition, dynamics shift, and continual policy shift, the results on D4RL, DSRL benchmarks, and the DeepMind Control Suite show that PSEC exhibits superior capacity to leverage prior knowledge to efficiently tackle new challenges, as well as expand its skill libraries to evolve the capabilities. Project website: https://ltlhuuu.github.io/PSEC/.

Published as a conference paper at ICLR 2025
SKILL EXPANSION AND COMPOSITION IN PARAMETER SPACE
Tenglong Liu'> Jianxiong Li Yinan Zheng?, Haoyi Niu², Yixing Lan't Xin Xult,
Xianyuan Zhan 2,3,4t
1 National University of Defense Technology. 2 Tsinghua University,
Shanghai Artificial Intelligence Laboratory, Beijjng Academy of Artificial Intelligence
1t @@nudtt edu, en, jx2iixx2mmlllmm ttiinnnuuaa... eduiiuuaaedd..... edu. cD, [xinxu, lany xing 6) @nudt. edu. en, zhanxianyuan@aia tsinghua. edu. cn
ABSTRACT
Humans excel at reusing prior knowledge to address new challenges and devel- oping skills while solving problems. This paradigm becomes increasingly pop- ular in the development of autonomous agents, as it develops systems that can self-evolve response to new challenges like human beings. However, previ- ous methods suffer from limited training efficiency when expanding new skills and fail to fully leverage prior knowledge to facilitate new task learning. In this paper, we propose Parametric Skill Expansion and Composition (PSEC), new framework designed to iteratively evolve the agents' capabilities and efficiently address challenges by maintaining manageable skill library. This library can progressively integrate skill primitives as plug-and-play Low-Rank Adapta- tion (LoRA) modules in parameter-efficient finetuning, facilitating efficient and flexible skill expansion. This structure also enables the direct skill compositions in parameter space by merging LoRA modules that encode different skills, lever- aging shared information across skills to effectively program new skills. Based on this, we propose a context-aware module to dynamically activate different skills to collaboratively handle new tasks. Empowering diverse applications including multi-objective composition, dynamics shift, and continual policy shift, the results D4RL, DSRL benchmarks, and the DeepMind Control Suite show that PSEC exhibits superior capacity to leverage prior knowledge to efficiently tackle new challenges, well as expand its skill libraries evolve the capabilities. Project website: https:: / //tthhuuuuu gi github. io/PSEC/.
INTRODUCTION
Humans excel at using existing skills and knowledge to tackle new tasks efficiently, while continu- ally evolving their capabilities to rapidly adapt to new tasks. (Driscoll et al., 2024; Courellis et al., 2024; Eppe et al., 2022; Eichenbaum, 2017). This fundamental approach to problem-solving high- lights a key aspect of human intelligence that is equally crucial for autonomous agents. However, most current decision-make makiin algorithms adhere to tabula rasa paradigm, where they trained from scratch without utilizing any prior knowledge or resources (Akkaya et al., 2019; Berner et al., 2019; Silver et al., 2016), leading to severe sample inefficiency and elevated cost when the agent encounters new tasks (Agarwal et al., 2022; Peng et al., 2019; Du & Kaelbling, 2024). Therefore, in this paper, we aim to explore the capability of autonomous agents leverage and expand upon their existing knowledge base in novel situations to enhance learning efficiency and adaptability.
While some existing studies, such as continual learning (Liu al., 2024a; Gai & Wang, 2024), compositional policies (Peng et al., 2019; Janner et al., 2022; Ajay et al., 2023), or finetuning- based methods (Agarwal et al., 2022), aim replicate this process. they jointly failed tackle several key limitations. /) Catastrophic forgetting: these approaches typically lack a fundamental mechanism to guarantee continuous improvement when acquiring new skills, making the autonomous agents very susceptible to overfitting new tasks while forgetting previously learned skills without proper regularization (Liu et al., 2023c; 2024a; Gai & Wang, 2024); 2) Limited efficiency in learning new tasks: Some methods avoid the catastrophic forgetting problem by adopting parameter-isolation
"Equal contribution.
Corresponding Authors.

---

Published as a conference paper at ICLR 2025
Given task of
with limited data
Solve the new task
Offline RI.II
+
og
Skill library II
Skill Library
Offline RLAL
LoR
ad adeptatin
Each skill is encoded as distinct LoRA modulc
Add to the library
Adaptively composing different skills program new skills (b) Skill Composition
(a) Skill Expansion
Previous skills
Target skill
Stage3: run
Flat ground
Stage2: walk
Stagel: stand
Uneven ground
Multi-ohjective composition
Continual policy shift
Dynamics shift
(c) Diverse applications of PSEC framework
Figure PSEC framework its application diverse scenarios. (a) maintain skill library that contains many skills primitives and can progressively expand by adding new LoRA modules. (b) Then we train a context-aware compositional network to adaptively compose different elements in the skill library to solve new tasks. (c) PSEC framework is versatile diverse applications where reusing prior knowledge is crucial.
approach via encoding new skills in independent new parameters. However, they typically do not fully utilize prior knowledge from old skills to enhance training in current tasks, lacking efficient way to learn new skills terms of both parameters and training samples (Peng et al., 2019; Zhang et al., 2023a), leading tremendous costs the number of skills progressively grows.
In order to deal with the above problems, we propose Parametric Skill Expansion and Composi- tion (PSEC), framework that facilitates efficient self-evolution of autonomous agents by maintain- ing a skill library that progressively integrates new skills, facilitating rapid adaptation to evolving demands. The key mechanism of PSEC is utilize the primitives in the skill library tackle new challenges by exploiting the shared information across different skills within the parameter space. As shown in Figure 1 (a), we adopt the Low-Rank Adaptation (LoRA) (Hu al., 2021) approach, which encodes skills as trainable parameters injected into existing frozen layers. This parameter- isolation approach naturally resolves the catastrophic forgetting problem, and significantly reduces computational burden due to the low-rank decomposition structure. This efficient modular design allows for managing skills as plug-and-play modules, and thus can directly blend different abilities within the parameter space to interpolate new skills (Clark et al., 2024), as shown in Figure 1 (b). The proposed PSEC approach can leverage more shared or complementary structures across skills for optimal compositions. Based this insight, context aware module is designed to adaptively compose skills and each primitive is modeled by diffusion models to ensure both flexibility and ex- pressiveness in composition. Through iterative expansion and composition, PSEC continually evolve and efficiently tackle new tasks, offering one promising pathway for developing human-level autonomous agents.
Empowering diverse settings including multi-objective composition, continual policy shift and dy- namies shift, PSEC demonstrates its capacity to evolve and effectively solve new tasks by leveraging prior knowledge, evaluated on the D4RL (Fu et al., 2020). DSRL (Liu et al., 2023a) and DeepMind Control Suite (Tassa al., 2018), showcasing significant potential for real-world applications.
2 RELATED WORKS
Compositional Policies. Some previous methods try to leverage prior knowledge relying on pre- trained primitive policies. More specifically, these methods used compositional networks in hier- archical structure to adaptively compose primitives to form complex behaviors (Peng et al., 2019; Qureshi al., 2020; Pertsch et al., 2021; Merel et al., 2019; 2020). However, their expressiveness is limited by the expressiveness of simple Gaussian primitives. Recently, due to the strong expressive- ness of the diffusion models and its inherent connection with Energy-Based Models (LeCun et al.,
2

---

Published as a conference paper at ICLR 2025
2006), many compositional policies have been approached by diffusion model. Diffusion models learn the gradient fields of an implicit energy function, which can be combined at inference time to generalize to new complex distribution readily (Janner et al., 2022; Wang al., 2024b; Du & Kaelbling, 2024; Liu et al., 2022; Luo et al., 2024b). However, these approaches rely on indepen- dently trained policies with fixed combination weights, which lack the flexibility adapt complex scenarios. Moreover, most previous methods can only combine skills after the policy distribution generation of each skill. Therefore, they fail fully utilize the shared features of different skills to achieve optimal compasitions. We systematically investigate the advantages of skill composition within the parameter space, and compose skills in a context-aware manner with each skill mod- eled as a diffusion model. This ensures both flexibility and expressiveness in composing complex behaviors.
Continual Learning for Decision Making. Current continual learning methods for decision mak- ing, including continual reinforcement learning (RL.) and imitation learning (IL), primarily focus on mitigating catastrophic forgetting of prior knowledge when learning new tasks. They can be roughly classified into three categories: structure-based (Smith et al., 2023; Wang et al., 2024d), regularization-basee (Kessler et al., 2020), and rehearsal-based methods (Liu et al., 2024a; Peng et al., 2023). Different from previous continual RL and IL approaches, our study focuses on lever- aging existing skills to facilitate efficient new task learning and enables the extension of skill sets. In addition, it naturally solves the catastrophic forgetting challenge due to the parameter isolation induced by the LoRA module (Liu et al., 2023c), directly bypassing the key challenges of existing continual learning methods.
3 METHODS
We propose PSEC, a generic framework that can efficiently reuse prior knowledge and self-evolve to address emerging new tasks. Next, we will elaborate on our problem setup and technical details.
3.1 PRELIMINARY
Diffusion Model for Policy Modeling. Recently, diffusion models have become popular for policy modeling because of their superior expressiveness to model complex distributions (Wang et al., 2023; Chen et al., 2022; Lu et al., 2023; Zheng al., 2025). Considering a policy distribution x (a]s)) and sample (s, a) drawn from an empirical dataset D of [[ass), the diffusion process (Ho et al., 2020) progressively introduces Gaussian noise to the sample over T' steps, producing a sequence of noisy samples dn. a1..... dy with ag = a following the forward Gaussian kernel:
q(oolac-1) = =N(ai: v B, ((a))ao) ~N Vprao, (1 p.)
(1)
where P: : 3g, P: I
i= Pt. and the noise is controlled by variance schedule B,,..., Be to ensure p(ar) = (0, 1). The denoise process aims to recover the sample from p(ar) by learning a conditional distribution pe s)).s. The policy To(a) (als) is typically modeled as:
(a(s))) = p(a.) II po (o po: /a, s) t=1
1, 8), Eo(at,4,8))
(2)
where Eg = BeT is set untrained time-dependent constants and po(a,, f, s) = 1 (at P.
vp: VI-A co(ai, s)) is reparameterized by ca. The trainable parameter 8, modeled by deep networks, can be optimized via minimizing the following objective by predicting the noise:
((s.aa) a) || c- (VTia + V VI- pic,t, t, 8 )|}
Can (f) = E.-
(3)
where L/ is uniform distribution over the discrete set (1,..., T). we(s a) is a flexible weight function that encodes human preference (Zheng et al., 2024). For example. w(s, s, a) f(A(s,))), f 2 0 with A(s, a) as the advantage function leads to weighted behavior cloning (BC) in offline reinforce- ment learning (RL) (Zheng et al., 2024; Kostrikov et al., 2022; Xu al., 2023), and w(s, a) = degenerates to traditional BC (Chen et al., 2023). After obtaining the approximated (ig and Eg, we can substitute them into Eq. (2) to iteratively denoise and obtain actions conditioned on the state.
Problem Setups. We consider a Markov Decision Process with s E S and a E AA are state and action space, P S A (SS) is transition dynamics, and r S - IR is reward function. We assume
3

---

Published as a conference paper at ICLR 2025
Actions
Interpolation between different LoRA modules
Actions
AW
Pretrained Weights Wo
Pretrained Weights Wo
AW
AW
X IZ2
X CE2
LoRA Adapter weights
Inputs
Inputs
Adjusting the alpha value
(a) Learning new skills using LoRA modules.
(b) Interpolation in LoRA modules.
Figure 2: (a) Each skill is encoded separate LoRA modules respectively. (b) By adjusting the composing weights dG, different LoRA modules can merge together to interpolate new skills.
the state space S and action space A remain unchanged during training, which is mild assumption in many relevant works (Peng et al., 2019; Ajay et al., 2023; Nair et al., 2020; Liu et al., 2024b;
Dai et al., 2024). We consider an agent with To as its initial policy and then is progressively tasked with new tasks T, i = 1,2, 1,2,... ..., with differences in the rewards r or dynamics P, to mirror woaldworld scenarios with non-stationary dynamics or new challenges continually emerge (Luo et al., 2024a). Each task is provided with several expert demonstrations D := {(s. a)} or a mixed-quality dataset with reward labels DT: ;= a(s,a, r;, So, we can either offline RL or imitation learning (IL) (Gong et al., 2024b) to adapt to the new challenges (Liu et al., 2024b). Inspired by previous maintain a policy library II to store the policies associated with different tasks and aim to utilize the prior knowledge to enable efficient policy learning and gradually expand it to incorporate new abilities across training.
works (Peng et al., 2019; Barreto al., 2018; Zhang et al., 2023a),
II {5o, T1, T2, T.....
(4)
We aim to explore 1) Efficient Expansion: How to manage the skill library II to learn new skills in efficient and manageable way, and 2) Efficient Composition: How to fully utilize the prior knowledge from primitives in the skill set 11 to tackle the emerging challenges.
3.2 EFFICIENT POLICY EXPANSION VIA
ADAPTATION
For the first objective, previous methods typically train each primitive from scratch in a tabula rasa paradigm (Peng al., 2019; Janner et al., 2022; Lu et al., 2023), failed to leverage the prior knowl- edge in IT to efficiently obtain a good skill primitive. This presents significant issues in terms of computational efficiency when the number of skills grows. To mitigate these challenges, we turn to Parameter-Efficieet Fine- Sine-Tuning (PEFT) (Ding et al., 2023), which has proven highly effective in various natural language processing and computer vision applications. One of the most popular PEFT implementations is LoRA (Hu et al., 2022). It injects trainable low-rank decomposed matrices into the pretrained layer to avoid overfitting with limited adaptation data and significantly reduces computational and memory burden. Inspired by this, we try to employ LoRA to efficiently learn new skills given solely limited data for the target skill.
Policy Expansion via Low-Rank Adaptation. We consider a pretrained policy To and denote Wo E dd Xdou as its associated weight matrix. Directly finetuning Wo to adapt to new skills might be extremely inefficient (Liu et al., 2023c), instead, we introduce tune-able LoRA module AW upon Wn, Le., Wo + A W- = Wi+ BA to do the adaptation and keep Wo frozen, where B E Rd xn. A E Rudd and < min(di.: dout). Specifically, the input feature of the linear layer is denoted as Iain E BRdu, and the output feature of the linear layer is hous. E Rddus the final output of a LoRA augmented layer can be calculated through the following forward process:
+
Aont (Wo + aA W Wh (Wo + aBA.)/; = Wohn. + aBAh
(5)
where a is a weight to balance the pre-trained model and LoRA modules. This operation naturally
prevents catastrophic forgetting in parameter isolation approach, and the low-rank decomposi- tion structure of A and B significantly reduces the computational burden. Benefiting from this lightweight characteristic, to encode different skill primitives T, respectively, as shown in Figure 2a. This flexible approach allows us to easily integrate new skills based existing knowledge, while also facilitating library
manage numerous LoRA modules {A W, B,A, ii E 1,2, k}
4

---

Published as a conference paper at ICLR 2025
Actions
Actions
Action compose
Actions
Denoise steps
Denoise T steps
0g X 0g
+ a2 x
axt
0g X
Predicted noise
Noise compose
Denoise T steps
Predicted noise
Predicted noise e
Predicted noise C2
Predicted noise eq
Parameter compose
4
62 X
Inputs
Inputs
Inputs
(a) Paaaameterleee composition
(b) Noise-level composition
(c) Action-level composition
Figure 3: Comparison between parameter- noise-, action-level composition. Parameter level composition offers more flexibility to leverage the shared complementary structure across skills to compose new skills. Noise- and action-level composition, however, is too late to benefit from this information.
management by removing suboptimal primitives and retaining the effective ones. More importantly, by adjusting the value of it holds the potential to interpolate the pretrained skill in Wo and other primitives in AW; (Clark et al., 2024) to generate novel skills, as shown in Eq. (6) and Figure 2b.
W = Wo Σ a, Wo + E o,
where a; is the weight to interpolate pre-trained weights and LoRA modules. This interpolation property has been explored in fields like text-to-image generation (Clark et al., 2024) and language modeling (Zhang al., 2023b), but its application in decision-mann makiin scenarios remains highly un- derexplored, despite LoRA has proven efficacy in skill acquisition (Liu et al., 2023c). Next, we will elaborate on how to effectively combine LoRA modules to adapt to decision-making applications.
3.3 CONTEXT-AWARE COMPOSITION IN PARAMETER SPACE
Effectively combining skills encoded different LoRA modules to solve new tasks is crucial. Pre- vious methods (Du & Kaelbling, 2024; Ajay et al., 2023; Janner et al., 2022) typically rely on fixed combinations of skills, resulting in limited compositional flexibility. This approach may be accept- able in static domains like language models, but falls short in dynamic skill composition is crucial. For example, in autonomous driving, the ability dynamically prioritize skills obstacle avoidance in potential collision scenarios, or acceleration when speeds suboptimal, is essential. Naively adopting a fixed set of a, like previous approaches (Du & Kael- bling, 2024; Ajay et al., 2023; Janner et al., 2022; Clark et al., 2024), however, cannot adequately support such flexible deployment of skills based real-time environmental demands.
applications where
Context-aware Composition. We propose a simple yet effective context-aware composition method that adaptively leverages pretrained knowledge optimally address the encountering tasks according to the agent's current context. Specifically, we introduce a context-aware modular ass; 0) I with a; as its i-th dimension. The composition method can expressed by Eq. (7):
& w Wo Σ (s(s0))A W. + Σ a,
Here, ass;0) adaptively adjusts output weights based on the agent's current situation with the parameter 9 optimized via minimizing the diffusion loss in Eq. (3). Note that the trainable parameter 0 lies solely in the composition network ag with the pretrained weights W2 and all LoRA modules AW, being kept frozen, thus & can be efficiently trained in terms of both samples and parameters.
Parameter-level v.s. Action-level Composition. Careful readers may notice that our context-aware composition is similar to previous works that adaptively compose Gaussian primitive skills to create complex behaviors (Peng et al., 2019; Qureshi et al., 2020), such as the one shown in Eq. (8) (Peng al., 2019):
5

---

Published as a conference paper at ICLR 2025
Stand Walk
Run
75
Too noisy
1-SRE Dimension 2
C-SNE Dimansion 2
SNN Dimension
25
-25
-50
-100
-00
Shared knowledge
50
-75
-50
50
-75
No shared faiture preserved
-50
I-SNE Dimension
1-SNE Dimension
USNEE Dimension 1
(a) Parameter space
(b) Noise space
(c) Action space
Figure 4: I-SNE projections of samples from different skills in parameter, noise. and action space. The parameter space exhibits a good structure for skill composition, where skills share common knowledge while retaining their unique features to avoid confusion. Noise and action spaces are either too noisy to clearly distinguish between skills or fail to capture the shared structure across them. See Appendix C.4 for details.
क (a|s)
(()) II (=(as))au(e)))
(a(as)) = NN (1,(s),
where ass; 8) is optimized combine the policy distributions n; 0, -.... to collaboratively build a new policy distribution π to solve the new task.
However, these two methods differ fundamentally in their stages of composition, mirroring the ad- vantages of early fusion over late fusion across various domains (Gadzicki et al., 2020; Wang et al., 2024e). PSEC employs a parameter-level composition, where different skills seamlessly in- tegrated within the parameter space. By contrast. Eq. (8) represents an action-level composition that explicitly combines the output distributions of various skills. In comparison, parameter-level composition will be more efficient, as it can leverage more shared or complementary information between different skills to enhance compositionality and overall performance before generating the final policy distribution (Shazeer et al., 2016; Wang et al., 2024d). Conversely, action-level compo- sition only merges skills after the action generation, which is too late to effectively leverage features across skills for optimal composition. Besides, previous action-level methods typically employ sim- ple Gaussian primitives to construct their skill library, significantly limiting its expressiveness.
Parameter-level v.s. Noise-level Composition. Some approaches use diffusion models for policy modeling and exhibit remarkable compositionality by identifying its connections to Energy-Based Models (Du & Kaelbling, 2024; Wang et al., 2024b; Janner et al., 2022; Ajay et al., 2023; Lu al., 2023). Specifically, the noise predicted by diffusion models can be regarded as the gradient field of some energy functions. It thus can be directly merged form new skills during sampling in noise-level composition, as shown in Eq. (9). This is equivalent to doing a logical operation on the energy functions to form complex behaviors (Du et al., 2023; Liu al., 2022; LeCun et al., 2006).
taa,,t,8=
Σ (,
Here, 6; represents the predicted noise derived from various skills, while e is the aggregated noise re- sulting from their composition. Utilizing for denoising Eq. (2) allows for the generation of a joint distribution of skills, thereby facilitating the effective composition of these diverse capabilities (Ajay al., 2023; 2024; Janner et al., 2022). However, these methods employ fixed weights a, for policy composition, limiting their flexibility and adaptability in dynamical scenarios where real-time ad- justment on the compositional weights required. our paper, PSEC not only employs diffusion models to enhance the expressiveness of primitives, but also adaptively adjusts the context-aware compositional weights to enhance compositional flexibility. Additionally, this noise-level composi- tion also tends to be less effective than parameter-level composition, as the latter integrates different skills at an earlier stage, leading to improved performance, as shown in Figure 3.
Empirical Observations. To evaluate the advantages of parameter-level composition over other levels of composition, we employ t-SNE (Van der Maaten & Hinton, 2008) to project the output features of LoRA modules into 2D space, alongside the noise and generated actions of various skills. Figure 4 illustrates that in the parameter space, different skills not only share common knowl- edge, but also retain their unique features to avoid confusion. In contrast, noise and action spaces

---

Published as a conference paper at ICLR 2025
Table 1: Normalized DSRL (Liu et al., 2023a) benchmark results. Costs below indicates safety. t: the higher the better. J: the lower the better. Results are averaged over 20 evaluation episodes and 4 seeds. Bold: Safe agents with costs below 1. Blue: Safe agents achieving the highest reward.
BC
CDT
CPQ
COpáDICE
FISOR
ASEC
NSEC
PSEC
Task
reward "cost [reward fooost .reward feost |reward froot [reward foost Ireward frost [reward foost Ireward fcost
easysparse
0.32
4.73
0.05
0.10
-0.06
0.24
0.94
18.21
0.38
0.53
0.95
0.55
0.08
0.55
0.02
easymean
0.22
2.68
0.27
0.24
-0.06
0.24
0.74
14.81
0.38
0.25
0.63
0.75
0.39
0.54
0.37
0.00
easydense
0.20
1.70
0.43
2.31
-0.06
0.29
0.60
11.27
0.36
0.25
0.85
5,28
0.76
1.45
0.51
0.01
mediumsparse
0.53
1.74
0.26
2:20
-0.08
0.64
7,26
0.42
0.22
0.93
2.52
0.60
0.08
0.76
0.03
mediummean
0.66
2.94
0.28
2.13
-0.08
0.28
0.73
8.35
0.39
0.08
0.74
1.00
0.82
2.87
0.61
0.01
mediumdense
0.65
3.79
0.29
0.77
-0.08
0.20
0.91
9.52
0.49
0,44
0.81
0.52
0.76
0.27
0.66
0.02
hardsparse
0.28
1.98
0.17
0.47
-0.04
0.28
0.34
7.34
0.30
0.01
0.30
0.41
0.34
1.21
0.34
0.04
hardmean
0.34
3,76
0,28
3.32
-0.05
0.24
0,36
7,51
0.26
0.09
0.46
1.05
0.38
0.32
0.39
0.07
harddense
0.40
5.57
0.24
1.49
-0.04
0.24
0.42
8.11
0.30
0.34
0.36
0.82
0.19
0.03
0.34
0.07
MetaDrive
0.40
3.21
0.25
1,45
-0.06
0.24
0.63
10.26
0.36
0.25
0.67
2.02
0.53
0.76
0.50
0.03
Average
AntRun
0.73
11.73
0.70
1.88
0.00
0.00
0.62
3.64
0.45
0.03
0.74
4.97
0.79
6.81
0.59
0.33
BallRun
0.67
11.38
0.32
0.45
0.85
13.67
0.55
11.32
0.18
0.000
0.35
4.35
01.58
7.46
0.15
0.95
CarRun
0.96
1.88
0.99
1.10
1.06
10.49
0.92
0.00
0.73
0.14
0.93
0.39
0.93
0.66
0.83
0.00
DroneRun
0.55
5.21
0.58
0.30
7.95
13.77
0.30
0.57
2.29
0.62
7.3
0.47
0.87
AntCirlle
0.65
19.45
0.48
7.44
0.00
0.00
0.18
13:41
0.20
0.00
0.46
5.55
0.36
2.08
0.20
0.00
BallCircle
0.72
10.02
0.68
2.10
0.40
4.37
0.70
9.06
0.34
0.00
0.54
1,58
0.58
2,08
0.34
0.22
CarCirele
0.65
11.16
0.71
2.19
0.49
7.73
2.86
01.40
2.62
0.36
0.20
DroneCircle
0.82
13.78
0.55
1.29
-0.27
1.29
0,24
2.19
0.48
0.00
0.65
3.60
0.71
4.93
0.33
0.07
BulletGym
0.72
10.58
0.63
2.09
0.32
5.28
0.55
7.64
0.39
0.10
0.58
3.20
0.62
4.24
0.41
0.33
Average
are either too noisy to clearly distinguish between skills or fail to capture the shared structure across them, making the compositions in noise and action space less effective than the parameter space.
EXPERIMENTS
PSEC enjoys remarkable versatility across various scenarios since many problems can be resolved by reusing pre- pre-trained policies and gradually evolving its capabilities during training. Thus, we present a comprehensive evaluation across diverse scenarios, including multi-ohjective composition, policy learning under policy shifts and dynamics shifts, to answer the following questions:
Can the context-aware modular effectively compose diffcrent skills?
* Can our parameter-level composition outperform noise- and action-level compositions?
Can the introduction of LoRA modules enhance training and sample efficiency?
Can PSEC framework iteratively evolve after incorporating more skills?
4.1 MULTI-OBJECTIVE COMPOSITION
In many real-world applications, a complex task can be decomposed into simpler objectives, where collaboratively combining these atomic skills can tackle the complex task. In this setting, we aim evaluate the advantages of parameter-level composition over other levels of composition in Fig- ure 3, and the effectiveness of the context-aware modular. We consider one practical multi- multi-objective composition scenario within the safe offline RL domain (Zheng et al., 2024). This setting requires solving constrained MDP (Altman, 2021) to tackle complex trilogy objective: avoiding distribu- tional shift, maximizing rewards, and meanwhile minimizing costs. These objectives can conflict, thus requiring a nuanced composition to optimize performance effectively (Zheng et al., 2024).
Setup. We evaluate on a popular safe offline RL benchmark, DSRL (Liu et al., 2023a). We set w(s a) 1 in Eq. (3) to train our initial policy mo as a behavior policy. Then, we set w(s,a) (s,(AA a)) and a(s,a) = cxp(-AA (s. a)) with (s a) and A (s. a) are the optimal reward and feasible value function learned by expectile regression (Zheng al., 2024) to train gi and nh that separately consider reward and safety performance respectively. During composition, we adopt few filtered near-expert demonstrations that jointly consider the trilogy objective, which is too
7

---

Published as a conference paper at ICLR 2025
Straight Road
Turn Right
Straight Road
Obstacle
Obstacle
Obstaple
compositional weights
Reward policy
0.6
Context-aware
Safety Policy
0.4
100
200
300
400
500
600
700
Environment Step
Figure 5: Output weights of context- aware modular evaluated on the MetaDrivv-easyee easymean task. The network dynamically adjusts the weights handle real-time demands: It prioritizes safety policies when the vehicle while avoiding boundary lines. When there are no obstacles and the
approaches obstacles or navigates task is simply to drive straight, the focus shifts to maximizing rewards and maintaining some safety insurance.
limited to imitate good policies. However, we can adopt these data to train a context-aware modular axx00 in Eq. (7) to adaptively compose 7,,1,2 to handle the conflicts in an efficient way.
Baselines. To demonstrate the effectiveness of the composition in parameter space, we compare two other composition methods: noise-level and action-level composition. We denote them as NESC and ASEC respectively, where we control the only differences to PSEC being the composition stage as shown in Figure 3 to ensure fair comparison. We also compare recent state-of-the-ar (SOTA) safe offline RL methods including FISOR (Zheng et al., 2024), CDT (Liu et al., 2023b), COptiDICE (Lee al., 2022a), CPQ (Xu al., 2022) and BC. These traditional safe offline RL methods typically use human-tuned trade-offs to balance the trilogy objective, which is equivalent to using fixed compo- sition weights compared to PSEC. All policies are trained on the full DSRL dataset to ensure fair comparison (see Appendix C.1 for details).
Main Results. Table I shows that PSEC achieves good balance between high returns and satisfac- tory safety performance, and simultaneously mitigates distributional shift across all tasks, enjoying highly competitive performance. In contrast, NSEC and ASEC exhibit skewed learning behaviors, where both of them fail to discover an effective composition to ensure both good safety performance and high returns, resulting in relatively poor safety outcomes despite high rewards. PSEC also outperforms all traditional safe offline RL baselines, demonstrating the necessity of context-aware composition over fixed composition when the task requires intricate balance between different el- ements. To further support this, we visualize the outputs of our context-aware modular a(s; 0) to illustrate its adaptive capabilities. Figure 5 demonstrates that the network dynamically adjusts the weightings combine different skills, enabling collaborative response real-time environmental changes. This adaptive behavior highlights the importance of dynamically adjusting the composi- tional weights rather than relying on a fixed combination of different skills to jointly solve new task like previous methods (Ajay et al., 2023; Zheng et al., 2024; Janner et al., 2022).
4.2 CONTINUAL POLICY SHIFT SETTING
We evaluate another practical scenario where the agent is progressively tasked with new tasks. We aim to continuously expand the skill libraries to test if the capabilities of agents to learn new skills can be gradually enhanced as prior knowledge grows and test the efficiency of LoRA.
Setup. We conduct experiments on the DeepMind Control Suite (DMC) (Tassa et al., 2018) envi- ronments, where an agent progressively required to stand, walk, and run. We investigate whether PSEC can leverage the standing skill to rapidly learn to walk, and then effectively combine standing and walking skills to adapt to running. For this purpose, we pretrain mo to learn the basic standing expert dataset DTo Subsequently, we provide small expert datasets D for walk and DTa for run, while maintaining w(ss, a) ;= 1 to adapt to mi and π2- After training T1, we integrate it into the skill library II to assist n2 training alongside To- See Appendix C.2 for detailed experimental setups.
skill by setting w(s, a) := 1 in Eq. (3) trained
Baselines. 1) We compare NSEC and ASEC to further demonstrate the superiority of parameter- over noise- and action-level composition. 2) We evaluate training from scratch (denoted as Scratch), replacing LoRA modules with multiplayer perceptions (MIP) to demonstrate the efficiency of
8

---

Published as a conference paper at ICLR 2025
5 > with different data quandty ot W task
5 without compostion on W task
Perforance and wio Cantext-Aware
PSEC
PBBCWVPP
PSEC
780
Scores
NSEC
NSEC
500
Scroth ASED
ABEC
200
AGEO
250
NSED
PSBD
60
100
0k
24
4k
6x
10k
NSEC
Number Trajectories
Training Steps
(a) Sample efficiency.
(b) Training efficiency.
(c) Context-aware efficiency.
Figure 6: Comparisons on sample and training efficiency and the effectiveness of context-aware modular. S, W. R denote stand, walk and run, respectively. Each value is averaged over 10 episodes and 5 seeds.
compositions and LoRA module. 3) We evaluate different PSEC variants without context-aware modular (denoted w/o CA) to further highlight the crucial role of dynamically combining skills.
Training and sample efficiency. To demonstrate the training and sample efficiency of PSEC, we conduct extensive evaluations across varying numbers of trajectories and different methods. Fig- ure 6(a) shows that PSEC achieves superior sample efficiency across different training sample sizes, particularly when data is searce (e.g., only 10 trajectories). Figure 6(b) shows that PSEC can quickly attain excellent performance even without composition, highlighting the effectiveness of the LoRA modules. Hence, we train less than 50k gradient steps for almost all tasks, while previous methods typically require millions of gradient steps and data to obtain reasonable results.
Continual Evolution. Table shows that PSEC ef- fectively leverages prior knowledge to facilitate effi- cient policy learning given solcly limited data. Notably, S+W-R outperforms S-*R, demonstrating that the learning capability of PSEC gradually evolves as the skill library grows. In contrast, training from scratch or replacing the LoRA modules with MLP fails to learn new skills given limited data, highlighting the effec- tiveness of both utilizing prior knowledge and the intro- duction LoRA to efficiently adapt to new skills and self-evolution. Moreover, note that even PSEC (MLP) outperforms NSEC and ASEC, further highlighting the advantages of parameter-level compositions.
Results in policy shift setting. S, W, R denote stand, walk and run. 10 trajec- tories are provided for W and R tasks
Table 2:
S W
S-+
S+W ++
Scratch NSEC
58.9
25.5
25.5
320.9
38.5
39.4
PSEC (MLP)
424.1
143.3
194.5
PSBC
688
221
247
Context-aware Composition r.s. Fixed Composition. We carefully tune the fixed composition (w/o CA) of different skills during composition. However, Figure 6(c) shows that the context-aware modular can consistently outperform the fixed ones across different levels of compositions. This demonstrates the advantages of the context-aware composition network to fully leverage the prior knowledge in the skill library to enable efficient policy adaptations.
4.3 DYNAMICS SHIFT SETTING
We evaluate PSEC in another practical setting to further validate its versatility, where the dynamics shift encompass diverse scenarios such as coosddmmodd (O'Neill al., 2024), sim-to-real transfer (Tobin et al., 2017), and policy learning in non-stationary environments (Xue et al., 2024).
Setup. We evaluate on the D4RL environments (Fu et al., 2020). where we modify the dynamics and morphology of locomotive robots to reflect the dynamic changes. Specifically, we first pretrain To using dataset D. collected from a modified dynamics Po and then equip with new small dataset DP: collected under the original D4RL. dynamics Pi. Friction, Thigh Size and Gravity de- note Po modifies the friction condition, the thigh size of cheetah/walker, and the gravity respectively. Based on the new small dataset D.P, set w(s, function trained by expectile regression on DP1 (Kostrikov et al., 2022) to obtain a new policy mi and then optimize the context-aware composition network a(s; 0) combine No,1 to collaboratively work under dynamics Pt. See Appendix C.3 for details.
exp(A, (s, a)) with A; (s. a) as the advantage
Baselines. One branch of baselines consists in training Ti from scratch the small dataset DP: which may face data scarcity challenges, including BC, offline RL methods like CQL (Kumar et al., 2020), IQL (Kostrikov et al., 2022), MOPO (Yu et al., 2020c). In addition, evaluate some gen-

---

Published as a conference paper at ICLR 2025
halfcheetah-m
halfcheetah-mr
halfcheetah-mme
40
40-
Scores
20
20-
0
0-
Methods
Methods
Methods
walker2d-mm
walker2d-mr
walker2d-me
Scores
50
20
0-
Methods
Methods Joint train(Frictio))
Methods
CQL
DOGE TSRL
Joint train((ravvtt) PSEC(Finctio)
PSEC(Thigh)
JOL
(rain(Thigh)
PSEC(Gravity)
Results in the dynamics shift setting over episodes and seeds. -m, -mr and -me refer to D sampling from medium, medium-replay and medium-expert--2 data in D4RL (Fu et al., 2020), respectively.
Figure 7:
eralizable offline RL methods including DOGE (Li et al., 2023) and TSRL. (Cheng et al., 2023) that are superior in the small sample regimes. Additionally, we evaluate the policy trained on the com- bination of D.o and D referred to as Joint train, to show the advantages of the PSEC framework over brute-force method of combining all data to address dynamic gaps.
Main Results. Figure demonstrates that PSEC effectively utilizes transferable knowledge from the pretrained policy Ty to enhance performance under changed dynamics. In contrast, traditional offline RL methods perform poorly with limited data in new dynamic settings. Moreover, PSEC surpasses specialized sample-efficient offline RL methods like TSRL and DOGE, showcasing its superior ability to leverage prior knowledge for increased training efficiency.
4.4 ABLATION STUDY
W with different LoRA ranks
We primarily ablate different LoRA ranks to assess the robustness of our methods in continual policy shift setting. Figure 8 demonstrates that under varied LoRA n ranks, PSEC consistently outperforms the MLP variant across various LoRA ranks, demonstrating the superior bustness of LoRA modules. Among the different rank set- tings, observe that therefore chosen as the default choice for the experiments. We hypothesize that using rank greater than 8 degener- ates because the training data is quite limited (e.g., only 10 demonstrations).
700
Scores
600
500
400
gives the best results and is
16
Rank
Figure Ablations on LoRA ranks.
5 CONCLUSION
We propose PSEC, framework that handles different skills as plug-and-play LoRA modules within an expandable skill library. This flexible approach enables the agents to reuse prior knowledge for efficient new skill acquisition and progressively evolve in response to new challenges like humans. By exploiting the interpolation property of LoRA, we propose a context-aware composi- tional network that adaptively activates and blends different skills directly in the parameter space by merging the corresponding LoRA modules. This parameter-leve composition enables the exploita- tion of more shared and complementary information across different skills, allowing for optimal compositions that collaboratively generate complex behaviors in dynamical environments. PSEC demonstrates exceptional effectiveness across diverse practical applications, such as multi-objective composition, continual policy shift and dynamic shift settings. making it highly versatile for real- world scenarios where knowledge reuse and monotonic policy improvements are crucial. One limi- tation is the pretrained policy To may encompass diverse distributions to ensure good LoRA tuning. However, this can be mitigated by utilizing the broad coverage. More discussions limitations and future works be found in Appendix A.
dataset to enhance distribution
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work supported by National Key Research and Development Program of China under Grant (2022YF22502904), National Natural Science Foundation of China under Grant 62403483, Grant U24A20279, and Grant U21 and funding from Wuxi Research Institute of Applied Tech- nologies, Tsinghua University under Grant 20242001120
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Ad- vances neural information processing systems, 35:28955-2888 28971, 2022.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenen- baum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 36, 2024.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with 2019.
robot hand. arXiv preprint arXiv:
Ferran Alet, Maria Bauza, Alberto Rodriguez, Tomas Lozano-Perez, and Leslie P Kaelbling. Mod- ular meta-learning in abstract graph networks for combinatorial generalization. arXiv preprint arXiv.1812.07768, 2018.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowiez, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Brandon Araki, Xiao Li, Kiran Vodrahalli, Jonathan DeCastro, Micah Fry, and Daniela Rus. The logical options framework. In International Conference on Machine Learning, pp. 307-317. PMLR, 2021.
Gasser Auda and Mohamed Kamel. Modular neural network classifiers: A comparative study. Journal of Intelligent and robotic Systems, 21:117-129, 1998.
Gasser Auda and Mohamed Kamel. Modular neural networks: a survey. International journal of 1999.
neural systems,
Chenjia Bai, Lingxiao Wang, Jianye Hao. Zhuoran Yang. Bin Zhao, Zhen Wang, and Xuelong Li. Pessimistic value iteration for multi-task data sharing in offline reinforcement learning. Artificial Intelligence, 326:100 10444,, 2024.
Bart Bakker and Tom Heskes. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 4:83-99, 2003.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference Machine Learning, pp. 501-510. PMLR, 2018.
Jonathan Baxter. A bayesian/information theoretic model of sampling. Machine learning, 28:7-39, 1997.
learning to learn via multiple task
11

---

Published as a conference paper at ICLR 2025
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint
2019.
Caruana. Multitask learning: A knowledge sooseddgebbsed source of inductive biasl. Proceedings of the Tenth International Conference on Machine Learning, pp. 41-48, Citeseer, 1993.
Rich Caruana. Multitask learning. Machine learning, 28:41-75,, 1997.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high- Sigh-fideiity generative behavior modeling. In The Eleventh International Conference on Learning Reprexentations, 2023.
Jie Cheng. Ruixi Qiao, Gang Xiong, Qinghai Miao, Yingwei Ma, Binhua Li, Yongbin Li, and Yisheng Lv. Scaling offline model-based rl via jointly-optimized world-action model pretrain- ing. arXiv preprint arXiv:2410.000 2024a.
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei-Yue Wang- Rime: Robust preference-hased reinforcement learning with noisy preferences. arXiv preprint arXiv:2402.1740 17257 2024b.
Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Zhang. Shoucheng Song, Han Wang, Youfang Lin, and Li Jiang. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. In sevethh Conference on Neural Information Processing Systems, 2023.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, 2024.
Hristos S Courellis, Juri Minxha, Araceli R Cardenas, Daniel L Kimmel, Chrystal M Reed, Taufik A Valiante, C Daniel Salzman, Adam Mamelak, Stefano Fusi, and Ueli Rutishauser. Abstract representations emerge in human hippocampal neurons during inference. Nature, pp. 1-9, 2024.
Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, and Li Shen. Is mamba compatible with trajectory optimization in offline rein- forcement learning? arXiv preprint arXiv:2405. /2094, 2024.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235. 2023.
Laura N Driscoll, Krishna Shenoy, and David Sussillo. Flexible multitask computation recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, pp. 1-15, 2024.
Yilun Du and Leslie Pack Kaelbling. Position: Compositional generative modeling: A single model is not all you need. In Forty-first International Conference on Machine Learning, 2024.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. International con- ference machine learning, pp.
PMLR, 2023.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross- lingual parameter sharing in neural network parser. In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (volume 2: short papers), pp. 845-850) 2015.
Howard Eichenbaum. Prefrontal- Prefrontal-hippocampae interactions in episodic memory. Nature Reviews Neuroscience, 1809)5547-558, 2017.
12

---

Published as a conference paper at ICLR 2025
Manfred Eppe, Christian Gumbsch, Matthias Kerzel, Phuong DH Nguyen, Martin V Butz, and Stefan Wermter. Intelligent problem-solving as integrated hierarchical reinforcement learning. Nature Machine Intelligence, 4(1): 111):11-20, 2022.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representa-
fions.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 122--135, PMLR, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Konrad Gadzicki, Razich Khamschashari, and Christoph Zetzsche. Early vs late fusion in multi- modal convolutional neural networks. In 2020 IEEE 23rd international conference on information fusson (FUSION), pp. 1-6. IEEE, 2020.
Sibo Gai and Donglin Wang. Single-task continual offline reinforcement learning. arXiv preprint arXiv:2404. 12639, 2024.
Xudong Gong, Dawei Feng, Kele Xu, Bo Ding, and Huaimin Wang. Goal-conditioned on-policy inforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Xudong Gong, Dawei Feng, Kele Xu, Yuanzhao Zhai, Chengkang Yao, Weijia Wang, Bo Ding, and Huaimin Wang. Iterative regularized policy optimization with imperfect demonstrations. In Forty-first International Conference on Machine Learning, 2024b.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Repre- sentations, 2019. URL https: openrevieweeeeeeeeeeeeeeee www.//oruurreeee///
Eorum? 1d=HkxttoC577.
Philippe -sttruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint ar3iv:23304 10573, 2023.
Bart LM Happel and Jacob MJ Murre. Design and evolution of modular neural network architec- tures. Neural networks, 766-7)995--104, 1994.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06. 09685, 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Effi- cient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269 2023.
Kaixin Huang, Li Shen, Chen Zhao, Chun Yuan, and Dacheng Tao. Solving continual offline rein- forcement learning with decision transformer. arXiv preprint arXiv:2401.0844 2024.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9915. PMLR, 2022.
Samuel Kessler, Jack Parker-Holder Philip Ball, Stefan Zohren, and Stephen Roberts. Unclear: A straightforward method for continual reinforcement learning. In Proceedings of the 37th Interna- fional Conference on Machine Learning, 2020.
13

---

Published as a conference paper at ICLR 2025
Junsu Kim, Seohong Park, and Sergey Levine. Unsupervised-to-onlin reinforcement learning. arXiv preprint arXiv:2408. 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 69800, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):321-3352, 2017.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference Learning Representations, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Yixing Lan, Xin Xu, Qiang Fang. Yujun Zeng, Xinwang Liu, and Xianjian Zhang. Transfer rein- forcement learning via meta-knowledge extraction using auto-pruned decision trees. Knowledge- Based Systems, 242: 108221, 2022. doi: 1.1016/J.KNOSYE 2022. 108221.
Daniel Lawson and Ahmed H Qureshi. Merging decision transformers: Weight averaging for form- ing multi-takk policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12942-12948. IEEE, 2024.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, al. A tutorial on energy- based learning. Predicting structured data, 1(0), 2006.
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference Learning Representations, 2022a.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-toonlinn reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022b.
Borja G León, Murray Shanahan, and Francesco Belardinelli. In a nutshell, the human asked for this: Latent goals for following temporal specifications. arXiv preprint arXiv:2/10.0944 2021.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Siyuan Li. Rui Wang, Minxue Tang, and Chongjie Zhang. Hierarchical reinforcement learning with advantage-baset auxiliary rewards. Advances in Neural Information Processing Systems, 32, 2019.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:1 1878-18889, 2021.
Jinmei Liu, Wenbin Li, Xiangyu Yue, Shilin Zhang, Chunlin Chen, and Zhi Wang. Contin- ual offline reinforcement learning via diffusion-based dual generative replay. arXiv preprint arXiv:2404. 10662, 2024a.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp- 423-439. Springer, 2022.
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao. Wei Pan, and Xin Xu. Adaptive advantage saaatagg-guuded policy regularization for offline reinforcement learning. In International Conference on Machine Learning, volume 235, pp. 31406 31424. PMLR, 2024b.
14

---

Published as a conference paper at ICLR 2025
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wen- hao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint wrXiv:2306.99300 2023a.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao Constrained decision transformer for offline safe reinforcement learning. In International Con- ference on Machine Leurning, 2023b.
Zuxin Liu, Jesse Zhang, Kavosh Asadi, Yao Liu, Ding Zhao, Shoham Sabach, and Rasool Fakoor. Tail: Task-specific adapters for imitation learning with large pretrained models. arXiv preprint arXiv:2310.000 10.05055, 2023c.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact eeergyyguided diffusion sampling in offline reinforcement learning. In Inter- national Conference on Machine Learning. PMLR, 2023.
Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. OMPO:
A unified framework for RL under policy and dynamics shifts. In Forty-first International Conference on Machine Learning, 2024a. URL hww.h:ttt//scctc https:// net/ wwt.:neeeeootiiieeeeee forum?id- R83VIZtHXA.
Yunhao Luo, Chen Sun, Joshua B Tenenbaum, and Yilun Du. Potential based diffusion motion planning. In Forty-first International Conference on Machine Learning, 2024b.
Haozhe Ma, Zhengding Luo, Thanh Vinh Vo, Kuankuan Sima, and Yze-Yun Leong. Highly effi- cient self-adaptive reward shaping for reinforcement learning, arXiv preprint arXiv:2408.0300 2024a.
Haozhe Ma. Kuankuan Sima, Thanh Vinh Vo, Di Fu, and Tze- Leong, Reward shaping for reinforcement learning with an assistant reward agent. In Forty-first International Conference on Machine Learning, volume 235, pp. 33925-338333 33939. PMLR, 2024b.
Josh Merel, Leonard Hasenelever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations, 2019. URL https:: /oppereve///.....ew.... Eorum? 1d=3J1 6T RRCY7.
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch & carry: reusable neural controllers for vision- guided whole-body tasks. ACM Transactions on Graphics (TOG), 39(4:33--1 2020.
J Daniel Morrow and Pradeep K Khosla. Manipulation task primitives for composing robot skills. In Proceedings of International Conference on Robotics and Automation, volume 4, pp. 3354-3359. IEEE, 1997.
Devang K Naik and Richard J Mammone. Meta-neural networks that learn by learning. In /Proceed- ings ]992] IJCNN International Joint Conference on Neural Networks, volume 1, pp. 437-442. IEEE, 1992.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint N0X1:22000609359,, 2020.
Abby O' O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acor Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 6892 6903. IEEE, 2024.
Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Turner, and Mo- hammad Emtiyaz Khan. Continual deep learning by functional regularisation of memorable past. Advances in neural information processing systems, 33:4453-44466 2020.
Leo Pape, Faustino Gomez, Mark Ring, and Jürgen Schmidhuber. Modular deep belief networks that do not forget. In The 2011 l International joint conference on neural networks, pp. 1191-1198. IEEE, 2011. 1.
15

---

Published as a conference paper at ICLR 2025
Liangzu Peng, Paris Giampouras, and René Vidal. The ideal continual learner: An agent that never forgets. In International Conference on Machine Learning, pp. 27585-2222 27610. PMLR, 2023.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learning composable hierarchical control with multiplicative compositional policies. Advances in neural information processing systems, 32, 2019.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188-204. PMLR, 2021.
Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining parameter- efficient modules for task-level generalisation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 687-702, 2023.
Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks. arXiv preprint 1rXiv:2410.13025 2024.
Ahmed H Qureshi, Jacob J Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, and Michael C Yip. Composing task-agnostic policies with deep reinforcement learning. In International Con- ference on Learning Representations, 2020.
Allen Ren, Justin Lidard, Lars I. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simehowitz. Diffusion policy policy optimiza- tion. arXiv preprint arXiv:2409. 2409000888888 2024.
Mark Bishop Ring. Continual learning in reinforcement environments. The University of Texas at Austin, 1994.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
S Ruder. An overview multi-task learning in deep neural networks. arXiv preprint NEKV:1706.05090 2017.
Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, and Sepp Hochreiter. Learning to modulate pre-trained models in Advances in Neural Information Processing Sys- lems, 36, 2024.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: A scalable framework for contin- ual learning, In International conference machine learning, pp.
Yee Whye
PMLR, 2018.
AMANDA J C SHARKEY. On combining artificial neural nets. Connection science, 8(3-4):299- 314, 1996.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of layer. In International Conference on Learning Representations, 2016.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Julian Schritwieser,, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 55007676):354355,, 2017.
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization text-to-image diffusion with c-lora. arXiv preprint arXiv:2304.066/ 2023.
Shagun Sodhani, Mojtaba Faramarzi, Sanket Vaibhav Mehta, Pranshu Malviya, Mohamed Abdel- salam, Janarthanan Janarthanan, and Sarath Chandar. An introduction to lifelong supervised learning. arXiv preprint arXiv:2207.04354 2022.
16

---

Published as a conference paper at ICLR 2025
Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter-comee crramettr-compositionall multi-task reinforcement learning. Advances in Neural Information Processing Systems, 35: 21495-21507, 2022.
Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka, Efficient multi-task and transfer reinforcement learning with cetprumttee-comssttionnn framework. IEEE Robotics and Automation Letters, 888):4569 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: /801 00090, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911),, 2008.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaöl Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350-354 2019.
Guan Wang, Haoyi Niu, Jianxiong Li, Li Jiang, Jianming Hu, and Xianyuan Zhan. Are expressive models truly necessary for offline rl? arXiv preprint arXiv:2412. 12533, 2024a.
Lirui Wang, Jialiang Zhao, Yilun Du, Edward H Adelson, and Russ Tedrake. Poco: Policy compo- sition from and for heterogencous robot learning. arXiv preprint arXiv:2402.02511, 2024b.
Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get a family: State-adaptive balances for offline-to-oniee reinforce- ment learning. Advances in Neural Information Processing Systems, 36, 2024c.
Yixiao Wang, Yifei Zhang, Mingxiao Huo, Thomas Tian, Xiang Zhang. Yichen Xie, Chenfeng Xu, Pengliang Ji, Wei Zhan, Mingyu Ding, and Masayoshi Tomizuka. Sparse diffusion policy: A sparse, reusable, and flexible policy for robot learning. In 8th Annual Conference on Robot Learning, 2024d. URL https: openreview net/ for/foommmtt de
Zhe Wang, Siqi Fan, Xiaoliang Huo, Tongda Xu, Yan Wang, Jingjing Liu, Yilun Chen, and Ya-Qin Zhang. Emiff: Enhanced multi-scale image feature fusion for ihfile-infrasstucture cooperative 3d object detection. ICRA, 2024e.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Milos. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Informa- fion Processing Systems, 34:28496-28510, 2021.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8753-8760, 2022.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regulariza- tion. In The Eleventh International Conference on Learning Representations, 2023.
Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamics shift. Advances in neural information processing systems, 36, 2024.
17

---

Published as a conference paper at ICLR 2025
Ruihan Yang. Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances Neural Information Processing Systems, 53:4767-47777 2020.
Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv preprint arXiv:1606. 160.00460004 2016.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-588 5836, 2020a.
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-11 1100. PMLR, 2020b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information 2020c.
Processing Systems, 33:14129-144
Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems, 23(8):1177-1199) 2012.
Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline- tonolie reinforce- ment learning. In The Eleventh International Conference Learning Representations, 2023a.
Jinghan Zhang, Junteng Liu, Junxian He, al. Composing parameter-efficient modules with arith- metic operation. Advances in Neural Information Processing Systems, 16:11299-12610 2023b.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guider diffusion model. arXiv preprint 10X70:2401.10700, 2024.
Yinan Zheng, Ruiming Liang, Kexin Zheng, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. biffusion-based planning for au- tonomous driving with flexible guidance. In The Thirteenth International Conference Learning Representations, 2025. URL https::////....... http///pppe/eeeeeeee//eeeeeee/ee net forum? in wM2 FVgMDH.
Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arXiv preprint arXiv:2402. 16843, 2024.
18

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE WORKS
In this section, we provide detailed discussions about the limitations and their potential solutions.
Assumption on the expressiveness of the pretrain policy. The main limitation of PSEC is the assumption that the pre-trained mo covers a diverse distribution, which allows for efficient fine-tuning using small add-on LoRA modules. If this assumption does not hold, learning new skills through parameter-efficien nificantly more parameters might be required to acquire new skills.
tne-tuning may prove challenging, as sig-
Potential solutions: Note that this assumption is mild in relevant papers that utilize LoRA to learn new skills (Hu et al., 2021; Liu et al., 2023c). To tackle this problem, forward solution is to increase the value of LoRA ranks to increase the learning capabilities of the newly introduced modules. Another simple solution is to leverage the cheap and abundant out-of-domain data to enhance the distribution coverage of the pretrained To to enable efficient LoRA adaptations.
straight-
Redundant skill expansion. In this paper, PSEC includes policies for all tasks in the skill library across its lifelong time. Although we adopt LoRA to reduce computational burden and memory usage, maintaining an extensive library of skill primitives may still lead to substantial computational costs.
Potential solutions: Note that not all skills should be incorporated into the skill library. particularly those that are redundant and can be synthesized from other primitives. An interesting direction for future research is to develop an evaluation metric to assess the interconnections between different skills, such as the skill diversity (Pertsch et al., 2021; Eysenbach et al.), to only include essential, non-composable atomic primitives. Such a strategy could significantly reduce the management costs associated with maintaining the skill library.
Hyperparameter-tuning Another limitation is PSEC introduces another LoRA modules to learn new skills, which can introduce additional hyperparameters required to be tuned, Potential solutions: This limitation is widely existed in relevant works that try to reuse prior knowledge to learn new skills (Liu ct al., 2023c; Clark et al., 2024; Wang et al., 2024d; Peng et al., 2019; Barreto et al., 2018), since almost all papers require additional parameters or regularization to adapt to the new skills. In this paper, we have ablated the robustness of PSEC against varied LoRA ranks, and demonstrate consistent superiority
the naive MLP modules in Figure 8, highlighting the robustness of PSEC for hyperparameter tuning.
We employ a simple context-aware mod- ular a(s; 0) to dynamically combine different primitives. This operation is simple and may not fully leverage the shared structure across skills for the target task.
Simple context-aware compositional modular:
Potential Solutions: However, our paper, we have demonstrated the superior advantages of this simple context-aware modular, as shown in Figure 6c. One interesting future direc- tion is to adopt a more advanced model architecture, training objective, or more flexible gating approach to optimize the modular.
B DISCUSSIONS ON MORE RELATED WORKS
Tabula Rasa. Tabula rasa learning is one popular paradigm for diverse existing Decision-making applications. such as robotics and games (Silver et al., 2017; Andrychowicz et al., 2020; Berner et al., 2019; Gong et al., 2024a; Vinyals et al., 2019). It directly learns policies from scratch without the assistance of any prior knowledge. However, it suffers from notable drawbacks related to poor sample efficiency and constraints on the complexity of skills an agent can acquire (Agarwal et al., 2022).
Finetune-based Methods. Some finetune-based methods aim to accelerate policy learning by lever- aging prior knowledge. This knowledge may come from pretrained policy or offline data (Liu et al., 2024b; Wang et al., 2024a; Dai et al., 2024; Gong et al., 2024b; Cheng al., 2024a), such Offline- to-online RL (Nair et al., 2020; Lee et al., 2022b; Agarwal et al., 2022: Cheng et al., 2024b) and transfer RL (Barreto et al., 2018; Li et al., 2019; Lan et al., 2022). Some methods maintain policy
19

---

Published as a conference paper at ICLR 2025
library that contains pretrained policies and adaptively selects one policy from this assist pol- icy training (Kim et al., 2024; Wang et al., 2024c; Barreto al., 2018). However, they are generally restricted to single-task scenarios where all policies serve the same task (Zhang et al., 2023a), or only sequentially activate one policy in the pretrained sets, which greatly limits the expressiveness of the pretrained primitives (Li et al., 2019). Our method, the contrary, both leverage multi- task knowledge to fulfill the new task, and can simultaneously activate all skills to compose more complex behaviors.
MoE in decision making. The recent SDP (Wang et al., 2024d) is particularly relevant to our work. Specifically, SDP employs Mixture of Experts (MoE) (Shazeer et al., 2016) encode skills as flexible combinations of forward path gated by distinct routers, allowing for efficient adaptation new tasks by fine-tuning newly introduced expert tokens and task-specific routers. However, SDP necessitates that the pretrained policy To be modeled with MoR layers, which imposes additional requirements on the model architecture. In contrast, our approach does not impose any constraints on the structure of the pretrained network and allows for the direct incorporation of new skills plug- and-play LoRA modules. Moreover, when we identify a skill that is underperforming, we can easily modify the skill library by simply removing its blug-and-play LoRA modules. In contrast, using MoE limits this flexibility in managing different skills, making it challenging to mitigate the side effects caused by suboptimal skills. Therefore, PSEC offers the skill library, making it more feasible to scale up and incorporate a larger number of skills.
flexible approach managing
LoRA in decision making. Other relevant works such as TAIL (Liu al., 2023c), LoRA- DT (Huang et al., 2024) and L2M (Schmied et al., 2024) also employ LoRA to encode skills. However, they solely investigate the parameter-isolation property of LoRA to prevent catastrophic forgetting, while overlooking the potential to merge different LoRA modules to interpolate new skills. Moreover, TAIL only studies the domain, L2M and LoRA-DT only study the RL domain, while PSEC both explore the effectiveness in RL and IL settings.
LoRA for composition in other domains. (Ponti et al., 2023; Clark al., 2024; Huang et al., 2023; Zhong et al., 2024; Prabhakar et al., 2024) use LoRA for multi-task learning but using fixed combination of LoRA modules, focusing on static settings like language model image generation, thus limiting its expressiveness of the pretrained LoRA modules and flexibility of composition. In contrast, PSEC combines different LoRA via a context-aware modular, maximizing the expressive- ness of pretrained skills to flexibly compose skills, which is crucial for decision making since the real-time adjustment required to handle the dynamical problems as shown in Figure 5.
Task 2
New
Task
Task 2
Task
New Task
Specific Layers
Specific Layers
Shared Layers
Constrained Layers
(a) Hard Parameter Sharing
(b) Soft Parameter Sharing
New Task
State
Specific Layers (e.g. LoRA modules)
Shared Layers
(c) PSEC (Ours)
Figure 9: Illustrative comparisons between PSEC other modularized multitask learning frameworks when deployed to continual learning settings.
Modularized skills for multitask learning. Multitask learning methods attempt to leverage the complementary benefits and commonalities across different tasks to enhance the cross-task general- ization and capabilities (Wang al., 2024d; Yang al., 2020; Sun al., 2022; 2023; Ruder, 2017;
20

---

Published as a conference paper at ICLR 2025
Ma et al., 2024b;a). To achieve effective skill sharing, two primitive paradigms are introduced, in- cluding Hard Parameter Sharing and Soft Parameter Sharing (Ruder, 2017), as shown in Figure 9. All these methods demonstrate modularized structure, where separate parameters required to solve different tasks. Not only enjoying the benefits of multitask learning. this modularized design allows for efficient adaptation new tasks by exploiting the shareable knowledge stored in different modules (Happel & Murre, 1994; SHARKEY, 1996; Auda & Kamel, 1998; 1999; Sodhani et al., 2022; Andreas et al., 2016; Alet et al., 2018; Ponti et al., 2023; Clark et al., 2024; Huang al., 2023; Zhong et al., 2024; Prabhakar et al., 2024).
Hard parameter sharing approaches (Caruana, 1993; Sun et al., 2022; 2023; Baxter, 1997; León et al., 2021) aim to learn shared feature that is strong and generalizable enough to capture the commonalities across all different tasks. This is achieved by developing multi-head style structure, where different heads solve different tasks and all heads share some common layers (Sun et al,, 2022; 2023; León et al., 2021; Bakker & Heskes, 2003). In this structure, zero-shot generalization to new tasks becomes possible if the shared layers can capture some generic features, following the spirits of meta learning (Finn et al., 2017; Gordon et al., 2019; Naik & Mammone, 1992; Lan et al., 2022). PSEC can be regarded as one specific type of hard parameter sharing method since different LoRA modules exploit a shared Ty. However, note that each LoRA module in PSEC is sequentially and independently optimized, thus making it easier to capture the task-specific features and avoid the potential gradient conflicss across different skills (Yu et al., 2020a; Liu et al., 2021). Previous methods, however, may introduce some gradient conflicts across different tasks that impede policy learning (Sun et al., 2022; Yang et al., 2020; Caruana, 1997), or suffer from collapsing to an entropic state and fail to encode task-specific features (Ponti et al., 2023).
Soft parameter sharing approaches (Yang et al., 2020; Wang et al., 2024d; Liu et al., 2024a; Duong et al., 2015; Yang & Hospedales, 2016; Ruder, 2017) are similar to the hard ones, with the differ- ences primarily in the shared features. Instead of directly employing shared layers (Bakker & Hes- kes, 2003; Caruana, 1993; Sun et al., 2022; León et al., 2021), soft parameter sharing approaches adopt regularizations to enforce a "shared" feature across tasks, such as minimizing the 1.2 distance or cosine similarity across the features for different tasks (Duong et al., 2015; Yang & Hospedales, 2016; Ruder, 2017), adopting flexible structures like MoE layers (Shazeer et al., 2016; Yuksel et al., 2012; Wang et al., 2024d), soft modular (Yang et al., 2020), or resorting moving average across dif- ferent features (Liu et al., 2024a; Lawson & Qureshi, 2024). These methods enjoy more flexibility than hard parameter sharing but may suffer from potential instability caused by improper regular- izations and outlier tasks. For instance, Liu et al. (2024a); Lawson & Qureshi (2024) may undergo performance degradation without appropriate averaging weights if they are trying to combine a sub- optimal skill learned on limited data.
Modularized skills for continual learning and compositions. More critically, the modularized design naturally facilitates continual evolvement by absorbing new skills in new modules in a purameter-isolation manner (Sodhani et al., 2022). This is one key advantage of modularized skills over traditional continual learning approaches since methods like EWC (Kirkpatrick et al., 2017), Rehearsal (Rolnick et al., 2019), Functional Regularization (Pan et al., 2020) often exhibit some catastrophic forgetting. The modularization method, however, can address this problem fundamen- tally by learning new parameters without disrupting pretrained ones. Along this line, numerous works also adopt modularized structure in a hard or soft manner as we discussed carlier (Ring, 1994; Pape et al., 201 I; Huang et al., 2023; Andreas et al., 2016; Alet et al., 2018; Clark et al., 2024; Zhong et al., 2024; Prabhakar et al., 2024; Liu et al., 2024a) like PSEC. However, PSEC differs fun- damentally in three key axes, including how to obtain different modules, how compose modules, and where to compose modules.
How to obtain different modules? Many previous methods typically assume a fixed set of modules during pretraining and jointly train all modules at once following a multitask learning paradigm (Ring, 1994; Pape et al., 2011; Schwarz et al., 2018; Ponti et al., 2023; Alet et al., 2018). Although this joint training approach enjoys the potential to exploit more shared features across tasks. The learned modules may fail to capture task-specific features, becoming general-purpost features and collapsing to highly entropic status, if the data distribution is very diverse and many outlier tasks exist (Ruder, 2017: Ponti et al., 2023). In contrast, PSEC independently trains each LoRA by exploiting a shared, frozen, and general-purpose no, avoiding lots of conflicts across different tasks and avoiding the
21

---

Published as a conference paper at ICLR 2025
risks of collapsing (Yu et al., 2020a; Sun et al., 2022). We conduct empirical evaluations in our rebuttal to demonstrate this.
How compose modules? PSEC can iteratively expand its skill library to include more skills and then combine them to form complex ones, which is one common advantage of all modularized approaches. So, previous works can also iteratively expand their modules to encode new skills and then compose the pretrained ones to tackle new tasks, such as (Ring, 1994; Morrow Khosla, 1997; Pape et al., 2011; 1; Ponti et al., 2023; Alet et al., 2018; Huang et al., 2023; LeCun et al., 2006; Liu et al., 2022; Du et al., 2023). However, most previous works typically resort to simple fixed combination of different modules, such as manually tuned weights (Liu et al., 2022; Du et al., 2023), thus significantly limiting the flexibility to handle decision-making satisfy the dynamic demands. For instance, 2 = C% C. C +...+ + C skills could be composed of n different (non-redundant) skills by using binary compositional weight (0 for deactivate and for activate). So, naively adopting a fixed combination of different skills can be very suboptimal. In contrast, PSEC introduces a context-awe aware composition to dynamically combine different skills, greatly enhancing the expressiveness of the skill libraries by interpolating or extrapolating across different skills.
scenarios where real-time composition adjustment is required to
Where to compose modules? Another key problem that should be investigated is where to compose different modules. Directly in the original output space (noise space (Ren et al., 2024; Zhang et al., 2023b) or action space (Peng et al., 2019; Qureshi et al., 2020; LeCun et al., 2006) the parameter space (Huang et al., 2023; Prabhakar al., 2024; Pape al., 2011; Zhong et al., 2024). PSEC systematically investigates the advantages of skill compo- sitions in parameter space over the noise space and action space, offering clear guidance for future research to expand and compose skills in parameter spaces rather than noise/action spaces. Also, intuitively, Figure shows that PSEC holds the potential to exploit more complementary features commonalities across tasks than naive hard parameter sharing or soft parameter sharing. Specifically, PSEC can fully leverage information across tasks to facilitate new task learning by employing the compositional network to combine all avail- able parameters. Hard/Soft parameter sharing, however, must rely on well-performed shared feature produced by the shared layers while discarding all other heads (Liu et al., 2024a; Lawson & Qureshi, 2024).
Some works use logical options for skill composition (Araki et al., 2021) but require significant human effort for skill management, limiting scalability. Additionally, Araki et al. (2021) focuses on efficient pretraining, not on fast adaptation/comtinual improvement. In contrast, PSEC targets the later setups and minimizes human effort by incorporating new skills LoRA modules, which are then combined through auto-Icarned compositional networks.
C EXPERIMENTAL SETUPS
C.I MULTI-OBJECTIVE COMPOSITION
Training details of PSEC. In this setting, we have four networks required train: the behavior policy no. the safety policy Ti that minimizes the cost, the reward policy T2 that maximizes the re- turn, and the context-awand modular ass; 0) E IR? For each task, we first pretrain To parameterized by Wo as behavior policy by minimizing the following objective on the full DSRL dataset D (Liu al., 2023a) ensure diverse pretrained distribution coverage:
|| c- -
ew. (VDca+ + 11-pic )l
Emg (Wo) = EE..d,
(10)
Then, we equip the agent with the same dataset 'D but provide feasible label h and reward labels r, forming the dataset hh {(x, a, h, 8')} and Dr [(s, a, r, 8')) Then we train क and лı based on these datasets by optimizing their newly introduced LoRA modules AW, and AW2 via minimizing the following objectives in Eq. (11-12):
Lxy (AW1) = Bo.. w (s(),a)) <wi (Vra+ vT- - )1" (11)
22

---

Published as a conference paper at ICLR 2025
Lx2 (AW)) = - w0 [Td (s. (5,,))
<ew, ew, (Vbra + v1 - pie, )1/. (12)
where the weights of LoRA augmented layer are W = Wo - 16AW, and W2 = Wo + 16AW;
as defined in Eq. (5). w² (s. == (s, and w (x, == (xp(A,(s a)) are the weight- ing function derived from the optimal feasible value function A; a) = Q7 (s,a) (s) and reward value function (7 a) = (s, (s) optimized via expectile regression follow- ing (Kostrikov et al., 2022; Zheng et al., 2024), where Q7 a) and Vi2 (s) can be obtained via minimizing Eq. (13-14), (7, a) and (s) can be obtained via minimizing Eq. (15-16):
Lv. (e,a) f (Q) Vs)()))
(13)
Qq. (a (N) [[(1 ()(1)))((s) + maxx)((()).. v.(')) Qn (
(14)
Lv, (( [L"
(15)
Co. ((s,,,"')) (r+ (s(s))))
(16)
where (T(x)= - < 0)l u² and Liv (u) = - 1(u > 0)(2² with r E (0.5, 1). By doing so, TT and T2 become one safety policy that avoids unsafe outcomes and one reward policy that tries to maximize the cumulative returns, respectively.
Then, we train our context aware modular network (ss; 8) to combine N0,12 to collaboratively tackle the safe offline RL problem. We filter the Top-30 trajectories with the highest rewards and costs below 5 from the dataset D form small near-expert dataset 'D' that obtains a good balance among distributional shift, reward maximization and safety constraint. Then, we train a(s; 0) by minimizing the following imitation learning loss based on the D":
[(0) E Bu- [|e-ew EW (VTva VI pie, (,5) ).)).
(17)
where W Wo E (s O)AW, defined in Eq. (7)-
We train Ty for 1M gradient steps with a batch size of 2048 to ensure a good performance of mo- Then, we only train Ti and n2 for 50K gradient steps, for the efficiency of LoRA modules. For (s;; 8), we only train it for 1K gradient steps since all decomposed policies including To.1. are ready to be composed, which can significantly reduce the computational burden leveraging these pretrained policies. Summarized hyperparameters can be found in Table 9.
Baselines. For FISOR (Zheng et al., 2024), CDT (Liu al., 2023b), COptiDICE (Lee al., 2022a), CPQ (Xu et al., 2022) and BC. we adopt the results from FISOR (Zheng et al., 2024). For NSEC and ASEC results, we only change the compositional stages, and meanwhile keep all other training details the same to ensure fair comparison. Specifically, the context-aware modular for NSEC is trained via the following reparameterization method instead of the one in Eq. (12):
2
NSEC = co + Σ (s 9)c,
(18)
1=1
where 10,1, is generated from networks with layers of Wo. W1 = Wo + 16AW, and W₂ = Wo + 16AW,, respectively. We can see that the composition in Eq. (18) between skills happens in the noise space, and thus we denote it as NSEC (noise skill expansion and composition).
For ASEC, we directly compose the generated actions of different policies:
CASEC = Qg +
a4 (s; 8)a
(19)
where 4g,1,2 are the actions generated from the denoising process in Eq. (2) using the predicted noise 60,1.2 generated by networks with layers of Wg. W1 Wo + 16AW, and W2 Wo + 16AW2, respectively. The composition happens in action space, and thus we denote it as ASEC (action skill expansion and composition).
23

---

Published as a conference paper at ICLR 2025
C.2 CONTINUAL POLICY SHIFT
To evaluate PSEC's ability to continually evolving its capabilities when tackling new challenges, conduct experiments on DeepMind Control Suite (DMC) (Tassa al., 2018), where a walker agent is progressively required to stand, walk, and run, shown in Figure 11. We use three expert datasets including walker-stand D walker-walk D and walker-run D 2 released by Bai et al. (2024) for the policy learning. Specifically, D DD and D contains 1 1000, 10 and 10 trajectories, respectively. D and T contain only handful of data because we aim to test if the agent can leverage the knowledge from the standing skill to efficiently adapt to new tasks. We first pretrain To on the large D to obtain the basic standing policy via minimizing the following behavior cloning loss:
||| [|e-ewo (Vr.a + v1 - pie, L, 8) )1]
LTo (Wo) = Bo
(20)
Stand Walk (S-> W) task. Then, we can integrate the walking skill Ti into the skill library II by optimizing the following objective:
[|s- ew, (vria + vI pie,
t,e,t, t, s
L. (AW1) e
(21)
where Wi Wo 16AW,, Then, we train a context-aware modular a walk(s; (s;01) E IR to combine To and to jointly tackle the walking task:
[(11) --, N.. [|e-ewwoak (Vria + pee, )1]
v
(22)
where W Wallk- Wo + a walk (s; 01 AWW In this setting, we hope the final policy parameterized by Wwalk can outperform the naive policy that is trained from scratch on the small data D to demonstrate the significance of utilizing the prior knowledge in To for efficient task adaptation.
Stand (S R) task. Here, the adaptation for the running policy T2 is similar. We can replace W; in Eq. (21) with W Wo + 16AW, and D DTa to train n2 parameterized by AW2 Additionally, we replace Wwalk in Eq. (22) as Wran = Wo + qumm(s A2) and D as D to train a run (s; 02) to combine To and T22 generate the rumning skill.
Stand+Walk-1 Walk->Run (S+W-R) task. After obtaining To. Ti and T2, the composition for the run- ning skill becomes very simple. We can replace W Wwalk in Eq. (22) as W = Wo+ E (s(ss))AM) to train as 0) E IR2 to combine To,1, to generate the running skill. In this setup, we aim to prove that utilizing the library that contains To.1, (S+W+RR) can outperform To.2 (S-RR to show the learning capability of PSEC can gradually grow after incorporating more skill primitives.
We train To for 1M gradient steps with batch size of 1024 to ensure good performance of mn- Then, we only train πj and Ta for 10K gradient steps with 10 trajectories thanks to the efficiency of LoRA. For a wa f), a (u: (s: 0), (s; decomposed policies including T0,1, in the skill library are ready to be composed, which can sig- nificantly reduce the computational burden leveraging these pretrained policies. The summarized hyperparameters can be found in Table 10.
s; 8). we only train them for IK gradient steps since the
Baselines. We compare PSEC with other composition methods NSEC and ASEC, the Scratch method, and the variant PSEC (MLP). NSEC and ASEC train the context-aware modular repre- sented by Eq. (18) and Eq. (19), respectively. Seratch method means training policy from scratch by IDQL (Hansen-Estruss PSEC (MLP) replaces the LoRA matrices with the MLP network in PSEC.
et al., 2023), since we build our model based on the IDQL method.
Experimental setups for Figure 6. For Figure 6(a), evaluate the sample efficiency of PSEC framework. Specifically, we evaluate on the S- W task with different data quantities of the W dataset DTi, including 10, 30, 50, and 100 trajectories, trained with 10K, 30K, 50K, and 100K training steps, respectively. We compare PSEC with other baselines to demonstrate the sample efficiency of parameter-level composition over other composition methods.
24

---

Published as a conference paper at ICLR 2025
For Figure 6(b), we visualize the training curves of PSEC, PSEC (MLP) and Scratch for the S->W task trained solely on Eq. (21) without the composition in Eq. (22) to demonstrate the efficiency of LoRA modules over the naive MLPs and the efficiency leverage pretrain policies. In this setting, D contains 10 trajectories and we train each method for 10K training steps.
For Figure 6(c), w/o CA represents the compositional weight is tuned by humans, rather than auto-generated by our context-aware modular ag. We compare PSEC, NSEC, ASEC with their corresponding w/o CA variants further demonstrate the importance of dynamical compositions.
We conduct similar experiments on the S-R task and the results are presented in Figure 12. Note that the running skill is more difficult. PSEC shows marked superiority this challenging setting.
5 W with different data quaniily W task and or wo CA
1000
750
Scores
500
250
0
10
30 Number of Trajectories
50
100
ASEC w/o
NSEC we NSEC
CA
ASEC
PSEC
Scratch
Figure 10: Results in the policy shift setting. Each value is averaged over 10 episodes and 5 seeds.
Stand
Walk
Run
Figure 11: Continual cvolution on DeepMind Control Suite for Continual policy shift.
S-RR with difteeet dets quartity of taek
S Rl with without compastion on tank
Perfrmance and wi Condeci-Aware on taok
900
300
PSEOMLP)
PSBC
Nobd
200
(V2) NSCD CA
ASEC
100
100
0
10
100
2%
6x
1k
10k
PSEC
NSEC Meihodd
ASEC
Number Trajeclories
Training Steps
(a) Sample efficiency.
(b) Training efficiency.
(c) Context-aware efficiency,
Figure 12: Comparisons on sample and training efficiency and effectiveness of context-aware modular. S, denote stand, run, respectively. Each value is averaged over 10 episades and 5 seeds.
C.3 DYNAMIC SHIFT
To further validate the versatility of PSEC, we conduct experiments in a practical and common setting: dynamic shift. We conduct experiments on the D4RL benchmark, where we modify the dynamics and morphology of locomotive robots to reflect the dynamics changes. Our goal is to leverage the policies based on the source datasets DDo and small amount of the target datasets DP. to adapt to the target task quickly. Specifically, the datasets DDo contain 20K transitions with 3 types of dynamic modifications on Po: 1) Friction: the friction coefficient the robot modi- fied; 2) Gravity: the gravity acceleration in the simulation environment is changed. 3) Thigh: the thigh is enlarged to double its original size to produce a morphology gap on the embodiment. The target datasets DPi are sampled from the D4RL benchmark with un-modified dynamics P1. includ- ing types: halcheetah- medium-v2 v2, valfcheetab-mmediu--roo medium halfcheetah-meedee -expertt-eeee-------e- shown in
malcheetah-meedim-- v2, walker2d-medium-22 walker2d-mediiu--rppaa
walker2d-medium-eveeee
25

---

Published as a conference paper at ICLR 2025
Figure 13. Each dataset type of D contains solely 10K transitions, which are too limited train good policies directly on the target dynamics P1 from scratch.
We first pretrain To with dataset D for 20k training steps by behavior cloning via minimizing the following objectives:
(C-- cwo (VTia + (CC-CooV mmo (Wo) = ( B.o. (
v1- Pic, t,
(23)
Then, we try to use the limited Pi to adapt mo to the target domain PSEC uses LoRA to train new policy mı with the pretrained source policy wo by minimizing the following objectives:
(AWI) ( w' (s, ew; (Vžna + V1- - )|] (24)
where Wi Wo + 16AW, Finally, PSEC the context-aware modular a<s;; 0) to integrate policy Tu, using the target dataset D to transfer to the target dynamics P1. The context-aware modular a(s 8) is trained for only Ik training steps by minimizing the following objectives:
[|e- (Vøja + VI (l-pictt, )){]
[ = E B...
(25)
where W = Wo + a(s; ()s; as defined in Eq. (7).
We train Ty for 1M gradient steps with batch size of 1024 to ensure a good performance of Tu- Then, we only train Ti for 20k gradient steps, for the efficiency of LoRA modules. For a(s; 0), we only train for 1K gradients steps since decomposed policies including T0,1 are ready to be composed, which can efficiently adapt to the target domain leveraging the pretrained source policies. Summarized hyperparameters can found Table 11.
Baselines. We compare PSEC with other methods in dynamic shift settings, including behavioral cloning (BC), offline RL approaches like CQL (Kumar et al., 2020), IQL (Kostrikov et al., 2022), and model-based methods such MOPO (Yu et al., 2020c). Additionally, eralizable offline RL methods, specifically DOGE (Li et al., 2023) and TSRL (Cheng et al., 2023), which have demonstrated superiority in small sample regimes. The baseline results for comparison are sourced from the TSRL. paper (Cheng et al., 2023), which reports state-of-the-a performance in these regimes. Furthermore, we assess policies trained on combinations the offline datasets DP. and D under various dynamics settings, referred to as Joint train (Gravity), Joint train (Friction), and Joint train (Thigh). These combinations involve training with one source dataset under dy- namic shifts (e.g., changes in gravity, friction, thigh size) and target datasets such as halfcheetah- medium-v2, halfcheetah-meddeee mediim-rrpllaa walker2d med-immmmuu-uuul--222222 and walker2d--medin-exppee medium--xpeer--te v2, 2iikkrrd-- In order to maintain fairness, the joint train method trained the same way as PSEC is trained on the source datasets. The results and training curves of PSEC across these settings presented in Table 8 and Figure 20, respec- tively. These comparisons showcase the effectiveness of PSEC under dynamic shifts and small sample conditions.
gen- evaluate
C.4 T-SNE EXPERIMENTAL SETUPS FOR FIGURE 4
To provide empirical support of the advantages of parameter-level composition over other levels of composition, we visualize the t-SNE (Van der Maaten & Hinton, 2008) projection of data samples in different spaces. Specifically, for each dataset DTo _ in the continual policy shift setting in Section C.2, we randomly sample 512 data samples (s, a). which forms three types of data that encode the standing, walking and running skill, respectively. In the action space, we directly utilize t-SNE projection to map these sampled data into a 2-dimentional space in Figure 4 (c). For the noise space, add 1 step of noise the sampled actions following the forward diffusion process in Eq. (1) and get the tuple (s, ai) for different skills. Then, we generate the noise based on this noisy tuples and visualize their t-SNE projections in Figure (b). In parameter-space, feed the noisy
26

---

Published as a conference paper at ICLR 2025
Source domain
Source domain
Halfcheetah Task
Walker2d Task
Friction
Friction
Target domain
Target domain
Thggh
Figure 13: The illustration of the source and target domains for the dynamic shift setting.
tuples (8, a1) into the trained networks and get the output features the middle LoRA augmented layers. Then, we project these features using t-SNE in Figure 4 (a).
D MORE EXPERIMENTAL RESULTS
Seasting stil
Stalling
Fall
Comportoooal weggls
1.5
1.0
1001
Skggs
Skag
(a) Fixed composition on R task
(b) PSEC on S
Figure 14: Outputs weights of the context-awart modular DeepMind Control
D.1 THE EFFECTIVENESS OF THE CONTEXT-AWARE MODULAR
Context-aware modular for the continual policy shift. To further explore the effectiveness of the context-aware module, employ analyze the trajectories generated by policies composed using fixed compositional weights. Specifically, for the S-R task in Section C.2, the fixed com- compose mo and
position method denote Wran Wo + aAW2, which uses fixed a Figure 14 (a) shows that naively using fixed compositional weights might accidentally stuck in some local suboptimal behavior such as standing still or falling down. We can clearly observe that our context-aware modular provides corresponding responses to correct these undesired behaviors. Therefore, it is necessary to adjust the weights of different strategies to fit the current states. Fig- 14 (b) presents the trajectories generated by PSEC. It clearly demonstrates that by utilizing the context-aware modular, the agent can make subtle adjustments between skills and stably run across the entire episodes.
D.2 THE PARAMETER EFFICIENCY OF PSEC
Parameter efficiency. To evaluate the parameter efficiency of PSEC, we compare its parameter count and performance various tasks against both the Scratch method and PSEC (MLP), The parameter count for PSEC includes the LoRA parameters and context-aware parameters specific to the walker-walk walker-run tasks. The Scratch method represents training the policy from scratch with standard MLP. PSEC (MLP), which substitutes the LoRA weights with standard MLP and retains the context-aware modular, has higher parameter count than the Scratch method. The
27

---

Published as a conference paper at ICLR 2025
parameter counts illustrated in Figure 15. In terms of performance, the results from the Deep- Mind Control Suite (DMC) tasks, as shown in Figures (b) and 12 (b), indicate that PSEC achieves significantly better performance despite having only 7.58% of the parameters used in the Scratch method. This performance advantage over both the Scratch method and PSEC (MLP) demonstrates that PSEC possesses strong parameter efficiency, effectively leveraging smaller number of param- eters for superior task performance. In this way, PSEC can leverage and expand upon its existing knowledge base in novel situations enhance learning efficiency and adaptability.
Comparison of model
PSEC (MLP)
Scratch
PSEC
(7.08% Scraten)
Parameter Count
Figure 15: Comparison of Model Parameters: The parameter count for PSEC is approximately 7.58% of Scratch, demonstrating significantly smaller model size while maintaining strong parameter efficiency, effec- tively leveraging a smaller number parameters superior task performance..
E MORE EXPERIMENTAL DETAILS
E.1 DESCRIPTION OF TASKS
We conduct experiments on 9 MetaDrive tasks and 8 Bullet-Safety-Gin tasks in the DSRL bench- mark (Liu et al., 2023a). The visualization of the environments is shown Figure 16. The tasks aim to learn policy from different level datasets such that the policy satisfies safety constraint (normalized cost < 1) and achieves higher rewards.
MetaDrive. It leverages the Panda3D game engine to simulate realistic driving scenarios. The tasks are categorized as (Road]((eeicae
where "Road" encompasses three levels difficulty for self-driving cars: easy, medium, and hard, while "Vehicle" represents four levels of surrounding traffic density: sparse, mean, and dense. In MetaDrive's autonomous driving tasks, costs are incurred from three safety-critical scenarios: (i) collision, (ii) out of road, and (iii) over-speed.
The environments are built on the PyBullet physics simulator. They feature four types agents: Ball, Car, Drone, and Ant, alongside task types: Circle and Run. Tasks designated as [Agent } {Task), }, combining the agent and the corresponding task type.
Bullet-Safety-Gym
E.2 ILLUSTRATION OF THE RECORDED DATA
To get more intuitive look the recorded data, we calculate the total reward and total cost for each trajectory in the datasets. These values are then plotted on a two-dimensional plane, where the x-axis corresponds to the total cost and the y-axis to the total reward. The results are shown in Figure 18 in the Appendix E of the paper. The plot highlights the dataset's diversity, particularly in how it captures a range of trajectory behaviors. The reward frontiers relative to cost illuminate the task's complexity, the shape of these frontiers can significantly influence the challenges faced by offline learners. Trajectories offering high rewards but incurring high costs pose an alluring yet risky opportunity, often testing the balance between optimizing performance and maintaining safety constraints. This duality underscores the importance of robust algorithms that can navigate the tf effectively.
28

---

Published as a conference paper at ICLR 2025
(a) MetaDrive
(b) Bullet-Safety-Gym
Figure 16: Visualization of the simulation environments and representative tasks of MetaDrive and Bullet- Safety-Gym. The figure is credited to Liu al. (2023a).
E.3 ADVANTAGE OF THE BENCHMARK
By generating diverse datasets across many environments with systematically varied complexities, the DSRL benchmark creates a rich and representative evaluation suite. This diversity ensures that method is tested under wide range of conditions, capturing different task structures, safety straints, and levels of stochasticity. Meanwhile, the DSRL benchmark includes multiple objectives, making it well-suited for testing the flexibility and efficiency of our method in handling new tasks. Providing diverse datasets across varying difficulty levels and incorporating multiple optimization goals enables comprehensive evaluation of our method's adaptability and performance across broad spectrum of scenarios.
F MORE EXPERIMENTS ON META-WORLD
To evaluate the effectiveness of PSEC on more complex experiments, conduct experiments on Meta-World benchmark (Yu et al., 2020b), which consists of 50 diverse tasks for robotic manipu- lation, such as grasping, manipulating objects, opening/closing window, pushing buttons, lock- ing/unlocking door, and throwing basketball. We compare PSEC with the strong baseline 1.2M (Schmied al., 2024). Next, we will elaborate on the three experiment settings our pa- per.
assembly
basketball
button press
button preso
button press
button press
coffee button
coffee pul
coffee push
bin picking
cl turn
olacssemble
door close
CoOr open
drower close
drower open
foucet open
foupet close
hammer
bOX close
hondle press
handle press
handle pull
handle pull
ever pull
peg insert
beg unplug
pick out of
pick place
Spor look
place
plate side
Ciee side
plate siide
3icte side
push bock
push
push woll
teach
Spor unlock
reach wall
shel place
SOCORT
elick push
stick pull
SweeO
window open
window close
hand Insert
Figure 17: Visualization the simulation environments and representative tasks world.
29

---

Published as a conference paper at ICLR 2025
F.1 CONTINUAL LEARNING SETTING
Following Continual world (Wolczyk et al., 2021) and L2M (Schmied et al., 2024), split the 50 tasks into 40 pre-training tasks and 10 fine-tuning unseen tasks (CW10). The training datasets the same the datasets collected by L.M.. We train 10K steps per task in CW10, which is only 10% training steps of L2M, with batch size of 1024. After every 10K update steps, we switch to the next task in the sequence. Then evaluate on all tasks in the task sequence. The results are shown in Table 3 and Table 4. We compare the performance of PSEC with L2M and other strong baselines. Thanks the efficiency of skill composition parameter space, PSEC can substantially outperform all L2M variants in a large margin, demonstrating that PSEC can achieve better performance on complex tasks.
Table 3: Success rates of different methods.
Table 4: Performance of PSEC on different tasks.
Tasks
PSEC
Methods
Success Rate
peg-unplug-side-v2
0.87
window-close-v2
0.88
L2M
0.65
shelf-place-v2
0.85
L2M-oracle
0.77
push-v2
0.89
L.2P-Pv2
0.40
handle-
penndle-reessssie--2
0.95
I.2P-PreT
0.34
stick-pull--2
0.74
L2P-PT
0,23
push-back-v2
0.85
EWC
0.17
faucet-close-v2
0.92
L2
0.10
push-wall-v2
0.86
PSEC (Ours)
0.87
hammer-v2
0.91
Mean
0.87
F.2 UNSEEN TASKS SETTING
To further evaluate the efficiency of PSEC on more challenging tasks, we pretrain on fewer (18) tasks and evaluate it more (12) unseen tasks than the first setting. Firstly, pretrain and finetune 18 tasks to obtain 18 LoRA modules. The performance on the 18 pretrained tasks is reported in Table 5. We compare the performance of PSEC with Scratch, ASEC and NSEC methods. The results show that PSEC can achieve enhanced skill learning even when the pretrained model is combined with LoRA for each task if the skill composed in parameter space. Then, evaluate PSEC with the obtained 18 LoRA modules on the unseen tasks. For the unseen tasks, we conduct two types of experiments: few-shot setting and zero-shot setting.
Few-shot. We perform few-shot learning by training the context-aware modular for Ik steps using only 10% of the total available data for unseen tasks. This setup simulates scenarios with limited data on new tasks. The results, summarized in Table 6, demonstrate that PSEC achieves high success rate on unseen tasks. This indicates that PSEC can effectively adapt new tasks, showcasing its capability for rapid transfer learning and efficient adaptation in data-scarce environments.
Zero-shot. No data from the unseen tasks is used to train the context-aware modular. Instead, the modular is trained for 2k steps using datasets from 18 pre-trained tasks. It is then evaluated directly on 12 unseen tasks, utilizing 4 seeds and 10 episodes per task. The results are shown in Table 7. Interestingly, even without access to unseen task data during training, PSEC demonstrates strong performance on several tasks. Notably, PSEC substantially outperforms NSEC and ASEC on this zero-shot transfer setting, highlighting the advantages of skill compositions in parameter spaces over noise and action spaces. Overall, the results demonstrate PSEC's ability to effectively utilize knowledge from previously learned skills to achieve strong zero-shot transfer.
30

---

Published as a conference paper at ICLR 2025
Table 5: Performance comparison pretrained tasks.
Tasks
Scratch
ASEC
NSEC
PSEC
peg-insert-side-v2
0.50
0.87
0.88
0.90
peg-umplug-side-v2
0.35
0.61
0.78
0.86
button-press-topdown-y
0.71
0.88
0.88
0.89
push-back-v2
0.26
0.61
0.76
0.88
Window-close-v2
0.65
0.84
0.84
0.88
door-open-v2
0.74
0.85
0.86
0.86
handle-press-v2
0.67
0.96
0.97
0.97
plate-slide-side-v2
0.27
0.23
0.53
0.74
handle-pull-side-v2
0.76
0.94
0.94
0.95
window-open-v2
0.87
0.75
0.88
0.89
door-close-v2
0.90
0.89
0.89
0.91
reach-v2
0.89
0.95
0.95
0.95
push-v2
0.15
0.58
0.81
0.92
stick-push-v2
0.44
0.54
0.17
0.79
drawer-close-v2
0.97
0.97
0.97
0.97
plate-slide-backv
0.90
0.94
0.94
0.95
coffee-button-v2
0.91
0.94
0.94
0.95
hand-insert-v2
0.30
0.68
0.63
0.89
Mean
0.62
0.78
0.81
0.90
Table 6: Few-shot performance compariion on 12 unseen tasks.
Tasks
ASEC
NSEC
PSEC
plate-slide-v2
0.14
0.66
0.89
sundle-press-side--2
0.73
0.65
0.92
putton-press-walll-22
0.09
0.03
0.72
button-peess--ppooonwwwll-l
0.87
0.88
0.89
push-wall-v2
0.57
0.68
0.88
reach-wall-v2
0,41
0.36
0.90
faucet-close-v2
0.41
0.49
0.90
button-
button-press-v2
0.02
0.14
0.23
0.17
0.19
0.92
handle-
handlc-pull-v2
0.15
0.21
0.93
faucet-open-v2
0.14
0.16
0.89
stick-pull-v2
0.00
0.00
0.32
G MORE VISUALIZATION ADVANTAGES OF PSEC OVER NSEC AND ASEC
To test whether the newly learned skills effectively utilize the shared knowledge of previous skills, evaluate the running policy obtained through context-aware modular combined with standing and walking skills on three rewards: stand, walk, and run. If the running skill can still get a rel- atively high stand walk reward, this represents the final combined running skill retaining these previous skills. We compare PSEC with other composition methods ASEC and NSEC. For each method, we rollout 10K steps and record the three rewards. The summarized rewards can be found in Figure 19. The results show that PSEC achieves high rewards across all tasks, whereas NSEC and ASEC cannot, demonstrating that the PSEC's running skill retains behaviors from walking and standing and suggesting superior skill sharing of PSEC compared to NSEC and ASEC.
31

---

Published as a conference paper at ICLR 2025
Table 7: Zero-shot performance comparison 12 unseen tasks.
Tasks
ASEC
NSEC
PSEC
plate-slide-v2
0.03
0,00
0.15
handle- pamullepress-side--2
0.50
0.60
0.62
mutton-press-walleee
0.00
0.00
0.40
button- propooprre--toooowwwwawall
0,85
0.87
0.89
push-wall-v2
0.53
0.53
0.71
reach-wall-v2
0.34
0.05
0.90
faucet-close-v2
0.00
0.00
0.16
button-press-v2
0.00
0,00
0.15
plate-slide-bacc-siiee-e
0.00
0.00
0.00
handle-pull-v2
0.00
0.00
0.00
faucet-open-v2
0.00
0.00
0.77
stick-pull-v2
0.00
0.00
0.00
Table 8: Results in the dynamics shift setting over 10 episodes and 5 seeds. -m, -mr and -me refer to D. sampling from medium, medium-replay and medium-expee V2 data D4RL (Fu al., 2020), respectively.
Metric
Halfcheetah-m Halfcheetah-mr Halfcheetah-me Walker2d-m Walker2d-mr Walker2d-me
BC
26.4 7.3
14.3 7.8
19.1 + 9.4
15.8 14.1
1.4 1.9
21.7 8.2
MОРO
-1.1 + 4.1
11.7 5.2
-1.1
3.7
3.7
0.1 +
CQL
35.4 = 3.8
8.1 8.199.4
26.5 - 10.8
18.8 18.8
8.5
19.1 14.4
IQL
29,9 = 0.2
22.7 6.4
10.5 = 8.8
2..5
10.7 11.9
26.5 + 8.6
DOGE
42.6 3.4
3.66
6.6
45.1 10.2
13.5
35.3 4.1
TSRL
38.4 3.1
28.1
39.9 21.1
49.7
26.0 11.3
46.4 + 13.2
Joint (ravi(Gr))ity))
1.4
5.4
39.4
15.7
+
Joint train(Friction)
15.8 = 1.0
14.9 1.2
16.5 1.1
8.1
7.6
7.4 0.5
Joint traighT
9.5 + 5.3
9.8 8.5
6,4 + 1.3
50.6 8.8
6.3 3 3.0
54.9 + 14.8
Dynamic shift
PSEC(Gravity)
29.2
42.4 1 1.0
$7.2 4.5
26.8 5.2
71.8
PSEC(Friction)
40.1 = 1.2
31.1 + 1.3
42.1 + 1.0
61.7 7.5
20.9 + 4.6
75.0 + 12.1
Body shift
PSEC(Thigh)
41.4 0.3
32.3 1.4
43.9 + 2.5
64.96 + 4.5
25.5 + 4.5
71.4 + 14.3
32

---

Published as a conference paper at ICLR 2025
Metadf/ve-easyspareevo
Metadrive-dasymenn-0
Mettarive--asyddenseeo
350
350
350
300
300
300
Rewar
Rewar
200
200
200
150
150
150
100
50
50
50
0
20
40
60
80
0
20
40
60
80
0
10
20
30
40
50
60
70 80
Costs
Costs
Costs
Metadfive-mediumsprrss
Metadrive-mediummean-0
Metadrive
250
$5""""
SFT
250
250
of
A bul 30
Rewards
Rewards
Rewards
150
150
100
100
100
50
50
50
10
30
40
30
40
Costs
Costs
Costs
Metadfive-hardssarseeo
Metadrive-hardmean--0
Metadrive-harddeensen
500
500
500
400
400
400
Rewards
Rewards
Rewards
300
380
300
E3
200
200
200
0-
20
40
20
40
20
40
Costs
Costs
Costs
SafetyAntRun
SafetyCarRun
SafetyBallRun
1000
1200
800
1000
Rewards
wards
Rewards
600
400
800
400
350
300
400-
250-
200
200
0
20
40
60
80
100
120 140
0
5
10
15
20
25
30
35
40
0
10
20
30
80 Costs
40
50
60
70
Costs
Costs
SafetyDroneRun
SafetyAntCircie
SafetyCarCircle
700
500-
600
400
Rewards
500
Rewards
300
Rewards
400
400
3.0
200
200
100
100
100
0
20
40
60
100
120
140
0
25
50
75 100 150175200
20
40
60
80
100
Costs
Costs
SafetyBallCircle
SafetyDroneCircle
1000
900
Rewards
600
Rewards
700
400
600
200
400
300
50
Costs
Costs
Figure 18: Illustration of the cost-reward plot for datasets from MetaDrive and Bullet-Safety-GGn
33

---

Published as a conference paper at ICLR 2025
Stard Reward Distributions INDE!
Walk Reward Distributions (KDE)
Run Bittnbuttons (DF)
AS2C
N00C
PSC
Donsty
Density
0.2
0.4
2.0
0.0
05
0.0
1.5
20
Rermard
Newwadd
Reward
Figure 19: We evaluate the final running policies of PSEC, NSEC and ASEC with the "stand," "walk," and "run" rewards with 10 episodes and 3 random seeds. Then plot the reward distribution by kernel density estimation (KDE). Each curve represents the probability density of rewards obtained for specific reward. The results show that PSEC achieves high rewards across all tasks, whereas NSEC and ASEC cannot, demonstrating that the PSEC's s running skill retains behaviors from walking and standing and suggesting superior skill sharing of PSEC compared to NSEC and ASEC.
Table 9: Hyperparameters for multi-objective composition tasks.
Hyper-parameters
Value
Normalized state
True
Target update Expectile 5
lc-3
0.9
shared hyperparameters
Discount 7
0,99
Actor learning rate
3e-4
Critic learning
3e-4
Number of added Gaussian noise T
hidden dim
256
hidden layers
2
activation function
ReLU
Mini-batch size
2048
Optimizer
Adam (Kingma & Ba, 2014)
Training steps
le6
(7,(s, a)) hidden dim
256
QT(s, hidden layers
QF(s, a) activation function
ReLU
V (s) hidden dim
256
(s) hidden layers
2
(s) activation function
ReLU
Actor hidden dim
256
Actor hidden layers
2
Actor Activation function
ReLU
Mini-batch size
2048
Optimizer
Adam
Training steps
5e4
Q7, (s. a) hidden dim
256
(T, hidden layers
2 ReLU
(s, a) activation function
Vi,(s) (s) hidden dim
256
V (s) hidden layers
2
V. (a) Activation function
ReLU
Actor hidden dim
256
Actor hidden layers
Actor Activation function
ReLU
Mini-batch size
2048
Optimizer
Adam
Training steps
5e4
hidden dim
256
hidden layers
2
a(s; 8)
activation function
ReLU
Mini-batch size
2048
Optimizer
Adam
Training steps
le3
LoRA
rank
16
34

---

Published as a conference paper at ICLR 2025
haifchestah-medium (F)
halfcheetah-medium (G)
halfchestah-medium (T)
40
40
Scores
Scores
Scores
30
30
30
20
20
Training Steps (1e5)
Training Steps (1e5)
Training Steps (1e5)
haltcheetah-meddiu-mrepyy (F)
halfcheetah-mediumrrellaaa (G)
haltcheetah-mediiurreellgg (T)
30
30
Scores
Scores
Scores
20
20
20
10
0
10
0
2
10
0
Training Steps (1e5)
Training Steps (1e5)
Training Steps (1e5)
balicheetah-mellumeepeee (F)
haffcheetah-meddiu-exrrrr (G)
haiicheetah-medlleeeepee
40
Scores
30
Scores
30
Scores
30
20
20
20
0
1
Training Steps (1e5)
Training Steps (1e5)
Training Steps (1c5)
walker2d-medium (F)
walker2d-medium (G)
walker2d-medium (T)
Scores
Scores
Scores
50
40
40
40
30
1
2
30
6
1
30
6
1
2
Training Steps (1e5)
Training Steps (1e5)
Training Steps (1e5)
wallerza-medium-repla (F)
walker2d-medium-rrplla (G)
walkerzd-medium-replay (T)
20
25
Scores
Scores
Scores
10
15 0
Training Steps (1e5)
Training Steps (1e5)
Training Steps (1e5)
palker2d-medduumeexpeed (F)
wallerr2d-mddlu-exxprr (G)
walker224-meduumeepper (T)
80
70
70
70
Scores
Scores
60
Scores
60
50
50
40
40
30
30
30
0
Training Steps (1e5)
Training Steps (1e5)
Training Steps (1e5)
Figure 20: Results of performance conducted on dynamic shift and body shift tasks. The lines and shaded areas indicate the averages and standard deviations calculated over random seeds.
35

---

Published as a conference paper at ICLR 2025
Table 10: Hyperparameters for continual policy shift.
Hyper-parameters
Value
Normalized state
True
Target update rate Expectile r
le-3
0.9
shared hyperparameters
Discount y
0,99
Actor learning rate
3e-4
Critic learning
3e-4
Number added Gaussian noise T
hidden dim
256
hidden layers
2
activation function
ReLU
Mini-batch size
1024
Optimizer
Adam
Training steps
le6
hidden dim
256
hidden layers
m
Activation function
ReLU
Mini-batch size
1024
Optimizer
Adam
Training steps
lc4
Actor hidden dim
256
Actor hidden layers
2
Actor Activation function
ReLU
Mini-batch size
1024
Optimizer
Adam
Training steps
lc4
hidden dim
256
hidden layers
0)
activation function
ReLU
Mini-hatch size
1024
Optimizer
Adam
Training steps
1e3
LoRA
rank
36

---

Published as a conference paper at ICLR 2025
Table 11: Hyperparameters for dynamic shift.
Hyper-parameters
Value
Normalized state
True
Target update rate Expectile r
le-3
0,9
shared hyperparameters
Discount
0,99
Actor learning rate
3c-4
Critic learning
3e-4
Number of added Gaussian noise T
hidden dim
256
hidden layers
2
activation function
ReLU
Mini-batch size
1024
Optimizer
Adam
Training steps
le6
(r(s, a) hidden dim
256
(*(s,a) a) hidden layers
Q, a) activation function
ReLU
V (&) hidden dim
256
V hidden layers
2
m
V (5) activation function
ReLU
Actor hidden dim
256
Actor hidden layers
2
Actor Activation function
ReLU
Mini-batch size
1024
Optimizer
Adam
Training steps
2c4
hidden dim
256
hidden layers
(s;0)
activation function
ReLU
Mini-hatch size
1024
Optimizer
Adam
Training steps
le3
LoRA
rank
37


Reviews:
[{"review_id": "7vTPFp2uOB", "reviewer": "Reviewer_ac7Q", "summary": "The paper introduces the Parametric Skill Expansion and Composition (PSEC) framework, which allows RL agents to learn and combine new skills by efficiently using a \"skill library\". Instead of relearning tasks from scratch, PSEC uses LoRA modules—compact, adaptable components—that can be added to this library as plug-and-play skills. This design enables agents to adapt to new tasks by combining skills directly within the model's parameters, allowing them to leverage shared knowledge from previous tasks while avoiding \"catastrophic forgetting\", since each \"skill\" is stored as an independent \"frozen\" module. \n\nAuthors test across different environments, including multi-objective tasks (where skills must be blended to meet multiple goals), settings with continual learning demands, and dynamic scenarios where the environment changes. Results show that PSEC enables efficient, flexible learning compared to vanilla RL.", "questions": "Please refer to the questions above.\n\n--- Post discussion ---\nAuthors correctly addressed my concerns during the rebuttal incorporating relevant discussion on related works and limitations, while I would aprreciate some of this to be present not only in the appendix, I believe the strenghts of the work outweight the weaknesses in the revised version", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to follow and clearly written. The experiment suite is diverse and proves the ability of SPEC to learn and compose skills.\n\nRegarding originality and significance I particularly found interesting the usage if diffusion models to adjust the weight levels of the compositions of skills and the study for skill composiiton on the differenc spaces (parameter, noise and action spaces). It made very clear the thought process of the authors to design the framework.", "weaknesses": "My biggest concern with this paper is that this is not the first paper proposeing using LoRa for multiple task leanring, e.g. [1,2] and while SPEC is clearly different from previous existing approaches, some level of  comparison, theorethical or empirical would be greatly benefitial. Specifially, at present is difficult to discern what are the novel components within SPEC wrt previous skill learning frameworks.\n\nIt would be also good if authors could provide their though contrasting SPEC with existing works on RL agents that learn skills compositionally such as [3-6]. Particularly, [5] even points as an advantage over previous frameworks not having to learn and storing a different set of weight for every skill/sub-task. Since SPEC goes back to this form of learning it would be good that authors include a discussion on this topic.\n\n[1] Ponti, Edoardo Maria, et al. \"Combining parameter-efficient modules for task-level generalisation.\" Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.\n\n[2] Huang, Kaixin, et al. \"Solving Continual Offline Reinforcement Learning with Decision Transformer.\" arXiv preprint arXiv:2401.08478 (2024).\n\n[3] Araki, Brandon, et al. \"The logical options framework.\" International Conference on Machine Learning. PMLR, 2021.\n\n[4] Hill, F., Tieleman, O., von Glehn, T., Wong, N., Merzic, H., & Clark, S. Grounded Language Learning Fast and Slow. In ICLR 2021.\n\n[5] León, Borja G., Murray Shanahan, and Francesco Belardinelli. \"In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications.\" ICLR 2022.\n\n[6] Vaezipoor, Pashootan, et al. \"Ltl2action: Generalizing ltl instructions for multi-task rl.\" International Conference on Machine Learning. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xn1GMLdiSP", "reviewer": "Reviewer_18jx", "summary": "This is a paper presents a new skill composition method, which \ncomposes skills in the parameter space. Concretely, the paper \nintroduces PSEC, which fine-tunes new skills using low-rank adaptation \n(LoRA) and argues the actions synthesized through this parameter-space \ncomposition greatly improves the performance, compared to two other \ncomposition methods termed noise-level composition and action-level \ncomposition.", "questions": "o See the concerns and questions from Weaknesses \n \no If I understand correctly from the paragraph \"Context-aware \nComposition\" (Line 244-254), only $W_{k}$ is updated at skill k. If \nthat's the case, What would be the impact of existing pre-trained LoRA \nweights? Is it possible that because there are two skills that are \ncontradictory to each other, the LoRA fine-tuning on a new skill might \nbe harder to train? It will be good to see such experiments, and see \nif the method still applies; or an additional experiment can conducted \nwhere the learning curriculum is shuffled. Will the result be \nconsistent with the existing ones where learning happens from standing \nto walking, running? \n \no Can this method scale to dozens of skills instead of three skills \npresented in this paper? An experiment on stress testing the number of \nskills it can handle will be able to support the effectiveness of the \nproposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "o This paper presents an interesting idea of how to compose skills in \nthe parameter space, and provide a clear categorization of skill \ncomposition: parameter-level, noise-level, and action level. This \npaper shows a way where neural network-based skills can be composed at \nthe parameter space level, instead of composing at the action output \nlevel. \n \no In the long run, the proposed method can open up the possibility for \nefficient skill learning based on large models in decision-making \ndomains", "weaknesses": "o While the general motivation in the first paragraph makes sense, the \nexample needs to be justified a bit better From online statistics, the \ntime difference between child learning to walk and learning to stand \nwithout support is on average 2-2.5 months. Not everyone would call \nthis \"rapid\" (Ln 32) \n \no While the idea of parameter-level composition is interesting and \nalso introduces fewer bottlenecks compared to methods like action \nlevel, the argument of parameter-level composition being superior is \nnot well-grounded. The explanation from line 256 to line 304 lacks \nconvincing evidence. \n \no The t-SNE plot (Figure 4) does not directly reveal that the \nparameter space \"shares knowledge\" - if 4(a) can be explained to have \nshared knowledge, why is 4(b) not a plot that shows \"shared \nknowledge\"? The authors might try to design a more specific \nvisualization that clearly illustrates the idea of \"shared \nknowledge.\" \n \no In the experiment sections, there are a lot of terms that authors \neither didn't specify clearly or misuse the words. \n \n    o It is unclear what \"versatility\" means. Authors should provide a \n    clear definition of this term(Line 354&473) \n \n    o Line 397, the authors shouldn't use the word \"generated\" to \n    describe other comparison methods, especially when the results \n    haven't been revealed. \n \n    o Line 374 \"popular safe offline RL benchmark\" is not \n    informative. The authors should replace this with  an explanation \n    of what problems does it contain (robot manipulation? autonomous \n    driving? a collecition of them?) and why is it a good benchmark to \n    test on (large-scale? diverse? or any other key features that make \n    them special) \n \n    o What does \"safety\" refer to? It's not explained well. The only \n    thing I can take a guess is the cost that shows in the table, \n    which does not really provide much information on how to interpret \n    those numbers. \n \no The paper touches on multi-task learning and continual learning, and \nuses a skill library. \nAll of these topics have been studied to some degree for decades.  Yet \nthe majority of citations are from 2023 and 2024, with only two \ncitations to work prior to 2016 - and those are just to energy-based \nmodels and t-SNE.   \nWithout putting the work in the proper context of prior work, its \nnovelty and significance can't be properly assessed.  \n \no Minor: There are many small grammatical errors throughout the paper \n- including in the first sentence of the abstract!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bieH71x0DX", "reviewer": "Reviewer_1bSW", "summary": "This paper introduces a novel framework called Parametric Skill Expansion and Composition (PSEC). PSEC employs Low-Rank Adaptation (LoRA) to learn and store new skills, and directly synthesizes skills in the parameter space. It incorporates a context-aware module to adaptively compose skills in dynamic environments and utilizes diffusion models for policy modeling. The framework has been evaluated across various scenarios, including multi-objective composition, policy shifts, and dynamics changes.", "questions": "* One of the main strengths claimed for PSEC is that it \"reduced computational costs and memory usage\". How does this compare to the baselines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The key features of PSEC include efficient skill learning and storage utilizing LoRA, direct skill synthesis in parameter space, adaptability through a context-aware module, and the ability for continuous skill expansion. Its effectiveness has been validated through various experiments, demonstrating versatility across different scenarios.", "weaknesses": "* While the strong assumption about the expressiveness of the pre-trained policy and the scalability issue of the skill library appear to be weaknesses, these seem to have been addressed in the paper's appendix.\n* A potential limitation of the authors' proposed framework is its applicability only to environments where data is available. This could be considered a weakness of the paper. It is conceivable that if the framework could be extended to unseen tasks through transfer learning or fast adaptation techniques, it would demonstrate greater differentiation from previous research.\n* Minor:\n    * Generally, skill learning, as in  [1] and [2], refers to 'temporal abstraction', but the skill defined by the authors seems to differ from this. This may lead to confusion with existing skill learning concepts.\n\n    * [1] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. \"Accelerating reinforcement learning with learned skill priors.\" Conference on robot learning. PMLR, 2021.\n    * [2] Eysenbach, Benjamin, et al. \"Diversity is all you need: Learning skills without a reward function.\" arXiv preprint arXiv:1802.06070 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ve7e1f9y2f", "reviewer": "Reviewer_oGS2", "summary": "This paper proposes Parametric Skill Expansion and Composition framework, which encodes skill primitives offline using LoRA modules and learning skill compositions with the generated LoRA library. The PSPEC framework is designed to cope with multi-objective composition as well as adapting to shift in policy and dynamics. The authors provides extensive experimentation on their proposed algorithm on the D4RL, DSRL, and DeepMind Control Suite.", "questions": "- I think the paper lacks general motivations and overview on the benchmarks used. Why does this paper uses an offline safe RL benchmark instead of just an offline RL benchmarks in general? What are the tasks and what do the recorded data look in the offline benchmarks? Why would they provide a characteristic examples to highlight the pros and cons of PSEC?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- PSEC has a very intuitive and stright-forward design that utilizes LoRA structure. While PSEC wouldn't be the only structure to use the LoRA as building blocks for skills composition, the authors provides tSNE analysis to show why their method encodes skills better than the alternatives\n- The paper is very pleasant to read and the paper structure too, is very straightforward. The authors also covers the related works very well which makes PSEC's design choices to be convincing.\n- The experimentations are very extensive.\n- I found the graphics very easy to understand and they summarize the key points very well. From intro to methods, I could very easily follow the keypoints by reading the details and captions in the graphics beforehand.", "weaknesses": "- LoRA is one of the key elements of the PSEC's structure, and while LoRA is mentioned in the abstract, the authors do not explain what LoRA is until page 2. I think it would be better to say in the abstract what LoRA stands for or simply refer to the module as a skills module if not explaining what LoRA is.\n- While the experiments are extensive, I think some of the assumptions and motivations were not well explained. I have included the details in the questions section.\n- I think one of the possible limitations of composing skills as a weighted some of LoRA actions is that some skills are not simply some of sub-skills. For example, the skill of throwing a ball is more of a fluid movements where the a person builds up the momentum and transfer the kinetic energy to the ball, instead of some combinations of walking forward and rotating torso. I am curious how the PSEC would perform with more complex tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)