Tackling Continual Offline RL through Selective Weights Activation on Aligned Spaces
Authors: Jifeng Hu
Sili Huang
Li Shen
Zhejian Yang
Shengchao Hu
Shisong Tang
Hechang Chen
Lichao Sun
Yi Chang
Dacheng Tao
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
continual learning
Primary area: reinforcement_learning

Abstract: Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based continual learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 17 baselines, our method reaches the SOTA performance.

Tackling Continual Offline RL through Selective Weights Activation on Aligned Spaces
Jifeng Hu'
Sili Huang
Li Shen
Zhejian Yang'
Shengchao Hu"
Shisong Tang
Hechang Chen
Lichao Sun
Yi Chang
Dacheng Tao?
Jilin University Minzu University of China SShnnzen Campus of Sun Yat-sen University
*Shanghai Jiao Tong University TTsnnguua University LLeiigh University
Nanyynng Technological University
(hujf21, zjyang22) (eae22mmmalll...mm jll. edu, en {chenhc, yichang) edu. cn huangsili@min Cmuc. edu, en mathshenli@gman com charles hharles-hu@sjtu edu on bongshisong1@ggall com Lis2210lehigh edu dacheng. tao@gmail........ com
Abstract
Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-basee based continual learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces various tasks, facilitating continual training in the same space. Then, we propose leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 17 baselines, our method reaches the SOTA performance.
we propose Vector-Quanten
Introduction
The endeavor of recovering segh-perffrmacce policies from abundant offline samples gathered by various sources and continually mastering future tasks learning and previous knowledge maintaining gives birth to the issue of continual offline reinforcement learning (CORL) [22 growing scenarios or offline datasets pose challenges for most continual RL methods that are trained on static data and are prone to showing catastrophic forgetting of previous knowledge and ineffective
2
39 42 Ever-
learning of new tasks [63 (5). Facing these challenges, three categories of methods, rehearsal-
based [42] 77 [0). regularization-brn based [82 [05] [044, and structure-based methods [12 67 g are
proposed to reduce forgetting and facilitate continual training.
However, most previous studies only focus on the continual learning (CL) setting with identical state and action spaces 63 82). It deviates from the fact that the ever ever-growing scenarios or offline datasets are likely to possess different state and action spaces with previous tasks for many reasons,
"Corresponding authors: Hechang Chen, Sili Huang, and Yi Chang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

different state and action spaces
identical state and switch to action spaces next task hopper
model
state
hopper
action
state space length
action space length
walker2d
walker2d
Weights
Assembling
diffusion model for multi-task
QSA
SWA
halfcheetah
ha halfeheetah
ant
ant
Figure 1: The high-level intuition of VQ-CD, Quantized space alignment (QSA) is used to expand the application range of VQ-CD, while selective weight activation (SWA) is used to reduce forgetting of historical tasks in continual learning.
such as the variation of demands and the number of sensors 998 [02]. Moreover, these datasets
often come from multiple behavior policies, which pose the additional challenge of modeling the multimodal distribution of various tasks (2] Benefiting from diffusion models' powerful expressive capabilities and highly competitive performance, an increasing number of researchers
are considering incorporating them to address the CORL problems 4 99, 17] from the perspective models with
of sequential modeling. There have been several attempts to combine diffusion- rehearsal- based and regularization- based techniques, which usually apply constraints to the continual
model learning process with previous tasks' data or well-trained weights [2 99 63 However,
constrained weight updating will limit the learning capability of new tasks and can not preserve the previously acquired knowledge perfectly [98]. Although structure-based methods can eliminate forgetting and strengthen the learning capability by preserving well-trained weights of previous tasks and reserving disengaged weights for ongoing tasks, they are still limited in simple architectures and CL settings with identical state and action spaces [102 93 66 Thus, in this paper, we seek to answer the question:
Can we merge the merits of diffusion models' powerful expression and structure-based parameters isolation to master CORI. problems with any task sequence?
We answer this in the affirmative through the key insight of allocating harmonious weights for each continual learning task. Tigure[]shows the intuitive design of our methods. Specifically, we propose fector-Quanticn Continual Diffuser called VQ-CD, which contains two complementary sections: the quantized spaces alignment (QSA) module and the selective weights activation diffuser (SWA) module. To expand our method to any task sequences under the continual learning setting, we adopt the QSA module to align the different state and action spaces. Concretely, we adopt vector quantization map the task spaces to a unified space for training based the contained codebook and recover to the original task spaces for evaluation. In the SWA module, we first perform task mask generation for each task, where the task masks are applied to the one-dimensional convolution kernel of the U-net structure diffusion model. Then, we use the masked kernels to block the influence of unrelated weights during the training and inference. Finally, after the training process, we propose the weights assembling to aggregate the task-related weights together for simplicity and efficiency. In summary, our main contributions are fourfold:
We propose the Veanor-Quantized Continual Diffuser (VQ-CD) framework, which can not only be applied to conventional continual tasks but also be suitable for any continual tasks setting, which makes it observably different from the previous CL method.
In the quantized spaces alignment (QSA) module of VQ-CD, we adopt ensemble vector quantized encoders based on the constrained codebook because it can be expanded expediently. During the inference, we apply task-related decoders to recover the various observation and action spaces. the selective weights activation (SWA) diffuser module of VQ-CD, we first perform task-related task masks, which will then be used to the kernel weights of the diffuser. After training, we propose assembling weights to merge all learned knowledge,
Finally, we conduct extensive experiments on 15 CL tasks, including conventional CL settings and any CL task sequence settings. The results show that our method surpasses or matches the SOTA performance compared with 17 representative baselines.

---

2 Related Work
Offline RL. Offline reinforcement learning is becoming an important direction in RL because supports learning on large pre-collected datasets and avoids massive demand for expensive, risky interactions with the environments [69 71 34 7 31 46 45]. Directly applying conventional RL methods in offline RL faces the challenge of distributional shift [78. 62 96
100]
44 caused by
the mismatch between the learned and data-collected policies, which will usually make the agent improperly estimate expectation return on out-of-distributiin actions [78 58 2 43 47). To tackle
this challenge, previous studies try to avoid the influences of out-of-distribution actions by adopting
constrained policy optimization 20 71 (88. behavior regularization [70, 55 23 37 importance
sampling 44 24 103], uncertainty estimation 33 89 61
and imitation learning [94 8 13 40)
Continual RL. Continual learning (CL) aims to solve the plasticity and stability trade-off under the
task setting, where the agent can only learn to solve each task successively [92]. CL can be classified into task-aware CL and task-free CL according to whether there explicit task boundaries [5] 92]. In this paper, we mainly focus on task-aware CL. There are three main technical routes to facilitate forward transfer (plasticity) and mitigate catastrophic forgetting (stability). Rehearsal- based approaches 80
102 82 store a portion of samples from previous tasks and use interleaving updates between new tasks' samples and previous tasks' samples. Simply storing samples increases the memory burden in many scenarios; thus, generative models such as diffusion models are introduced to mimic previous data distribution and generate synthetic replay for knowledge maintenance [101 75 22]. Regularization-base
66
93
approaches [51] 52 104 [05] seek to find a proficiency compromise between previous and new tasks by leveraging constraint terms on the total loss function. Usually, additional terms of learning objectives will be adopted to penalize sigmificant changes in the behaviors of models' outputs or the updating of models' parameters [55] 51]. In the
82 [6 researchers usually consider parameter isolation by using sub-networks or task-related neurons to prevent forgetting.
structure- bruseure-based approaches [93 53 93
102
Diffusion RL. Recently, diffusion- based models have shown huge potential in RL. under the
38 36) A typical use of diffusion models is to mimic the joint distribution of states and actions, and we usually use state-action value functions as the classifier or class-free guidance when generating decisions [72 29
perspective of sequential modeling (30 32 4
Diffusion models, as representative generative models, can also be used as environmental dynamics to model and generate synthetic samples to improve sample efficiency or maintain previous knowledge in CL 197 28 65 16 63). It is noted that the diffusion model's 's powerful expression ability on multimodal distribution also makes suitable for being used as policies model the distribution of actions and as planners to perform morgooorzoon planning [1 50 IT]. Besides, diffusion models can also be used as multi-task learning models to master several tasks simultaneously 26 or as multi-agent models to solve more complex RI. scenarios [106].
73 33
3 Preliminary
3.1
Continual Offline RL
We focus on the task-aware CL in the continual offline RL in this paper 92 82 76 90] Suppose that we have I successive tasks, and task j arises behind task for any < j. Each task i, E [1 /] is represented by Markov Decision Process (MDP) M = (S', AA', Pt, R', 7),, where we use supscript i to differentiate different tasks, I is the number of total tasks, S the state space. space, respectively. P S x A + A(S) denotes the transition function, R : S x AAx x S + R is the reward function, and 7 E [0. 1) the discount factor. Conventional CL tasks have the same state and action spaces for all tasks, i.e.. S = [S*), = A' E ] IJ. While for any tasks sequences, we have |S + (8'), AA'l * [ J. In the offline RI., we can only access pre-collected datasets (D))EE of each task. The goal of continual offline RL is to find an optimal policy that can maximize the objective Σ Ex I (s a))
the action
98
on all tasks.
3.2 Conditional Generative Behavior Modeling
In this paper, we adopt the diffusion-based model with the U-net backbone as the generative model to fit the joint distribution q(T.) = [ ddr I:K of state sequences Ta and an inverse dynamics
3

---

Continual Triining Taski
Invere Dynamic Looe
Diffasion Loos
Selective Weights Aclivation Diffoser?
length
action sequence
[9
aligned
State
Evalmation
Loss
Kill Hu kii-cl
length
state sequence
spares
Return
Action
Generation Process
Enet block
gemerated state sequence
length
return sequence
Weights
Inverse Dynamk
blocks
[k]
[R p
Weights mark
Different tasks
spaces
Quantized Spaces Aligunient
Favironment
VQD
Weights Assembling
masked
masked convolution
input
Task Mask Generntion
nssembled
kerne
Intally
Cinlebook
weights
VOE
weights
nellape
convolution and slide
BB
M
mask
Task
Task
Task
weights
task pre-defined
musii block
Mi
lank
mask
Figure 2: The framework of VQ-CD. It contains two sections: The Quantized Space Aligmment (QSA) module and the Selective Weights Activation (SWA) module, where QSA enables our method to adapt to general task-aware continual learning task settings by transferring action spaces to the same spaces. SWA uses selective neural network weight activation to maintain knowledge previous tasks through task-related weight masks. After the training, we perform Weights Assembling to integrate the total weights and save the memory budget.
different state and
model fine, (St, St+1) produce actions ae, where & [1 K] is the diffusion step. t is the RL time step, y is the parameters inverse dynamics model, and we omit the identification tasks for the sake of simplicity because the training is same for all tasks. Through specifying the pre-defined forward diffusion process q(T I ( N(T,; k Vours" --1 B) and the trainable reverse process
(u(T5-1177 = NN (T( k), )
k,
30 we can train the diffusion model with the simplified
loss function
( = E (1, 2..... (...))))))
co(T ks +C)111
where T. VOrT? + v1 ogC, no (T( Vac
1
Be ED s k:), Σ = 1-āx-1 Br. is VI-oE
I-ûc
the approximate discretization pre-defined parameters 212 3g 1 ai, ag = II aes U is the uniform distribution, is standard Gaussian noise, I is the identity matrix, 7 D is the state sequences stored in the task replay buffer D, B is binomial distribution, À = 0.25 is the parameter of B, C is condition, which is usually selected discounted returns or value function in RL, and 0 is the total parameters of model eg. The following is
#k-1
(+
BRE.
(2)
generation function, where we use 7h to denote the generated state sequences, è = ca( k 0) w(ea(Fk, h; C) en (fte k; 0)), w is the guidance scale, 0 means b = We use inverse dynamics model (.) (.) to produce actions, where the training loss is
[(bb) --Dlle - f
st+1)11
(3)
4 Method
Our method enables training on general task-aware CL task sequences through two sections (as shown Figure : the selective weights activation diffuser (SWA) module and the quantized spaces alignment (QSA) module. Algorithm shows how to generate the actions during inference. The detailed training process shown in Algorithm T of Appendix A.T. In the following parts, we introduce these two modules in detail.

---

Table The comparison of VQ-CD, diffusion- based baselines, and LoRA methods on Ant-dir tasks, where the continual task sequence is 10-15-19-25. The results are average on 30 evaluation rollouts with 30 random seeds.
Method
VQ-CD
Col>
Multitsk Col>
IL.
CoD- LoRA
Diffuser-woo rehearsal
CoD- RCR
MTDIFF
DD-w/o rehearsal
(ours)
rehearsal
Mean
558 14
478. 19515
15
402.53
g
24
140.44.9.
84.01
-1115
return
4.1 Quantized Spaces Alignment
To make our method suitable for solving any CL task sequence setting, we propose aligning the different state and action spaces with the quantization technique. Specifically, we propose solve the following quantized representation learning problem
min Og 0g
LoSA(2;0e) Ber 02. 0g),
(4)
S.C.
|| 12
< P.
where (x) = E [| fvQD [1]] + [|sg((2)) 12 + + [[|s((z)) [[]ss[[s(((+a)) - En 11] is the total quantized loss, sg(-) represents the stop gradient operation, 0, and 02 are the parameters of the vector quantized encoder (VQE) and vector quantized decoder (VQD). fg is the parameters of the codebook, p limits the range of codebook embeddings, x can represent the states or actions for each specific CL task, 2q fe, (Ee) is the quantized representation which is consisted of fixed number of fixed-length quantized vectors, and &e = QvQe (x; 0e) is the output of the encoder. Here, we propose searching the constrained optimal solution of the above problem for the consideration of the diffusion model training within limited value range, just like the limit normalization in CV [30 [4 and RL 64]. There are many methods to force optimization under restricted constraints, such as converting the constraints to a penalty term (9]. In our method, for simplicity and convenience, propose to directly clip the quantized vector 2g to meet the constraints after every codebook updating step. Moreover, meet the potential demand for extra tasks beyond the predefined CL tasks, design the codebook as easy to equip. where the quantized spaces of different tasks are separated so that we can expediently train new task-related encoders, decoders, and quantized vectors.
For tasks where the state and action spaces are different, we can use the well-trained QSA module to obtain the aligned state feature s¹_ fa,,, (fiQE, (s'; 0e)) and the action feature a², = fa,o, (fVQE, (at:0)) (e)) for each task i. Thus, we can use T and Tuzo to represent the state and action feature sequences. Now, the action is produced through of fiQD. (Jinn(sz,ts (2,,1+1) Oa)-
4.2 Selective Weights Activation
In this section, we introduce how to selectively activate different parameters of the diffusion model to reduce catastrophic forgetting and reserve disengaged weights for ongoing tasks.
Task Mask Generation. Suppose that the diffusion model contains L. blocks, and the weights (i.e., parameters) of block 1 are denoted by Wi, (1....,.), There influence of the weights on the model outputs. One is masking the output neurons O = f((:; Wi) of each block, where fi(-) the neural network function of block This strategy is friendly to MLP-based neural networks for two reasons: 1) the matrix calculation, such as W3 I, is relatively simple so that we can easily recognize the disabled weights; 2) we do not need to apply any special operation on the optimizer because the output masking will cut off gradient flow naturally. However, we can not arbitrarily apply the above masking strategy more expressive network structures, such as convolution- parameters and outputs. Thus, we search for another masking strategy: masking the parameters Wi with M,, which permits us to control each parameter accurately.
ways to disable the
networks, because we can not easily distinguish the dependency between
Specifically, suppose that the total available mask positions of block are Mj, In this paper, Mi is ones matrix, and the entries with 0 mean that we will perform masking. Before training on task i, we first pre-define the specific mask Mr, of task i on block by randomly sampling unmasked positions from the remaining available mask positions. Then, with the increase of the tasks, the remaining
available mask positions decrease until M, El= M..
5

---

Selective Weights Forward and Backward Propagation. After obtaining the mask My,, we can perform forward propagation with masked weights
co C) f
(5)
O++1 = f+410, k; M1,++1 W(41), Toad /Tayy
where co is the noise prediction model introduced in Equation and M,,+1 o Wi+1 represents the pairwise product. 75.A and T4. denote the perturbed state or action sequences of task at diffusion step k. Through forward designing, we can selectively activate different weights for different tasks through the mask M,,+1, thus preserving previously acquired knowledge and reserving disengaged weights for other tasks. Though we can expediently calculate the masked output O++1 during forward propagation with weights neurons masking, it poses a challenge distinguishing the dependency from weights to loss and updating the corresponding weights during the backward propagation. order to update the corresponding weights, we realize two methods. 1) Intuitively, we propose to update the neural network with the sparse optimizer rather than the dense optimizer [15], where the position and values of the parameters are recorded to update the corresponding weights. However, in the implementation, we find that the physical time consumption of the sparse optimizer intractable (Refer to Table of Appendix B.8 for more details.), which encourages us to find a more straightforward and convenient method. 2) Thus, we propose extracting and assembling the corresponding weights at the end of the training rather than updating the corresponding weights during training. This choice brings two benefits: (1) It can significantly reduce the time consumption spent on training. (2) is friendly to implementation on complex network structures.
Weights Assembling. Assembling weights after training permits us to save the total acquired knowledge and do not need extra memory budgets. Concretely, after training on task we will obtain the weights W,, which can be extracted with the mask M, from the total weights W * 22). including all the diffusion model weights. We use W, to denote the weights related task i, Q the training step on each CL task, and Wi S2 represents the total weight checkpoint at training step i * Q. Then, the end of the training, we can assemble weights {W E I} by simply adding these weights together because of the exclusiveness property, i.e., W = Σ W; Ei= M₁ Wii +2].
Experiments
In this section, we will introduce environmental settings, evaluation metrics, and baselines in the following sections. Then, we will report and analyze the comparison results, ablation study, and parameter sensitivity analysis. Other implementation details are shown in Appendix AA.2 and A.3
5.1 Environmental Settings
Following previous studies [98), we select MuJoCo Ant-dir and Continual World (CW) to formulate traditional CL settings with the same state and action spaces. In Ant-dir, select 10-15-19-22 19-29-25 and 4-18-26-344 4-18-26-33-42-49, for training and evaluation. which contains 10 robotic manipulation tasks. Additionally, we propose to leverage D4RL tasks [18] to construct the CL settings with diverse state and action spaces, where the task datasets in D4RL. (Hopper, Walker2d, and HalfCheetah) contains difficulty settings (random, medium, expert, medium- expert, medium-replay, and full-replay).
CW, we adopt the task setting of CW10,
5.2 Evaluation Metrics
Considering the various reward structures of different environments, we should adopt different performance comparison metrics. For Ant-dir, we adopt the average episodic return over all tasks as the performance comparison, i.e., the final performance P mean (>, I2;) is calculated based on the task i's return R,. In the CW environment, previous works 995 6
usually adopt the success rate V as the performance metric. Thus, adopt the average success rate all tasks as the final performance, ie., = mean(E: Mz). . For the D4RL environments, we use the normalized score & 19 42)
as R. R--Roantee Rowndom
the metric to calculate the performance P = mean( Ce Φ.), where & Respert Rrandom * 100. Usually, use the interface of these environments to obtain the score expediently.

---

VQ-CD
PackNet
CRIL
CuGRO
DGR
step (05)
on,
Datakkk4
eval on task42
1: task 18
eval task34
2: task 26
evel .n, laski8
evel oon Jesket
S: task
Legend
Multitask
t-DGR
Finetune
EWC
Figure 3: The comparison of VQ-CI> and several baselines on the continual tasks setting (Ant-dir task 4-18-26-34-40 24--4449.... We train on each task for 500k steps. We report the normalized evaluation performance VQ-CD the top left corner, where the coordinates, e.g., task 4, represent evaluation on task 4 at different training tasks. To show the overall performance all tasks, show the normalized evaluation performance on the six tasks after finishing the training at the right part.
Table 2: The feature difference between the aligned features produced by the space alignment module. We randomly sample thousands aligned state and action features to calculate the difference.
Method
VQ-CD
AE-CD
feature difference
stute difference
action difference
state difference
action difference
[Hopper-fe, Walker2d-fr Halfcheetah-fi
8.8311
4.54
11.31
14.062209
[Hopper-mr; Walker2d-mr. Halechetthh-ror
9.03
4,45
48. 18..14
15.391371
[Hopper-m. Walker2d-m. httei2ddm..aaaeheeeeeeee Halfcheettab-mm
8.53+115
4.22+am
42.27+2429
13.59#26
[Hopper-me, Walker2d-ma Halfchectah---m
8.93+200
+0.56
57.91 43654
13.93+320
5.3
Baselines
We select various representative CL baselines, which can be classified into diffusion-based and non- diffusion-based methods. example, the diffusion- methods consist of CRIL (2I), DGR (80) t-DGR [99], MTDIFF [25). CuGRO [63]. CoD 355. and CoD variants. The non-diffusion-banl based methods include L2M [791, EWC (55), PackNet [66], Finetune, IL-rehearsal [88), and Multitask. From the perspective of mainstream CL classification standards, these baselines can also be sorted as rehearsal- methods (L2M, EWC, CuGRO, and Finetune), and structure- based methods (PackNet, Multitask, and MTDIFF).
methods (CRIL, DGR, t-DGR, CoD, and IL-rehearsal), regularization. based
5.4 Experimental Results
In this section, we mainly separate the experimental settings into two categories, the traditional CL settings with the same state and action spaces and the arbitrary CL settings with different state and action spaces, to show the effectiveness of our method. Besides, we also investigate the influence of the alignment techniques, such auto-encoder, variational auto-encoder, vector-quantized variational auto-encoder (we adopt this in our method). More deeply, we investigate how to deal with the potential demand for additional tasks beyond the pre-defined task length by releasing nonsignificant masks or expanding more available weights (Refer to Appendix B.7 for more details.).
The traditional CL settings correspond to the first question we want to answer: Can VQ-CD achieve superior performance compared with previous methods in the traditional CL tasks?
We use Ant-dir and Continual World 98 to formulate the continual task sequence, where we select two types of task sequence Ant-dir and "hammer-v2, push-wall-v2 faucet-close-v2 push-back-v2. stick-pull-v2, handle-press-siee to construct CW10 CL setting. For simplicity, we do not align the state and action spaces with quantized alignment techniques because the traditional CL setting naturally has the same spaces.
push-v2, shelf-place-v2, window-close-v2. reg-unplug-side-v22

---

[Hopper-fr, Walker2d-Ff Halfcheetah-fri
[Hopper-mr, Walker2d-mr, Halfcheetah-mal
40
mean episode score
mean episode score
20
20-
0
5.0
15.0
10.0
15.0
train step (e5)
train step (e5)
[Hopper-m, Walker2d-m, Halfcheetah-m)
[Hopper-me, Walker2d-me, Halfcheetah-me)
60
80
mean episode score
mean episode score
40
40
20-
20
5.0
10.0
15.0
5.0
10.0
15.0
step (e5)
train step (e5)
VQ-CD
PackNet
Finetune
DGR
Multitask
b-DGR
CuGRO
EWC
CRIL
Figure 4: The comparison on the arbitrary CL settings. We select the D4RL tasks to formulate the CL task sequence. In order to align the state and action spaces,
the pre-trained QSA module (the same as our method) to provide aligned spaces during training. The experiments are conducted on various dataset qualities, where the results show that our method surpasses the baselines not only the expert datasets but also at the non-expert datasets, which illustrates the wide task applicability of method. The datasets characteristic "fr", "mr", "m", and "me" represent "full-replay", "medium- replay". "medium", and "medium-expent", respectively. "Hopper", "Walker2d", and "Halfcheetah" the different environments.
The comparison results between our method and several diffusion- -ased baselines are shown in Table where these baselines include rehearsal-based (CoD and IL-rehearsal), parameter-sharing (CoD-LoRA), multitask training (Multitask CoD and MTDIFF), and representative diffusion RL methods (Diffuser-w/o rehearsal, CoD-RCR, and DD-w/o rehearsal). Our method surpasses all baselines in the Ant-dir setting by large margin in Figure which directly shows the effectiveness of our method. As another experiment of CL setting with the same state and action spaces, we report the results in Figure I1] Compared with the upper bound performance of Multitask, our method reaches the same performance after continually masters new tasks and sustains the performance, while the basclines show varying degrees of performance attenuation, which can be found in the fluctuation of the curves. Moreover, the final performance difference between character, which can be reflected by the overall upward trend these curves. More experiments of shuffling task orders can be found in Appendix B.2
training. With increase in new tasks, our method
method and the Multitask method indicates the forgetting
The arbitrary CL settings correspond the second question we want answer: Can we use the proposed space alignment method to enable VQ-CD to adapt to incoming lasks with various spuces?
To answer the above question,
select D4RL formulate the CL task sequence because of the various state and action spaces, and the results are shown in Figure 4 Considering the dataset qualities of D4RL [18), we choose different dataset quality settings and report the ft.IIrandom
mean episode score that is calculated with Ruspert-Rreae Rrandom
100. Generally, from the four sub-
experiments (a, b, c, and d), we can see that our method (VQ-CD) surpasses these baselines in all CL settings. Especially in the CL settings (Figure and b), where the datasets contain

---

qualiuality trajectories, our method achieves large performance margin even compared with the Multitask method. We can attribute the reason to the return-based action generation that helps our method distinguish different quality trajectories and generate high-reward actions during evaluation, as well as the selective weights acti- vation that can reserve the previous knowlede ledge and reduce forgetting. While other methods just possess the ability to continue learning and lack the ability to separate different qualities and ac- tions, thus leading poor performance. For trajectory qualities that are similar across the datasets (Figure 4 and d), improvement gains between our method and baselines. However, it should be noted that method can still reach better performance than other baselines. Apart from the pre-trained QSA alignment, we also conduct experiments (Fig- ure [2) on baselines that adopt padding align state and action spaces in Appendix B.6
WA MUFCD
lower
(Hooper mnc. Frendc
Vullened
The ablation study of space alignment module and diffusion network structure. For each type of ablation study, we fix the other same and retrain the model on four D4RL CL settings.
Figure 5:
5.5 Ablation Study
In this section, we want to investigate the influence of different modules of VQ-CD. Thus, the experiments contain two investigation directions: space alignment module ablation study and diffuser network structure ablation study. To show the importance of vector quantization. we change the space alignment module with auto-encoder (AE) and variational auto-encoder (VAE). Based on this modification, we retrain our method and report the results in Figure 5 The results show significant improvements the D4RL CL settings, illustrating the importance and effectiveness of vector quantization in our method. Compared with AE-CD, VAE-CD performs poorly on all D4RL CL settings. The reason lies in that the implicit Gaussian constraint on each dimension may hurt the space alignment. Compared with the codebook in VQ-CD, AE-CD may cause bigger difference between aligned features produced by AE (shown in Table 2). posing challenges for the diffusion model to model the distribution of the aligned features and leading low performance. As for the diffuser network structure, we conduct the selective weights activation on the mip-based and unet-based diffusion models. The latter structure is beneficial to making decisions with temporal information inside the trajectories, leading to higher performance evaluation.
5.6 Parameter Sensitivity Analysis
When performing on the aligned feature with diffusion models, the hyperparameters of the state and action of the quantized spaces alignment module matter. Usually, the complexity of states more significant than the actions, so the codebook size controls the performance of reconstruction. Thus, we investigate the effect of different codebook sizes and report the results in Figure 6 Obviously, a small codebook size limits performance, and negative effect arises when it exceeds certain value, such as 512. Additionally, we also conduct the experiments parameter sensitivity analysis on actions latent and put the resutls in Figure Appendix B.3
state 128
Mate 256
state, 512
state 768
state 1024
(Hosper Walker2d i. Halchesch
Hoppper m. Wilke 20hm. Halchesthhin mn)
mean apiode
g
train step (e5)
train step (e5)
Figure 6:
The effeetts of different codebook sizes about the states.

---

Discussion
The Interplay of VQ and CD. In this paper, we investigate broadening the application scenarios of the same state and action spaces to tasks of arbitrary state and action spaces by space alignment. Vector quantization verified as one effective way to achieve space alignment compared with AE, VAE, and padding. Furthermore, we adopt the diffusion model to perform continual learning based on VQ due its strong model expressiveness and competitive performance. The ablation study illustrates that integrating VQ and CD induces the proposed powerful method VQ-CD.
The Intuition of Constraint in QSA Module. In Equation We add a constraint to encourage more concentrated distribution of the quantized representation vectors as shown in Table 2 which benefits the diffusion model in learning the data distribution in a limited range [30 4 However, this may not necessarily benefit other methods that do not focus on modeling distributions (Refer to Figure 4 because concentrated representations can make originally dissimilar state and action vectors from different tasks appear more similar, making them harder to distinguish and learn. We use the clip operation rather than convert the constraint to a penalty because our goal ensure that the magnitude of the quantized representation vectors does not exceed a certain value, rather than minimize the norm of the constraint. More discussion can be found in Appendix C
Conclusion and Limitation
In this paper, we propose Prettr-uuantized Continual Diffuser, called VQ-CD, which opens the door to training on any CL task sequences. The advantage of this general ability to adapt to any CL task sequences stems from the two sections of our framework: the selective weights activation diffuser (SWA) module and the quantized spaces alignment (QSA) module. SWA preserves the previous knowledge by separating task-related parameters with task-related masking. QSA aligns the different state and action spaces so that we can perform training in the same aligned space. Finally, we show the superiority of our method by conducting extensive experiments, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). The results illustrate that our method achieves the SOTA performance by comparing with 17 baselines on 15 continual learning task settings. For limitations, our method belongs to task-aware CORL, where an additional mechanism is
CORL and is not suitable for task boundary-agnostt needed to detect whether a task change has occurred. Clearly, this demands extra task similarity measurement mechanisms for detection. This is currently a limitation of our approach. However, we are confident that further progress will be reflected in our future research.
Acknowledgement
We would like to thank Lijun Bian for her contributions to the figures and tables of this manuscript. We thank Siyuan Guo for his contributions to the writing suggestions of this manuscript. This work is supported in part by the National Key R&D Program of China (No. 2023YFF0905400,
No. 20212D01 12500); the National Natural Science Foundation of China (No. 624761 10, No. U2341229); the National Key R&D Program of China (No. YFFO99000,, No. 2021ZD01 12500); the Key R&D Project of Jilin Province (No. 20240304200SF); NSFC Grant (No. 62576364).
10

---

References
[1] David Abel, André Barreto, Benjamin Van Roy. Doina Precup, Hado van Hasselt, and Satinder Singh. A definition of continual reinforcement learning. arXiv preprint arXiv:2307.1100 2023.
[2] Suzan Fce Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution generalization in offline reinforcement learning. IEEE Robotics and Automation Letters, 2024.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International conference on machine learning, pages 104-114. PMLR, 2020.
[4] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making arXiv preprint arXiv:2211.1663: 2022.
[5] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11254-11263, 2019.
[6] Nishanth Anand and Doina Precup. Prediction and control in continual reinforcement learning. arXiv preprint urXiv:2312.11666 2023.
[7] Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594. PMLR, 2023.
[8] Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33: 14879-14890, 2020.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[10] Arslan Chaudhry, Mare Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Effi- cient lifelong learning with a-gem. arXiv preprint arXiv:1812.00422 2018.
[11] Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401.02644 2024.
[12] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fieelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
[13] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems,
1353-183633, 2020.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
[15] P Kingma Diederik. Adam: A method for stochastic optimization. (No Tite), 2014. 2014.
[16] Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model. arXiv preprint arXiv:2402.03570, 2024.
[17] Mohamed Elsayed and A Rupam Mahmood. Addressing loss of plasticity and catastrophic forgetting in continual learning. arXiv preprint arXiv:2404.00781, 2024.
[18] Justin Fu, Aviral Kumar, Ofr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[19] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.

---

[20] Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[21] Chongkai Gao, Haichuan Gao, Shangqi Guo, Tianren Zhang, and Feng Chen. Cril: Continual robot imitation learning via generative and prediction model. In 2021 JEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 6747-5754. IEEE, 2021.
[22] Rui Gao and Weiwei Liu. Ddgr: Continual learning with deep diffusion-based generative replay, In International Conference on Machine Learning, pages 10744-10763. PMLR, 2023.
[23] Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pages 7513-7530. PMLR, 2022.
[24] Assaf Hallak and Shie Mannor. Consistent on-line off-policy evaluation. In International Conference on Machine Learning, pages 1372-1383. PMLR, 2017.
[25] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang. Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is effective planner and data synthesizer for multi-task reinforcement learning. arXiv preprint arXiv:2305. 18459, 2023.
[26] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang. Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems, 36, 2024.
[27] Longxiang He, Deheng Ye, Junbo Tan, Xueqian Wang, and Li Shen. Robust policy expansion for offline-to-onie tonoine rl under diverse data corruption. arXiv preprint arX1v:2509.2474, 2025.
[28] Charles A Hepburn and Giovanni Montana. Model-based trajectory stitching for improved behavioural cloning and its applications. Machine Learning, 113(2):647-674, 2024.
[29] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 12Xi5 2022.
[30] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 16100,668111181 2020.
[31] Jifeng Hu, Yanchao Sun, Hechang Chen, Sili Huang, Yi Chang, Lichao Sun, et al. Distribu- tional reward estimation for effective multi-agent deep reinforcement learning. Advances in Neural Information Processing Systems, 35: 15:12619-1262,, 2022.
[32] Jifeng Hu, Yanchao Sun, Sili Huang, Yuan Guo, Hechang Chen, Li Shen, Lichao Sun, Yi Chang, and Dacheng Tao. Instructed diffuser with temporal condition guidance for offline reinforcement learning. arXiv preprint arXiv:2306. 04875, 2023.
[33] Jifeng Hu, Sili Huang. Siyuan Guo, Zhaogeng Liu, Li Shen, Lichao Sun, Hechang Chen, Yi Chang, and Dacheng Tao. Decision flow policy optimization. arXiv preprint 0tti:25005.03500 2025.
[34] Jifeng Hu, Sili Huang, Zhejian Yang, Shengchao Hu, Li Shen, Hechang Chen, Lichao Sun, Yi Chang, and Dacheng Tao. Analytic energy guided policy optimization for offline reinforce- ment learning. arXiv preprint 2221022505.0182,, 2025.
[35] Jifeng Hu, Li Shen, Sili Huang, Zhejian Yang. Hechang Chen, Lichao Sun, Yi Chang, and Dacheng Tao. Continual diffuser (cod): Mastering continual offline with experience rehearsal. IEEE Transactions Neural Networks and Learning Systems, 2025.
[36] Shengchao Hu, Ziqing Fan, Chaoqin Huang. Li Shen, Ya Zhang, Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement learning. In International Conference on Machine Learning, pages 19165-19181. PMLR, 2024.
[37] Shengchao Hu, Ziqing Fan, Li Shen, Ya Zhang, Yanfeng Wang. and Dacheng Tao. Harmodt: Harmony multi-task decision transformer for offline reinforcement learning. In International Conference on Machine Learning, pages 19197. PMLR, 2024.
12

---

[38] Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, and Dacheng Tao. On transforming reinforce- ment learning with transformers: The development trajectory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(1 (6(12):85008839) 2024.
[39] Shengchao Hu, Yuhang Zhou, Ziqing Fan, Jifeng Hu, Li Shen, Ya Zhang, and Dacheng Tao. Continual task learning through adaptive policy self-composition. arXiv preprint 1rXiv:241..11366 2024.
[40] Shengehao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer for offline reinforcement learning. SCIENCE INFORMATFOOM SCIENCES, 68(6), 2025.
[41] Shengchao Hu, Wanru Zhao, Weixiong Lin, Li Shen, Ya Zhang, and Dacheng Tao. Prompt tuning with diffusion for few-shot pre- rre-traine policy generalization. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pages 2556-2558. 2025.
[42] Kaixin Huang, Li Shen, Chen Zhao, Chun Yuan, and Dacheng Tao. Solving continual offline reinforcement learning with decision transformer. arXiv preprint urXiv:2401.086 1.04488 2024.
[43] Sili Huang, Bo Yang, Hechang Chen, Haiyin Piao, Zhixiao Sun, and Yi Chang. Ma-trex: Mutli-agent trajectory-ranked reward extrapolation via inverse reinforcement learning. In International Conference on Knowledge Science, Engineering and Management, pages 3-14. Springer, 2020.
[44] Sili Huang, Yanchao Sun, Jifeng Hu, Siyuan Guo, Hechang Chen, Yi Chang, Lichao Sun, and Bo Yang. Learning generalizable agents via saliency-guidedd features decorrelation, Advances in Neural Information Processing Systems, 96:39363-39381, 2023.
[45] Sili Huang, Hechang Chen, Haiyin Piao, Zhixiao Sun, Yi Chang, Lichao Sun, and Bo Yang. Boosting weak-to-strong agents in multiagent reinforcement learning via balanced ppo. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[46] Sili Huang, Jifeng Hu, Zhejian Yang, Liwei Yang, Tao Luo, Hechang Chen, Lichao Sun, and Bo Yang. Decision mamba: Reinforcement learning via hybrid selective sequence modeling. Advances in Neural Information Processing Systems, 77:72688-72709 2024.
[47] Sili Huang, Jifeng Hu, Hechang Chen, Peng Cui, Haiyin Piao, Lichao Sun, and Bo Yang. Generalizable causal reinforcement learning for out-of-distributinn environments. IEEE Transactions on Industrial Informatics, 2025.
[48] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint 20X.::2200.09911,, 2022.
[49] Nan Jiang and Lihong Li. Doubly robust aff-ppoicy value evaluation for reinforcement learning. In International conference on machine learning, pages 652 661. PMLR, 2016
[50] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[51] Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. arXiv preprint 102.:00020002555 2019.
[52] Samuel Kessler, Jack Parker- Holder, Philip Ball, Stefan Zohren, and Stephen J Roberts. Unclear: straightforward method for continual reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning. 2020.
[53] Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J Roberts. Same state, different task: Continual reinforcement learning without interference. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 7143-7151, 2022.
[54] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1xXi:14122.6980 2014.
13

---

[55] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trrbska-Barwinske et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (1<(13)3521-3526) 2017.
[56] Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu. Parameter-level soft- masking for continual learning. In International Conference on Machine Learning, pages 14922-15055. PMLR, 2023.
[57] Lukasz Korycki and Bartosz Krawezyk. Class- Class-incremental experience replay for continual learning under concept drift. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3649-360 -658,, 2021.
[58] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06166 2021.
[59] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1 1179-1191, 191, 2020.
[60] Dongsu Lee, Chanin Eom, and Minhae Kwon. Ad4rl: Autonomous driving benchmarks for offline reinforcement learning with value-based dataset. arXiv preprint arXhv:2404.02429, 2024.
[61] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference Robot Learning, pages 1702-1712. PMLR, 2022.
[62] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review. and Perspectives on Open Problems, 5, 2020.
[63] Jinm ei Liu, Wenbin Li, Xiangyu Yue, Shilin Zhang, Chunlin Chen, and Zhi Wang. Continual offline reinforcement learning via diffusion-based dual generative replay. arXiv preprint arXiv:2404.10662. 2024.
[64] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference Machine Learning, pages 22825-22855. PMLR, 2023.
[65] Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024.
[66] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018.
[67] Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. arXiv preprint arXiv:2312.08977 2023.
[68] Christos N Mavridis and John S Baras. Vector quantization for adaptive state aggregation in reinforcement learning. 2021 American Control Conference (ACC), pages 2187-2192. IEEE, 2021.
[69] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-533 2015.
[70] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint 1rXiv:1912.02077,,00 2019.
[71] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 2020.
[72] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference Machine Learning, pages 8162-8171. PMLR, 2021.
14

---

[73] Tim Pearce, Tabish Rashid, Anssi Kanervisto. Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301. 10677, 2023.,0023.
[74] Liangzu Peng, Paris Giampouras, and René Vidal. The ideal continual learner: An agent that never forgets. In International Conference on Machine Learning, pages 27585-27610. PMLR, 2023.
[75] Daiqing Qi. Handong Zhao, and Sheng Li. Better generative replay for continual federated learning. arXiv preprint 2x02:22302.13001,, 2023.
[76] Yunpeng Qing, Jingyuan Cong, Kaixuan Chen, Yihe Zhou, Mingli Song, et al. Advantage- aware policy optimization for offline reinforcement learning. arXiv preprini arXiv22403.07262, 2024.
[77] Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv: 1063, 2018.
[78] Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint
2015.
[79] Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, and Sepp Hochreiter. Learning to modulate pre-trained models in rl. Advances in Neural Information Processing Systems, 36, 2024.
[80] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[81] Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe. Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offfine reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
[82] James Seale Smith, Yen- Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. arXiv preprint arXiv:2304.0002) 2023.
[83] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint xrxiv:2010.02502, 2020.
[84] Yanchao Sun, Shuang Ma. Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor. Smart: supervieer multi-task pretraining with control transformers. arXiv preprint arXiv:2301. 230.09330.09166, 2023.
[85] Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 185-):185-11999 2022.
[86] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[87] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
[88] Weikang Wan, Yifeng Zhu, Rutav Shah, and Yuke Zhu. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 537-544. IEEE, 2024.
[89] Jianhao Wang, Zhizhou Ren, Teny Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.0 10062, 2020.
[90] Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision transformer for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15706-15714, 2024.
15

---

[91] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06 2022.
[92] Zhenyi Wang, Li Shen, Tiehang Duan, Qiuling Suo, Le Fang, Wei Liu, and Mingchen Gao. Distributionally robust memory evolution with generalized divergence for continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[93] Zhi Wang, Chunlin Chen, and Daoyi Dong. A dirichlet process mixture of robust task models for scalable lifelong reinforcement learning. IEEE Transactions Cybernetics, 2022.
[94] Ziyu Wang. Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[95] Maciej Wolezyk, Michat Zajac, Razvan Pascanu, Fukasz Kucinski, and Piotr Mitoć. Continual world: robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 34:28496-285100 2021.
[96] Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 144:27355227400 2021.
[97] Shin'y ya Yamaguchi and Takuma Fukuda. On the limitation of diffusion models for synthesizing training datasets. arXiv preprint arXiv:2311 130000, 2023.
[98] Yijun Yang, Tianyi Zhou, Jing Jiang, Guodong Long. and Yuhui Shi. Continual task allocation in meta-policy network via sparse prompting. In International Conference on Machine Learning, pages 39623-39638. PMLR, 2023.
[99] William Yue, Bo Liu, and Peter Stone. t-dgr: for continual learning in decision making. arXiv preprint arXiv:2401.0255 2024.
A trajectory-based deep generative replay method
[100] Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint arX1v:2210.09241, 2022.
[101] Mengyao Zhai, Lei Chen, Frederick Tung, Jiawei He, Megha Nawhal, and Greg Mori. Lifelong gan: Continual learning for conditional image generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2759-2768, 2019.
[102] Qizhe Zhang, Bocheng Zou, Ruichuan An, Jiaming Liu, and Shanghang Zhang. Split & merge: Unlocking the potential of visual adapters via sparse training. arXiv preprint arXiv:2312.0299. 2023.
[103] Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09072, 2020.
[104] Tiantian Zhang, Xueqian Wang. Bin Liang, and Bo Yuan. Catastrophic interference in reinforcement learning: A solution based on context division and knowledge distillation. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[105] Tiantian Zhang, Zichuan Lin, Yuxing Wang, Deheng Ye. Qiang Fu, Wei Yang, Xueqian Wang, Bin Liang, Bo Yuan, and Xiu Li. Dynamics-adaptive continual reinforcement learning via progressive contextualizatio. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[106] Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models, arXiv preprint ar0iv:2305. 17330, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section andd Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
The source code available at here
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss//nips.c/ppbbicc guiddes/oodeuubnisioonool
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Appendix A.2 Appendix
A.3
Appendix
A.5 and Appendix A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Section 5.4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Appendix A.3 and Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/heeunpp.cccpubbbehehhtiih
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of the package should be provided. For popular datasets, paperswithcode, wwaeecodde.ooomaataeeee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
21

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
22

---

Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:://eeeeiiis.cccoommeeeeee/s for what should or should not be described.
23

---

Appendix of "Tackling Continual Offline RL through Selective Weights Activation on Aligned Spaces"
A Algorithm
A.1 Pseudocode of VQ-CD
Algorithm 1: Vector-Quantizze Continual Diffuser (VQ-CD)
Input:
Noise prediction model eg, inverse dynamic model finn.sh state and action quantized model (0,, Od, 0q), tasks set M, E (1,-.., 7, each task training step Ω max diffusion step K, sequence length Te, state dimension d,, action dimension da reply buffer D,, E {L,... (L....]] noise schedule æo: and Bo:
Output: Eg. fine.w. 0e, 0q. 0g
Initialization: v/, 0g, 0d. and 0g
Separate the state-action trajectories of D,, E (1, [} into state-action sequences with length Te and calculate the discounted returns R2 Er== 7d - for each step t for each task i do // Quantized Spaces Alignment (QSA) Pretraining for each train epoch do for each train step do
Sample states and actions from task i's buffer D; Calculate the quantization loss and reconstruction loss
Updating the parameters 0₂ of fiog (rog(::0e)) 02), , 0a of (ivo(;; 0a), and 0g of fö, (-) by solving the problem of Equation
10
end end
11
12
Save the task i's well-trained fi 0e), fvon(: 02), and få, (-) // Selective Weights Activation (SWA) Diffuser Training Generate the task-related mask M, for task i for each train epoch do for each train step m do
14
16
Sample sequences 7! = (S44T, CutI, RE } +cd from task i's buffer D;
17
18
Obtain the quantized state and action feature o = Sh (iVQ,, (s+; (z)) and
a²,, fá,a, (fiQr,, (at:0.) with the QSA module
19
Train the inverse dynamic model five according to Equation 3
Formulate s² a²_, as sequences -i.0 {s 2514, az 22y.:11T.
20
21
Sample the corresponding discounted returns Ri from task i's buffer D, Sample diffusion time step Uniform(A ()) and return coefficient b ~ B())
22
Sample Gaussian noise ~ N I).c bxxxxxxd.,,
23
24
Obtain 5.24 = V S,2g + -1-60 V1-
25
Perform the forward propagation with Equation
26
Train Eg according to Equation]]
27
end end
Save task i's related models as 6i42,,8 29
30 end
// Weights Assembling
32
Construct new models ég with the same structure as ep
for each task do
34
Extract the task-related parameters W; with mask information M, from 6,62,9
Fill the corresponding task-related parameters W, M, W, into éo
end
24

---

Algorithm 2: Evaluation Process
for For each environmental step in task do Receive the environmental state si Set the return condition R. = 0.8, St,2q
Sa, (RVQ,, (s 0.])
8 Construct Tzgg = [st,2, s1t1, 112,,2
where st ~ ~N(0,)) for 1' > t.
for For from K to do
Calculate é with
Obtain P---1 with Equation
Replace the first state of f with 81,74 end
Extract 1,+11,2g from *gzy
11
Obtain at,z, = finoo (5t,2 : 1041.80) Interact with aj fVQD. (ae, 0a)
12
end
The training of VQ-CD (Pseudocode is shown in Algorithm 1 contains three stages. 1) We first pre-train the QSA module for space alignment, as shown in lines 4-12, where we mainly want to solve the constrained problem of Equation 4 Then, in lines 13-29, for each task i, generate the task-related mask M, followed by a standard diffusion model training process (Refer to Equation ] and Equation 55 the training loss) on the aligned state and action spaces. 3) Finally, we assemble
the task-related weights W, together with the mask information {M,ii [1 []} according to
the training steps for each CL task, and Wii Ω is the weights checkpoints of G=122.0- It is noted that the gre-trannin of the QSA module and the training of the
W =ΣM, o W[i * 22). where
SWA module can be merged together, i.e., for each task we can first train the QSA module related
to task i and then train the SWA module. The source code is available at https://gituu.com//I L
Lel/VVeeto_uunttieee Continual_Diffuun
A.2 Hyperparameters
We classify the hyperparameters shown in Table 3 into three categories: QSA module-related, SWA module-related, and maining-relltted hyperparameters. We use the learning rate schedule when pre- training the QSA module, so the VQ learning rate decreases from le-3 to le-4. In our experiments, the maximum diffusion steps are set as 200, and the default structure is Unet. Usually, it is time- consuming for the diffusion-based model to generate actions in RL. Thus, we consider the speed-up technique DDIM [3] and realize it in our method to improve the generation efficiency during evaluation. For all models, we use the Adam optimizer to perform parameter updating.
A.3 Computation
We conduct the experiments on NVIDIA GeForce RTX 3090 GPUs and NVIDIA A10 GPUs, and the CPU type is Intel(R) Xeon(R) Gold 6230 CPU @ 2. 10GHz. Each run of the experiments spanned about 24-72 hours, depending on the algorithm and the length of task sequences.
A.4 Generation Speed-up Technique
The time and memory consumption of diffusion models is attributed the mechanism of the diffusion generation process that requires multiple computation rounds to generate data (30). Fortunately, previous studies provide useful speed-up strategies to accelerate the generation process [72 83). In
this paper, we adopt DDIM the default generation speed-up technique and reduce the reverse diffusion generation step to 10 compared to the original 200 generation steps. Table we use the CL setting of Ant-dir task-4-18 1--1188--34-42--
as example to compare the time consumption of different generation steps. Compared with the original 200 diffusion steps, we can see that incorporating DDIM will significantly (19.76x) improve the efficiency of generation. In the experiments, we find that 10 diffusion steps setting performs well on performance and generation efficiency. Thus, we set the default sampling speed-up stride to 20, and the diffusion step is 200/20=10 steps.
25

---

Table The hyperparameters of VQ-CD.
Hyperparameter
Value
network backbone
MLP
hidden dimension of QSA module
256
commitment cost coefficient codebook embedding limit state codebook size per task number of state latent state latent dimension action codebook size per task
0.25
3.0
512
10
QSA section
2
512
number of action latent action latent dimension alignment type VQ learning rate
5
2
VQ/AE/VAE
[lc-4,1e-3]
network backbone
Unet/MLP
hidden dimension
256
sequence length T. diffusion learning rate guidance value mask rate condition dropout À max diffusion step K sampling speed-up stride condition guidance sampling type of diffusion
8
3e-4
0.95
SWA section
1/1
0.25
200
20
1.2
DDIM
loss function
MSE
batch size
32
Training
optimizer
Adam
discount factor Y
0.99
Table 4: The comparison of generation speed with different generation steps under the CL setting of Ant-dir task-4- task-4-18-26-32 1--2--6--4444 In the main body of our manuscript, we use the 10 diffusion steps setting for all experiments.
Diffusion steps
200 (original)
100
50
25
20
10
sampling speed-up stride
(original)
8
10
20
Time consumption of per generation (s)
5.73-029
2.88
1.41
0.71
0.58
0.29
Speed-up up ratio
1x
1.99x
4.06x
8.07x
9,88x
19.76x
A.5
Computational Cost Analysis
In T report the GPU memory consumption during the training process. We mainly consider the experiments on the D4RL, Ant-dir, and CW CL tasks. We can change the first block of the diffusion model to make model suitable for longer CL task sequence. For example, expand the dimension length from 512 to 1024 when switching the CL training task from task-10-11519-255 to
We compare the computational cost, including generation time and memory consumption, with diffusion-based methods, such CuGRO, and transformer-based methods, such as 1.2M, and report the results in Table and Table 7 The results show that, compared to the baselines, our method achieves lower time overhead and better performance with similar memory usage.
26

---

Table 5: The GPU memory consumption.
domain
CL task setting
GPU memory consumption (GB)
Hopper-fi, Walkerr2A-ft.Hallchhttaaa
4.583
Hopper-mr. Nalkerz2l-mm.//allchhhctaa--m--
4.583
D4RL
Hopper-m, Walkker2dbm.HHalchhhtaaa--m
4.583
[Hopper-me, Wallkerr---nne//alichhhaaaanee
4.583
task- 10k-100-5-19-5
4.711
Ant-dir
task-4-18-26344442449
5.955
CW
CW10
5.897
Table 6: The computational cost of generation speed with different generation steps in D4RL [Hopper- m, faakerdd-m,Halihheetllaae tasks.
method
basc
VQ-CD
CuGRO
time consumption
5.73
0.29
0.33
speed-up ratio score
1x
19.8
17.4x
45.4
27.6
Table The comparison of GPU memory consumption. We conduct the experiment with NVIDIA GeForce RTX 3090 GPUs and Intel(R) Xeon(R) Gold 6230 CPU @ 10GGZ
GPU
Parameters
Physical training time (h)
domain
CL task setting
Method
overhcad (GB)
Performance
(M)
DARL
Hopper- Wwl.eer2ddff..alfcheeeee
VQ-CD L2M
89.1
55
48.0
D4RL
Hopper-fr, Www.eerr2--f..aalchcceuuea
57.8
92
13.4
D4RL
Ho halporftiWlllkk22-ttaahhhhhhhhe
L2M-large
8.8
96.0
94
16.0
A.6
Baselines Implementation
All the comparison methods used in this paper utilize their official codebases. Specifically,
For L2M, we use the official source code: https//githhbb.cmm/miijj/LLLLLL
For CuGRO,
the official source code: https://gthhb.coomm//UURRRtCCCC
For CoD, we use the official source code: https//giihbbboomm//fftttuonnnuu
Diffuser
For MTDIFF, we use the official source code: http:://ponneeewwwweee/////////a
A.7
Network Details
In the diffusion model (SWA module), we utilize a UNet network structure, incorporating residual connections at both the input and output of each block. Additionally, residual connections are applied between the down- wwwn-sampling and ua-sampling blocks, meaning that the output of the down- sampling block serves the input to the up-sampling block. The convolution kernels in the UNet are one-dimensional, with their shapes corresponding to the shape of the mask matrix.
In the QSA module, there are no shared parameters. The primary purpose of the QSA module is to align the state and action spaces across different environments. Consequently, for different tasks, the internal components of the QSA module, vector quantized encoder (VQE), vector quantized decoder (VQD), and codebook are task-specific, and none of their parameters are shared. Thanks to the alignment provided by the QSA module, the inverse dynamics model in the SWA module can be shared. This is because the state and action spaces of different environments are mapped into an alignment space with the same value range.
27

---

State_ 128
state_256
state __12
State_ 768
state_ _late_1024
[Hopper-fr, fr, Walker2d-ff fr. Halfcheetah-fr)
[Hoppermm, Walker2d-mr, Halfcheetah-mt
2.5
2.5
state loss (e-1)
state loss (e-1)
1.5
2.5
00-0
%
10
15
%
10
train step (e5)
train step (e5)
Figure 7: The QSA module loss under different codebook sizes about states. We explore five codebook size settings: 128, 256, 512, 768, and 1024. The red line represents the experimental codebook size setting for states.
action_3_2
action_5_2
action_ ____
action
action_11 _1__
(Hopper-fr, WalkerZd-fr Halfcheetah-fr]
[Hopper-mr, WalkerZd-mr, Halfcheetah--mi)
10
10
action loss (e-4)
action loss (e-4)
35
35
10
10
15
train step (e5)
train step (e5)
Figure 8: The QSA module loss under different latent numbers about actions. The setting includes 3, 7, 9, and 11, which correspond to the aligned action space sizes 10, 14, 18, and 22, The red line represents the experimental latent numbers setting for actions.
Additional Experiments
B.1 QSA Module Loss Analysis
Under the same hyperparameter settings in Section 5.6 we report the loss of the QSA Module to investigate the effects of codebook size and latent number. For the states, we investigate the influence of codebook size, where we set codebook size as 128, 256, 512, 768, 1024, and select D4RL CL setting [Hopper-fr, Walker2d-fr. Halfcheetah-fr] and [Hopper-mr, Walker2d-mr, Halfcheetah-mr] as example. The results are shown in Figure 7 where we train the QSA module on each task for 5e5 steps. We can see that for states, codebook size of 512 is good enough for aligning the different tasks' state spaces. larger codebook size, such as 768 and 1024 in Figure 7 and b, will not bring significant loss improvements. Smaller codebook sizes not provide sufficient latent vectors to map the state spaces to uniform space.
For the action, we select the latent number to explore the QSA action loss and report the results in Figure 8 We can see the same trend that has been seen in QSA state loss (Figure Though the lower loss value of the more latent number indicates that we should use more action latent vectors, we find that gap between action latent number settings and small when we increase computation resources. Besides, we also see inconspicuous performance gains in the final performance Figure TO which urges us to use 5 as the default action latent number setting. For the action latent vector dimension, directly use 2 as the default setting.
28

---

cc0 tank 4-4-15 44-44-----4
bank 4--8-25-44-2--4
432
200
272
Dlaki stop (6:5
tran stop (65)
Train stop (65)
6co baok 18:4 25 34 42.40
lack 25-4-18
task 34 18-4 42-40
kco
200
train step (e:)
train step (65)
train step (85)
taak 42 12-54-4-88-2--9
13----30 20-2000-
task 45 -0--54---------
420
200
200
200
Irein (K5)
(45)
VO.CD
DGR
bOGR
OUGRO
EWC
CRIL
Figure 9: The experiments of Ant-dir with shuffled task order. We investigate the influence of shuffled task order in the Ant-dir environment, where the experiments include inserting new tasks into the predefined task order -4-18-26-34 42-49 and disrupting the tasks order.
action_3_2
action_5_2
action ___
action_9_2
action_ll_2
[Hopper-fr fr. Walker2d-fr fr, Hallcheetah-11
(Hoppor me. WalkerZd mr. Halfcheethh-my mm]
mean spisode score
mean episode score
40
20
20
%
train step (e5)
train step (e5)
Figure 10: The effects of the number of latent vectors about the actions.
B.2 Experiments of Task Order Shuffling
To investigate the influence of task order in CORL, we choose Ant-dir as the testbed and change the task order for new CL training. We change the task order by inserting new tasks into the predefined task order '4- Figure that our method achieves the best performance almost all CL task orders. The task order will affect the final performance of other baselines. For example, CRIL. performs better in the task orders 'task 18-4-26-34-42-44' and "task 49-42-34-26-1 18-4" than other task order experiments. Another example is PackNet, which achieves the best performance only in the task order 'task
and disrupting the task order. We can see from the results shown in
29

---

VQ-CD
PackNet CuGRO
Finetune
DGR
Multitask
t-DGR
EWC
CRIL
1.0
[Continual world 10 tasks]
0.8
mean success rate
0.6
0.4
0.2
0
train step (e5)
Figure 11: The experiments the CW10 tasks, which contain various robotics control tasks. We train each method on each task for 5e5 steps and use the mean success rate on all tasks as the performance metric. Generally,
the superiority of our method from the above figure.
34- 14-184-22--42499 Different from the baselines, whose performance fluctuates with the changing of task orders, our method (VQ-CD) shows stable training performance no matter what task orders are defined.
B.3 Experiments of Parameter Sensitivity
In Figure 6 we investigate the effect of different codebook sizes and find that small codebook size limits performance, and a negative effect arises when it exceeds a certain value. For the actions, we believe the actions can be decomposed into several small latent vectors, and the number of latent vectors is crucial for reconstructing actions. Similarly, we also see the same trend in Figure T0 which shows that more latent vectors not always better.
B.4 The Benefits of Inverse Dynamics
Following previous studies 4 the inverse dynamics is introduced to produce actions based on state sequence generated by the diffusion model. We choose to model the distribution of the state sequence rather than the state-action sequence on the basis two reasons: 1) Usually, in many robotics control scenarios, the actions often represented as joint torques, which are frequereyyyccy and less smooth, making it hard to model and predict the action sequence. 2)The state is usually continuous in RI., but the mode of action is diverse, such discrete and continuous. Modeling state sequences separately makes the Liffusion-based model more generic to extensive RL. scenarios. Using the diffusion model to model the state sequences and producing actions with the inverse dynamics are not related to accommodating different action spaces across tasks.
To further investigate the benefit of producing actions with inverse dynamics rather than generating (8, a) together with diffusion models, we conduct the experiments modeling state and action sequences together with diffusion models and only modeling state sequences with diffusion models. Table shows that when using inverse dynamics, our method can achieve better performance compared with directly producing action with diffusion models.
B.5 Experiments on Continual World
We select CW10 another experiment of CL setting with the same state and action spaces, where the task number is 10. We report the results in Figure Compared with the upper bound performance of Multitask, our method reaches the same performance after the CL training. With the increase in new tasks, our method continually masters new tasks and sustains the performance, while the baselines show varying degrees of performance attenuation, which can be found the fluctuation of the curves.
30

---

[Hopper-fr, Walker2d-fr, Halfcheetah-fri
[Hopper-mr, Walker2d-ma -m, Halfcheetah-mi
mean episode score
mean episode score
40
20
20
0
0
5.0
10.0
15.0
5.0
10.0
15.0
train step (e5)
train step (e5)
(Hopper-m, Walker2d-m, Halfcheetah-m)
[Hopper-me, Walker2d-me, Halfcheetah-me!
60
80
mean episode score
mean episode score
60
40
20
10.0
15.0
5.0
10.0
15.0
train step (e5)
train step (e5)
VQ-CD
PackNet
Finetune
DGR
L2M
t-DGR
CuGRO
EWC
CRIL
Multitask
Figure 12:
The comparison on the arbitrary CL. settings. We select the D4RL tasks to formulate the
CL task sequence. We leverage state and action padding align the spaces. The experiments conducted on various dataset qualities, where the results show that our method surpasses the baselines only the expert datasets but also at the non-expert datasets. The datasets characteristic "fr", "mr", "m", and "me" represent "full-replay", "medium-replay" "medium", and "medium-expert", respectively. "Hopper", "Walker2d", and "Halfcheetah" are the different environments.
Table The comparison of producing actions with the diffusion model and the inverse dynamics.
producing action with diffuusion model
producing action with inverse dynamics
lask
Ant-dir 4-1-2663344422
498.2
524.1
[Hopper-m, Walker2ddm,Halichhee
39.5
45.5
B.6 Experiments of Baselines Equipped QSA
In Section 5.4, report the comparison of method and baselines in the arbitrary CL settings, where in the D4RL CL settings, we adopt the pre-trained QSA module to align the state and action spaces. Apart from the pre-trained QSA module, we can also use the state and action padding to align the different state and action spaces. In Figure [2] we report the results of baselines equipped with state and action padding. From the results, we can also see that our method still achieves the best performance compared with these basclines. Considering the results of Figure 12] Figure 5 (VQ-MLPCD), and Figure 4 (VQ baselines), we can see the importance of complementary sections: QSA and SWA.
B.7 Supporting Tasks Training Beyond the Pre-defined Task Sequence
After training on pre-defined task sequences, we may hope the model has the capacity to support training on potential tasks, which means that we need more weights or weight masks. Releasing weight masks that are used to learn previous tasks is straightforward choice when the total weights fixed. We conduct the experiments of mask pruning Ant-dir 'task 4-
and
31

---

500
performance 18 prune
task performance task mask prune
maan episode return
mask rate 2
mask prune rate (16
episnde
3.0
200
2.0
200
2.0
100
1.0
100
1.0
15
01
23 mask prune threshold (e-3)
27
mask prune threshold (e-3)
500
5.0
1000
task 34 performance
maan episode return
400
task 26 performance prune
360
task 34 mask prune
mask prume rate (%)
4.0
episode
3.0
200
2.0
mask
800
2.0
100
1.0
200
0.0
1]
15
mask prune threshold (e-3)
mask prune threshold (e-3)
500
5.0
task 49 performance 49 prune
mean apisade return
task 42 mask
(60
400
4.0
mask prune rate (%0
200
4.0
epicade
3.0
300
200
mask
100
1.0
1.0
0.0
0.0
11 prune threshold (e-3)
mask threshold (e-3)
Figure 13: The mask pruning experiments of Ant-dir 'task 1--18-66-44422444 We investigate the task pruning according the absolute weight values, i.e., we release the weights train on potential new tasks according to the mask prune threshold.
Table 9: The comparison of time consumption per update between sparse and dense (normal) optimizers. We compare these two types of optimizers on the CL settings and find that when we first the normal optimizer, such as Adam, train the model and then use weights assembling obtain the final model, the total physical time consumption is significantly smaller than sparse optimizer (e.g.. sparse Adam).
time consumption update (s) sparse optimizer
domain
CL task setting
dense optimizer
0.089
0.198++0224
hetpre/wwaaaaleeodo..uccccchaaa mr]
0.096
0.1977::.223
D4RL
Hopper-m, hatppemmwaaallee2--m--lichhhhhhe
0.089
0.195:10:224
0.090+0.223
0.206++0.225
task-10-15-19-25
0.062+00061
0.239+40282
Ant-dir
task-4-18-26-34442.99
0.064
0.21410270
CW
CW10
0.061 +0.06
0.218
report the performance and weight mask prune rate when pruning weight masks according to certain absolute value thresholds in Figure T3] The results illustrate that we can indeed release some weight masks under the constraint of preserving 90% or more performance compared with the unpruned model. On the other hand, we can also that this mask pruning method can only provide finite
32

---

Mask visualization: example based on [Hogper-m, Walker2d-m, Halfchoctah-m)
Dat Jor dinst anaili
Mall anfine
Figure 14: Mask matrices visualization. select [Hopper-m, Walker2d-m, Halfcheetah-ml as an example to report the mask results. For each mask matrix, we only draw the first 100 channels of the weights mask matrix if the mask matrix is too large.
capacity for tasks beyond the predefined task sequence. We postpone the systematic investigation of mask pruning to future work.
B.8 Time Consumption of Different Optimizers
In the CL. settings of our experiments, we compare two types of optimizers and find that when we first use the normal optimizer, such as Adam, to train the model and then use weights assembling to obtain the final model, the total physical time consumption is significantly smaller than sparse optimizer (e.g., sparse Adam). Thus, we propose the weights assembling to obtain the final well-trained model after the training rather than suffering huge time burden of sparse optimizer during the training.
B.9 Mask Visualization
We select [Hopper-m, Walker2d-m, Halfcheetah-ml to visualize the weights mask of our method in Figure [4] To make it easy to show the mapping relation between masks and the weights, we draw the network structure and mask matrices, where we only report the first 100 channels the mask matrices.
B.10 Alignment Space Visualization
In order to further demonstrate the effectivenesss of our method. We conduct the visualization experiments of aligned state feature and report the visualization results in Figure 15 From the experimental results, we can see that the state features learned by the AE method are not well well-mapped to separate regions but are instead mapped multiple areas. In contrast, the features obtained by our method are better partitioned into individual regions, which is more conducive for the model to capture the data distribution.
C Further Discussion of Experiments
The Interplay of VQ and CD. In this paper, we investigate broadening the application scenarios of the same state and action spaces to tasks of arbitrary state and action spaces by space alignment. Vector quantization is verified one effective way to achieve space alignment compared with AE,
33

---

Hopper-m, Walker2d-m, m, Halfcheetah-m
Hopper-m, Walker2d-m, mm Halfcheetah-m
Peoser- walaerZam halkoeetthiv
walkare--n
100
50
50
-50
-103
-100
-100
-100
150
-150
100
AE-CD
VQ-CD
Hopper-me, Walker2d me, Halfcheetah-me
Hopper-me, Walker2d-me, Halfcheetah-me
hosparen
walkr2d
Walle saltchettahvm
haPtheetst--m
303
50
0-
-50
-103
-100
-150
-100
-350
230
AE-CD
VQ-CD
Figure 15: Visualization of aligned state feature. We use the QSA module to process the different state spaces and align them in the same space. Then we use t-SNE 87 to visualize aligned state features.
VAE, and padding. Furthermore, we adopt the diffusion model perform continual learning based on VQ due to its strong model expressiveness and competitive performance. The ablation study illustrates that integrating VQ and CD induces the proposed powerful method VQ-CD.
The Intuition of Constraint in QSA Module. In Equation (4). We add a constraint to encourage a more concentrated distribution of the quantized representation vectors as shown in Table 2 which benefits the diffusion model in learning the data distribution in a limited range [30] 4 However, this may not necessarily benefit other methods that do focus on modeling distributions (Refer to Figure 4 because concentrated representations can make originally dissimilar state and action vectors from different tasks appear more similar, making them harder to distinguish and learn. We use the clip operation rather than convert the constraint to penalty because our goal is to ensure that the magnitude of the quantized representation vectors does not exceed certain value, rather than minimize the norm of the constraint.
Further Discussion of Experiments. In Figure 4 (a) and (b), we can that VQ-CD surpasses Multitask. The reason as follows. 1) The Action Quality Discrimination Ability Differences: The datasets contain trajectories collected from the entire training process, i.c., from a random policy to well-trained policy. Our method leverages accumulated discounted returns to guide the generation of state sequences, encouraging the generation of higher-return state sequences. Consequently, the actions generated by the inverse dynamics model also yield higher returns. In contrast, Multitask does not currently incorporate returns, resulting lower performance. In Figures and (d), the variance of trajectory returns in the dataset is smaller, allowing Multitask to achieve better learning outcomes. 2) Architecture Differences: The multitask baseline implemented using the MLP architecture, which is often insufficient to model the complex state-action mappings across diverse tasks. In contrast, our VQ-CD method leverages a diffusion-based policy model, which has demonstrated superior expressiveness and stability in high-dimensional generative modeling, This difference leads
34

---

Loss
Current training part
Current training dataset
Waiting for train
QSA module loss
Train
SWA A module
L LSs (x;0,
switch to
VQD I
VODI
task
Tasks datasets
Task dataset
QSA module
d=6
d-6
Task i dataset
Codebook
Task I dataset
VQE
VQE
VQE
switch next task
Task I dataset
Figure 16: A graphical depiction of QSA training. From the figure, it is intuitively clear that the training of the QSA module can fully adhere to the CL training setup.
to the performance differences in the experiments. 3) Representation Ambiguity Across Tasks: As shown in Table the QSA module maps task-specific ilate-action pairs into shared aligned latent space, where the distances between different tasks' state- and action-aligned vectors are relatively small. This results high semantic proximity across tasks. For Multitask training, all tasks' state- and action-aligne vectors are blended together, which makes it challenging to learn the mapping of state-aligned vector to the correct action-aligne cetion-aligned vector, leading degraded performance. In contrast, VQ-CD separates alignment with QSA and generation via the conditional diffusion model, which enables more precise modeling and avoids such interference.
The Motivation of Adopting Diffusion Models. We select diffusion models as our foundational method primarily based the following reasons:
Natural support for multi-modal action distribution modeling: Diffusion models can effectively avoid the limitations of traditional Gaussian models that cannot model multi-peak action distribu- tions.
Powerful model expressiveness: Diffusion models support the modeling of complex data or
trajectory distributions.
Stable log-likelihood training: Diffusion models can effectively prevent mode collapse and training instability issues.
Competitive model performance: Diffusion-based RL. methods have also shown huge potential in many robotic control scenarios.
SWA is specifically designed for diffusion models with oic-dimensional convolutional structures because diffusion models exhibit different dependency pattern: every output of a given channel is influenced by all weights of the convolution kernel in that channel due to the weight-sharing nature of convolutions. Thus, traditional output neuron masking is not suitable for diffusion models with one-dimensional convolutional structures because we can not enable task-related parameters through masking certain output neurons. However, SWA applies masking directly on convolution kernel parameters, allowing us to selectively activate neurons in the convolution kernel to control the training of task-related parameters. QSA is applicable to any model. Different state and action spaces lead to significant distribution differences between tasks, making it impossible to use single diffusion model for learning. QSA aligns these varying state and action spaces into the same latent space, enabling effective continual training under the same diffusion backbone.
D Discussion of Future Research Directions
Adaptation to Dramatic Shifts across Tasks. Our work currently focuses on tasks with certain similarities, rather than dramatic shifts in state representation formats across tasks. Usually, the
35

---

applicability of continual learning across tasks with completely different state spaces is quite narrow,
and previous studies rarely explore this area [85]. The purpose of continual learning is to progressively master new tasks by discovering common knowledge between tasks. There are several substantial challenges facing high-dimensional image-based observations, particularly in terms of representation alignment, codebook generalization, and action generation. 1) For representation, we can introduce visual encoder, c.g. VIT, to extract compact latent representations from images before passing them into the diffusion models, thereby aligning the modalities. 2) For the codebook, the QSA module can be modified to support modality-specifee encoders while maintaining a shared codebook space or adopting cedaaliyy-ooditionna quantization. 3) For the inverse dynamies, if to directly produce actions with image-like states, then we need to upgrade inverse dynamics to a ssiioncondditione architecture.
inverse dynamics
Theoretical Analysis about the QSA and SWA. We can see that the current version lacks theoretical proof to demonstrate that space-aligned representations can effectively extract common knowledge between tasks. However, our method has a theoretical foundation for the representation In paper, the QSA module maps tasks' states and actions into a shared discrete latent space using the codebook. This process enables structural alignment between otherwise heterogeneous observation spaces, making it easier for the diffusion model to generalize across tasks. We can obtain that under Robbins-Moon stepsizes and Lipschitz gradients, the expected gradient of the QSA loss lim,- [[[ Vos. ||] vanishes as the training iteration
with vector quantization 7
68
increases, where n is the training iteration [86].
For the convergence proof of SWA, according to the previous studies of diffusion model [30] the denoising sctre-matching loss (L()) [F[l|c co (T log-likelihood. Under the Lipschitz constraint, the loss landscape is smooth, and optimization using Adam or SGD with diminishing learning rates satisfies the Robbins-Monro conditions. Thus, the training process converges
2)) is a lower bound on the negative
stationary point of the objective.
Extension to General Online RL Tasks. Our method is currently designed for the offline RL setting primarily due to the multiple-step sampling process inherent in diffusion models, which leads to significant computational latency during interaction in online RL. To enable online RL extensions in the future, several promising research lines can be explored: 1) Parallelized sampling techniques to reduce per-decision latency. 2) Advanced samplers (e.g.. DDIM and DPM-solver) to accelerate generation. 3) Distilling the diffusion model into the consistency model that can realize single-step generation.
36


Reviews:
[{"review_id": "AHTtHdPGwy", "reviewer": "Reviewer_tiFq", "summary": "The paper addresses the challenge of continual multi-task RL where an agent must learn multiple tasks without forgetting previously known knowledge. The authors propose a novel framework Vector-Quantized Continual Diffuser, named VQ-CD,\n1. suitable for multi-tasks and a new continuing task  with a different space\n2. A quantized space alignment (QSA) which maps the space to a quantized latent space  to align with any state/action space\n3. Selective weights activation (SWA) diffuser module to reduce the knowledge forgetting by adjust the activation of the weights.\n\nThe evaluation is comprehensive and conducted across several benchmarks, including Ant-dir and Continual World, and D4RL.", "questions": "1. What motivated the architectural decision to separate state sequence modeling (diffusion model) from action prediction (inverse dynamics model) rather than employing a unified model for complete trajectories? Is this separation related to accommodating different action spaces across tasks?\n\n\n2. Given the strong performance demonstrated within individual benchmarks, have you considered training across different benchmark environments simultaneously? What key challenges would you anticipate in such a cross-benchmark continual learning scenario?\n\n\n3. How would VQ-CD perform when faced with dramatic shifts in state representation formats across tasks? For instance, what adaptations would be necessary to handle a transition from low-dimensional vector state spaces to high-dimensional image-based observations while maintaining knowledge transfer capabilities?\n\n\n4. The VQ-CD framework integrates multiple technical components (quantized space alignment, selective weight activation, task-specific regularization). Which of these elements do you consider most critical to the method's performance gains?\n\n5. The authors justify selecting parameter masking over output neuron masking based on purported difficulties in \"distinguishing dependencies from weights to loss during backward propagation.\" However, modern automatic differentiation frameworks like PyTorch are specifically designed to handle such dependencies transparently. Could you clarify why dependency tracking poses a significant challenge in your architecture when automatic differentiation systems should resolve this issue inherently?", "limitations": "1. High-Dimensional State Representation Evaluation? Testing on high-dimensional observation spaces, such as image-based environments, would be even better to see the power of the proposed method.\n\n\n2. Despite empirical success, the framework lacks theoretical analysis for example, explaining why the quantized latent space representation achieves effective knowledge transfer between tasks\n\n3. Computational Efficiency Concerns", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- A novel approach that combines the quantized latent space, powerful diffusion models, and several improvements to tackle the continual multi-task offline RL problem, which increasingly becoming more important\n- Experimental results demonstrate remarkable performance advantages over existing methods across multiple benchmark environments\n- Clear presentation with logical structure and comprehensive technical details enhances readability and reproducibility\n\nWeakness:\n- Computational efficiency and resource requirements are not adequately addressed in the current analysis", "was_revised": true, "final_justification": "5 AC", "rebuttal": "Dear Reviewer tiFq,\n\nWe appreciate the reviewer's efforts for reviewing and respond to your concerns below.\n\n### [I]. Response to Weaknesses\n\n> **[1/1] W1:** Computational efficiency analysis\n\nWe compare the computational cost, including generation time and memory consumption, with diffusion-based methods, such as CuGRO, and transformer-based methods, such as L2M, and report the results in Table 1 and Table 2. The results show that, compared to the baselines, our method achieves lower time overhead and better performance with similar memory usage.\n\n**Table 1**: The computational cost of generation speed with different generation steps in D4RL [Hopper-m,Walker2d-m,Halfcheetah-m] tasks.\n||base|VQ-CD|CuGRO|\n|-|-|-|-|\n|Time consumption|5.73|0.29|0.33|\n|speed-up ratio|1×|19.8×|17.4×|\n|score|-|45.4|27.6|\n\n**Table 2**: The comparison of GPU memory consumption. We control the hardware the same for all methods. The GPU is NVIDIA GeForce RTX 3090, and the CPU is Intel(R) Xeon(R) Gold 6230 @ 2.10GHz.\n|domain|CL task setting|Method|GPU memory overhead (GB)|Approximate physical training time consumption (h)|Performance|\n|-|-|-|-|-|-|\n|D4RL|[Hopper-m,Walker2d-m,Halfcheetah-m]|VQ-CD (Ours)|4.6|55|48.0|\n|D4RL|[Hopper-m,Walker2d-m,Halfcheetah-m]|L2M|6.7|92|13.4|\n|D4RL|[Hopper-m,Walker2d-m,Halfcheetah-m]|L2M-large|8.8|94|16.0|\n\n### [II]. Response to Questions\n\n> **[1/5] Q1:** The motivation of inverse dynamics.\n\nFollowing previous studies [1], inverse dynamics is introduced to produce actions based on the state sequence generated by the diffusion model. We choose to model the distribution of state sequence rather than state-action sequence on the basis of two reasons: \n- Usually, in many robotics control scenarios, the actions are often represented as joint torques, which are high-frequency and less smooth, making it hard to model and predict the action sequence. \n- The state is usually continuous in RL, but the mode of action is diverse, such as discrete and continuous. Modeling state sequences separately makes the diffusion-based model more generic to extensive RL scenarios.\n\nUsing the diffusion model to model the state sequences and producing actions with the inverse dynamics are not related to accommodating different action spaces across tasks.\n\nTo further investigate the benefit of producing actions with inverse dynamics rather than generating (s,a) together with diffusion models, we conduct the experiments of modeling state and action sequences together with diffusion models and only modeling state sequences with diffusion models. Table 3 results show that when using inverse dynamics, our method can achieve better performance compared with directly producing action with diffusion models.\n\n**Table 3**: The comparison of producing actions with the diffusion model and the inverse dynamics.\n|type|producing action with diffusion model|producing action with inverse dynamics|\n|-|-|-|\n|performance in Ant-dir 4-18-26-34-42-49|498.2|524.1|\n|performance in [Hopper-m,Walker2d-m,Halfcheetah-m]|39.5|45.4|\n\n> **[2/5] Q2:** Training across different benchmark environments. What key challenges?\n\nGenerally speaking, previous methods rarely consider continual learning across benchmarks, because totally different RL tasks will bring several challenges [2,3]. We discuss them below and propose potential solutions.\n- First, there are inconsistencies in state and action spaces between different tasks, which would prevent a single model from adapting to each task. Fortunately, the quantized space alignment can be used to learn in a unified latent space.\n- The reward function structures vary significantly between different tasks, which will affect the joint distribution modeling between returns and state sequences with diffusion models. To address this issue, returns need to be normalized. \n- The task difference between different tasks is substantial, such as autonomous driving tasks and robot control tasks. Acquiring prior knowledge of tasks and designing task-specific action decoders for each task might be a direct and effective solution. \n\nGiven the problems posed by these challenges, we will consider training across different benchmark environments simultaneously in our future research.\n\n> **[3/5] Q3:** Adaptation from low-dimensional vector state spaces to high-dimensional image-based observations\n\nThank you for your comments. \nUsually, the applicability of continual learning across tasks with completely different state spaces is quite narrow, and previous studies rarely explore this area [3,4]. The purpose of continual learning is to progressively master new tasks by discovering common knowledge between tasks.\n\nThere are several substantial challenges facing high-dimensional image-based observations, particularly in terms of representation alignment, codebook generalization, and action generation.\n- **For representation:** We can introduce a visual encoder, e.g., ViT, to extract compact latent representations from images before passing them into the diffusion models, thereby aligning the modalities.\n- **For codebook:** The QSA module can be modified to support modality-specific encoders while maintaining a shared codebook space or adopting modality-conditional quantization.\n- **For inverse dynamics:** If we use inverse dynamics to directly produce actions with image-like states, then we need to upgrade inverse dynamics to a vision-conditioned architecture.\n\n> **[4/5] Q4:** Discussion of technical components used in VQ-CD\n\nQuantized space alignment and selective weight activation each play important roles. Quantized space alignment is used to expand the application range of VQ-CD, while selective weight activation is used to reduce forgetting of historical tasks in continual learning. The advantages of these two modules are complementary, and performance gains rely on both modules simultaneously.\n\n> **[5/5] Q5:** Why dependency tracking poses a significant challenge?\n\nOur decision to prefer parameter masking over output neuron masking stems from the 1D convolutional network structure of the diffusion model.\n\nIn classical MLP-based architectures, each output neuron is directly and exclusively associated with a subset of weights, making output masking an effective mechanism to selectively deactivate certain parts of the model. In contrast, 1D convolutional networks exhibit a different dependency pattern: every output of a given channel is influenced by all weights of the convolution kernel in that channel due to the weight-sharing nature of convolutions.\n\nConsequently, even if we apply output neuron masking at the activation level, the autograd engine of PyTorch will still compute gradients and update all convolutional weights associated with the masked outputs, as those weights contribute to other unmasked outputs. This deviates from the purpose of selective activation of task-specific parameters. Therefore, to achieve accurate control over which convolutional parameters are updated per task, we adopt parameter masking at the kernel level.\n\n### [III]. Response to Limitations\n\n> **[1/3] L1:** High-Dimensional State Representation Evaluation?\n\nThank you for your comments. Although our current work focuses on low-dimensional state spaces, the core design of VQ-CD, particularly the modular quantized space alignment (QSA), can be extended to high-dimensional modalities like images.\n- **For space alignment.** We can incorporate the visual encoder (e.g., ResNet or ViT) to extract latent features from images before quantization, enabling modality alignment without changing the core pipeline.\n- **For codebook generalization.** The QSA module already supports modular encoder-decoder pairs and scalable codebooks, allowing us to integrate image-based encoders and maintain aligned latent spaces across modalities.\n- **For action generation:** The inverse dynamics can be upgraded to a vision-conditioned architecture to handle transitions between visual observations.\n\nWhile image-based evaluation is beyond the current scope, we believe these extensions are feasible within our framework, and we will investigate them in future work.\n\n> **[2/3] L2:** Why the quantized latent space representation is effective\n\nWe acknowledge that the current version lacks theoretical proof to demonstrate that space-aligned representations can effectively extract common knowledge between tasks. However, our method has a theoretical foundation for the representation with vector quantization [5,6]. In our paper, the QSA module maps tasks' states and actions into a shared discrete latent space using the codebook. This process enables structural alignment between otherwise heterogeneous observation spaces, making it easier for the diffusion model to generalize across tasks.\n\nAbout the convergence of QSA, we can prove that under Robbins–Monro stepsizes and Lipschitz gradients, the expected gradient of the QSA loss $\\lim_{n\\rightarrow\\infty} \\mathbb{E}[||\\nabla_{\\theta_e,\\theta_d} \\mathcal{L}_{QSA}||]$ vanishes as the training iteration increases, where $n$ is the training iteration [7].\n\nWe will investigate further theoretical analysis in our future work. Besides, we will add the discussion of the limitations in the main body.\n\n> **[3/3] L3:** Computational Efficiency\n\nWe sincerely appreciate your suggestions. Please refer to W1 for the experiments on computational efficiency.\n\n### References\n\n[1] Is conditional generative modeling all you need for decision-making?\n\n[2] Continual world: A robotic benchmark for continual reinforcement learning.\n\n[3] Three types of incremental learning.\n\n[4] Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay.\n\n[5] Theory and experiments on vector quantized autoencoders.\n\n[6] Vector quantization for adaptive state aggregation in reinforcement learning.\n\n[7] Neural discrete representation learning.\n\nPlease do not hesitate to let us know if you have any further concerns.\n\nSincerely,\n\nAuthors of Paper 6316"}, {"review_id": "DqkdTa5zCe", "reviewer": "Reviewer_YNrb", "summary": "This paper tackles the problem of continual offline RL where action and observation space can be different among tasks. To this end, it proposed a method termed Vector-Quantized Continual Diffuser VQ-CD. VQ-CD comprises two parts, namely the quantization spaces alignment where vector quantization is used to align the different state and action spaces of different tasks, allowing continual learning in the same space. The second part is to use a unified diffusion model attached to by the inverse dynamic model to master all tasks by selectively activating different weights according to task-related sparse masks. The paper also provided experiments to demonstrates that VQ-CD can achieve SOTA performance on15 continual learning tasks (compared with 17 baselines).", "questions": "1.      The method developed in this work is for offline RL settings. I wonder how difficult or what are the factors that can enable this method to be applied to online RL settings. It might be better to provide a little more discussion regarding why the proposed method is a favorable solution for offline RL, rather than RL in general?\n2.      On line 120, should q(\\tau_s) be q(\\tau_s^0)? Also for \\tau_s^{1:K}, where k\\in[1:K] is referred to as the diffusion step, whereas on line 163, \\tau^_s^I, where $i$ indicates task index, are these two subscripts consistent with each other? On line 148, what is the expression for the stop gradient operation, sg(.)?\n3.      The tasks considered in this work are mostly continuous control problems similar to each other. Will the proposed method work if tasks are of different types, for example after playing some video game for a few hours, you may consider to do some exercise like swimming or running, and then do homework, like reading comprehension or math?\n4.      For table 1, is there a particular reason the continual task sequence is chosen to be 10-15-19-25. Will the results or conclusion be different if a different task sequence is chosen?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "riginality.\nStrength:\n1.      Novelty: comparing to many previous works on continual learning where action space and observation space are the same and tasks are mainly distinguished through varied reward or dynamics. This works considers a different setting, where observation and action spaces are different.\n2.       Comprehensive empirical evaluations are provided to demonstrate the superior performance of the proposed method over SOTA\nWeaknesses:\n1.      Most results are empirical, no theoretical analysis, such as convergence property, or sample/computation complexity are provided\n2.      The evaluation tasks are limited to existing continuous control problems\n3.      Although the proposed method works well empirically, it is unclear to me how this method can be applied to a general continue learning setting, like the one I provided in Question 3. If might be better if at the beginning of a paper, some intuitive examples can be provided to illustrate how the proposed method works, besides Figure 1.", "was_revised": true, "final_justification": "Given the quality of this paper and possible room for improvement, I maintain my rating.", "rebuttal": "Dear Reviewer YNrb,\n\nWe appreciate the reviewer's efforts for reviewing and respond to your concerns below.\n\n### [I]. Response to Weaknesses\n\n> **[1/3] W1:** Most results are empirical, no theoretical analysis, such as convergence property, or sample/computation complexity are provided\n\n#### **[W1.1] Theoretical Analysis**\n\nWe discuss theoretical analysis from two perspectives.\n\n**Convergence proof of QSA:** Inspired by the proof of VQ-VAE [1], under Robbins–Monro stepsizes and Lipschitz gradients, the expected gradient vanishes as the training iteration increases. In orther words, $\\lim_{n\\rightarrow\\infty} \\mathbb{E}[||\\nabla_{\\theta_e,\\theta_d} \\mathcal{L}_{QSA}||]$, where $n$ is the training interation. The proof contains three steps.\n- The encoder and decoder parameters are updated via projected stochastic gradient descent, ensuring that each step remains within a bounded feasible set and that the expected loss does not increase.\n- The vector quantization step follows an EM-style update, which guarantees non-increasing reconstruction loss.\n- Since the total loss is bounded below by zero and decreases in expectation, the update sequence converges, and the expected gradient norm tends to zero as training progresses.\n\n**Convergence proof of SWA:** According to the previous studies of diffusion model [2,3], the denoising score-matching loss ($\\mathcal{L}(\\theta)=\\mathbb{E}[||\\epsilon-\\epsilon_{\\theta}(\\tau_s^k, k, b*\\mathcal{C})||_2^2]$) is a lower bound on the negative log-likelihood. Under the Lipschitz constraint, the loss landscape is smooth, and optimization using Adam or SGD with diminishing learning rates satisfies the Robbins–Monro conditions. Thus, the training process converges to a stationary point of the objective.\n\n#### **[W1.2] Computation Complexity**\n\nFollowing your suggestions, we provide the computation complexity.\n\nWe compare the computation complexity by comparing with diffusion-based methods, such as CuGRO, and transformer-based methods, such as L2M, and report the results in Table 1 and Table 2. The results show that, compared to the baselines, our method achieves lower time overhead and better performance with similar memory usage.\n\n**Table 1**: The computational cost of generation speed with different generation steps in D4RL [Hopper-m,Walker2d-m,Halfcheetah-m] tasks.\n||base|VQ-CD|CuGRO|\n|-|-|-|-|\n|Time consumption|5.73|0.29|0.33|\n|speed-up ratio|1×|19.8×|17.4×|\n|score|-|45.4|27.6|\n\n**Table 2**: The comparison of GPU memory consumption. We control the hardware the same for all methods. The GPU is NVIDIA GeForce RTX 3090, and the CPU is Intel(R) Xeon(R) Gold 6230 @ 2.10GHz.\n|domain|CL task setting|Method|GPU memory overhead (GB)|Approximate physical training time consumption (h)|Performance|\n|-|-|-|-|-|-|\n|D4RL|[Hopper-m,Walker2d-m,Halfcheetah-m]|VQ-CD (Ours)|4.6|55|48.0|\n|D4RL|[Hopper-m,Walker2d-m,Halfcheetah-m]|L2M|6.7|92|13.4|\n|D4RL|[Hopper-m,Walker2d-m,Halfcheetah-m]|L2M-large|8.8|94|16.0|\n\n> **[2/3] W2:** The evaluation tasks are limited to existing continuous control problems\n\nWe appreciate your comments. Though our current evaluation focuses on continuous control benchmarks, we would like to clarify that the chosen tasks already cover a wide spectrum of challenges: \n1) **Task diversity:** Our experiments include multiple types of continual learning tasks across locomotion and robotic manipulation, with different state and action spaces and difficulty levels. \n2) **Space heterogeneity:** In D4RL experiments, we test across tasks with heterogeneous state and action spaces, such as Hopper, Walker2d, and HalfCheetah, under various dataset qualities (e.g., medium and full-replay), which simulate realistic distribution shifts. \n\nMore importantly, the core framework of VQ-CD is task-agnostic and modular, which makes it conceptually applicable to other types of tasks, such as visual RL tasks, as long as it is provided with appropriate encoder and decoder adaptations. We agree that evaluating on broader domains would further enhance generality, and we consider investigating these directions in future work.\n\n> **[3/3] W3:** Although the proposed method works well empirically, it is unclear to me how this method can be applied to a general continue learning setting, like the one I provided in Question 3. If might be better if at the beginning of a paper, some intuitive examples can be provided to illustrate how the proposed method works, besides Figure 1.\n\nGenerally speaking, previous methods rarely consider continual learning on totally different RL tasks, such as playing video games followed by exercise [4,5]. Such high-level task heterogeneity involves dramatic shifts in not just state and action spaces, but also in task dynamics, goals, and modality. There are essential challenges that we need to resolve.\n- First, there are inconsistencies in state and action spaces between different tasks, which would prevent a single model from adapting to each task. QSA may provide an effective way to align the different state and action spaces.\n- Second, the reward function structures vary significantly between different tasks, which will affect the joint distribution modeling between returns and state sequences with diffusion models. Currently, reward normalization is a simple but effective method.\n- The task difference between different tasks is substantial, such as autonomous driving tasks and robot control tasks. Acquiring prior knowledge of tasks and designing task-specific action decoders for each task might be a direct and effective solution. \n\nWe fully agree that intuitive examples in the introduction can largely improve accessibility for readers. In the revised version, we will incorporate a motivating example, e.g., learning to control different robots to master different skills, to concretely illustrate how VQ-CD aligns representations, learns knowledge, and reduces forgetting.\n\n### [II]. Response to Questions\n\n> **[1/4] Q1:** It might be better to provide a little more discussion regarding why the proposed method is a favorable solution for offline RL, rather than RL in general?\n\nOur method is currently designed for the offline RL setting primarily due to the multiple-step sampling process inherent in diffusion models, which leads to significant computational latency during interaction in online RL. To enable online RL extensions in the future, several promising research lines can be explored:\n- Parallelized sampling techniques to reduce per-decision latency.\n- Advanced samplers (e.g., DDIM [3] and DPM-solver [4]) to accelerate generation.\n- Distilling the diffusion model into the consistency model [5] that can realize single-step generation.\n\n> **[2/4] Q2:** On line 120, should q(\\tau_s) be q(\\tau_s^0)? Also for \\tau_s^{1:K}, where k\\in[1:K] is referred to as the diffusion step, whereas on line 163, \\tau^_s^I, where $i$ indicates task index, are these two subscripts consistent with each other? The explanation of sg(.)?\n\n1. Your understanding is right, $q(\\tau_s)$ and $q(\\tau_s^0)$ are same each other. Usually, we omit the diffusion step subscript if it is 0.\n2. In line 163, $i$ denotes the task index and the diffusion step supscript of $\\tau^i_{s_{z_q}}$ is 0, thus we omit the diffusion step.\n3. Stop Gradient refers to blocking the gradient propagation path of certain tensors to the loss function during backpropagation, ensuring that these tensors will not be updated in the current training process.\n\n> **[3/4] Q3:** Will the proposed method work if tasks are of different types, for example video game followed by some exercise like swimming or running, and then do homework, like reading comprehension or math?\n\nWe sincerely thank the reviewer's advice. Please refer to **W3** for a detailed discussion.\n\n> **[4/4] Q4:** For table 1, is there a particular reason the continual task sequence is chosen to be 10-15-19-25. Will the results or conclusion be different if a different task sequence is chosen?\n\nWe arbitrarily selected several tasks from Ant-dir to form a continual learning sequence. To verify that our method is independent of task selection, we also select other tasks to form task sequences, such as the task sequence 4-18-26-34-42-49 and longer sequences such as 4-9-18-22-26-34-42-49, and report the performance comparison with other baselines in Table 3. \nFurthermore, to verify that our method is independent of the sequence order of tasks, we also arbitrarily rearranged the order of tasks for experiments.\n\nExperimental results show that our method outperforms baselines across various arbitrarily selected task sequences, and achieves similar performance across them, indicating that our method's performance is independent of task selection and insensitive to the sequence order of tasks. Meanwhile, some baselines are more sensitive to the sequence order of tasks. For example, PackNet performs better when the task sequence is 4-26-18-34-42-49, which differs significantly from its performance on the 18-4-26-34-42-49.\n\n**Table 3**: The experiments of arbitrary task selections.\n|Task sequences|VQ-CD|CRIL|CuGRO|PackNet|DGR|t-DGR|Finetune|EWC|\n|-|-|-|-|-|-|-|-|-|\n|4-18-26-34-42-49|524.1|408.4|385.6|289.8|100.2|72.3|56.4|50.9|\n|4-9-18-22-26-34-42-49|523.6|328.1|293.9|241.3|105.0|88.6|84.3|84.9|\n|18-4-26-34-42-49|516.7|435.1|340.1|97.5|83.8|77.4|56.6|58.1|\n|49-4-34-18-26-42|518.7|403.8|275.7|136.6|187.3|186.5|278.3|278.5|\n|4-26-18-34-42-49|517.2|421.1|344.6|291.0|99.5|56.4|58.2|57.2|\n\n### References\n\n[1] Neural discrete representation learning.\n\n[2] Denoising diffusion probabilistic models.\n\n[3] Denoising diffusion implicit models.\n\n[4] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps.\n\n[5] Consistency Models.\n\n\nPlease do not hesitate to let us know if you have any further concerns.\n\nSincerely,\n\nAuthors of Paper 6316"}, {"review_id": "GKNZYKME9y", "reviewer": "Reviewer_aMmx", "summary": "This paper tackled continual offline RL via two components: quantized spaces alignment (QSA) and selective weights activation (SWA). QSA adopts ensemble vector quantized encoders to align state-action spaces and task-related decoders to recover original spaces. SWA selectively activates different weights according to task-related sparse masks.", "questions": "- The SWA and QSA modules seems to have nothing to do with diffusion models, and can be applied to any other network architectures. In what ways do the two modules match the utilization of diffusion models?\n\n- The QSA is trained using Eq. (4). However, in experiments, the spaces are aligned using simple state-action padding. Any explanation on this inconsistency?\n\n- It is hard to figure out how the encoder and decoder in QSA are trained. It seems to map different state-action spaces into a fixed-length latent space, and then decode the original space during inference. What is the intrinsic principle when designing the QSA module, and in what way QSA can extract common knowledge for tasks with different state-action spaces?\n\n- The network masking strategy is trivial and heuristic. Also, the strategy is friendly to MLPs. What is the scalability when applied to transformers?\n\n- In the arbitrary CL setting of Figure 3, are there only three tasks on the stream? Also, the performance does not degrade when switching to a new task, which is bit of counter-intuitive.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The way of aligning state-action spaces is necessary and interesting.\n- The comparison to multiple baselines is comprehensive.\n\nWeaknesses:\n- The motivation of using diffusion models is unclear and weak. The core modules of QSA and SWA seem applicable to any network architectures, not just for diffusion models.\n\n- The method is too complicated, failing to yield an elegant formulation. The presentation is also too complicated and confusing in many parts (such as Fig. 1, too many components, failing to identify the critical parts). \n\n- The SWA module involve many hand-designed heuristics, leading to the impression of insufficient contribution on the methodology side.", "was_revised": true, "final_justification": "The authors provided further clarification and partially addressed my concerns. I increased my original score.", "rebuttal": "Dear Reviewer aMmx,\n\nWe appreciate the reviewer's efforts for reviewing and respond to your concerns below.\n\n### [I]. Response to Weaknesses\n\n> **[1/3] W1:** The motivation of using diffusion models is unclear and weak. The core modules of QSA and SWA seem applicable to any network architectures, not just for diffusion models.\n\nFirstly, we would like to clarify why we chose diffusion models rather than other types of models. We select diffusion models as our foundational method primarily based on the following reasons.\n- **Natural support for multi-modal action distribution modeling.** Diffusion models can effectively avoid the limitations of traditional Gaussian models that cannot model multi-peak action distributions.\n- **Powerful model expressiveness.** Diffusion models support the modeling of complex data or trajectory distributions.\n- **Stable log-likelihood training.** Diffusion models can effectively prevent mode collapse and training instability issues.\n- **Competitive model performance.** Diffusion-based RL methods have also shown huge potential in many robotic control scenarios.\n\nSecondly, SWA is specifically designed for diffusion models with one-dimensional convolutional structures because diffusion models exhibit a different dependency pattern: every output of a given channel is influenced by all weights of the convolution kernel in that channel due to the weight-sharing nature of convolutions. Thus, traditional output neuron masking is not suitable for diffusion models with one-dimensional convolutional structures because we can not enable task-related parameters through masking certain output neurons. However, SWA applies masking directly on convolution kernel parameters, allowing us to selectively activate neurons in the convolution kernel to control the training of task-related parameters.\n\nFinally, QSA is applicable to any model. Different state and action spaces lead to significant distribution differences between tasks, making it impossible to use a single diffusion model for learning. QSA aligns these varying state and action spaces into the same latent space, enabling effective continual training under the same diffusion backbone.\n\n> **[2/3] W2:** The method is too complicated, failing to yield an elegant formulation. The presentation is also too complicated and confusing in many parts (such as Fig. 1, too many components, failing to identify the critical parts).\n\nThanks for your helpful suggestions. We have drawn a simpler and more intuitive structural diagram to showcase the core parts of our method. Due to this year's updated rebuttal policy, we can not include new figures at this stage. But we will add the diagram and demonstrate the functions of QSA and SWA through intuitive examples in the revised version.\n\n> **[3/3] W3:** The SWA module involve many hand-designed heuristics, leading to the impression of insufficient contribution on the methodology side.\n\nWe would like to clarify that the design of SWA is based on explicit motivation grounded in continual learning theory, particularly in the context of parameter isolation to mitigate forgetting. In diffusion models, previous parameter isolation strategies, such as output neuron masking, are improper for training due to the sophisticated 1D convolution structure. SWA addresses this by introducing task-specific binary masks over convolution kernels, enabling selective activation of parameters per task. Moreover, we explored other alternatives (e.g., sparse training) in Table 1 but found that the current SWA strikes the best balance between simplicity, effectiveness, and compatibility with diffusion models.\n\n**Table 1**: The comparison of time consumption per update between sparse and dense optimizers.\n|CL task|time consumption per update of dense training (s)|time consumption per update of dense training (s)|\n|-|-|-|\n|[Hopper-fr,Walker2d-fr,Halfcheetah-fr]|0.089|0.198|\n|[Hopper-mr,Walker2d-mr,Halfcheetah-mr]|0.096|0.197|\n|[Hopper-m,Walker2d-m,Halfcheetah-m]|0.089|0.195|\n|CW10|0.061|0.218|\n\n### [II]. Response to Questions\n\n> **[1/5] Q1:** The SWA and QSA modules seems to have nothing to do with diffusion models, and can be applied to any other network architectures. In what ways do the two modules match the utilization of diffusion models?\n\nIn **W1**, we detailed the importance of QSA and SWA for diffusion models. Please refer to **W1** for the detailed explanation.\n\n> **[2/5] Q2:** The QSA is trained using Eq. (4). However, in experiments, the spaces are aligned using simple state-action padding. Any explanation on this inconsistency?\n\nWe want to clarify that we conducted experiments on the spaces aligned with QSA and displayed them in Figure 11 of the main body. We sincerely apologize for the confusion caused, and in the revised version, we will swap the positions of Figure 3 and Figure 11, first showing the QSA alignment results in the main text.\n\n> **[3/5] Q3:** It is hard to figure out how the encoder and decoder in QSA are trained. It seems to map different state-action spaces into a fixed-length latent space, and then decode the original space during inference. What is the intrinsic principle when designing the QSA module, and in what way QSA can extract common knowledge for tasks with different state-action spaces?\n\nThe core principle behind QSA is to provide a shared representation basis, so that all tasks with different state and action spaces can be mapped into and decoded from the same quantized latent space. This provides a common ground for the inverse dynamics model, which can operate solely on the quantized latent states $s_{z_q}$ and learn the cross-task transition dynamics independent of the raw state $s$.\n\nThis alignment is crucial for continual learning, as it reduces input heterogeneity and promotes shared policy reasoning. Our ablation study (shown in Figure 4 of the main body) and feature difference comparison (Please refer to Table 2 of the main body) further validate that QSA significantly improves cross-task representation consistency compared to other space alignment methods.\n\n> **[4/5] Q4:** The network masking strategy is trivial and heuristic. Also, the strategy is friendly to MLPs. What is the scalability when applied to transformers?\n\nWe would like to emphasize that the Selective Weights Activation (SWA) module is carefully designed to address the unique challenges of diffusion models in continual learning, rather than being a trivial or ad hoc solution. In contrast to transformer-based methods, diffusion-based methods that are built on U-Net convolutional backbones are more suitable for modeling dense temporal state-action distributions. Diffusion models' powerful expression and superior stability have been supported by recent works [1]. The design of SWA is grounded in convolution-based diffusion models, enabling parameter isolation by introducing task-specific binary masks over convolution kernels, and shows potential in addressing various continual learning tasks.\n\nRegarding scalability to transformers: 1) While transformers may permit more granular activation control (e.g., attention head masking), they also introduce significant memory and computation overhead, especially in long-horizon RL. 2) Our SWA is architecture-agnostic in principle, and could be adapted to transformer blocks by masking feedforward or attention parameters.\n\n> **[5/5] Q5:** In the arbitrary CL setting of Figure 3, are there only three tsks on the stream? Also, the performance does not degrade when switching to a new task, which is bit of counter-intuitive.\n\nThe experiments in Figure 3 contain three tasks, and the y-axis represents the average performance across these three tasks. This means that when the model is trained on the first task, the tested performance is divided by 3. Since SWA preserves knowledge of historical tasks through parameter isolation, when switching to a new task, the model's knowledge of historical tasks is preserved to the greatest extent possible, so performance hardly declines.\n\n### References\n\n[1] Is Conditional Generative Modeling all you need for Decision-Making?\n\n\nPlease do not hesitate to let us know if you have any further concerns.\n\nSincerely,\n\nAuthors of Paper 6316"}, {"review_id": "9saFUUz7ss", "reviewer": "Reviewer_Yyw3", "summary": "This paper proposes VQ-CD, a framework for continual offline reinforcement learning that handles heterogeneous task spaces by combining vector quantization-based space alignment and selective weight activation. The method achieves strong performance across both traditional and arbitrary CL settings.", "questions": "Please see weaknesses 1- 5.\n\nI am currently leaning toward a borderline reject. But I am open to increasing my score if the authors can adequately address these issues during the rebuttal.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The paper conducts comprehensive experiments across both traditional and arbitrary CL settings to demonstrate the effectiveness of the proposed VQ-CD framework.\n- Detailed ablation studies provide evidence for the contribution of each module.\n\n**Weaknesses**\n- Some claims may be overstated. For example, Line 138 states that the method supports “our method enables training on **any** CL task sequence” and “QSA enables our method to adapt for **any** continual learning task setting.” However, it appears that VQ-CD assumes that task identities and boundaries are known. Can the method be applied in task-free CL settings?\n- The SWA module contributes significantly even when task spaces are consistent. Could the authors elaborate on how SWA compares structurally and functionally to existing parameter-isolation methods such as PackNet?  Is there a risk of capacity saturation or degradation in performance as tasks accumulate, especially in longer task sequences?\n-  The authors mention that each task occupies a separate region in the quantized space. As the number of tasks increases, how does this design affect the size and efficiency of the codebook? Is there any mechanism to reuse or compress unused entries, or does the method assume a large fixed codebook with sufficient capacity?\n- In Figure 2, the performance of the multitask baseline, which is often considered the upper bound in continual learning, is unexpectedly poor. In Figure 3, VQ-CD even substantially outperforms multitask training. Could the authors provide insights or possible reasons for this?\n- Given that VQ-CD builds a shared latent space across tasks, is the model able to generalize to unseen tasks with similar structure but new state/action configurations?", "was_revised": true, "final_justification": "Thank you for the rebuttal. It has addressed my concerns, and I will raise my score accordingly.", "rebuttal": "Dear Reviewer Yyw3,\n\nWe appreciate your further suggestions and respond to your concerns below.\n\n### [I]. Response to Weaknesses\n\n> **[1/5] W1:** Some claims may be overstated. For example, Line 138 states that the method supports “our method enables training on any CL task sequence” and “QSA enables our method to adapt for any continual learning task setting.” However, it appears that VQ-CD assumes that task identities and boundaries are known. Can the method be applied in task-free CL settings?\n\nThanks for your suggestions. We will revise the corresponding content to restrict the scope of application in task-aware continual learning. Detecting the task boundaries demands extra task similarity measurement mechanisms for detection. Admittedly, this is currently a limitation of our approach. It is one of the challenges we are actively addressing.\n\nTask-aware continual learning is quite prevalent in many applications, such as CW10 and CW20 continual learning settings, as explicit task boundaries can often be conveniently determined using various methods, such as leveraging large models or human feedback on existing datasets or constructing new datasets with explicit task boundaries based on classical RL algorithms [1,2].\n\nCompared with task-aware continual learning, task boundary-agnostic continual learning requires an additional mechanism to detect whether a task change has occurred. Clearly, this demands extra task similarity measurement mechanisms for detection. Admittedly, this is currently a limitation of our approach. However, it is also one of the challenges we are actively addressing. We are confident that further progress will be reflected in our future research.\n\n> **[2/5] W2:** The SWA module contributes significantly even when task spaces are consistent. Could the authors elaborate on how SWA compares structurally and functionally to existing parameter-isolation methods such as PackNet? Is there a risk of capacity saturation or degradation in performance as tasks accumulate, especially in longer task sequences?\n\nThank you for the valuable suggestions. We first discuss the differences between VQ-CD and PackNet.\n- Structurally, PackNet applies structured pruning to identify task-relevant weights and masks out the rest for later tasks, often involving iterative pruning-retraining cycles and reliance on weight magnitudes. In contrast, our SWA module adopts lightweight task-specific binary masks directly applied to convolution kernel weights, enabling efficient activation of relevant parameters during both training and inference. This approach avoids pruning heuristics and simplifies implementation within diffusion probabilistic models.\n- Functionally, PackNet was designed primarily for classification tasks with feedforward MLPs, while SWA of VQ-CD is designed to accommodate more sophisticated U-Net-based diffusion models. Besides, compared with PackNet, diffusion models' powerful expression and superior stability have been supported by recent studies [3].\n\nFor the capacity saturation, it is an inherent challenge in all parameter-isolation methods, and we mitigate it in two ways: 1) Diffusion models are typically overparameterized, and early tasks often underutilize capacity, leaving room for subsequent tasks. 2) Mask pruning introduced in Appendix B.6 of the main body allows us to purposefully free up a portion of parameters for training new tasks. 3) For extremely long-horizon task sequences, we acknowledge the need for future work on adaptive mask reallocation, soft-mask learning, or latent reparameterization strategies to further improve scalability.\n\n> **[3/5] W3:** The authors mention that each task occupies a separate region in the quantized space. As the number of tasks increases, how does this design affect the size and efficiency of the codebook? Is there any mechanism to reuse or compress unused entries, or does the method assume a large fixed codebook with sufficient capacity?\n\nThank you for your thoughtful suggestions. We try to address your concerns by discussing more details of our method.\n\nFor different tasks, we maintain separate codebooks currently. If we think of the QSA module as a base station, these codebooks are like memory cards. When we need to test on task i, we load the corresponding codebook for task i from the saved checkpoints into the QSA module, enabling our model to perform space alignment for task i. When we encounter a new task j, we need to construct a new codebook for task j training and save it as a checkpoint for future use. Therefore, as the number of tasks increases, the memory overhead of QSA during inference does not increase with the addition of tasks, because QSA only needs to load the task-relevant codebook for evaluation.\n\nTo further address potential codebook saturation or inefficiency in longer task sequences, we believe several extensions are promising directions: 1) Codebook LoRA strategy, which means new tasks can reuse several old codebook latents. 2) Codebook regularization based on usage frequency that encourages all tasks to share a single codebook, where new tasks during QSA training should minimize the use of latents frequently called by previous tasks. \n\n> **[4/5] W4:** In Figure 2, the performance of the multitask baseline, which is often considered the upper bound in continual learning, is unexpectedly poor. In Figure 3, VQ-CD even substantially outperforms multitask training. Could the authors provide insights or possible reasons for this?\n\nThere are two main reasons that contribute to this result:\n- **Architecture differences:** The multitask baseline is implemented using the MLP architecture, which is often insufficient to model the complex state-action mappings across diverse tasks. In contrast, our VQ-CD method leverages a diffusion-based policy model, which has demonstrated superior expressiveness and stability in high-dimensional generative modeling [3]. This difference leads to the performance differences in the experiments.\n- **Representation ambiguity across tasks:** As shown in Table 2 of the main body, the QSA module maps task-specific state-action pairs into a shared aligned latent space, where the distances between different tasks' state- and action-aligned vectors are relatively small. This results in high semantic proximity across tasks. For Multitask training, all tasks' state- and action-aligned vectors are blended together, which makes it challenging to learn the mapping of a state-aligned vector to the correct action-aligned vector, leading to degraded performance. In contrast, VQ-CD separates alignment with QSA and generation via the conditional diffusion model, which enables more precise modeling and avoids such interference.\n\n> **[5/5] W5:** Given that VQ-CD builds a shared latent space across tasks, is the model able to generalize to unseen tasks with similar structure but new state/action configurations?\n\nThank you for your forward-thinking insights. \nWhile our method currently focuses on CORL, we agree that the structure of our framework naturally supports generalization to new tasks with similar temporal or semantic structure. Given the shared latent space across tasks, the QSA module learns to project diverse task-specific state and action vectors into a unified discrete latent space. This design inherently promotes composability and reuse of latent codes for unseen tasks. \n\nWhile we have not explicitly evaluated zero-shot or few-shot generalization in this paper, the proposed framework can be extended to support such scenarios with appropriate adaptations. For example, a new task with different state and action spaces but similar temporal semantics can be encoded via the shared latent space. Then the decision can be produced by diffusion models followed by the QSA decoder.\n\n### References\n\n[1] Continual World: A Robotic Benchmark For Continual Reinforcement Learning.\n\n[2] Continual offline reinforcement learning via diffusion-based dual generative replay.\n\n[3] Diffusion policies as an expressive policy class for offline reinforcement learning.\n\n\nPlease do not hesitate to let us know if you have any further concerns.\n\nSincerely,\n\nAuthors of Paper 6316"}]

Author rebuttal:

Meta-review: Within the continual offline learning framework, the paper proposes VQ-CD, which aligns different observation and action spaces via vector quantization and uses selective weight activation to preserve prior tasks. This approach will be quite useful to the community with strong results across different benchmarks. I recommend accepting this paper.

However, reviewers noted a lack of high-dimensional image settings. In addition, I also suggest clarifying for a broader audience the difference between offline and online continual learning and what aspect of the work is specific to offline learning or not, etc, that will help the continual learning community put this work into perspective. Also, please address the remaining issues noted by the reviewers such as analysis on computational efficiency and resource requirements, some of which are already discussed by the authors.
Decision: Accept (poster)