Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces

## Abstract

Abstract Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based lifelong learning systems by modeling the joint distributions of trajectories.
However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments.
In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks.
Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation.
In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space.
Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks.
Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces).
Compared with 16 baselines, our method reaches the SOTA performance.

## 1 Introduction

The endeavor of recovering high-performance policies from abundant offline samples gathered by various sources and continually mastering future tasks learning and previous knowledge maintaining gives birth to the issue of continual offline reinforcement learning (CORL) (Levine et al., ; Ada et al., ; Huang et al., ).
Ever-growing scenarios or offline datasets pose challenges for most continual RL methods that are trained on static data and are prone to showing catastrophic forgetting of previous knowledge and ineffective learning of new tasks (Liu et al., ; Zhang et al., ; Korycki & Krawczyk, ).
Facing these challenges, three categories of methods, rehearsal-based (Huang et al., ; Peng et al., ; Chaudhry et al., ), regularization-based (Smith et al., ; Zhang et al., ; ), and structure-based methods (Zhang et al., ; Marouf et al., ; Borsos et al., ), are proposed to reduce forgetting and facilitate continual training.

However, most previous studies only focus on the continual learning (CL) setting with identical state and action spaces (Liu et al., ; Smith et al., ).
It deviates from the fact that the ever-growing scenarios or offline datasets are likely to possess different state and action spaces with previous tasks for many reasons, such as the variation of demands and the number of sensors (Yang et al., ; Zhang et al., ).
Moreover, these datasets often come from multiple behavior policies, which pose the additional challenge of modeling the multimodal distribution of various tasks (Ada et al., ; Lee et al., ).
Benefiting from diffusion models’ powerful expressive capabilities and highly competitive performance, an increasing number of researchers are considering incorporating them to address the CORL problems (Ajay et al., ; Yue et al., ; Elsayed & Mahmood, ) from the perspective of sequential modeling.
There have been several attempts to combine diffusion-based models with rehearsal-based and regularization-based techniques, which usually apply constraints to the continual model learning process with previous tasks’ data or well-trained weights (Smith et al., ; Yue et al., ; Liu et al., ).
However, constrained weight updating will limit the learning capability of new tasks and can not preserve the previously acquired knowledge perfectly (Yang et al., ).
Although structure-based methods can eliminate forgetting and strengthen the learning capability by preserving well-trained weights of previous tasks and reserving disengaged weights for ongoing tasks, they are still limited in simple architecture and CL settings with identical state and action spaces (Zhang et al., ; Wang et al., ; Mallya & Lazebnik, ).
Thus, in this paper, we seek to answer the question:

*Can we merge the merits of diffusion models’ powerful expression and structure-based parameters isolation to master CORL problems with any task sequence?*

We answer this in the affirmative through the key insight of allocating harmonious weights for each continual learning task.
Specifically, we propose Vector-Quantized Continual Diffuser called VQ-CD, which contains two complementary sections: the quantized spaces alignment (QSA) module and the selective weights activation diffuser (SWA) module.
To expand our method to any task sequences under the continual learning setting, we adopt the QSA module to align the different state and action spaces.
Concretely, we adopt vector quantization to map the task spaces to a unified space for training based on the contained codebook and recover it to the original task spaces for evaluation.
In the SWA module, we first perform task mask generation for each task, where the task masks are applied to the one-dimensional convolution kernel of the U-net structure diffusion model.
Then, we use the masked kernels to block the influence of unrelated weights during the training and inference.
Finally, after the training process, we propose the weights assembling to aggregate the task-related weights together for simplicity and efficiency.
In summary, our main contributions are fourfold:

- •
We propose the Vector-Quantized Continual Diffuser (VQ-CD) framework, which can not only be applied to conventional continual tasks but also be suitable for any continual tasks setting, which makes it observably different from the previous CL method.
- •
In the quantized spaces alignment (QSA) module of VQ-CD, we adopt ensemble vector quantized encoders based on the constrained codebook because it can be expanded expediently. During the inference, we apply task-related decoders to recover the various observation and action spaces.
- •
In the selective weights activation (SWA) diffuser module of VQ-CD, we first perform task-related task masks, which will then be used to the kernel weights of the diffuser. After training, we propose assembling weights to merge all learned knowledge.
- •
Finally, we conduct extensive experiments on 15 CL tasks, including conventional CL settings and any CL task sequence settings.
The results show that our method surpasses or matches the SOTA performance compared with 16 representative baselines.

## 2 Related Work

Offline RL.
Offline reinforcement learning is becoming an important direction in RL because it supports learning on large pre-collected datasets and avoids massive demand for expensive, risky interactions with the environments (Mnih et al., ; Nair et al., ; Cheng et al., ; Ball et al., ).
Directly applying conventional RL methods in offline RL faces the challenge of distributional shift (Schaul et al., ; Levine et al., ; Xie et al., ; Yue et al., ) caused by the mismatch between the learned and data-collected policies, which will usually make the agent improperly estimate expectation return on out-of-distribution actions (Schaul et al., ; Kostrikov et al., ; Ada et al., ).
To tackle this challenge, previous studies try to avoid the influences of out-of-distribution actions by adopting constrained policy optimization (Peng et al., ; Fujimoto et al., ; Nair et al., ; Kostrikov et al., ), behavior regularization (Nachum et al., ; Kumar et al., ; Ghosh et al., ), importance sampling (Jiang & Li, ; Hallak & Mannor, ; Zhang et al., ), uncertainty estimation (Agarwal et al., ; Wang et al., ; Lee et al., ), and imitation learning (Wang et al., ; Siegel et al., ; Chen et al., ).

Continual RL.
Continual learning (CL) aims to solve the plasticity
and stability trade-off under the task setting, where the agent can only learn to solve each task successively (Zhang et al., ; Wang et al., ).
CL can be classified into task-aware CL and task-free CL according to whether there are explicit task boundaries (Aljundi et al., ; Wang et al., ).
In this paper, we mainly focus on task-aware CL.
There are three main technical routes to facilitate forward transfer (plasticity) and mitigate catastrophic forgetting (stability).
Rehearsal-based approaches (Shin et al., ; Mallya & Lazebnik, ; Wang et al., ; Zhang et al., ; Smith et al., ) store a portion of samples from previous tasks and use interleaving updates between new tasks’ samples and previous tasks’ samples.
Simply storing samples increases the memory burden in many scenarios; thus, generative models such as diffusion models are introduced to mimic previous data distribution and generate synthetic replay for knowledge maintenance (Zhai et al., ; Qi et al., ; Gao & Liu, ).
Regularization-based approaches (Kaplanis et al., ; Kessler et al., ; Zhang et al., ; ) seek to find a proficiency compromise between previous and new tasks by leveraging constraint terms on the total loss function.
Usually, additional terms of learning objectives will be adopted to penalize significant changes in the behaviors of models’ outputs or the updating of models’ parameters (Kirkpatrick et al., ; Kaplanis et al., ).
In the structure-based approaches (Wang et al., ; Kessler et al., ; Wang et al., ; Zhang et al., ; Smith et al., ; Konishi et al., ), researchers usually consider parameter isolation by using sub-networks or task-related neurons to prevent forgetting.

Diffusion RL.
Recently, diffusion-based models have shown huge potential in RL under the perspective of sequential modeling (Sohl-Dickstein et al., ; Ho et al., ; Rombach et al., ; Janner et al., ; Ajay et al., ; Beeson & Montana, ).
A typical use of diffusion models is to mimic the joint distribution of states and actions, and we usually use state-action value functions as the classifier or class-free guidance when generating decisions (Nichol & Dhariwal, ; Ho & Salimans, ; Pearce et al., ; Liu et al., ).
Diffusion models, as representative generative models, can also be used as environmental dynamics to model and generate synthetic samples to improve sample efficiency or maintain previous knowledge in CL (Yamaguchi & Fukuda, ; Hepburn & Montana, ; Lu et al., ; Ding et al., ; Liu et al., ).
It is noted that the diffusion model’s powerful expression ability on multimodal distribution also makes it suitable for being used as policies to model the distribution of actions and as planners to perform long-horizon planning (Wang et al., ; Kang et al., ; Chen et al., ).
Besides, diffusion models can also be used as multi-task learning models to master several tasks simultaneously (He et al., ) or as multi-agent models to solve more complex RL scenarios (Zhu et al., ).

## 3 Preliminary

### 3.1 Continual Offline RL

We focus on the task-aware CL in the continual offline RL in this paper (Zhang et al., ; Abel et al., ; Wang et al., ; Smith et al., ; Qing et al., ; Wang et al., ).
Suppose that we have $I$ successive tasks, and task $j$ arises behind task $i$ for any $i<j$.
Each task $i,i\in[1:I]$ is represented by a Markov Decision Process (MDP) $\mathcal{M}^{i}=\langle\mathcal{S}^{i},\mathcal{A}^{i},\mathcal{P}^{i},
\mathcal{R}^{i},\gamma\rangle$, where we use supscript $i$ to differentiate different tasks, $I$ is the number of total tasks, $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, respectively, $\mathcal{P}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ denotes the transition function, $\mathcal{R}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ is the reward function, and $\gamma\in[0,1)$ is the discount factor.
Conventional CL tasks have the same state and action spaces for all tasks, i.e., $|\mathcal{S}^{i}|=|\mathcal{S}^{j}|,|\mathcal{A}^{i}|=|\mathcal{A}^{j}|,
\forall~{}i,j\in[1:I]$.
While for any tasks sequences, we have $|\mathcal{S}^{i}|\neq|\mathcal{S}^{j}|,|\mathcal{A}^{i}|\neq|\mathcal{A}^{j}|$.
In the offline RL setting, we can only access pre-collected datasets $\{D^{i}\}_{i\in[1:I]}$ from each task $\mathcal{M}^{i}$.
The goal of continual offline RL is to find an optimal policy that can maximize the discounted return $\sum_{i}^{I}\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t}^{i},a_{t}^{i
})]$ (Fujimoto & Gu, ; Yang et al., ; Sun et al., ) on all tasks.

### 3.2 Conditional Generative Behavior Modeling

In this paper, we adopt a diffusion-based model with the U-net backbone as the generative model to fit the joint distribution $q(\tau_{s})=\int q(\tau_{s}^{0:K})d\tau_{s}^{1:K}$ of state sequences $\tau_{s}$ and an inverse dynamics model $\Psi(s_{t},s_{t+1})$ to produce actions $a_{t}$, where $\tau_{s}$ is the tailored state sequences to facilitate training, $k\in[1:K]$ is the diffusion step, $t$ is the RL time step, and we omit the identification of tasks for the sake of simplicity because the training is same for all tasks.
Through specifying the pre-defined forward diffusion process $q(\tau_{s}^{k}|\tau_{s}^{k-1})=\mathcal{N}(\tau_{s}^{k};\sqrt{\alpha_{k}}\tau_
{s}^{k-1},\beta_{k}\bm{I})$ and the trainable reverse process $p_{\theta}(\tau_{s}^{k-1}|\tau_{s}^{k})=\mathcal{N}(\tau_{s}^{k-1};\mu_{\theta
}(\tau_{s}^{k},k),\Sigma^{k})$ (Ho et al., ), we can train the diffusion model with the simplified loss function

$$ $\mathcal{L}(\theta)=\mathbb{E}_{k\sim U(1,2,...,K),\epsilon\sim\mathcal{N}(0, \bm{I}),\tau_{s}^{0}\sim D}[||\epsilon-\epsilon_{\theta}(\tau_{s}^{k},k)||_{2} ^{2}],$ (1) $$

where $\tau_{s}^{k}=\sqrt{\bar{\alpha}}\tau_{s}^{0}+\sqrt{1-\bar{\alpha}}\epsilon$, $\mu_{\theta}(\tau_{s}^{k})=\frac{1}{\sqrt{\alpha_{k}}}(\tau_{s}^{k}-\frac{
\beta_{k}}{\sqrt{1-\bar{\alpha}_{k}}}\epsilon_{\theta}(\tau_{s}^{k},k))$, $\Sigma^{k}=\frac{1-\bar{\alpha}_{k-1}}{1-\bar{\alpha}_{k}}\beta_{k}\bm{I}$, $\alpha_{k}$ is the approximate discretization pre-defined parameters (Chen et al., ; Lu et al., ), $\beta_{k}=1-\alpha_{k}$, $\bar{\alpha}_{k}=\prod_{\iota=1}^{k}{\alpha}_{\iota}$
, $U$ is the uniform distribution, $\epsilon$ is standard Gaussian noise, $\bm{I}$ is the identity matrix, we set the markov chain of SDE start from the sequential date of replay buffer $D$, i.e., $\tau_{s}^{0}=\tau_{s}$, and $\theta$ is the parameters of model $\epsilon_{\theta}$.

To achieve better control of action generation in RL, researchers propose classifier-guided and classifier-free guidance for diffusion models generation process $p_{\theta}(\tau_{s}^{k-1}|\tau_{s}^{k},\mathcal{C})$ (Dhariwal & Nichol, ; Liu et al., ), where the conditions are $\mathcal{C}$, which is usually selected as Q function in RL.
Inspired by the Bayes expression $p_{\theta,\phi}(\tau_{s}^{k-1}|\tau_{s}^{k},\mathcal{C})\propto p_{\theta}(
\tau_{s}^{k-1}|\tau_{s}^{k})p_{\phi}(\mathcal{C}|\tau_{s}^{k})$ and the properties of exponential family distribution, classifier-guided methods propose to add guidance on the mean value with a separated training model, i.e., $p(\tau_{s}^{k-1}|\tau_{s}^{k},\mathcal{C})=\mathcal{N}(\mu_{\theta}+\Sigma^{k}
\cdot\nabla log~{}p_{\phi}(\mathcal{C}|\tau_{s}),\Sigma^{k})$.
Classifier-free methods propose to model the correlation between the state sequences and conditions in the training phase by learning unconditional and conditional noise $\epsilon_{\theta}(\tau_{s}^{k},\emptyset,k)$ and $\epsilon_{\theta}(\tau_{s}^{k},\mathcal{C},k)$, where $\emptyset$ is usually the zero vector (Ajay et al., ).
At each diffusion step $k$, the perturbed noise is calculated by $\epsilon_{\theta}(\tau_{s}^{k},\emptyset,k)+\omega(\epsilon_{\theta}(\tau_{s}^
{k},\mathcal{C},k)-\epsilon_{\theta}(\tau_{s}^{k},\emptyset,k))$.

## 4 Method

Our method enables training on any CL task sequences through two sections (as shown in Figure 1): the selective weights activation diffuser (SWA) module and the quantized spaces alignment (QSA) module.
The detailed algorithm is shown in Algorithm 1 of Appendix A.1.
In the following parts, we introduce these two modules in detail.

Figure: Figure 1: The framework of VQ-CD. It contains two sections: The Quantized Space Alignment (QSA) module and the Selective Weights Activation (SWA) module, where QSA enables our method to adapt for any continual learning task setting by transferring the different state and action spaces to the same spaces. SWA uses selective neural network weight activation to maintain the knowledge of previous tasks through task-related weight masks. After the training, we perform weights assembling to integrate the total weights and save the memory budget.
Refer to caption: extracted/5936748/figures/framework.png

### 4.1 Quantized Spaces Alignment

To make our method suitable for solving any CL task sequence setting, we propose aligning the different state and action spaces with the quantization technique.
Specifically, we propose to solve the following quantized representation learning problem

$$ $\displaystyle\underset{\theta_{e},\theta_{d},\theta_{q}}{\text{min}}$ $\displaystyle\mathcal{L}_{QSA}(x;\theta_{e},\theta_{d},\theta_{q}),$ (2) s.t. $\displaystyle||z_{q}||_{2}^{2}<\rho,$ $$

where $\mathcal{L}_{QSA}(x)=\mathbb{E}\left[||x-f_{VQD}(z_{q};\theta_{d})||_{2}^{2}
\right]+\mathbb{E}\left[||\mathbf{sg}(z_{q})-z_{e}||_{2}^{2}\right]+\mathbb{E}
\left[||\mathbf{sg}(z_{e})-z_{q}||_{2}^{2}\right]$ is the total quantized loss, $\mathbf{sg}(\cdot)$ represents the stop gradient operation, $\theta_{e}$ and $\theta_{d}$ are the parameters of the vector quantized encoder (VQE) and vector quantized decoder (VQD), $\theta_{q}$ is the parameters of the codebook, $\rho$ limits the range of codebook embeddings, $x$ can represent the states or actions for each specific CL task, $z_{q}=f_{\theta_{q}}(z_{e})$ is the quantized representation which is consisted of fixed number of fixed-length quantized vectors, and $z_{e}=f_{VQE}(x;\theta_{e})$ is the output of the encoder.
Here, we propose searching the constrained optimal solution of the above problem for the consideration of the diffusion model training within a limited value range, just like the limit normalization in CV (Ho et al., ; Dhariwal & Nichol, ) and RL (Ajay et al., ; Lu et al., ).
There are many methods to force optimization under restricted constraints, such as converting the constraints to a penalty term (Boyd & Vandenberghe, ).
In our method, for simplicity and convenience, we propose to directly clip the quantized vector $z_{q}$ to meet the constraints after every codebook updating step.
Moreover, to meet the potential demand for extra tasks beyond the predefined CL tasks, we design the codebook as easy to equip, where the quantized spaces of different tasks are separated so that we can expediently train new task-related encoders, decoders, and quantized vectors.

For tasks where the state and action spaces are different, we can use the well-trained QSA module to obtain the aligned state feature $s^{i}_{z_{q}}=f^{i}_{s,\theta_{q}}(f^{i}_{VQE_{s}}(s^{i};\theta_{e}))$ and the action feature $a^{i}_{z_{q}}=f^{i}_{a,\theta_{q}}(f^{i}_{VQE_{a}}(a^{i};\theta_{e}))$ for each task $i$.
Thus, we can use $\tau^{i}_{s_{z_{q}}}$ and $\tau^{i}_{a_{z_{q}}}$ to represent the state and action feature sequences.

### 4.2 Selective Weights Activation

In this section, we introduce how to selectively activate different parameters of the diffusion model to reduce catastrophic forgetting and reserve disengaged weights for ongoing tasks.

Task Mask Generation.
Suppose that the diffusion model contains $L$ blocks, and the weights (i.e., parameters) of block $l$ are denoted by $W_{l},l\in\{1,...,L\}$.
There are two ways to disable the influence of the weights on the model outputs.
One is masking the output neurons $O_{l}=f_{l}(\cdot;W_{l})$ of each block, where $f_{l}(\cdot)$ is the neural network function of block $l$.
This strategy is friendly to MLP-based neural networks for two reasons: 1) the matrix calculation, such as $W_{l}*x$, is relatively simple so that we can easily recognize the disabled weights; 2) we do not need to apply any special operation on the optimizer because the output masking will cut off gradient flow naturally.
However, we can not arbitrarily apply the above masking strategy to more expressive network structures, such as convolution-based networks, because we can not easily distinguish the dependency between parameters and outputs.
Thus, we search for another masking strategy: masking the parameters $W_{l}$ with $M_{l}$, which permits us to control each parameter accurately.

Specifically, suppose that the total available mask positions of block $l$ are $M_{l}$.
In this paper, $M_{l}$ is a ones matrix, and the entries with 0 mean that we will perform masking.
Before training on task $i$, we first pre-define the specific mask $M_{i,l}$ of task $i$ on block $l$ by randomly sampling unmasked positions from the remaining available mask positions.
Then, with the increase of the tasks, the remaining available mask positions decrease until $M_{l}=\sum_{i=1}^{I}M_{i,l}$.

Selective Weights Forward and Backward Propagation.
After obtaining the mask $M_{i,l}$, we can perform forward propagation with masked weights

$$ $\displaystyle\epsilon_{\theta}(\tau_{\cdot}^{k},k)=f_{L}(f_{L-1}(...(f_{1}( \cdot))))$ (3) $\displaystyle O_{l+1}=f_{l+1}(O_{l},k;M_{i,l+1}\circ W_{l+1}),O_{0}=\tau^{i,k} _{s_{z_{q}}}/\tau^{i,k}_{a_{z_{q}}},$ $$

where $\epsilon_{\theta}$ is the noise prediction model introduced in Equation 1, and $M_{i,l+1}\circ W_{l+1}$ represents the pairwise product.
$\tau^{i,k}_{s_{z_{q}}}$ and $\tau^{i,k}_{a_{z_{q}}}$ denote the perturbed state or action sequences of task $i$ at diffusion step $k$.
Through forward designing, we can selectively activate different weights for different tasks through the mask $M_{i,l+1}$, thus preserving previously acquired knowledge and reserving disengaged weights for other tasks.
Though we can expediently calculate the masked output $O_{l+1}$ during forward propagation with weights or neurons masking, it poses a challenge to distinguishing the dependency from weights to loss and updating the corresponding weights during the backward propagation.
In order to update the corresponding weights, we realize two methods.
1) Intuitively, we propose to update the neural network with the sparse optimizer rather than the dense optimizer Diederik (), where the position and values of the parameters are recorded to update the corresponding weights.
However, in the implementation, we find that the physical time consumption of the sparse optimizer is intractable (Refer to Table 6 of Appendix B.5 for more details.), which encourages us to find a more straightforward and convenient method.
2) Thus, we propose extracting and assembling the corresponding weights at the end of the training rather than updating the corresponding weights during training.
This choice brings two benefits: (1) It can significantly reduce the time consumption spent on training. (2) It is friendly to implementation on complex network structures.

Weights Assembling.
Assembling weights after training permits us to save the total acquired knowledge and do not need extra memory budgets.
Concretely, after training on task $i$, we will obtain the weights $W_{i}$, which can be extracted with the mask $M_{i}$ from the total weights $W[i*\Omega]$, including all the diffusion model weights.
We use $W_{i}$ to denote the weights related to task $i$, $\Omega$ is the training step on each CL task, and $W[i*\Omega]$ represents the total weight checkpoint at training step $i*\Omega$.
Then, at the end of the training, we can assemble weights $\{W_{i}|i\in I\}$ by simply adding these weights together because of the exclusiveness property, i.e., $W=\sum_{i=1}^{I}W_{i}=\sum_{i=1}^{I}M_{i}\circ W[i*\Omega]$.

## 5 Experiments

In this section, we will introduce environmental settings, evaluation metrics, and baselines in the following sections.
Then, we will report and analyze the comparison results, ablation study, and parameter sensitivity analysis.
Other implementation details are shown in Appendix A.2 and  A.3.

### 5.1 Environmental Settings

Following previous studies (Zhang et al., ; Yang et al., ), we select MuJoCo Ant-dir and Continual World (CW) to formulate traditional CL settings with the same state and action spaces.
In Ant-dir, we select several tasks, such as 10-15-19-25 and 4-18-26-34-42-49, for training and evaluation.
In CW, we adopt the task setting of CW10, which contains 10 robotic manipulation tasks for CL performance comparison.
Additionally, we propose to leverage D4RL tasks (Fu et al., ) to construct the CL settings with diverse state and action spaces.
We select the Hopper, Walker2d, and HalfCheetah as elements to construct CL tasks, where each environment among Hopper, Walker2d, and HalfCheetah contains 6 qualities (random, medium, expert, medium-expert, medium-replay, and full-replay) datasets.

### 5.2 Evaluation Metrics

Considering the various reward structures of different environments, we should adopt different performance comparison metrics.
For Ant-dir, we adopt the average episodic return over all tasks as the performance comparison, i.e., the final performance $P=\text{mean}(\sum_{i}R_{i})$ is calculated based on the task $i$’s return $R_{i}$.
In the CW environment, previous works (Wołczyk et al., ; Anand & Precup, ) usually adopt the success rate $\Psi$ as the performance metric.
Thus, we adopt the average success rate on all tasks as the final performance, i.e., $P=\text{mean}(\sum_{i}\Psi_{i})$.
For the D4RL environments, we use the normalized score $\Phi$ (Wang et al., ; Huang et al., ) as the metric to calculate the performance $P=\text{mean}(\sum_{i}\Phi_{i})$, where $\Phi_{i}=\frac{R_{i}-R_{random}}{R_{expert}-R_{random}}*100$.
Usually, we can use the interface of these environments to obtain the score expediently.

### 5.3 Baselines

We select various representative CL baselines, which can be classified into diffusion-based and non-diffusion-based methods.
For example, the diffusion-based methods consist of CRIL (Gao et al., ), DGR (Shin et al., ), t-DGR (Yue et al., ), MTDIFF (He et al., ), CuGRO (Liu et al., ), CoD (Hu et al., ), and CoD variants.
The non-diffusion-based methods include EWC (Kirkpatrick et al., ), PackNet (Mallya & Lazebnik, ), Finetune, IL-rehearsal (Wan et al., ), and Multitask.
From the perspective of mainstream CL classification standards, these baselines can also be sorted as rehearsal-based methods (CRIL, DGR, t-DGR, CoD, and IL-rehearsal), regularization-based methods (EWC, CuGRO, and Finetune), and structure-based methods (PackNet, Multitask, and MTDIFF).

Figure: Figure 2: The comparison of VQ-CD and several baselines on the continual tasks setting (Ant-dir task 4-18-26-34-42-49). We train on each task for 500k steps. We report the normalized evaluation performance of VQ-CD in the top left corner, where the coordinates, e.g., task 4, represent evaluation on task 4 at different training tasks. To show the overall performance on all tasks, we show the normalized evaluation performance on the six tasks after finishing the training at the right part.
Refer to caption: extracted/5936748/figures/ant_dir.png

**Table 1: The comparison of VQ-CD, diffusion-based baselines, and LoRA methods on Ant-dir tasks, where the continual task sequence is 10-15-19-25. The results are average on 30 evaluation rollouts with 30 random seeds.**
| Method | VQ-CD<br>(ours) | CoD | Multitask<br>CoD | IL-<br>rehearsal | CoD-<br>LoRA | Diffuser-w/o<br>rehearsal | CoD-<br>RCR | MTDIFF | DD-w/o<br>rehearsal |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mean<br>return | 558.22$\pm$1.14 | 478.19$\pm$15.84 | 485.15$\pm$5.86 | 402.53$\pm$17.67 | 296.03$\pm$11.95 | 270.44$\pm$5.54 | 140.44$\pm$32.11 | 84.01$\pm$41.10 | -11.15$\pm$45.27 |

### 5.4 Experimental Results

In this section, we mainly separate the experimental settings into two categories, the traditional CL settings with the same state and action spaces and the arbitrary CL settings with different state and action spaces, to show the effectiveness of our method.
Besides, we also investigate the influence of the alignment techniques, such as auto-encoder, variational auto-encoder, vector-quantized variational auto-encoder (we adopt this in our method).
More deeply, we investigate how to deal with the potential demand for additional tasks beyond the pre-defined task length by releasing nonsignificant masks or expanding more available weights (Refer to Appendix B.4 for more details.).

The traditional CL settings correspond to the first question we want to answer: *Can VQ-CD achieve superior performance compared with previous methods in the traditional CL tasks?*

We use Ant-dir and Continual World (Zhang et al., ; Yang et al., ) to formulate the continual task sequence, where we select “10-15-19-25” as the task sequence in Ant-dir and “hammer-v2, push-wall-v2, faucet-close-v2, push-back-v2, stick-pull-v2, handle-press-side-v2, push-v2, shelf-place-v2, window-close-v2, peg-unplug-side-v2” to construct CW10 CL setting.
For simplicity, we do not align the state and action spaces with quantized alignment techniques because the traditional CL setting naturally has the same spaces.
The comparison results between our method and several diffusion-based baselines are shown in Table 1, where these baselines include rehearsal-based (CoD and Il-rehearsal), parameter-sharing (CoD-LoRA), multitask training (Multitask CoD and MTDIFF), and representative diffusion RL methods (Diffuser-w/o rehearsal, CoD-RCR, and DD-w/o rehearsal).
Our method surpasses all baselines in the Ant-dir task setting by a large margin, which directly shows the effectiveness of our method.
As another experiment of CL setting with the same state and action spaces, we report the results in Figure 3.
Compared with the upper bound performance of Multitask, our method reaches the same performance after the CL training.
With the increasing of new tasks, our method continually masters new tasks and sustains the performance while the baselines show varying degrees of performance attenuation, which can be found in the fluctuation of the curves.
Moreover, the final performance difference between one method and the Multitask method indicates the forgetting character, which can be reflected by the overall upward trend of these curves.
More experiments of shuffling task orders can be found in Appendix B.2.

Figure: Figure 3: The experiments on the CW10 tasks, which contain various robotics control tasks. We train each method on each task for 5e5 steps and use the mean success rate on all tasks as the performance metric. Generally, we can see the superiority of our method from the above figure.
Refer to caption: extracted/5936748/figures/offline_cw10.png

Figure: Figure 4: The comparison on the arbitrary CL settings. We select the D4RL tasks to formulate the CL task sequence. We leverage state and action padding to align the spaces. The experiments are conducted on various dataset qualities, where the results show that our method surpasses the baselines not only at the expert datasets but also at the non-expert datasets, which illustrates the wide task applicability of our method. The datasets characteristic “fr”, “mr”, “m”, and “me” represent “full-replay”, “medium-replay”, “medium”, and “medium-expert”, respectively. “Hopper”, “Walker2d”, and “Halfcheetah” are the different environments.
Refer to caption: extracted/5936748/figures/d4rl.png

The arbitrary CL settings correspond to the second question we want to answer: *Can we use the proposed space alignment method to enable VQ-CD to adapt to incoming tasks with various spaces?*

To answer the above question, we select D4RL to formulate the CL task sequence because of the various state and
action spaces, and the results are shown in Figure 4.
Considering the dataset qualities of D4RL (Fu et al., ), we choose different dataset quality settings and report the mean episode score that is calculated with $\frac{R_{i}-R_{random}}{R_{expert}-R_{random}}*100$.
Generally, from the four sub-experiments (a, b, c, and d), we can see that our method (VQ-CD) surpasses these baselines in all CL settings.
Especially in the CL settings (Figure 4 a and b), where the datasets contain low-quality trajectories, our method achieves a large performance margin even compared with the Multitask method.
We can attribute the reason to the return-based action generation that helps our method distinguish different quality trajectories and generate high-reward actions during evaluation, as well as the selective weights activation that can reserve the previous knowledge and reduce forgetting.
While other methods just possess the ability to continue learning and lack the ability to separate different qualities and actions, thus leading to poor performance.
For trajectory qualities that are similar across the datasets (Figure 4 c and d), we can see lower improvement gains between our method and baselines.
However, it should be noted that our method can still reach better performance than other baselines.
Apart from the padding alignment, we also conduct experiments (Figure 11) on baselines that adopt our pre-trained QSA module to align state and action spaces in Appendix B.3.

### 5.5 Ablation Study

In this section, we want to investigate the influence of different modules of VQ-CD.
Thus, the experiments contain two investigation directions: space alignment module ablation study and diffuser network structure ablation study.
To show the importance of vector quantization, we change the space alignment module with auto-encoder (AE) and variational auto-encoder (VAE).
Based on this modification, we retrain our method and report the results in Figure 5.
The results show significant improvements in the D4RL CL settings, illustrating the importance and effectiveness of vector quantization in our method.
Compared with AE-CD, VAE-CD performs poorer on all D4RL CL settings.
The reason lies in that the implicit Gaussian constraint on each dimension may hurt the space alignment.
Compared with the codebook in VQ-CD, AE-CD may cause a bigger difference between aligned features produced by AE (shown in Table 2), posing challenges for the diffusion model to model the distribution of the aligned features and leading to low performance.
As for the diffuser network structure, we conduct the selective weights activation on the mlp-based and unet-based diffusion models.
The latter structure is beneficial to making decisions with temporal information inside the trajectories, leading to higher performance evaluation.

Figure: Figure 5: The ablation study of space alignment module and diffusion network structure. For each type of ablation study, we fix the other same and retrain the model on four D4RL CL settings.
Refer to caption: extracted/5936748/figures/d4rl_alignment.png

**Table 2: The feature difference between the aligned features produced by the space alignment module. We randomly sample thousands of aligned state and action features to calculate the difference.**
| Method | VQ-CD | AE-CD |  |  |
| --- | --- | --- | --- | --- |
| feature difference | state difference | action difference | state difference | action difference |
| $[\text{Hopper-fr,Walker2d-fr,Halfcheetah-fr}]$ | 8.83$\pm$1.98 | 4.54$\pm$0.74 | 51.31$\pm$26.91 | 14.06$\pm$2.09 |
| $[\text{Hopper-mr,Walker2d-mr,Halfcheetah-mr}]$ | 9.03$\pm$1.97 | 4.45$\pm$0.74 | 48.12$\pm$21.94 | 15.39$\pm$3.71 |
| $[\text{Hopper-m,Walker2d-m,Halfcheetah-m}]$ | 8.53$\pm$1.56 | 4.22$\pm$0.79 | 42.27$\pm$24.29 | 13.59$\pm$2.63 |
| $[\text{Hopper-me,Walker2d-me,Halfcheetah-me}]$ | 8.93$\pm$2.00 | 4.05$\pm$0.56 | 57.91$\pm$36.94 | 13.93$\pm$3.20 |

Figure: Figure 6: The effects of different codebook sizes about the states.
Refer to caption: extracted/5936748/figures/offline_d4rl_performance_observation.png

Figure: Figure 7: The effects of the number of latent vectors about the actions.
Refer to caption: extracted/5936748/figures/offline_d4rl_performance_action.png

### 5.6 Parameter Sensitivity Analysis

When performing on the aligned feature with diffusion models, the hyperparameters of state and action of the quantized spaces alignment module matter.
Usually, the complexity of states is more significant than the actions, so the codebook size controls the performance of reconstruction.
Thus, we investigate the effect of different codebook sizes and report the results in Figure 6.
Obviously, a small codebook size limits performance, and a negative effect arises when it exceeds a certain value, such as 512.
As for the actions, we believe the actions can be decomposed into several small latent vectors, and the number of latent vectors is crucial for reconstructing actions.
Similarly, we also see the same trend in Figure 7, which inspires us that more latent vectors are not always better.

## 6 Conclusion

In this paper, we propose Vector-Quantized Continual Diffuser, called VQ-CD, which opens the door to training on any CL task sequences.
The advantage of this general ability to adapt to any CL task sequences stems from the two sections of our framework: the selective weights activation diffuser (SWA) module and the quantized spaces alignment (QSA) module.
SWA preserves the previous knowledge by separating task-related parameters with task-related masking.
QSA aligns the different state and action spaces so that we can perform training in the same aligned space.
Finally, we show the superiority of our method by conducting extensive experiments, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces).
The results illustrate that our method achieves the SOTA performance by comparing with 16 baselines on 15 continual learning task settings.

## Appendix A Algorithm

### A.1 Pseudocode of VQ-CD

Figure: Algorithm 1 Vector-Quantized Continual Diffuser (VQ-CD)

The training of VQ-CD (Pseudocode is shown in Algorithm 1) contains three stages.
1) We first pre-train the QSA module for space alignment, as shown in lines 4-13, where we mainly want to solve the constrained problem of Equation 2.
2) Then, in lines 15-31, for each task $i$, we generate the task-related mask $M_{i}$ followed by a standard diffusion model training process (Refer to Equation 1 and Equation 3 for the training loss) on the aligned state and action spaces.
3) Finally, we assemble the task-related weights $W_{i}$ together with the mask information $\{M_{i}|i\in[1:I]\}$ according to $W=\sum M_{i}\circ W[i*\Omega]$, where $\Omega$ is the training steps for each CL task, and $W[i*\Omega]$ is the weights checkpoints of $\epsilon_{i*\Omega,\theta}$.
It is noted that the pre-training of the QSA module and the training of the SWA module can be merged together, i.e., for each task $i$, we can first train the QSA module related to task $i$ and then train the SWA module.

### A.2 Hyperparameters

We classify the hyperparameters into three categories: QSA module-related, SWA module-related, and training-related hyperparameters.
We use the learning rate schedule when pre-training the QSA module, so the VQ learning rate decreases from 1e-3 to 1e-4.
In our experiments, the maximum diffusion steps are set as 200, and the default structure is Unet.
Usually, it is time-consuming for the diffusion-based model to generate actions in RL.
Thus, we consider the speed-up technique DDIM (Song et al., ) and realize it in our method to improve the generation efficiency during evaluation.
For all models, we use the Adam (Kingma & Ba, ) optimizer to perform parameter updating.

**Table 3: The hyperparameters of VQ-CD.**
|  | Hyperparameter | Value |
| --- | --- | --- |
| QSA section | network backbone | MLP |
| hidden dimension of QSA module | 256 |  |
| commitment cost coefficient | 0.25 |  |
| codebook embedding limit $\rho$ | 3.0 |  |
| state codebook size per task | 512 |  |
| number of state latent | 10 |  |
| state latent dimension | 2 |  |
| action codebook size per task | 512 |  |
| number of action latent | 5 |  |
| action latent dimension | 2 |  |
| alignment type | VQ/AE/VAE |  |
| VQ learning rate | [1e-4,1e-3] |  |
| SWA section | network backbone | Unet/MLP |
| hidden dimension | 256 |  |
| sequence length $T_{e}$ | 8 |  |
| diffusion learning rate | 3e-4 |  |
| guidance value | 0.95 |  |
| mask rate | $1/I$ |  |
| condition dropout | 0.25 |  |
| max diffusion step $K$ | 200 |  |
| sampling speed-up stride | 20 |  |
| condition guidance $\omega$ | 1.2 |  |
|  | sampling type of diffusion | DDIM |
| Training | loss function | MSE |
| batch size | 32 |  |
| optimizer | Adam |  |
| discount factor $\gamma$ | 0.99 |  |

### A.3 Computation

We conduct the experiments on NVIDIA GeForce RTX 3090 GPUs and NVIDIA A10 GPUs, and the CPU type is Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz.
Each run of the experiments spanned about 24-72 hours, depending on the algorithm and the length of task sequences.

**Table 4: The comparison of generation speed with different generation steps under the CL setting of Ant-dir task-4-18-26-34-42-49. In the main body of our manuscript, we use the 10 diffusion steps setting for all experiments.**
| Diffusion steps | 200 (original) | 100 | 50 | 25 | 20 | 10 |
| --- | --- | --- | --- | --- | --- | --- |
| sampling speed-up stride | 1 (original) | 2 | 4 | 8 | 10 | 20 |
| Time consumption of<br>per generation (s) | 5.73$\pm$0.29 | 2.88$\pm$0.21 | 1.41$\pm$0.16 | 0.71$\pm$0.18 | 0.58$\pm$0.17 | 0.29$\pm$0.15 |
| Speed-up ratio | 1× | 1.99× | 4.06× | 8.07× | 9.88× | 19.76× |

**Table 5: The GPU memory consumption.**
| domain | CL task setting | GPU memory consumption (GB) |
| --- | --- | --- |
| D4RL | $[\text{Hopper-fr,Walker2d-fr,Halfcheetah-fr}]$ | 4.583 |
| $[\text{Hopper-mr,Walker2d-mr,Halfcheetah-mr}]$ | 4.583 |  |
| $[\text{Hopper-m,Walker2d-m,Halfcheetah-m}]$ | 4.583 |  |
| $[\text{Hopper-me,Walker2d-me,Halfcheetah-me}]$ | 4.583 |  |
| Ant-dir | task-10-15-19-25 | 4.711 |
| task-4-18-26-34-42-49 | 5.955 |  |
| CW | CW10 | 5.897 |

### A.4 Generation Speed-up Technique

The time and memory consumption of diffusion models is attributed to the mechanism of diffusion generation process that requires multiple computation rounds to generate data Ho et al. ().
Fortunately, previous studies provide useful speed-up strategies to accelerate the generation process (Nichol & Dhariwal, ; Song et al., ).
In this paper, we adopt DDIM as the default generation speed-up technique and reduce the reverse diffusion generation step to 10 compared to the original 200 generation steps.
In Table 4, we use the CL setting of Ant-dir task-4-18-26-34-42-49 as an example to compare the time consumption of different generation steps.
Compared with the original 200 diffusion steps, we can see that incorporating DDIM will significantly (19.76×) improve the efficiency of generation.
In the experiments, we find that 10 diffusion steps setting performs well on performance and generation efficiency.
Thus, we set the default sampling speed-up stride to 20, and the diffusion step is 200/20=10 steps.

### A.5 Memory Consumption

In Table 5, we report the GPU memory consumption during the training process.
We mainly consider the experiments on the D4RL, Ant-dir, and CW CL tasks.
We can change the first block of the diffusion model to make our model suitable for a longer CL task sequence.
For example, we expand the dimension length from 512 to 1024 when switching the CL training task from ‘task-10-15-19-25’ to ‘task-4-18-26-34-42-49’.

## Appendix B Additional Experiments

### B.1 QSA Module Loss Analysis

Under the same hyperparameter settings in Section 5.6, we report the loss of the QSA Module to investigate the effects of codebook size and latent number.
For the states, we investigate the influence of codebook size, where we set codebook size as 128, 256, 512, 768, 1024, and select D4RL CL setting [Hopper-fr, Walker2d-fr, Halfcheetah-fr] and [Hopper-mr, Walker2d-mr, Halfcheetah-mr] as the example.
The results are shown in Figure 8, where we train the QSA module on each task for 5e5 steps.
We can see that for states, a codebook size of 512 is good enough for aligning the different tasks’ state spaces.
A larger codebook size, such as 768 and 1024 in Figure 8 a and b, will not bring significant loss improvements.
Smaller codebook sizes can not provide sufficient latent vectors to map the state spaces to a uniform space.

For the action, we select the latent number to explore the QSA action loss and report the results in Figure 9.
We can see the same trend that has been seen in QSA state loss (Figure 8).
Though the lower loss value of the more latent number indicates that we should use more action latent vectors, we find that the gap between action latent number settings 5 and 7 is small when we increase computation resources.
Besides, we also see inconspicuous performance gains in the final performance in Figure 7, which urges us to use 5 as the default action latent number setting.
For the action latent vector dimension, we directly use 2 as the default setting.

Figure: Figure 8: The QSA module loss under different codebook sizes about states. We explore five codebook size settings: 128, 256, 512, 768, and 1024. The red line represents the experimental codebook size setting for states.
Refer to caption: extracted/5936748/figures/offline_d4rl_parasensitivity_loss_observation.png

Figure: Figure 9: The QSA module loss under different latent numbers about actions. The setting includes 3, 5, 7, 9, and 11, which correspond to the aligned action space sizes 6, 10, 14, 18, and 22. The red line represents the experimental latent numbers setting for actions.
Refer to caption: extracted/5936748/figures/offline_d4rl_parasensitivity_loss_action.png

### B.2 Experiments of Task Order Shuffling

To investigate the influence of task order in CORL, we choose Ant-dir as the testbed and change the task order for new CL training.
We change the task order by inserting new tasks into the predefined task order ‘4-18-26-34-42-49’ and disrupting the task order.
We can see from the results shown in Figure 10 that our method achieves the best performance in almost all CL task orders.
The task order will affect the final performance of other baselines.
For example, CRIL performs better in the task orders ‘task 18-4-26-34-42-49’ and ‘task 49-42-34-26-18-4’ than in other task order experiments.
Another example is PackNet, which achieves the best performance only in the task order ‘task 34-18-4-26-42-49’.
Different from the baselines, whose performance fluctuates with the changing of task orders, our method (VQ-CD) shows stable training performance no matter what task orders are defined.

Figure: Figure 10: The experiments of Ant-dir with shuffled task order. We investigate the influence of shuffled task order in the Ant-dir environment, where the experiments include inserting new tasks into the predefined task order ‘4-18-26-34-42-49’ and disrupting the tasks order.
Refer to caption: extracted/5936748/figures/perturb_task_sequence.png

### B.3 Experiments of Baselines Equipped QSA

In Section 5.4, we report the comparison of our method and baselines in the arbitrary CL settings, where in the D4RL CL settings, we adopt state and action padding to align the state and action spaces.
Apart from the state and action padding, we can also use the pre-trained QSA module to align the different state and action spaces.
In Figure 11, we report the results of baselines equipped with QSA.
When introducing the QSA, the model is actually trained on the feature space rather than the original state and action spaces, which makes it hard to learn for these baselines proposed from the traditional CL setting.
From the results, we can also see that our method still achieves the best performance compared with these baselines.
Considering the results of Figure 5 (VQ-MLPCD) and Figure 11 (VQ baselines), we can see the importance of complementary sections: QSA and SWA.

Figure: Figure 11: The comparison on the arbitrary CL settings. We select the D4RL tasks to formulate the CL task sequence. In order to align the state and action spaces, we use the pre-trained QSA module (the same as our method) to provide aligned spaces during training. The experiments are conducted on various dataset qualities, where the results show that our method surpasses the baselines not only at the expert datasets but also at the non-expert datasets, which illustrates the wide task applicability of our method. The datasets characteristic “fr”, “mr”, “m”, and “me” represent “full-replay”, “medium-replay”, “medium”, and “medium-expert”, respectively. “Hopper”, “Walker2d”, and “Halfcheetah” are the different environments.
Refer to caption: extracted/5936748/figures/vq_d4rl.png

**Table 6: The comparison of time consumption per update between sparse and dense (normal) optimizers. We compare these two types of optimizers on the CL settings and find that when we first use the normal optimizer, such as Adam, to train the model and then use weights assembling to obtain the final model, the total physical time consumption is significantly smaller than sparse optimizer (e.g., sparse Adam).**
| domain | CL task setting | time consumption per update (s) |  |
| --- | --- | --- | --- |
| dense optimizer | sparse optimizer |  |  |
| D4RL | $[\text{Hopper-fr,Walker2d-fr,Halfcheetah-fr}]$ | 0.089$\pm$0.219 | 0.198$\pm$0.224 |
| $[\text{Hopper-mr,Walker2d-mr,Halfcheetah-mr}]$ | 0.096$\pm$0.223 | 0.197$\pm$0.223 |  |
| $[\text{Hopper-m,Walker2d-m,Halfcheetah-m}]$ | 0.089$\pm$0.211 | 0.195$\pm$0.224 |  |
| $[\text{Hopper-me,Walker2d-me,Halfcheetah-me}]$ | 0.090$\pm$0.223 | 0.206$\pm$0.225 |  |
| Ant-dir | task-10-15-19-25 | 0.062$\pm$0.064 | 0.239$\pm$0.282 |
| task-4-18-26-34-42-49 | 0.064$\pm$0.061 | 0.214$\pm$0.270 |  |
| CW | CW10 | 0.061$\pm$0.065 | 0.218$\pm$0.286 |

### B.4 Supporting Tasks Training Beyond the Pre-defined Task Sequence

After training on pre-defined task sequences, we may hope the model has the capacity to support training on potential tasks, which means that we need more weights or weight masks.
Releasing weight masks that are used to learn previous tasks is a straightforward choice when the total weights are fixed.
We conduct the experiments of mask pruning on Ant-dir ‘task 4-18-26-34-42-49’ and report the performance and weight mask prune rate when pruning weight masks according to certain absolute value thresholds in Figure 12.
The results illustrate that we can indeed release some weight masks under the constraint of preserving 90% or more performance compared with the unpruned model.
On the other hand, we can also see that this mask pruning method can only provide finite capacity for tasks beyond the pre-defined task sequence.
We postpone the systematic investigation of mask pruning to future works.

Figure: Figure 12: The mask pruning experiments of Ant-dir ‘task 4-18-26-34-42-49’. We investigate the task pruning according to the absolute weight values, i.e., we release the weights to train on potential new tasks according to the mask prune threshold.
Refer to caption: extracted/5936748/figures/ant_dir_mask_prun.png

### B.5 Time Consumption of Different Optimizers

In the CL settings of our experiments, we compare two types of optimizers and find that when we first use the normal optimizer, such as Adam, to train the model and then use weights assembling to obtain the final model, the total physical time consumption is significantly smaller than sparse optimizer (e.g., sparse Adam).
Thus, we propose the weights assembling to obtain the final well-trained model after the training rather than suffering huge time burden of sparse optimizer during the training.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Can the learned masks be visualized? Visualizing the masks could reveal task similarities and therefore can provide important insight for this proposed method. 
- Can this method detect non-periodic task changes?
soundness: 2
strengths: - Overall this paper contains a lot of new ideas. Using a VQ-VAE model to encode different tasks seems to be a reasonable way to enable the generalization of diffusion-based RL methods. 

- The idea of selecting activation weights in a U-Net diffusion model effectively addresses catastrophic forgetting by isolating task-specific weights. 

- The authors provided pretty comprehensive evaluation to demonstrate the effectiveness of this work both qualitatively and quantitatively.
summary: This paper introduces a method for the problem of continual offline reinforcement learning. The proposed method combines two modules: the Quantized Spaces Alignment (QSA) module, which standardizes different state and action spaces by using vector quantization to map them into a unified representation, and the Selective Weights Activation (SWA) module, which preserves knowledge from prior tasks by activating only relevant model weights for each new task through a masking mechanism. Extensive experiments show that the proposed method consistently outperforms existing methods, validating the effectiveness of this proposed method.
weaknesses: - The proposed method is a little bit complicated and might require significant computational resources and engineering efforts. 

- This method is designed for task-aware settings with explicit task boundaries, which could reduce its applicability in task-free continual learning scenarios where such boundaries are not predefined.

- It seems that the propose method, through effective, is a combination of existing learning methods, e.g. VQ-VAE, Diffusion, tasks masks.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. What is the "diffusion-based lifelong learning systems" in Abstract?
2. Figure 1 appears somewhat cluttered and hard to understand. Labeling the sequence within the figure would enhance its clarity.
soundness: 1
strengths: 1. Traditional continual reinforcement learning approaches are significantly limited as they only consider tasks with consistent state and action spaces. This paper successfully addresses the challenge of tasks with diverse state and action spaces through the innovative use of a quantized spaces alignment module. 
2. The baselines compared in the paper are thorough and comprehensive, and the experimental environments considered are sufficiently diverse. These aspects robustly demonstrate the superiority of VQ-CD.
summary: This paper propose Vector-Quantized Continual Diffuser (VQ-CD) for any offline continual reinforcement learning tasks, including the tasks with different observation and action space. Specifically, quantized spaces alignment (QSA) module aligns different state and action spaces (with even different dimensions) to perform continual learning within the same space. Selective weights activation diffuser (SWA) module preserves the previous knowledge by separating task-related parameters with task-related masking. VQ-CD achieves SOTA performance on 15 continual tasks comparing with 16 baselines.
weaknesses: The paper appears to lack a significant amount of crucial details, which hinders my understanding of the algorithms presented. 
1. The paper does not clarify how the 'return' is utilized. Based on Section 3.2, I guess that the algorithm employs the 'return' as a condition. So, is it the 'return' or the Q-function that is being used? Additionally, how is the target return or target function determined during the inference phase? 
2. Within the QSA module, which parameters are shared, and which need to be retrained for different tasks?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - $w$ in line 193 has not been defined, what is it?
- Is there any intuition about why the constraint in eq. (2) is important?
- The constraint in eq. (2) could easily be incorporated in the loss function, why is clipping used?
- What are "nonsignificant masks" (line 352)
- What is the source of the data for CW10 and MuJoCo Ant-dir?
- How is it possible that VQ-CD is better than Multitask on two of the D4RL tasks?
- Why would the gaussian constraint for VAE hurt space alignment?
- How does reconstruction look for VAE-CD in Table 2? Only VQ-CD and AE-CD are shown there.
- Why does multitask baseline performance decrease with VQ state/action spaces (Figure 11)?
soundness: 3
strengths: The idea of leveraging vector quantization to learn a unified state/action space is strong and enables joint training on spaces of different dimensionality.

Empirical evidence seems strong and the method is compared against plenty of baselines.

Ablation studies highlight the importance of combining VQ with CD.
summary: This work proposes a novel method for continual offline reinforcement learning called VQ-CD that combines vector quantization with diffusion for continual learning.
The vector quantization part enables training of a unified state and action space such that VQ-CD  can be trained jointly on environments that provide different state/action spaces.
The policy is based on a diffusion model and weight masking that relies on task information.
The weight masks are constructed in a manner that they can be readily merged into a single model after training.
VQ-CD exhibits strong performance on Ant-dir, CW10, and D4RL environments with different state/action spaces compared to baselines.
weaknesses: **Significance of results:**

While I appreciate the number of experiments conducted and baselines the authors trained, I am not entirely certain the compared methods are state of the art.
One particular baseline that comes to mind is L2M [1], which constructs unified state and action spaces  (similar to what QSA does) and incorporates a task matching mechanism to train separate weights for each task (similar to SWA).
Therefore, L2M should be added as a baseline. Moreover, the unified state/action space of L2M can be used as an additional ablation study to the VQ approach, as it is simpler and does not require training.


**Limitations:**

The authors do not elaborate on limitations, however there are notable ones that should be mentioned:
- Training needs to be conducted on each task separately, resulting in separate model checkpoints with different masked weights, this scales linearly with the number of tasks
- VQ-CD relies on task information, other methods [1] can be used in a task-agnostic manner

**Presentation:**

First, the title overclaims the contribution of the paper as it does not "solve" the field, please consider renaming.
Please also rephrase "any CL tas sequence settings", to something like "CL with different state/action spaces", as "any CL task" is ill-defined.
Figure 1 only shows the VQ-CD training pipeline, but line 465 talks about "return-based action generation" which is never explained in the methodology.
In fact, it is never explained whether the diffusion part constitutes the policy and how the inverse dynamics model is actually used.
Can the authors elaborate in more detail about what is the policy and how the inverse dynamics model is used?
Figure 2 is never referred to in the text.
Can the authors adjust Figure 4 so that the task boundaries are visible? As of now it is not clear at which steps the task is switched as there are now clear differences in the learning curves.
Line 423 sounds like state and action padding is being used for VQ-CD, but I assume it is only used for baselines, is this correct?
Line 526 sounds like one action vector is being decomposed into several latent vectors, however VQ only quantizes into a single latent.
It is not clear what the labels in Figure 7 indicate, can the authors elaborate and make it more explicit?

**References:**

[1] Learning to Modulate pre-trained Models in RL, Schmied et al., NeurIPS 2023

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: - For evaluation: is it necessary to know what task you are currently performing evaluation for?
- If no: how do you select the right mask at test time?
soundness: 3
strengths: The paper presents an interesting approach to continual RL using DDPMs, a rarely used technique that has great merit. In general, modeling of policies by generative models is a very attractive idea. Experimental verification uses established benchmarks and compares to established SOTA methods, producing good results. The ablation study and the parameter importance investigation are interesting as well. The idea of aligning perceptual and action spaces in a common representation is worthy of attention.
summary: The paper proposes a composite method to perform continual offline reinforcement learning by means of denoising diffusion probabilistic models (DDPMs). Each CORL task can have different observation and action spaces, which are aligned through a mechanism termed quantized space alignment (QSA). Continual learning performance is ensured by using a masking strategy (selective weights activation, SWA) that assigns distinct weights in the DDPM to distinct tasks. The paper presents an experimental evaluation on various  established continual RL benchmarks that show that the proposed method performs better than SOTA, as well as an ablation and parameter importance study to identify relevant parameters.
weaknesses: It is really hard to understand the big picture of the article. Figure 1 is not really helping since it does not indicate data flow through the model. It is only by studying the algorithm in the appendix that I could understand what the various components actually do, in which order. That algorithm should be moved to the main text, or els Fig.1 should be improved and simplified. 
        
 This is not really continual learning, as you are processing all data in one go in Alg.1 during the QSA pretraining step. In a CL setting, you would have access to data from task i only when training on task i, but not before. So, from a CL perspective, adapting QSA in one go prior to real RL training is cheating because you are looking into the future.
    
 In the same vein: *if* you have access to all collected data, why use CL at all and not train everything at once?
    
 It is not clear how this method could scale to more tasks when the mask is filled. It would be very helpful to comment on this, and why you do not simply train a separate DDPM for each task.    
        
 Important details are simply missing: 1) how do you use a diffusion model and the inverse dynamics model to generate actions? What role does $\psi$ play? What are the dimensions of obs and action spaces for the different tasks? How do you apply DDPMs to data that are not images (e.g., actions)?
    
 It is unclear how the SOTA methods are implemented, I can find no information at all except the references. What parameter settings? What libraries? Without this information, comparing to SOTA is meaningless since nothing can be reproduced.
    
It is not made clear what the benefit is of having a single RL controller learn policies for tasks with different state/action spaces. The paper would benefit from examples here.
        
The language of the paper is often imprecise or slightly incorrect, please be sure to fix this.

More comments:
3.2 the notion of "state sequences" and is unclear, and this paragraph is hard to understand. Maybe explain this in more detail. what is $\tau_s^{0:K}$ ?
3.2 Same goes for the notion of "inverse dyanmics model": what is this for? It is never mentioned again...
3.2 " tailored state sequences to facilitate training", what does this mean? Unclear...
General: very few details given for DDPM usage: input space, action space, ...
General: Continual offline reinforcement learning requires a more in-depth introduction and justification, as offline and reinforcemenet learning seem to be contradictory concepts. What is the advantage here? What can ORL do that RL cannot, and vice versa?

Meta-review: **summary**

This paper introduces Vector-Quantized Continual Diffuser (VQ-CD), a novel method for continual offline RL, capable of handling tasks with differing observation and action spaces. VQ-CD integrates two core modules: Quantized Spaces Alignment (QSA), which uses vector quantization to unify diverse state and action spaces into a shared representation, and Selective Weights Activation (SWA), which employs task-specific masking to preserve prior knowledge by isolating task-related parameters. Experiments on benchmarks like Ant-dir, CW10, D4RL, and others demonstrate that VQ-CD achieves state-of-the-art performance across various continual tasks, outperforming existing baselines.

---

**strengths**

* Novel approaches: (1) interesting idea of leveraging vector quantization to learn a unified state/action space and (2) leveraging generative models for policy learning

* Comprehensive Validation: Demonstrates state-of-the-art performance across diverse benchmarks with extensive experiments, including ablation studies and parameter analysis.

---

**weaknesses**

* [w1] Clarity of Writing: Some parts of the paper are unclear, making it difficult to fully understand the methodology.
* [w2] Baselines: The selection or implementation of baselines could be improved or better justified.
* [w3] Problem Setup Justification: The justification for the problem setup is unconvincing and could be elaborated further.

---

**decision**

All reviewers agreed that the idea is interesting and highlighted the diverse experimental environments as a strength. However, they also raised concerns about the baselines and the problem setup. During the discussion phase, some comments were partially addressed, but the paper requires revision to fully resolve these issues.

Decision: Reject