Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning

## Abstract

Abstract Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. While recent advances in pre-trained models provide a solid foundation for DIL, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. Specifically, sequential model updates can overwrite both the representation and the classifier with knowledge from the latest domain. Thus, it is crucial to develop a representation and corresponding classifier that accommodate all seen domains throughout the learning process.
To this end, we propose DUal ConsolidaTion ( Duct ) to unify and consolidate historical knowledge at both the representation and classifier levels.
By merging the backbone of different stages, we create a representation space suitable for multiple domains incrementally. The merged representation serves as a balanced intermediary that captures task-specific features from all seen domains. Additionally, to address the mismatch between consolidated embeddings and the classifier, we introduce an extra classifier consolidation process. Leveraging class-wise semantic information, we estimate the classifier weights of old domains within the latest embedding space. By merging historical and estimated classifiers, we align them with the consolidated embedding space, facilitating incremental classification. Extensive experimental results on four benchmark datasets demonstrate Duct ’s state-of-the-art performance.

## 1 Introduction

Recent years have seen the rise of deep learning, demonstrating its strong potential in real-world applications He et al. (); Deng et al. (); Floridi & Chiriatti ().
However, in a dynamic and ever-changing world, data often evolves in a stream format Aggarwal (); Ye et al. (), necessitating continuous updates with data from new domains Gama et al. ().
For instance, autonomous vehicles must handle driving tasks across different seasons and weather conditions Bojarski et al. (), and face recognition systems are supposed to recognize users despite the varying lighting conditions Zhao et al. ().
Correspondingly, Domain-Incremental Learning (DIL) van de Ven et al. () addresses this challenge by absorbing new knowledge from new data distributions while preserving existing knowledge.
However, sequentially updating the model with new domains often biases the embedding Yu et al. () and classifier Zhao et al. (); Wu et al. () modules toward the latest domain, leading to catastrophic forgetting Mermillod et al. (); De Lange et al. () of previous knowledge.
Therefore, developing methods for learning new domains without forgetting existing knowledge becomes a critical challenge for the machine learning community.

To combat catastrophic forgetting, recent works leverage strong pre-trained models (PTMs)Han et al. () as the initialization for DILWang et al. (; ; ). PTMs’ robust representation abilities provide a powerful starting point, showing promising results in DIL benchmarks. These approaches freeze the pre-trained backbone to preserve existing knowledge and append lightweight modules Jia et al. (); Chen et al. (); Lian et al. () to capture new patterns.
However, since incoming domains will also overwrite the appended modules, the representations still suffer from forgetting, leading to corresponding bias in the classifier.

In DIL, there are two primary sources of forgetting: the overwriting of features and classifiers. Continually adjusting the features to capture the latest task biases the representations toward the newest domain.
For example, suppose the previous task involves real-world photos while the subsequent task includes quick-draw images. The features will be overwhelmingly adjusted to highlight non-objective details, which are less helpful in distinguishing the previous domain, thus leading to forgetting.
In light of this, an ideal feature space in DIL should fit all domains and equally highlight the domain-specific features for all seen tasks.
However, as features become less generalizable due to overwriting, the classifier also becomes biased toward the latest domain.
Since the classifier plays the pivotal role of mapping embeddings to classes, using the biased classifier can cause detrimental effects to decision-making.
Consequently, it is imperative to consolidate existing knowledge both in the feature and classifier to resist forgetting.

There are two main challenges to achieving this goal. 1) Building a unified embedding space that suits all tasks continually. Since the training data arrives in a stream format, we cannot access all training instances simultaneously, preventing us from achieving the ‘oracle embedding’ that favors all seen domains. 2) Calibrating the prediction of biased classifiers. Even if we achieve a holistic embedding suitable for all tasks, we still cannot build an accurate mapping between features and classes due to the lack of previous instances. Consequently, a classifier consolidation process is needed to align the classifier weights with the changing features continually.

In this paper, we propose DUal ConsolidaTion (Duct) to address these challenges. To counter feature drift, we introduce a representation merging technique that continuously integrates historical backbones. This merged embedding consolidates task-specific information from all previous domains, providing informative features without forgetting. Consequently, we achieve non-forgettable features incrementally and resist feature-level forgetting.
To map updated features to classes, we design a classifier consolidation process that merges historical classifier weights with calibrated weights. By extracting class-wise semantic relationships in the joint space, we develop a classifier transport mechanism that estimates classifiers of previous domain classes in the latest embedding space. Through consolidating features and classifiers in this coordinated way, Duct balances all seen domains and robustly resists forgetting, achieving competitive performance on various benchmarks.

## 2 Related Work

Continual Learning/Incremental Learning is a popular topic in the machine learning field, aiming to incorporate new knowledge within streaming data De Lange et al. (); Masana et al. (); Wang et al. (); Zhao et al. (); Zheng et al. (); Qi et al. (). It can be further classified into three subcategories, i.e., task-incremental learning (TIL) De Lange et al. (), class-incremental learning (CIL) Masana et al. (), and domain-incremental learning (DIL) van de Ven et al. (). Among them, TIL and CIL are faced with emerging new classes and are required to learn them without forgetting.
By contrast, DIL mainly focuses on the scenario where label space is fixed while incoming data involves new domains Wang et al. (); Shi & Wang ().
In continual learning, typical solutions include using data replay Bang et al. (); Aljundi et al. (); Isele & Cosgun (); Rolnick et al. (); Ratcliff () to review former knowledge, using knowledge distillation Hinton et al. (); Rebuffi et al. (); Li & Hoiem () or parameter regularization Kirkpatrick et al. (); Zenke et al. (); Chaudhry et al. (); Aljundi et al. (; ) to maintain existing knowledge. Recent advances also resort to bias correction Hou et al. (); Zhao et al. (); Ahn et al. (); Castro et al. (); Wu et al. (); Belouadah & Popescu () to rectify the model bias or expand the network structure as data evolves Rusu et al. (); Aljundi et al. (); Yan et al. (); Wang et al. (); Zhou et al. ().

Domain-Incremental Learning with Pre-Trained Models: With the rapid development of pre-training techniques, PTMs provide a strong initialization for DIL models to boost the feature generalizability Smith et al. (); Wang et al. (; ; ).
Current PTM-based DIL methods mainly resort to visual prompt tuning Jia et al. () to adjust the pre-trained features. With the pre-trained features frozen, it learns a prompt pool as external knowledge and selects instance-specific prompts to encode task information. L2P Wang et al. () utilizes the query-key matching mechanism to select instance-specific prompts, while DualPrompt Wang et al. () extends it by learning task-specific and instance-specific prompts jointly.
By contrast, S-Prompts Wang et al. () learns domain-specific prompts for DIL and retrieves prompts via KNN search.
To alleviate the prompt selection cost, CODA-Prompt Smith et al. () modifies the prompt selection process into an attention-based weighted combination.
Apart from prompt tuning, there are also some methods that directly build classifiers upon the pre-trained features Zhou et al. (); McDonnell et al. ().

Model Merging:
Recent advances in pre-training have made weight interpolation among different models a useful technique in model editing Ainsworth et al. (); Frankle et al. (); Ilharco et al. (); Matena & Raffel (); Wortsman et al. (; ). Most work focuses on simple averaging of multiple models, aiming to enhance model in terms of its performance on the single task and robustness to out-of-distribution data.
In this paper, we consider merging the pre-trained weights and task vector parameters to cultivate multitask representation for all seen domains.

## 3 From Old Domains to New Domains

In this section, we introduce the background information about domain-incremental learning and the baselines for applying pre-trained models to domain-incremental learning.

### 3.1 Domain-Incremental Learning

In domain-incremental learning, the model is faced with the data stream with a sequence of tasks $\left\{\mathcal{D}^{1},\mathcal{D}^{2},\cdots,\mathcal{D}^{B}\right\}$, where $\mathcal{D}^{b}=\left\{\mathcal{X}_{b},\mathcal{Y}_{b}\right\}$ is the $b$-th training task, i.e., $\mathcal{X}_{b}=\left\{{\bf x}_{i}\right\}_{i=1}^{n_{b}}$ and $\mathcal{Y}_{b}=\left\{y_{i}\right\}_{i=1}^{n_{b}}$.
Each training instance ${\bf x}_{i}\in\mathbb{R}^{D}$ belongs to class $y_{i}\in Y$.
This paper follows the exemplar-free setting Zhu et al. (); Wang et al. (), i.e., during the $b$-th training stage, we can only access data in the current dataset $\mathcal{D}^{b}$. In DIL, the label space $Y$ is unchanged throughout the learning process, while the distribution of input instance keeps changing from domain to domain, i.e., $p(\mathcal{X}_{b})\neq p(\mathcal{X}_{b^{\prime}})$ for $b\neq b^{\prime}$. The target of DIL is to fit a model $f({\bf x})$ that can discriminate the classes among any seen domains:

$$ $f^{*}=\underset{f\in\mathcal{H}}{\operatorname{argmin}}\enspace\mathbb{E}_{( \mathbf{x},y)\sim\mathcal{D}_{t}^{1}\cup\cdots\mathcal{D}_{t}^{b}}\mathbb{I}(y \neq f(\mathbf{x}))\,,$ (1) $$

where $\mathcal{H}$ is the hypothesis space, $\mathbb{I}(\cdot)$ is the indicator function which outputs $1$ if the expression holds and 0 0 otherwise. $\mathcal{D}_{t}^{b}$ denotes the data distribution of task $b$.
Consequently, DIL models are supposed to classify instances of new domains while not forgetting previous ones.

Following Wang et al. (; ; ); Smith et al. (), we assume a pre-trained Vision Transformer (ViT) Dosovitskiy et al. () is available as the initialization for $f({\bf x})$.
For simplicity, we decouple the network structure into the embedding function $\phi(\cdot):\mathbb{R}^{D}\rightarrow\mathbb{R}^{d}$ (i.e., the final [CLS] token) and the linear classifier $W\in\mathbb{R}^{d\times|Y|}$.
In this way, the output is denoted as $f({\bf x})=W^{\top}\phi({\bf x})$, and we utilize a cosine classifier in this paper.
The classifier is further decoupled into
$W=[{\bf w}_{1},{\bf w}_{2},\cdots,{\bf w}_{|{Y}|}]$, where ${\bf w}_{j}$ denotes the classifier weight of the $j$-th class.
Although the label space is static during the learning process, data distribution of the same class across different domains can yield significant domain gaps. Hence, we follow existing works Wang et al. (; ; ); Smith et al. () to expand the classifier $W\in\mathbb{R}^{d\times b|Y|}$ as new tasks emerge, and the final prediction is made by $\left(\operatorname*{arg\,max}_{i}{\bf w}_{i}^{\top}\phi({\bf x})\right)
\textbf{ mod }|Y|$.

### 3.2 Baselines in Domain-Incremental Learning

In domain-incremental learning, a naive solution facing the incoming datasets of new domains is to directly optimize the embedding and classifier:

$$ $\min_{W\cup\phi}\sum_{({\bf x},y)\in{D^{b}}}\ell\left(W^{\top}\phi({\bf x}),y \right)\,.$ (2) $$

With pre-trained weights as initialization, Eq. 2 can quickly capture the discriminative features within the new domain. However, since the embedding $\phi(\cdot)$ is kept changing among different domains, it quickly loses the generalization ability to previous domains and suffers from forgetting.

To resist feature-level forgetting, a feasible solution is to freeze the pre-trained weights and append lightweight modules to encode domain-specific knowledge, e.g., prompt Jia et al. (). A representative work L2P Wang et al. () organizes a set of prompts as the prompt pool (i.e., $\mathbf{Pool}$) and optimizes the model via:

$$ $\min_{\mathbf{Pool}\cup W}\sum_{({\bf x},y)\in{D^{b}}}\ell\left(W^{\top}\bar{ \phi}\left({\bf x};\mathbf{Pool}\right),y\right)+\mathcal{L}_{\mathbf{Pool}}\,,$ (3) $$

where $\bar{\phi}\left({\bf x};\mathbf{Pool}\right)$ denotes the prompted feature representation with backbone frozen, and $\mathcal{L}_{\mathbf{Pool}}$ denotes the prompt selection loss Wang et al. ().
Eq. 3 dynamically retrieves instance-specific prompts to adjust the representations and learns the classifier to map the features to corresponding classes. Since the backbone weights are frozen, it can alleviate the representation drift during updating.

Discussions:
Although Eq. 2 and Eq. 3 adopt different policies for model updating, both of them suffer from the feature-level forgetting.
Specifically, tuning the model via Eq. 2 can encode task-specific knowledge into the embedding,
but also comes with the risk of feature being fully overwritten by new domains since there is no restriction on retaining previous knowledge.
Besides, although Eq. 3 freezes the pre-trained weights, the trainable prompts appended may also incur forgetting.
Since the prompted feature relies on instance-specific prompt selection, optimizing the prompts for new domains still has the risk of overwriting previous ones.
With the features being biased toward the latest domain, the classifier is also adjusted upon it. Consequently, the classifiers of previous tasks are incompatible with the unstable and ever-changing features, resulting in the classifier-level forgetting.

## 4 Duct : Dual Consolidation for Domain-Incremental Learning

Observing the risk of feature-level and classifier-level overwriting, we aim to resist forgetting in DIL from two aspects.
Firstly, alleviating feature-level forgetting requires obtaining a unified embedding space at a low cost.
Since sequentially updating the backbone or prompts will both result in forgetting, a proper solution is needed to make full use of all historical features and consolidate them into a unified one.
Secondly, as the representation changes from task to task, the mismatch between classifier and embedding becomes more severe as the data evolves.
Consequently, a classifier consolidation step is needed to calibrate them to be compatible with the embeddings.

In the following sections, we introduce the learning paradigm for representation consolidation and classifier consolidation. Lastly, we provide detailed guidelines for training and inference.

### 4.1 Representation Consolidation

Observing the challenge of striking a balance between all seen domains, we need to opt for another feature updating policy to avoid sequential overwriting and resist forgetting.
Let us assume an ideal scenario where we can separately train each model for each incoming domain, obtaining a set of models $\{\phi_{1}(\cdot),W_{1}\},\cdots,\{\phi_{B}(\cdot),W_{B}\}$. Those models are obtained by optimizing the same PTM via Eq. 2, thus having the discriminability for each specific domain and can be seen as the ‘domain expert.’
In an oversimplified scenario where we know which domain the instance is from, we can directly utilize the corresponding expert for prediction.
However, since such auxiliary information is unavailable in DIL, we need to obtain a omnipotent embedding space that suits all domains. In this way, there is no need to decide which embedding to use since the omnipotent embedding is strong enough to capture all domain-specific features.

Correspondingly, we take inspiration from the model merging community Ilharco et al. (); Ramé et al. () that an ideal model for multiple domains can be achieved by combining multiple task vectors.
We denote the relative change of the embedding function as $\delta_{\phi_{i}}=\phi_{i}-\phi_{0}$, where $\phi_{0}$ is the weight of the pre-trained model.
Hence, $\delta_{\phi_{i}}$ represents the relative change of the weights in the $i$-th domain, and we can build the unified embedding via:

$$ $\phi^{m}_{i}=\phi_{0}+\alpha_{\phi}\sum_{i}\delta_{\phi_{i}}\,,$ (4) $$

which combines the pre-trained weight with every task vector. Since all the fine-tuned embeddings start from the same pre-trained weight, Ilharco et al. (); Zhang & Bottou () verify the task vectors share low similarity due to the semantic gap between different domains.
Consequently, adding such weights enables the merged model to highlight all task-specific features, and Eq. 4 provides a feasible way to obtain the universal feature representation that suits all seen domains.

Representation consolidation with task similarity: Although Eq. 4 provides a simple way to consolidate multiple task vectors to build a unified embedding, it still lacks consideration in measuring task-wise similarities. For example, if two distinct domains are more similar, highlighting those features would be more helpful for recognizing classes of corresponding domains. Hence, we introduce task-similarity ($\textbf{Sim}_{0,i}$) into the representation consolidation process and replace Eq. 4 into:

$$ $\phi^{m}_{i}=\phi_{0}+\alpha_{\phi}\sum_{i}\textbf{Sim}_{0,i}\delta_{\phi_{i}}\,,$ (5) $$

Specifically, since the merged embedding is built upon pre-trained weights and task vectors, we consider the similarity of pre-trained model $\phi_{0}$ and subsequent models $\phi_{i}$ to adjust the merging process. A naive way to is to directly measure the cosine similarity between $\phi_{0}$ and $\phi_{i}$. However, since pre-trained weights are with millions of dimensions, the similarity in such space is ineffective due to curse of dimensionality. To this end, we utilize the current training task as an indicator for task similarity. Specifically, for every class in $\mathcal{D}^{b}$, we utilize the backbone $\phi_{i}$ to extract class centers in its embedding space:

$$ $\textstyle{\bm{c}}^{i}_{p}=\sum_{j=1}^{|\mathcal{{D}}^{b}|}\mathbb{I}(y_{j}=p) \phi_{i}({\bf x}_{j})/\sum_{j=1}^{|\mathcal{{D}}^{b}|}\mathbb{I}(y_{j}=p)\,.$ (6) $$

Afterward, we can obtain two sets of class centers in the embedding space, i.e., ${\bm{C}^{0}}=[{\bm{c}}^{0}_{1},{\bm{c}}^{0}_{2},\cdots,{\bm{c}}^{0}_{|{Y}|}]$, ${\bm{C}^{i}}=[{\bm{c}}^{i}_{1},{\bm{c}}^{i}_{2},\cdots,{\bm{c}}^{i}_{|{Y}|}]$. Since those class centers are representative points in the embedding space, we calculate the pair-wise class similarity to indicate task similarities:

$$ $\textstyle\textbf{Sim}_{0,i}=\frac{1}{|{Y}|}\sum^{|{Y}|}_{j=1}\text{sim}({\bm{ c}}^{0}_{j},{\bm{c}}^{i}_{j})\,,$ (7) $$

where we utilize cosine similarity to calculate center-wise similarity $\text{sim}(\cdot,\cdot)$.

Effect of representation consolidation: Figure 1(top) visualizes the representation consolidation process, where the merged backbone can unify multiple domains.
Through Eq. 5, we are able to build the joint embedding space that suits all tasks by combining them in the parameter space. Since tasks in DIL emerge one-by-one, the representation consolidation process can also be made incrementally, i.e., $\phi^{m}_{i}=\phi^{m}_{i-1}+\alpha_{\phi}\textbf{Sim}_{0,i}\delta_{\phi_{i}}$. In this way, we can get rid of the extensive memory cost and only keep at most two backbones in memory.
In each training stage, we first fine-tune the model via Eq. 2 and conduct representation consolidation to aggregate the representations. In this way, we obtain a unified embedding space across all seen tasks.

Figure: Figure 1: Illustration of Duct. Top: Representation consolidation. We utilize the pre-trained model as initialization and optimize it for each domain, obtaining the task vectors. Afterward, we combine the pre-trained model and all seen task vectors to build the unified embedding space. Bottom: Classifier consolidation. To align the classifiers with consolidated features, we design the new classifier retraining and old classifier transport to consolidate classifiers. Class-wise semantic information is utilized in classifier transport.
Refer to caption: x1.png

### 4.2 Classifier Consolidation

In Eq. 5, we design the feature consolidation process, which aggregates multiple fine-tuned models by accumulating task vectors.
However, a fatal problem still exists, i.e., there is a mismatch between the classifiers and the consolidated embeddings.
Since the classifiers are optimized to match the embeddings to the corresponding class, the matching degree drastically decays as the backbone is replaced with another one.
Recalling the background information in Section 3.1 that we utilize an independent classifier for each domain, we need to design the extra classifier consolidation process to align the classifiers to the embedding space.
In each training stage, we denote the classifier for previous domains as ‘old classifier’ ($W_{o}$) and the classifier for the current domain as ‘new classifier’ ($W_{n}$). We then discuss how to align them with the consolidated features.

New Classifier Retraining: In each training stage, we have the training data $\mathcal{D}^{b}$ in hand. Consequently, it is intuitive to coordinate the new classifier with the consolidated features via:

$$ $\min_{W_{n}}\sum_{({\bf x},y)\in{D^{b}}}\ell\left(W_{n}^{\top}\bar{\phi}_{i}^{ m}({\bf x}),y\right)\,,$ (8) $$

where the consolidated feature $\bar{\phi}_{i}^{m}(\cdot)$ is frozen to resist further mismatch, and we can align the new classifier ${W_{n}}$ to match the features with Eq. 8.

Old Classifier Transport: Eq. 8 aligns the new classifier to the latest embedding space. However, a fatal problem still exists, i.e., the old classifier is also incompatible with the merged embedding space.
Consequently, the previous knowledge of old domains shall be forgotten in the incremental learning process.
If we have plenty of training instances of previous domains, a similar calibration process can be done as in Eq. 8.
However, due to the exemplar-free restrictions, we cannot save any previous instances in the memory.
Hence, we need to find another way to estimate the relative calibration weight for the old classifier.
For example, if we have the retrained classifier to classify a ‘lion’ in the clip art style, it can be mostly reused to classify the lion in the real photo style. We call such a relationship ‘semantic information,’ and the goal is to utilize such information to assist old classifier alignment.
We denote the calibrated classifier using semantic information as $\hat{W}_{o}=\mathcal{T}(W_{n},\textbf{S})$, i.e., the estimated classifier is obtained by transforming the new classifier using semantic information S.
Hence, there remain two core problems to be solved: 1) How to define the transformation function $\mathcal{T}$? 2) How to define the semantic information S?

Implementing $\mathcal{T}$ via Optimal Transport: The function $\mathcal{T}$ encodes the correlation between the set of features among two tasks.
Considering that the weight matrix of a linear layer reveals the relative relationship among feature-class pairs and the final logit is the aggregation of all features, we can design the recombination of exiting classifiers with a linear mapping $T\in\mathbb{R}^{\beta\times\gamma}$.
$T$ encodes the cross-task correlation between the current and the previous domains, where larger values in $T$ denote higher class-wise similarity.
Since the decision is made by matching the classifier with corresponding features, we can utilize and recombine the weights of similar classes in the current domain to obtain those of previous domains. For example, important features that discriminate lions in the clip art style should be assigned higher coefficients to help classify lions in the photo style and vice versa.

We denote $\bm{\mu}_{1}\in\Delta_{\beta}$ and $\bm{\mu}_{2}\in\Delta_{\gamma}$, where
$\Delta_{d}=\left\{\bm{\mu}:\bm{\mu}\in\mathbb{R}_{+}^{d},\bm{\mu}^{\top}
\mathbf{1}=1\right\}$ is the $d$-dimensional simplex.
$\bm{\mu}_{1}$ and $\bm{\mu}_{2}$ denote the importance of each class across domains, and we set them as
uniform distribution.
To map a set of classes to another between old and new domains, a cost matrix $Q\in\mathbb{R}_{+}^{\beta\times\gamma}$ is further introduced to guide the transition.
The larger weight of $Q_{i,j}$ indicates we need to pay more cost when reusing the classifier of $i$-th class to assist the $j$-th class. Consequently, the matrix $T$ can be formulated as the coupling of two distributions, aiming to connect classes from different domains at the lowest transportation cost. Hence, $T$ can be obtained via minimizing:

$$ $\displaystyle\min_{T}\langle T,Q\rangle\quad\text{ s.t. }T\mathbf{1}=\bm{\mu}_ {1},T^{\top}\mathbf{1}=\bm{\mu}_{2},T\geq 0\,,$ (9) $$

which is the Kantorovich formulation of optimal transport (OT) Villani (); Peyré et al. ().
Optimizing Eq. 9 enables us to show how to move the probability mass across domains with low cost. Since the target is to reuse the retrained classifier $W_{n}$ to adjust previous classifier $W_{o}$, we set $\bm{\mu}_{1}\in\Delta_{|Y|}$ and $\bm{\mu}_{2}\in\Delta_{|Y_{o}|}$ with uniform class marginals, where $|Y_{o}|$ denotes the number of classes in $W_{o}$. Solving Eq. 9 produces the alignment between different domains, and we can apply $T$ to the new classifier to obtain the estimated old classifier, i.e., $\hat{W}_{o}=W_{n}T$.

Defining the Transportation Cost with Semantic Information: Solving Eq. 9 requires a proper definition of the cross-domain cost, i.e., $Q$. The higher cost indicates it is less effective to transport the classifier to the target class and vice versa. To this end, we design a simple yet effective way to measure such class-wise information. Specifically, before the training of each stage, we utilize the pre-trained backbone $\phi_{0}$ to extract class centers in its embedding space via Eq. 6, i.e., $[{\bm{c}}^{0}_{1},{\bm{c}}^{0}_{2},\cdots,{\bm{c}}^{0}_{|{Y}|}]$.
These class centers indicate the most representative embedding of the corresponding classes.
Since $\phi_{0}$ is optimized with extensive datasets, we rely on its generalizability to reflect task-wise similarity.
If two classes are similar, their embeddings should also be situated near each other.
Consequently, we calculate the Euclidean distance between class centers as the transportation cost, i.e., $Q_{i,j}=\left\|{\bm{c}}^{0}_{i}-{\bm{c}}^{0}_{j}\right\|_{2}^{2}$.
Here classes $i$ and $j$ are from different domains.
With the pair-wise transportation cost, we are able to optimize Eq. 9 for the transportation plan $T$. Afterward, we utilize the interpolation between old classifiers and the transported classifier for consolidation:

$$ $W_{o}^{m}=(1-\alpha_{W})W_{o}+\alpha_{W}\hat{W}_{o}=(1-\alpha_{W})W_{o}+\alpha _{W}W_{n}T\,.$ (10) $$

Effect of classifier consolidation: We visualize the classifier consolidation process in Figure 1 (bottom). The classifier consolidation process takes two steps, i.e., new classifier retraining and old classifier transport. The first step is designed to align the new classifiers with the unified embedding, while the second step aims to recombine the old classifiers with the calibrated classifier. In this way, we obtain a compatible classifier with the consolidated features, thus favoring the recognition in the unified embedding space.

### 4.3 Summary of Duct

Figure: Algorithm 1 Duct for DIL

We summarize the training steps in Algorithm 1. Specifically, facing a new training task, we extract the class centers with $\phi_{0}$. Afterward, we optimize the model and separately consolidate the features and classifiers. The consolidated model ($[W_{o}^{m};W_{n}]^{\top}\phi^{m}_{i}$) is then utilized for testing. We utilize Sinkhorn’s algorithm Sinkhorn & Knopp () to solve the OT problem in Eq. 9.

Discussions on memory cost: During the training process, we need to keep two models in the memory, i.e., the historical consolidated backbone $\phi^{m}_{i}$ and the online backbone $\phi_{i}$ for learning the current task.
However, after the learning process of each domain, the current running model is merged into $\phi^{m}_{i}$ (Eq. 5) without needing to be kept in memory.
During inference, we utilize the consolidated backbone $\phi^{m}_{i}$ and classifiers $[W_{o}^{m};W_{n}]$ for classification, which is the same as a single backbone. Consequently, Duct can be fairly compared with others.

## 5 Experiment

In this section, we conduct experiments on benchmark datasets to compare Duct to existing state-of-the-art methods. We also provide ablation studies on the components and parameter robustness to analyze the effect of different modules. Visualizations of the embedding space also verify the effectiveness of our proposed method.

Figure: (a) Office-Home ViT-B/16 IN1K
Refer to caption: x2.png

**Table 1: Average and last performance of different methods among five task orders. The best performance is shown in bold. All methods are implemented with ViT-B/16 IN1K. Methods with $\dagger$ indicate implemented with exemplars (10 per class).**
| Method | Office-Home | DomainNet | CORe50 | CDDB |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\bar{\mathcal{A}}$ | ${\mathcal{A}_{B}}$ | $\bar{\mathcal{A}}$ | ${\mathcal{A}_{B}}$ | $\bar{\mathcal{A}}$ | ${\mathcal{A}_{B}}$ | $\bar{\mathcal{A}}$ | ${\mathcal{A}_{B}}$ |  |  |
| Finetune | 78.32$\pm$3.28 | 76.16$\pm$1.39 | 28.17$\pm$6.47 | 38.82$\pm$7.65 | 75.44$\pm$1.68 | 76.19$\pm$2.36 | 52.08$\pm$1.35 | 50.11$\pm$1.62 |  |
| Replay^† Ratcliff () | 84.23$\pm$2.31 | 83.75$\pm$0.68 | 64.78$\pm$2.98 | 61.16$\pm$1.19 | 85.56$\pm$0.38 | 92.21$\pm$0.63 | 66.91$\pm$18.0 | 63.21$\pm$11.6 |  |
| iCaRL^† Rebuffi et al. () | 81.66$\pm$2.43 | 81.11$\pm$1.23 | 59.89$\pm$2.86 | 57.46$\pm$2.31 | 74.43$\pm$3.18 | 79.86$\pm$2.96 | 68.43$\pm$18.7 | 70.50$\pm$16.5 |  |
| MEMO^† Zhou et al. () | 71.18$\pm$2.76 | 63.09$\pm$1.80 | 61.92$\pm$5.39 | 58.41$\pm$3.20 | 64.80$\pm$3.16 | 68.24$\pm$2.41 | 60.87$\pm$13.4 | 58.09$\pm$11.7 |  |
| SimpleCIL Zhou et al. () | 75.69$\pm$5.03 | 75.72$\pm$0.00 | 42.95$\pm$4.84 | 44.08$\pm$0.00 | 70.92$\pm$0.74 | 74.80$\pm$0.00 | 60.80$\pm$4.49 | 63.40$\pm$0.00 |  |
| L2P Wang et al. () | 79.72$\pm$4.19 | 80.03$\pm$1.29 | 50.45$\pm$4.10 | 48.72$\pm$2.83 | 83.57$\pm$0.35 | 87.87$\pm$0.51 | 67.33$\pm$5.98 | 64.45$\pm$5.83 |  |
| DualPrompt Wang et al. () | 80.20$\pm$3.81 | 80.85$\pm$0.14 | 52.28$\pm$3.35 | 50.46$\pm$3.17 | 84.53$\pm$0.89 | 87.27$\pm$1.06 | 68.33$\pm$7.52 | 71.41$\pm$1.24 |  |
| CODA-Prompt Smith et al. () | 84.70$\pm$2.94 | 85.07$\pm$0.34 | 59.85$\pm$4.49 | 59.99$\pm$0.88 | 87.92$\pm$0.41 | 91.57$\pm$0.69 | 69.19$\pm$6.11 | 74.18$\pm$1.33 |  |
| EASE Zhou et al. () | 81.16$\pm$3.52 | 76.33$\pm$2.16 | 50.50$\pm$2.27 | 43.72$\pm$1.70 | 86.30$\pm$0.04 | 87.02$\pm$1.21 | 67.78$\pm$2.44 | 64.96$\pm$8.36 |  |
| RanPAC McDonnell et al. () | 82.30$\pm$3.34 | 82.28$\pm$0.00 | 55.20$\pm$3.93 | 54.80$\pm$0.36 | 79.16$\pm$0.55 | 81.38$\pm$0.12 | 78.92$\pm$4.85 | 80.48$\pm$0.68 |  |
| S-iPrompt Wang et al. () | 81.50$\pm$3.17 | 80.51$\pm$0.21 | 61.16$\pm$4.57 | 60.46$\pm$0.90 | 81.95$\pm$0.57 | 83.38$\pm$0.70 | 68.51$\pm$7.20 | 72.76$\pm$0.43 |  |
| Duct | 86.27$\pm$2.95 | 86.91$\pm$0.06 | 67.16$\pm$3.75 | 67.01$\pm$1.35 | 91.95$\pm$0.15 | 94.47$\pm$0.33 | 84.14$\pm$4.37 | 85.10$\pm$0.52 |  |

### 5.1 Experimental Setup

Dataset: Following the benchmark settings Wang et al. (; ; ); Smith et al. () in PTM-based domain-incremental learning, we evaluate the performance on Office-Home Venkateswara et al. (), DomainNet Peng et al. (), CORe50 Lomonaco & Maltoni (), and CDDB-Hard Li et al. ().
Specifically, Office-Home has four domains, DomainNet has six domains, CORe50 has 11 domains, and CDDB-Hard has five domains. We consider five task orders to shuffle these domains in the DIL setting for a holistic evaluation and report the details in
Section B.2.

Comparison methods: In the comparison, we consider two types of methods, including exemplar-based methods, i.e., Replay Ratcliff (), iCaRL Rebuffi et al. (), MEMO Zhou et al. (), and SOTA exemplar-free DIL methods, i.e., SimpleCIL Zhou et al. (), L2P Wang et al. (), DualPrompt Wang et al. (), CODA-Prompt Smith et al. (), EASE Zhou et al. (), RanPAC McDonnell et al. (), and S-iPrompt Wang et al. (). We utilize the same pre-trained backbone for all compared methods.

Implementation details:
We deploy the experiments using PyTorch Paszke et al. () and Pilot Sun et al. () on NVIDIA 4090.
Following Wang et al. (); Zhou et al. (), we consider two typical pre-trained weights, i.e., ViT-B/16-IN21K and ViT-B/16-IN1K. Both are pre-trained with ImageNet21K Russakovsky et al. (), while the latter is further finetuned on ImageNet1K.
We optimize Duct using SGD optimizer with a batch size of 128 for 15 epochs. The learning rate is set to $0.001$. We select 10 exemplars per class for exemplar-based methods using herding Welling () algorithm.
In Duct, we set the consolidation parameter $\alpha_{\phi}=0.5,\alpha_{w}=0.5$.
The code will be made publicly available upon acceptance.

Performance Measure: Following Wang et al. (; ), we denote the accuracy among all seen domains after learning the $b$-th task as $\mathcal{A}_{b}$, we mainly consider $\mathcal{A}_{B}$ (the performance after learning the last stage) and $\bar{\mathcal{A}}=\frac{1}{B}\sum_{b=1}^{B}\mathcal{A}_{b}$ (the average performance among all incremental stages) for comparison. We also consider the forgetting measure Chaudhry et al. () to measure the relative forgetting degree in DIL.

### 5.2 Benchmark Comparison

We report the results on four benchmark datasets in Figure 2. As we can infer from these results, Duct consistently outperforms other compared methods by $\bf 1\sim 7$% in the final accuracy.
It must be noted that the differences of the starting point are due to the different tuning techniques, which cannot be directly aligned since the number of trainable parameters are different in those methods.
Specifically, sequentially finetuning the model suffers from catastrophic forgetting, even starting with the pre-trained model. To compensate for the forgetting phenomena, we observe that several exemplar-based methods (Replay, iCaRL, and MEMO) show stable improvements to the baseline. However, since some of these works are designed for the class-incremental learning scenario, the expansion or distillation target may not be optimal for the current setting.
We then compare Duct to the methods specially designed for PTMs and find prompt-based methods (L2P, DualPrompt, CODA-Prompt, and S-iPrompt) yield inferior performance to ours. We also observe that Duct is compatible with various pre-trained weights and shows stable improvements with ViT-B/16 IN1K (a$\sim$d) and IN21K (e$\sim$f).

Besides, we also report the incremental performance among five task orders (average and standard deviation) in Table 1 and Figure 3(a). As we can infer from the table, Duct works robustly among different task orders, showing stable improvements against other state-of-the-art methods.

Figure: (a) Multiple trails
Refer to caption: x8.png

### 5.3 Further Analysis

Forgetting Measure: Figure 2 and Table 1 mainly focus on the accuracy measure, which utilizes all seen domains to measure the relative performance. Apart from the accuracy measure, we also follow Wang et al. (; ) to utilize the forgetting measure, which captures the stability of existing knowledge among previous domains. As shown in Figure 3(b), we report the forgetting measure among all compared methods on the CDDB dataset. As we can infer from the figure, several methods suffer from severe forgetting, e.g., Finetune and MEMO, indicating that they are unsuitable for the domain-incremental learning scenario. We also observe that prompt-based methods freeze the backbone to resist feature drift, which tackles the forgetting phenomena. However, Duct still shows the least forgetting (i.e., 0.12) among all compared methods, indicating its strong performance.

Parameter Robustness: There are two major consolidation steps in Duct, i.e., the representation consolidation in Eq. 5 and the classifier consolidation in Eq. 10. These consolidation steps include the merging parameters between a set of models/classifiers, i.e., the backbone merge ratio $\alpha_{\phi}$ and the classifier merge ratio $\alpha_{W}$. In this section, we conduct ablations on the choice of these parameters to investigate the robustness of Duct with change of them. Specifically, we choose $\alpha_{\phi}$ and $\alpha_{W}$ among $\{0.1,0.25,0.5,0.75,1.0\}$, resulting in 25 parameter combinations. We conduct experiments with these parameter combinations on the Office-Home dataset and show the average performance in Figure 3(c). As we can infer from the figure, Duct is generally robust to parameter changes. Besides, since the merge parameters are designed to trade-off between old and new knowledge, we find a small value (e.g., 0.1) or large value (e.g., 1.0) works poorly. The poor performance of these marginal parameters also indicates the importance of the merging process since a small value or large value equals to ignoring part of the important information during consolidation.
By contrast, both of them prefer a mild value, i.e., choosing them around {0.25, 0.5} shows better performance. Hence, we suggest setting $\alpha_{\phi}=0.5,\alpha_{w}=0.5$ as the default setting.

Figure: Table 2: Ablation study on differnt modules in Duct.
Refer to caption: extracted/5892119/pics/duct/tsne_unmerged.png

Compositional Components: In this section, we conduct experiments to investigate the importance of each module in Duct on CDDB dataset. As shown in Table 2, ‘Baseline’ denotes directly freezing the embedding and extracting class centers as the classifier, which shows poor performance due to the domain gap to the downstream domains. We then utilize Eq. 4 to sequentially merge the backbone to consolidate the representations and denote it as Variation 1. It shows that adding the representation consolidation process drastically improves the performance, indicating the superiority of such unified representation in DIL.
However, when switching Eq. 4 to Eq. 5, we find Variation 2 further improves the performance, indicating that task similarity is helpful in obtaining a universal embedding space.
Afterward, we combine it with the classifier retraining process in Eq. 8, which is denoted as Variation 3. It shows that the mismatch between consolidated features and the classifiers weakens the performance, and aligning the new classifiers to the embedding helps DIL. When comparing Duct to Variation 3, we find the old classifier transport is also vital to recover former knowledge and resist forgetting, which improves the final accuracy by 2%. Ablation studies verify the efficacy of different modules in Duct.

Visualizations: In this section, we visualize the embedding space to show the effectiveness of Duct on the DomainNet dataset. We consider a two-stage domain-incremental learning scenario, each containing five classes, and utilize t-SNE Van der Maaten & Hinton () to visualize the representation space before and after Duct.
We use dots to represent the classes in the first domain and triangles to represent classes in the second domain. We utilize the shadow regions to denote the decision boundary obtained by the classifier weights to represent different classes. As shown in Figure 4, although the embedding space before Duct shows competitive performance, there are two major flaws: 1) There exists the confusion region between yellow and red dots, indicating that previous classes are partially forgotten as data evolves. 2) The embedding space of the same class is still located in different regions, e.g., the purple and green classes, which is not ideal for domain-incremental learning. However, when applying Duct to consolidate the features and classifiers, the above problems are addressed in Figure 5. Specifically, since the consolidated embedding suits all tasks, the forgetting of previous domains is alleviated. Besides, the unified embedding space adequately places the same class of different domains together, favoring the final inference.

## 6 Conclusion

Domain-incremental learning is a desired ability for real-world learning systems to obtain new knowledge. In this paper, we propose dual consolidation (Duct) to achieve this goal. Since the forgetting in DIL occurs in two aspects, i.e., the embedding and the classifier, we separately design the consolidation technique to handle them. Specifically, we consider merging the pre-trained model with domain-specific task vectors to achieve the unified embedding. To compensate for the mismatch between consolidated features and classifiers, we design the classifier consolidation process by introducing semantic-guided transport. Extensive experiments validate the effectiveness of Duct.

Limitations: Possible limitations include the utilization of PTMs since feature consolidation relies on generalizable initialization. Future work includes extending Duct to non-PTM scenarios.

## Appendix A Supplied Experimental Results

In this section, we supply additional experiments to show the effectiveness of Duct, including the running time comparison, the detailed performance among different task orders, and more results with other pre-trained weights.

### A.1 Running time comparison

In this section, we report the running time comparison among different compared methods. As shown in Figure 7, Duct costs competitive running time against other compared methods while having the best performance.

Figure: Figure 7: Running time comparison among different methods. Duct shows the best performance while having competitive training costs.
Refer to caption: x11.png

### A.2 Performance of different task orders

In the main paper, we conduct experiments on the benchmark datasets with five task orders and report the average performance.
In this section, we report the performance on each order in Table 3, 4, 5, 6, 7. The task sequences are reported in Section B.2.

**Table 8: Details on domain size, train/test split, and instance number of the benchmark datasets. The dataset split and selection follows Wang et al. (; ; ); Smith et al. ().**
|  | Domains | Size | Test set |
| --- | --- | --- | --- |
| CDDB-Hard | biggan | 4.0k | standard splits |
| gaugan | 10.0k | 75%:25% |  |
| san | 440 | (’san’ - 80%:20%) |  |
| whichfaceisreal | 2.0k |  |  |
| wild | 10.5k |  |  |
| CORe50 | s1 | 14.9k | s3, s7, s10 |
| s2 | 14.9k | Indoor:Outdoor |  |
| s4 | 14.9k |  |  |
| s5 | 14.9k |  |  |
| s6 | 14.9k |  |  |
| s8 | 14.9k |  |  |
| s9 | 14.9k |  |  |
| s11 | 14.9k |  |  |
| DomainNet | clipart | 48.1k | standard splits |
| infograph | 51.6k | 70%:30% |  |
| painting | 72.2k |  |  |
| quickdraw | 172.5k |  |  |
| real | 172.9k |  |  |
| sketch | 69.1k |  |  |
| Office-Home | Art | 2.4k | random splits |
| Clipart | 4.3k | 70%:30% |  |
| Product | 4.4k |  |  |
| Real World | 4.3k |  |  |

### A.3 Different backbones

In the main paper, we mainly report the performance with ViT-B/16-IN1K.
Since Duct is robust with the change of backbones, we report the performance with ViT-B/16-IN21K in this section.
Apart from the benchmark datasets reported in Figure 2, the performance on the rest of the benchmarks is shown in Figure 8.

Figure: (a) Office-Home
Refer to caption: x12.png

**Table 9: Task orders of CDDB-Hard.**
| CDDB-Hard | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 |
| --- | --- | --- | --- | --- | --- |
| Order 1 | san | whichfaceisreal | biggan | wild | gaugan |
| Order 2 | wild | whichfaceisreal | san | gaugan | biggan |
| Order 3 | biggan | gaugan | wild | whichfaceisreal | san |
| Order 4 | gaugan | biggan | wild | whichfaceisreal | san |
| Order 5 | whichfaceisreal | san | gaugan | biggan | wild |

**Table 10: Task orders of CORe50.**
| CORe50 | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Task 7 | Task 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Order 1 | s11 | s4 | s2 | s9 | s1 | s6 | s5 | s8 |
| Order 2 | s2 | s9 | s1 | s6 | s5 | s8 | s11 | s4 |
| Order 3 | s4 | s1 | s9 | s2 | s5 | s6 | s8 | s11 |
| Order 4 | s1 | s9 | s2 | s5 | s6 | s8 | s11 | s4 |
| Order 5 | s9 | s2 | s5 | s6 | s8 | s11 | s4 | s1 |

**Table 11: Task orders of DomainNet.**
| DomainNet | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 |
| --- | --- | --- | --- | --- | --- | --- |
| Order 1 | clipart | infograph | painting | quickdraw | real | sketch |
| Order 2 | infograph | painting | quickdraw | real | sketch | clipart |
| Order 3 | painting | quickdraw | real | sketch | clipart | infograph |
| Order 4 | quickdraw | real | sketch | clipart | infograph | painting |
| Order 5 | real | quickdraw | painting | sketch | infograph | clipart |

**Table 12: Task orders of Office-Home.**
| Office-Home | Task 1 | Task 2 | Task 3 | Task 4 |
| --- | --- | --- | --- | --- |
| Order 1 | Art | Clipart | Product | Real_World |
| Order 2 | Clipart | Product | Real_World | Art |
| Order 3 | Product | Clipart | Real_World | Art |
| Order 4 | Real_World | Product | Clipart | Art |
| Order 5 | Art | Real_World | Product | Clipart |

## Appendix B Dataset Details

In this section, we introduce the details about datasets, including the dataset information (i.e., the number of tasks and instances) and split information (i.e., the five splits adopted in the main paper).

### B.1 Dataset introduction

We report the details about benchmark datasets in Table 8, and they are listed as follows.

- •
DomainNet Peng et al. ()(^1^11https://ai.bu.edu/M3SDA/) is a dataset of common objects in six different domains. All domains include 345 categories of objects, such as Bracelets, planes, birds, and cellos. The domains include clipart — a collection of clipart images; real — photos and real-world images; sketch — sketches of specific objects; infograph — infographic images with specific objects; painting — artistic depictions of objects in the form of paintings, and quickdraw — drawings of the worldwide players of the game ‘Quick Draw!’. We use the officially recommended version ‘Cleaned’ in this paper.
- •
Office-Home Venkateswara et al. ()(^2^22https://hemanthdv.github.io/officehome-dataset/) is a benchmark dataset for domain adaptation which contains four domains where each domain consists of 65 categories. The four domains are Art — artistic images in the form of sketches, paintings, ornamentation, etc.; Clipart — a collection of clipart images; Product — images of objects without a background; and Real-World — images of objects captured with a regular camera. It contains 15,500 images, with an average of around 70 images per class and a maximum of 99 images in a class.
- •
CDDB-Hard Li et al. ()(^3^33https://coral79.github.io/CDDB_web/). As a dataset mixed with real-world and model-generated images, the continual deepfake detection benchmark (CDDB) aims to simulate real-world deepfakes’ evolution.
The authors put out three scenarios for evaluation, ‘EASY,’ ‘Hard,’ and ’Long,’ respectively.
Following Wang et al. (), we choose the ’Hard’ track for evaluation since it poses a more challenging problem due to its complexity.
- •
CORe50 Lomonaco & Maltoni ()(^4^44https://vlomonaco.github.io/core50/index.html#dataset). Composed of 11 sessions characterized by different backgrounds and lighting, CORe50 is built for continual object recognition. Numerous RGB-D images are divided into 8 indoor sessions for training and 3 outdoor sessions for testing, and each session includes a sequence of about 300 frames for all 50 objects.

### B.2 Domain Sequences

In domain-incremental learning, different algorithms’ performances may be influenced by the order of domains. Consequently, we randomly shuffle the domains and organize five domain orders in the main paper, which are further utilized for a holistic evaluation.
The task orders are reported in Table 9, 10, 11, 12.

## Appendix C Compared Methods

In this section, we introduce the methods that were compared in the main paper. Note that we re-implement all methods using the same pre-trained model as initialization. They are listed as follows.

- •
Finetune is a simple baseline in DIL, which directly optimizes the model with cross-entropy loss. It will suffer catastrophic forgetting since there is no restriction on preserving previous knowledge.
- •
Replay Ratcliff () is an exemplar-based method, which saves a set of exemplars from previous domains (i.e., in this paper, we save 10 exemplars per class) and replay them when learning new domains. Hence, forgetting can be alleviated since the model can revisit informative instances from previous domains when learning new ones. Of note, for classes with fewer than 10 instances, repeatable sampling is allowed to conform to the requirement.
- •
iCaRL Rebuffi et al. () is a knowledge distillation-based continual learning algorithm, which saves the previous model in memory. During updating, apart from the cross-entropy loss for learning new tasks, it also introduces the knowledge distillation loss between old and new models to avoid forgetting. It also requires saving an increasing number of exemplars.
- •
MEMO Zhou et al. () is an expansion-based continual learning algorithm that partially expands the network to catch new features. As for the implementation, we follow the original paper to decouple the network and expand the last transformer block for each new task.
- •
SimpleCIL Zhou et al. () proposes this simple baseline in pre-trained model-based continual learning. It freezes the backbone representation, extracts the class center of each class, and utilizes a cosine classifier updated by assigning class centers to the classifier weights.
- •
L2P Wang et al. () is the first work introducing prompt tuning in continual learning. With the pre-trained weights frozen, it learns a prompt pool containing many prompts. During training and inference, instance-specific prompts are selected to produce the instance-specific embeddings. However, as alluded to before, learning new domains will lead to the overwriting of existing prompts, thus triggering forgetting.
- •
DualPrompt Wang et al. () extends L2P in two aspects. Apart from the prompt pool and prompt selection mechanism, it further introduces prompts instilled at different depths and task-specific prompts. During training and inference, the instance-specific and task-specific prompts work together to adjust the embeddings.
- •
CODA-Prompt Smith et al. () aims to avoid the prompt selection cost in L2P. It treats prompts in the prompt pool as bases and utilizes the attention results to combine multiple prompts as the instance-specific prompt.
- •
EASE Zhou et al. () designs lightweight feature expansion technique with adapters to learn new features as data devolves. To fetch a classifier with the same dimension as ever-expanding features, it utilizes class-wise similarity to complete missing class prototypes.
- •
RanPAC McDonnell et al. () extends SimpleCIL by randomly projecting the features into the high-dimensional space and learning the online LDA classifier for final classification.
- •
S-iPrompt Wang et al. () is specially designed for pre-trained model-based domain-incremental learning. It learns task-specific prompts for each domain and saves domain centers in the memory with K-Means. During inference, it first forwards the features to select the nearest domain center via KNN search. Afterward, the selected prompt will be appended to the input.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1.	DUCT introduces approach to domain-incremental learning by addressing both feature and classifier forgetting simultaneously, providing a fresh perspective on solving the catastrophic forgetting problem.
2.	DUCT cleverly combines ideas from model merging and optimal transport. The merging of task vectors with the pre-trained model and the use of optimal transport for classifier alignment are creative applications of existing techniques to the DIL context.
3.	The paper presents comprehensive experimental results across four benchmark datasets and five task orders, demonstrating the robustness and effectiveness of DUCT.
4.	This paper is well-organized, with clear sections and logical flow. The methodology is explained in detail, and the experimental setup and results are presented clearly.
summary: This paper introduces DUCT, a dual consolidation technique for domain-incremental learning (DIL) that effectively mitigates catastrophic forgetting. DUCT addresses the challenge of balancing knowledge across domains by Representation Consolidation and Classifier Consolidation. The paper demonstrates DUCT’s effectiveness through extensive experiments on four benchmark datasets, showing it consistently outperforms state-of-the-art methods in terms of accuracy and forgetting measure.
weaknesses: 1.	There are many vision language models like CLIP that can perform zero shot. Can the author report the results of CLIP and CLIP related fine-tuning methods such as Coop, CoCoop, etc., to demonstrate the advantages of the article's method compared to these general models.
2.	I wonder what data is used to calculate the class center of the pretrained model? If using pretrained data such as ImageNet, the first issue is how to ensure consistency with downstream task categories to calculate class center similarity? The second question is how to reduce the overhead caused by large number of categories and data size?
3.	The author should conduct experiments on more backbones to demonstrate the effectiveness of the method, such as convolutional neural networks like Resnet.
4.	Task similarity is calculated based on all categories, may it lead to the influence of some categories being overly magnified while the influence of others is ignored.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: see the weakness
soundness: 2
strengths: The authors analyze the current challenge of forgetting in domain incremental learning (DIL) and its underlying causes.
They propose a model-merging approach that demonstrates promising accuracy.
summary: The DUal ConsolidaTion (DUCT) framework addresses feature drift and forgetting by integrating historical backbones through a representation merging technique, creating stable task-specific embeddings. Additionally, DUCT’s classifier consolidation process merges calibrated and historical weights, preserving semantic class relationships and resisting forgetting, yielding strong performance on diverse benchmarks.
weaknesses: 1. Equations 4 and 5 attempt to build a unified embedding through weighted summation of model weights, raising questions about feasibility. Given the complexity and lack of interpretability in deep network weights, is this combination effective, or could it intensify conflicts within the feature space? More comprehensive theoretical analysis is required.
2. While the authors suggest that DIL could benefit applications like autonomous vehicles and face recognition, their experiments focus on classification tasks. Testing on more realistic applications could be more convincing.
3. The proposed DUCT method relies on model merging. However, as domains accumulate, previously merged models may become overly complex, containing information from multiple domains, while models from newer domains include only the latest domain data. This could lead to an imbalance between older and newer domains, creating potential confusion and forgetting.
4. The authors tested DUCT on ViT-B/16, but other methods, like S-Prompts, report results on the more powerful CLIP backbone. Does DUCT maintain its effectiveness on a stronger backbone?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Why do you need the two stage merging? Could you just absorb the linear weights into $\phi$ and use equation (5)?

Is there anyway you could calculate an upper bound for Table1 (e.g. performance of finetuning on the union of all datasets?)
soundness: 3
strengths: (1) The manuscript is well-organized despite the complicate method. 

(2) The method is novel.

(3) Results are good.
summary: I authors propose DUCT, a method for domain incremental learning (DIL). DIL is the setting where a sequence of tasks is presented during model finetuning. The training algorithm does not have access to data from prior tasks. The authors decompose the task overfitting problem into two components: (1) representation overfitting and (2) classifier overfitting. The authors tackle to two problems separately and propose novel model-merging-inspired techniques to solve both.
weaknesses: (1) Many notations are not introduced before hand. This makes the math hard to follow and the method ambiguous. For example, what is $\phi_i^m$ and $\alpha_\phi$ in equation 4? Furthermore, equations are not introduced in the correct order. For example, Eq. 5 depends on a value that is not defined until Eq. 7. 

(2) It is unclear why the proposed method is better than model merging (intuitively).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1.What is the impact on the results if the proposed method does not use task vectors during integration?
2.In lines 198-200 of page 4, the authors claim that the proposed method can capture the domain-specific features of all domains. This is an interesting claim. How do the authors prove it?
soundness: 2
strengths: 1.This paper is well-motivated. It is easy to understand the motivation of the method. It is reasonable to consider both feature extraction and classification in incremental learning.
2.The experimental results show that the proposed method performs better than the previous methods, which successfully demonstrates the effectiveness of the proposed method.
summary: This paper is motivated by the forgetting problem of features and the mismatch problem of the classifier in domain-incremental learning. Then this paper proposes to address the above problems by unifying the historical knowledge at both the feature and classifier level. In particular, this paper proposes to merge the backbone of different stages and utilize optimal transport to adapt the classifier of old domains to both the new domain and the merged backbone. This paper conducts multiple experiments to demonstrate the effectiveness of the proposed method.
weaknesses: 1.The contributions are slightly limited. Integrating the models to balance different stages of tasks has been applied in incremental learning, such as [1].
2.Many descriptions and notations are confusing. For example, in line 143 of page 3, a lack of explanation of “b” in “b|Y|”. In lines 179-185, the author seems to have used different words (features, representation, and embedding) to convey the same meaning, and I don't quite understand why the author did so. In Eq.(4), there lack of interpretation of , maxima value, and initial values in the summation notation. In line 257 of page 5, the description of “at most two backbones in memory” is confusing since I find there are at least three models (,, ) in memory according to lines 256. Finally, the authors are conflicted on which way to integrate, as shown in lines 256 and Algorithm 1.
[1] Zheng Z, Ma M, Wang K, et al. Preventing zero-shot transfer degradation in continual learning of vision-language models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 19125-19136.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. As shown in fig. 2, the initial performance of DUCT on the first domain is not optimal. Please further elaborate on this issue.

2. The parameter sensitivity analysis in fig. 3(c) indicates that DUCT still achieves decent performance when the head-merge ratio $\alpha_W$ is small. What if the ratio is set to zero? 

3. Can the proposed method be applied to class-incremental learning, given that it treats classes from the incoming domain as new categories?
soundness: 3
strengths: 1. The problem of exemplar-free domain-incremental learning is more challenging yet more practical. The authors did a good job maintaining the historical knowledge without replaying the past data.
2. The proposed method has strong performance, achieving a significant accuracy improvement compared to existing DIL methods.
3. The algorithm is simple, which could make a broader impact to the community.
summary: The paper addresses the problem of exemplar-free Domain-Incremental Learning (DIL) given a pre-trained embedding model. The proposed method, DUCT, jointly consolidate historical knowledge at both the representation level and the classifier level. At the representation level, the authors modify the technique of task vectors via considering the task similarity. At the classifier level, the authors propose to retrain a new classifier, and leverage the new classifier to modify the old classifier via optimal transport. To evaluate its effectiveness, DUCT is compared with DIL baselines on four cross-domain benchmarks. DUCT achieves state-of-the-art performance on all the experiments. An ablation study as well as other analytical experiments are reported to provide a more in-depth analysis of the proposed method.
weaknesses: 1. The major concern is that, as shown in the ablation study in table 2, the main reason for the accuracy boost in DUCT could be attributed to task vector, which is an existing technique for addressing multiple tasks simultaneously. In reviewer's opinion, despite that the authors make certain modification on the weighting strategy, the paper fails to provide new insights to this technique on why it is effective in addressing DIL problem. One possible aspect the reviewer can think of is to explain why applying DUCT 'places the same class of different domains together', as suggested in the visualization in fig. 5.

2. The notation in the paper needs improvement. First, in equation 5, $\phi^m_i$ should not use $i$ as subscript as it indicates the index of the summation. Second, $\beta$ and $\gamma$ should be explained once they appear in line 300.