Title: A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm
Venue: NeurIPS 2023
Accepted: yes
Keywords: Domain Incremental Learning

Continual Learning

Theory
Abstract: Domain incremental learning aims to adapt to a sequence of domains with access to only a small subset of data (i.e., memory) from previous domains. Various methods have been proposed for this problem, but it is still unclear how they are related and when practitioners should choose one method over another. In response, we propose a unified framework, dubbed Unified Domain Incremental Learning (UDIL), for domain incremental learning with memory. Our UDIL **unifies** various existing methods, and our theoretical analysis shows that UDIL always achieves a tighter generalization error bound compared to these methods. The key insight is that different existing methods correspond to our bound with different **fixed** coefficients; based on insights from this unification, our UDIL allows **adaptive** coefficients during training, thereby always achieving the tightest bound. Empirical results show that our UDIL outperforms the state-of-the-art domain incremental learning methods on both synthetic and real-world datasets. Code will be available at https://github.com/Wang-ML-Lab/unified-continual-learning.

A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm

Abstract

Domain incremental learning aims to adapt to a sequence of domains with access to only a small subset of data (i.e., memory) from previous domains. Various methods have been proposed for this problem, but it is still unclear how they are related and when practitioners should choose one method over another. In response, we propose a unified framework, dubbed Unified Domain Incremental Learning (UDIL), for domain incremental learning with memory. Our UDIL unifies various existing methods, and our theoretical analysis shows that UDIL always achieves a tighter generalization error bound compared to these methods. The key insight is that different existing methods correspond to our bound with different fixed coefficients; based on insights from this unification, our UDIL allows adaptive coefficients during training, thereby always achieving the tightest bound. Empirical results show that our UDIL outperforms the state-of-the-art domain incremental learning methods on both synthetic and real-world datasets. Code will be available at https://github.com/Wang-ML-Lab/unified-continual-learning.

1 Introduction

Despite recent success of large-scale machine learning models [35, 48, 36, 28, 92, 22, 33], continually learning from evolving environments remains a longstanding challenge. Unlike the conventional machine learning paradigms where learning is performed on a static dataset, domain incremental learning, i.e., continual learning with evolving domains , hopes to accommodate the model to the dynamically changing data distributions, while retaining the knowledge learned from previous domains [90, 60, 41, 97, 27]. Naive methods, such as continually finetuning the model on newcoming domains, will suffer a substantial performance drop on the previous domains; this is referred to as 'catastrophic forgetting' [46, 58, 81, 105, 52]. In general, domain incremental learning algorithms aim to minimize the total risk of all domains, i.e.,

where L t calculates model h θ 's expected prediction error ℓ over the current domain's data distribution D t . L 1: t -1 is the total error evaluated on the past t -1 domains' data distributions, i.e., {D i } t -1 i =1 .

The main challenge of domain incremental learning comes from the practical memory constraint that no (or only very limited) access to the past domains' data is allowed [52, 46, 105, 74]. Under such a constraint, it is difficult, if not impossible, to accurately estimate and optimize the past error L 1: t -1 . Therefore the main focus of recent domain incremental learning methods has been to develop effective surrogate learning objectives for L 1: t -1 . Among these methods [46, 81, 2, 105, 58, 10, 75,

77, 21, 25, 65, 66, 9, 72, 82, 95, 53], replay-based methods, which replay a small set of old exemplars during training [90, 75, 8, 4, 80, 11], has consistently shown promise and is therefore commonly used in practice.

One typical example is ER [75], which stores a set of exemplars M and uses a replay loss L replay as the surrogate of L 1: t -1 . In addition, a fixed, predetermined coefficient β is used to balance current domain learning and past sample replay. Specifically,

While such methods are popular in practice, there is still a gap between the surrogate loss ( β L replay ) and the true objective ( L 1: t -1 ), rendering them lacking in theoretical support and therefore calling into question their reliability. Besides, different methods use different schemes of setting β [75, 8, 4, 80], and it is unclear how they are related and when practitioners should choose one method over another.

To address these challenges, we develop a unified generalization error bound and theoretically show that different existing methods are actually minimizing the same error bound with different fixed coefficients (more details in Table 1 later). Based on such insights, we then develop an algorithm that allows adaptive coefficients during training, thereby always achieving the tightest bound and improving the performance. Our contributions are as follows:

We propose a unified framework, dubbed Unified Domain Incremental Learning (UDIL), for domain incremental learning with memory to unify various existing methods.

Our theoretical analysis shows that different existing methods are equivalent to minimizing the same error bound with different fixed coefficients. Based on insights from this unification, our UDIL allows adaptive coefficients during training, thereby always achieving the tightest bound and improving the performance.

Empirical results show that our UDIL outperforms the state-of-the-art domain incremental learning methods on both synthetic and real-world datasets.

2 Related Work

Continual Learning. Prior work on continual learning can be roughly categorized into three scenarios [90, 15]: (i) task-incremental learning, where task indices are available during both training and testing [52, 46, 90], (ii) class-incremental learning, where new classes are incrementally included for the classifier [74, 100, 30, 45, 44], and (iii) domain-incremental learning, where the data distribution's incremental shift is explicitly modeled [60, 41, 97, 27]. Regardless of scenarios, the main challenge of continual learning is to alleviate catastrophic forgetting with only limited access to the previous data; therefore methods in one scenario can often be easily adapted for another. Many methods have been proposed to tackle this challenge, including functional and parameter regularization [52, 46, 81, 2], constraining the optimization process [77, 21, 58, 10], developing incrementally updated components [104, 38, 53], designing modularized model architectures [73, 95], improving representation learning with additional inductive biases [9, 66, 65, 25], and Bayesian approaches [24, 63, 49, 1]. Among them, replaying a small set of old exemplars, i.e., memory, during training has shown great promise as it is easy to deploy, applicable in all three scenarios , and, most importantly, achieves impressive performance [90, 75, 8, 4, 80, 11]. Therefore in this paper, we focus on domain incremental learning with memory , aiming to provide a principled theoretical framework to unify these existing methods.

Domain Adaptation and Domain Incremental Learning. Loosely related to our work are domain adaptation (DA) methods, which adapt a model trained on labeled source domains to unlabeled target domains [68, 67, 57, 78, 79, 108, 71, 16, 17, 64, 94, 51]. Much prior work on DA focuses on matching the distribution of the source and target domains by directly matching the statistical attributions [67, 89, 87, 71, 64] or adversarial training [108, 57, 26, 109, 17, 102, 101, 54, 94]. Compared to DA's popularity, domain incremental learning (DIL) has received limited attention in the past. However, it is now gaining significant traction in the research community [90, 60, 41, 97, 27]. These studies predominantly focus on the practical applications of DIL, such as semantic segmentation [27], object detection for autonomous driving [60], and learning continually in an open-world setting [18]. Inspired by the theoretical foundation of adversarial DA [5, 57], we propose, to the best of our knowledge, the first unified upper bound for DIL . Most related to our work are previous DA methods that flexibly align different domains according to their associated given

or inferred domain index [94, 101], domain graph [102], and domain taxonomy [54]. The main difference between DA and DIL is that the former focuses on improving the accuracy of the target domains , while the latter focuses on the total error of all domains , with additional measures taken to alleviate forgetting on the previous domains. More importantly, DA methods typically require access to target domain data to match the distributions, and therefore are not directly applicable to DIL.

3 Theory: Unifying Domain Incremental Learning

In this section, we formalize the problem of domain incremental learning, provide the generalization bound of naively applying empirical risk minimization (ERM) on the memory bank, derive two error bounds (i.e., intra-domain and cross-domain error bounds) more suited for domain incremental learning, and then unify these three bounds to provide our final adaptive error bound. We then develop an algorithm inspired by this bound in Sec. 4. All proofs of lemmas, theorems, and corollaries can be found in Appendix A.

Problem Setting and Notation. We consider the problem of domain incremental learning with T domains arriving one by one. Each domain i contains N i data points S i = { ( x ( i ) j , y ( i ) j ) } N i j =1 , where ( x ( i ) j , y ( i ) j ) is sampled from domain i 's data distribution D i . Assume that when domain t ∈ [ T ] ≜ { 1 , 2 , . . . , T } arrives at time t , one has access to (1) the current domain t 's data S t , (2) a memory bank M = { M i } t -1 i =1 , where M i = { ( ˜ x ( i ) j , ˜ y ( i ) j ) } ˜ N i j =1 is a small subset ( ˜ N i ≪ N i ) randomly sampled from S i , and (3) the history model H t -1 after training on the previous t -1 domains. For convenience we use shorthand notation X i ≜ { x ( i ) j } N i j =1 and ˜ X i ≜ { ˜ x ( i ) j } ˜ N i j =1 . The goal is to learn the optimal model (hypothesis) h ∗ that minimizes the prediction error over all t domains after each domain t arrives. Formally,

̸

where for domain i , we assume the labels y ∈ Y = { 0 , 1 } are produced by an unknown deterministic function y = f i ( x ) and ϵ D i ( h ) denotes the expected error of domain i .

3.1 Naive Generalization Bound Based on ERM

Definition 3.1 ( Domain-Specific Empirical Risks ) . When domain t arrives, model h 's empirical risk ̂ ϵ D i ( h ) for each domain i (where i ≤ t ) is computed on the available data at time t , i.e.,

̸

̸

Note that at time t , only a small subset of data from previous domains ( i < t ) is available in the memory bank ( ˜ N i ≪ N i ). Therefore empirical risks for previous domains ( ̂ ϵ D i ( h ) with i < t ) can deviate a lot from the true risk ϵ D i ( h ) (defined in Eqn. 3); this is reflected in Lemma 3.1 below.

Lemma 3.1 ( ERM-Based Generalization Bound ) . Let H be a hypothesis space of VC dimension d . When domain t arrives, there are N t data points from domain t and ˜ N i data points from each previous domain i < t in the memory bank. With probability at least 1 -δ , we have:

Lemma 3.1 shows that naively using ERM to learn h is equivalent to minimizing a loose generalization bound in Eqn. 33. Since ˜ N i ≪ N i , there is a large constant ∑ t -1 i =1 1 ˜ N i compared to 1 N t , making the second term of Eqn. 33 much larger and leading to a looser bound.

3.2 Intra-Domain and Cross-Domain Model-Based Bounds

In domain incremental learning, one has access to the history model H t -1 besides the memory bank { M i } t -1 i =1 ; this offers an opportunity to derive tighter error bounds, potentially leading to better algorithms. In this section, we will derive two such bounds, an intra-domain error bound (Lemma 3.2) and a cross-domain error bound (Lemma 3.3), and then integrate them two with the ERM-based bound in Eqn. 33 to arrive at our final adaptive bound (Theorem 3.4).

Lemma 3.2 ( Intra-Domain Model-Based Bound ) . Let h ∈ H be an arbitrary function in the hypothesis space H , and H t -1 be the model trained after domain t -1 . The domain-specific error ϵ D i ( h ) on the previous domain i has an upper bound:

̸

where ϵ D i ( h, H t -1 ) ≜ E x ∼D i [ h ( x ) = H t -1 ( x )] .

Lemma 3.2 shows that the current model h 's error on domain i is bounded by the discrepancy between h and the history model H t -1 plus the error of H t -1 on domain i .

One potential issue with the bound Eqn. 34 is that only a limited number of data is available for each previous domain i in the memory bank, making empirical estimation of ϵ D i ( h, H t -1 ) + ϵ D i ( H t -1 ) challenging. Lemma 3.3 therefore provides an alternative bound.

Lemma 3.3 ( Cross-Domain Model-Based Bound ) . Let h ∈ H be an arbitrary function in the hypothesis space H , and H t -1 be the function trained after domain t -1 . The domain-specific error ϵ D i ( h ) evaluated on the previous domain i then has an upper bound:

̸

where d H ∆ H ( P , Q ) = 2sup h ∈H ∆ H | Pr x ∼P [ h ( x ) = 1] -Pr x ∼Q [ h ( x ) = 1] | denotes the H ∆ H -divergence between distribution P and Q , and ϵ D t ( h, H t -1 ) ≜ E x ∼D t [ h ( x ) = H t -1 ( x )] .

Lemma 3.3 shows that if the divergence between domain i and domain t , i.e., d H ∆ H ( D i , D t ) , is small enough, one can use H t -1 's predictions evaluated on the current domain D t as a surrogate loss to prevent catastrophic forgetting. Compared to the error bound Eqn. 34 which is hindered by limited data from previous domains, Eqn. 35 relies on the current domain t which contains abundant data and therefore enjoys much lower generalization error. Our lemma also justifies LwF-like cross-domain distillation loss ϵ D t ( h, H t -1 ) which are widely adopted [52, 23, 100].

3.3 A Unified and Adaptive Generalization Error Bound

Our Lemma 3.1, Lemma 3.2, and Lemma 3.3 provide three different ways to bound the true risk ∑ t i =1 ϵ D i ( h ) ; each has its own advantages and disadvantages. Lemma 3.1 overly relies on the limited number of data points from previous domains i < t in the memory bank to compute the empirical risk; Lemma 3.2 leverages the history model H t -1 for knowledge distillation, but is still hindered by the limited number of data points in the memory bank; Lemma 3.3 improves over Lemma 3.2 by leveraging the abundant data D t in the current domain t , but only works well if the divergence between domain i and domain t , i.e., d H ∆ H ( D i , D t ) , is small. Therefore, we propose to integrate these three bounds using coefficients { α i , β i , γ i } t -1 i =1 (with α i + β i + γ i = 1 ) in the theorem below.

Theorem 3.4 ( Unified Generalization Bound for All Domains ) . Let H be a hypothesis space of VC dimension d . Let N = N t + ∑ t -1 i ˜ N i denoting the total number of data points available to the training of current domain t , where N t and ˜ N i denote the numbers of data points collected at domain t and data points from the previous domain i in the memory bank, respectively. With probability at

least 1 -δ , we have:

̸

where ̂ ϵ D i ( h, H t -1 ) = 1 ˜ N i ∑ x ∈ ˜ X i ✶ h ( x ) = H t -1 ( x ) , ̂ ϵ D t ( h, H t -1 ) = 1 N t ∑ x ∈X i ✶ h ( x ) = H t -1 ( x ) , and Ω ≜ { α i , β i , γ i } t -1 i =1 .

̸

Theorem 3.4 offers the opportunity of adaptively adjusting the coefficients ( α i , β i , and γ i ) according to the data (current domain data S t and the memory bank M = { M i } t -1 i =1 ) and history model ( H t -1 ) at hand, thereby achieving the tightest bound. For example, when the H ∆ H divergence between domain i and domain t , i.e., d H ∆ H ( D i , D t ) , is small, minimizing this unified bound (Eqn. 8) leads to a large coefficient β i and therefore naturally puts on more focus on cross-domain bound in Eqn. 35 which leverages the current domain t 's data to estimate the true risk.

UDIL as a Unified Framework. Interestingly, Eqn. 8 unifies various domain incremental learning methods. Table 1 shows that different methods are equivalent to fixing the coefficients { α i , β i , γ i } t -1 i =1 to different values (see Appendix B for a detailed discussion). For example, assuming default configurations, LwF [52] corresponds to Eqn. 8 with fixed coefficients { α i = γ i = 0 , β i = 1 } ; ER [75] corresponds to Eqn. 8 with fixed coefficients { α i = β i = 0 , γ i = 1 } , and DER++ [8] corresponds to Eqn. 8 with fixed coefficients { α i = γ i = 0 . 5 , β i = 0 } , under certain achievable conditions. Inspired by this unification, our UDIL adaptively adjusts these coefficients to search for the tightest bound in the range [0 , 1] when each domain arrives during domain incremental learning, thereby improving performance. Corollary 3.4.1 below shows that such adaptive bound is always tighter, or at least as tight as, any bounds with fixed coefficients.

Corollary 3.4.1. For any bound g ( h, H t -1 , Ω fixed ) (defined in Eqn. 8) with fixed coefficients Ω fixed , e.g., Ω fixed = Ω ER = { α i = β i = 0 , γ i = 1 } t -1 i =1 for ER [75], we have

Corollary 3.4.1 shows that the unified bound Eqn. 8 with adaptive coefficients is always preferable to other bounds with fixed coefficients. We therefore use it to develop a better domain incremental learning algorithm in Sec. 4 below.

4 Method: Adaptively Minimizing the Tightest Bound in UDIL

̸

Although Theorem 3.4 provides a unified perspective for domain incremental learning, it does not immediately translate to a practical objective function to train a model. It is also unclear what coefficients Ω for Eqn. 8 would be the best choice. In fact, a static and fixed setting will not suffice, as different problems may involve different sequences of domains with dynamic changes; therefore ideally Ω should be dynamic (e.g., α i = α i +1 ) and adaptive (i.e., learnable from data). In this section, we start by mapping the unified bound in Eqn. 8 to concrete loss terms, discuss how the coefficients Ω are learned, and then provide a final objective function to learn the optimal model.

4.1 From Theory to Practice: Translating the Bound in Eqn. 8 to Differentiable Loss Terms

(1) ERM Terms. We use the cross-entropy classification loss in Definition 4.1 below to optimize domain t 's ERM term ̂ ϵ D t ( h ) and memory replay ERM terms { γ i ̂ ϵ D i ( h ) } t -1 i =1 in Eqn. 8.

Definition 4.1 ( Classification Loss ) . Let h : R n → S K -1 be a function that maps the input x ∈ R n to the space of K -class probability simplex, i.e., S K -1 ≜ { z ∈ R K : z i ≥ 0 , ∑ i z i = 1 } ; let X be a collection of samples drawn from an arbitrary data distribution and f : R n → [ K ] be the function that maps the input to the true label. The classification loss is defined as the average cross-entropy between the true label f ( x ) and the predicted probability h ( x ) , i.e.,

Following Definition 4.1, we replace ̂ ϵ D t ( h ) and ̂ ϵ D i ( h ) in Eqn. 8 with ̂ ℓ X t ( h ) and ̂ ℓ X i ( h ) .

(2) Intra- and Cross-Domain Terms. We use the distillation loss below to optimize intradomain ( { ̂ ϵ D i ( h, H t -1 ) } t -1 i =1 ) and cross-domain ( ̂ ϵ D t ( h, H t -1 ) ) model-based error terms in Eqn. 8.

Definition 4.2 ( Distillation Loss ) . Let h, H t -1 : R n → S K -1 both be functions that map the input x ∈ R n to the space of K -class probability simplex as defined in Definition 4.1; let X be a collection of samples drawn from an arbitrary data distribution. The distillation loss is defined as the average cross-entropy between the target probability H t -1 ( x ) and the predicted probability h ( x ) , i.e.,

Accordingly, we replace ̂ ϵ D i ( h, H t -1 ) with ̂ ℓ X i ( h, H t -1 ) and ̂ ϵ D t ( h, H t -1 ) with ̂ ℓ X t ( h, H t -1 ) .

(3) Constant Term. The error term ∑ t -1 i =1 ( α i + β i ) ϵ D i ( H t -1 ) in Eqn. 8 is a constant and contains no trainable parameters (since H t -1 is a fixed history model); therefore it does not need a loss term.

(4) Divergence Term. In Eqn. 8, ∑ t -1 i =1 β i d H ∆ H ( D i , D t ) measures the weighted average of the dissimilarity between domain i 's and domain t 's data distributions. Inspired by existing adversarial domain adaptation methods [57, 26, 109, 17, 102, 101, 94], we can further tighten this divergence term by considering the embedding distributions instead of data distributions using an learnable encoder. Specifically, given an encoder e : R n → R m and a family of domain discriminators (classifiers) H d , we have the empirical estimate of the divergence term as follows:

where U i (and U t ) is a set of samples drawn from domain D i (and D t ), d : R m → S t -1 is a domain classifier, and ∆ i ( x ) = [ d ( e ( x ))] i -[ d ( e ( x ))] t is the difference between the probability of x belonging to domain i and domain t . Replacing the indicator function with the differentiable cross-entropy loss, ∑ t -1 i =1 β i ̂ d H ∆ H ( e ( U i ) , e ( U t )) above then becomes

Algorithm 1 Unified Domain Incremental Learning (UDIL) for Domain t Training

Require: history model H t - 1 = P t - 1 ◦ E t - 1 , current model h θ = p ◦ e , discriminator model d ϕ ;

Require: dataset from the current domain S t , memory bank M = { M i } t - 1 i =1 ;

Require: training steps S , batch size B , learning rate η ;

Require: domain alignment strength coefficient λ d , hyperparameter for generalization effect C .

1: h θ ← H t - 1

▷ Initialization of the current model.

2:

Ω

≜

{

α

i

, β

i

, γ

i

} ← {

1

/

3

,

1

/

3

,

1

/

3

}

, for

∀

i

∈

[

t

-

1]

▷

Initialization of the replay coefficient

Ω

.

3: for s = 1 , · · · , S do

4: B t ∼ S t ; B i ∼ M i , ∀ i ∈ [ t -

1] ▷ Sample a mini-batch of data from all domains.

Ω) ▷ Discriminator training with Eqn. 16.

▷ Find a tighter bound with Eqn. 15.

7:

θ

θ

-

η

· ∇

θ

(

V

l

(

h

θ

,

◦

Ω)

-

λ

d

V

d

(

d, e,

◦

Ω))

▷ Model training with Eqn. 14 and Eqn. 16.

8: end for

9: H t ← h

10: M← BalancedSampling ( M , S t )

11: return H t

▷ For training on domain t +1 .

4.2 Putting Everything Together: UDIL Training Algorithm

Objective Function. With these differentiable loss terms above, we can derive an algorithm that learns the optimal model by minimizing the tightest bound in Eqn. 8. As mentioned above, to achieve a tighter d H ∆ H , we decompose the hypothesis as h = p ◦ e , where e : R n → R m and p : R m → S K -1 are the encoder and predictor, respectively. To find and to minimize the tightest bound in Theorem 3.4, we treat Ω = { α i , β i , γ i } t -1 i =1 as learnable parameters and seek to optimize the following objective (we denote as ◦ x = sg ( x ) the 'copy-weights-and-stop-gradients' operation):

Details of V l , V 0 -1 , and V d . V l is the loss for learning the model h , where the terms ̂ ℓ · ( · ) are differentiable cross-entropy losses as defined in Eqn. 10 and Eqn. 11:

V 0 -1 is the loss for finding the optimal coefficient set Ω . Its loss terms use Definition 3.1 and Eqn. 12 to estimate ERM terms and H ∆ H -divergence, respectively:

In Eqn. 15, ̂ ϵ · ( · ) uses discrete 0-1 loss , which is different from Eqn. 14, and a hyper-parameter C = √ 8 d log (2 eN/d ) + 8 log (2 /δ ) is introduced to model the combined influence of H 's VCdimension and δ .

V d follows Eqn. 12 to minimize the divergence between different domains' embedding distributions (i.e., aligning domains) by the minimax game between e and d with the value function:

(a)Example:3D-Balls

(b)Labels

(c)Domains

(d)Joint (91.083%,Oracle)

(e)ER (82.255%)

(f) DER++ (79.322%)

(g) CLS-ER (85.844%)

(h) UDIL (86.872%,Ours)

Here in Eqn. 16, if an optimal d ∗ and a fixed Ω is given, maximizing V d ( d ∗ , e, Ω) with respect to the encoder e is equivalent to minimizing the weighted sum of the divergence ∑ t -1 i =1 β i d H ∆ H ( e ( D i ) , e ( D t )) . This result indicates that the divergence between two domains' embedding distributions can be actually minimized. Intuitively this minimax game learns an encoder e that aligns the embedding distributions of different domains so that their domain IDs can not be predicted (distinguished) by a powerful discriminator given an embedding e ( x ) . Algorithm 1 below outlines how UDIL minimizes the tightest bound. Please refer to Appendix C for more implementation details, including a model diagram in Fig. 2.

5 Experiments

In this section, we compare UDIL with existing methods on both synthetic and real-world datasets.

5.1 Baselines and Implementation Details

We compare UDIL with the state-of-the-art continual learning methods that are either specifically designed for domain incremental learning or can be easily adapted to the domain incremental learning setting. For fair comparison, we do not consider methods that leverage largescale pre-training or prompt-tuning [99, 98, 53, 88]. Exemplar-free baselines include online Elastic Weight Consolidation ( oEWC ) [81], Synaptic Intelligence ( SI ) [105], and Learning without Forgetting ( LwF ) [52]. Memory-based domain incremental learning baselines include Gradient Episodic Memory ( GEM ) [58], Averaged Gradient Episodic Memory ( A-GEM ) [10], Experience Replay ( ER ) [75], Dark Experience Replay ( DER++ ) [8], and two recent methods, Complementary Learning System based Experience Replay ( CLS-ER ) [4] and Error Senesitivity Modulation based Experience Replay ( ESM-ER ) [80] (see Appendix C.5 for more detailed introduction to the baseline methods above). In addition, we implement the fine-tuning ( Fine-tune ) [52] and joint-training ( Joint ) as the performance lower bound and upper bound (Oracle).

We train all models using three different random seeds and report the mean and standard deviation. All methods are implemented with PyTorch [70], based on the mammoth code base [7, 8], and run on a single NVIDIA RTX A5000 GPU. For fair comparison, within the same dataset, all methods adopt the same neural network architecture, and the memory sampling strategy is set to random

balanced sampling (see Appendix C.2 and Appendix C.6 for more implementation details on training). We evaluate all methods with standard continual learning metrics including 'average accuracy', 'forgetting', and 'forward transfer' (see Appendix C.4 for detailed definitions).

5.2 Toy Dataset: High-Dimensional Balls

To gain insight into UDIL, we start with a toy dataset, high dimensional balls on a sphere (referred to as HD-Balls below), for domain incremental learning. HD-Balls includes 20 domains, each containing 2,000 data points sampled from a Gaussian distribution N ( µ , 0 . 2 2 I ) . The mean µ is randomly sampled from a 100-dimensional unit sphere, i.e., { µ ∈ R 100 : ∥ µ ∥ 2 = 1 } ; the covariance matrix Σ is fixed. In HD-Balls , each domain represents a binary classification task, where the decision boundary is the hyperplane that passes the center µ and is tangent to the unit sphere. Fig. 1(a-c) shows some visualization on HD-Balls .

Column 3 and 4 of Table 2 compare the performance of our UDIL with different baselines. We can see that UDIL achieves the highest final average accuracy and the lowest forgetting. Fig. 1(d-h) shows the embedding distributions (i.e., e ( x ) ) for different methods. We can see better embedding alignment across domains generally leads to better performance. Specifically, Joint, as the oracle, naturally aligns different domains' embedding distributions and achieves an accuracy upper bound of 91 . 083% . Similarly, our UDIL can adaptively adjust the coefficients of different loss terms, including Eqn. 12, successfully align different domains, and thereby outperform all baselines.

5.3 Permutation MNIST

We further evaluate our method on the Permutation MNIST ( P-MNIST ) dataset [50]. P-MNIST includes 20 sequential domains, with each domain constructed by applying a fixed random permutation to the pixels in the images. Column 5 and 6 of Table 2 show the results of different methods. Our UDIL achieves the second best ( 92 . 666% ) final average accuracy, which is only 0 . 284% lower than the best baseline DER++. We believe this is because (i) there is not much space for improvement as the gap between joint-training (oracle) and most methods are small; (ii) under the permutation, different domains' data distributions are too distinct from each other, lacking the meaningful relations among the domains, and therefore weakens the effect of embedding alignment in our method. Nevertheless, UDIL still achieves best performance in terms of forgetting ( 2 . 853% ). This is mainly because our unified UDIL framework (i) is directly derived from the total loss of all domains, and (ii) uses adaptive coefficients to achieve a more balanced trade-off between learning the current domain and avoiding forgetting previous domains.

5.4 Rotating MNIST

We also evaluate our method on the Rotating MNIST dataset ( R-MNIST ) containing 20 sequential domains. Different from P-MNIST where shift from domain t to domain t +1 is abrupt, R-MNIST 's domain shift is gradual. Specifically, domain t 's images are rotated by an angle randomly sampled from the range [9 ◦ · ( t -1) , 9 ◦ · t ) . Column 7 and 8 of Table 2 show that our UDIL achieves the highest average accuracy ( 86 . 635% ) and the lowest forgetting ( 8 . 506% ) simultaneously, significantly improving on the best baseline DER++ (average accuracy of 84 . 258% and forgetting of 13 . 692% ). Interestingly, such improvement is achieved when our UDIL's β i is high, which further verifies that UDIL indeed leverages the similarities shared across different domains so that the generalization error is reduced.

5.5 Sequential CORe50

CORe50 [55, 56] is a real-world continual object recognition dataset that contains 50 domestic objects collected from 11 domains (120,000 images in total). Prior work has used CORe50 for settings such as domain generalization (e.g., train a model on only 8 domains and test it on 3 domains), which is different from our domain-incremental learning setting. To focus the evaluation on alleviating catastrophic forgetting, we retain 20% of the data as the test set and continually train the model on these 11 domains; we therefore call this dataset variant Seq-CORe50 . Table 3 shows that our UDIL outperforms all baselines in every aspect on Seq-CORe50 . Besides the average accuracy over all domains, we also report average accuracy over different domain intervals (e.g., D 1:3 denotes average accuracy from domain 1 to domain 3) to show how different model's performance drops over time. The results show that our UDIL consistently achieves the highest average accuracy until the end. It is also worth noting that UDIL also achieves the best performance on another two metrics, i.e., forgetting and forward transfer.

6 Conclusion

In this paper, we propose a principled framework, UDIL, for domain incremental learning with memory to unify various existing methods. Our theoretical analysis shows that different existing methods are equivalent to minimizing the same error bound with different fixed coefficients. With this unification, our UDIL allows adaptive coefficients during training, thereby always achieving the tightest bound and improving the performance. Empirical results show that our UDIL outperforms the state-of-the-art domain incremental learning methods on both synthetic and real-world datasets. One limitation of this work is the implicit i.i.d. exemplar assumption, which may not hold if memory is selected using specific strategies. Addressing this limitation can lead to a more powerful unified framework and algorithms, which would be interesting future work.

References

H. Ahn, S. Cha, D. Lee, and T. Moon. Uncertainty-based continual learning with adaptive regularization. Advances in neural information processing systems , 32, 2019.

R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV) , pages 139-154, 2018.

M. Anthony, P. L. Bartlett, P. L. Bartlett, et al. Neural network learning: Theoretical foundations , volume 9. cambridge university press Cambridge, 1999.

E. Arani, F. Sarfraz, and B. Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. arXiv preprint arXiv:2201.12604 , 2022.

S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine learning , 79:151-175, 2010.

C. M. Bishop. Pattern recognition and machine learning . springer, 2006.

M. Boschini, L. Bonicelli, P. Buzzega, A. Porrello, and S. Calderara. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems , 33:15920-15930, 2020.

H. Cha, J. Lee, and J. Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference on computer vision , pages 9516-9525, 2021.

A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420 , 2018.

A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486 , 2019.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

T. Chen, H. Shi, S. Tang, Z. Chen, F. Wu, and Y. Zhuang. Cil: Contrastive instance learning framework for distantly supervised relation extraction. arXiv preprint arXiv:2106.10855 , 2021.

X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Z. Chen and B. Liu. Lifelong machine learning , volume 1. Springer, 2018.

Z. Chen, J. Zhuang, X. Liang, and L. Lin. Blending-target domain adaptation by adversarial metaadaptation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2248-2257, 2019.

S. Dai, K. Sohn, Y.-H. Tsai, L. Carin, and M. Chandraker. Adaptation across extreme variations using unlabeled domain bridges. arXiv preprint arXiv:1906.02238 , 2019.

Y. Dai, H. Lang, Y. Zheng, B. Yu, F. Huang, and Y. Li. Domain incremental lifelong learning in an open world. arXiv preprint arXiv:2305.06555 , 2023.

M. Davari, N. Asadi, S. Mudur, R. Aljundi, and E. Belilovsky. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16712-16721, 2022.

M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7):3366-3385, 2021.

D. Deng, G. Chen, J. Hao, Q. Wang, and P.-A. Heng. Flattening sharpness for dynamic gradient projection memory benefits continual learning. Advances in Neural Information Processing Systems , 34:18710-18721, 2021.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

P. Dhar, R. V. Singh, K.-C. Peng, Z. Wu, and R. Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5138-5146, 2019.

S. Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach. Uncertainty-guided continual learning with bayesian neural networks. arXiv preprint arXiv:1906.02425 , 2019.

J. Gallardo, T. L. Hayes, and C. Kanan. Self-supervised training enhances online continual learning. arXiv preprint arXiv:2103.14010 , 2021.

Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research , 17(1):2096-2030, 2016.

P. Garg, R. Saluja, V. N. Balasubramanian, C. Arora, A. Subramanian, and C. Jawahar. Multi-domain incremental learning for semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 761-771, 2022.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

F. Graf, C. Hofer, M. Niethammer, and R. Kwitt. Dissecting supervised contrastive learning. In International Conference on Machine Learning , pages 3821-3830. PMLR, 2021.

Y. Guo, B. Liu, and D. Zhao. Online continual learning through mutual information maximization. In International Conference on Machine Learning , pages 8109-8126. PMLR, 2022.

Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 , pages 87-102. Springer, 2016.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural computation , 18(7):1527-1554, 2006.

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

W. Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding , pages 409-426, 1994.

C.-Y. Hung, C.-H. Tu, C.-E. Wu, C.-H. Chen, Y.-M. Chan, and C.-S. Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems , 32, 2019.

D. Jung, D. Lee, S. Hong, H. Jang, H. Bae, and S. Yoon. New insights for the stability-plasticity dilemma in online continual learning. arXiv preprint arXiv:2302.08741 , 2023.

H. Jung, J. Ju, M. Jung, and J. Kim. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122 , 2016.

T. Kalb, M. Roschani, M. Ruf, and J. Beyerer. Continual learning for class-and domain-incremental semantic segmentation. In 2021 IEEE Intelligent Vehicles Symposium (IV) , pages 1345-1351. IEEE, 2021.

P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems , 33:18661-18673, 2020.

D. Kim and B. Han. On the stability-plasticity dilemma of class-incremental learning. arXiv preprint arXiv:2304.01663 , 2023.

G. Kim, C. Xiao, T. Konishi, Z. Ke, and B. Liu. A theoretical study on solving continual learning. Advances in Neural Information Processing Systems , 35:5065-5079, 2022.

G. Kim, C. Xiao, T. Konishi, and B. Liu. Learnability and algorithm for continual learning. arXiv preprint arXiv:2306.12646 , 2023.

J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

V. Kothapalli, E. Rasromani, and V. Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041 , 2022.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

R. Kurle, B. Cseke, A. Klushyn, P. Van Der Smagt, and S. Günnemann. Continual learning with bayesian neural networks for non-stationary data. In International Conference on Learning Representations , 2019.

Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

M. Li, H. Zhang, J. Li, Z. Zhao, W. Zhang, S. Zhang, S. Pu, Y. Zhuang, and F. Wu. Unsupervised domain adaptation for video object grounding with cascaded debiasing learning. In Proceedings of the 31th ACM International Conference on Multimedia , 2023.

Z. Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017.

Z. Li, L. Zhao, Z. Zhang, H. Zhang, D. Liu, T. Liu, and D. N. Metaxas. Steering prototype with prompt-tuning for rehearsal-free continual learning. arXiv preprint arXiv:2303.09447 , 2023.

T. Liu, Z. Xu, H. He, G. Hao, G.-H. Lee, and H. Wang. Taxonomy-structured domain adaptation. In ICML , 2023.

V. Lomonaco and D. Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In S. Levine, V. Vanhoucke, and K. Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learning , volume 78 of Proceedings of Machine Learning Research , pages 17-26. PMLR, 13-15 Nov 2017.

V. Lomonaco, D. Maltoni, and L. Pellegrini. Rehearsal-free continual learning over small non-iid batches. In CVPR Workshops , volume 1, page 3, 2020.

M. Long, Z. CAO, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems , 30, 2017.

U. Michieli and P. Zanuttigh. Knowledge distillation for incremental learning in semantic segmentation. Computer Vision and Image Understanding , 205:103167, 2021.

M. J. Mirza, M. Masana, H. Possegger, and H. Bischof. An efficient domain-incremental learning approach to drive in all weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3001-3011, 2022.

S. I. Mirzadeh, M. Farajtabar, R. Pascanu, and H. Ghasemzadeh. Understanding the role of training regimes in continual learning. Advances in Neural Information Processing Systems , 33:7308-7320, 2020.

M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning . MIT press, 2018.

C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner. Variational continual learning. arXiv preprint arXiv:1710.10628 , 2017.

L. T. Nguyen-Meidine, A. Belal, M. Kiran, J. Dolz, L.-A. Blais-Morin, and E. Granger. Unsupervised multi-target domain adaptation through knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1339-1347, 2021.

Z. Ni, H. Shi, S. Tang, L. Wei, Q. Tian, and Y. Zhuang. Revisiting catastrophic forgetting in class incremental learning. arXiv preprint arXiv:2107.12308 , 2021.

Z. Ni, L. Wei, S. Tang, Y. Zhuang, and Q. Tian. Continual vision-language representation learning with off-diagonal information, 2023.

S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. IEEE transactions on neural networks , 22(2):199-210, 2010.

S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering , 22(10):1345-1359, 2010.

V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1406-1415, 2019.

Q. Pham, C. Liu, and S. Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems , 34:16131-16144, 2021.

R. Ramesh and P. Chaudhari. Model zoo: A growing" brain" that learns continually. arXiv preprint arXiv:2106.03027 , 2021.

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 , 2018.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

G. Saha, I. Garg, and K. Roy. Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762 , 2021.

K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3723-3732, 2018.

S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8503-8512, 2018.

F. Sarfraz, E. Arani, and B. Zonooz. Error sensitivity modulation based experience replay: Mitigating abrupt representation drift in continual learning. arXiv preprint arXiv:2302.11344 , 2023.

J. Schwarz, W. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In International conference on machine learning , pages 4528-4537. PMLR, 2018.

J. Serra, D. Suris, M. Miron, and A. Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning , pages 4548-4557. PMLR, 2018.

H. Shi, D. Luo, S. Tang, J. Wang, and Y. Zhuang. Run away from your teacher: Understanding byol by a novel self-supervised approach. arXiv preprint arXiv:2011.10944 , 2020.

H. Shi, Y. Zhang, Z. Shen, S. Tang, Y. Li, Y. Guo, and Y. Zhuang. Towards communication-efficient and privacy-preserving federated representation learning. arXiv preprint arXiv:2109.14611 , 2021.

H. Shi, Y. Zhang, S. Tang, W. Zhu, Y. Li, Y. Guo, and Y. Zhuang. On the efficacy of small self-supervised contrastive models without distillation signals. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 2225-2234, 2022.

M. S. Sorower. A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis , 18(1):25, 2010.

B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer VisionECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14 , pages 443-450. Springer, 2016.

V. Thengane, S. Khan, M. Hayat, and F. Khan. Clip model is an efficient continual learner. arXiv preprint arXiv:2210.03114 , 2022.

E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 , 2014.

G. M. van de Ven, T. Tuytelaars, and A. S. Tolias. Three types of incremental learning. Nature Machine Intelligence , pages 1-13, 2022.

V. N. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Measures of complexity: festschrift for alexey chervonenkis , pages 11-30, 2015.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

J. S. Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS) , 11(1):37-57, 1985.

H. Wang, H. He, and D. Katabi. Continuously indexed domain adaptation. arXiv preprint arXiv:2007.01807 , 2020.

L. Wang, X. Zhang, Q. Li, J. Zhu, and Y. Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 254-271. Springer, 2022.

L. Wang, X. Zhang, H. Su, and J. Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487 , 2023.

Y. Wang, Z. Huang, and X. Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. arXiv preprint arXiv:2207.12819 , 2022.

Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision , pages 631-648. Springer, 2022.

Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 139-149, 2022.

Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 374-382, 2019.

Z. Xu, G.-Y. Hao, H. He, and H. Wang. Domain-indexing variational bayes: Interpretable domain index for domain adaptation. In International Conference on Learning Representations , 2023.

Z. Xu, G.-H. Lee, Y. Wang, H. Wang, et al. Graph-relational domain adaptation. arXiv preprint arXiv:2202.03628 , 2022.

C. Yaras, P. Wang, Z. Zhu, L. Balzano, and Q. Qu. Neural collapse with normalized features: A geometric analysis over the riemannian manifold. Advances in neural information processing systems , 35:11547-11560, 2022.

J. Yoon, E. Yang, J. Lee, and S. J. Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 , 2017.

F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning , pages 3987-3995. PMLR, 2017.

M.-L. Zhang and Z.-H. Zhou. Ml-knn: A lazy learning approach to multi-label learning. Pattern recognition , 40(7):2038-2048, 2007.

M.-L. Zhang and Z.-H. Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering , 26(8):1819-1837, 2013.

Y. Zhang, T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. In International conference on machine learning , pages 7404-7413. PMLR, 2019.

M. Zhao, S. Yue, D. Katabi, T. S. Jaakkola, and M. T. Bianchi. Learning sleep stages from radio signals: A conditional adversarial architecture. In International Conference on Machine Learning , pages 4100-4109. PMLR, 2017.

J. Zhou, C. You, X. Li, K. Liu, S. Liu, Q. Qu, and Z. Zhu. Are all losses created equal: A neural collapse perspective. Advances in Neural Information Processing Systems , 35:31697-31710, 2022.

Z. Zhu, T. Ding, J. Zhou, X. Li, C. You, J. Sulam, and Q. Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems , 34:29820-29834, 2021.

Appendix

In Sec. A, we present the proofs for the lemmas, theorems, and corollaries presented in the main body of our work. Sec. B discusses the correspondence of existing methods to specific cases within our framework. In Sec. C, we provide a detailed presentation of our final algorithm, UDIL, including an algorithmic description, a visual diagram, and implementation details. We introduce the experimental settings, including the evaluation metrics and specific training schemes. Finally, in Sec. D, we present additional empirical results with varying memory sizes and provide more visualization results.

A Proofs of Lemmas, Theorems, and Corollaries

Before proceeding to prove any lemmas or theorems, we first introduce three crucial additional lemmas that will be utilized in the subsequent sections. Among these, Lemma A.1 offers a generalization bound for any weighted summation of ERM losses across multiple domains. Furthermore, Lemma A.2 provides a generalization bound for a weighted summation of labeling functions within a given domain. Lastly, we highlight Lemma 3 in [5] as Lemma A.3, which will be used to establish the upper bound for Lemma 3.3.

Lemma A.1 ( Generalization Bound of α -weighted Domains ) . Let H be a hypothesis space of VC dimension d . Assume N j denotes the number of the samples collected from domain j , and N = ∑ j N j is the total number of the examples collected from all domains. Then for any α j > 0 and δ ∈ (0 , 1) , with probability at least 1 -δ :

Proof. Suppose each domain D j has a deterministic ground-truth labeling function f j : R n →{ 0 , 1 } . Denote as ̂ ϵ α ≜ ∑ j α j ̂ ϵ D j ( h ) the α -weighted empirical loss evaluated on different domains. Hence,

̸

where R j,k = ( α j N j N ) · ✶ h ( x k ) = f j ( x k ) is a random variable that takes the values in { α j N j N , 0 } . By the linearity of the expectation, we have ϵ α ( h ) = E [ ̂ ϵ α ( h )] . Following [3, 62], we have

̸

where in Eqn. 20, ̂ ϵ ′ α ( h ) is the α -weighted empirical loss evaluated on the 'ghost' set of examples {X ′ j } ; Eqn. 22 is yielded by applying Hoeffding's inequalities [37] and introducing the growth function Π H [3, 62, 91] at the same time; Eqn. 24 is achieved by using the fact Π H (2 N ) ≤ ( e · 2 N / d ) d ≤ (2 N ) d , where d is the VC-dimension of the hypothesis set H . Finally, by setting Eqn. 24 to δ and solve for the error tolerance ϵ will complete the proof.

Lemma A.2 ( Generalization Bound of β -weighted Labeling Functions ) . Let D be a single domain and X = { x i } N i be a collection of samples drawn from D ; H is a hypothesis space of VC dimension

d . Suppose { f j : R n →{ 0 , 1 }} j is a set of different labeling functions. Then for any β j > 0 and δ ∈ (0 , 1) , with probability at least 1 -δ :

Proof. Denote as ϵ β ( h ) ≜ ∑ j β j ϵ D ( h, f j ) the β -weighted error on domain D and { f j } j the set of the labeling functions , and ̂ ϵ β ≜ ∑ j β j ̂ ϵ D ( h, f j ) as the β -weighted empirical loss evaluated on different labeling functions. We have

̸

̸

where R i = ∑ j β j ✶ h ( x i ) = f j ( x i ) ∈ [0 , ∑ j β j ] is a new random variable. Then we have

̸

where in Eqn. 28, ̂ ϵ ′ β ( h ) is the β -weighted empirical loss evaluated on the 'ghost' set of examples X ′ ; Eqn. 30 is yielded by applying Hoeffding's inequalities [37] and introducing the growth function Π H [3, 62, 91] at the same time; Eqn. 31 is achieved by using the fact Π H (2 N ) ≤ ( e · 2 N / d ) d ≤ (2 N ) d , where d is the VC-dimension of the hypothesis set H . Finally, by setting Eqn. 31 to δ and solve for the error tolerance ϵ will complete the proof.

Lemma A.2 asserts that altering or merging multiple target functions does not impact the generalization error term, as long as the sum of the weights for each loss ∑ j β j remains constant and the same dataset X is used for estimation. Next we highligt the Lemma 3 in [5] again, as it will be utilized for proving 3.3.

Lemma A.3. For any hypothesis h, h ′ ∈ H and any two different domains D , D ′ ,

Proof. By definition, we have

̸

̸

Now we are ready to prove the main lemmas and theorems in the main body of our work.

Lemma 3.1 ( ERM-Based Generalization Bound ) . Let H be a hypothesis space of VC dimension d . When domain t arrives, there are N t data points from domain t and ˜ N i data points from each previous domain i < t in the memory bank. With probability at least 1 -δ , we have:

Proof. Simply using Lemma A.1 and setting α i = 1 for every i ∈ [ t ] completes the proof.

Lemma 3.2 ( Intra-Domain Model-Based Bound ) . Let h ∈ H be an arbitrary function in the hypothesis space H , and H t -1 be the model trained after domain t -1 . The domain-specific error ϵ D i ( h ) on the previous domain i has an upper bound:

̸

where ϵ D i ( h, H t -1 ) ≜ E x ∼D i [ h ( x ) = H t -1 ( x )] .

Proof. By applying the triangle inequality [5] of the 0-1 loss function, we have

Lemma 3.3 ( Cross-Domain Model-Based Bound ) . Let h ∈ H be an arbitrary function in the hypothesis space H , and H t -1 be the function trained after domain t -1 . The domain-specific error ϵ D i ( h ) evaluated on the previous domain i then has an upper bound:

̸

where d H ∆ H ( P , Q ) = 2sup h ∈H ∆ H | Pr x ∼P [ h ( x ) = 1] -Pr x ∼Q [ h ( x ) = 1] | denotes the H ∆ H -divergence between distribution P and Q , and ϵ D t ( h, H t -1 ) ≜ E x ∼D t [ h ( x ) = H t -1 ( x )] .

Proof. By the triangle inequality used above and Lemma A.3, we have

Theorem 3.4 ( Unified Generalization Bound for All Domains ) . Let H be a hypothesis space of VC dimension d . Let N = N t + ∑ t -1 i ˜ N i denoting the total number of data points available to the training of current domain t , where N t and ˜ N i denote the numbers of data points collected at domain t and data points from the previous domain i in the memory bank, respectively. With probability at least 1 -δ , we have:

̸

where ̂ ϵ D i ( h, H t -1 ) = 1 ˜ N i ∑ x ∈ ˜ X i ✶ h ( x ) = H t -1 ( x ) , ̂ ϵ D t ( h, H t -1 ) = 1 N t ∑ x ∈X i ✶ h ( x ) = H t -1 ( x ) , and Ω ≜ { α i , β i , γ i } t -1 i =1 .

̸

Proof. By applying Lemma 3.2 and Lemma 3.3 to each of the past domains, we have

Re-organizing the terms will give us

Then applying Lemma A.1 and Lemma A.2 jointly to Eqn. 37 will complete the proof.

B UDIL as a Unified Framework

In this section, we will delve into a comprehensive discussion of our UDIL framework, which serves as a unification of numerous existing methods. It is important to note that we incorporate methods designed for task incremental and class incremental scenarios that can be easily adapted to our domain incremental learning. To provide clarity, we will present the corresponding coefficients { α i , β i , γ i } of each method within our UDIL framework (refer to Table 4). Furthermore, we will explore the conditions under which these coefficients are included in this unification process.

Learning without Forgetting (LwF) [52] was initially proposed for task-incremental learning, incorporating a combination of shared parameters and task-specific parameters . This framework can be readily extended to domain incremental learning by setting all 'domain-specific' parameters to be the same in a static model architecture. LwF was designed for the strict continual learning setting, where no data from past tasks is accessible. To overcome this limitation, LwF records the predictions of the history model H t -1 on the current data X t at the beginning of the new task t . Subsequently, knowledge distillation (as defined in Definition 4.2) is performed to mitigate forgetting:

where H t -1 ( x ) , h ( x ) ∈ R K are the class distribution of x over K classes produced by the history model and current model, respectively. The loss for learning the current task L new is defined as

LwF uses a 'loss balance weight' λ o to balance two losses, which gives us its final loss for training:

In LwF, the default setting assumes the presence of two domains (tasks) with λ o = 1 . However, it is possible to learn multiple domains continuously using LwF's default configuration. To achieve this, the current domain t can be weighed against the number of previous domains ( 1 versus t -1 ). Specifically, if there is no preference for any particular domain, λ o should be set to t -1 . Remarkably, this is equivalent to setting { β i = 1 , α i = γ i = 0 } in our UDIL framework (Row 2 in Table 4).

Experience Replay (ER) [75] serves as the fundamental operation for replay-based continual learning methods. It involves storing and replaying a subset of examples from past domains during training. Following the description and implementation provided by [8], ER operates as follows: during each training iteration on domain t , a mini-batch B t of examples is sampled from the current domain, along with a mini-batch B ′ t from the memory. These two mini-batches are then concatenated into a larger mini-batch ( B t ∪ B ′ t ) , upon which average gradient descent is performed:

Suppose that each time the mini-batch of past-domain data is perfectly balanced, meaning that each domain has the same number of examples in B ′ t . In this case, Eqn. 43 can be further decomposed as follows:

where B ′ i = { ( x , y ) | ( x , y ) ∈ ( B ′ t ∩ M i ) } is the subset of the mini-batch that belongs to domain i .

Now, by dividing both sides of Eqn. 44 by ( | B t | + | B ′ t | / | B t | ) and comparing it to Theorem 3.4, we can include ER in our UDIL framework when the condition | B t | = | B ′ t | / ( t -1) is satisfied. In this case, ER is equivalent to { α i = β i = 0 , γ i = 1 } in UDIL (Row 3 in Table 4). It is important to note that this condition is not commonly met throughout the entire process of continual learning. It can be achieved by linearly scaling up the size of the mini-batch from the memory (which is feasible in the early domains) or by linearly scaling down the mini-batch from the current-domain data (which may cause a drop in model performance). It is worth mentioning that this incongruence highlights the intrinsic bias of the original ER setting towards current domain learning and cannot be rectified by adjusting the batch sizes of the current domain or the memory. However, it does not weaken our claim of unification.

Dark Experience Replay (DER++) [8] includes an additional dark experience replay, i.e., knowledge distillation on the past domain exemplars, compared to ER [75]. Now under the same assumptions (balanced sampling strategy and | B t | = | B ′ t | / ( t -1) ) as discussed for ER, we can utilize Eqn. 44 to transform the DER++ loss as follows:

In this scenario, DER++ is equivalent to { α i = γ i = 1 / 2 , β i = 0 } in UDIL (Row 4 in Table 4).

Incremental Classifier and Representation Learning (iCaRL) [74] was initially proposed for class incremental learning. It decouples learning the representations and final classifier into two individual phases. During training, iCaRL adopts an incrementally increasing linear classifier. Different from traditional design choice of multi-class classification where the softmax activation layer and multiclass cross entropy loss are used jointly, iCaRL models multi-class classification as multi-label classification [107, 86, 106]. Suppose there are K classes in total, and we denote the one-hot label vector of the input x as y ∈ R K where y j ≜ ✶ f ( x )= j . Then the multi-label learning objective treats each dimension of the output logits as a score for binary classification, which is computed as follows:

When a new task that contains K ′ new classes is presented, iCaRL adds additional K ′ new dimensions to the final linear classifier.

In iCaRL training, the new classes and the old classes are treated differently. For new classes, it trains the representations of the network with the ground-truth labels ( f ( x ) in the original paper), and for old classes, it uses the history model's output ( H t -1 ( x ) ) as the learning target (i.e., pseudo labels). Each data point is treated with the same level of importance during iCaRL's training procedure. Hence after translating the loss function of iCaRL in the context of class-incremental learning to domain-incremental learning, we have

where in ̂ ℓ ′ X i ( h, H t -1 ) , we follow the same definition as in the distillation loss in Eqn. 4.2 and replace the ground-truth label with the soft label for iCaRL.

Similar to ER [75] and DER++ [8] as analyzed before, the equation above does not naturally fall into the realm of unification provided in UDIL. To achieve this, we need to sample the data from the current domain and exemplars in the memory independently, i.e., assuming B t and B i . By applying the same rule of α i + β i + γ i = 1 , we now have that for iCaRL, the corresponding coefficients are { α i = B i / B t = 1 , β i = γ i = 0 } (Row 5 in Table 4).

Complementary Learning System based Experience Replay (CLS-ER) [4] involves the maintenance of two history models, namely the plastic model H ( p ) t -1 and the stable model H ( s ) t -1 , throughout the continual training process of the working model h . Following each update of the working model, the two history models are stochastically updated at different rates using exponential moving averages (EMA) of the working model's parameters :

where α ( p ) ≤ α ( s ) is set such that the plastic model undergoes rapid updates, allowing it to swiftly adapt to newly acquired knowledge, while the stable model maintains a 'long-term memory' spanning multiple tasks. Throughout training, CLS-ER assesses the certainty generated by both history models and employs the logits from the more certain model as the target for knowledge distillation.

In the general formulation of the UDIL framework, the history model H t -1 is not required to be a single model with the same architecture as the current model h . In fact, if there are no constraints on memory consumption, we have the flexibility to train and preserve a domain-specific model H i for each domain i . During testing, we can simply select the prediction with the highest certainty from each domain-specific model. From this perspective, the 'two-history-model system' employed in CLS-ER can be viewed as a specific and limited version of the all-domain history models. Consequently, we can combine the two models used in CLS-ER into a single history model H t -1 as follows:

where ( x , y ) ∈ M is an arbitrary exemplar stored in the memory bank.

At each iteration of training, CLS-ER samples a mini-batch B t from the current domain and a mini-batch B ′ t from the episodic memory. It then concatenates B t and B ′ t for the cross entropy loss minimization with the ground-truth labels, and uses B ′ t to minimize the MSE loss between the logits of h and H t -1 . To align the loss formulation of CLS-ER with that of ESM-ER [80], here we consider the scenarios where the losses evaluated on B t and B ′ t are individually calculated, i.e., we consider ̂ ℓ B t ( h ) + ̂ ℓ B ′ t ( h ) instead of ̂ ℓ B t ∪ B ′ t ( h ) . Based on the assumption from [34], the MSE loss on the logits is equivalent to the cross-entropy loss on the predictions under certain conditions. Therefore, following the same balanced sampling strategy assumptions as in ER, the original CLS-ER training objective can be transformed as follows:

Therefore, by imposing the constraint α i + β i + γ i = 1 , we find that λ = t -2 . Substituting this value back into λ / t -1 yields the equivalence that CLS-ER corresponds to { α i = λ / λ +1 , β i = 0 , γ i = 1 / λ +1 } in UDIL, where λ = t -2 (Row 6 in Table 4).

Error Sensitivity Modulation based Experience Replay (ESM-ER) [80] builds upon CLS-ER by incorporating an additional error sensitivity modulation module. The primary goal of ESM-ER is to mitigate sudden representation drift caused by excessively large loss values during current-domain learning. Let's consider ( x , y ) ∼ D t , which represents a sample from the current domain batch. In ESM-ER, the cross-entropy loss value of this sample is evaluated using the stable model H ( s ) t -1 and can be expressed as:

To screen out those samples with a high loss value, ESM-ER assigns each sample a weight λ by comparing the loss with their expectation value, for which ESM-ER uses a running estimate µ as its replacement. This can be formulated as follows:

where β is a hyperparameter that determines the margin for a sample to be classified as low-loss. For the sake of analysis, we make the following assumptions: (i) β = 1 ; (ii) the actual expected loss value E x ,y [ ℓ ( x , y )] is used instead of the running estimate µ ; (iii) a hard screening mechanism is employed instead of the current re-scaling approach. Based on these assumptions, we determine the sample-wise weights λ ⋆ according to the following rule:

Under the assumption that the loss value ℓ ( x , y ) follows an exponential distribution, denoted as ℓ ( x , y ) ∼ Exp( λ 0 ) , where the probability density function is given by f ( ℓ ( x , y ) , λ 0 ) = λ 0 e -λ 0 ℓ ( x ,y ) , we can calculate the expectation of the loss as E x ,y [ ℓ ( x , y )] = 1 / λ 0 . Based on this, we can now determine the expected ratio r of the unscreened samples in a mini-batch using the following equation:

The ratio r represents the proportion of effective samples in the current-domain batch, as the weights λ ⋆ ( x ) of the remaining samples are set to 0 due to their high loss value. Consequently, the original training loss of ESM-ER can be transformed as follows:

After applying the constraint of α i + β i + γ i = 1 , we obtain λ = r · ( t -1) -1 . Substituting this value back into λ / r ( t -1) , we find that ESM-ER is equivalent to { α i = λ / λ +1 , β i = 0 , γ i = 1 / λ +1 } in UDIL, where λ = r · ( t -1) -1 = (1 -e -1 )( t -1) -1 should be set (Row 7 in Table 4).

Bias Correction (BiC) [100] was the first to apply class incremental learning at a large scale, covering extensive image classification datasets including ImageNet (1,000 classes, [76]) and MSCeleb-1M (10,000 classes, [31]). Similar to iCaRL [74], BiC treats each data example (from new classes' data and the memory) with the same level of importance. Different from its previous work, the distillation loss in BiC ( L d in the original paper) implicitly includes the cross-domain distillation loss described by Lemma 3.3, as it computes the old classifier's output evaluated on the new data (considering only old classes). In addition, BiC further re-balances the classification loss ( L c in the original paper) and the distillation loss ( L d ) based on the number of old classes n and new classes m as follows:

Encoder

𝑒

Predictor

𝑝

Discriminator

𝑑

History Model

𝐻

!"#

𝜖

𝒟

$

(ℎ, 𝐻

"#$

)

,

𝜖

𝒟

%

(ℎ, 𝐻

"#$

)

𝜖

𝒟

$

(ℎ)

,

𝜖

𝒟

%

(ℎ)

𝑉

%

(𝑑, 𝑒)

ℳ={𝑀

&

}

&'#

!"#

𝒮

!

Ω = {𝛼

&

, 𝛽

&

, 𝛾

&

}

&'$

"#$

𝑡

Domain ID

By interpreting the distillation loss L d as the combination of the intra-domain loss (Lemma 3.2) and the cross-domain distillation loss (Lemma 3.3), and substituting ( n, m ) with ( t -1 , 1) due to the consistent number of classes in each domain, we arrive at the BiC model for DIL:

The equation above relies on the number of the current-domain examples and the exemplars (i.e., memory), which is not constant in practice. Hence we replace ( N t , ˜ N i ) with the mini-batch size ( | B t | , | B i | ) . After re-organizing the equation above, we have

The immediate observatio is that BiC exhibits a significant bias towards retaining knowledge from past domains . This is evident in the coefficient of coefficient of cross-domain distillation summed over the past domains ∑ t -1 i =1 β i = ( t -1) , which violates the constraints posed in this work ( α i + β i + γ i = 1 ). However, if we slightly relax BiC's formulation and focus on the relative ratios of the three coefficient, we get α i : β i : γ i = ( t -1) B i : ( t -1) B t : B i . Further applying the constraint α i + β i + γ i = 1 and assuming B i = B t to enforce a balanced sampling strategy over different domains, we arrive at { α i = β i = t -1 / 2 t -1 , γ i = 1 / 2 t -1 } (Row 8 in Table 4).

C Implementation Details of UDIL

This section delves into the implementation details of the UDIL algorithm. The algorithmic description of UDIL is presented in Algorithm 1 and a diagram is presented in Fig. 2. However, there are several practical issues to be further addressed here, including (i) how to exert the constraints of probability simplex ( [ α i , β i , γ i ] ∈ S 2 ) and (ii) how the memory is maintained. These two problems will be addressed in Sec. C.1 and Sec. C.2. Then, Sec. C.3 will introduce two auxiliary losses that improve the stability and domain alignment for the encoder during training. Next, Sec. C.4

will cover the evaluation metrics used in this paper. Finally, Sec. C.5 and Sec. C.6 will present a detailed introduction to the main baselines and the specific training schemes we follow for empirical evaluation.

C.1 Modeling the Replay Coefficients Ω = { α i , β i , γ i }

Instead of directly modeling Ω in a way such that it can be updated by gradient descent and satisfies the constraints that α i + β i + γ i = 1 and α i , β i , γ i ≥ 0 at the same time, we use a set of logit variables { ¯ α i , ¯ β i , ¯ γ i } ∈ R 3 and the softmax function to indirectly calculate Ω during training. Concretely, we have:

where Z i = exp(¯ α i ) + exp( ¯ β i ) + exp(¯ γ i ) is the normalizing constant. At the beginning of training on domain t , the logit variables { ¯ α i , ¯ β i , ¯ γ i } = { 0 , 0 , 0 } are initialized to all zeros, since we do not have any bias towards any upper bound. During training, they are updated in the same way as the other parameters with gradient descent.

C.2 Memory Maintenance with Balanced Sampling

Different from DER++ [8] and its following work [4, 80] that use reservoir sampling [93] to maintain the episodic memory, UDIL adopts a random balanced sampling after training on each domain. To be more concrete, given a memory bank with fixed size |M| , after domain t 's training is complete, we will assign each domain a quota of |M| / t . For the current domain t , we will randomly sample ⌊ |M| / t ⌋ exemplars from its dataset; for all the previous domains i ∈ [ t -1] , we will randomly swap out ⌈ |M| / t -1 -|M| / t ⌉ exemplars from the memory to make sure each domain has roughly the same number of exemplars. To ensure a fair comparison, we use the same random balanced sampling strategy for all the other baselines. The following Algorithm 2 shows the detailed procedure of random balanced sampling.

Algorithm 2 Balanced Sampling for UDIL

Require: memory bank M = { M i } t -1 i =1 , current domain dataset S t , domain ID t . 1: for i = 1 , · · · , t -1 do 2: for j = 1 , · · · , ⌈ |M| / t -1 -|M| / t ⌉ do 3: ( x , y ) ← RandomSample ( M i ) 4: ( x ′ , y ′ ) ← RandomSample ( S t ) 5: Swap ( x ′ , y ′ ) into M , replacing ( x , y ) 6: end for 7: end for 8: return M

C.3 Improving Stability and Domain Alignment for the Embedding Distribution

In this section, we will examine the training loss employed to align the embedding distribution across distinct domains. As discussed in Sec. 4.2, we decompose the model h into an encoder e and a predictor p as h = p ◦ e . In order to establish a tighter upper bound proposed in Theorem 3.4, we introduce a discriminator d that aims to distinguish embeddings based on domain IDs. Specifically, during the training process of UDIL, given a set of coefficients Ω = { α i , β i , γ i } , both the encoder e and the discriminator d engage in the following minimax game:

where ( ◦ · ) represents the "copying-weights-and-stopping-gradients" operation, and λ d is a hyperparameter introduced to control the strength of domain embedding alignment. More specifically, the

value function of the mini-max game V d is defined as follows:

As previously mentioned, the practical effect of Eqn. 62 is to align the embedding distribution of different domains, thus enhancing the model's generalization ability to both previously encountered domains and those that may be encountered in the future.

However, actively altering the embedding space can lead to the well-known stability-plasticity dilemma [43, 96, 20, 39, 61]. This dilemma arises when the model needs to modify a significant number of parameters to achieve domain alignment for a new domain, potentially resulting in a mismatch between the predictor p and the encoder e , which, in turn, can lead to catastrophic forgetting. Furthermore, it is worth noting that the adversarial training scheme described in Eqn. 62 is primarily designed for unsupervised domain alignment [108, 57, 26, 109, 17, 102, 101, 94], where the semantic labels in the target domain(s) are not available during training. This implies that the current adversarial training technique does not fully leverage the label information in domain incremental learning.

To tackle the aforementioned challenges, i.e., (i) maintaining a stable embedding distribution across all domains and (ii) accelerating domain alignment with label information, we incorporate two additional auxiliary losses: (i) the past embedding distillation loss [40, 65] and (ii) the supervised contrastive loss [42, 29]. It's important to note that these auxiliary losses, in conjunction with the adversarial feature alignment loss V d , operate exclusively on the encoder space of the model and do not impact the original objectives outlined in Theorem 3.4, provided that encoder e and predictor p remain sufficiently strong. Therefore, the two losses are used to simply stabilize the training, without compromising the theoretical significance of our work.

In past embedding distillation , also known as representation(al) distillation or embedding distillation in previous work on continual learning [40, 65, 59], the model stores the embeddings of the memory after being trained on domain t -1 . It then uses them to constrain the encoder's behavior: the features produced on these samples should not change too much during the current domain training. After decoupling the history model H t -1 = P t -1 ◦ E t -1 , where E t -1 is the history encoder and P t -1 is history predictor, we define the past embedding distillation loss V p as

The loss above promotes the stability of the embedding distribution from past domains. When combined with the adversarial embedding alignment loss in Eqn. 62, it encourages the embedding distribution of the current domain to match those of the previous domains, but not vice versa.

As supervised variations of contrastive learning [12, 32, 14, 83, 65, 13, 84, 85], the supervised contrastive loss [42, 29] will compensate for the fact that Eqn. 62 does not utilize the label information to align different domains, and therefore lack in the efficiency of aligning two domains' embedding distribution. The supervised contrastive learning pulls together the embeddings of the same label and pushes apart those with distinct labels. Notably, it is done in a 'domain-agnostic' manner, i.e., the domain labels are not considered. Denoting as P = 1 t ∑ t i D i the combined data distribution of all domains and as P ·| y the data distribution given the class label y ∈ [ K ] , the supervised contrastive loss for embedding alignment is defined as follows:

where s e ( u , v ) = ∥ e ( u ) -e ( v ) ∥ 2 2 is the squared Euclidean distance for any u , v ∈ X .

Introducing the supervised contrastive loss to continual learning is novel, as existing methods often harness its ability to create a compact representation space, thereby mitigating representation overlapping in class incremental learning [65, 30, 9, 96]. However, in this work, the primary motivation for using the supervised contrastive loss to facilitate domain alignment lies in its optimal solution [29, 110, 19]. This optimal point referred to as neural collapse [47, 69, 111, 103], where the embeddings of the same class collapse to the same point, and those of different classes are sparsely distributed. It is easy to envision that when the optimal state of the supervised constrastive loss is attained across different domains, it concurrently achieves perfect domain alignment .

The loss function that fosters stability and domain alignment in the encoder e can be summarized as follows:

Here, two hyper-parameters, λ p and λ s , balance the influence of each individual loss on the encoder's embedding distribution. In practice, the final performance of UDIL is not significantly affected by the values of λ p and λ s , and a wide range of values for these parameters can yield reasonable results.

C.4 Evaluation Metrics

In continual learning, many evaluation metrics are based on the Accuracy Matrix R ∈ R T × T , where T represents the total number of tasks (domains). In the accuracy matrix R , the entry R i,j corresponds to the accuracy of the model when evaluated on task j after training on task i . With this definition in mind, we primarily focus on the following specific metrics:

Average Accuracy (Avg. Acc.) up until domain t represents the average accuracy of the first t domains after training on these domains. We denote it as A t and define it as follows:

In most of the continual learning literature, the final average accuracy A T is usually reported. In our paper, this metric is reported in the column labeled ' overall '. The average accuracy of a model is a crucial metric as it directly corresponds to the primary optimization goal of minimizing the error on all domains, as defined in Eqn. 3.

Additionally, to better illustrate the learning (and forgetting) process of a model across multiple domains, we propose the use of the " Avg. of Avg. Acc. " metric A t 1 : t 2 , which represents the average of average accuracies for a consecutive range of domains starting from domain t 1 and ending at domain t 2 . Specifically, we define this metric as follows:

This metric provides a condensed representation of the trend in accuracy variation compared to directly displaying the entire series of average accuracies { A 1 , A 2 , · · · , A T } . We report this Avg. of Avg. Acc. metric in all tables (except in Table 2 due to the limit of space).

Average Forgetting (i.e., 'Forgetting' in the main paper) defines the average of the largest drop of accuracy for each domain up till domain t . We denote this metric as F t and define it as follows:

where f t ( j ) is the forgetting on domain j after the model completes the training on domain t , which is defined as:

Typically, the average forgetting is reported after training on the last domain T . Measuring forgetting is of great practical significance, especially when two models have similar average accuracies. It indicates how a model balances stability and plasticity . If a model P achieves a reasonable final average accuracy across different domains but exhibits high forgetting, we can conclude that this model has high plasticity and low stability. It quickly adapts to new domains but at the expense of performance on past domains. On the other hand, if another model S has a similar average accuracy to P but significantly lower average forgetting, we can infer that the model S has high stability and low plasticity. It sacrifices performance on recent domains to maintain a reasonable performance on past domains. Hence, to gain a comprehensive understanding of model performance, we focus on evaluating two key metrics: Avg. Acc. and Forgetting . These metrics provide insights into how models balance stability and plasticity and allow us to assess their overall performance across different domains.

Forward Transfer W t quantifies the extent to which learning from past t -1 domains contributes to the performance on the next domain t . It is defined as follows:

where r i is the accuracy of a randomly initialized model evaluated on domain i . For domain incremental learning, where the model does not have access to future domain data and does not explicitly optimize for higher Forward Transfer, the results of this metric are typically random. Therefore, we do not report this metric in the complete tables presented in this section.

C.5 Introduction to Baselines

We compare UDIL with the state-of-the-art continual learning methods that are either specifically designed for domain incremental learning or can be easily adapted to the domain incremental learning setting. Exemplar-free baselines include online Elastic Weight Consolidation ( oEWC ) [81], Synaptic Intelligence ( SI ) [105], and Learning without Forgetting ( LwF ) [52]. Memory-based domain incremental learning baselines include Gradient Episodic Memory ( GEM ) [58], Averaged Gradient Episodic Memory ( A-GEM ) [10], Experience Replay ( ER ) [75], Dark Experience Replay ( DER++ ) [8], and two recent methods, Complementary Learning System based Experience Replay ( CLS-ER ) [4] and Error Senesitivity Modulation based Experience Replay ( ESM-ER ) [80]. In addition, we implement the fine-tuning ( Fine-tune ) [52] and joint-training ( Joint ) as the performance lower bound and upper bound (Oracle). Here we provide a short description of the primary idea of the memory-based domain incremental learning baselines.

GEM [58]: The baseline method that uses the memory to provide additional optimization constraints during learning the current domain. Specifically, the update of the model cannot point towards the direction at which the loss of any exemplar increases.

A-GEM [10]: The improved baseline method where the constraints of GEM are averaged as one, which shortens the computational time significantly.

ER [75]: The fundamental memory-based domain incremental learning framework where the mini-batch of the memory is regularly replayed with the current domain data.

DER++ [8]: A simple yet effective replay-based method where an additional logits distillation (dubbed 'dark experience replay') is applied compared to the vanilla ER.

CLS-ER [4]: A complementary learning system inspired replay method, where two exponential moving average models are used to serve as the semantic memory, which provides the logits distillation target during training.

ESM-ER [80]: An improved version of CLS-ER, where the effect of large errors when learning the current domain is reduced, dubbed 'error sensitivity modulation'.

C.6 Training Schemes

Training Process. For each group of experiments, we run three rounds with different seeds and report the mean and standard deviation of the results. We follow the optimal configurations (epochs and learning rate) stated in [8, 80] for the baselines in P-MNIST and R-MNIST dataset. For HD-Balls and Seq-CORe50 , we first search for the optimal training configuration for the joint learning, and then grid-search the configuration in a small range near it for the baselines listed above. For our UDIL framework, as it involves adversarial training for the domain embedding alignment, we typically need a configuration that has larger number of epochs and smaller learning rate. We use a simple grid search to achieve the optimal configuration for it as well.

Model Architectures. For the baseline methods and UDIL in the same dataset, we adopt the same backbone neural architectures to ensure fair comparison. In HD-Balls , we adopt the same multi-layer perceptron with the same separation of encoder and decoder as in CIDA [94], where the hidden dimension is set to 800. In P-MNIST and R-MNIST , we adopt the same multi-layer perceptron architecture as in DER++ [8] with hidden dimension set to 800 as well. In Seq-CORe50 , we use the ResNet18 [33] as our backbone architecture for all the methods, where the layers before the final average pooling are treated as the encoder e , and the remaining part is treated as the predictor p .

(a)Joint (91.083%,Oracle)

(b) GEM (76.619%)

(C)A-GEM (59.654%)

(d)ER (82.255%)

(e)DER++(79.322%)

(f) CLS-ER (85.844%)

(g)ESM-ER (82.192%)

(h) UDIL (86.872%,Ours)

Hyperparameter Setting. For setting the hyper-parameter embedding alignment strength coefficient λ d and parameter C that models the combined effect of VC-dimension d and error tolerance δ , we use grid search for each dataset, where the range λ d ∈ [0 . 01 , 100] and C ∈ [0 , 1000] are used.

D Additional Empirical Results

This section presents additional empirical results of the UDIL algorithm. Sec. D.1 will show the additional results on different constraints with varying memory sizes. Sec. D.2 provides additional qualitative results: visualization of embedding distributions, to showcase the importance of the embedding alignment across domains.

D.1 Empirical Results on Varying Memory Sizes

Here we present additional empirical results to validate the effectiveness of our UDIL framework using varying memory sizes. The evaluation is conducted on three real-world datasets, as shown in Table 5, Table 6, and Table 7. By increasing the memory size from 400 to 800 in Table 5 and 6 and from 500 to 1000 in Table 7, we can investigate the impact of having access to a larger pool of past experiences on the continual learning performance, which might occur when the constraint on memory capacity is relaxed. This allows us to study the benefits of a more extensive memory in terms of knowledge retention and performance improvement. On the other hand, by further decreasing the memory size to the extreme of 200 in Table 5 and 6, we can explore the consequences of severely limited memory capacity. This scenario simulates situations where memory constraints are extremely tight, and the model can only retain a small fraction of past domain data, for example, a model deployed on edge devices. To ensure a fair comparison, here we use the same best configuration found in the main body of this work.

The results in all three tables demonstrate a clear advantage of our UDIL framework when the memory size is limited. In P-MNIST and R-MNIST , when the memory size |M| = 200 , the overall performance of UDIL reaches 91 . 483% and 82 . 796% respectively, which outperforms the second best model DER++ by 0 . 757% and a remarkable 6 . 125% . In Seq-CORe50 , when the memory size |M| = 500 is set, UDIL holds a 3 . 474% lead compared to the second best result. When the memory size is larger, the gap between UDIL and the baseline models is smaller. This is because when the memory constraint is relaxed, all the continual learning models should be at least closer to the performance upper bound, i.e., joint learning or 'Joint (Oracle)' in the tables, causing the

indistinguishable results among each other. Apparently, DER++ favors larger memory more than UDIL, while UDIL can still maintain a narrow lead in the large scale dataset Seq-CORe50 .

D.2 Visualization of Embedding Spaces

Here we provide more embedding space visualization results for the baselines with the utilization of memory, shown in Fig. 3. As one of the primary objectives of our algorithm, embedding space alignment across multiple domains naturally follows the pattern shown in the joint learning and therefore leads to a higher performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for domain incremental learning that unifies ideas from previous methods in this area. A theoretical analysis is first performed to formally state the problem of domain incremental learning. The theoretical formalization identifies three ways to bound the risk of error on a particular domain: 1) by a naive ERM-based bound, 2) by an intra-domain model-based bound, 3) by a cross-domain model-based bound. Based on the identified limitations of these three bound, the authors proposed a unified view of the generalization bound for domains observed in incremental learning. They propose to integrate these three bounds using coefficients that vary depending on the domains at hand. Existing methods in domain incremental learning are then categorized according to the components of the unified framework. In the second part of the paper, the authors propose a method where the coefficients are both learnable and dynamic. Experimental results on both toy data and benchmark datasets, along with comparison to SOTA methods in the area, are provided.\n\n---Rebuttal---\nI read the authors rebuttal. My review was already positive, so I left my score unchanged.\n\nSTRENGTHS:\n- Regarding originality, the paper presents originality both at the theoretical and experimental levels. Regarding theoretical originality, to the best of my knowledge the paper presents the first comprehensive analysis of generalization bounds in domain incremental learning while also categorizing SOTA methods in the area according to different views of these bounds. Regarding experimental originality, the paper presents a method that is dynamic and adaptable to relevant aspects such as the memory size and the domain relatedness in domain incremental learning. \n\n- Regarding significance, the paper provides new insights on domain incremental learning through rigorous and extensive theoretical analysis. This is of high relevance for researchers in this subfield. \n\n- Regarding quality and clarity, the paper is very well written and it is easy to follow. Extensive details are provided both in the paper and in the supplementary material.\n\nWEAKNESSES:\n- From the experimental results presented in Tables 2 and 3, the proposed method seems to be competitive but not significantly better than counterparts (especially DER++). \n\n- Details on the computational complexity/cost of the method are not provided in the formulation of the method or in the experiments. Considering that the method needs to learn the coefficients, and therefore the problem in Eq. 13 is more difficult than the optimization problem in counterpart methods, what would be the extra-cost in terms of computation?\n\nQUESTIONS:\nPlease refer to the \"weaknesses\" section for questions.\n\nLIMITATIONS:\nLimitations are properly addressed thorough the paper. There are not potential negative societal impacts of this work that need to be stated explicitly.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the author proposed a unified theoretical framework in order to reveal how the tasks in the domain incremental learning are related. Moreover, the framework unifies various existing methods from the perspective of domain generalization bound and it showed that the unified generalization bound can be organized in the format of the combination of data, memory bank, and history model. Meanwhile, the author showed that the fixed coefficient of the above combination is always less effective compared to the adaptive ones. As a result, the author proposed an adaptive adjusting coefficients method to tightly bound the generalization upper bound. Extensive experiments also showed the effectiveness of the unified framework.\n\nSTRENGTHS:\n1. This paper is well-organized and the mathematical proofs are sound. \\\\\n2. The author combined theoretical analysis of domain adaptation to deal with the domain incremental problem, which is impressive. \\\\\n3. The proposed unified framework can explain a variety of previous works, which is very powerful. \\\\\n\nWEAKNESSES:\nsee questions\n\nQUESTIONS:\n1. The author should emphasize what the ghost example in supplementary material is ( if it is some kind of subset of the previous domain? or some stored replay examples ), otherwise the transition from Equation 19 to 20 is not smooth, e.g. it provided a loose scale-up. \\\\\n2. From Theorem 3.4, we can find that the first bounding term is actually a \nsum up of loss from the first model to the current one (ERM), so is that appropriate to store the model at every training stage in the sample replay setting, and is the history model froze during the training? Meanwhile, will that cause a lot of additional memory burden? \\\\\n3. The intra-domain and cross-domain constraint is actually an every step alignment with the history learned models, thus as long as the domain discrepancy is relatively large from one to another, the $h$ would have a high probability of severe change, making the second term of Equation 8 in the left large, would that be a problem since it seems no explicit parameter regularization? \n4. Also, since in Equation 8, every step of updates would come from a bottom-up calculation from the first domain to the current one, it seems the calculation would be expensive as the growth of the domain, so I have concerns over the calculation burden. Also, as a consequence, would the method can be implemented in some large-scale real-world datasets which used in domain incremental learning, e.g. DomainNet?\n5. The Theorem 3.4 seems like a combination loss over all domains, thus, I have a little concern over the novelty of this Theorem. ( Considering it is the main Theorem).\n\nLIMITATIONS:\nyes, the authors adequately addressed the limitations yet", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission investigates domain incremental learning problem, where the model is adapted to a sequence of new tasks which have different distributions. Starting from some (domain related) generalization bound theory, the combination of three different bounds could be used to address domain incremental learning, and meanwhile several existing methods could be unified in the combined bound. An objective to find the efficient combination weight is also introduced. Experiments on both toy dataset and real dataset show the efficacy of the method.\n\nSTRENGTHS:\n- The motivation is clear, it is reasonable to combine the three different bounds to deal with mode general case.\n\n- It is good to derive a decent method from theoretic perspective for domain incremental learning task. And it is creditable that some hyperparameters such as $C$ could be also estimated inside the bound.\n\n- It is admirable that the resulting bound could include several existing methods, which connects the intuitive methods and the corresponding theory.\n\nWEAKNESSES:\nOverall the submission is a good paper, and I have no obvious concerns about the submission, while I simply hope maybe the authors could add more narrations in the main paper, on the related works of domain incremental learning, as there are relatively fewer existing works in this topic. And I am also interested in the effectiveness on the huge dataset, such as DomainNet which has relatively larger domain gap and more categories.\n\nQUESTIONS:\nAs the generalization bounds considered in the submission are actually related to domain adaptive task (from [3]), I am curious about whether the method could be extended to the more practical scenario, unsupervised domain incremental learning, where the new domain only contains unlabeled samples. I think there is possibility to extend the work, as some unsupervised domain adaptation methods are directly utilizing the $d_{\\mathcal{H}\\Delta\\mathcal{H}}$, such as MCD[a], though in the current setting there are only limited source samples.\n\n\n\n*reference*\n\n[a] Saito K, Watanabe K, Ushiku Y, et al. Maximum classifier discrepancy for unsupervised domain adaptation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3723-3732.\n\nLIMITATIONS:\nNone potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-15"}