A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm
Authors: Haizhou Shi
Hao Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Domain Incremental Learning
Continual Learning
Theory
TL;DR: We propose a principled domain incremental learning framework that unifies multiple existing methods and achieves the tightest generalization bound.

Abstract: Domain incremental learning aims to adapt to a sequence of domains with access to only a small subset of data (i.e., memory) from previous domains. Various methods have been proposed for this problem, but it is still unclear how they are related and when practitioners should choose one method over another. In response, we propose a unified framework, dubbed Unified Domain Incremental Learning (UDIL), for domain incremental learning with memory. Our UDIL **unifies** various existing methods, and our theoretical analysis shows that UDIL always achieves a tighter generalization error bound compared to these methods. The key insight is that different existing methods correspond to our bound with different **fixed** coefficients; based on insights from this unification, our UDIL allows **adaptive** coefficients during training, thereby always achieving the tightest bound. Empirical results show that our UDIL outperforms the state-of-the-art domain incremental learning methods on both synthetic and real-world datasets. Code will be available at https://github.com/Wang-ML-Lab/unified-continual-learning.

A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm
Haizhou Shi
Hao Wang
Department of Computer Science Rutgers University Piscataway, NJ 08854
Department of Computer Science Rutgers University Piscataway, NJ 08854
haizhou. shi @rutgurutteseeee edu
1e4.80cssrutgeee edu
Abstract
Domain incremental learning aims to adapt to a sequence of domains with access to only small subset of data (i.e., memory) from previous domains. Various methods have been proposed for this problem, but it is still unclear how they are related and when practitioners should choose one method over another. In response, we propose unified framework, dubbed Unified Domain Incremental Learning (UDIL), for domain incremental learning with memory. Our UDIL unifies various existing methods, and our theoretical analysis shows that UDIL always achieves tighter generalization eπrоr bound compared to these methods. The key insight is that different existing methods correspond to our bound with different fixed coefficients; based on insights from this unification, our UDIL allows adaptive coefficients during training, thereby always achieving the tightest bound. Empirical results show that our UDIL outperforms the state-of-the-art domain incremental learning methods on both synthetic and real-world datasets. Code will be available at https://github.. www.//mm/////eeeeeeee Wang--a--g--aaa
Introduction
Despite recent success of large-scale machine learning models [35, 48, 36, 28, 92, 22, 33], continually learning from evolving environments remains longstanding challenge. Unlike the conventional machine learning paradigms where learning is performed on a static dataset, domain incremental learning, Le., continual learning with evolving domains, hopes accommodate the model to the dynamically changing data distributions, while retaining the knowledge learned from previous domains [90, 60, 41, 97, 27]. Naive methods, such as continually finetuning the model on new- coming domains, will suffer a substantial performance drop on the previous domains; this is referred to as "catastrophic forgetting" [46, 58, 81, 105, 52]. In general, domain incremental learning algorithms aim to minimize the total risk of all domains, i.e.,
(0) = L.(0) (0) ((L.)),,, ho
Σ (M
(1)
where L, calculates model he" is expected prediction error & over the current domain's 's data distribution
D,. Lra-1 is the total error evaluated on the past t domains' data distributions, i.c., (D.)-=
The main challenge of domain incremental learning comes from the practical memory constraint that no (or only very limited) access to the past domains' data is allowed [52, 46, 105, 74]. Under such constraint, it is difficult, if not impossible, to accurately estimate and optimize the past error f14-1- Therefore the main focus of recent domain incremental learning methods has been to develop effective surrogate learning objectives for L1.-1. Among these methods [46, 81, 105, 58, 10, 75,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

27,21,2, 25, 65, 66, 72, 82, 53], during training [90, 75, 8, 4, 80, 11], has consistently shown promise and is therefore commonly used in practice.
rep.ayy replay-based methods, which replay a small set of old exemplars
One typical example is ER [75], which stores set of exemplars M and uses replay loss Creplay as the surrogate of Lia-1. 1- In addition, a fixed, predetermined coefficient 3 is used to balance current domain learning and past sample replay. Specifically,
[(0 (6) + B Ceppay (0) (0) + B E (/ ha()))
(2)
While such methods are popular in practice, there is still gap between the surrogate loss (3oyylyay) and the true objective (C1+-1),. rendering them lacking in theoretical support and therefore calling into question their reliability. Besides, different methods use different schemes of setting 3 [75, 8, 4, 80], and it is unclear how they are related and when practitioners should choose one method over another.
To address these challenges, we develop unified generalization error bound and theoretically show that different existing methods are actually minimizing the same error bound with different fixed coefficients (more details in Table 1 later). Based on such insights, we then develop an algorithm that allows adaptive coefficients during training, thereby always achieving the tightest bound and improving the performance. Our contributions are as follows:
We propose a unified framework, dubbed Unified Domain Incremental Learning (UDIL), for domain incremental learning with memory to unify various existing methods. Our theoretical analysis shows that different existing methods are equivalent minimizing the same error bound with different fixed coefficients. Based on insights from this unification, our UDIL allows adaptive coefficients during training, thereby always achieving the tightest bound and improving the performance.
Empirical results show that our UDIL outperforms the fate--o--th--e domain incremental learning methods on both synthetic and real-world datasets.
2 Related Work
Continual Learning. Prior work on continual learning can be roughly categorized into three scenarios 190, 15]: (i) task-incremental learning, where task indices are available during both training and testing [52, 46, 90], (ii) class-incremental learning, where new classes incrementally included for the classifier [74, 100, 30, 45, 44], and (iii) -nntennnn learning, where the data distribution's 's incremental shift is explicitly modeled [60, 41, 97, 27]. Regardless of scenarios, the main challenge of continual learning is to alleviate catastrophic forgetting with only limited access the previous data; therefore methods in one scenario can often be easily adapted for another. Many methods have been proposed to tackle this challenge, including functional and parameter regularization [52. 46, 81, 2]. constraining the optimization process [77, 21, 58, 10]. developing incrementally updated components [104, 38. 53], designing modularized model architectures [73, 95], improving representation learning with additional inductive biases [9, 66, 65, 25], and Bayesian approaches [24, 63, 49, IJ. Among them, replaying a small set of old exemplars, i.c., memory, during training has shown great promise as it is easy to deploy, applicable all three scenarios, and, most importantly, achieves impressive performance [90, 75, 8, 4, 80, [1]. Therefore in this paper, we focus on domain incremental learning with memory, aiming to provide a principled theoretical framework to unify these existing methods.
Domaín Adaptation and Domain Incremental Learning. Loosely related our work are domain adaptation (DA) methods, which adapt a model trained on labeled source domains to unlabeled target domains [68, 67, 57, 78, 79, 108, 71, 16. 17, 64, 94, 51]. Much prior work on DA focuses on matching the distribution of the source and target domains by directly matching the statistical attributions [67, 89, 87, 71, 64] or adversarial training [108, 57, 26, 109, 17, 102, 101, 54, 94]. Compared to DA'S popularity, domain incremental learning (DIL) has received limited attention in the past. However, is now gaining significant traction in the research community [90, 60, 41, 97, 27]. These studies predominantly focus on the practical applications of DIL, such as semantic segmentation [27], object detection for autonomous driving [60], and learning continually in an wpendworld setting [18]. Inspired by the theoretical foundation of adversarial DA [5, 57], we propose, to the best of our knowledge, the first unified upper bound for DII.. Most related to our work are previous DA methods that flexibly align different domains according to their associated given

---

or inferred domain index [94, 101], domain graph [102], and domain taxonomy [54]. The main difference between DA and DIL is that the former focuses on improving the accuracy of the targer domains, while the latter focuses on the total error of all domains, with additional measures taken to alleviate forgetting on the previous domains. More importantly, DA methods typically require access to target domain data to match the distributions, and therefore are not directly applicable to DIL.
3 Theory: Unifying Domain Incremental Learning
In this section, we formalize the problem of domain incremental learning, provide the generalization bound of naively applying empirical risk minimization (ERM) on the memory bank, derive two error bounds (i.c., intra-domain and cross-domain error bounds) more suited for domain incremental learning, and then unify these three bounds provide our final adaptive error bound. We then develop an algorithm inspired by this bound in Sec. 4. All proofs of lemmas, theorems, and corollaries can be found Appendix A.
Problem Setting and Notation. We consider the problem of domain incremental learning with (i)
T domains arriving one by one. Each domain i contains N; data points S, {(xx)) y )=
where (x) (1) ",3 is sampled from domain i's data distribution D; Assume that when domain 1 EE [T] A (1.2....,7]) arrives at time I, one has access to (1) the current domain I's data St, (2) (11)
memory bank .M = {M, where M, = { 33 د is small subset (Ñ; < N₂) randomly domains. For convenience we use shorthand notation X; ( and X, (x))) ب The goal is to learn the optimal model (hypothesis) h² that minimizes the prediction error over all domains after each domain arrives. Formally.
sampled from S,, and (3) the history model H₃-1 after training on the previous
1=
h" arg min
ED. ED (h),
ED, (h) A EEND, [h(x)) + f((x))
where for domain i, we assume the labels ) = } are produced by an unknown deterministic function y = (z) and eD, (h) denotes the expected error of domain i.
3.1 Naive Generalization Bound Based on ERM
Definition 3.1 (Domain-Specifee Empirical Risks). When domain t arrives, model h's empirical risk εD, (h) for each domain (where t) is computed on the available data at time i.e.
1 Enex, 1((0)(((())
ifi
TD (h)
(4)
N. Enex, 11(m)/5(()))
<i
Note that at time i, only small subset of data from previous domains (i 8) is available in the memory bank (N; N2). Therefore empirical risks for previous domains (ED, (h) with i () can deviate a lot from the true risk ép, (h) (defined in Eqn. 3); this is reflected in Lemma 3.1 below.
Lemma 3.1 lEse- Generalization Bound). Let H be hypothesis spuce of VC dimension d. When domain t arrives, there are Ne data points from domain t and N, data points from each previous domain < in the memory bank. With probability at least 1 - 5, we have:
(-1
ED. (h) < ED.
ED (h)
+
+
(8d ( (2EN) + log (2)).
(5)
N;
Lemma shows that naively using ERM to learn à equivalent to minimizing a loose generalization bound in Eqn. 33. Since N, & N₁, there is a large constant Ei=1 compared to N, making the second term of Eqn. 33 much larger and leading looser bound.

---

3.2 Intra-Domain and Cross-Domain Model-Based Bounds
In domain incremental learning, one has access to the history model H besides the memory bank { this offers an opportunity to derive tighter enror bounds, potentially leading to better algorithms. In this section, we will derive two such bounds, an intra-domain error bound (Lemma 3.2) and a cross-domain error bound (Lemma 3.3), and then integrate them two with the ERM-based bound Eqn. 33 to arrive at our final adaptive bound (Theorem 3.4).
Lemma 3.2 Intra-Domain Model-Based Bound). Let h E H be an arbitrary function in the hypothesis space H. and H be the model trained after domain t The domain-specife error éD, (h) the previous domain has upper bound:
(h) S ED. (D, H1-1) + eD. (H.(H)
(6)
where ED, (D, H1-1) & Ee-D, # Hz-1(z)).
Lemma 3.2 shows that the current model It's error on domain i is bounded by the discrepancy between h and the history model H plus the error of H,-1 on domain i.
One potential issue with the bound Eqn. 34 is that only a limited number of data is available for each previous domain in the memory bank, making empirical estimation of éD, (h, Hr-1) + ED. (H--1) challenging. Lemma 3.3 therefore provides an alternative bound.
Lemma 3.3 Dooss- Model-Based Bound). Let h H be arbitrary function in the hypothesis space H. and Hi-1 be the function trained after domain - 1. The domain specffc error CD, (h) evaluated on the previous domain i then has upper bound:
(b) S ED. (h, H--1) 2 dd D.) + cD. (He-1);
where (HAH(P,00 Q) = 2suPHEhAH [Prz~plhit (x(()) = 1] Pramol((I)) 1]1 denotes the HAH- divergence between distribution P and Q, and eD, (h, H,-1) & En D. (h(x)) # H(-1(x)).-
Lemma 3.3 shows that if the divergence between domain i and domain t, i.e., dHAH (D; D.). is small enough, one can use Hi-1's predictions evaluated on the current domain 27 as surrogate loss prevent catastrophic forgetting. Compared the error bound Eqn. 34 which is hindered by limited data from previous domains, Eqn. 35 relies on the current domain which contains abundant data and therefore enjoys much lower generalization error. Our lemma also justifies LwF-like cross domainn distillation loss éD, (h, H,-1) which are widely adopted [52, 23, 100].
3.3 A Unified and Adaptive Generalization Error Bound
Our Lemma 3.1, Lemma 3.2, and Lemma 3.3 provide three different ways to bound the true risk
Σi_ ED. (h); each has its own advantages and disadvantages. Lemma 3.1 overly relies on the limited number of data points from previous domains
in the memory bank to compute the empirical risk; Lemma 3.2 leverages the history model H4- for knowledge distillation, but is still hindered by the limited number of data points in the memory bank; Lemma 3.3 improves over Lemma 3.2 by leveraging the abundant data D; in the current domain f, but only works well if the divergence between domain and domain t, i.e., dHAu(() (D;, Dz), is small. Therefore, we propose to integrate these three bounds using coefficients {a, 8r, ) (with a + Bg + 7/ = 1) in the theorem below.
Theorem 3.4 (Unified Generalization Bound for All Domains). Let H be hypothesis space of t-1 N; denoting the total number of data points available the
VC dimension d. Let N = Nr + + training of current domain I, where N, and N; denote the numbers of data points collected at domain t and data points from the previous domain i in the memory bank, respectively. With probability at

---

Table 1: UDIL unified framework for domain incremental learning with memory. Three methods (LwF [52], ER [75], and DER++ [8] are by default compatible with DIL setting. For the remaining four CIL methods (iCaRL [74], CLS-ER [4], EMS-ER [80], and BiC [100]), adapt their original training objective to DIL settings before the analysis. For CLS-ER [4] and EMS-ER [80]. X and X' are the intensity coefficients of the logits distillation. For BiC [100], L is the current number of
the incremental domain. The conditions under which the unification of each method is achieved are
provided in detail in Appendix B.
UDII (Ours)
LWF [52]
ER [75]
DER+ [8]
iCaRL. [74]
CLS-ER [4]
EMS-ER [80]
BiC [100]
0, 8
0.
0
0.5
V/+4))
*/044)
1/(2-11
0.1
1
0
(-1) 1)
7
0.
0
0.5
0
1/44))
1/4x))
1- 1/2-1)
least] 1 - 6, we have:
ED, (1)
In (h) + a,,E, (h, He (H, (1) + Σ B, (h, Ho-11
1=1
+
BadHan (D,,
ass aast + Bi)eD, (H(-1)
+
(1+ 8.2
+
(mo+ou??
(8dlog ( a + 8 log (2))
N₁
A (h, H2),,
(8)
where (h, H1-1) = 1 N, Exsr, IMayM 1(2) (h, H(-1) = 1 N. Emex. La(m)y 1a2) and Ω {a, 71!=
Theorem 3.4 offers the opportunity of adaptively adjusting the coefficients (ax, Br, and x) according to the data (current domain data St and the memory bank M { and history model (H₃-1) at hand, thereby achieving the tightest bound. For example, when the HAH divergence between domain and domain t, i.e., dHAn (D,, D.), is small, minimizing this unified bound (Eqn. 8) leads to a large coefficient/ B3 and therefore naturally puts on more focus on cross- domain bound in Eqn. 35 which leverages the current domain I's data estimate the true risk.
UDIL as a Unified Framework. Interestingly, Eqn. 8 unifies various domain incremental learning methods. Table shows that different methods are equivalent to fixing the coefficients (a;, 3,, 74 }= to different values (see Appendix B for detailed discussion). For example, assuming default configurations, LWF [52] corresponds to Eqn. with fixed coefficients {a; = % 0, 3 = 1); ER [75] corresponds to Eqn. 8 with fixed coefficients {a, B, 0, 1}. and DER++ [8] corresponds to Eqn. 8 with fixed coefficients {ar = 75 0.5. 3 = 0). under certain achievable conditions. Inspired by this unification, our UDIL adaptively adjusts these coefficients to search for the tightest bound the range [0, 1] when each domain arrives during domain incremental learning, thereby improving performance. Corollary 3.4. below shows that such adaptive bound is always tighter, or at least as tight as, any bounds with fixed coefficients.
Corollary 3.4.1. For any bound g(, H Drsee) (defined in Eqn. 8) with fixed coefficients Paxeds e.g., Pgised DER {az = B 0, 74 1 for ER [75], we have
Σ 1=1
CD, (h) < min (h, H2) < g(h, H Ppsed): Yh, H E H. f2
Corollary 3.4.1 shows that the unified bound Eqn. 8 with adaptive coefficients is always preferable to other bounds with fixed coefficients. We therefore use it to develop a better domain incremental learning algorithm in Sec. 4 below.
5

---

4 Method: Adaptively Minimizing the Tightest Bound in UDIL
Although Theorem 3.4 provides a unified perspective for domain incremental learning, it does not immediately translate to a practical objective function to train a model. It is also unclear what coefficients $2 for Eqn. would be the best choice. In fact, static and fixed setting will not suffice, as different problems may involve different sequences of domains with dynamic changes; therefore ideally Ω should be dynamic (e-g.. 04 & ax+1) and adaptive (i.e., learnable from data). In this section, we start by mapping the unified bound in Eqn. 8 to concrete loss terms, discuss how the coefficients are learned, and then provide final objective function to learn the optimal model.
4.1 From Theory to Practice: Translating the Bound in Eqn. 8 to Differentiable Loss Terms
(1) ERM Terms. We use the cross-entropy classification loss in Definition 4.1 below to optimize domain t's ERM term εD, (h) and memory replay ERM terms &yED, (h)) in Equ. 8.
Definition 4.1 (Classification Loss). Let h IR + SK 1 be a function that maps the input æ E IR" to the space of K -class probability simplex, i.e., SK-1 & {z E ERK: 2, 2 0. E. 23 =); let X' be a collection of samples drawn from an arbitrary data distribution and f R - [K] be the function that maps the input to the true label. The classification loss is defined as the average cross-entropy between the true label (x) and the predicted probability h(()), i.e.
(x) - PI E
1 )())== log (M())),) -
K
(10)
Following Definition 4.1, we replace ēD, (h) and ēD, (h) in Eqn. 8 with Ex (h) and εx; (h).
(2) Intra- and Cross-Domain Terms. We use the distillation loss below to optimize intra- domain ((ED, (h, H)=)) and cross-domain (ED, (h, H₁-1)) model-based error terms in Eqn. 8. Definition 4.2 (Distillation Loss). Let h. H R² - g both be functions that map the input I2" to the space of K-class probability simplex as defined in Definition 4.1; let X be collection of samples drawn from an arbitrary data distribution. The distillation loss is defined as the average cross-entropy between the target probability H₂-1(2) (2)) and the predicted probability h(x)), ie..
Ex,h, H A [2
E.Ex
Σ
K
[H, [H)-1(z)]] logg (I)())),)
(11)
j=n
Accordingly, we replace TD (h, H1-1) with Ex, (h, He-1) and ēD (h, He-1) with Exx (h, Hz-1).
(3) Constant Term. The error term E (a; + 3,),,, (H₂-1) in Equ. is a constant and contains no trainable parameters (since H₁-1 is fixed history model); therefore it does not need a loss term.
(4) Divergence Term. In Eqn. 8. > BidHAH (D,, D.) measures the weighted average of the dissimilarity between domain i's and domain f's data distributions. Inspired by existing adversarial domain adaptation methods [57, 26, 109, 17, 102, 101, 94], can further tighten this divergence term by considering the embedding distributions instead of data distributions using an learnable encoder. Specifically, given encoder R" Rm and family of domain discriminators (classifiers) Ha. we have the empirical estimate of the divergence term as follows:
1-1
t-1
E ((dxss(e((),e((4)) = Σ 3 i=1
min dEH
B
Σ lA((2)2 IE
Σ AA ZE24.
T4
Dee
where U, (and Ui) is set of samples drawn from domain D, (and D:). d Rm + gt- is domain classifier, and (x(x) = [d(e(x))] [d(e(=))] is the difference between the probability of æ belonging to domain and domain t. Replacing the indicator function with the differentiable cross-entropy loss, E (e())) ((4)) above then becomes
t-1 1
2
B 2 min
8
Σ
|- log ([d(e(=)))1]+ + 1 E ([(e((((((( N. #ES.
(12)
dEH.
N:
i=1
TEA:

---

Algorithm Unified Domain Incremental Learning (UDIL) for Domain / Training
Require: history model H Pt-1 o E current model hg e, discriminator model dø:
Require: dataset from the current domain St, memory bank M = {M Require: training steps S, batch size B, learning rate 7/; domain alignment strength coefficient Ad, hyperparameter for generalization effect C.
Require:
1: hg H4,1
> Initialization of the current model. Initialization of the replay coefficient f2
2: Ω _ {a,, Bi m + (//s, 1/s, 1/s), for Vi
3: for s == -,... do
4:
B Sw Bi M, Vi [
Sample mini-batch of data from all domains.
5:
Φ - "1 Aa VoVa(d,e,
Discriminator training with Eqn. 16.
6:
Q- - (2)
D Find a tighter bound with Eqn. 15.
7:
0 0+0-1 Va(Vi(ho, 12) XaVa(d,e,
Model training with Eqn. 14 and Eqn. 16.
8: end for
9: H₂
10: M +
BalancedSampling(MM,
return H₁
For training on domain t +
4.2 Putting Everything Together: UDIL Training Algorithm
With these differentiable loss terms above, learns the optimal model by minimizing the tightest bound in Eqn. 8. As mentioned above, to achieve tighter dzAH, decompose the hypothesis as h p where R" + R" and p: IRm + SK-1 are the encoder and predictor, respectively. To find and to minimize the tightest bound in Theorem 3.4, we treat Ω = {as, Bi, 7 as learnable parameters and seek to optimize the following objective (we denote ã sg(z) the 'copy- srrrgghtt-and--t---gaadiee -to--ggaeeenttsss operation):
Objective Function.
derive algorithm that
min max [O.h-poe)
Vi .) + (h, - XaVa(d, e,
8
(13)
S.t.
a + + 74
Ei {1, 2, ... 1- ViE 2, .- f1,
04, 11 0,
Details of VI, Vo., and Va. Vi is the loss for learning the model h, where the terms (.() are differentiable cross-entropy losses as defined in Eqn. 10 and Eqn. 11:
Vi(h, = - [x (h) a. (h, H-) + Es. (h)
E B.) (h, H(-1). (14)
Vo-1 the loss for finding the optimal coefficient set Q. Its loss terms use Definition 3.1 and Eqn. 12 to estimate ERM terms and H H. diverreeee,, respectively:
Vo- (o, ②2 Σ 1-1
(h) (h) av (h,HH-H) Ho-1) + 2 3.YED, (h. H,-1)
+ 1/2 Σ
:=1 BidHAN (2(2)),((S.) + Σ
(as + (1)=,((1,_1
(
(1+E No
B.?
(xstae)) i= N.
+C
(15)
In Eqn. 15, 2.(.) uses discrete 0-1 loss, which is different from Eqn. 14, and a hyper-parameter
C. /8d log (2eN/d) + 8 log (2/8) is introduced to model the combined influence of H's VC- dimension and 8.
V, follows Eqn. 12 to minimize the divergence between different domains' embedding distribu- tions (i.e., aligning domains) by the minimax game between e and with the value function:
e, e 2)
[- log (Id(e(z))),)
+
| log ([d(e()))\)), (16)
SE
7

---

(a) Example: 3D-Balls
(b) Labels
(c) Domains
(4) Joint (91.0833) 01.08 Oracle)
(c) ER (82.255%)
(9 DER+ (79.322%)
(g) CLS-ER (85.844%)
(b) UDIL (86.872% Ours)
Figure I: Results on HDD-Balls. In (a-b), data is colored according to labels; in (c-h), data is colored according to domain ID. All data is plotted after PCA [6]. (a) Simplified HD-Balls dataset with domains in the 3D space (for visualization purposes only). (b-c) Embeddings of HD-Balls's raw data colored by labels and domain ID. (d-h) Accuracy and embeddings learned by Joint (oracle), UDIL, and three best baselines (more in Appendix C.5). Joint, as the oracle, naturally aligns different domains, and UDIL outperforms all baselines in terms of embedding alignment and accuracy.
Here in Eqn. 16, if an optimal d' and fixed Ω given, maximizing Va(d", e2) with respect the en- coder is equivalent to minimizing the weighted sum of the divergence Σ (,dhsx(e(PP.). eD,))) This result indicates that the divergence between two domains' embedding distributions can be ac- tually minimized. Intuitively this minimax game learns an encoder e that aligns the embedding distributions of different domains so that their domain IDs can not be predicted (distinguished) by powerful discriminator given an embedding e(x:). Algorithm below outlines how UDIL minimizes the tightest bound. Please refer to Appendix C for more implementation details, including model diagram in Fig. 2.
5 Experiments
In this section, we compare UDIL with existing methods on both synthetic and real-world datasets.
5.1 Baselines and Implementation Details
We compare UDIL with the state-of-the-are continual learning methods that are either specifi- cally designed for domain incremental learning or can be easily adapted to the domain incre- mental learning setting. For fair comparison, we do not consider methods that leverage large- scale pre-training prompt-tuning [99, 98, 53, 88). Exemplar-free baselines include online Elas- tic Weight Consolidation (oEWC) [81], Synaptic Intelligence (SI) [105], and Learning without Forgetting (LwF) [52]. Memory-based domain incremental learning baselines include Gradient
Episodic Memory (GEM) [58], Averaged Gradient Episodic Memory (A-GEM) [10], Experience Replay (ER) [75], Dark Experience Replay (DER++) [8], and two recent methods, Complementary Learning System based Experience Replay (CLS-ER) [4] and Error Senesitivity Modulation based Experience Replay (ESM-ER) [80] (see Appendix C.5 for more detailed introduction to the baseline methods above). addition, we implement the fine-tuning (Fine-tune) [52] and joint-training (Joint) as the performance lower bound and upper bound (Oracle).
We train all models using three different random seeds and report the mean and standard deviation. All methods are implemented with Py PoTcc [70], based on the mammoth code base [7, 8]. and run on single NVIDIA RTX A5000 GPU. For fair comparison, within the same dataset, all methods adopt the same neural network architecture, and the memory sampling strategy is set to random

---

Table 2: Performances (%) on HD-Balls, P-MNIST, and R-MNIST. We two metrics, Average Accuracy and Forgetting, to evaluate the methods' effectiveness. "f" and "4" mean higher and lower numbers are better, respectively. We use boldface and underlining denote the best and the second-best performance, respectively. We use "." to denote "not appliable".
HD-Balls
P-MNIST
R-MNIST
Method
Buffer
Avg. Acc
Forgetting
Avg. Acc e
Forgetting
Avg. Acc
Forgetting (U)
Fine-tune
43.520+0079 =2.319
70. 20.
27.522=3042
47.803
52.281
oEWC [8]]
54.131 193
39.743
1..47
11.06
48.203==0.807
51.181
SI [105]
=2.303=a0087
43.175+0004
79. 1.335
17.409
48.25 11.381
11.053+1.507
LwF [52]
51.523
25.155
73.54512.666
24.556
54.709.0.515
45.473
GEM [58]
09.747=a0.55
13.591440779
10.149
6,975
76.619
21.289
A-GEM [10]
02.777:1000
12.878
0..560
8.577
59.65 39.654.10.00
39. 19.1191..111
ER [75]
82.255 1.552
1.524
18.039
1.100
76. 10.66
20. 6..66.6477744
DER+ [8]
400
79.432+1.347
13.762+11514
92.950=0.361
3.378+40245
84.258+
13.692+0.50
CLS-ER [4]
85.844
4.297
91.591
3.795440.44
81.77 1:0.34
45.455
ESM-ER [80]
31.995=3333
13,245
89.829
39.88992..60 20.00
6.888.10.738
82..192:10.0
16. 195
UDIL (Ours)
86.872=00195
3.428
92.666+0.10
2.853++11107
86.635
8.506=1181
Joint (Oracle)
91.083=0.13
96.368+n.42
97.150=0.:06
balanced sampling (see Appendix C.2 and Appendix C.6 for more implementation details on training). We evaluate all methods with standard continual learning metrics including 'average accuracy', 'forgetting', and 'fowwrrd transfer' (see Appendix C.4 for detailed definitions).
5.2 Toy Dataset: High-Dimensional Balls
To gain insight into UDIL, we start with a toy dataset, high dimensional balls on sphere (referred to as HD-Balls below), for domain incremental learning. HD-Balls includes 20 domains, each containing 2,000 data points sampled from a Gaussian distribution N(j,, 0.2² The mean l is randomly sampled from a 100-dimensional unit sphere, i.c., {u E 12000 ||ea||2 = 13: the covariance matrix L' is fixed. In HD-Balls, each domain represents a binary classification task, where the decision boundary is the hyperplane that passes the center μs and is tangent to the unit sphere. Fig. ((a-c) shows some visualization HD-Balls.
Column 3 and of Table 2 compare the performance of our UDIL with different baselines. We can see that UDIL achieves the highest final average accuracy and the lowest forgetting. Fig. ((d-h) shows the embedding distributions (i.e., e(x)) for different methods. We can see better embedding alignment across domains generally leads to better performance. Specifically, Joint, as the oracle, naturally aligns different domains' embedding distributions and achieves an accuracy upper bound of 91. 00.88% Similarly, our UDIL can adaptively adjust the coefficients of different loss terms, including Eqn. 12, successfully align different domains, and thereby outperform all baselines.
5.3 Permutation MNIST
We further evaluate our method on the Permutation MNIST (P-MNIST) dataset [50]. P-MNIST in- cludes 20 sequential domains, with each domain constructed by applying a fixed random permutation to the pixels in the images. Column 5 and 6 of Table 2 show the results of different methods. Our UDIL achieves the second best (92.666%) final average accuracy, which is only 0.284% lower than the best baseline DER++. We believe this is because (i) there is not much space for improvement as the gap between joint-training (oracle) and most methods are small; (ii) under the permutation, different domains' data distributions are too distinct from each other, lacking the meaningful rela- tions among the domains, and therefore weakens the effect of embedding alignment in our method. Nevertheless, UDIL still achieves best performance in terms forgetting (2.853%). This mainly because our unified UDIL. framework (i) is directly derived from the total loss of all domains, and (ii) uses adaptive coefficients to achieve a more balanced trade-off between learning the current domain and avoiding forgetting previous domains.

---

Table 3: Performances (%) evaluated on Seq-CORe50. We use three metrics, Average Accuracy, Forgetting, and Forward Transfer, to evaluate the methods' effectiveness. "I" and "L" mean higher and lower numbers are better, respectively. We use boldface and underlining denote the best and the second-best performance, respectively. We use "_" to denote "not appliable" and "x" to denote out-of-memory (OOM) error when running the experiments.
Method
Butfer
Dass
Dun
Drw
Pa0s11
Overall Forgetting (i)
Avg. Ace
Avg. Ace (T)
Fwd. Transfer
Fine Lume
18.707
29.406:25M
28.689
31.832:1104
73.296.11
15.153
oEWC [81]
74.567:13.00
35.915
30.174:a
28. 18.291=1522
30.813:1114
74.563:20.07
15.041=0349
[105]
74.661
34.345
30.127.2991
28. 185839
32,46911331
11.144.1188
14.8
LwF [52]
80.383 10.1900
28.357
31.386=0.28
2
31.692::0.65
72.990=1.350
15.356
GEM [58]
79.852#5854
38.961=1718
39,258=2614
36.859=0s42
37.701=0273
22.724
19.030=av95
A GEGEM [10]
80,3481:900
41.472.3394
43.213:11522
39. 181
43.181:2025
33.775 1:000
19:053
ER [75]
20
79.343
65.151
65.03421151
66.605
32.750,p0.49
2
DER+ [8]
500
12.444
88.6522:155
80.391
:8.038
78.629:0025)
21.910:1.m4 *
22.488:
CI.-ER [4]
19.83421177 335
18.909
70.59
1
ESM-ER ER [80]
84,905
51.905
53.815
50.178::257
52.75
25. 4..4
21 2
UDI. UDDI)(Our))
18.152.1145
89.814_1.M
83.052..0.511
81.547:-630
82. 10.
19.589 _01.33
31.215
Joint (Oracle)
99.1337.0000
5.4
Rotating MNIST
We also evaluate our method on the Rotating MNIST dataset (R-MNIS7) containing 20 sequential domains. Different from P-MNIST where shift from domain to domain + is abrupt, R-MNIS7's domain shift is gradual. Specifically, domain t's images are rotated by an angle randomly sampled from the range 99- highest average accuracy (86.635% and the lowest forgetting (8. 8.506%) simultaneously, significantly improving on the best baseline DER++ (average accuracy of 84. 258% and forgetting of 13.65 (3.692%). Interestingly, such improvement is achieved when our UDIL's B, is high, which further verifies that UDIL indeed leverages the similarities shared across different domains so that the generalization error is reduced.
1), 9 t). Column and of Table 2 show that our UDIL achieves the
5.5
Sequential CORe50
CORe50 [55, 56] is real-world continual object recognition dataset that contains 50 domestic objects collected from 11 domains (120,000 images total). Prior work has used CORe50 for settings such as domain generalization (e.g., train model on only 8 domains and test it on 3 domains), which different from our comain-incremental learning setting. To focus the evaluation on alleviating catastrophic forgetting, we retain 20% of the data as the test set and continually train the model on these 11 domains; we therefore call this dataset variant Seq-CORe5te Table shows that our UDIL outperforms all baselines in every aspect on Seq- 500 Besides the average accuracy over all domains, we also report average accuracy over different domain intervals (e.g., Daa denotes average accuracy from domain 1 to domain 3) to show how different model's performance drops over time. The results show that our UDIL consistently achieves the highest average accuracy until the end. It is also worth noting that UDIL also achieves the best performance another two metrics, i.c., forgetting and forward transfer.
6 Conclusion
In this paper, we propose a principled framework, UDIL, for domain incremental learning with memory unify various existing methods. Our theoretical analysis shows that different existing methods are equivalent to minimizing the same error bound with different fixed coefficients. With this unification, our UDIL allows adaptive coefficients during training, thereby always achieving the tightest bound and improving the performance. Empirical results show that our UDIL outperforms the state-of-the-an domain incremental learning methods on both synthetic and real-world datasets. One limitation of this work is the implicit i.i.d. exemplar assumption, which may not hold if memory selected using specific strategies. Addressing this limitation can lead to a more powerful unified framework and algorithms, which would be interesting future work.
10

---

Acknowledgement
The authors thank the reviewers/AC for the constructive comments to improve the paper. HS and HW are partially supported by NSF Grant IIS-2127918. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed implied, of the sponsors.
References
[1] H. Ahn, S. Cha, D. Lee, and T. Moon. Uncertainty- based continual learning with adaptive regularization. Advances in neural information processing systems, 32, 2019.
[2] Aljundi, Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[3] M. Authony, P. L. Bartlett, P. L. Bartlett, et al. Neural network learning: Theoretical foreriicaooo foundations, volume 9. cambridge university press Cambridge. 1999.
[4] E. Arani, F. Sarfraz, and B. Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. arXiv preprint arXiv:2201.16666 2022.
[5] S. Ben-David J. Blitzer, K. Crammer. A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine learning, 79:151-177 2010.
[6] C. M. Bishop. Pattern recognition and machine learning. springer, 2006.
[7] M. Boschini, L. Bonicelli, P. Buzzega, A. Porrello, and S. Calderara.
continual learning into the extended der-verse. IEEE Transactions Patiern Analysis and Machine Intelligence, 2022.
[8] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33:159 13:15920-15590 2020.
[9] H. Cha, Lee, Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference on computer vision, pages 9516-9525, 2021.
[10] A. Chaudhry, M. Ranzato, Rohrbach, and M. Elhoseiny. Efficient lifelong learning with a-gem, arXiv preprint 10I.:v11112000220.0 2018.
[11] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato. On tiny episodic memories in continual learning. arXiv preprint 190.//4999000800 2019.
[12] T. Chen, S. Kornliihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 15597-6077. PMIR, 2020.
[13] T. Chen, H. Shi, S. Tang, Z. Chen, F. Wu, and Y. Zhuang. Cil: Contrastive instance learning framework for distantly supervised relation extraction. arXiv preprint arXiv:2106. 10855, 2021.
[14] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0423 04297, 2020.
[15] Z. Chen and B. Liu. Lifelong machine learning, volume 1. Springer, 2018
[16] 7. Chen, J. Zhuang, X. Liang, and L. Lin. Blending- ttrgett domain adaptation by adversarial meta- adaptation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2248-2257. 2019.
[17] Dai, Sohn, Y.H. Tsai, L. Carin, and M. Chandraker. Adaptation across extreme variations using unlabeled domain bridges. arXiv preprint arXiv:1906.0222 2019.
[18] Dai, Lang, Y. Zheng, B. Yu, F. Huang, and Y. Li. Domain incremental lifelong learning an open world. arXiv preprint arXiv:2305.066 06555, 2023.
[19] M. Davari, N. Asadi, S. Mudur, R. Aljundi, and E. Belilovsky. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16712-16722 2022.
11

---

[20] M. De Lange, R. Aljundi, M. Masana. S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions puttern analysis and machine intelligence, 44(7):3366-35) 2021.
[21] D. Deng, G. Chen, Hao, Q. Wang, and P.A. Heng. Flattening sharpness for dynamic gradient projection memory benefits continual learning. Advances in Neural Information Processing Systems, 14:18710-18721, 2021.
[22] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint urXiv:1810.04800 2018.
[23] P. Dhar, R. V. Singh, K.-C. Peng. Z. Wu, and R. Chellappa. Learning without memorizing, In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages
2019.
[24] S. Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach. Uncertainty-guicl continual learning with bayesian neural networks. arXiv preprint arXiv:1906.00 02425, 2019.
[25] J. Gallardo, T. L.. Hayes, and C. Kanan. Self-supervised training enhances online continual learning. arXiv preprint 103 2021.
[26] Ganin, Ustimova, H. Ajakan, P. Germain, H. Larochelle, F, Lavioleue, Marchand, and Lem- pitsky. Domain-adveesarial training of neural networks. The journal of machine learning research, 17(1):2096-200 2030,,220000200 2016.
[27] P. Garg, R. Saluja, V. N. Balasubramanian, C. Arora, A. Subramanian, and C. Jawahar. Multi domain incremental learning for semantic segmentation. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 761-771, 2022.
[28] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11))139-144, 2020.
[29] Graf. Hofer, M. Niethammer, and Kwitt. Dissecting supervised contrastive learning. In International Conference on Machine Learning, pages 3821-3830. PMLR, 2021.
[30] Y. Guo, B. Liu, and D. Zhao. Online continual learning through mutual information maximization. In International Conference on Machine Learning, pages 8109 8126. PMLR, 2022.
[31] Y. Guo, L.. Zhang, Y. Hu, X. Hc, and J. Gao. Ms-celeb-Im: A dataset and benchmark for large-scale face recognition. In Computer Vision-ECCT ECCV 2016: 14th European Conference, Amstendam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 87-102. Springer, 2016.
[32] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2020.
[33] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recogmition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: (503:/5003.0531, 2015.
[35] G. E. Hinton, S. Osindero, and Y.W. Teh. A fast learning algorithm for deep belief nets. Neural computation, 1577:1557- 2006.
[36] S. Hochreiter and J. Schmidhuber. Long short-tent term memory. Neural computation, 9(8: 1780, 1997.
[37] W. Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pages 409 029,426,11 1994.
[38] C.-Y. Hung, C.-H. Tu, C.-F. Wu, C.-H. Chen, Y.-.. Chan, and C.-S. Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[39] D. Jung, D. Lee, S. Hong. H. Jang, H. Bae, and S. Yoon. New insights for the stability-plasticity dilemma in online continual learning, arXiv preprint arXiv:2302.087411 2023.
[40] H. Jung. J. Ju. M. Jung, and J. Kim. Less-forgetting learning in deep neural networks. arXiv preprint 100.00600.0012..... 2016.
12

---

[41] T. Kalb, M. Roschani, M. Ruf, and J. Beyerer. Continual learning for class-and domain-incremmnnta semantic segmentation. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 1345-1351. IEEE, 2021.
[42] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18677 2020.
[43] D. Kim and B. Han. On the piaiiity-pltiiittyy dilemma of daas-incrementa learning. arXiv preprint arXiv:2304.01663 2023.
[44] G. Kim, C. Xiao, T. Konishi, Z. Ke, and B. Liu. A theoretical study solving continual learning. Advances in Neural Information Processing Systems, 35:5065-5079, 2022.
[45] G. Kim, C. Xiao, T. Konishi, and B. Liu. Learnability and algorithm for continual learning. arXiv preprint arXiv:2306. 12646, 2023.
[46] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan,
T. Ramalho, A. Barwinka, et al. Overcoming catastrophic forgetting in neural networks. Proceedings the national academy of sciences, 11 (1111333:521-3555 2017.
[47] V. Kothapalli, E. Rasromani, and V. Awatramani, Neural collapse: A review on modelling principles and generalization, arXiv preprint
2022.
[48] A. Krizhevsky, Sutskever, and G. E. Hinton. Imagonet classification with deep convolutional neural networks. Communications of the ACM, 60(6):849900 90, 2017.
[49] R. Kurle, B. Cseke, A. Klushyn, P. Van Der Smagt, and S. Günnemann. Continual learning with bayesian neural networks for non-stationary data. In International Conference on Learning Representations, 2019.
[50] LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: 2, 2010.
[51] M. Li, H. Zhang, J. Li. Z. Zhao, W. Zhang. S. Zhang. S. Pu, Y. Zhuang, and F. Wu. Unsupervised domain adaptation for video object grounding with cascaded debiasing learning. In Proceedings of the 31th ACM International Conference on Multimedia, 2023.
[52] Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, (0(11))2935-22 29997 2017.
[53] Z. Li, L. Zhao, Z. Zhang, H. Zhang, D. Liu, T. Liu, and D. N. Metaxas. Steering prototype with prompt-tuning for rehearsal free continual learning. arXiv preprint arXiv:2303.09447, 2023.
[54] T. Liu, Z. Xu, H. He, G. Hao, G.-H. Lee, and H. Wang. Taxonomy-structured domain adaptation. In ICML, 2023.
[55] V. Lomonaco and D. Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In S. Levine, V. Vanhoucke, and K. Goldberg, editors, Proceedings of the 1st Annual Conference Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 17-26. PMLR, 13-15 Nov 2017.
[56] V. Lomonaco. D. Maltoni, and L. Pellegrini. Rehearsal-free continual learning over small non-iid batches. CVPR Workshops, volume 1, page 3, 2020.
[57] M. Long, 7. CAO, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. I Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett, editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[58] D. Lopez-Paz Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[59] U. Michieli and P. Zanuttigh. Knowledge distillation for incremental learning in semantic segmentation. Computer Vision and Image Understanding, 205: 103167, 2021.
[60] M. J. Mirza, M. Masana, H. Possegger, and H. Bischof. An efficient domain-incremental learning approach to drive in all weather conditions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3001-3011, 2022.
[61] S. I. Mirzadeh, M. Farajtabar, R. Pascanu, and H. Ghasemzadeh. Understanding the role of training regimes in continual learning. Advances in Neural Information Processing Systems, 33:7308-755 73200 2020.
13

---

[62] M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of muchine learning. MIT press, 2018.
[63] C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner. Variational continual learning. arXiv preprint arXiv: 10628, 2017.
[64] L. T. Nguyen- Meidine, A. Belal, M. Kiran, J. Dolz, L.-A. Blais-Morin, and E. Granger. Unsupervised multi-target domain adaptation through knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1339-1347, 2021.
[65] 7- Ni, H. Shi, S. Tang, L.. Wei, Q. Tian, and Y. Zhuang. Revisiting catastrophic forgetting in class incremental learning. arXiv preprint vrXi:22107.13306 2021.
[66] Z. Ni, L. Wei, S. Tang. Y. Zhuang, and Q. Tian. Continual vision-language representation learning with off-diagonal information, 2023.
[67] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. TEEE transactions on neural nenworks, 22(2):99-2110, 2010.
[68] S. J. Pan and Q. Yang. A survey transfer learning. IEFF Transactions knowledge and data
engineering. 2(110)1345-135 2010.
[69] V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652 24663,, 2020.
[70] A. Paszke, S. Gross, F. Massa, A. Lever, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein. Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances neural information processing systems, 32, 2019.
[71] X. Peng, Q. Bai. X. Xia, Z. Huang. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEF/CVF international conference on computer vision, pages 1415, 2019.
[72] Q Pham, C. Liu, and S. Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34:16131-16144 2021.
[73] R. Ramesh and P. Chaudhari. Model zoo: A growing" brain" that learns continually. arXiv preprint arXiv:2 20f.::/00600022,, 2021.
[74] S.-A. Rebufti, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 2001-2010, 2017.
[75] M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv: 1990. 2018.
[76] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy. A. Khosla, M. Bermstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252 2015.
[77] G. Saha, 1. Garg. and K. Roy. Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762, 2021.
[78] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723-3732, 2018
[79] Sankaranarayanan, Balaji, C. D. Castillo, and Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8503-8512, 2018
[80] F. Sarfraz, E. Arani, and B. Zonooz. Error sensitivity modulation based experience replay: Mitigating abrupt representation drift in continual learning. arXiv preprint arXiv:2302.1 11344, 2023.
[81] J. Schwarz, W. Czarnecki. J. Luketina, A. Baroskaa. Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In International conference on machine learning, pages 1528--4331 PMLR, 2018.
[82] J. Serra, D. Suris, M. Miron, and A. Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In International Conference Machine Learning, pages 4548 4557. PMLR, 2018
14

---

[83] H. Shi, D. Luo, S. Tang. J. Wang, and Y. Zhuang. Run away from your teacher: Understanding byol by novel self-supervised approach. arXiv preprint arXiv:201 I. 10944, 2020.
[84] H. Shi, Y. Zhang, Z. Shen, S. Tang, Y. Li, Y. Guo, and Y. Zhuang. Towards communication-efficcn and federated representation learning. arXiv preprint arXiv:2109. 14611, 2021.
[85] H. Shi, Y. Zhang, S. Tang, W. Zhu, Y. Li, Y. Guo, and Y. Zhuang. On the efficacy of small self-supervised contrastive models without distillation signals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2225-2234, 2022.
[86] M. S. Sorower. A literature survey on algorithms for multi-label learning. Oregon State University. Corvallis. 18(1):25, 2010.
[87] B. and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision- ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443-450. Springer, 2016.
[88] Thengane, S. Khan, M. Hayat, and Khan, Clip model an efficient continual learner. arXiv preprint 21010:2210.03114, 2022.
[89] Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 1412::441..3474, 2014
[90] G. M. van de Ven, T. Tuytelaars, and A. S. Tolias. Three types of incremental learning. Nature Machine Intelligence, pages 1-13, 2022.
[91] N. Vapnik and Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Measures of complexity: festschrift for alexey chervonenkis, pages 11 11-30, 2015.
[92] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I.. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[93] J. S. Vitter. Random sampling with a reservoir, ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 57, 1985.
[94] H. Wang, H. He. and D. Katabi. Continuously indexed domain adaptation. arXiv preprint arXiv:2007.0180) 2020.
[95] L. Wang. X. Zhang. Q. Li, J. Zhu, and Y. Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 254-271. Springer, 2022.
[96] L. Wang, X. Zhang, H. Su, and J. Zhu, A comprchensive survey of continual learning: Theory, method and application. arXiv preprint ar arXiv:2302.0044 2023.
[97] Y. Wang, Z. Huang, and X. Hong. S S-prompts learning with pre- trained transformers: An occam's razor for domain incremental learning, arXiv preprint arXiv:2207.128/) 2022.
[98] Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. Dualprompt: Complementary prompting for rehearsal free continual learning. In European Conference on Computer Vision, pages 631 648. Springer, 2022.
[99] Z. Wang. Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren. G. Su, V. Perot, J. Dy, and T. Plister. Learning prompt for continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[100] Wu, Y. Chen, Wang. Ye, Z. Liu, Guo, and Y. Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 382, 2019.
[101] Z. Xu, G. Hao, H. He, and H. Wang. Domain Domain-indexing variational bayes: Interpretable domain index for domain adaptation. In International Conference on Learning Representations, 2023.
[102] Z. Xu, G.-H. Lee, Y. Wang, H. Wang, et al. Graph-relational domain adaptation. arXiv preprint 2mXiv:2202.03628 2022.
[103] C. Yaras, P. Wang, Z. Zhu, L. Balzano, and Q Qu. Neural collapse with normalized features: A geometric analysis over the riemannian manifold. Advances in neural information processing systems. 15:11547-1156, 2022.
15

---

[104] J. Yoon, E. Yang. J. Lee. and S. J. Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
[105] F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987-3995. PMLR, 2017.
[106] M.-L. Zhang and Z.-H. Zhou. MI-knn: A lazy learning approach to multi-label learning. recognition, 40(7):2038-2044 2007.
Puttern
[107] M.-I.. 7hang and 7.-H. Zhou. review multi-label learning algorithms. IEFE transactions on knowledge and data engineering, 1818) 2013.
[108] Y. Zhang, T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. In International conference on machine learning, pages 7404-7413, PMLR, 2019.
[109] M. Zhao, S. Yue, D. Katabi, S. Jaakkola, and M. T. Bianchi. Learning sleep stages from radio signals: conditional adversarial architecture. In International Conference Machine Learning, pages 4100-4109. PMLR, 2017.
[1 [0] J. Zhou, C. You, X. Li, Liu, S. Liu, Q Qu, and 7. 7hu, Are all losses created equal: neural collapse perspective. Advances in Neural Information Processing Systems, 35:31697-31710, 2022.
[111] Z. Zhu, Ding, J. Zhou, X. Li, C. You, Sulam, and Q. Qu. A geometric analysis of neural collapse with unconstrained features. Advances Neural Information Processing Systems, 34:29820-29834, 2021.
16

---

Appendix
In Sec. A, we present the proofs for the lemmas, theorems, and corollaries presented in the main body of our work. Sec. B discusses the correspondence of existing methods to specific cases within our framework. In Sec. C, we provide detailed presentation of our final algorithm, UDIL., including an algorithmic description, a visual diagram, and implementation details. We introduce the experimental settings, including the evaluation metrics and specific training schemes. Finally, in Sec. D, we present additional empirical results with varying memory sizes and provide more visualization results.
A Proofs of Lemmas, Theorems, and Corollaries
Before proceeding to prove any lemmas or theorems, we first introduce three crucial additional lemmas that will be utilized in the subsequent sections. Among these, Lemma A. offers a generalization bound for any weighted summation of ERM losses across multiple domains. Furthermore, Lemma A.2 provides generalization bound for a weighted summation labeling functions within a given domain. Lastly, we highlight Lemma 3 in [5] Lemma A.3, which will be used to establish the upper bound for Lemma 3.3.
Lemma A.1 (Generalization Bound of a-weighted Domains). Let H be a hypothesis space of VC dimension d. Assume N, denotes the number of the samples collected from domain j. and N E, N₃ is the total number of the examples collected from all domains. Then for any a and 8 E (0, 1), with probability at least 8:
Σ agje (h) S
Σ a1) +
(8d log (22N) + 8 log (3))
(17)
Proof. Suppose each domain Dj has deterministic ground-truth labeling function fj R" - {0,1}, 1}. Denote Ea - E, as (h) the w-weighted empirical loss evaluated on different domains. Hence,
N
(h(h) = (ajēD,(h) (h) = E Cast, Σ xEX,
a
ΣΣ k=
Rg
(18)
where RíAA- = ( ( 1 (wx) is random variable that takes the values in { (o,N, 0}. By the linearity of the expectation, have ca (h) [A) Following [3, 62], we have
P (3h E H. s.t. (aa c. c)
(19)
< 2.P P
sup Ea (h) - 2,2 (h)l 2 5 WCH
(20)
N,
< 2.P
Σ
RR, - R2,a) 25
(21)
N
Rya.RRy k
3
km
< (2N) exp
2(N2/2)2
(22)
(, NYN,Y
22xx(2N) exp
2 8 ,
(23)
< 2(2N) exp
(24)
3E,("//M))
where in Eqn. 20, (h) is the w-wehttted empirical loss evaluated on the "ghost" set of examples
{XG); Eqn. 22 is yielded by applying Hoeffding's inequalities [37] and introducing the growth
function 1l [3, 62, 91] the same time; Eqn. 24 is achieved by using the fact (2N) < (e-2N/a)d < (2N)d, where d is the VC-dimension of the hypothesis set H. Finally, by setting Eqn. 24 to and solve for the error tolerance e will complete the proof.
Lemma A.2 (Generalization Bound of 3-weighted Labeling Functions). Let D be a single domain and X {x be collection of samples drawn from D; H is hypothesis space of VC dimension
17

---

d. Suppose {f, : + (0,1);; is a set of different labeling functions. Then for any B3 > 0 and
8 E (0, 1), with probability least 1 ò:
Beeo(h, $3) (1 + E B3) N (8d (2eN) + 8 log (3))
(25)
Proof. Denote as (g(h) - (h,, f2] the 3-weighted error on domain and (fz), the set of the labeling functions, and ē3 - Σ, (h, fz) the B-weighted empirical loss evaluated different labeling functions. We have
N
(h(h) = Σ 3
3
1 MaaDE(())
Σ
E By ((
& N 2m
Ra
(26)
N 2=1
where R₁
E3 ((2))))11))) E0. Bz] is random variable.
Then we have
{3h E H. s.t. (h) c0(h)) e
(27)
< 2 P sup
( [1) 2
(28)
IIH
N
< 2
Σ RR:
R$)
(29)
Ru,
< 21IH (2N)
exp
-2(8g/232 N.-22, 233
(30)
A 2(2N)* exp
(31)
8,,, 9,)
where in Eqn. 28, (h) the 3-weighted empirical loss evaluated on the "ghost" set of examples
X"; Eqn. 30 yielded by applying Hoeffding's inequalities [37] and introducing the growth function I1 [3, 62, 91] at the same time; Eqn. 31 is achieved by using the fact (2N) I (e-2N/a)á < (2N)), where d the VC-dimension the hypothesis set H. Finally, by setting Eqn. 31 to å and solve for the error tolerance will complete the proof.
Lemma A.2 asserts that altering or merging multiple target functions does not impact the generaliza- tion error term, as long the sum of the weights for each loss Σ, 33 remains constant and the same dataset X' is used for estimation. Next we highligt the Lemma 3 in [5] again, as it will be utilized for proving 3.3.
Lemma A.3. For any hypothesis h, J E H and any two different domains D, D,D',
leD h²) <D (h, 161 S (dHAH(D,D))
(32)
Proof. By definition, have
(HAH(D,D)) = 2 sup K
PPaDD # 11 11(x))) - P (aaD)(((())
2 sup leD h1) ED (h, h!)l
2 2 (cp h/) cD ( (h, h')l
Now we are ready to prove the main lemmas and theorems in the main body of our work.
Lemma 3.1 (ERM-Based Generalization Bound). Let H be hypothesis space of VC dimension d. When domain & arrives, there are N₂ data points from domain and N, data points from each previous domain i < in the memory bank. With probability at least 1 - 8, we have:
(h) < Σ (h) + /
(
Nr
N.
(&dlog (2g) + 8log (2))
(33)
1=1
18

---

Proof. Simply using Lemma A.1 and setting 1 for every i E [t] completes the proof.
Lemma 3.2 Intra-Domain Model-Based Bound). Let h H be an arbitrary function in the hypothesis space H. and H be the model trained after domain I - 1. The domain-specific error ED, (h) on the previous domain has an upper bound:
(h) S ED, (h, H1-1) + ED. (Hle-1);
(34)
where ED, (h, 111-1) Em-D,Ihar + + (+)))
Proof. By applying the triangle inequality [5] of the 0-1 loss function, have
cD, (h) ED. (h, fa)
S cD. (h, H1-1) + CD. (H
CD. (h, H1-1) CD. (H(-1).
Lemma 3.3 Pross-Domain Model-Based Bound). Let h H be arbitrary function in the hypothesis space H. and H1-1 be the function trained after domain - 1. The domain aamain-ppecific error ED, (h) evaluated on the previous domain i then has upper bound:
ED. (h) S CD. (h, H1-1) duau D:) + CD. (Hi-1),
(35)
where dHAH (P, Q) SHuPhEHAH PPzz~Plh(C (x) 1] - Pla~o) 1|] denotes the HAH- divergence between distribution and Q, and eD, (h, H,-1) a EnwD, (1(x)) Hr)1(
Proof. By the triangle inequality used above and Lemma A.3, we have
cD. (h) S CD H1-1) cD. (H
ep, (h, H + eD. (H--1) CD. (h, eD ep, H1-1) S (H,-)) + <D. (h, H1-1) (D, H1-1)l cD, (h, H(-1)
S <D,(h, (h, H(-1) [ddAa D.) + CD. (H--1).
Theorem 3.4 (Unified Generalization Bound for All Domains). Let be a hypothesis space of VC dimension Let N = N, N.+ N; denoting the total number of data points available to the training of current domain where N₂ and N; denote the numbers of data points collected at domain t and data points from the previous domain in the memory bank, respectively. With probability at least 1 - 8, we have:
Σ ED (1)
<
(nED, (h) + asED, (h, H. H1--)) (
(E) + 2B, (h, Ho-1)
+
2
Bidnan D.) + (a: + Bi)CD (H)e
+
(1+5 11+51133 3.)²
+
(u+aa)?
(Sdlog (2eN) + log (2)
N.
A g(,, H;-1,
(36)
where ēD. (h, H1-1) N.
1 Exsi, An(o)) ēD. ED, (h, H(-1) N. ExEx, Ln(m)) and
2 (ar 3r, 1=11
Proof. By applying Lemma 3.2 and Lemma 3.3 to each of the past domains, we have
ED. (h) (a; + 7)p,
S (h) + t,lle, H4 + ep. (H1r1))
+ Bie,, (h, H,-1) cD, (h, H(-1) + ]dnAH(D, D.)
19

---

Table 4: Unification of existing methods under UDIL, when certain conditions achieved.
Transformed Objective
Condition
UDIL (Ours)
[0,1] [0. [0,
LwF [52]
(ear(h) = Ex. (h) Autxe H,-1)
A
ER [75]
Cex(1) = Ee (M) + Σ
(1)(A)
[B/]
12
DER+ [8]
1/2
1/a
A.ER.. (h) - (h) + 2
Fey + (b, He-.)l
[R]] [8
iCaRI. [74]
0
((cua((()) = Pb)
Σ
10c//e-11 M841
(ty Hi
[8]]
[81 (1-1)
CLS-ER
X+1
X+1
CCISER(1)) (b)) Σ
2-t (l He-1)
1=1-2
ESM-ER (80)
(rs) (h.(h)+
( रা oo (b, He-i)
[== -=-1+r(t--
r==l-
(mc(M) =fn, (k)
Σ 2--ll (b, H(-1)
BIC [100]
[B] = [Bc]
+ (1- -1 He-1)+
Re-organizing the terms will give
(h) (h, H--i)l { (D, + ② 3,)eD, (h, Ho-1)
+
Σ (1)
+ 2 Σ BidnaH(D D.) + Σ (a: +
(37)
Then applying Lemma A.1 and Lemma A.2 jointly to Eqn. 37 will complete the proof.
B UDIL as a Unified Framework
In this section, we will delve into a comprehensive discussion of our UDIL framework, which serves as unification numerous existing methods. It is important to note that we incorporate methods designed for task incremental and class incremental scenarios that can be easily adapted to our domain incremental learning. To provide clarity, we will present the corresponding coefficients fa,, 8r, 7 of each method within our UDIL framework (refer to Table 4). Furthermore, we will explore the conditions under which these coefficients are included in this unification process.
Learning without Forgetting (LwF) [52] was initially proposed for task-incremental learning, incorporating a combination of shared parameters and lask-specific parameters. This framework can be readily extended to domain incremental learning by setting all "domain-specific"" parameters to be the same in a static model architecture. LwF was designed for the strict continual learning setting, where no data from past tasks is accessible. To overcome this limitation, LwF records the predictions of the history model H; on the current data X; at the beginning of the new task Subsequently, knowledge distillation (as defined in Definition 4.2) is performed to mitigate forgetting:
Loaah, H,-1)
H. (z)) [log([(<))))) = Zx. (h, H,-1),
(38)
ZEX k=1
where Hz-1(w), h(x)) E R are the class distribution of over classes produced by the history model and current model, respectively. The loss for learning the current task Lnew is defined as
Lnew (1))
Σ
1 [og((><=))))] = Ex. (h))
(39)
N.
S
k=
20

---

LwF uses "loss balance weight" X to balance two losses, which gives us its final loss for training:
(h) _ Lnew (h) + Xo= (h,a H(-1).
(40)
In LwF, the default setting assumes the presence of two domains (tasks) with A, 1. However, is possible to learn multiple domains continuously using LwF's default configuration. To achieve this, the current domain & can be weighed against the number of previous domains (1 versus / 1).
Specifically, if there is no preference for any particular domain, A, should be set to L - 1. Remarkably, this is equivalent to setting {3: 1, a; = 7/4 = 0} in our UDIL framework (Row 2 in Table 4).
Experience Replay (ER) [75] serves as the fundamental operation for replay-based continual learning methods. It involves storing and replaying subset of examples from past domains during training. Following the description and implementation provided by [8] ER operates as follows: during each training iteration on domain along with a mini-batch B' from the memory. These two mani-batchees are then concatenated into a larger mini-batch (B, U Bj), upon which average gradient descent is performed:
mini-batch B, of examples is sampled from the current domain,
(hR(h) = (B,UB((h)
(41)
Σ
Σ
1 yー [og(((())))))
(42)
[8+++B8
(m.y)) B, UB k=1
B. 1 BI (h) +
[B [B+ (A).
(43)
Suppose that each time the mini-batch of past-domain data is perfectly balanced, meaning that each domain has the same number of examples in By. In this case, Eqn. 43 can be further decomposed as follows:
(ER(H) (h) [B.++ BT
BBd
B.. (h)
Σ
miveu B++BB (h)),
(44)
where B¹ = ((xe,))(e v) (B) Mi)} is the subset of the mini-batch that belongs to domain i. Now, by dividing both sides of Eqn. 44 by (181)-18:1/ B.)) and comparing to Theorem 3.4, we can include ER in our UDIL framework when the condition [Be] (B//(tt1)) is satisfied. In this case, ER equivalent to {a; = 3, = 0, 74- = 1} in UDIL (Row 3 in Table 4). is important to note that this condition is not commonly met throughout the entire process of continual learning. It can be achieved by linearly scaling up the size of the mini-batch from the memory (which feasible the early domains) or by linearly scaling down the mini-batch from the current-domain data (which may cause a drop in model performance). It is worth mentioning that this incongruence highlights the intrinsic bias of the original ER setting towards current domain learning and cannot be rectified by adjusting the batch sizes of the current domain or the memory. However, it does not weaken our claim of unification.
Dark Experience Replay (DER++) [8] includes an additional dark experience replay, i.e., knowledge distillation on the past domain exemplars, compared to ER [75]. Now under the same assumptions (balanced sampling strategy and B.| = (111((-1)) as discussed for ER, we can utilize Eqn. 44 to transform the DER++ loss as follows:
LDER++ (h)
[B,
B.
B, (h) +
B B
InfifC 1) E
(h)
Σ
oh B.++Be B (h, He-1).
(45)
In this scenario, DER++ is equivalent to {a; 7/ 1/2,8, 0} in UDIL (Row 4 in Table 4).
Incremental Classifier and Representation Learning (iCaRL) [74] was initially proposed for class incremental learning. It decouples learning the representations and final classifier into two individual phases. During training, iCaRL. adopts an incrementally increasing linear classifier. Different from traditional design choice of multi-class classification where the softmax activation layer and multi- class cross entropy loss are used jointly, iCaRL models multi-class classification as multi-label classification [107, 86, 106]. Suppose there are K classes in total, and we denote the one-hot label vector of the input x as y E K where yj A [(=)= Then the multi-label learning objective treats each dimension of the output logits as a score for binary classification, which is computed as follows:
ex (h) 2 1
Σ
K
[5 log (I1(=)1,,) + (1 - -y) log (1 [b(=)1,)]
(46)
DEX
1=1
21

---

When a new task that contains K' new classes is presented, iCaRL adds additional K' new dimensions to the final linear classifier.
In iCaRL training. the new classes and the old classes are treated differently. For new classes, it trains the representations of the network with the ground-truth labels (f():) in the original paper), and for old classes, it uses the history model's output (H2-1 (œ)) as the learning target (i.e., pseudo labels). Each data point treated with the same level of importance during iCaRL's training procedure. Hence after translating the loss function of iCaRL in the context of class-incremental learning to comain-incremente learning, we have
L-1 No
(h) = No- (x) +
(h)
Σ i=1
Bx (h, HD-1);
(47)
where in Ex (h, H1-1), we follow the same definition as in the distillation loss in Eqn. 4.2 and replace the ground-truth label with the soft label for iCaRI..
Similar to ER [75] and DER++ [8] as analyzed before, the equation above does not naturally fall into the realm of unification provided in UDIL. To achieve this, we need to sample the data from the current domain and exemplars in the memory independently. i.e., assuming B, and B,. By applying the same rule of o; + 3: + Ti 1, we now have that for iCaRL, the corresponding coefficients are fa; = B./B. = 1, 3, = 74- = 0} (Row 5 in Table 4).
Complementary Learning System based Experience Replay (CLS-ER) [4] involves the mainte- nance of two history models, namely the plastic model H and the stable model H[ throughout the continual training process of the working model h. Following each update of the working model,
the two history models are stochastically updated at different rates using exponential moving averages (EMA) of the working model's ' parameters:
H - (.) (0) H
(1- h, i E {p,s),
(48)
where (P) I (+) is set such that the plastic model undergoes rapid updates, allowing to swiftly adapt to newly acquired knowledge, while the stable model maintains "long-term memory" spanning multiple tasks. Throughout training, CL-S-ER assesses the certainty generated by both history models and employs the logits from the more certain model as the target for knowledge distillation.
In the general formulation of the UDIL framework, the history model - is not required to be single model with the same architecture as the current model h. In fact, if there are no constraints on memory consumption, we have the flexibility to train and preserve a domain-speeiff each domain i. During testing, we can simply select the prediction with the highest certainty from each domain- speeiin- model. From this perspective, the sawo-history-moe model system" employed in CLS-ER can be viewed as a specific and limited version of the all-domain history models. Consequently, we can combine the two models used in CLS-ER into single history model H as follows:
model H; for
H (x)
[H > |1[-] (x)ly
Hz-1(x) A
(49)
H (z) O.W.
where (x. E .M is an arbitrary exemplar stored in the memory bank.
At each iteration of training, CL.S-ER samples a mini-batch B, from the current domain and mini-batch BY from the episodic memory. then concatenates B, and B/ for the cross entropy loss minimization with the ground-truid truth labels, and uses B₁ to minimize the MSE loss between the logits of h and 11-1- To align the loss formulation of CLS-ER with that of ESM-ER [80], here we consider the scenarios where the losses evaluated on B3 and BY are individually calculated, i.e., we consider
Eg, (h) + br (h) instead of B, (h). Based on the assumption from [34], the MSE loss on the logits is equivalent to the cross-entrops loss on the predictions under certain conditions. Therefore, following the same balanced sampling strategy assumptions as in ER, the original CLS-ER training objective can be transformed as follows:
(CL)-ER(h) = EE. + mnp(h) + (h,
(50)
1-1
1-1
- [bb,(h) 120+~>+ 2 mx (h) + Σ En H1-1).
(51)
1=1
1=1
22

---

Therefore, by imposing the constraint a, + B; + 1, we find that À = -- Substituting this value back into 1/1-1 yields the equivalence that CLS-ER corresponds to {a: A/a+1, S 0,74 1/>-13 in UDII,, where À= = - 2 (Row 6 in Table 4).
Error Sensitivity Modulation based Experience Replay (ESM-ER) [80] builds upon CLS-ER by incorporating an additional error sensitivity modulation module. The primary goal of ESM-ER is to mitigate sudden representation drift caused by excessively large loss values during current-domain learning. Let's consider (æt y) ~ Dr, which represents a sample from the current domain batch. In
ESM-ER, the cross-entropy loss value of this sample is evaluated using the stable model 41!") and can be expressed as:
l(e, v) = - log([ (z))l)-
(52)
To screen out those samples with a high loss value, ESM-ER assigns each sample a weight À by comparing the loss with their expectation value, for which ESM-ER uses running estimate μ as its replacement. This can be formulated as follows:
if v) S B.
X(x)
(53)
E(2.3))
O.W.
where & is a hyperparameter that determines the margin for a sample to be classified as low-loss. For the sake of analysis, we make the following assumptions: (i) 3 = 1: (ii) the actual expected loss value E [f(zr, y)] is used instead of the running estimate je; (iii) hard screening mechanism is employed instead of the current re-scaling approach. Based on these assumptions, we determine the sample wise weights x according to the following rule:
(xxx) = {
[1
if (xx v) S Be Beyll(a v)l
(54)
o.W.
Under the assumption that the loss value {(x, y) follows an exponential distribution, denoted as f(x,)) ~ Exp(Ao). where the probability density function is given by f(((x, y), An) = Age (m))))))))) we can calculate the expectation of the loss as Ez.y (f( y)] = 1/>e- Based on this, now determine the expected ratio of the unscreened samples in mini- using the following equation:
S( e dy = e
1 Age >o de(x,y) =
(55)
The ratio r represents the proportion of effective samples in the current-domain batch, as the weights (xxx of the remaining samples are set to 0 due to their high loss value. Consequently, the original training loss ESM-ER can be transformed as follows:
LESM-ER (h) = - Er. +
XEez (h-1)
(56)
t-1
1-1
[r. + Σ - Zo (h) + Σ
1-1 Epg (h, He-1).
d
(57)
1=1
After applying the constraint of a; + 3, + 7/4 1, we obtain X r. (1 - 1)-11 1. Substituting this value back into A/r(e-1), we find that ESM-ER is equivalent to {a. = 1/2+1, 3, = 0, Y = ./>+1} in UDIL, where À 1)-
(6-1)
e )t - 1) - should be set (Row 7 in Table 4).
Bias Correction (BiC) [100] was the first to apply class incremental learning at a large scale, covering extensive image classification datasets including ImageNet (1,000 classes, [76]) and MS- Celeb-1M (10,000 classes, [31]). Similar to iCaRL [74], BiC treats each data example (from new classes' data and the memory) with the same level of importance. Different from its previous work, the distillation loss BiC (La in the original paper) implicitly includes the cross-domain distillation loss described by Lemma 3.3, as it computes the old classifier's output evaluated on the new data (considering only old classes). In addition, BiC further re-balances the classification loss (Teg in the original paper) and the distillation loss (La) based the number of old classes and new classes m as follows:
LBic
72 n4m
Lad + m Le-
(58)
mmm
23

---

Eo (h, (n, (1,))))))
Histery Model Hi-1
M [M,)
Predictor p
(o,(h) (h)
Encoder
Diseriminator
Va(d.e)
Domain ID
Figure 2: Diagram of UDIL. M = {Mi represents the memory bank that stores all the past exemplars. S; corresponds the dataset from the current domain and the current model h = p is depicted separately in the diagram. The three different categories of losses are illustrated in the dark rectangles, while the weighting effect of the learned replay coefficient Ω = {ax, 3,, 7/4 7> is depicted using dashed lines.
By interpreting the distillation loss Lag as the combination of the intra-domain loss (Lemma 3.2) and the cross-domain distillation loss (Lemma 3.3), and substituting (n, m) with (t 1, due the consistent number of classes in each domain, we arrive at the BiC model for DIL:
(b) = No 1x.2x,1 (h, H+11)
Σ
No Ex (h, He-1)
t-1 No Ex
1 No (x,(h) (h) + E
(59)
The equation above relies the number of the current-domain examples and the exemplars (i.e., memory), which is constant in practice. Hence we replace (Nt, N.) with the mini-batch size (lB,). [B:)). After re-organizing the equation above, we have
LBic(h) =Bp,(h)=(B,(h +
[1-1)BB. Exy
(1,111 Hz-1)
+ (t ( EB, (h, H Σ
B. B.
(b)
(60)
The immediate observatio is that BiC exhibits significant bias towards retaining knowledge from past domains. This is evident in the coefficient of coefficient of cross-domain distillation summed over the past domains Σ B; = (t. - 1). which violates the constraints posed this work (a, + 33 + 7 1).
However, if slightly relax BiC's formulation and focus on the relative ratios of the three coefficient,
we get a; 8 = (t B) (t 1)B, : B;. Further applying the constraint a, + 8 + = 1 and assuming B, B, enforce balanced sampling strategy over different domains, arrive
faz 3 1/21-1, 7- = 1/2--1 (Row 8 in Table 4).
C Implementation Details of UDIL
This section delves into the implementation details of the UDIL algorithm. The algorithmic descrip- tion of UDIL is presented in Algorithm and diagram is presented in Fig. 2. However, there are several practical issues to be further addressed here, including (i) how to exert the constraints of probability simplex (Ja;, Bi 7)] E $2) and (ii) how the memory is maintained. These two problems will be addressed in Sec. C.I and Sec. C.2. Then, Sec. C.3 will introduce two auxiliary losses that improve the stability and domain alignment for the encoder during training. Next, Sec. C.4

---

will cover the evaluation metrics used in this paper. Finally, Sec. C.5 and Sec. C.6 will present detailed introduction to the main baselines and the specific training schemes we follow for empirical evaluation.
C.1 Modeling the Replay Coefficients ? {ar, %
Instead of directly modeling in a way such that can be updated by gradient descent and satisfies the constraints that o; + = 1 and or,, fe TV at the same time, set of logit variables {ūs, Bi, 75) E IR3 and the softmax function to indirectly calculate Ω during training. Concretely, we have:
a, 3 = softmax
Ok 3
expla.)/z. z. exp( (2o)8)//21
(61)
7/
exp(3:)/2,1
where Z; exp(ā;) + exp(3:) (3,) + exp(7; is the normalizing constant. At the beginning of training on domain f, the logit variables {ūt, 8r, 7r) = {0, 0,0) are initialized to all zeros, since we do not have any bias towards any upper bound. During training, they updated in the same way the other parameters with gradient descent.
C.2 Memory Maintenance with Balanced Sampling
Different from DER++ [8] and following work [4, 80] that use reservoir sampling [93] to maintain the episodic memory, UDIL adopts random balanced sampling after training on each domain. To be more concrete, given a memory bank with fixed size IM]. after domain t's training is complete, we will assign each domain quota of [MI/t, For the current domain t, we will randomly sample [MI/e]] exemplars from its dataset; for all the previous domains E [t - 1]. we will randomly swap out [IMl/--1 MI/] exemplars from the memory to make sure each domain has roughly the same number of exemplars. To ensure fair comparison, we use the same random balanced sampling strategy for all the other baselines. The following Algorithm 2 shows the detailed procedure of random balanced sampling.
Algorithm Balanced Sampling for UDIL
Require: memory bank M [-)) current domain dataset S,, domain ID t.
1: for = 1,.., 1 do
2: for j = 1,
M / do
4: (x. y) + RandomSample( (M,)
3:
w''1// + RandomSample((S)
5: Swap (x', y') into M, replacing (x,
6: end for
7:
end for
8: return M
C.3 Improving Stability and Domain Alignment for the Embedding Distribution
In this section, we will examine the training loss employed to align the embedding distribution across distinct domains. As discussed in Sec. 4.2, we decompose the model h into encoder and predictor p as h = p a e. In order to establish tighter upper bound proposed in Theorem 3.4, we introduce discriminator d that aims to distinguish embeddings based domain IDs. Specifically, during the training process of UDIL, given a set of coefficients S = {a, 3,, ). both the encoder e and the discriminator engage in the following minimax game:
min max A.Vald,e,
(62)
where ()) represents the "copying-wett weights-wwiitnannnee and- ann-stopping--eaaaeaamda -rrpping-gradients" operation, and d a hyperpa- rameter introduced to control the strength of domain embedding alignment. More specifically, the
25

---

value function of the mini-max game Va is defined as follows:
et
Σ |- log ([d(e())))))
Σ log ([d(e())))))) =
(63)
mES
As previously mentioned, the practical effect of Eqn. 62 is to align the embedding distribution of different domains, thus enhancing the model's 's generalization ability to both previously encountered domains and those that may be encountered in the future.
However, actively altering the embedding space can lead to the well-known dilemma [43, 96, 20, 39, 61]. This dilemma arises when the model needs to modify a significant number of parameters to achieve domain alignment for a new domain, potentially resulting in a mismatch between the predictor p and the encoder e, which, in turn, can lead to catastrophic forgetting. Furthermore, it is worth noting that the adversarial training scheme described in Eqn. 62 is primarily designed for unsupervised domain alignment [108, 57, 26, 109, 17, 102, 101, 94], where the semantic labels in the target domain(s) are not available during training. This implies that the current adversarial training technique does not fully leverage the label information in domain incremental learning.
To tackle the aforementioned challenges, i.e., (i) maintaining a stable embedding distribution across all domains and (ii) accelerating domain alignment with label information, we incorporate two additional auxiliary losses: (i) the past embedding distillation loss [40, 65] and (ii) the supervised contrastive loss [42, 29]. It's important to note that these auxiliary losses, in conjunction with the adversarial feature alignment loss Va. operate exclusively on the encoder space of the model and do not impact the original objectives outlined in Theorem 3.4, provided that encoder and predictor p remain sufficiently strong. Therefore, the two losses are used to simply stabilize the training, without compromising the theoretical significance of our work.
In past embedding distillation, also known as representation(a)) distillation or embedding distillation in previous work on continual learning [40, 65, 59], the model stores the embeddings of the memory after being trained on domain i 1. It then uses them to constrain the encoder's s behavior: the features produced on these samples should not change too much during the current domain training. After decoupling the history model H = Pt-1 E,- where Eg-1 is the history encoder and P is history predictor, we define the past embedding distillation loss Vpp
Vi((e) Σ EaND, [lle(x) - Ex)1
(64)
The loss above promotes the stability of the embedding distribution from past domains. When combined with the adversarial embedding alignment loss in Eqn. 62, it encourages the embedding distribution of the current domain to match those of the previous domains, but not vice versa.
As supervised variations of contrastive learning [12, 32, 14, 83, 65, 13, 84, 85]. the supervised contrastive loss [42, 29] will compensate for the fact that Eqn. 62 does not utilize the label information to align different domains, and therefore lack in the efficiency of aligning two domains' embedding distribution. The supervised contrastive learning pulls together the embeddings of the same label and pushes apart those with distinct labels. Notably, it is done in "domain-agnostic" manner, i.e., the domain labels are not considered. Denoting as P 1 Σ Dj the combined data distribution of all domains and as P.ly the data distribution given the class label y E [K], the supervised contrastive loss for embedding alignment is defined as follows:
V.(e)= E.,, E,,,, #2)~PP 18 ~P
log expl (. (1 +D expl (2) (.)
eNp S. m2))
(65)
where su, v) = ||e(u) ((v))) 2 is the squared Euclidean distance for any u, V E X.
Introducing the supervised contrastive loss to continual learning is novel, as existing methods often harness its ability to create a compact representation space, thereby mitigating representation overlapping in class incremental learning [65, 30, 9, 96]. However, in this work, the primary motivation for using the supervised contrastive loss to facilitate domain alignment lies in its optimal solution [29, 1 110, 19]. This optimal point referred to as neural collapse [47, 69, 111, 103], where the embeddings of the same class collapse to the same point, and those of different classes are sparsely distributed. is easy to envision that when the optimal state of the supervised constrastive loss attained across different domains, it concurrently achieves perfect domain alignment.
26

---

The loss function that fosters stability and domain alignment in the encoder can be summarized as follows:
= (-.o)))=AVV(dd...3) + Vp X.V)(e).
(66)
Here, two hyper-parameters, Ap and A balance the influence of each individual loss the encoder's embedding distribution. In practice, the final performance of UDIL is not significantly affected by the values of Ap and Xg. and a wide range of values for these parameters can yield reasonable results.
C.4 Evaluation Metrics
In continual learning. many evaluation metrics are based on the Accuracy Matrix R E RTXT, where T represents the total number of tasks (domains). In the accuracy matrix R. the entry R corresponds to the accuracy of the model when evaluated on task j after training on task i. With this definition in mind, we primarily focus the following specific metrics:
Average Accuracy (Avg. Acc.) up until domain represents the average accuracy of the first domains after training these domains. We denote it as Ar and define it as follows:
A 2
Σ R1 1=1
(67)
In most of the continual learning literature, the final average accuracy Ar is usually reported. In our paper, this metric is reported in the column labeled "overall". The average accuracy of a model is a crucial metric it directly corresponds to the primary optimization goal of minimizing the error on all domains, as defined in Eqn. 3.
Additionally, to better illustrate the learning (and forgetting) process of model across multiple domains, we propose the use the Avg. of Avg. Acc." metric Ar,,,, which represents the average of average accuracies for consecutive range of domains starting from domain ti and ending at domain 12. Specifically, we define this metric as follows:
Araita
A
(68)
This metric provides condensed representation of the trend in accuracy variation compared to directly displaying the entire series of average accuracies A2, ...................... ). We report this Avg. of Avg. Acc. metric in all tables (except in Table 2 due to the limit of space).
Average Forgetting (i.e., 'Forgetting" in the main paper) defines the average of the largest drop of accuracy for each domain up till domain t. We denote this metric as F, and define it as follows:
Fe F 1-1 1
(69)
j=1
where ft(j) is the forgetting on domain after the model completes the training on domain t, which defined as:
fe()) 2 max [Ri. R. [E[t-]]
(70)
Typically, the average forgetting is reported after training on the last domain T. Measuring forgetting of great practical significance, especially when two models have similar average accuracies. indicates how a model balances stability and plasticity. If model P achieves a reasonable final average accuracy across different domains but exhibits high forgetting, we can conclude that this model has high plasticity and low stability. quickly adapts to new domains but the expense of performance on past domains. On the other hand, if another model S has similar average accuracy to P but significantly lower average forgetting. we can infer that the model S has high stability and low plasticity. It sacrifices performance on recent domains to maintain reasonable performance on past domains. Hence, to gain a comprehensive understanding of model performance, we focus on evaluating two key metrics: Avg. Acc. and Forgetling. These metrics provide insights into how models balance stability and plasticity and allow us to assess their overall performance across different domains.
27

---

Forward Transfer W quantifies the extent to which learning from past t domains contributes to the performance on the next domain It defined as follows:
W -
Σ 1=2
R 1 - To.
(71)
where r; the accuracy of a randomly initialized model evaluated on domain For domain incremental learning, where the model does not have access to future domain data and does not explicitly optimize for higher Forward Transfer, the results of this metric are rypically random. Therefore, we do not report this metric in the complete tables presented in this section.
C,5 Introduction to Baselines
We compare UDIL with the state-of-the-art continual learning methods that are either specifically designed for domain incremental learning or can be easily adapted to the domain incremental learn- ing setting. Exemplar-fres baselines include online Elastic Weight Consolidation (oEWC) [81], Synaptic Intelligence (SD) [105], and Learning without Forgetting (LwF) [52]. Memory-based do-
main incremental learning baselines include Gradient Episodic Memory (GEM) [58]. Averaged Gradient Episodic Memory (A-GEM) [10], Experience Replay (ER) [75]. Dark Experience Re- play (DER++) [8], and two recent methods, Complementary Learning System based Experience Replay (CLS-ER) [4] and Error Senesitivity Modulation based Experience Replay (ESM-ER) [80]. In addition, we implement the fine-tuning (Fine-tune) [52] and joint-training (Joint) as the perfor- mance lower bound and upper bound (Oracle). Here provide short description of the primary idea of the memory-baa based domain incremental learning baselines.
The baseline method that uses the memory provide additional optimization constraints during learning the current domain. Specifically, the update of the model cannot point towards the direction which the loss of any exemplar increases.
GEM [58]:
A-GEM [10]: The improved baseline method where the constraints of GEM are averaged as one, which shortens the computational time significantly.
ER [75]: The fundamental memory- -baedd domain incremental learning framework where the mini-bater of the memory is regularly replayed with the current domain data.
DER++ [8]: A simple yet effective replay-based method where an additional logits distilla- tion (dubbed "dark experience replay") is applied compared to the vanilla ER.
CLS-ER [4]: A complementary learning system inspired replay method, where two expo- nential moving average models are used to serve as the semantic memory, which provides the logits distillation target during training.
ESM-ER [80]: An improved version of CLS-ER, where the effect of large errors when learning the current domain reduced, dubbed "error sensitivity modulation".
C.6 Training Schemes
Training Process. For each group of experiments, we run three rounds with different seeds and report the mean and standard deviation of the results. We follow the optimal configurations (epochs and learning rate) stated in [8, 80] for the baselines in P-MNIST and R-MNIST dataset. For IID-Balls and Seq-CORe500 we first search for the optimal training configuration for the joint learning, and then grid-search the configuration in small range near it for the baselines listed above. For our UDIL framework, as it involves adversarial training for the domain embedding alignment, we typically need configuration that has larger number of epochs and smaller learning rate. We use simple grid search to achieve the optimal configuration for it as well.
Model Architectures. For the baseline methods and UDIL in the same dataset, we adopt the same backbone neural architectures to ensure fair comparison. In HD--Blll, we adopt the same multi-layer perceptron with the same separation of encoder and decoder as in CIDA [94], where the hidden dimension set to 800. In P-MNIST and R-MNIST, we adopt the same multi-layer perceptron architecture as in DER++ [8] with hidden dimension set to 800 as well. In Seq-CORe50, we use the ResNet18 [33] as our backbone architecture for all the methods, where the layers before the final average pooling are treated the encoder and the remaining part is treated as the predictor
28

---

(a) Joint (91.083%, Oracle)
(b) GEM (76.619%)
(C) A-GEM (39. 634%)
(d) ER (82.253%)
(e) DER++ (79.322%)
CLS-ER (5.844%)
ESM ER 192%)
UDIL (86.872% Ours)
Figure 3: More Results HD-Balls. Data is colored according domain ID. All data is plotted after PCA [6]. (a-h) Accuracy and embeddings learned by Joint (oracle), UDIL, and six baselines with memory. Joint, the oracle, naturally aligns different domains, and UDIL outperforms all baselines in terms of embedding alignment and accuracy.
Hyperparameter Setting. For setting the hyper-paramese embedding alignment strength coeffi- cient As and parameter C' that models the combined effect of VC-dimension d and error tolerance 6, use grid search for each dataset, where the range Xa [0.01, 100) and C' E [0, 1000] are used.
D Additional Empirical Results
This section presents additional empirical results of the UDIL algorithm. Sec. D.1 will show the additional results on different constraints with varying memory sizes. Sec. D.2 provides additional qualitative results: visualization of embedding distributions, showcase the importance of the embedding alignment across domains.
D.1 Empirical Results on Varying Memory Sizes
Here we present additional empirical results validate the effectiveness of our UDIL framework using varying memory sizes. The evaluation is conducted on three real-world datasets, as shown in Table 5, Table and Table By increasing the memory size from 400 800 Table and and from 500 to 1000 in Table 7, we can investigate the impact of having access to a larger pool of past experiences on the continual learning performance, which might occur when the constraint on memory capacity is relaxed. This allows us to study the benefits of more extensive memory in terms of knowledge retention and performance improvement. On the other hand, by further decreasing the memory size to the extreme of 200 in Table 5 and 6, we can explore the consequences of severely limited memory capacity. This scenario simulates situations where memory constraints are extremely tight, and the model can only retain small fraction of past domain data, for example, model deployed on edge devices. To ensure a fair comparison, here we use the same best configuration found in the main body of this work.
The results all three tables demonstrate clear advantage of our UDIL framework when the memory size is limited. In P-MNIST and R-MIST,, when the memory size [M] = 200, the overall performance of UDIL reaches 91.483% and 82. 82.796% respectively, which outperforms the second best model DER++ by 0.757% and a remarkable 6.125%. In Seq-
when the memory size |M] 500 set, UDIL holds a 3.474% lead compared to the second best result. When the memory size is larger, the gap between UDIL and the baseline models is smaller. This is because when the memory constraint is relaxed, all the continual learning models should be least closer to the performance upper bound, i.c., joint learning 'Joint (Oracle)' in the tables, causing the
29

---

indistinguishable results among each other. Apparently, DER++ favors larger memory more than UDIL, while UDIL can still maintain a narrow lead in the large scale dataset Seq-CORe50.
D.2
Visualization of Embedding Spaces
Here we provide more embedding space visualization results for the baselines with the utilization of memory, shown in Fig. 3. As one of the primary objectives of our algorithm, embedding space aligmment across multiple domains naturally follows the pattern shown in the joint learning and therefore leads higher performance.
30

---

Table 5: Performances (%) evaluated on P-MNIST. Average Accuracy (Avg. Acc.) and Forgetting are reported measure the methods' performance. "f" and "4" mean higher and lower numbers better, respectively. We use boldface and underlining to denote the best and the second-best performance, respectively. We use s_" denote "not appliable".
"not
D1.5
Do:10
D11:5
D16:20
Overall
Method
Buffer
Avg. Acc (t)
Avg. Acc
Forgetting (1)
Fine-tune
92.506
$7.088
81.295
72.807
20.102
27.522
oEWC [81]
92.415
87.988
83.098
78.670.0.90
78.476.21223
18. 1.130
SI [105]
92.282+0a66
87.986=11622
83. 83.698=158
79.669#0.70
79.045=1.357
17.409=11446
LwF [52]
95.025=0044
91.402+1.50
83.984=2.10
76.046#2.004
23.445
24.556:22789
GEM [58]
93.3 14.300em 3:73
89.813 13:9111
87.25 #00.524
86.729
9.430
A-GEM [10]
93.3261:0 363
91,466.10 1666610.600 0080
89 12,048 1.05
86.5 1...0.04
85.712.10 12 228
10.485.10.19
ER [75]
94.087=a0 762
92.397 00.4664
$9,999
1060
87.492 4.4444
86.963 303
9.273-+0.25
DER+ [8]
200
94.708 451
94.582+0..0 15.15
93.271 +0.588
90.980
90.333 587
6. 10
CLS-ER [4]
94.761 +40
93,943 19.197
92.725
005566
91.150++0.357
90.726:0021
5.428++0.252
ESM-ER [80]
95.198
94.029
+0.427
91.710 10
88.181 +11221
86.851 250
10.007 =0.964
UDIL (Ours)
95.747 104486
94.695 10.256
93.756 1.0343
92.254
91.483 10.270
4.399 1.0.144
GEM [58]
93.557 225
92.635 306
91.246
89.565 -..34
89.097 =0 149
6.975
A-GEM [10)
93.432
92.064=20.29
90.038 900726
87. 87.988 #40,33
87.560=00.m7
8.577 +0.55
ER [75]
93.525 101
+0,492
90.426=0456
88.728 00.355
88. =0033
7.180+0a29
DER+ [8]
400
94.952 100403
95.089 10.075
94.458 1.0328
93.257 1.0.44
02.95010.30
3.378 1..245
CLS-ER [4]
94.262 6.6649
93. 03.1955.0 10.144
92.623
02.6223...199 1.0195
31.83910.11
91.598 1:0.17
2.795110.14
ESM-ER [80]
95.4133:
94.654.20.31
93.353
91.02 01.222227774
89 899.829=00om 608
6.888810.758
UDIL (Ours)
95.992:10.49
95. 026:10.250
14.2122::0000 280
93. 93.094-10.306
92.666220. 108
2.853
GEM [58]
93.717 17717
13.116.00.06
92.166
1416666.335 L.0335
91.076100342
90.6091.p 10.364
5.39310..17
A-GEM [10]
93.612
92.523 10,375
90.718
88.543 10.391
88.020 a0.51
8,081 100,667
ER [75]
93.827=2087
92,457 20.217
91.688 207277
90.61 0.0289
20..252:0000
5. 2.18880.005
DER+ [8]
800
95.295
95,539 10,041
95.099 1.0187
94,423
94.227 0.266
2. 16610.16)
CLS-ER [4]
94.463=a557
93.567 +0,093
93. 13.18220.13 137
92.744
92.578=0.155
2.803++0.188
ESM-ER [80]
95.567=a 150
95.1 26
94,301 +:0
92.981 +0.397
=0.4088
4.1 70+0.377
UDIL (Ours)
06.312
95.207 10.196
94.64 60.14
93,997 10.194
93.724
1.633-14.035
Joint (Oracle)
96.368
31

---

Table 6: Performances (%) evaluated R-MNIST. Average Accuracy (Avg. Acc.) and Forgetting are reported to measure the methods' performance. "f" and "4" mean higher and lower numbers better, respectively. We use boldface and underlining to denote the best and the second-best performance, respectively. We use "." to denote "not appliable".
Dr.5
Do.00
D11.15
Di6:2
Overall
Method
Buffer
Avg. Acc
Avg. Ace
Forgetting
Fine-tune
92.66
76.617=3011
60.21223688
49.793=1.552
52.281 1..
oEWC [81]
91.765+22.26
76.226#7..22
60.320=1wz
50.505
48.203 40.277
51.181
[1[1
91.867 12.272
76.801 17.391
60,95613350
50.301
48,251 L11388
51.053
LwF [52]
95.174.1.154
83.044.55.35
65. 14.009
$5.980.1.296
54.709.1.50
45.473
GEM [58]
93.441 10,610
88.620
21.034=2270a
13.112+11922
70.545 #0,023
27. 27.684:40.445
A-GEM [10]
92.667 11,352
82.772+5.50
70.579
60.462+2200
57.958 9..955
=0.969
ER [75]
94,705 10.790
89.171 128883
79 922966...33 132265
71.787 1.1.80
69.627 100911
28, 74,7499... 10,999
DER+ [8]
200
94.904-20414
1186
241115
18.373=+1.244
76.67 +00390
20.743
CLS-ER [4]
95.131 50.53
(2.421 1:.7
24.773=22665 666
77.735 A80
75.609
20.418
22.483 49.46
ESM-ER [80]
95.378+0.551
90.800+22522
83 =3.448
76.987#1.219
75.203 14.143
23.564440.157
UDIL (Ours)
95,097
93.101
89.194
84.704.11.122
82.796 117796.1.880
12.971
GEM [58]
93.842
90.663
85.392 886
79.06 1.558
76.619 200.5
21.2899:0.579
A-GEM [10]
12.72
83.5644
72.61 3305
22.223++2.01
a0.154
39.196:40.71
ER [75]
94.916=11457
91.491 1:7778
86.029
78.688 1.3.6 1.323
+6.794
20.696
DER+ [8]
400
95.246.0..2
11.627 1.147
90.01 1
10.90110.982
84,258 10.544
13.692.0.500
CLS-ERR [4]
95.233
92.7740 1.266
89. 1:330
83.678::11..
81.77
25.354
15.455
ESM-ER [80]
95.825
93.378 1.40
89 10 604
83
1618211.66
82, 192.992.... 164
19.
UDIL (Ours)
95.274+40400
94.04
91.511 =4.990
=7.809
86.635 +0.65
8.506=1.181
GEM [58]
94.212.10.22
92.482.11.125
89.191 1.1.346
84.866.11.317
82.772.1..0
14.781
A-GEM [10]
92.902 194
84.61 1 +44451
=5.150
64.55
(0:244777
.236 61.240
37.528::1.009
ER [75]
95. 0.2221
12.997
89.319 1.365
14.8
81.877 1557
+5.285
DER+ [8]
800
95.496=1.261
94.960=0.568
936013
90.820=:16687
89.746=#0.85
7.82 +0,371
CLS-ER [4]
95, 4........144
93.927 10.881 1..881
91.275
109900
87.816.10 1...66
86.418 1.0.15
10.598 10,228
ESM-FR [80]
96.086=20361
94.7-4622915
209774
89.745
88.66222t0266
9.409:2025
UDIL (Ours)
95.354-10.50
94.7 11 20.563
92.776=0005
90.399
89. 191 20.685
1:1304
Joint (Oracle)
97.15010.006
32

---

Table 7: Performances (%) evaluated on Seq-CORe50. Avg. Acc. and Forgetting are reported to measure the methods' performance. "f" and "4" mean higher and lower numbers better, respectively. We use boldface and underlining to denote the best and the second-bos best performance, respectively. We use "." to denote "not appliable" and "*" to denote out-of- oof-mmory (OOM) error when running the experiments.
D3.3
D4.
D7.9
D10:11
Overall
Method
Buffer
Avg. Ace
Avg. Acc
Forgetting
Fine-tune
73.707=13.44
34.551
29.406+1.59
28.689+3.14
31.832+1054
73.296
oEWC [81]
74.56711133900
35.91510.200
30.17413.195
28.291 12.522
30.813
14.963
[105]
114161
34.345+1.00
30.127+2.97
28.83993.631
1.34
13.14441.588
LwF [52]
80.383::0.190
28.357
31.386
28.711
31.692.0.768
12.990
GEM [58]
79.852+6.664
#1.761
39,258+2.114
+6.859+0.842
37.701 #0.275
22.724
A-GEM [10]
80.348 894
41.472+3.994
43.213-t1.542
39.181 19.111 +:3.99
43.181 #2.025
33.775
ER [75]
90.8 177
79.343
68.151 1.0,226
11.034
66.605
1:0.14
32.750
DER+ [8]
1..444
18.652.11.84
80.391 10.107
28.038
78.62920788
11.910.1.004
CLS-ER
19.83411.33 1.333
1..70
70.591 110.322
ESM-ER [80]
84.905+6477
5 51.905+33225
13.
27.
52.751
25.444440.5500
UDIL (Ours)
98.152
89.81411.302
83.052
81.547
82.103
19.589
GEM [58]
78.717 444811
43.269++3.199
40.908+2.200
40.408 4..1+...66
41.576::1.59
18.5337+1.225
A-GEM [10]
78.9 78.917 984
41.172++4.99
44.576xt1.701
960
12.627+ 1.059
33. 1.
[75]
90.048
14.66811.95s
77.561 1.2281
72.26811.722
72.988110566
25,997 100094
DER+ [8]
1000
89.5 10
92.492+0.mm
88. 79.
108
86,39224.714
CLS-ER
92.004-10.894
85,0 1::044:1:276
ESM-ER [80]
25.4204433
54, 54.855245511
61.714+10 1.400
58.932+0.u99
20. +0.6644
UDIL (Ours)
98.648 1..64811174
93.447.11.1
90.545
87.923 11.2231...23
88.155
12.882.10.40
(Oracle)
99.137
33


Reviews:
[{"review_id": "XERZxMNwZE", "reviewer": "Reviewer_Ps1S", "summary": "This submission investigates domain incremental learning problem, where the model is adapted to a sequence of new tasks which have different distributions. Starting from some (domain related) generalization bound theory, the combination of three different bounds could be used to address domain incremental learning, and meanwhile several existing methods could be unified in the combined bound. An objective to find the efficient combination weight is also introduced. Experiments on both toy dataset and real dataset show the efficacy of the method. ", "questions": "As the generalization bounds considered in the submission are actually related to domain adaptive task (from [3]), I am curious about whether the method could be extended to the more practical scenario, unsupervised domain incremental learning, where the new domain only contains unlabeled samples. I think there is possibility to extend the work, as some unsupervised domain adaptation methods are directly utilizing the $d_{\\mathcal{H}\\Delta\\mathcal{H}}$, such as MCD[a], though in the current setting there are only limited source samples.\n\n\n\n*reference*\n\n[a] Saito K, Watanabe K, Ushiku Y, et al. Maximum classifier discrepancy for unsupervised domain adaptation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3723-3732.", "limitations": "None potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear, it is reasonable to combine the three different bounds to deal with mode general case.\n\n- It is good to derive a decent method from theoretic perspective for domain incremental learning task. And it is creditable that some hyperparameters such as $C$ could be also estimated inside the bound.\n\n- It is admirable that the resulting bound could include several existing methods, which connects the intuitive methods and the corresponding theory.\n\n", "weaknesses": "Overall the submission is a good paper, and I have no obvious concerns about the submission, while I simply hope maybe the authors could add more narrations in the main paper, on the related works of domain incremental learning, as there are relatively fewer existing works in this topic. And I am also interested in the effectiveness on the huge dataset, such as DomainNet which has relatively larger domain gap and more categories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive and encouraging comments as well as the insightful questions. We are glad that you find our motivation ``\"clear\"``/``\"reasonable\"``,  our method ``\"decent\"``/``\"creditable\"``, our theoretical analysis ``\"admirable\"``, and our submission ``\"a good paper\"`` with ``\"no obvious concerns\"``. Below we address your questions one by one. \n\n**Q1. Maybe the authors could add more narrations in the main paper about the related work of DIL?**\n\nThis is a good suggestion. In the revised version of our paper, we will follow your suggestion to include a more detailed introduction to the field of Domain-Incremental Learning (DIL). We observe that while DIL is one of the three main categories in continual learning, there is a lack of systematic study specifically focused on this topic, in comparison to Task-Incremental Learning (TIL) and Class-Incremental Learning (CIL). We hope our work could contribute to the understanding and development of DIL methods and address the challenges in this exciting but underexplored area. \n\n**Q2. I am curious about whether the method could be extended to the more practical scenario, unsupervised domain incremental learning?**\n\nThanks for this insightful question. Indeed it would be very interesting to extend our UDIL framework to unsupervised domain incremental learning. We expect that the availability of the next domain's (unlabeled) data would offer more legitimacy for aligning domains' feature distributions (similar to MCD (cited as [44] in our paper) in the unsupervised domain adaptation setting). For now, our work primarily focuses on the standard DIL setting with memory, providing a unified perspective that incorporates previously existing DIL techniques. We hope that this contribution will lay a solid foundation for future research, including the exploration of the unsupervised domain incremental learning idea you proposed. \n\n\n**Q3. I am also interested in the effectiveness on the huge dataset, such as DomainNet which has relatively larger domain gap and more categories.**\n\nPlease refer to **Q2 of the General Response** above. "}, {"review_id": "s904EmXIUQ", "reviewer": "Reviewer_jN5u", "summary": "In this paper, the author proposed a unified theoretical framework in order to reveal how the tasks in the domain incremental learning are related. Moreover, the framework unifies various existing methods from the perspective of domain generalization bound and it showed that the unified generalization bound can be organized in the format of the combination of data, memory bank, and history model. Meanwhile, the author showed that the fixed coefficient of the above combination is always less effective compared to the adaptive ones. As a result, the author proposed an adaptive adjusting coefficients method to tightly bound the generalization upper bound. Extensive experiments also showed the effectiveness of the unified framework.", "questions": "1. The author should emphasize what the ghost example in supplementary material is ( if it is some kind of subset of the previous domain? or some stored replay examples ), otherwise the transition from Equation 19 to 20 is not smooth, e.g. it provided a loose scale-up. \\\\\n2. From Theorem 3.4, we can find that the first bounding term is actually a \nsum up of loss from the first model to the current one (ERM), so is that appropriate to store the model at every training stage in the sample replay setting, and is the history model froze during the training? Meanwhile, will that cause a lot of additional memory burden? \\\\\n3. The intra-domain and cross-domain constraint is actually an every step alignment with the history learned models, thus as long as the domain discrepancy is relatively large from one to another, the $h$ would have a high probability of severe change, making the second term of Equation 8 in the left large, would that be a problem since it seems no explicit parameter regularization? \n4. Also, since in Equation 8, every step of updates would come from a bottom-up calculation from the first domain to the current one, it seems the calculation would be expensive as the growth of the domain, so I have concerns over the calculation burden. Also, as a consequence, would the method can be implemented in some large-scale real-world datasets which used in domain incremental learning, e.g. DomainNet?\n5. The Theorem 3.4 seems like a combination loss over all domains, thus, I have a little concern over the novelty of this Theorem. ( Considering it is the main Theorem).", "limitations": " yes, the authors adequately addressed the limitations yet ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized and the mathematical proofs are sound. \\\\\n2. The author combined theoretical analysis of domain adaptation to deal with the domain incremental problem, which is impressive. \\\\\n3. The proposed unified framework can explain a variety of previous works, which is very powerful. \\\\", "weaknesses": "see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive and encouraging comments as well as the insightful questions. We are glad that you find our unified framework ``\"powerful\"`` and able to ``\"explain a variety of previous works\"``, our theory ``\"sound\"``/``\"impressive\"``, and our paper ``\"well-organized\"``. Below we address your questions one by one. **We will include the following discussions in our revision as suggested.**\n\n**Q1. ...what the ghost example in supplementary material is ...**\n\nThe term \"ghost sample\" (or \"ghost data\") refers to a second dataset that is sampled from the same data distribution, but does not actually exist; rather, it serves solely as a tool for analysis. This is a *standard proving technique* [1, 2] for analyzing generalization bound. The reason for introducing ghost data is that measuring the true error $\\hat{\\epsilon}_{\\alpha}(h)$ requires an infinite number of samples, making it challenging to apply the concept of VC-dimension for analysis. Therefore, in the inequality (Eq. 20), we follow the literature [1, 2] to introduce \"ghost data\" to facilitate analysis of the generalization bound. \n\n**Q2. ...is that appropriate to store the model at every training stage in the sample replay setting, and is the history model froze during the training? ...will that cause a lot of additional memory burden?**\n\nThese are good questions. Actually we *only* need to store *one* model, i.e., the model trained after domain $t-1$. Note that the history model $H_{t-1}$ in Theorem 3.4 is *not* a collection of past models trained from the first domain to the most recent one, but rather the model trained after domain $t-1$, which has a constant memory consumption. As a result, when we calculate and optimize the first term in Theorem 3.4, we compute the discrepancy between the current model $h$ and the history model $H_{t-1}$ evaluated over different domains, requiring only one forward pass and backward pass. Therefore, memory cost is not a concern, making our UDIL scalable. \n\nYes, the history model is frozen during the training. \n\n**Q3. ...as long as the domain discrepancy is relatively large from one to another, the $h$ would have a high probability of severe change, making the second term of Equation 8 in the left large, would that be a problem since it seems no explicit parameter regularization?**\n\nThis is a good question. To avoid this problem, there is actually some implicit measure imposed by the *optimizable* term $\\frac{1}{2}d\\_{\\mathcal{H}\\Delta\\mathcal{H}}(\\mathcal{D}\\_i, \\mathcal{D}\\_t)$ (note that this term is not a constant; instead it can be optimized in Eqn. 13) in Eqn. 7 (and Eqn. 8). \n\n**(1)** To see this, assuming $h$ consists of an encoder $e$ and a predictor $p$, minimizing this $d\\_{\\mathcal{H}\\Delta\\mathcal{H}}$ term (i.e., the third term of UDIL's objective function in Eqn. 8 or Eqn. 13) is equivalent to aligning previous domains $\\mathcal{D}\\_i$ and the current domain $\\mathcal{D}\\_t$ in the encoding space; this effectively regularizes the encoder $e$ of the model $h$ and minimizes the domain discrepancy in the encoding space, such that $h$ would not change severely. \n\n**(2)** Furthermore, our adaptively updated coefficient $\\Omega$ is inherently designed for addressing this issue: treating Eqn. 8 as the objective function to minimize, if the discrepancy between two domains $d\\_{\\mathcal{H}\\Delta\\mathcal{H}}(\\mathcal{D}\\_i, \\mathcal{D}\\_t)$  is too large for the adversarial training to align their features, the corresponding coefficient $\\beta_i$ in the third term of Eqn. 8, $\\beta\\_i \\cdot d\\_{\\mathcal{H}\\Delta\\mathcal{H}}(\\mathcal{D}\\_{i}, \\mathcal{D}\\_{t})$, will be adapted to a very small value. Since $\\beta\\_i$ also controls the second term of Eqn. 8, $\\beta\\_i \\cdot \\hat{\\epsilon}_{\\mathcal{D}\\_t}(h, H\\_{t-1})$, smaller $\\beta\\_i$ will \"de-emphasize\" the cross-domain loss term $\\hat{\\epsilon}\\_{\\mathcal{D}\\_t}(h, H\\_{t-1})$, thereby minimizing the negative effect of optimizing this term. \n\n**Q4. ...calculation...expensive as the growth of the domain...large-scale realistic dataset...DomainNet...** \n\nPlease refer to **Q1 & Q2 of the General Response** above. \n\n**Q5. ...Theorem 3.4 seems like a combination loss over all domains...the novelty of this Theorem.**\n\nWe are sorry for the confusion. We should have better highlighted our novelty in the main paper. While Theorem 3.4 presents a combination of losses over all domains, it has novelty that can be outlined in three aspects: \n\n(i) Theorem 3.4 introduces the first unified theoretical framework that provides a generalization bound encompassing various existing practical techniques (intra-domain and cross-domain distillation loss) in Domain Incremental Learning (DIL). This comprehensive approach, as illustrated in Table 1 of the paper, sets it apart from prior work.\n\n(ii) Another key innovation of Theorem 3.4 is the provision for automatically learning adaptive coefficients for different loss terms. This capability plays a pivotal role in achieving the tightest generalization bound and ultimately enhancing the model's performance.\n\n(iii) Theorem 3.4 systematically decomposes the joint-training objective of the general DIL problem into an Empirical Risk Minimization (ERM) term, an intra-domain term, and a cross-domain term, taking into account the domain discrepancy. This offers valuable insights for more inclusive (encompassing more terms into the bound) and stronger (developing a tighter bound) frameworks in the future. \n\nReviewer 4VUe also commends the novelty of our UDIL, acknowledging it as a ``\"significant\"`` contribution and ``\"the first comprehensive analysis of generalization bounds in domain incremental learning while also categorizing SOTA methods in the area according to different views of these bounds\"``. \n\n**References**\n\n[1] Abu-Mostafa, et al. *Learning from data*. AMLBook, 2012.\n\n[2] Mohri, et al. *Foundations of machine learning*. MIT press, 2018."}, {"review_id": "IMcL9rkxTQ", "reviewer": "Reviewer_4VUe", "summary": "This paper proposes a framework for domain incremental learning that unifies ideas from previous methods in this area. A theoretical analysis is first performed to formally state the problem of domain incremental learning. The theoretical formalization identifies three ways to bound the risk of error on a particular domain: 1) by a naive ERM-based bound, 2) by an intra-domain model-based bound, 3) by a cross-domain model-based bound. Based on the identified limitations of these three bound, the authors proposed a unified view of the generalization bound for domains observed in incremental learning. They propose to integrate these three bounds using coefficients that vary depending on the domains at hand. Existing methods in domain incremental learning are then categorized according to the components of the unified framework. In the second part of the paper, the authors propose a method where the coefficients are both learnable and dynamic. Experimental results on both toy data and benchmark datasets, along with comparison to SOTA methods in the area, are provided.\n\n---Rebuttal---\nI read the authors rebuttal. My review was already positive, so I left my score unchanged.", "questions": "Please refer to the \"weaknesses\" section for questions.", "limitations": "Limitations are properly addressed thorough the paper. There are not potential negative societal impacts of this work that need to be stated explicitly.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Regarding originality, the paper presents originality both at the theoretical and experimental levels. Regarding theoretical originality, to the best of my knowledge the paper presents the first comprehensive analysis of generalization bounds in domain incremental learning while also categorizing SOTA methods in the area according to different views of these bounds. Regarding experimental originality, the paper presents a method that is dynamic and adaptable to relevant aspects such as the memory size and the domain relatedness in domain incremental learning. \n\n- Regarding significance, the paper provides new insights on domain incremental learning through rigorous and extensive theoretical analysis. This is of high relevance for researchers in this subfield. \n\n- Regarding quality and clarity, the paper is very well written and it is easy to follow. Extensive details are provided both in the paper and in the supplementary material.", "weaknesses": "- From the experimental results presented in Tables 2 and 3, the proposed method seems to be competitive but not significantly better than counterparts (especially DER++). \n\n- Details on the computational complexity/cost of the method are not provided in the formulation of the method or in the experiments. Considering that the method needs to learn the coefficients, and therefore the problem in Eq. 13 is more difficult than the optimization problem in counterpart methods, what would be the extra-cost in terms of computation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive and encouraging comments as well as the insightful questions. We are glad that you find our work ``\"original\"`` at both ``\"theoretical and experimental levels\"``, our paper ``\"very well-written and easy to follow\"``, and acknowledge that our UDIL is ``\"the first comprehensive analysis of generalization bounds in domain incremental learning\"``. Below we address your questions one by one. \n\n\n**Q1. In Table 2 and 3, the proposed method seems to be competitive but not significantly better than counterparts (especially DER++).**\n\nThis is an important question. For datasets Rotating-MNIST and Seq-CORe50, where different domains' data exhibits a more natural distributional shift, UDIL shines with notable improvements of 2 and 4 absolute accuracy points over the second best model, DER++. These improvements are significant, particularly considering the number of domains (20 in Rotating-MNIST) and the number of classes (50 per domain in Seq-CORe50) taken into account.\n\nFor the toy dataset HD-Balls, our UDIL demonstrates superior performance, *outperforming DER++ by up to 7 absolute accuracy points* and surpassing the second best method, CLS-ER, by 1 absolute accuracy point. We suspect the smaller (but *still statistically significant*) margin over CLS-ER is due to the relatively simple data distribution of HD-Balls. \n\nFor the three datasets above, we ran the corresponding hypothesis tests, and the $p$-values are in the range of $(1.23\\times 10^{-3}, 5.24\\times 10^{-3})$, much lower than the threshold of $0.05$, and therefore verifies the significance of our UDIL's performance improvement. \n\nFor Permutation-MNIST, our UDIL's performance is very close to that of the best baseline, DER++ (92.666% versus 92.950%). Notably, the joint-training accuracy upper-bound, i.e., oracle, on this dataset only reaches an accuracy of 96.368%, *leaving little room for significant improvement*. Consequently, our model does not exhibit a dominantly superior performance on this dataset. We also did the hypothesis test for this dataset. The $p$-value of DER++ better than UDIL is $0.152$, which is much larger than the threshold $0.05$, therefore rejecting that the DER++ is significantly superior, and showing that UDIL is comparable to DER++ even on this dataset. \n\nOverall, the experiments demonstrate the efficacy of UDIL, particularly in scenarios where distributional shifts are more pronounced, highlighting its potential as a strong candidate for handling domain incremental learning challenges. \n\n\n**Q2. Details on the computational complexity/cost of the method are not provided in the formulation of the method or in the experiments. ...needs to learn the coefficients...Eq. 13 is more difficult than the optimization problem in counterpart methods, what would be the extra-cost in terms of computation?**\n\nThank you for the suggestion. The overhead of optimizing the coefficients is *minimal* since it is a small Linear Programming (LP) problem, which involves only $3(t-1)$ variables where $t$ is the number of domains. For example, our Seq-CORe50 dataset has 11 domains, which leads to an LP problem with *only 30 variables*. Solving this LP problem usually takes less than a second; therefore the introduced overhead is negligible. \n\nIt is worth noting that when optimizing (learning) the coefficients in Eqn. 13, the 0-1 loss does not involve any gradient calculation on the model $h$; all loss terms associated with the coefficients are treated as constants. This is why such optimization can be treated as a simple LP problem. \n\nWe also provide a more detailed discussion on our UDIL's computation complexity in **Q1 of the General Response** above, and will include a similar discussion in the our revision as suggested. "}]

Author rebuttal:
# General Response 

We thank all the reviewers for their valuable and constructive comments. We are glad they find the problem we address ``"original"``/``"of high relevance"`` (4VUe, Ps1S), our proposed unified framework ``"admirable"``/``"reasonable"``/``"powerful"``/``"original"`` (Ps1S, jN5u, 4VUe), our theoretical analysis ``"impressive"``/``"sound"``/``"rigorous"`` (Ps1S, jN5u, 4VUe), our paper ``"clear"``/``"well-organized"``/``"well written"`` (Ps1S, jN5u, 4VUe), and acknowledge that our experiments show the ``"efficacy"``/``"effectiveness"`` of our method (Ps1S, jN5u) and that our UDIL is ``"the first comprehensive analysis of generalization bounds in domain incremental learning"``. Below we address reviewers' two common questions. 

**Q1. \[UDIL's Computational Complexity?\]** *"...in Equation 8, every step of updates would come from a bottom-up calculation from the first domain to the current one, it seems the calculation would be expensive as the growth of the domain...the calculation burden..."* **(jN5u)**; *"Details on the computational complexity/cost of the method are not provided in the formulation of the method or in the experiments. ...Eq. 13 is more difficult than the optimization problem in counterpart methods, what would be the extra-cost in terms of computation?"* **(4VUe)** 

This is a good question. In fact, each step of update does not grow with the number of domains $t$, which makes our algorithm scalable. 

Note that in general, UDIL's memory bank has a **constant** number of data points; as the number of domain increases, the number of data points for each domain decreases (refer to Appendix C.2). 
Therefore, while one needs to calculate the loss from the first domain to the current one, such calculation takes constant time and **does not grow with the number of domains**. 

Below, we provide a more detailed analysis of UDIL's computational complexity. As shown in Eqn. 13, our framework calculates three loss functions (stopping-gradient operation omitted for clarity):

$
V_l(h, \Omega) + V_{\text{0-1}}(h, \Omega) + V_d(d, e, \Omega).
$

The computational cost of optimization of the first term $V_l$ is the same as our replay-based baselines such as vanilla experience replay (ER) and dark experience replay (DER++); this term estimates and optimizes the classification loss with respect to the model parameters. 

Therefore, UDIL's computational overhead comes from the last two terms, $V_{\text{0-1}}(h, \Omega)$ and $V_d(d, e, \Omega)$:

**(1)** $V_{\text{0-1}}(h, \Omega)$ optimizes the coefficients to find the tightest upper bound for UDIL. Its 0-1 loss does not involve any gradient calculation on the model $h$, i.e., all loss terms associated with the coefficients are treated as constants; therefore the **overhead of optimizing the coefficients is minimal** since it is a **small Linear Programming problem** (only $3(t-1)$ variables where $t$ is the number of domains). For example, our Seq-CORe50 dataset has 11 domains, which leads to an LP problem with *only 30 variables*. Solving this LP problem usually takes *less than a second*; therefore the introduced overhead is *negligible*. 

**(2)** $V_d(d, e, \Omega)$ involves training a small multiclass domain classifier $d$ on top of the encoder $e$. Its computational cost is almost the same as $V_l(h, \Omega)$. Consequently, the computational complexity of our proposed framework *remains in the same order as the baselines such as ER and DER++*. 

The ``"bottom-up calculation from the first domain to the current one"``, as mentioned by Reviewer jN5u, is achieved by concatenating the mini-batches of data from different domain, and can be effectively calculated without any additiional overhead. 

In summary, we can see that our UDIL's computational complexity is in the same order as the strongest baselines such as ER and DER++; therefore UDIL is scalable to large datasets. 

**Q2. \[UDIL's Application to Large-Scale Datasets?\]** *"I am also interested in the effectiveness on the huge dataset, such as DomainNet which has relatively larger domain gap and more categories"* **(Ps1S)**; *"would the method can be implemented in some large-scale real-world datasets which used in domain incremental learning, e.g. DomainNet?"* **(jN5u)**.

Following your suggestion, we ran additional experiments on the DomainNet dataset. Considering the constraints of time and computational resources, we compared our UDIL with only DER++ in these additional experiments, as DER++ is our strongest baseline across four datasets. We also report the performance of the fine-tuning baseline and joint-training oracle. Due to the constraint of computational resources, we select the ResNet-18 as our backbone network. We grid-searched the learning rate in the range of (1e-4, 5e-4, 1e-3, 5e-3, 1e-2) with Adam optimizer for all four methods, and we inherit the rest of hyper-parameters of UDIL from the Seq-CORe50 dataset. The following table shows the average accuracy (Avg. Acc.) and forgetting rates of UDIL and two baseline models Fine-Tune and DER++, as well as the oracle model Joint-Training. 

| **Model** | **Avg. Acc. (↑)** | **Forgetting (↓)** |
|:-----------:|:--------------------:|:---------------:|
| Fine-Tune | 19.675 | 24.971 |
| DER++ | 22.996 | 17.043 | 
| UDIL (Ours) | **24.232** | **14.171** | 
| Joint-Training (Oracle) | 38.582 | - |
| | |

As shown in the table, there is a noticeable and significant performance boost (considering DomainNet is a 347-way classification problem) when applying UDIL, which further validates the effectiveness of our proposed methods on the large-scale realistic dataset. We will conduct more systematic experiments on the DomainNet dataset in the revised version of our work, including a more thorough search over the hyper-parameters and experiments on other baseline models.

Meta-review: The paper has been positively evaluated by three reviewers (5,6,7 scores). The provided rebuttal and discussion did not result in further changes in the score. The reviewers agree that the paper offers new insights on domain incremental learning, and that the theoretical analysis is of interest to the community. The AC recommends acceptance of the paper. The authors should incorporate the additional results and discussion on computational complexity in the final version.
Decision: Accept (poster)