Lifelong Domain Adaptation via Consolidated Internal Distribution
Authors: Mohammad Rostami
Venue: NeurIPS 2021 Poster (2021)
Keywords: lifelong machine learning
continual learning
unsupervised domain adaptation
internal distribution

Abstract: We develop an algorithm to address unsupervised domain adaptation (UDA) in continual learning (CL) settings.  The goal is to update a model continually to learn distributional shifts across sequentially arriving tasks with unlabeled data while retaining the knowledge about the past learned tasks. Existing  UDA  algorithms address the challenge of domain shift, but they require simultaneous access to the datasets of the source and the target domains. On the other hand, existing works on CL can handle tasks with labeled data.  Our solution is based on consolidating the learned internal distribution for improved model generalization on new domains and benefitting from experience replay to overcome catastrophic forgetting.

Lifelong Domain Adaptation via Consolidated Internal Distribution
Mohammad Rostami USC Information Sciences Institute Los Angeles, CA 90007 rostam i mêusc. edu
Abstract
We develop an algorithm to address unsupervised domain adaptation (UDA) in continual learning (CL) settings. The goal is to update a model continually to learn distributional shifts across sequentially arriving tasks with unlabeled data while retaining the knowledge about the past learned tasks. Existing UDA algorithms address the challenge of domain shift, but they require simultancous access to the datasets of the source and the target domains. On the other hand, existing works CI. can handle tasks with labeled data. Our solution is based on consolidating the learned internal distribution for improved model generalization on new domains and benefiting from experience replay to overcome catastrophic forgetting.
1 Introduction
Deep neural networks relax the need to for manual feature engineering by learning to generate discriminative features in an end-to-end blind training procedure 212. Despite significant advances in deep learning, however, robust generalization of deep neural network on unseen data is still a primary challenge when domain shift exists between the training and the testing data [3] Domain
shift is natural challenge in continual learning (CL) [5 6 where the goal is to learn adaptively and autonomously when the underlying input distribution drifts over extended time periods. Distributional shifts over time usually lead to performance degradation of trained models which turn necessitates model retraining to acquire knowledge about new distributions. Current CL algorithms mainly consider tasks, i.e., domains, with fully labeled datasets. Hence, these algorithms require annotating massive training datasets for observed domains. Persistent manual data annotation. however, is practically prohibitive because of being an economically costly and time-consuning process Too relax this constraint, our goal is to develop algorithm for continual adaptation of model for tackling the challenge of domain shift in a CL. setting using solely unannotated datasets.
Unsupervised Domain adaptation (UDA) is highly relevant learning setting to problem of interest. The goal in UDA to train a model for a target domain with unannotated data by transferring knowledge from a related source domain in which annotated data is accessible 3 A primary group of UDA algorithms map the training data points for both domains into a shared latent embedding space and align the distributions of the source and the target domains in that space (8). Hence, source-trained classifier that receives its input from the shared embedding space would generalize on the target domain well. The domain alignment procedure has been implemented either using 12 13 or by directly minimizing the distance between the Existing UDA algorithms are not suitable for continual learning because the underlying model can be trained if datasets from both domains are accessible. Moreover, these methods usually consider only single target domain and a single source domain. Finally, simply updating the underlying model to generalize in the current encountered domain is not sufficient. Because upon updating the model, the network likely would forget the past learned domains as the result of retroactive interference, referred as the phenomenon of catastrophic forgetring [19 20 in the
generative adversarial learning 9 10
two distributions [14]
15
16
17
18
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

CL literature. Consequently, we also need to tackle catastrophic forgetting in our unexplored learning setting. Our method can be considered as an improvement over existing UDA and CL methods.
Contributions: we develop an algorithm for lifelong unsupervised adaptation of a model on new domains using solely unannotated data. Our idea is based on consolidating the internally learned distribution that encodes the learned knowledge by the model when the initial source domain is learned. We use this multimodal distribution to update the model such that the learned distributions for all subsequent unannotated domains are coupled. To overcome catastrophic forgetting, we store important representative samples for all tasks and replay them back when the model updated. We provide a theoretical analysis to demonstrate that our method mitigates catastrophic forgetting and also leads to improved generalization. We validate our method using standard UDA benchmarks.
2 Background and Related Work
Lifelong UDA setting lies on the intersection of both UDA and CL learning settings.
Unsupervised domain adaptation: an effective approach for domain alignment in UDA is to use a probability distance measure distributional discrepancy and then train an encoder to minimize
the cross-domain distance at its output as a latent shared embedding space [21 22 23 24 23 26).
The Wasserstein distance (WD) [27] is a suitable metric for this purpose due possessing non- vanishing gradients. This property is beneficial because deep learning optimization problems are primarily solved using gradient- based optimization techniques [23]. In this work, we rely on the sliced Wasserstein distance (SWD) [24] variant of the Wasserstein distance because it can be computed efficiently using a closed form solution via empirical samples of the data distribution.
Continual learning: Existing CL methods primarily use either model regularization or experience replay to tackle catastrophic forgetting. The idea of model regularization (20 5 is to identify the network weights that contribute significantly in retaining knowledge about a past learned task and then consolidate them when the model is updated to learn the subsequent tasks. Alternatively, the model can be expanded progressively to learn the new tasks using added weights (28). We rely on experience replay which is implemented using pseudo-rehearsal [19]. The idea is to identify important training data points that contribute significantly to learning task and store them in memory buffer [29]. These samples then would be replayed along with the current task's data to represent the past learned distributions 30 B1 to retain the acquire knowledge about the past tasks. The major challenge is to identify the important data points. We rely on the consolidated internal distribution to identify the important data points. To mitigate the sample selection challenge, some CL algorithms rely on generative experience replay, where the memory buffer is replaced with the ability of generating
synthetic pseudo-samples that are similar to the samples of the past learned tasks [3 32 33 34)
Domain Adaptation in Continual Learning Settings: Existing works in this direction are highly limited. Wulfmeier et al. (35) study addressing gradual shifts in changing environments. Bobu et al. (36) and Wu et al. [37 explore addressing the problem of domain shift in continual learning settings but both works assume that datasets for all learned tasks are observable at each time-step. This assumption limits practicality of these works. Porav al. (38 explore a setting similar to our setting but the proposed method relies on image translation which makes the work highly domain specific. In our work, we relax these limitations and develop more general algorithm.
3 Problem Statement
Consider a classification problem in source domain S with labeled training dataset Ds = (Xo, Yo). where X,, = [xq. =&] X c d Yo = [vf..... g&1 E y C R and the training data points are drawn independently from an unknown source distribution, i.e., x. ~ po(z)) Given a deep neural network fo with learnble weights 9, we can solve for an optimal model using the standard empirical risk minimization (ERM): 0, = arg ming E, ((fe(a)), y?), where L(-) is a suitable loss function. If the training dataset is large enough and the network structure is complex enough, the ERM optimal model will generalize well on unseen data points, drawn from po(x) (3). The challenge in CL is that the input distribution may be non-stationary. Hence, the testing samples may be drawn from drifted distributions. The resulting distributional gap will lead to poor generalization during the testing time. Our goal is to update the source- -rained model fa, continually using solely unlabeled data to avoid poor generalization without forgetting past experiences. To model this process,

---

we consider a set of sequentially arriving target domains T t = 1... .T, with unlabeled datasets Dj (X,), where X, E I M,, o ~ (x), and vt,tz Pt, * Pt, (see Figure I. Since these domains are unlabeled using ERM is implausible. As stated, common UDA methods cannot address CL settings.
To address challenges of "domain shift" and "catastrophic forgetting", we consider that the base network f-(-) can be decomposed into deep encoder Φv(-) X - IR and a classifier subnetwork (-) Z + y/ i.e., fo = two o Φv. where 0 = (w, v). Our method is based on consolidating the internal distribution that is formed in the embedding space 2. We assume that as the result of initial training on the source domain, the source classes become separable in 2. If each time-steps, we update the model such that the internal distribution remains stable, i.e., the distance between the distributions @(Po (x))) and (x))))) is minimized, then the model continues to generalize well on the target domains, despite initially being trained with the source domain labeled dataset. This strategy has been used extensively by the existing UDA algorithms but we are constrained with the accessibility of Dss in CL. i.e., the term (Pp((=))) cannot be computed.
4 Proposed Solution
t-2
Our solution is based on consolidating the in- termediate distribution that is learned in the dis- criminative embedding space to retain model generalizability. Upon learning the initial source domain, the encoder is trained to map the input source distribution into a multi-modal distribu- tion (z(2) in the embedding space. Each mode in this distribution corresponds to one of the input classes and the training data points that belong to a particular class, are mapped to the same cluster (see Figure []] middle). The inter- nally learned multimodal distribution is repre- sented empirically by the source domain data
New Domains eacountered sequentially
Current Unlabeled Data
Di
Experience Explay Path
3. The model is trained joiatly the imput samples and the mwmory buller samples
Memory Buffer
Encoder
r D. (.C.
S.
The GMM updated upon learning the current tasks
Lifekong
Learning Loop
Embeddiing Space
Ciasoliiated Insenaal Distriouiinn
Pscudo- Dataset
T/ = (Zip, Np)
Jlewiffer
Label Space Classffferr
(XL Y
4. Supervised pathway remains stable continual
6, Data selection for sturge accomuhation to implement the pseudo- rehcarsal process
representations ((du(x))) ( > We use GMM (z) with ls component the paramet- ric model to estimate the internal distribution:
training on the
pscudo
Lifelong Domain Adaptation Agent
p)(=) = Σ CfN E,) 1=1
(1) Figure I:
Architecture and learning procedure for the proposed lifelong UDA framework. For an en- larged version, please refer to the Appendix.
where al denotes the mixture weights, and uz
and ΣΦ denote the mean and co-variance matrices for the components. Computing these parameters is generally challenging iterative procedure. But since the labels are accessible, we can compute the parameters for each mode independently. Let SJ denote the support set for the mode j. i.c.,
S ((a), v E Dsl arg max v
j}. The MAP estimates for the GMM parameters would be:
a
N ,
Σ
59 oo()))
Σ
599 (bo(a=) (2)) (()((=)) - (3) (2)
(m Sees
We can update the estimates in Eq. for <> 1, to be described later. Let (2) denote the estimated GMM at the time-step 1. Our idea to consolidate this distribution to retain model generalization. In the absence of source data, we adapt the model such that the encoder always aligns the target distributions with this distribution in the embedding space. We implement this process by drawing
random samples from the GMM and build a labeled pseudo-dataset: DGP (Zip. Y,), where D = |zj, ... x,, E IRPN,, Yp [DP ..... DRI11 E RAXN,, and 2 ~ (z). We determine the labels according to the model predictions. We include only those samples for which the model prediction confidence-level is more than a preset threshold 7 to exclude outlier samples. To retain the model generalization power, solve the following problem at time t:
min
. + (x)))
(3)
3

---

where -)...) denotes a probability discrepancy measure and À is a trade-off parameter. The first term enforces the classifier to preserve its generalizability on the internal distribution. The second term enforces alignment of the target domain distribution with the internally learned distribution in the embedding space. We use the SWD metric [27] for the term D). Upon updating the model, we can also update the internal distribution ýi (2) using the pseudo-dataset samples.
Solving Eq. 33) would help the model to generalize well on the new domain T However, catastrophic forgetting will remain untackled because the encoder subnetwork is updated unconditionally to mini- mize the domain discrepancy term in Eq. To tackle catastrophic forgetting, we rely on experience replay. When a task is learned, we select small subset of the training data points as representative samples, to be stored in a memory buffer. These samples are replayed during model adaptation to mitigate catastrophic forgetting. Possible selection strategies includes mean of features (MoF) A0). ring buffer [1], and reservoir sampling [42) Existence of the internal distribution makes MoF a natural choice. Upon model update at time-step & and then updating the GMM, we can compute the distance of representations for all the data points of T from their corresponding cluster mean, i.e., d²ا, = ||, < Vzf s.t. of arg max f& (af) = j. Given memory budget of N, samples, pick the M,, No/k samples per class that have the closest distance to the mean to form the buffer-stored dataset D DI-1
1 U (X}, YY,). These samples are the most informative samples to estimate the internal distributions. We update Eq. (3) to tackle catastrophic forgetting as follows:
min Σ (hool)))) 0. 17,90
Σ (hha(dooa))) g.) + (x.))
+ (x)))
(4) Adding the second supervised term in Eq. helps to mitigate catastrophic forgetting. The fourth term helps to consolidate the internal distribution across all the previous tasks. Our proposed solution for lifelong domain adaptation, named Lifelong Domain Adaptation Using Consolidated Internal Distribution (LDAACID), is presented and visualized in Algorithm Tand Figure I respectively.
5 Theoretical Analysis
Algorithm 1 LDAuCID (A, T,,NN)
1: Source Training:
We provide an analysis within standard PAC- learning framework [39]. Consider the set of classifier sub-networks H (+) :
2: Input: source labeled datasct Ds = (Xu, Yo)
3:
O = (wos 0b) = arg ming Σ, ((o(),)), y!!)
2 + R E } the hypothesis class
4:
Internal Distribution Estimation:
within PAC-learning. Let eo denote the ex- pected error on the source domain, e de- note the expected error on the target do- mains, and ef' denote the expected error on the pseudo-dataset for a given h E H. Also, let Po(2)) = E (()() and (a())) = 1 EM_, (z.( denote the empirical M.
5:
Use Eq. 2 and estimate aI, 43,, and En
6:
Memory Buffer Initialization
7:
D.
Pick the No/k samples with the least di,, arg max fox (z) (2).
D. 0.4
8: Continual Unsupervised Domain Adaptation:
9: for t = 1...., do
source and target distributions in the embedding space. We provide the following theorem.
10:
Input: target unlabeled datasct Dj = (X,) Pseudo-Datavet Generation:
11:
Theorem 1: Consider LDAuCID algorithm at learning time-step t = T. Then for all the previ- ously learned tasks < T. the following holds:
12:
D'p (Zzp. Yp)
13:
([* ...., N 1 UN DWDD where:
2. ~ ~ (2), Sis No and
8ht arg maxy (hao, (z]" if with confidence max, (hva, (z.t > for itr = 1, l,.., ,,. do
el_ + W( ).P + Σ W(G
14:
15:
draw data batches from D4 and Dr Update the model by solving Eq. 4 end for
<(ww)) ) + v (2lo8(())00 / 3/1((
I Mi + *VN,
I
16:
17:
18:
Internal Distribution Estimate Update:
19:
Use Eq. 2 similar to step 5 above. Memory Buffer Update
No
20:
(5)
21:
DJ. = DJ (X;, Yn'), where (X,, 87) is computed similar to step 7 above.
where e(w") denotes expected error for the best joint-trained optimal model in the hypothesis
22:
end for
4

---

space, i.e., the model trained as w" = arg min cc(w) = arg mi + (h()), w(.-)
denotes the WD distance, and and E are two constants.
Proof: Included in the Appendix.
Theorem I explains why LDAuCID algorithm is effective. Major terms in the right-hand side of 5 as an upperbound for the expected error for each task (domain), are continually minimized by LDAuCID. The first term is minimized because the internal distribution random samples are used to minimize the empirical error term as the first term of Eq. 4 The second term is minimized as the third terms of Eq. 4 when the task distribution is aligned with the empirical internal distribution in the embedding space at time t. The third term which is a summation which models the effect of continual learning. The 8th term in this sum is minimized when the task T is learned and deliberately update the internal distribution. This additive term grows as more tasks are learned after learning a particular task which can potentially make the upperbound looser, leading forgetting effects. The fourth term is a constant small term the tasks are related, i.e., share the same classes. LDAuCID does not minimize this terms. It just suggests that the model should work well for joint-training scenario in order to perform well in sequential learning setting. The last term is a constant term which is negligible if we have sufficient numbers of source and target training data points. It also states that storing more samples in the memory buffer leads to better performance, as expected. We conclude if the upperbound in Eq. (5) sufficiently tight, the domains are relevant, and the hypothesis space is suitable space to learn the tasks, then adapting the model using LDAuCID can both tackle catastrophic forgetting and also improve model generalization on the target domains.
6 Empirical Validation
Our implemented code is accessible as an Appendix.
6.1 Data Sets and Tasks
We are not aware of any prior work that addresses UDA in a lifelong learning setting that we can directly compare against. For this reason, we use four existing UDA benchmark datasets, adopt them to build sequential UDA tasks, and validate our method on these classic datasets. For comparison purpose, we run our algorithm in the learning setting of the existing UDA methods with one source and one target domains. For fair comparison against these works, we have followed the evaluation protocols that are used by most of the recent classic UDA papers.
Digit recognition tasks: the common MNIST (M), the USPS (41), and the SVHN (S) datasets are used as three domains. Majority of the existing UDA methods report their results on three tasks defined between these domains: M - u, + .M, and S + M tasks. We perform experiments on two S M + U1 and S U + M sequential UDA tasks to cover the three classic UDA tasks.
ImageCLEF CDEF-DA Dataset: this dataset consists of the 12 shared image classes between the Caltech- 256 (C), the ILSVRC 2012 (I), and the Pascal VOC 2012 (P) visual recognition datasets. The dataset is fully balanced with 50 images per class, i.e., 600 images for each domain. There are six possible binary UDA tasks. We perform experiments on - I P and C I P ternary domain tasks.
this a more challenging object recognition dataset which consists of 15, 500images in office and home settings. The images are grouped into 65 classes. There are 4 domains with relatively large gaps: Artistic images (A), Clip Art (C), Product images (P), and Real-World images (R) which leads to possibility of defining 12 pair-wise binary UDA tasks. We preform experiments on the sequential UDA tasks A + C + P - R and R P + A tasks.
Office-Home Dataset:
this object recognition dataset is built using the 10 shared classes between the Office-31 and Caltech-256 datasets. There are four visual domains: Amazon (A), Caltech (C), DSLR (D), and Webcam (W) with 2533 images in total. There are 12 definable binary UDA tasks. We perform experiments on the sequential UDA tasks A - C - D > W and W - D - A.
Office-Caltech Dataset:
6.2 Network Structure and Evaluation Protocol:
We use the VGG16 network as the base model for the digit recognition tasks. Following the literature, use the Decafó features for the Office-Caltcch tasks. For the other two datasets, we use the
5

---

44
55207
3230
(8) S 7 M + 4
(b) C + I + P
(c) A +
C
P
+
R
(d) A + C +
D -
W
16
16
350 70301s
Sporne
Spaets
7pants
(e)
M
(0) -I C
(g) - (h) W A
Figure 2: Learning curves for sequential UDA tasks on (a.e) digit, (b.f) ImageClef, (c.g) Office-Home,
and (d,h) Office-Caltech datasets. (Best viewed in color).
ResNet-50 network which is pre-trained the ImageNet dataset the backbone of the network. To study the model learning dynamics over time, we generate learning curves by reporting the model performance on the testing split of the learned tasks versus the training epochs. Our purpose is simulate continual training during execution time. Providing the learning curves allows to study dynamics of learning. For comparison purpose and after learning a target domain task, we report the average classification rate and the standard deviation on the target domain for five runs for each task. We train the base model using the source labeled data. We report the performance of the model before adaptation as simple ablation study to demonstrate the effect of domain shift. Then we adapt the model using the target unlabeled data using LDAuCID algorithm and report the performance of LDAuCID on the target domain. For more details on implementation, please refer to the Appendix.
6.3 Results
Learning curves for the above eight sequential UDA tasks are visualized in Figure 2 curves, the model has been trained for 100 epochs, i.e., notion for time, for each task and then we have moved forward to learn subsequent task. We have stored 10 samples per class per domain in the memory buffer for experience replay. We first observe that domain shift leads to an initial performance drop on the target domain in all the eight tasks, however there a jumpstart in performance. That is, the learning curves for the subsequent target tasks initialize significantly higher than random label assignment. For example, we observe in Figure 2a the initial testing accuracies for the domain M and 1I are in the + +60% range. This initial jumpstart occurs due similarities between the domains, leading to knowledge transfer from past experiences. The rising behavior the learning curves after this initial performance, when the model is being trained on T', demonstrates that LDAuCID effectively improves performance of the source-trained model. This improvement occurs for all the target domain tasks. We note that this the final performance improvements on the Office-Home dataset is less because domain gap between the domains are larger for this dataset.
In these learning
The second important observation is that catastrophic forgetting has been mitigated quite well. We observe that after moving forward to learn the subsequent domains for all the eight tasks, the model performance on the previously leamed tasks is relatively stable and forgetting effect is not dramatic. We note that the forgetting effects for the SVHN dataset S - 2 + M and S - M - U tasks are severe. This is likely because this dataset is more challenging and hence larger number of samples in the memory buffer may be required. We conclude that LDAuCID algorithm has successfully improved the model generalization while mitigating catastrophic forgetting.
To measure ability of our algorithm, we also compare LDAuCID against existing UDA. We are not aware of any prior work that addresses UDA in a lifelong learning setting. For this reason,

---

Method
M-u
4-M
M
Method
M
u M
8 8-4 M
GIA 10]
0.9 13 0.9
CDAN 13
93.9
96.9
88.5
CoGAN
46
0.. 89.1
SHOT 47
89.6+5.00
96.8+0.4
91.9+-0.4
ADDA
8
0.2 90.1
76.0 1.8
CyCADA
48
9
96.5 0.1
90.4 + 0,4
RevGrad
13
77.1
73.0
73.9
JDDA 44
97.0
93.1 +0.2
DRCN
21
91.8
0.1
7..7
0.2
OPDA 27
70.0
60.2
ETD
96.41 0.3
97.91 9.131
0.4
MML 50
77.9
60.5
62.9
Source Only
90.1+2.6
80.2+5.7
67.3+2.6
LDAuCID
96.8 + 0.2
98.4 + 0.1
91.4 2.2
Table 1: Classification accuracy for UDA tasks between MNIST, USPS, and SVHN datasets.
employ our algorithm in the special case of having only two domains and compare its performance with the classic UDA algorithms. We consider that only one target domain exists and report our performance on this domain. We still address UDA in sequential learning setting, but is the best we can do for comparison. For comparison, we included the classic UDA methods: GtA [TO] DANN 43 ADDA (3) MADA (II, DAN [14), DRCN [21]. RevGrad [15], JAN [66), JDDA [44)
and UDAwSD 45 We include results of these works in case the original paper have reported performance on the corresponding dataset. In the Tables, bold font denotes the best performance.
Comparative results for six bi-domain digit recognition tasks are summarized in Table | In each table, we have included the Source Only performance as baseline. It reports the performance of the cource-trained model on the target domain without adapting the model. Despite the fact that the classic UDA methods use the full source dataset for joint-training, we observe that LDAuCID outperforms the UDA methods in two of the tasks and its performance on the remaining task is competitive. We observe that ETD is outperforming LDAuCID and on average possesses the highest performance. This not unexpected because this method is quite close to our method. ETD relies on version of WD distance for improvement for classic UDA joint-training scheme.
Table 2 presents the results for the ImageCLEF-DA dataset tasks. We note LDAuCID has lead to significant improvement on this dataset. As mentioned, this dataset is fully balanced both in terms of number of data points per domain and also per class. Note that our method we rely on the empirical versions of the distributions for domain alignment (check the Appendix for details). A balanced dataset makes the empirical distribution less biased approximation of the real distribution. We conclude that balanced datasets across the domains can boost performance of our method.
Table summarizes the comparative results for the Office-Home dataset. We observe that CDAN has the best average performance but LDAuCID is still competitive and on four of the tasks outperforms CDAN. As discussed, domains of this dataset has larger gap compared to the rest of the datasets we used. This could be verified in Figure 22 from the lower jumpstart performance value. As a result, minimizing the second term of the upperbound in Eq. 5s more challenging for the Office-Home dataset. Performance of CDAN is higher likely because CDAN aligns the two distributions class- conditionally. We conclude that a possible direction improve our method further in the future is to benefit from class-conditional alignment techniques, e.g, pseudo-labeling the target domain data.
From inspecting Tables 1-4, conclude that LDAuCID performs quite competitively on all the UDA tasks, despite addressing lifelong UDA constrains in which most of the source domain data points not accessible. Although LDAuCID likely needs be upperbounded by classic UDA algorithms in terms of performance, but we observe it outperforms many of these UDA methods on some of the standard UDA tasks. Although our motivation was to address UDA in CL. setting, our observations conclude LDAuCID can be used for addressing classic UDA. We hope that development of subsequent methods for lifelong UDA learning regime would make more thorough comparison possible.
6.4 Analytic and Ablative Studies
To study the effect of the algorithm empirically, we have analyzed the geometry of data points in the embedding space. Data geometry approximates the distributions that are learned in the output of the encoder, i.e., empirical version of the internally learned distribution. For this purpose, we take advantage of the UMAP (54) visualization tool to reduce the dimension of the data representations in the embedding space for 2D visualization purpose. In Figure 3 we have visualized the testing splits of the source domain and the two target domains along with a number of randomly drawn samples of the internally learned GMM distribution for the S + - U digit recognition task. In this figure,
7

---

Method
I+P
P
I-C
C-I
C+P
P
Average
Source Only
7..8
8.1
91.3
7..
635
9..2
80.8
DANN
43
82.0 0.4
96.9
99.1
79.7
6..2
6.54
82.2
MADA
11
75.0 0.3
87.
96.
160000 0.3 88.8
75.2
9..
85.9
CDAN
13
76.7 0.3
90.6
97.0
0.4 90.5
74.5
93.5+
87.1
DAN 14
74.5 0.4
82.2
92.8
0.2 86.3
69.2
0.4
8.8
82.4
RevGrad
[S]
75.0
8..0
96.
0.4
87.0
74.3
9.6
85.0
JAN 16
76.8 1 0.4
88.0 0.2 94.7
0.2
89.5 0.5
74.2
0.3
91.7
85.7
ETD 49
81.0
91.7
97.9
93.3
79.5
95.0
89.7
LDAuCID
87.8 1 1.4 99.1
1 0.2 100 1 0.0 99.8 1 0.0 88.8
1.0 99.5 0.3
95.8
Table 2: Classification accuracy for UDA tasks for ImageCLEF-DD dataset.
Method
A-C
A-PP
A-R
C-+A
C->P
C-R
R
P-A
P-C
P-R
R
R-A
R-+C
R-+P
Average
Source Only
9
34.9
50.0
58.0
37.4
41.9
46.2
38.5
31.2
60.4
53.9
41.2
59.9
46.1
DANN
70.1
47.0
63.2
76.8
CDAN
13
49.0
69.3
74.5
55.4
66.0
68.4
55.6
48.3
75.9
68.4
$5.4
80.5
63.9
DAN 14
43.6
57.0
67.9
45.8
56.5
60.4
44,0
43.6
61.7
63.1
51.5
74.3
56.3
JAN
16
45.9
61.2
68.9
50.4
59.7
61.0
45.8
43.4
70.3
63.9
52.4
76.8
58.3
DJT
223
39.7
50.4
62.4
39.5
54.3
53.1
36.7
39.2
63.5
52.2
45.4
70.4
50.6
LDAuCID
74.1
48.7
61.9
49.6
42.1
60.3
76.6
Table 3: Classification accuracy for UDA tasks of Sifice-Home dataset.
Method
A-C
A-+
A-+W
W-+
W-+
W-+C
A
D-W
D-C
C-+
C-+
W
C-D
Average
Source Only
84.6
81.1
75.6
79,8
98.3
79.6
84.6
96.8
80.5
92.4
84.2
87.7
DANN 43
87.8
82.5
77.8
83.0
100
81.3
84.7
99.0
82.1
93.3
89.5
91.2
87.7
MMAN
51
88.7
97.5
96.6
94,2
100
89.4
94.3
99.3
87.9
93.7
98.3
98.1
94.6
RevGrad
85.7
90.8
92.8
88.1
87.9
DAN 14
84.1
91.7
91.8
92.1
100
81.2
90.0
98.5
80.3
92.0
90.6
89.3
90.1
CORAL 52
86.2
91.2
90.5
88.4
100
88.6
85.8
97.9
85.4
93.0
92.6
89.5
90.8
WDGRL
53
87.0
93.7
89,5
93.7
100
89.4
91.7
90.2
93.5
91.6
94.7
92.7
LDAuCID
99.6
100.0
86.3
96.1
100
99.8
88.5
100.0
95.7
99.3
96.4
99.8
96.8
Table 4: Performance comparison for UDA tasks of Office-Caltech dataset.
each point represents one data point and each color represents one of the ten digit classes. Each row of Figure repeeeeens the data geometry at the embedding space at the end time-step t. For example, the second row shows data geometry after learning the SVHN and MNIST datasets. This means that we check the aub-figures vertically for each of the domains, i.e., a column of the figure, we can tasks versus time. Checking all the columns,
inspect the effect of learning learning a task, the learned knowledge is retained when the model is adapted in the future, i.e., the next bottom rows, because classes stay relatively separable. This stability suggests that catastrophic forgetting has been continually mitigated when the model is updated. We can also see at the last row, all the domains share the same distribution similar to the internally learned GMM distribution. This observation suggests that our method successfully aligns the distributions to share the same internal distribution. Finally and as an example, we compare the distribution of the MNIST dataset in the first row versus the second row, we see that as the result of domain adaptation, its distribution in the embedding space becomes more similar the SVHN dataset (source domain). We conclude that Figure 3 empirically confirms the result that we could deduce analytically from Theorem 1.
see that upon
Similar to most learning algorithms, setting up the proper values for the hyperparamters is important for optimal performance. We have studied the effect of the values for the hyperparameters and T on the model performance for the binary UDA task S - M in Figure [e and Figure 33 to suggest how to tune these parameters for practical usage. We have visualized the average performance with a dark curve. The lighter shaded region around the curve denotes the standard deviation. We observe in Figure 3e that the value for the parameter À does not have a significant effect on performance. This observation is according expectation because the ERM loss term C(-) in Eq. (3) is relatively small prior to beginning of the alignment process as result of pre-training on the source domain. In other words, since the dominant term for optimization is the domain-ale alignmenn term in Eq. (3), careful finetuning of the trade-off parameter À is not necessary. We also observe in Figure 3f that when the confidence parameter is 1, the model performance on target domain improves. This accords with our intuition because potential outlier GMM distribution samples cause label pollution which is a challenge for domain alignment UDA. From the results presented this section, we can deduce that our method is effective and the empirical observations support our theoretical analysis.

---

94.0
90
Accurecy
80
92.5
92.0
70
90.5
90.0
-7
02
kud(Al
97.1 (a) SVHN (b) MNIST
USPS
(d) GMM
(e)
(I)
Figure 3: (a-d) UMAP visualization for the testing split of the domains in the UDA task S + M - +4
and the fitted GMM time-steps 0, 2, Visualizations at each of the rows are computed after learning the tth task prior to the time-step + 1. Enlarged version is included in the Appendix. (e-f) Effect of the values for the hyperparameters À and on model generalization in the target domain for the binary UDA task S M. (Best viewed in color and on screen.)
Imbalanced Dataset
No = 50
N₂ 100
Time-step t= t=
Time-step
t=0
[ ==2
Time-step t= f=
1=
89.8
84.8
80.2
92.3
83.6
82.0
93.3
83.6
83.1
M
93.2
93.0
M
94.2
95.6
M
95.6
93.1
U
91.9
u
94.6
Table 5: Analytic experiments using the S + M - U digit recognition task.
We have provided additional controlled experiments the S + M + U task in Table draw better intuitions about our proposed algorithm. Since we observed that our algorithm more effective for balanced datasets, first study the robustness of algorithm with respect data imbalance. To this end, we introduced data imbalance for all domains in the S - M + l task by considering that for digit i, only 1+1 portion of the data points available. We have reported the observed performance in Table where each row shows performance for each task at the end of learning period for task We observe that despite reduced performance, our method still addresses catastrophic forgetting and leads to improved performance in the target domains for this imbalanced dataset.
Finally, note another unexplored aspect in our algorithm is the effect of No on performance set its value arbitrarily. We set N₂ = 10 in previous experiments because compared to the sizes the datasets, N₂ would be very small. We studied effect of increasing N, in Table 5 for N,, 50 and N₂ 100. As expected, we observe that using a larger buffer size leads to improvements. In practice, the buffer size should be selected to be large as possible given the hardware storage limitations.
7 Conclusions
We develop an algorithm for domain adaptation a continual learning setting. The core assumption that when train neural classifier, the input distribution mapped an internal distribution, in an embedding space modeled by a neural network hidden layer. Our method is based on consolidating this internally learned distributional such that all learned tasks share a similar distribution in the embedding space. As result, the model generalizability is retained when new tasks are learned. Additionally, catastrophic forgetting mitigated using experience replay by storing and then replaying the input samples that are more informative for estimating the internally learned distribution. We used a simple approach to estimate the internal distribution but we foresee that better estimations of the internal distribution can boost the continual learning performance. Future research includes studying the effect the tasks' order on continual learning performance and extensions to incremental learning setting, where new classes can be learned beyond the initial training phase.

---

Acknowledgement
We thank the anonymous reviewers whose constructive and continual feedback helped improve the presentation, clarity, and analysis of the proposed work.
References
[1] Yaniv Morgenstern, Mohammad Rostami, and Dale Purves. Properties of artificial networks evolved to contend with natural spectra. Proceedings of the National Academy of Sciences, 111(Supplement 1110868-18777 2014.
[2] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):433 444, 2015.
[3] Eric Tzeng, Judy Hoffman, Kate Sacnko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 7167-7170
2017.
[4] Mohammad Rostami, David Isele, and Eric Eaton. Using task descriptions in lifelong machine learning for improved performance and zero-shot transfer. Journal of Artificial Intelligence Research, 67:673-704, 2020.
[5] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995. PMLR, 2017.
[6] Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures Artificial Intelligence and Machine Learning, 12(3):1 207, 2018.
[7] Mohammad Rostami, David Huber, and Tsai "sai-Ching Lu. A crowdsourcing triage algorithm for geopolitical event forecasting. In Proceedings of the 12th ACM Conference on Recommender Systems, pages 377-381.
ACM, 2018.
[8] Mohammad Rostami. Transfer Learning Through Embedding Spaces. CRC Press, 2021.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 770-778,
2016.
[10] Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa. Generate adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8503-85 8212, 2018.
[11] Zhongyi Pei, Zhangjic Can, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adaptation. In Proceedings Thirty-Secona AAAI Conference on Artificial Intelligence, pages 3934-3941, 2018.
[12] Yabin Zhang, Hui Tang, Kui Jía, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5031 5031-5040, 2019.
[13] Mingsheng Long, Zhangjie Cao, Jianmin Wang, Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Syxtems, pages 1640-1650, 2018.
[14] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Proceedings of International Conference on Machine Learning, pages 97-105, 2015.
[15] Yaroslav Ganin and Victor Lempitsky, Unsupervised domain adaptation by backpropagation. Proceed- ings of International Conference on Machine Learning, pages 1180-1 189, 2015.
[16] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 2208-2217. JMLR. org, 2017.
[17] Guoliang Kang. Lu Jiang. Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4902, 2019.
[18] Mohammad Rostami and Aram Galstyan. Domain agnostic internal distributions for unsupervised model adaptation. arXiv preprint 07X1:7200.001197 2020.
10

---

[19] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences. 3(4):128-135, 1999.
[20] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka forgetting in neural networks. Proceedings the national academy of sciences. 114(13):521-33526 2017.
et al. Overcoming catastrophic
[21] Muhammad Ghifary. W Bastiaan Kleijn, Mengje Zhang, David Balduzzi, and Wen Li. Deep reconstruction- classification networks for unsupervised domain adaptation. In Europaan Conference Computer Vision, pages 597-613. Springer, Spriin,, 2016.
[22] Pietro Morerio, Jacopo Cavazza, and Vittorio Murino, Minimal-entropy correlation alignment for unsuper- vised deep domain adaptation. In ICLR, 2018
[23] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference Computer Vision (ECCV), pages 447-463, 2018.
[24] Chen- Yu Lee. Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10285-10295, 2019.
[25] Serban Stan and Mohammad Rostami. Unsupervised model adaptation for continual semanntc segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2593-222 2001,261122200 2021.
[26] Mohammad Rostami, Soheil Kolouri, Eric Eaton, and Kyungnam Kim. Sar image classification using few-shot cross-domain transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[27] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(9))1853-188 1865, 2016.
[28] Andrei Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell, Progressive neural networks. arXiv preprint arXiv: 16kiv:6606.440 04601, 2016.
[29] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 1xXi::151..5555,225 2015.
[30] Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123- 146, 1995.
[31] Hanul Shin, Jung Kwon Lec, Jachong Kim, and Jiwon Kim. Continual Icarning with deep generative replay. In Advances in Neural Information Processing Systems, pages
2017.
[32] Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint arXiv: 10368, 2017.
[33] Mohammad Rostami, Soheil Kolouri, and Praveen K Pilly. Complementary learning for overcoming catastrophic forgetting using experience replay. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pages 3339-33455 3345. AAAI Press, 2019.
[34] Mohammad Rostami, Soheil Kolouri, Praveen Pilly, and James McClelland. Generative continual concept learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5545-5552, 2020.
[35] Markus Wulfmeeerr Alex Bewley, and Ingmar Posner. Incremental adversarial domain adaptation for continually changing environments. In 2018 IEEE International conference robotics and automation (ICRA). pages 4489-4495. IEEE, 2018.
[36] Andreca Bobu, Eric Tzeng, Judy Hoffman, and Trevor Darrell. Adapting to continuously shifting domains. 2018.
[37] Zuxuan Wu, Xin Wang, Joseph E Gonzalez, Tom Goldstein, and Larry S Davis. Ace: Adapting to changing environments for semantic segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2121-2130, 2019.
[38] Horia Porav, Tom Bruls, and Paul Newman, Don't worry about the weather: Unsupervised condition- dependent domain adaptation. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 33 40. JEEE, 2019.

---

[39] Shai Shalev-Shward and Shai Ben-David. Understanding machine learning: From theory to algorithmms Cambridge university press, 2014.
[40] Sylvestre-Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[41] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances neural information processing systems, pages 6467 6476, 2017.
[42] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. International Conference on Learning Representations, 2018.
[43] Y. Ganin, Ustinova, H. Ajakan, Germain, H. Larochelle, Laviolette, M. Marchand, Lempitsky. avveraaldversaaial training of neural networks. The Journal of Machine Learning Rexearch, 17(1):2096- 2030, 2016.
[44] Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3296 3303, 2019.
[45] Rui Li. Qianfen Jiao, Wemming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9641-9650, 2020.
[46] Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in neural information processing systems, pages 469 477,4477.200 2016.
[47] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028 6039. PMLR, 2020.
[48] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cyclc-consistent adversarial domain adaptation. In International Conference on Machine Learning, pages 1989-1998, 2018.
[49] Mengxue Li, Yi-Ming Zhai, You-We Wei Luo, Peng-Fei Fei Ge, and Chuan-Xiaa Ren. Enhanced transport distance for unsupervised domain adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Partern Recognition, pages 13936-13944, 2020.
[50] Vivien Seguy, Bharath Bhushan Damodaran, Remi Flamary, Nicolas Courty, Antoine Rolet, and Mathicu Blondel. scale optimal transport and mapping estimation. In International Conference Learning Representattins, 2018.
[51] Xinhong Ma, Tianzuu Zhang, and Changsheng Xu. Deep multi- adversarial networks for unsuper- vised domain adaptation. JEFE Transactions on Multimedia, 21(9):2419-244 2019.
[52] Baochen Sun. Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
[53] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[54] Leland McInnes. John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform manifold approxi- mation and projection. Journal of Open Source Software, 3(29):861, 2018.
12


Reviews:
[{"review_id": "y9ebcG-YxOK", "reviewer": "Reviewer_T4HK", "summary": "This paper presents a novel method for unsupervised domain adaptation in continual learning setting using only unannotated data. The proposed method models the internally learned distribution of initial annotated source domain as multimodal distribution and updates it using the learned distributions of subsequent unannotated target datasets to couple all the domains together i.e., consolidated internal distribution. \n\nTo minimize catastrophic forgetting, the proposed method also performs experience replay during domain adaptation on target dataset using important representative samples of all historical tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\n1. This work proposes a novel method which combines Unsupervised Domain Adaptation and Continual Learning methods in a joint learning framework via consolidated internal distribution. \n\n2. In terms of novelty, the proposed method has the following strengths:\n> - Creating internally learned distribution using initial annotated source dataset as multimodal distribution and consolidating it over a sequence of subsequent unannotated target datasets via domain alignment in shared embedding space using Sliced Wasserstein Distance (SWD) metric.\n> - Identifying important representative data points for each historical dataset and using them during current dataset's training for knowledge retention of past datasets/domains and to minimize catastrophic forgetting.\n\n---\n\nQuality:\n\n1. The motivation is well founded and the claims are sound.\n\n2. Theoretical formulation and analysis are done in detail and look fine except for the comments below.\n> - Step-19 in algorithm 1 denotes update of GMM parameters using newly learned unannotated target dataset $t$, but equation 2 only explains the initialization of GMM parameters using initial annotated source dataset. Therefore, there is a need of another equation for updating GMM parameters using target dataset $t$.\n> - See comments regarding inconsistent use of notations below.\n\n---\n\nClarity:\n\n1. As the proposed method deals with unsupervised domain adaptation and continual learning for classification tasks only, it would be appropriate to mention this in the introduction itself before \"costly and time-consuming manual data annotation\" statement.\n\n2. In Figure 1, if equations are also referenced in the steps (wherever applicable) as done in algorithm 1, then it should enhance the understanding of the proposed method.\n\n3. A table of notations should improve the readability.\n\n4. Inconsistent use of notations:\n> - $N$ is used to signify the number of samples in annotated dataset $D_S$, but $M_t$ is used to signify number of samples in unannotated dataset $D_T^t$ (in line 78).\n> - Pseudo-dataset samples are denoted with $z_i^t$ in line 116 and with $z_i^p$ in equation 3 and $z_i^{p,t}$ in algorithm 1.\n\n5. Typographical errors:\n> - Line 35: forgetting the CL literature -> forgetting in the CL literature\n> - Line 41: update the modal -> update the model\n> - Line 72: challenge is CL -> challenge in CL\n> - Line 73: non-stationery -> non-stationary\n> - Line 75: is update -> is to update\n> - Line 134 form form -> form\n> - Line 195: Damian -> Domain\n> - Figure 2 caption: (a,c) -> (a,e)\n> - Table 1 caption: MINIST -> MNIST\n\n6. Methodology:\n> 1. As selection of important representative samples for experience replay is done based on the distance to cluster mean in input representation space ($x$) and not in embedding space ($z$), using model prediction confidence scores in addition to distance to cluster mean might help in selecting better representative samples for experience replay.\n> 2. While updating GMM parameters, updating parameters using all the samples of target dataset in embedding space might be sub-optimal due to low confidence model predictions. So, it would be interesting to see how model performance varies if parameters are updated based on the samples with model prediction confidence score above a certain threshold hyperparameter.\n\n7. Empirical Evaluation:\n> 1. In Figure 2, accuracy of base model trained on each dataset of UDA sequential tasks as initial source dataset can be added and compared with the learning curves of sequential UDA tasks to study the tradeoff between knowledge transfer from past domains and knowledge retention of past domains i.e., minimizing catastrophic forgetting.\n> 2. As equation 4 does not contain any hyperparameter to control the intensity of minimizing catastrophic forgetting, the only control parameter is the number of samples $N_b$. So why store only 10 samples per class for experience replay? Is it supported by any ablation study?\n> 3. There is no clear description of \"Source Only\" method shown in all the Tables. If \"Source Only\" method is training of base model using only initial annotated source dataset then, Figures 2(a,e) both suggest model accuracy after training 100 epochs on source dataset $S$ is around 90. So, why \"Source Only\" performance in Table 1 in $S \\rightarrow M$ column is only 67.3 which is far less than 90?\n> 4. Similarly, in Table 2, why \"Source Only\" performance so much different in columns $C \\rightarrow I$ and $C \\rightarrow P$ with the same source dataset $C$? However, performance of LDAuCID method on target datasets is similar with columns containing same target datasets.\n> 5. A clear indication in Figure 3 (left) regarding which row belongs to which timestep would improve understanding better.\n\n8. Evaluation Setup:\n> 1. Why chose max epochs as 100 per task as 100 epochs might not be enough for convergence on initial annotated source dataset which builds the internally learned distribution in embedding space?\n\n---\n\nSignificance:\n\t\n1. Evaluation setup is comprehensive and results show significant improvement in 2 tasks out of 4.\n2. This is a significant achievement given the fact that this is the first algorithm which implements Unsupervised Domain Adaptation in Continual Learning setting.\n3. Evaluation datasets, tasks and base models have been selected based on the related works in UDA literature.\n4. To keep the evaluation fair with other UDA algorithms without continual learning, pairwise tasks have been used.\n5. Although, flat learning curves in Figures 2 (c,d,f,g,h) of the initial source dataset during adapting model on subsequent target datasets might indicate that the algorithm puts more weight on minimizing catastrophic forgetting. Hence, there might be a need for a hyperparameter apart from number of samples $N_b$ to control catastrophic forgetting during domain adaptation on subsequent target datasets.\n\n--\n\nUpdates: \n\n(24-08-2021): increased score from 6 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Addressed by authors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "14"}, "extra_text": {}}, {"review_id": "kwOtINwptQJ", "reviewer": "Reviewer_7ykJ", "summary": "Unsupervised Domain Adaptation (UDA) is studied in a task sequential continual learning setting. The source domain is assumed to have supervised data and several target domains without labels are being sequentially adapted to. An algorithms is engineered for the specific case of supervised classification source datasets which can be neatly clustered in some high-level representation; then, learning on related (target) datasets which can be classified in the same (source) classes can be done sequentially and without the use of labels. Catastrophic forgetting is mitigated using a standard experience replay approach, but also through the assumptions of the conceptual framework, i.e. strong overlap between source and target classes, which is known to be the easier setting in CL works. Empirical validation is done primarily in the using classic domain adaptation benchmarks, with the difference of using a single source domain for multiple target domains in a sequential fashion. The theoretical analysis and ablation study round up the work, including a discussion of algorithm hyper-parameters.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of the approach is reduced in terms of continual learning research, especially due to assumptions about source and target datasets, hence I need to primarily assess the merits of the work based on domain adaptation literature. While labels are not used for the target datasets, the assumption of identical class set is very strong. \n\nThe empirical evaluation datasets used here, with domains in the hundreds of examples, make it difficult to imagine what the method would do on more realistic current image data streams. Real datasets are not this neat and definitely not this small, even supervised datasets. It is also hard to separate the improvements due to ImageNet pre-training and those due to specific domain adaptation algorithms. I do not believe it is enough to say that this is what domain adaptation works use, but it may be necessary to educate the readers (and indeed the reviewers) on why these datasets are still challenging and why algorithmic improvement is needed; for example, with large scale source dataset learning (even unsupervised source learning with e.g. SimCLR(v1 or v2)), freezing such models and learning linear output layers is enough for learning many other datasets. Why do we need to keep learning from small source datasets? Why not focus on domains where there is no choice but unsupervised domain adaptation?\n\n\nI need further clarifications for the following:\n* What is the upper bound of performance? Multitask training on both source and target domains using target labels and an empirically weighted joint objective?\n* When is \"aligning internal distributions\" possible or desirable? Many datasets have very different classes, e.g. MNIST and Flowers. Is aligning distributions useful there?\n* How robust is the method when there is data imbalance between source classes in target datasets?\n\n\n\nMisc\n* Could you please explain what is meant in rows 30-31? Why are existing UDA algorithms not suitable for continual learning?\n* Row 41: modal or model?\n* Row 42: could you please explain what is meant by coupled learned distributions?\n* Row 195: Damian or domain?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not believe the discussion of limitations is sufficient. I would suggest improvements and clarifications along these lines:\n* Consolidating internal distributions from datasets which are very different seems counterintuitive; even if the target dataset is a subset of source dataset, say source=ImageNet, target=Dogs++, there are at least 100+ overlapping classes, but it is at least possible that aligning distributions may lead to problems. This should be discussed in the main text.\n* Even if continual UDA is possible, it is not entirely clear whether it is needed or beneficial for performance improvements compared to target task specific  source -> target UAD experiments with tuned hyper-parameters.\n* Self-supervised learning on data rich source datasets (e.g. ImageNet++) seems to be a strong alternative, certainly for the datasets where the proposed method was demonstrated; I can agree that there are cases where such transfer is not helpful, e.g. (I suppose) skin cancer classification, but such domains are not explored.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "YBClCd3liR", "reviewer": "Reviewer_RVm3", "summary": "This paper proposes a problem formulation at the intersection between continual learning (CL) and unsupervised domain adaptation (UDA). It introduces a method to perform UDA throughout the lifespan of a model, mitigating catastrophic forgetting of previous information. The proposed method is validated on benchmarks proposed ad-hoc for this problem, as well as on standard UDA benchmarks - showing that is can compete with state-of-the-art approaches even in standard settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well written paper on an important topic, that has not received the proper amount of attention yet (CL+UDA). The proposed solution builds on standard techniques typically used in UDA (minimizing distance between domains in the space of learned representations) and in CL (pseudo-rehearsal): they are merged in a very reasonable way, into a method that is empirically shown to perform well on sequences of tasks - where by \"task\" we intend \"UDA problem\".\n\nI liked the idea of using GMM for pseudo-rehearsals, with respect to related work on CL that use more complex strategies based on VAEs/GANs: it is cleaner, simpler to implement, and it fits well with the Mean-of-Features memory approach that is used.\n\nMy main criticism is that other works have studied CL+UDA problems before, and this work claims to be the first of its kind (I'm reporting the main references below). In particular, [A] proposes a solution that is very similar to the one proposed in this paper. It is originally devised for domains that shift gradually, but it could be easily applied to the benchmarks studied in this paper. Can the Authors comment on this? While [B] is presented as a method for semantic segmentation, the proposed pipeline seems to be fully generalizable to classification problems.\n\nMinor: in the Digits experiment, SVHN is never used as the final domain. MNIST $\\rightarrow$ SVHN adaptation is very challenging in UDA, hence it would be interesting to see what happens when SVHN is the final domain in a learning sequence.\n\nI enjoyed reading this paper, and I believe it will help bringing attention to a research direction that is currently a narrow niche, CL+UDA. I did not give a higher score because the related work section - and, in turn, comparisons and discussions - does not include the few works that have addressed the problem tackled in this paper. I look forward reading the Author response on this point, and updating my score accordingly.\n\nReferences for related work section ([A] and [B] seem the most relevant to me, but the others can further enrich the section)\n\n[A] Bobu et al., Adapting to Continuously Shifting Domains, ICLR Workshops 2018\n\n[B] Wu et al., ACE: Adapting to Changing Environments for Semantic Segmentation, ICCV 2019 \n\n[C] Wulfmeier et al., Incremental Adversarial Domain Adaptation for Continually Changing Environments, ICRA 2018\n\n[D] Porav et al., Don't Worry About the Weather: Unsupervised Condition-Dependent Domain Adaptation, ITSC 2019", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The theoretical analysis provided hints on the conditions in which the proposed methods will work. Societal impact is not discussed but I agree with the Authors that there is not much to discuss in that regard, for what concerns the submitted manuscript.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "/"}, "extra_text": {}}, {"review_id": "11ssF6dN-qR", "reviewer": "Reviewer_22Be", "summary": "The paper aims to address an unsupervised domain adaptation (UDA) in the continual learning (CL) setting, specifically the sequential unsupervised domain adaptation task. The learned internal distribution estimated as GMM are consolidated using pseudo-rehearsal to mitigate the domain shift and catastrophic forgetting.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity: The paper writing and formatting is clear and well organized. However, some grammatical errors and typos are found, e.g., line 72, “The challenge is CL is ...”; line 75, “Our goal is update ...”; line 134, duplicated “form”; line 196, the two sequential tasks are exactly the same. Also, the mathematical notations, especially super/subscripts, in the equations and algorithms have typos and sometimes are not consistent, which cause confusion and need to be corrected, e.g. in Eq(3), the superscript for $z_i$ should be $p$ instead of $t$; $e^J_t$ in line 149 and $e^t_J$ in line 16; $t$ instead of 0 is used as superscripts in Algorithm line 7; Algorithm line 13, the pseudo-dataset size “N” should be $N_p$. Please check the grammar through the paper and clean up the typos in the mathematical notations. \n\nReproducibility: All the experiments are using public datasets, and the code and experiment settings are provided in the paper and supplements, so the proposed method and experiments are expected to be reproducible. \n\nOriginality: The task of the paper is a combination of UDA and CL, and is formed as a sequential UDA task, which is a novel task setting in domain adaptation. The novelty of the method is the internal distribution estimation and pseudo-dataset sampling. However, the loss function and pseudo-rehearsal are existing techniques in UDA and CL and have limited novelty. \n\nSignificance: Since this task is a novel setting, limited horizontal comparisons are made between the proposed method and other existing methods. From the results in Table 1-4, the proposed method is at least comparable to other classic UDA algorithms. But since there’s no comparisons with other existing continual learning methods, it’s hard to say whether this method has significant advantages in CL. Moreover, the accumulation of memory buffer limits the proposed method being used in CL in practice, since in reality, lifelong learning will take unlimited data, but you cannot accumulate unlimited representative data in the memory buffer. \n\nQuality: The sequential UDA task is novel, and the challenges such as catastrophic forgetting and domain shift in CL are mitigated from the learning curves in Fig.2 as well as the visualization in Fig.3. The proposed algorithm and loss function are based on existing techniques and have both theoretical and experimental support. \nHowever, I think several important questions are not fully explained in this paper: \n1. Although the motivation for improving the domain shift in CL with unlabeled data is valuable and significant, the sequential domain adaptation in the problem statement is simply an ideal situation and not a real task in practice. It’s very hard to find such a situation in real life where multiple domains are learned separately and sequentially. In most CL cases, after initial training on labeled source domain, the upcoming continual unlabeled data points are sampled from the source domain and other target domains in random order and are always mixed together, instead of the training setting in the paper, where only one target domain is trained at each time step and no data points from other domains are included. \n2. The paper does not explain how the internal GMM distribution is updated each step after initialization from the source domain. What data samples are used for updating the GMM at each time step? Pseudo-datasets or unlabeled data points from the target domain? Also, in the Algorithm 1, for $t=1$, how to get $p^1_J$ for pseudo-dataset sampling in line 13? I assume $p^1_J$ at $t=1$ should be the same distribution as $p^0_J$ for the source domain. \n3. Memory buffer are used and accumulated to store the representative data points from each domain. However, since lifelong continual learning takes unlimited data, the memory buffer will also become extremely large after longtime accumulation. Thus, the memory buffer strategy is not preferred in practice. Also, the paper stores 10 samples per class per domain, but how is this number determined? Since this number is quite important for the performance (obviously larger buffer will have better results), I think some explanation is needed. \n4. Although horizontal comparisons are made with other UDA methods in two-domain setting, however, since the motivation of the paper is also about improving CL method, I think some comparisons with other classic CL methods are needed in order to demonstrate the advantage of the proposed method for lifelong continual learning and sequential domain adaptation. \n5. As Results explained, “A balanced dataset makes the empirical distribution a less biased approximation of the real distribution.” as shown in Table 2. This shows a limitation of GMM internal distribution estimation: this kind of distribution estimation is sensitive to class-wise balance. So I’m a bit concerned about the performance of the method on unbalanced datasets, which is always the case in practice. \n6. Two loss terms are added from Eq(3) to Eq(4), however there’s no ablation study for each loss term in the experiments. More ablation study for the loss is definitely needed in order to show the advantage of adding memory buffer terms in the loss function. \n\nDespite the above questions, the paper presents a valuable task combining both UDA and CL, and domain shift is the main challenge for lifelong unsupervised learning which is mitigated in this paper. So I think the paper lies on the borderline and look forward to the authors' response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper mentions some limitations in the results section. 1. Because of the distribution estimation, the algorithm works well on balanced datasets but the performance degrades when the dataset is unbalanced. (shown in Table 1-2); 2. As mentioned in line 266-268, the method has a possible improving direction to benefit from class-conditional alignment techniques. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers identified a number of strengths and weaknesses in this submission. The authors response led to a detailed discussion, over the course of which some of the reviewers increased their scores. Ultimately, three of the four reviewers recommend acceptance, with the four still being in the borderline range. I see not reason to overrule this vote and therefore recommend acceptance.
Decision: Accept (Poster)