Online Continual Learning  Under  Domain Shift
Authors: Quang Pham
Chenghao Liu
Steven HOI
Venue:  (2021)
Keywords: Continual Learning
Domain Generalization

Abstract: Existing continual learning benchmarks often assume each task's training and test data are from the same distribution, which may not hold in practice. Towards making continual learning practical, in this paper, we introduce a novel setting of online continual learning under conditional domain shift, in which domain shift exists between training and test data of all tasks: $P^{tr}(X, Y) \neq P^{te}(X,Y)$, and the model is required to generalize to unseen domains at test time. To address this problem, we propose \emph{Conditional Invariant Experience Replay (CIER)} that can simultaneously retain old knowledge, acquire new information, and generalize to unseen domains. CIER employs an adversarial training to correct the shift in $P(X,Y)$ by matching $P(X|Y)$, which results in an invariant representation that can generalize to unseen domains during inference. Our extensive experiments show that CIER can bridge the domain gap in continual learning and significantly outperforms state-of-the-art methods. We will release our benchmarks and implementation upon acceptance.

Under review as a conference paper at ICLR 2021
ONLINE CONTINUAL LEARNING UNDER CONDITIONAL DOMAIN SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
Existing continual learning benchmarks often assume each task's training and test data are from the same distribution, which may not hold in practice. Towards making continual learning practical, in this paper, we introduce a novel setting of online continual learning under conditional domain shift, in which domain shift exists between training and test data of all tasks: Plr(X,Y) * (tc(X, Y), and the model is required to generalize to unseen domains at test time. To address this problem, we propose Conditional Invariant Experience Replay (CIER) that simultaneously retain old knowledge, acquire new information, and general- ize to unseen domains. CIER employs an adversarial training to correct the shift in P(X,Y) by matching P(X which results in an invariant representation that can generalize to unseen domains during inference. Our extensive experiments show that CIER can bridge the domain gap in continual learning and significantly outperforms state-of of-the-art methods. We will release our benchmarks and imple- mentation upon acceptance.
INTRODUCTION
Continual learning is a promising framework towards human-level intelligence by developing mod- els that can continuously learn over time (Ring) 1997 Parisi et al. 2019). Unlike traditional learning paradigms, continual learning methods observe a continuum of tasks and have simultaneously per- form well on all tasks with limited access to previous data. Therefore, they have to achieve good trade-off between retaining old knowledge (French) 1999) and acquiring new skills, which referred to as the stability-llastici dilemma (Abraham & Robins, 2005). Continual learning is not only challenging research problem but also has tremendous impacts on many applications (Diethe al.] 2019). In particular, the deployed model may encounter new problems over time, and re-training every time new data arrive is infcasible, especially for large, complex neural networks.
Despite recent success, existing continual learning benchmarks assume the data of each class drawn from the same distribution during training and testing, i.e. plr(X, = pte(,, Y). This restrictive assumption is unlikely to hold in practice and prohibits the use of continual learning strategies in numerous real-world applications. For example, consider a continual learning agent already trained recognize certain objects in the indoor environment; then, it moves to new outdoor environment to acquire new skills. The agent may fail to recognize the learned objects because they are placed in a completely different background. Such an environment change is referred to as domain shift (Khosla et al. 2012] Long et al. very natural for continual learning in practice (Diethe et al.,
Diethe et al.) 2019). and is 2019p. To formalize the continual
2015
learning under domain shift problem, we need to consider the interaction between domain and the observation. In many computer vision applications, the causal structure usually Y X, i.e., the object class is the cause for image features (Lopez-Paz et al. 2017). This setting ignores the effect of conditional domain shift (Zhang al. 2013). where both domain and the object class are the causes for the image features. That is, images may come from different domains, but they have the same label. Figure [ shows the causal graph for continual learning under conditional domain shift problem and an illustrative example of the domain shift between training and testing data.
Continual learning under conditional domain shift requires the model to perform previous tasks in new domains, which poses great challenge since we do not know in hindsight which domain will be tested. Therefore, the model needs to learn the concepts presented in the data while ignoring the domains. For example, placing object in different background does not change the label of that

---

Under review as a conference paper at ICLR 2021
Training images
Testing images
n Et
bicycle airplane
n Vt
d
bird
N
T
clipart
infograph
painting
quickdraw
real
sketch
(b)
Figure I: (a) A causal model for the online continual learning under conditional domain shift. T the number of tasks and N₁ is the number of training samples task t. The causal interaction between the domains and images domain shift, sampled images are extracted from the DomainNet dataset (Peng et al. 2019)
considered in the traditional setting. (b) An example of the
D2.
Shotte 2M,
ae!
Task I
Sampling
Episodic Memory
Label predictor A
Task 2
Data for Task
Domain critics (1))
Task
Bhat
D6
Data Sources
Feature extmactor 0g
JCg
Forward pass
Backward pass
Critics' training loop
Figure 2: Overview of Conditional Invariant Experience Replay (CIER). CIER employs a set of domain critics (blue) to learn conditional invariant representation via an adversarial training loss in every experience replay round. Best viewed in colors.
object. As result, the model needs to achieve an invariant representation using data from various source domains observed during training, i.e. the joint distribution P(X, Y) is the same for all source domains. To this end, we develop Conditional Invariant Experience Replay (CIER) that can simultaneously retain previous knowledge, learn new tasks, and generalize new domains. CIER employs episodic memory perform experience replay and an adversarial loss to achieve an invariant representation. Particularly, CIER formulates a multiplayer minimax game such that the conditional distribution P(X is stable across each class's observed source domains. Therefore, if the prior distribution P(Y) is stable in the target domains, i.e., no class imbalance, CIER can achieve the invariant representation in the joint distribution P(X, Y) and generalize to unseen domains. Fig. 2 provides a high level overview of the proposed CIER method.
In summary, formalize the continual learning under conditional domain shift problem and struet three novel benchmarks using real data with different levels of domain shift and diversity the number of domains, tasks, and classes. Then, we develop CIER, a novel continual learning method that learns a conditional invariant representation and generalizes to novel domains. Our extensive experiments demonstrate the limitations of existing continual learning methods when being tested on unseen domains and show that the proposed CIER can mitigate such domain gaps effectively.
2 RELATED WORK
2.1 CONTINUAL LEARNING
Continual learning aims at developing model that can continuously learn different tasks over a data continuum. In literature, there are different continual learning protocols with different properties of the continuum. First, a setting can be either task-free (Rebuffi et al. 2017; Aljundi et al., (019b) or task-aware (Kirkpatrick et al. 2017 Lopez-Paz & Ranzato 2017) based whether task-indicator
2

---

Under review as a conference paper at ICLR 2021
is presented to the model. Second, learning can follow the online (Lopez- Paz & Ranzato, 2017) or batch (Kirkpatrick et al., 2017) setting. In online continual learning, the data within each task is also online and the model is only allowed to learn each example once. Differently, in batch continual learning, the model is allowed to learn a task over many epochs. Generally, the task-free and online settings are more difficult because the model has to predict on all tasks, and optimizing deep neural network online over one epoch is particularly challenging (Loezz-Paz & Ranzato 2017 Sahoo et al.) 2018 Aljundi al., 2019a). However, we argue that existing protocols do not consider the domain shift between training and testing data of each task, which is important in real scenarios. This limitation motivates us in developing the continual learning under conditional domain shift problem as practical and more challenging problem to evaluate continual learning methods.
Continual learning methods aim at achieving a good trade-off between alleviating catastrophic for- getting and acquiring new knowledge. Existing methods can be broadly categorized into two main groups: (i) dynamic architecture approaches that employ a separate subnetwork for each task, and (ii) static architecture approaches that maintain a fixed network throughout learning. While dynamic architecture methods (Rusu et al.] 2016, Serra et al. 2018: von Oswald et al.) 2020; Yoon et al,
2018 Li et al., | Xu & Zhu 2018) usually do not suffer from catastrophic forgetting, they not practical because of the high computational cost. In the fixed architecture method family, experi-
ence replay (ER) (Lin, 1992) and its variants (Chaudhry al., 2019b] Aljundi et al. 2019a Riemer let al., 2019] Rolnick et al.) 2019) have recently gained much interest thanks to their strong per- formance across different continual learning protocols, significantly outperforms other approachecs such as regularization (Kirkpatrick et al. 2011 Zenke et al. [217 Aljundi et al. 2018 Ritter et al.) 2018). representation learning (Rebulli et 2017). However, existing continual learning methods lack the ability to generalize to unseen domains beyond training, which motivates us to develop a novel method of CIER that can achieve an invariant representation in continual learning.
2.2 DOMAIN GENERALIZATION AND DOMAIN ADAPTATION
Domain generalization aims at training a model that can generalize to a target domain given data from the source domains (Khosla et al. 2012 Li al. 2017 2018 Carlucci et al. 2019). As sim- pler problem setting, domain adaptation assumes unsupervised data from the target domain is also given during training (Long et al. 2015; Ganin & Lempitsky| 2015). A general approach for such problems is learning an invariant representation by correcting the shift in P(X, Y) across domains. Particularly, (Long et al., 2013 Ganin & Lempitsky 2015) assume that the conditional distribution P(Y is the same across domains; therefore. the domain shift can be addressed by matching only P(X). However, invariant in P(Y|X) is a strong assumption and is unlikely to hold in many ap- plications. A more robust approach is matching the conditional distribution P(XIY) (Zhang et al.) 2013 Li et al. 2018). In which case, if the class-prior distribution P(Y) is stable, i.e., there is no class-imallance among domains, matching PYX will result in the invariant in P(X, Y) because of the decomposition P(X, Y)= = P(Y) P(X))) We emphasize that no imbalance data is a mild assumption, especially when do not know about the target domains. This result motivated us in the development of CIER. However, different from such methods, CIER can accommodate novel classes and domains over time while maintaining good performances over all classes
3 PROBLEM FORMULATION
3.1 ONLINE CONTINUAL LEARNING UNDER CONDITIONAL DOMAIN SHIFT
Notations. Let X C D and C N be the input and label space respectively. domain is defined as joint distribution P(X, Y) over X' x y, and the k-th domain (X,, Y') is denoted as pk: Often, we do not have access to the domain pk instead, we can label each data point with a scalar d E N indicating that it comes from domain d. During training, learner O receives a continuum of data {z,, d4, J. where (z,, y1) is the i-th training sample and d, is the domain indicator associated with {æ,, 34 In the following, we use (t) as the number of observed classes until time t; dyy (t) denotes the number of observed domains of class y until time t; and Y denotes the samples of the j-th class and has domain dy.
We consider the fask-free, online continual learning setting where the continuum is locally iid, i.. each training sample (æ 9:) is drawn iid from an underlying task distribution P., (X, Y) with the task identifier 1, not revealed to the learner. Moreover, each task consists of subset of labels sam-
3

---

Under review as a conference paper at ICLR 2021
pled without replacement from the label space y. To formalize the online continual learning under conditional domain shift problem, we assume that each class y is associated with a set of source domains Di = {dite dere ,2. ,...,,, d. and a set of target domains y = {d dest. Additionally, the source and target domains are disjoint for all classes, i.e. Di n Digt = 0. Vy. We consider three scenarios with decreasing difficulties to evaluate the learner 0 based the degree of shift between the training and testing domains.
d'gt }.
Scenario 1 (significant shift):
Train on source domains, test target domains, the most challenging setting where all domains of the test data are unobserved during training.
Scenario 2 (mild shift): Train source domains, test both source and target domains. Here we assume that domains of test data can be draw from both source and target domains.
Scenario (no shift): Train target domains, test on target domains. This is the traditional continual learning setting, where there is no discrepancy among domains of training and testing data. We design this scenario so that it has the same test data as the first one.
In all scenarios, the model is required to perform well on the test data. We emphasize that the contin- ual learning setting we considered is the most challenging in the literature: both task and data arrive online, and the model has to predict all observed classes so far during evaluation. Moreover, we provide the model with the domain identifiers during training; however, only the imput z is provided at test time. Finally, we aim at creating a model that can predict at any moment, therefore, we do make any assumption regarding the testing domains, including the availability of unlabeled data, and the models are not allowed to train during inference.
3.2 EVALUATION PROTOCOT.
We use three standard metrics to measure the model's performance: Average Accuracy (Lopez-Paz & Ranzato 2017), Forgetting Measure (Chaudhry et al.) 2019a), and Learning Accuracy (Riemer al. 2019) Let '' be the total number of tasks, ag,j be the model's accuracy evaluated on the test set of task Ty after it finished learning the task T; The aforementioned metrics are defined as:
Average Accuracy (higher is better): ACC + Ei= @T
Forgetting Measure (lower is better): FM 1 T-1
T-T Ej=1 maxie(L...TT-I) 01 argj-
Learning Accuracy (higher is better): LA 1 ΣT= Czf
The Averaged Accuracy (ACC) measures the model's overall performance across all tasks and is a common metric to compare among different methods. Forgetting Measure (FM) measures the model's forgetting as the averaged performance degrades of old tasks. Finally, Learning Accuracy (LA) measures the model's ability to acquire new knowledge. Note that the task identifiers are not given the model at any time and are only used to measure the evaluation metrics.
4 PROPOSED METHOD
4.1 CONDITIONAL INVARIANT AS A MULTIPLAYER MINIMAX GAME
Learning invariant representation is equivalent correcting the shift in the joint distribution Y) = P(Y) However, is often expensive to obtain the join distribution; there- fore, we propose learn conditional invariant in the conditional distribution P(X Then if the class-prior distributions P(Y)) satisfy p"'()) can expect the model to generalize unseen target domains. Therefore, we propose to learn a conditional invariant deep network Fo(-) using only data from source domains by enforcing pk (P((X))))= = ('F(())) Yk, Dere.
(YY),, Vm E Dere, VI E Díst, we
To achieve this goal, we formulate learning conditional invariant feature extractor as a multiplayer minimax game. Particularly, each class y/ is associated with a domain critic K, predicting the domain associated with each sample of y/- Therefore, we propose the following minimax problem:
B(4) Veonal ( FF P,K,),
min max F 0² 0'..... (2(())
(1)
4

---

Under review as a conference paper at ICLR 2021
where (eond(F, K)) is a domain discrepaney measure for class yj. At time t, Eq. I is (y(t) + pl-players minimax game with y() crities trained to distinguish the features generated from the feature extractor F. while F is trained simultaneously maximize all critics' losses. In this work, we consider two versions of the domain discrepancy measure Vonnd- First, we consider the Jensen- Shannon divergence (JSD), in which case the domain discrepancy is calculated as Veona (, Ký) EmH EawY y log Kj (F)())). When using JSD, the domain critics K are implemented
softmax
layer, i.e. Ee1 (1) (z) 1, Vj = ..... ..... y(t). JSD is natural choice the domain divergence as has been successfully applied in other applications, e.g. (Goodfellow et al. 2014 et al. 2018).
Another popular choice to measure domain divergence is the Wasserstein distance between two distributions (Villani 2008). Different from JSD, the Wasserstein metric has much smoother value range, which may provide more meaningful gradients to update the feature extractor. This motivates us to use the Wasserstein distance as the domain divergence in the minimax game in Eq. | In practice, estimating the Wasserstein-p distance for arbitrary value of p highly computationally expensive. Fortunately, the Kantorovich-Rubinsee compute the Wasserstein-1 distance, which we refer the Wasserstein distance throughout the rest of this paper. The duality of the Wasserstein-1 distance can be efficiently computed as:
duality (Villani 2008 provides an efficient way
W (w, = sup Ex-af fXx=
(2)
11
where the supremum is taken over all Lipschitz continuous functions f M - with Lipschitz constant less than or equal to one. The total domain divergence of class Vj is calculated by summing the Wasserstein distances of all possible source domain pairs:
W Woond (F, K2)
sup | K1
Estimating the Wasserstein distance in Eq. requires taking the supremum over all I-Lipschitz func- tions of the critics. Here we follow the strategy of Arjovsky et al. (2017) to enforce this constraint by clipping the critics' parameters to be pre-defined range.
4.2 CONDITIONAL INVARIANT EXPERIENCE REPLAY (CIER)
We now describe Conditional Invariant Experience Replay (CIER), which learns a conditional in- variant feature representation of the data for continual learning. CIER utilizes an episodic memory to store and replay past data during learning new samples by minimizing the classification error and learning the conditional invariant features. To achieve the conditional invariant property, propose to optimize the minimax objective Eq. [ Taach time a new sample arrives. Moreover, because pre- vious data might not be fully available when learning new tasks, we restrict Eq. to consider only data in the episodic memory. Therefore, CIER always learns conditional invariant features of all
(1(1) classes and source domains observed so far. To formalize CIER, let (0)))() be the parameters of vtt) domain critics K,, f be the feature extractor, and 0, be the classifier. Then, the classification loss Leis(0r ( 0z) is the standard cross-entropy between the true and predicted labels. We propose two variants of CIER: CIER-ISD using the JSD, and CIER-W W using Wasserstein as the domain divergence. The critics' losses La can be empirically estimated
4,(0)
JSD: : CLL Σ j
log Koy (Fo, (1)) ;
Σ
Kgy (z) (3) =1, 1,
(4)
BYYY
: (.
Σ
Σ Likk
(F),
Σ
Koj (Fo,
N
w>yk
where N₁. denotes the number of samples from class j that belong domain & stored in the episodic memory. Let Liotal = aLas (0,, 0.) + (1 (0 (0.) be the total loss with the trade-off parameter a. When incoming data arrive, the whole network can be optimized by a two-stages
5

---

Under review as a conference paper at ICLR 2021
Table 1: Summary of the benchmarks used in our experiments
#src domains
#igt domains
#total domains
#train imgs
#test imgs
#classes
iAnimal
23
10
33
4,500
1,500
10
iVehicle
28
11
39
4,000
1,290
9
iDomainNet
3
205,141
88,204
345
optimization process. First, the crities trained maximize the critic loss in Eq. Eq. 5 In the second stage, the classifier and the feature extractor are trained to correctly classify the current data as well learning conditional invariant features. Importantly, the feature extractor 0 trained to minimize the domain's ss discrepancy measure while the critics are trained to maximize it, which implements the minimax game to learn conditional invariant features described in Sec. 4.1 Notably, calculating the critics loss La requires y(t) forward passes, however, only one backward pass is required to update the critics' parameters. We provide the pseudo-code of both CIER-JSD and CIER-W in Appendix
5
EXPERIMENT
5.1 EXPERIMENTAL SETUP
Benchmark. We consider the task-free, online continual learning setting as described in Sec. the model observes a continuum data consists of T' tasks, where the input each time step is tuple of an image, domain indicator, and label. At test time, the model only observes the images and it has to predict on all observed classes so far. We consider two datasets: NICO (He et al. 2020) and DomainNet (Peng et al. 2019), and create three continual learning benchmarks in our experiments. The NICO dataset consists of two superclasses of Vehicle and Animal with and 10 categories; each has total of approximately 10,000 images over 30 domains. We create continual learning benchmark for each superclass using its subclasses, resulting in the incremental Vehicle (iVehicle and incremental Animal (iAnimal) benchmarks. Similarly, we take DomainNet, large scale dataset of 345 classes with 569,010 images over six domains, and create the iDomainNet benchmark.
To ensure the domain shift between training and testing data, for each class, we randomly choose 50% of its domains as the source domains for training: the remaining domains are considered as target domains. The source and target domains are used to construct three continual learning sce- narios as described in Sec. 3 significant shift, mild shift, and no shift. All images are resized, center-cropped to 3 x484 84 and normalized to be in [0, 1). No other data augmentation is used in all experiments. Table [ summaries the statistic of each benchmark. We can see that while the NICO dataset consists of many domains, only has a small number of classes and a moderate amount of training data. On the other hand, the DomainNet dataset has a large number of classes and training images but only contains six domains. Consequently, our continual learning benchmarks are diverse in both number of training instances, domains, and tasks.
Baselines. Since there have not been any approaches in the literature that directly address the contin- ual learning under domain shift problems, consider three -tate-of-the-ar online continual learn- ing methods, namely ER (Chaudhry et al. (019b)). MER (Riemer et al. 2019), and MIR (Aljundi) ct al. 2019a). Furthermore, we also propose baseline called Adversarial Experience Replay (AER) that augments ER with an adversarial domain classification loss to match the marginal distribution P(X). AER is principally different from our proposed CIER since CIER matches the conditional distributions P(XJY) across domains. Due to space constraints, we refer to our Appendix B for the results of more baselines.
Implementation Details. We use a pre-trained ResNet18 (He et al., 2016) backbone in all bench- mark. All methods use a Reservoir sampling buffer with sizes equal to 9,000; 10,000; and 34,500 samples in the iVehicle, iAnimal, and iDomainNet respectively. All methods are optimized online (over one epoch) using SGD with mini-batches of size 10 on i'ehiicl, iAnimal, and 32 on iDo- mainNet. We cross-validate the hyper-parameters of all methods on small subset of data (5%) using grid search. Different from (Aljundi et al.) 2019a), we follow the standard implementation in (Chaudhry et al. 2019b), which concatenates the current and previous data when calculating the perience replay loss. This results in the naachhhommmlizttionn layer applied on both old and new data in the forward pass instead of the old and new data batch-normalized separately. We observe that

---

Under review as a conference paper at ICLR 2021
Table Evaluation metries on the iVehicle and iAnimal benchmarks, respectively. All methods use the same pre-trained Resnet18 backbone. Scenario (significant shift): train on source domains, test on target domains. Scenario (mild shift): train on source domains, test on source and target domains. Scenario 3 (no shift): train on target domains, test target domains.
Veehille 3 tasks
iAnimal 5 tasks
Scenario
ACC
FM
LA
ACC
FM
LA
MER
22.56
63.20+1.43
46.38=1.00
31.50+22.05
71.62=1.44
ER
48.70:10.90
23.22::3.15
64.26:t1.31
45.76110.8
34.12::1.46
73.08
MIR
42.00
30.00::2.28
6..88
44.02:10.91
41.96:1:.55
1..5
AER
08.96::0.39
2..30+:..2
64.46::0.98
1.49
32.82+22.02
72.24=0.99
CIER-JSD
51.52:10.59 19.6211.98
1.27
49.04
31.2011.116
1..0
CIER-W W
51.38
22.0442.63
6..08
48.3610.67
31.02+1.77
13.16
Scenario 2
ACC
FM
LA
ACC
FM
LA
MER
16.90++1.05
18.90+1.400
69.22++:.21
52.66+1.53
28.28+11226
75.30
ER
16.30
21.0412.44
7..34
51.60-0.80
31.68:1.52
1..9
MIR
55.46:10.88
2..24+1.34
72.30-L0.34
49.1611.52
40.52+1.32
0.99
AER
1..14
1..78+1.60
7..32
49.98-10.8
33.1011.18
76.44400.0
CIER-JSD
09.4
18.82422.0
71.98
53.8610.72
31.30+1.46
78.90=1.05
CIER-W W
58.20
2..56
71.92+1.31
54.66+2.51
29.08
07.92
Scenario
ACC
FM
LA
ACC
FM
LA
MER
63.144-0.84
20.28 : 1.61
76.70+
6..74++0.62
26.42++1.14
82.88
ER
61.80+-0.94
1.94
78.10+1.07
59.32
30.20+0.800
83.48
MIR
59.08
30.9842.33
79.76+2.33
58.32+1.31
35.68+1.26
86.88
AER
62.82:10.87
20.6412.50
1..5
59,5210.53
28.8411.70
82.56-10.97
CIER-JSD
65.78+1122
23.6242.06
81.54
63.78
26.1011.711
83.56+2.06
CIER-W
64.82+1.28
24.10+:1.33
80.90+
0.83
29.30+10.00
1.26
concatenating current and episodic memory during for experience replay yields better results for all methods; therefore, we will use this implementation the main paper. We provide the results of the other implementation with batch-normalization in the Appendix B.2
5.2 RESULTS ON THE IVEHICLE AND IANIMAL BENCHMARKS
We create a sequence of three tasks for the iVehicle and five tasks for the iAnimal benchmarks and report the results in Table 2 Because of the standard ER implementation with natch-normalization MIR performs slightly worse than ER across all settings. Under the significant domain shift, the performance of existing methods drops significantly compared to the traditional setting with no domain shift. Both ER, AER, CIER-JSD, and CIER-W has similar FM, indicating that they are equally affected by catastrophic forgetting. Moreover, CIER-JSD and CIER-W W achieve higher LA, resulting in higher overall performance, indicated by the ACC metric. The results show that CIER can achieve a good trade-off between preventing catastrophic forgetting and facilitating knowledge transfer while being able generalize to unseen domains.
5.3 RESULTS THE IDOMAINNET BENCIMARK
We construct a sequence of five and 15 tasks for the iDomainNet benchmark and report the results T We also observe a significant performance drop across all continual learning methods. Moreover, since iDomainNet only has three source domains. it is much more challenging to learn an invariant representation. Consequently, we observe a slightly small gap between CIER and ER compared to the iVehicle and iAnimal benchmarks: 1% versus 2-4%. This result suggests that invariant representation learning requires more source domains to obtain a powerful representation that can generalize well at test time. Different from iAnimal and iVehicle, CIER-W consistently achieves the best performance in iDomainNet, showing that the Wasserstein distance is a better choice for the domain divergence when there are only small number of source domains. When there are many source domains, it is challenging to extend the Wasserstein distance to multiple distributions while the JSD can be naturally extended to such cases.
7

---

Under review as a conference paper at ICLR 2021
Table Evaluation metrics on the iDomainNet benchmarks with five and 15 tasks, respectively. All methods use the same pre-trained R 8 backbone. Scenario (significant shift): train source domains, test on target domains. Scenario (mild shift): train on source domains, test on source and larget domains. Scenario 3 (no shift): train on target domains, test target domains
iDomainNet - 5 tasks
iDomainNet 15 tasks
Scenario
ACC
FM
LA
ACC
FM
LA
MER
13.32=00.15
9.66:10223
20.92=0.24
12.52+0.221
21.74:1021
32.82-+0.18
ER
15.06:10.18
9.88+1052
2.29
12.60-10.22
22.56:10.23
34.62:1..2
MIR
15.1
10.26:10.29
23.40=0.16
12.28
23.14:0.13
34.8
AER
15.04=0.17
9.7611022
22.88=0.19
12.30-10.20
22.2011035
33.02:10366
CIER-ISD
15.88=0.20 10.52::0.10
24.26=0.23
13.3410.09
22.28+0.30
34.9010.24
CIER- W
16.02.10.15
10.72.0..9
24.66=0.29
13.5410.19
22.4440.22
35.1810.100
Scenario 2
ACC
FM
LA
ACC
FM
LA
MER
23.96=#0009
13.06+0.11
3..4
23.28:H0.15
25.98
07.24
ER
26.96:10.09
12.50+:0133
36.92=0.12
23.9811021
25.9011.24
49.16:10.19
MIR
26.88
12.68
37.04=0.18
23.8010.10
26.72+0.10
49.70+0.10
AER
26.62.10.00
12.08
36.28
22.6811.110
26.52110.1
47.44
CIER-ISD
27.16=00117 13.14
38.48=0.07
24.74410004
26.12+0.009
49.82+10.07
CIER-W W
28.16=0.18
03.24
38.74
24.9210.22
26.04410.29
49.9210.09
Scenario 3
ACC
FM
LA
ACC
FM
LA
MER
39.06-+0.11
0.22
5.26
37.30+
32.46
0.
ER
0..12+0000
17.06++0.12
54.80-0.14
38.42+0.23
33.14+0.30
70.34+0.09
MIR
10.88=0.14
55.00=0.10 17.66+0.20
37.8410.41
34.1010.48
70.6410.09
AER
40.62.10.04
17.28100.11
54.44=0.24
34.2411.11
35.6210.13
67.50110.15
CIER-ISD
43.74=0.17
17.68+00.25
57.86=0.17
38.62+0.32
33.66:10.33
7..44
CIER-W
03.9410.70
17.78+0.10
58.16=0.90
39.12+1.21
33.34-0.26
71.0410.13
Finally, we note that all methods have lower ACC when moving from 5 to 15 tasks. Since both cases have the same amount of training data; having more tasks means each task is easier to learn, resulting in higher LA. However, learning on long sequence of tasks also suffers significantly from catastrophic forgetting indicated by higher FM, resulting lower final performances.
5.4 MODEL AND COMPUTATIONAL COMPLEXITY
Table 4: Model and computational complexity of CIER compared to ER and MIR. Computational complexity is the averaged training time per incoming batch, all methods have very small stddev, i... smaller than 0.0ls
ER
MIR
CIER-ISD
CIER-W
# params
Time
params
Time
# params
Time
# params
Time
iVehicle iAnimal
11,181,129
0.42s
11,181,129
0.53s
11,443,013
0.53s
11,380,562
0.48s
11,181.642
0.42s
11,181,642
0.54s
11,437,872
0.54s
11,381,332
0.48s
iDomainNet
11.353,497
0.57s
11,353,497
0.59s
11,816.612
0.87s
11,639,282
0.68s
We study the model and computational complexity of CIER, and report the number of parameters and the averaged training time per incoming data on all benchmarks in Table Among all methods, we observe significant additional computational cost the iDomainNet benchmark. The overhead incurs because of the larger batch size and the larger number of classes: 345 classes compared to CIER-JSD, each class's 's critic is a D-ways classification
nine in iVhehicle and 10 iAnimal. model where D is the number of domains observed, which results in more expensive parameters and computational complexity. On the other hand, CIER- RR's critics only predict domains scores, which scalars. Thefore, CIER-W enjoys much lower complexities compared CIER-JSD. However,
8

---

Under review as a conference paper at ICLR 2021
the complexity overhead in both methods significant compared to ER and especially MIR. Particularly, CIER-W W is only 15%-20% slower than ER, and has 1.7%-2.5% additional parameters.
5.5 SUMMARY
From the experiments, observe that all continual learning methods suffer when domain shift happens. Second, having many source domains important in achieving an invariant representation, as demonstrated by the gap of CIER compared other methods from the considered benchmarks. Moreover, the Wasserstein distance is powerful when learning with a small number of source do- mains while the Jensen-Shannon divergence (JSD) is better choice when more source domains are available. Finally, while CIER-JSD and CIER-V W achieved promising results, there is still a gap compared to the ideal scenario of having no domain shift between training and testing domains.
CONCLUSION
We address the limitations of existing continual learning methods when testing on unseen domains with domain shift. This motivates to develop a novel problem setting named Continual learning un- der domain shift and three novel benchmarks, which support diverse range of domains, classes, and tasks. To address this challenge, we propose CIER that simultaneously performs experience replay maintain a good performance across all tasks and learns conditional invariant representation via an adversarial learning objective. As result, CIER balances among all three aspects: prevent- ing catastrophic forgetting, facilitating knowledge transfer, and generalization Through extensive experiments, we demonstrate that existing state-of-the-a ofttte-rrttt-ttt-- methods are often too optimistic and struggle to generalize to unseen domains. On the other hand, CIER can bridge the gap in the domain shifts and offer significant improvements. We believe our study provides more realistic setting to evaluate continual learning methods and opens new future research directions.
unseen domains.
REFERENCES
Wickliffe C Abraham and Anthony Robins. Memory retention-the synaptic stability versus plastic- ity dilemma. Trends in neurosciences, 28(2):73-78, 2005.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 139-154, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Cacciaa Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 11849-11866 2019a.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11254-11263, 2019b.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. wrXiv:1701.07875 2017.
Wasserstein
gan.
arXiv preprint
Fabio M Carlucci, Antonio D' Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference Com- puter Vision and Pattern Recognition, pp. 222992233, 2019.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. International Conference on Learning Representations (ICLR), 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc Aurelio Ranzato. On tiny episodic memories in continual learning, arXiv preprint arXiv: 2019b.
Tom Diethe, Tom Borchert, Eno Thereska, Borja Balle, and Neil Lawrence. Continual learning in practice. arXiv preprint /x03v09903005202 2019.

---

Under review as a conference paper at ICLR 2021
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, pp. 107383, 2020.
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In European Conference on Computer Vision, pp. 158-171. Springer, 2012.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barrskkka, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning. pp. 3925- 3934, 2019.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-321, 1992.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Dis- covering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6979-6987, 2017.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1803. X1vx11803.00999 2018
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision pp. 1406-1415, 2019.
10

---

Under review as a conference paper at ICLR 2021
Sylvestre-Alvies Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. International Conference on Learning Representations (ICLR). 2019.
Mark Ring. Child: first step towards continual learning. Machine Learning, 28(1):77-104, 1997.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, pp- 3738-3748, 2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pp. 348- 358, 2019.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatriek, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 046771 2016.
Doyen Sahoo, Quang Pham, Jing Lu, and Steven C. H. Hoi. Online deep learning: Learning deep neural networks the fly. In Proceedings of the Sweett-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, 2018.
Joan Serra, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. Proceedings of the 35th International Conference on Machine Learning- Volume 80, pp. 4557. org, 2018.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. International Conference on Learning Representations (ICLR), 2020.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Pro- cessing Systems, pp. 899-908, 2018
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. International Conference on Learning Representations (ICLR), 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 3987- 3995. JMLR. org, 2017.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pp. 819-827, 2013.
11

---

Under review as a conference paper at ICLR 2021
This Appendix is organized follows. We deseribe the pseudo-code of the CIER algorithms in Appendix A Then, A provides the details our implementation used in the main paper and additional results of more basclines and model complexity. Finally, we include some illustrative examples extracted from our benchmarks in Appendix
A THE CIER ALGORITHMS
Alg. | depicts the training procedure of the proposed CIER-JSD and CIER-W. All experiments conducted in this work use the hyperpameter provided in [ For fair comparisons, optimize all methods, including CIER and other baselines, using two updates per samples.
Algorithm I: Our proposed CIER algorithm. All experiments in this paper used the default values Tinnne Reritic 2, a 0.95, c 0.01.
Require: a, the trade-off parameter. c, the clipping parameter. N, the replay batch size, Nister number of iterations per incoming training data.
Init: fy the deep network feature extractor parameters. 0., the classifier parameters. (0))=- the critics parameters of |Y| classes. M the episodic memory with reservoir sampling.
for f <-- to do
for j < to bbatches do
I/ Receive data of task T sequentially Bj = (2c, d, 3). receive a mini batch of data from the current task
for
to Minner do
// Inner update of the current sample
MM = (x' .''.'vv) random sampling from the episodic memory M
BM + By UB
// Mixing data for experience replay // Training critics loop
for n +- 0 to Maritic do Calculate the critics' loss Lw on BMM, using Eq. or Eq.
8
9
0 < 0 + 00
aLa Vi = 1...,,,, |Y|
Domain critics SGD step
CIER-W: 0 + clip(0'). c, c), Vi = I,..., |
|/ Parameters clipping
11
Calculate the total loss Leonal and the classification loss Lets on By
Phrcra a0, 0 - Of -0,
// Feature extractor SGD step
0. + 0. ahatz 00.
// Classifier SGD step
Add B, to M using reservoir sampling
return 0,, 10 O.
15
B EXPERIMENTS DETAILS
In this section, we first details the baselines used in our experiments. Then, we prove the details of experience replay implementations using batch normalization, and highlight the differences between them. For comparison, also include the results of both implementation the iVehicle and iAnimal benchmarks. Finally, we provide the results of more continual learning methods on the i'ehicle and iAniaal benchmarks.
B.1 EXPERIENCE REPLAY IMPLEMENTATIONS WITH BATCH NORMALIZATION
We first describe two experience replay (ER) implementations considered in this work. Let B, be the current data observed from the continuum and By be a random batch of data sampled from the episodic memory. For convenience, use to denote the total number of parameters in the model. including the feature extractor and the classifier. Then, an experience replay update can be implemented with or without concatenating B, and Bas as followings.
12

---

Under review as a conference paper at ICLR 2021
Batch Normalization with concatenation. This is the standard implementation of ER used in (Chaudhry et al. 2019b). The update step is described as:
B B BM,
(6)
Cet - B), 0 OG)
(7)
(8)
In this implementation, the batch normalization layer is applied on the merged batch B, which results in one forward pass, and the mini-batch mean and mini-batch variance being estimated once using the concatenation of the current data B, and old data B in the memory.
Batch Normalization without concatenation. This an alternative implementation of ER used in (Aljundi et al. 2019a). The update step is described as:
Lots + ( (,) + L,0,BM), SGD 0).
(9)
(10)
In this implementation, the loss is calculate on current and previous data separately. This results in two forward passes, and the mini-batch mean and mini-batch variance being estimated twice per gradient update: once on the current data B, and once on the past data BM-
Since ER is the base of many methods used in our work, including MIR (Aljundi et al. (019a). our CIER-ISD and CIER-W, W, this results in two alternative implementations of such methods.
B.2 RESULTS OF THE TWO EXPERIENCE REPLAY IMPLEMENTATIONS
We report the results of both experience replay implementations the iVehicle in Table 5 emphasize that these implementations, beside the difference concatenating the current and past samples, other factors exactly the same. Interestingly, subtle change in the use batch normal- ization can result in significant changes of the final results. One possible reason of this discrepancy is the use of the pre-trained ResNet, which was pre-trained by concatenating all samples in each mini-batch. In addition, when using BN without concatenation, we observe the same behaviour of MIR as in (Aljundi et al., (2019)): MIR improves ER by lowering forgetting measure FM, resulting in slightly better overall performance across benchmarks. Moreover, we still observe the same trend in both implementations that CIER can improve over ER and MIR in terms of LA, which results in better ACC. Nevertheless, implementing ER using BN with concatenation showed to to superior than the alternative. Therefore, we decided to report the result of this strategy in the main paper.
We
B.3 RESULTS OF ADDITIONAL BASELINES
In this section, we provide a broader set of baselines than the ones included in the main paper. First, we provide a short description all baselines considered.
Learning without Forgetting (LwF) (Li & Hoiem 2017). LwF prevents catastrophic
forgetting by a variant of knowledge distillation.
AGEM (Chaudhry al., 2019a). AGEM prevents catastrophic forgetting and transferring knowledge by preventing the old tasks' losses from increasing via a quadratic constraint. Since the original AGEM experimented with the online, task-aware setting, we extend AGEM to the task-free setting by enforcing the constraint on a randomly selected memory data of all observed classes.
Meta Experience Replay (MER) (Riemer et al. 2019). MER balances both catastrophic forgetting and facilitating knowledge transfer by penalizing the gradient direction of every samples stored in the episodic memory. MER implements this objective via a variant of the Reptile algorithm (Nichol et al. 2018).
Experience Replay (ER) (Chaudhry al. 2019b). ER is a simple, yet very effective strategy for the online, task- -ree continual learning problem. ER works by simply mixing current data and previous data in the memory and peforms gradient update.
Maximally Interfered Retrieval ER (MIR) (Aljundi al. 2019a). MIR improves ER by replacing the random memory sampling with a strategy to selects previous samples causing the most classification loss increase.
13

---

Under review as a conference paper at ICLR 2021
Table 5: Evaluation metrics of two experience replay implementations on the ieehicle benchmarks. BN denotes Batch Normalization. Scenario 1: train on source domains, test on target domains. Scenario 2: train on source domains, test on source and target domains. Scenario 3: train on target domains, test on target domains
iVehicle
BN with concatenation
BN without concatenation
Scenario
ACC
FM
LA
ACC
FM
LA
ER
48.70=E0.90
23.22:3.15
64.26+1.31
41.70=1.10
20.12::.10
55.04+2.95
MIR
45.88:12.00
30.00:12.28
65,88:11.44
1..9
20.04:12.01
55.30
CIER-JSD
51.52:10.59
19.62:11.98
6..78
43.26
20.66:13.29
57.02
CIER-W W
51.38+11.00
22.04+2..63
6..0
42.69=2.45
20.54:4.65
55.70=
Scenario 2
ACC
FM
LA
ACC
FM
LA
ER
56.30::1.29
21.0442.44
1..34
50.09
21.40::4.56
65.20=2.72
MIR
55.46++00.8
25.24++.34
0..3
50.66++2.71
16.60+22.61
61.74+2.86
CIER-JSI
0.84
18.82+2.80
71.98+1.81
1..8
2..3
2.33
CIER-W
18.20
2..5611.226
71.92:1.31
50.5612..1
20.8410.59
64.40
Scenario
ACC
FM
LA
ACC
FM
LA
ER
61.80:10.94
24.40
18.10
58.48
16.92.12.94
69. 69.78:12.39
MIR 59.08=:0.25
30.98+2.33
79.76+22.33
13.70:2.36 59.34+0.50
68.48
CIER-JSD
1.278
23.62=2.06 01.54
15.58=1.54 60.16=0.76
70.52 16.90++3.61
CIER-W 6..82+1.28 1..10+1.33
80.90+00.11
1..84
71.48 2.54
Table 6: Evaluation metrics of two experience replay implementations on the iAnimal benchmarks. BN denotes Batch Normalization. Scenario 1: train on source domains, test on target domains. Scenario 2: train on source domains, test on source and target domains. Scenario 3: train on target domains, test on target domains
iAnimal
BN with concatenation
BN without concatenation
Scenario
ACC
FM
LA
ACC
FM
ER
45.76:1-0.88
34.12+1.46
73.08
11.84-11.88
29.60-1..65
65.38
MIR
44.02+40.91
4..5
4..26 1.25
66.48 28.72+4422
CIER-ISI
49.94
1.16
43.52+0041 1..00+
29.69+0.61
67.48
CIER-W
W
48.66
31.02:1.77
13.16
42.99+1.33
31.76+11.99
67.66110.9
Scenario
ACC
FM
LA
ACC
FM
LA
ER
51.60+H0.80
31.68
76.92+1..74
43.90+3..7
32.72+3.99
70.06
MIR
19.16:
4..52+1.32
81.58
44.38
68.98 30.03:14.44
CIER-JSD
53.86:+0.72
11.30
78.90
44.48
33.52+3.00
7..68
CIER-W
54.66
29.08
77.92
45.14 33.90
1..24
Scenario
ACC
FM
LA
ACC
FM
LA
ER
59.32+0.91
30.20=0.80
83.48+0.74
15.14
25.46+1.50 1..4
MIR
18.32+1.31
3..6 86.8810.94
56.02+1.11
22.6242.22
1..78
CIER-ISD
63.78::1.13
26.10:11.71 83.5642.06
56.48+2.31
24.0812.69 73.94
CIER-W
62.32+0.80
29.30+0.90
85.68+1.20
56.16++2.26
27.08
1..54+1.40
Adversarial Experience Replay (AER) (this work). AER improves over ER by matching the distribution P(X) across domains. Particularly, AER employs a domain classifier to classify the domains from the backbone's features. The domain classifier is trained to maximize the domain classification loss while the feature extractor is trained to minimize it, resulting an invariant P(X). As we discussed in Sec. (2 this method corresponds to an invariant assumption in XYY]X); however, when this assumption is violated, matching P(X) may not provide additional benefits the traditional ER.
14

---

Under review as a conference paper at ICLR 2021
Table Evaluation metrics on the iVehicle and iAnimal benchmarks. All methods use the same pre- trained Resnet18 backbone. Scenario 1: train on source domains, test on target domains. Scenario 2: train on source domains, test on source and target domains. Scenario train on target domains, test on target domains. This is the full version of Table 2 in the main paper
Vehicle 3 tasks
iAnimal 5 tasks
Scenario
ACC
FM
LA
ACC
FM
LA
LwF
24.10:t1.60
55.86=1.44
61.38+1.80
16.44+1.60
78.4412.79
79. 1.31
AGEM
23.76:10.43
69.6811.32
70.18:11.19
1..34
85.961:2.05
84.14
MER
48.12:11.12
22.58:42.56
63.20::1.43
46.38
31.50:12.05
71.62:#1.41
ER
48.70=:.90
23.2243.15
64.26=#1.31
45.76=0088
34.12+1.46
73.08=1.64
MIR
45.88:2.00
30.0012.28
65.88
44.02:10.1
41.96+1.55
77.56+1.22
AER
08.96:0.39
2..30+1.32
64.46:10.98
45.98=1447
32.8242.02
72.24=0.99
CIER-JSD
0..5 19.62+1.98
6..78
49.04
31.10
74.00+1.553
CIER-W W
1..0
22.04+2.63
66.08+1.33
48.36+00677
3170
13.16+1.57
Scenario 2
ACC
FM
LA
ACC
FM
LA
LWF
29.76:10.86
4..96
73.08
14.28
83.62+1.20
11.18
AGEM
25.86:10.65
77.1
77.3411.20
15.16
88.3411.01
8.10
MER
56.90
18.90=1.40
69.22+1.21
52.66+1.53
28.2811.26
1..30
ER
16.30
21.04:2.44
70.34
51.60=0.80
31.68+1.52
76.92
MIR
5..4
25.24+1.34
72.30+F0.34
49.16+1.52
40.52+1.32
81.58
AER
55.14+1.00
2..78+1.60
7..3
49.9
33.10+1.18
7.644+0.60
CIER-ISD
59.44 18.8242.80
71.98
53.86110722
31.30
78.90
CIER-W W
58.20+1.21
20.56:2.226
71.92+1.31
54.66
29.08
77.92+0.75
Scenario
ACC
FM
LA
ACC
FM
LA
LwF
38.30
63.48
80.62+
17.64
85.02+22.58
85.62+
AGEM
1.78
11.52+1.59
82.24
16.60=0.57
91.8810.38
90.10=10.86
MER
63.14:10.84
2..28
76.70.L0.88
61.74.10.62
26.42:11.14
02.48
ER
6..90
2..40
78.10++.077
59.32-10991
30.20+0.80
83.48=10.74
MIR
59.08:10.25
30.9842.33
79.76+2.33
58.3241.31
35.6811.26
86.88
AER
62.82.0.87 20.6412.50
7..56
59.5210.53
82.56=0.97 28.8410.70
CIER-ISD
65.78:11.28 23.6242.06
81.54
63.78-1.133
26.10.11.71
83.5642.06
CIER-W
64.82+1.28
24.10+11.3
80.90+L0.61
62.32+00.80
29.30-10.90
1.20
Due to the computational expensive of the iDomainNet benchmark, only report the results of all baselines in the i'Veiice and iAnimal benchmarks Table The results show that, in gen- eral, continual learning benchmarks cannot achieve good balance among three important aspects: preventing catastrophic forgetting, facilitating knowledge transfer, and generalization to unseen do- mains. This is reflected by comparing the three evaluated metrics. While AGEM can facilitate knowledge transfer (high LA), it suffers from catastrophic forgetting (high FM), resulting in poor performances (low ACC). ON the other hand, experience replay and variants can achieve good trade-off between preventing catastrophic forgetting and facilitating knowledge transfer; but they struggle to generalize unseen domain. Our result suggests that balancing among all three objec- tives is important for continual learning and remains open problem for future research.
C SAMPLE IMAGES
We provide sample images of the iAnimal, iVehicle, and iDOmainNet benchmarks in Fig. Fig. and Fig. 5 respectively. For each benchmark, we show the sample images in the source and target domains of three randomly chosen classes. Each class's source and target domains used to construct the three evaluation scenarios described in Sec.
15

---

Under review as a conference paper at ICLR 2021
Cow
Dog
Cat
(a) Source domain images
Cow
Dog
Cat
(b) Target domain images
Figure Sample images from the iANimal benchmark. Images are extracted from the Animal
superclass of the NICO dataset (He et al. (020)).
Boat
Air- plane
Bus
(a) Source domain images
Boat
Air-
plane
Bus
(b) Target domain images
Figure 4: Sample images from the ieehicle benchmark. Images are extracted from the Vehicle superclass of the NICO dataset (He et al.) 2020).
16

---

Under review as a conference paper at ICLR 2021
Saw
Dolphin
Guitar
(a) Source domain images
Mmn
Saw
MMN
Dolphin
Guitar
(b) Target domain images
Figure 5: Sample images from the iDomainNet benchmark. Images are extracted from the Domain- Net dataset (Peng et al. 2019).
17


Reviews:
[{"review_id": "xW6Ao5S-J7V", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper talks about online continual learning in scenarios where there might be domain-shift during test time. Though I find the problem to be important, I believe that the solution proposed in this paper is straightforward (which is fine) and imposes a new set of constraints (knowing a priori the domain id during training) making the problem quite impractical for scenarios where continual learning might be useful. Please find below my comments in detail:\n\n1. Problem formulation: I do agree with the authors that task-free set-up (also known as single-head) is something we should focus more on, and like the fact that they remove this constraint of knowing task id during train/test. However, while relaxing this constraint, they added a new assumption of knowing the domain id during training. I find this rather a more strict requirement. The right problem formulation would be where there is a stream of samples coming (online) with a relatively blurry task boundary, and these samples might belong to different domains. Knowing domains a priori is very impractical in continual learning set-up. Could you please scenarios where it’s feasible to know domain id a priori? \n\n2. Approach: While I understand the ER part, I do not understand clearly section 4.1. What is the need of all GAN literature here? Why do you call D_j a domain critic? Am I wrong in saying that the only role of D_j for class y_j is to apply cross-entropy loss over domains (clipping if needed)? Please correct me if I am wrong. The section was a bit unclear to follow. \n\n3. How do you make sure that the assumption of data balance is valid given that the memory is very small compared to the current dataset. Also, it seems like that the memory sizes of 9k, 10k, and 34k are too big. Please comment. \n\n4. Training time: Since the set-up is online, training efficiently is extremely crucial. In Algorithm 1, every parameter update requires |Y| extra backprops for the domain critics SGD step. Could you please comment on the training time?\n\n5. References: I would suggest the authors to correct their citations a bit. For example, when talking about task-free, I would also cite iCarl, synaptic intelligence (SI), and RWalk. Similarly, the forgetting measure used in this paper was not proposed in Chaudhry 2019a, it was proposed in RWalk. \n\nOverall, even though the problem is important, I find the final experimental set-up to be impractical. There already are too many impractical formulations for continual learning, I would rather refrain myself from encouraging a new one. On top of this, there are a few technical aspects (training time, memory budget etc) I do not completely agree with, therefore, would request authors to answer above questions for clarity.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vOYQrDrVXT1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I'd like to preface this review by saying that I am not an expert in this area of continual learning. Hence, I will adjust my review after reading the authors' response as well as other reviewers' comments accordingly.\n\nThis work proposes CIER, a continual learning method that adjusts to domain shift during test time. The authors claim that existing methods correct distribution shift in P(X, Y), which makes the stronger assumption that P(Y | X) is the same across domains. In contract, this work follow prior work by Zhang 2013 and corrects distribution shift in P(X | Y), which makes the weaker assumption that P(Y) is the same across domains. The contribution of this work include using an adversarial objective via a critic that attempts to distinguish the domain of the learned representation F(X | Y), as well as the adoption of an experience replay buffer from which to sample examples to optimize the minimax adversarial objective.\n\nMy concerns are as follows:\n1. I am skeptical of the central claim of this work, which is that continual learning does not address domain shift. I am not an expert in this area, but to my knowledge, all reinforcement learning work that train on one environment and adapts to another environment must continual learn in a different environment (e.g. https://arxiv.org/abs/1803.11347, https://arxiv.org/abs/1905.04819, https://arxiv.org/abs/1910.08210). Can the authors comment on whether they think these works that both continually learn and adapt to new domains are relevant? If so, would it make sense to compare to them? Moreover, these works (and others in domain adaptation) do not assume that P(Y) is identical across domains, hence I feel like this is a rather strong assumption.\n2. The manuscript rehash the story of learning an invariant F(X | Y) repeatedly, however the terms in which it does this are not precise. For example, how do the authors define: domain, task, test domain, target domain, invariance, stability? I suggest that the authors define these precisely (for example w/ mathematical definitions), and give concrete examples grounded in application settings where possible.\n3. When the authors say that the distribution is stable, they mean that there is no class imbalance between domains. However stability in a distribution leads to a number of consequences (https://en.wikipedia.org/wiki/Stable_distribution) - are these consequences necessary? If so, in what ways? If not, would it make sense to just say \"there is no class imbalance between domains\"?\n4. The manuscript contains some traces of notation abuse which make it hard to read. For example, D and its cursive variant represent critic and domain. the lowercase t represents both task and time, the authors alternate between test domain and target domain. I suggests that the authors make their terminology consistent.\n5. In section 4.1, the authors propose learning a representation conditionally invariant in P(X|Y), however they also say that this was done by Zhang 2013 and Li 2018, hence this is not a contribution, correct?\n\nIn summary, due to the limitations posed by the assumption of this work (P(Y) the same across domains), the incremental contribution of this work (adversarial loss + experience replay), and the lack of clarity and precision in the manuscript,  I recommend a weak rejection. Due to my lack of experience in continual learning, I cannot assess the strength of the benchmarks and the significance of the experimental results, and will defer to other reviewers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "W55a8pZsWJQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates continual learning under domain shift and proposes a conditional invariant experience replay (CIER) method accordingly. CIER aims to retain old knowledge, acquire new information, and generalize to unseen domains. In particular, CIER uses adversarial training to correct the domain shift. Experimental results on three benchmark datasets are reported and discussed.\n\nPros.\n1. The problem setting of continual learning with domain shift is well motivated. The authors explained the rationality of this new problem setting by using a causal model. \n2. Three scenarios are considered and evaluated, including the significant shift, mild shift and no shift. Experimental results show that the proposed methods outperform baselines in many cases.\n3. Overall the paper is well written and clearly organized. The technical details are easy to follow.    \n\nCons. \n1. Although the problem setting is new, my main concern is the limited novelty of the proposed CIER method. In detail, experience replay has been extensively studied in continual learning, while the domain adversarial learning has also been widely used in domain adaptation literature for many years. The proposed CIER method, as illustrated in Figure 2, simply combines these two components. \n2. Ablation studies of the proposed method are missing. Thus, it is difficult to understand the contribution of each component, the effects of different sampling buffer sizes, etc. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "RC8I-tnTVvO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the domain adaptation problem when the source data comes from multiple domains continuously and the test domain for adaptation is unknown. The assumption used for domain shift is that the domain label would change the features but not the labels. So the main idea is to learn invariant representations across all the domains and avoid spurious correlation on the domain label. The proposed method then involves a multiplayer minimax game. The adversaries are the domain discriminator for each class, which tries to maximize the domain discrepancy. The minimizer player is the representation learner. The paper introduces two discrepancy measure based on the Jensen-Shannon divergence and the one dimensional Wasserstein distance. In the experiment, the data set for continual learning is constructed using domain shift data such that it mimics the online learning setting. The results are competitive in comparison with a limited set of baselines.\n \n \nStrong points:\n1. The paper focused on an interesting and important topic.\n2. The multiplayer adversarial game in terms of minimizing domain discrepancy seems to be novel.\n \nWeak points:\n1. The online or continual learning perspective is merely solved by keeping an episodic source data buffer, which I think is overly simplified. In general, I have a question about how this adversarial method would work when there is not an online/continual learning component. Given a fixed target, this a static set of source domains, it seems the method should be still valid. So I am not sure how the domain generalization side and the online side of the method interact with each other. Investigating the online setting before the batched setting seems problematic to me\n \n2. The evaluation in experiments shows that the FM measure of the proposed method is not very competitive. I believe it is related to how the memory is sampled and the length of the task sequence. However, it also indicates the method is not very satisfactory for avoiding forgetting. Otherwise, it could also be the case the conditional domain shift assumption is not valid in the data.\n \n3. The idea to learn an invariant representation is not novel. For example, the invariant risk minimization (IRM) method is exactly dealing with the same problem. Using an episodic source data buffer, it seems you can also apply IRM to solve the problem. I think it should be included as a baseline. In general, more invariant learning approaches should be discussed in the related work.\n \n4. Writing can be significantly improved.\n \nGiven the weak points, I recommend rejection for this paper.\n \nHere are some of my questions and additional suggestions:\n1. I am curious about how the invariant feature looks like. Basically, after learning the adversarial domain predictors, how the representation learning looks like. It would be nice to have some visualization on that.\n\n2. I do recommend focusing on a batched problem before going to the online version. I feel even though it is not the case that the batched method will work for online cases, at least in domain generalization we cannot expect a bad learner to work online when the data is even more limited. Even for an online paper, showing the performance for the batched version seems to be necessary.\n\n3. How \\alpha is chosen? And how it affects learning? In general, more ablation studies would improve paper quality. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presented an online continual learning method where there may be a shift in data distribution at test time. The paper proposes a Conditional Invariant Experience Replay (CIER) approach to correct the short which matches the distribution of inputs conditioned on the outputs. This is based on an adversarial training scheme.

The reviewers found the problem setting interesting but found the approach to be lacking in novelty and problem formulation somewhat restrictive (e.g.,  requiring domain id during training). The author feedback was taken into account but the reviewers stayed with their original assessment and, even after the rebuttal phase, none of the reviewers is in favor of accepting the paper.

The authors are advised to consider the feedback from the reviewers which will hopefully help to improve the paper for a future submission to another venue.
Decision: Reject