Online Continual Learning  Under  Domain Shift

1 INTRODUCTION . Continual learning is a promising framework towards human-level intelligence by developing models that can continuously learn over time ( Ring , 1997 ; Parisi et al. , 2019 ) . Unlike traditional learning paradigms , continual learning methods observe a continuum of tasks and have to simultaneously perform well on all tasks with limited access to previous data . Therefore , they have to achieve a good trade-off between retaining old knowledge ( French , 1999 ) and acquiring new skills , which is referred to as the stability-plasticity dilemma ( Abraham & Robins , 2005 ) . Continual learning is not only a challenging research problem but also has tremendous impacts on many applications ( Diethe et al. , 2019 ) . In particular , the deployed model may encounter new problems over time , and re-training every time new data arrive is infeasible , especially for large , complex neural networks . Despite recent success , existing continual learning benchmarks assume the data of each class are drawn from the same distribution during training and testing , i.e . P tr ( X , Y ) = P te ( X , Y ) . This restrictive assumption is unlikely to hold in practice and prohibits the use of continual learning strategies in numerous real-world applications . For example , consider a continual learning agent already trained to recognize certain objects in the indoor environment ; then , it moves to a new outdoor environment to acquire new skills . The agent may fail to recognize the learned objects because they are placed in a completely different background . Such an environment change is referred to as domain shift ( Khosla et al. , 2012 ; Long et al. , 2015 ; Diethe et al. , 2019 ) , and is very natural for continual learning in practice ( Diethe et al. , 2019 ) . To formalize the continual learning under domain shift problem , we need to consider the interaction between domain and the observation . In many computer vision applications , the causal structure is usually Y → X , i.e. , the object class is the cause for image features ( Lopez-Paz et al. , 2017 ) . This setting ignores the effect of conditional domain shift ( Zhang et al. , 2013 ) , where both the domain and the object class are the causes for the image features . That is , images may come from different domains , but they have the same label . Figure 1 shows the causal graph for continual learning under conditional domain shift problem and an illustrative example of the domain shift between training and testing data . Continual learning under conditional domain shift requires the model to perform previous tasks in new domains , which poses a great challenge since we do not know in hindsight which domain will be tested . Therefore , the model needs to learn the concepts presented in the data while ignoring the domains . For example , placing an object in different background does not change the label of that object . As a result , the model needs to achieve an invariant representation using data from various source domains observed during training , i.e . the joint distribution P ( X , Y ) is the same for all source domains . To this end , we develop Conditional Invariant Experience Replay ( CIER ) that can simultaneously retain previous knowledge , learn new tasks , and generalize to new domains . CIER employs an episodic memory to perform experience replay and an adversarial loss to achieve an invariant representation . Particularly , CIER formulates a multiplayer minimax game such that the conditional distribution P ( X|Y ) is stable across each class ’ s observed source domains . Therefore , if the prior distribution P ( Y ) is stable in the target domains , i.e. , no class imbalance , CIER can achieve the invariant representation in the joint distribution P ( X , Y ) and generalize to unseen domains . Fig . 2 provides a high level overview of the proposed CIER method . In summary , we formalize the continual learning under conditional domain shift problem and construct three novel benchmarks using real data with different levels of domain shift and diversity in the number of domains , tasks , and classes . Then , we develop CIER , a novel continual learning method that learns a conditional invariant representation and generalizes to novel domains . Our extensive experiments demonstrate the limitations of existing continual learning methods when being tested on unseen domains and show that the proposed CIER can mitigate such domain gaps effectively . 2 RELATED WORK . 2.1 CONTINUAL LEARNING . Continual learning aims at developing a model that can continuously learn different tasks over a data continuum . In literature , there are different continual learning protocols with different properties of the continuum . First , a setting can be either task-free ( Rebuffi et al. , 2017 ; Aljundi et al. , 2019b ) or task-aware ( Kirkpatrick et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ) based on whether a task-indicator is presented to the model . Second , learning can follow the online ( Lopez-Paz & Ranzato , 2017 ) or batch ( Kirkpatrick et al. , 2017 ) setting . In online continual learning , the data within each task is also online and the model is only allowed to learn each example once . Differently , in batch continual learning , the model is allowed to learn a task over many epochs . Generally , the task-free and online settings are more difficult because the model has to predict on all tasks , and optimizing deep neural network online over one epoch is particularly challenging ( Lopez-Paz & Ranzato , 2017 ; Sahoo et al. , 2018 ; Aljundi et al. , 2019a ) . However , we argue that existing protocols do not consider the domain shift between training and testing data of each task , which is important in real scenarios . This limitation motivates us in developing the continual learning under conditional domain shift problem as a practical and more challenging problem to evaluate continual learning methods . Continual learning methods aim at achieving a good trade-off between alleviating catastrophic forgetting and acquiring new knowledge . Existing methods can be broadly categorized into two main groups : ( i ) dynamic architecture approaches that employ a separate subnetwork for each task , and ( ii ) static architecture approaches that maintain a fixed network throughout learning . While dynamic architecture methods ( Rusu et al. , 2016 ; Serra et al. , 2018 ; von Oswald et al. , 2020 ; Yoon et al. , 2018 ; Li et al. , 2019 ; Xu & Zhu , 2018 ) usually do not suffer from catastrophic forgetting , they are not practical because of the high computational cost . In the fixed architecture method family , experience replay ( ER ) ( Lin , 1992 ) and its variants ( Chaudhry et al. , 2019b ; Aljundi et al. , 2019a ; Riemer et al. , 2019 ; Rolnick et al. , 2019 ) have recently gained much interest thanks to their strong performance across different continual learning protocols , significantly outperforms other approachecs such as regularization ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Aljundi et al. , 2018 ; Ritter et al. , 2018 ) , representation learning ( Rebuffi et al. , 2017 ) . However , existing continual learning methods lack the ability to generalize to unseen domains beyond training , which motivates us to develop a novel method of CIER that can achieve an invariant representation in continual learning . 2.2 DOMAIN GENERALIZATION AND DOMAIN ADAPTATION . Domain generalization aims at training a model that can generalize to a target domain given data from the source domains ( Khosla et al. , 2012 ; Li et al. , 2017 ; 2018 ; Carlucci et al. , 2019 ) . As a simpler problem setting , domain adaptation assumes unsupervised data from the target domain is also given during training ( Long et al. , 2015 ; Ganin & Lempitsky , 2015 ) . A general approach for such problems is learning an invariant representation by correcting the shift in P ( X , Y ) across domains . Particularly , ( Long et al. , 2015 ; Ganin & Lempitsky , 2015 ) assume that the conditional distribution P ( Y |X ) is the same across domains ; therefore , the domain shift can be addressed by matching only P ( X ) . However , invariant in P ( Y |X ) is a strong assumption and is unlikely to hold in many applications . A more robust approach is matching the conditional distribution P ( X|Y ) ( Zhang et al. , 2013 ; Li et al. , 2018 ) . In which case , if the class-prior distribution P ( Y ) is stable , i.e. , there is no class-imbalance among domains , matching P ( X|Y ) will result in the invariant in P ( X , Y ) because of the decomposition P ( X , Y ) = P ( Y ) P ( X|Y ) . We emphasize that no imbalance data is a mild assumption , especially when we do not know about the target domains . This result motivated us in the development of CIER . However , different from such methods , CIER can accommodate novel classes and domains over time while maintaining good performances over all classes . 3 PROBLEM FORMULATION . 3.1 ONLINE CONTINUAL LEARNING UNDER CONDITIONAL DOMAIN SHIFT . Notations . Let X ⊂ RD and Y ⊂ N be the input and label space respectively . A domain is defined as a joint distribution P ( X , Y ) over X ×Y , and the k-th domain P k ( X , Y ) is denoted as P k. Often , we do not have access to the domain P k , instead , we can label each data point with a scalar d ∈ N indicating that it comes from domain d. During training , a learner θ receives a continuum of data { xi , di , yi } , where ( xi , yi ) is the i-th training sample and di is the domain indicator associated with { xi , yi } . In the following , we use y ( t ) as the number of observed classes until time t ; dy ( t ) denotes the number of observed domains of class y until time t ; and x ∼ Y kj denotes the samples of the j-th class and has domain dk . We consider the task-free , online continual learning setting where the continuum is locally iid , i.e . each training sample ( xi , yi ) is drawn iid from an underlying task distribution Pti ( X , Y ) with the task identifier ti not revealed to the learner . Moreover , each task consists of a subset of labels sam- pled without replacement from the label space Y . To formalize the online continual learning under conditional domain shift problem , we assume that each class y is associated with a set of source domains Dsrcy = { dsrc1 , dsrc2 , . . . , dsrcs } , and a set of target domains Dtgty = { d tgt 1 , d tgt 2 , . . . , d tgt m } . Additionally , the source and target domains are disjoint for all classes , i.e . Dsrcy ∩Dtgty = ∅ , ∀y . We consider three scenarios with decreasing difficulties to evaluate the learner θ based on the degree of shift between the training and testing domains . • Scenario 1 ( significant shift ) : Train on source domains , test on target domains , the most challenging setting where all domains of the test data are unobserved during training . • Scenario 2 ( mild shift ) : Train on source domains , test on both source and target domains . Here we assume that domains of test data can be draw from both source and target domains . • Scenario 3 ( no shift ) : Train on target domains , test on target domains . This is the traditional continual learning setting where there is no discrepancy among domains of training and testing data . We design this scenario so that it has the same test data as the first one . In all scenarios , the model is required to perform well on the test data . We emphasize that the continual learning setting we considered is the most challenging in the literature : both task and data arrive online , and the model has to predict all observed classes so far during evaluation . Moreover , we provide the model with the domain identifiers during training ; however , only the input x is provided at test time . Finally , we aim at creating a model that can predict at any moment , therefore , we do not make any assumption regarding the testing domains , including the availability of unlabeled data , and the models are not allowed to train during inference .

Summary: This paper investigates continual learning under domain shift and proposes a conditional invariant experience replay (CIER) method accordingly. CIER aims to retain old knowledge, acquire new information, and generalize to unseen domains. In particular, CIER uses adversarial training to correct the domain shift. Experimental results on three benchmark datasets are reported and discussed.