Lifelong Domain Adaptation via Consolidated Internal Distribution

1 Introduction . Deep neural networks relax the need to for manual feature engineering by learning to generate discriminative features in an end-to-end blind training procedure [ 1 , 2 ] . Despite significant advances in deep learning , however , robust generalization of deep neural network on unseen data is still a primary challenge when domain shift exists between the training and the testing data [ 3 , 4 ] . Domain shift is a natural challenge in continual learning ( CL ) [ 5 , 6 ] where the goal is to learn adaptively and autonomously when the underlying input distribution drifts over extended time periods . Distributional shifts over time usually lead to performance degradation of trained models which in turn necessitates model retraining to acquire knowledge about new distributions . Current CL algorithms mainly consider tasks , i.e. , domains , with fully labeled datasets . Hence , these algorithms require annotating massive training datasets for new observed domains . Persistent manual data annotation , however , is practically prohibitive because of being an economically costly and time-consuming process [ 7 ] .To relax this constraint , our goal is to develop an algorithm for continual adaptation of a model for tackling the challenge of domain shift in a CL setting using solely unannotated datasets . Unsupervised Domain adaptation ( UDA ) is a highly relevant learning setting to our problem of interest . The goal in UDA is to train a model for a target domain with unannotated data by transferring knowledge from a related source domain in which annotated data is accessible [ 3 ] . A primary group of UDA algorithms map the training data points for both domains into a shared latent embedding space and align the distributions of the source and the target domains in that space [ 8 ] . Hence , a source-trained classifier that receives its input from the shared embedding space would generalize on the target domain as well . The domain alignment procedure has been implemented either using generative adversarial learning [ 9 , 10 , 11 , 12 , 13 ] or by directly minimizing the distance between the two distributions [ 14 , 15 , 16 , 17 , 18 ] . Existing UDA algorithms are not suitable for continual learning because the underlying model can be trained if datasets from both domains are accessible . Moreover , these methods usually consider only a single target domain and a single source domain . Finally , simply updating the underlying model to generalize in the current encountered domain is not sufficient . Because upon updating the model , the network likely would forget the past learned domains as the result of retroactive interference , referred as the phenomenon of catastrophic forgetting [ 19 , 20 ] in the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) CL literature . Consequently , we also need to tackle catastrophic forgetting in our unexplored learning setting . Our method can be considered as an improvement over existing UDA and CL methods . Contributions : we develop an algorithm for lifelong unsupervised adaptation of a model on new domains using solely unannotated data . Our idea is based on consolidating the internally learned distribution that encodes the learned knowledge by the model when the initial source domain is learned . We use this multimodal distribution to update the model such that the learned distributions for all subsequent unannotated domains are coupled . To overcome catastrophic forgetting , we store important representative samples for all tasks and replay them back when the model is updated . We provide a theoretical analysis to demonstrate that our method mitigates catastrophic forgetting and also leads to improved generalization . We validate our method using standard UDA benchmarks . 2 Background and Related Work . Lifelong UDA setting lies on the intersection of both UDA and CL learning settings . Unsupervised domain adaptation : an effective approach for domain alignment in UDA is to use a probability distance to measure distributional discrepancy and then train an encoder to minimize the cross-domain distance at its output as a latent shared embedding space [ 21 , 22 , 23 , 24 , 25 , 26 ] . The Wasserstein distance ( WD ) [ 27 ] is a suitable metric for this purpose due to possessing nonvanishing gradients . This property is beneficial because deep learning optimization problems are primarily solved using gradient-based optimization techniques [ 23 ] . In this work , we rely on the sliced Wasserstein distance ( SWD ) [ 24 ] variant of the Wasserstein distance because it can be computed efficiently using a closed form solution via empirical samples of the data distribution . Continual learning : Existing CL methods primarily use either model regularization or experience replay to tackle catastrophic forgetting . The idea of model regularization [ 20 , 5 ] is to identify the network weights that contribute significantly in retaining knowledge about a past learned task and then consolidate them when the model is updated to learn the subsequent tasks . Alternatively , the model can be expanded progressively to learn the new tasks using added weights [ 28 ] . We rely on experience replay which is implemented using pseudo-rehearsal [ 19 ] . The idea is to identify important training data points that contribute significantly to learning a task and store them in a memory buffer [ 29 ] . These samples then would be replayed along with the current task ’ s data to represent the past learned distributions [ 30 , 31 ] to retain the acquire knowledge about the past tasks . The major challenge is to identify the important data points . We rely on the consolidated internal distribution to identify the important data points . To mitigate the sample selection challenge , some CL algorithms rely on generative experience replay , where the memory buffer is replaced with the ability of generating synthetic pseudo-samples that are similar to the samples of the past learned tasks [ 31 , 32 , 33 , 34 ] . Domain Adaptation in Continual Learning Settings : Existing works in this direction are highly limited . Wulfmeier et al . [ 35 ] study addressing gradual shifts in changing environments . Bobu et al . [ 36 ] and Wu et al . [ 37 ] explore addressing the problem of domain shift in continual learning settings but both works assume that datasets for all learned tasks are observable at each time-step . This assumption limits practicality of these works . Porav et al . [ 38 ] explore a setting similar to our setting but the proposed method relies on image translation which makes the work highly domain specific . In our work , we relax these limitations and develop a more general algorithm . 3 Problem Statement . Consider a classification problem in a source domain S with a labeled training dataset DS = ( X0 , Y0 ) , where X0 = [ x01 , . . . , x 0 N ] ∈ X ⊂ Rd×N , Y0 = [ y01 , ... , y0N ] ∈ Y ⊂ Rk×N , and the training data points are drawn independently from an unknown source distribution , i.e. , x0i ∼ p0 ( x ) . Given a deep neural network fθ with learnable weights θ , we can solve for an optimal model using the standard empirical risk minimization ( ERM ) : θ̂0 = argminθ ∑ i L ( fθ ( x0i ) , y0i ) , where L ( · ) is a suitable loss function . If the training dataset is large enough and the network structure is complex enough , the ERM optimal model will generalize well on unseen data points , drawn from p0 ( x ) [ 39 ] . The challenge in CL is that the input distribution may be non-stationary . Hence , the testing samples may be drawn from drifted distributions . The resulting distributional gap will lead to poor generalization during the testing time . Our goal is to update the source-trained model fθ̂0 continually using solely unlabeled data to avoid poor generalization without forgetting past experiences . To model this process , we consider a set of sequentially arriving target domains T t , t = 1 . . . T , with unlabeled datasets DtT = ( Xt ) , where Xt ∈ Rd×Mt , xti ∼ pt ( x ) , and ∀t1 , t2 : pt1 ̸= pt2 ( see Figure 1 ) . Since these domains are unlabeled using ERM is implausible . As stated , common UDA methods can not address CL settings . To address challenges of “ domain shift ” and “ catastrophic forgetting ” , we consider that the base network fθ ( · ) can be decomposed into a deep encoder ϕv ( · ) : X → Z ⊂ Rp and a classifier subnetwork hw ( · ) : Z → Y , i.e. , fθ = hw ◦ ϕv , where θ = ( w , v ) . Our method is based on consolidating the internal distribution that is formed in the embedding space Z . We assume that as the result of initial training on the source domain , the source classes become separable in Z . If at each time-steps , we update the model such that the internal distribution remains stable , i.e. , the distance between the distributions ϕ ( p0 ( x0 ) ) and ϕ ( pt ( xt ) ) is minimized , then the model continues to generalize well on the target domains , despite initially being trained with the source domain labeled dataset . This strategy has been used extensively by the existing UDA algorithms but we are constrained with the accessibility of DS in CL , i.e. , the term ϕ ( p0 ( x0 ) ) can not be computed . 4 Proposed Solution Our solution is based on consolidating the intermediate distribution that is learned in the discriminative embedding space to retain model generalizability . Upon learning the initial source domain , the encoder is trained to map the input source distribution into a multi-modal distribution pJ ( z ) in the embedding space . Each mode in this distribution corresponds to one of the input classes and the training data points that belong to a particular class , are mapped to the same cluster ( see Figure 1 , middle ) . The internally learned multimodal distribution is represented empirically by the source domain data representations { ( ϕv ( x0i ) , y0i ) } Ni=1 . We use a GMM p0J ( z ) with k component as the parametric model to estimate the internal distribution : p0J ( z ) = k∑ j=1 α0jN ( z|µ0j , Σ0j ) , ( 1 ) where α0j denotes the mixture weights , and µ 0 j and Σ0j denote the mean and co-variance matrices for the components . Computing these parameters is generally a challenging iterative procedure . But since the labels are accessible , we can compute the parameters for each mode independently . Let S0j denote the support set for the mode j , i.e. , S0j = { ( x0i , y0i ) ∈ DS | argmaxy0i = j } . The MAP estimates for the GMM parameters would be : α̂0j = |S0j | N , µ̂j = ∑ ( x0i , y 0 i ) ∈S 0 j 1 |S0j | ϕv ( x 0 i ) , Σ̂ 0 j = ∑ ( x0i , y 0 i ) ∈S 0 j 1 |S0j | ( ϕv ( x 0 i ) − µ̂0j ) ⊤ ( ϕv ( x 0 i ) − µ̂0j ) . ( 2 ) We can update the estimates in Eq . 2 for t > 1 , to be described later . Let p̂tJ ( z ) denote the estimated GMM at the time-step t. Our idea is to consolidate this distribution to retain model generalization . In the absence of source data , we adapt the model such that the encoder always aligns the target distributions with this distribution in the embedding space . We implement this process by drawing random samples from the GMM and build a labeled pseudo-dataset : D̂tP = ( Z t P , Ŷ t P ) , where ZtP = [ z t 1 , . . . , z t Np ] ∈ Rp×Np , ŶP = [ ŷp , t1 , ... , ŷ p , t Np ] ∈ Rk×Np , and zti ∼ p̂tJ ( z ) . We determine the labels according to the model predictions . We include only those samples for which the model prediction confidence-level is more than a preset threshold τ to exclude outlier samples . To retain the model generalization power , we solve the following problem at time t : min v , w N∑ i=1 L ( hw ( z p i ) , ŷ p , t i ) + λD ( ϕv ( pt ( Xt ) ) , p̂ t J ( Z t P ) ) , ( 3 ) where D ( · , · ) denotes a probability discrepancy measure and λ is a trade-off parameter . The first term enforces the classifier to preserve its generalizability on the internal distribution . The second term enforces alignment of the target domain distribution with the internally learned distribution in the embedding space . We use the SWD metric [ 27 ] for the term D ( · , · ) . Upon updating the model , we can also update the internal distribution p̂tJ ( z ) using the pseudo-dataset samples . Solving Eq . ( 3 ) would help the model to generalize well on the new domain T t. However , catastrophic forgetting will remain untackled because the encoder subnetwork is updated unconditionally to minimize the domain discrepancy term in Eq . 3.To tackle catastrophic forgetting , we rely on experience replay . When a task is learned , we select a small subset of the training data points as representative samples , to be stored in a memory buffer . These samples are replayed during model adaptation to mitigate catastrophic forgetting . Possible selection strategies includes mean of features ( MoF ) [ 40 ] , ring buffer [ 41 ] , and reservoir sampling [ 42 ] . Existence of the internal distribution makes MoF a natural choice . Upon model update at time-step t and then updating the GMM , we can compute the distance of representations for all the data points of T t from their corresponding cluster mean , i.e. , dtj , l = ∥µtj − ϕ ( xtl ) ∥22 , ∀xtl s.t . ŷtl = argmax fθ̂t ( x t l ) = j . Given a memory budget of Nb samples , we pick the Mb = Nb/k samples per class that have the closest distance to the mean to form the buffer-stored dataset Dtb = D t−1 b ∪ ( Xtb , Ŷ tb ) . These samples are the most informative samples to estimate the internal distributions . We update Eq . ( 3 ) to tackle catastrophic forgetting as follows : min v , w N∑ i=1 L ( hw ( z p i ) , ŷ p , t i ) + Nb∑ i=1 L ( hw ( ϕv ( x b i ) ) , ŷ b i ) + λD ( ϕv ( pt ( Xt ) ) , p̂ t J ( Z t P ) ) + λD ( ϕv ( pt ( X t b ) ) , p̂ t J ( Z t P ) ) . ( 4 ) Adding the second supervised term in Eq . ( 4 ) helps to mitigate catastrophic forgetting . The fourth term helps to consolidate the internal distribution across all the previous tasks . Our proposed solution for lifelong domain adaptation , named Lifelong Domain Adaptation Using Consolidated Internal Distribution ( LDAuCID ) , is presented and visualized in Algorithm 1 and Figure 1 , respectively . 5 Theoretical Analysis Algorithm 1 LDAuCID ( λ , τ , Nb ) 1 : Source Training : 2 : Input : source labeled dataset DS = ( X0 , Y0 ) 3 : θ̂0 = ( ŵ0 , v̂0 ) = argminθ ∑ i L ( fθ ( x 0 i ) , y 0 i ) 4 : Internal Distribution Estimation : 5 : Use Eq . ( 2 ) and estimate α0j , µ 0 j , and Σ 0 j 6 : Memory Buffer Initialization 7 : D0b = ( X0b , Ŷ 0b ) Pick the Nb/k samples with the least dtj , l = ∥µtj − ϕ ( xtl ) ∥22 , ŷ0 , ib = argmax f θ̂t ( x 0 , Nb b ) 8 : Continual Unsupervised Domain Adaptation : 9 : for t = 1 , . . . , T do 10 : Input : target unlabeled dataset DtT = ( Xt ) 11 : Pseudo-Dataset Generation : 12 : D̂tP = ( ZtP , Ŷ t P ) = 13 : ( [ zp , t1 , . . . , z p , t N ] , [ ŷ p , t 1 , . . . , ŷ p , t N ] ) , where : zp , ti ∼ p̂ t−1 J ( z ) , 1 ≤ i ≤ Np and ŷp , ti = argmaxj { hŵt ( z p , t i ) } if with confidence τ : maxj { hŵt ( z p , t i ) } > τ 14 : for itr = 1 , . . . , ITR do 15 : draw data batches from DtT and D̂P 16 : Update the model by solving Eq . ( 4 ) 17 : end for 18 : Internal Distribution Estimate Update : 19 : Use Eq . ( 2 ) similar to step 5 above . 20 : Memory Buffer Update 21 : Dtb = Dt−1b ∪ ( X t b , Ŷ t b ) , where ( X t b , Ŷ t b ) is computed similar to step 7 above . 22 : end for We provide an analysis within a standard PAClearning framework [ 39 ] . Consider the set of classifier sub-networks H = { hw ( · ) |hw ( · ) : Z → Rk , v ∈ RV } as the hypothesis class within PAC-learning . Let e0 denote the expected error on the source domain , et denote the expected error on the target domains , and eJt denote the expected error on the pseudo-dataset for a given h ∈ H. Also , let p̂0 ( x ) = 1N ∑N n=1 δ ( ϕv ( x s n ) ) and p̂t ( x ) = 1 Mt ∑Mt m=1 δ ( ϕv ( x t m ) ) denote the empirical source and target distributions in the embedding space . We provide the following theorem . Theorem 1 : Consider LDAuCID algorithm at learning time-step t = T . Then for all the previously learned tasks t < T , the following holds : et ≤ eJT−1 +W ( ϕ ( p̂t ) , p̂tJ ) + T−2∑ s=t W ( p̂sJ , p̂ s+1 J ) + e ( w∗ ) + √ ( 2 log ( 1 ξ ) /ζ ) ( √ 1 Mt + √ 1 Np + √ 1 Nb ) , ( 5 ) where e ( w∗ ) denotes expected error for the best joint-trained optimal model in the hypothesis space , i.e. , the model trained as w∗ = argminw ec ( w ) = argminw { et ( h ) + etJ ( h ) } , W ( · , · ) denotes the WD distance , and ζ and ξ are two constants . Proof : Included in the Appendix . Theorem 1 explains why LDAuCID algorithm is effective . Major terms in the right-hand side of Eq . 5 , as an upperbound for the expected error for each task ( domain ) , are continually minimized by LDAuCID . The first term is minimized because the internal distribution random samples are used to minimize the empirical error term as the first term of Eq . ( 4 ) . The second term is minimized as the third terms of Eq . ( 4 ) when the task distribution is aligned with the empirical internal distribution in the embedding space at time t. The third term which is a summation which models the effect of continual learning . The sth term in this sum is minimized when the task T s is learned and we deliberately update the internal distribution . This additive term grows as more tasks are learned after learning a particular task which can potentially make the upperbound looser , leading to forgetting effects . The fourth term is a constant small term if the tasks are related , i.e. , share the same classes . LDAuCID does not minimize this terms . It just suggests that the model should work well for a joint-training scenario in order to perform well in a sequential learning setting . The last term is a constant term which is negligible if we have sufficient numbers of source and target training data points . It also states that storing more samples in the memory buffer leads to better performance , as expected . We conclude if the upperbound in Eq . ( 5 ) is sufficiently tight , the domains are relevant , and the hypothesis space is a suitable space to learn the tasks , then adapting the model using LDAuCID can both tackle catastrophic forgetting and also improve model generalization on the target domains .

Summary: Unsupervised Domain Adaptation (UDA) is studied in a task sequential continual learning setting. The source domain is assumed to have supervised data and several target domains without labels are being sequentially adapted to. An algorithms is engineered for the specific case of supervised classification source datasets which can be neatly clustered in some high-level representation; then, learning on related (target) datasets which can be classified in the same (source) classes can be done sequentially and without the use of labels. Catastrophic forgetting is mitigated using a standard experience replay approach, but also through the assumptions of the conceptual framework, i.e. strong overlap between source and target classes, which is known to be the easier setting in CL works. Empirical validation is done primarily in the using classic domain adaptation benchmarks, with the difference of using a single source domain for multiple target domains in a sequential fashion. The theoretical analysis and ablation study round up the work, including a discussion of algorithm hyper-parameters.