Lifelong Domain Adaptation via Consolidated Internal Distribution

1 Introduction . Deep neural networks relax the need to for manual feature engineering by learning to generate discriminative features in an end-to-end blind training procedure [ 1 , 2 ] . Despite significant advances in deep learning , however , robust generalization of deep neural network on unseen data is still a primary challenge when domain shift exists between the training and the testing data [ 3 , 4 ] . Domain shift is a natural challenge in continual learning ( CL ) [ 5 , 6 ] where the goal is to learn adaptively and autonomously when the underlying input distribution drifts over extended time periods . Distributional shifts over time usually lead to performance degradation of trained models which in turn necessitates model retraining to acquire knowledge about new distributions . Current CL algorithms mainly consider tasks , i.e. , domains , with fully labeled datasets . Hence , these algorithms require annotating massive training datasets for new observed domains . Persistent manual data annotation , however , is practically prohibitive because of being an economically costly and time-consuming process [ 7 ] .To relax this constraint , our goal is to develop an algorithm for continual adaptation of a model for tackling the challenge of domain shift in a CL setting using solely unannotated datasets . Unsupervised Domain adaptation ( UDA ) is a highly relevant learning setting to our problem of interest . The goal in UDA is to train a model for a target domain with unannotated data by transferring knowledge from a related source domain in which annotated data is accessible [ 3 ] . A primary group of UDA algorithms map the training data points for both domains into a shared latent embedding space and align the distributions of the source and the target domains in that space [ 8 ] . Hence , a source-trained classifier that receives its input from the shared embedding space would generalize on the target domain as well . The domain alignment procedure has been implemented either using generative adversarial learning [ 9 , 10 , 11 , 12 , 13 ] or by directly minimizing the distance between the two distributions [ 14 , 15 , 16 , 17 , 18 ] . Existing UDA algorithms are not suitable for continual learning because the underlying model can be trained if datasets from both domains are accessible . Moreover , these methods usually consider only a single target domain and a single source domain . Finally , simply updating the underlying model to generalize in the current encountered domain is not sufficient . Because upon updating the model , the network likely would forget the past learned domains as the result of retroactive interference , referred as the phenomenon of catastrophic forgetting [ 19 , 20 ] in the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) CL literature . Consequently , we also need to tackle catastrophic forgetting in our unexplored learning setting . Our method can be considered as an improvement over existing UDA and CL methods . Contributions : we develop an algorithm for lifelong unsupervised adaptation of a model on new domains using solely unannotated data . Our idea is based on consolidating the internally learned distribution that encodes the learned knowledge by the model when the initial source domain is learned . We use this multimodal distribution to update the model such that the learned distributions for all subsequent unannotated domains are coupled . To overcome catastrophic forgetting , we store important representative samples for all tasks and replay them back when the model is updated . We provide a theoretical analysis to demonstrate that our method mitigates catastrophic forgetting and also leads to improved generalization . We validate our method using standard UDA benchmarks . 2 Background and Related Work . Lifelong UDA setting lies on the intersection of both UDA and CL learning settings . Unsupervised domain adaptation : an effective approach for domain alignment in UDA is to use a probability distance to measure distributional discrepancy and then train an encoder to minimize the cross-domain distance at its output as a latent shared embedding space [ 21 , 22 , 23 , 24 , 25 , 26 ] . The Wasserstein distance ( WD ) [ 27 ] is a suitable metric for this purpose due to possessing nonvanishing gradients . This property is beneficial because deep learning optimization problems are primarily solved using gradient-based optimization techniques [ 23 ] . In this work , we rely on the sliced Wasserstein distance ( SWD ) [ 24 ] variant of the Wasserstein distance because it can be computed efficiently using a closed form solution via empirical samples of the data distribution . Continual learning : Existing CL methods primarily use either model regularization or experience replay to tackle catastrophic forgetting . The idea of model regularization [ 20 , 5 ] is to identify the network weights that contribute significantly in retaining knowledge about a past learned task and then consolidate them when the model is updated to learn the subsequent tasks . Alternatively , the model can be expanded progressively to learn the new tasks using added weights [ 28 ] . We rely on experience replay which is implemented using pseudo-rehearsal [ 19 ] . The idea is to identify important training data points that contribute significantly to learning a task and store them in a memory buffer [ 29 ] . These samples then would be replayed along with the current task ’ s data to represent the past learned distributions [ 30 , 31 ] to retain the acquire knowledge about the past tasks . The major challenge is to identify the important data points . We rely on the consolidated internal distribution to identify the important data points . To mitigate the sample selection challenge , some CL algorithms rely on generative experience replay , where the memory buffer is replaced with the ability of generating synthetic pseudo-samples that are similar to the samples of the past learned tasks [ 31 , 32 , 33 , 34 ] . Domain Adaptation in Continual Learning Settings : Existing works in this direction are highly limited . Wulfmeier et al . [ 35 ] study addressing gradual shifts in changing environments . Bobu et al . [ 36 ] and Wu et al . [ 37 ] explore addressing the problem of domain shift in continual learning settings but both works assume that datasets for all learned tasks are observable at each time-step . This assumption limits practicality of these works . Porav et al . [ 38 ] explore a setting similar to our setting but the proposed method relies on image translation which makes the work highly domain specific . In our work , we relax these limitations and develop a more general algorithm . 3 Problem Statement . Consider a classification problem in a source domain S with a labeled training dataset DS = ( X0 , Y0 ) , where X0 = [ x01 , . . . , x 0 N ] ∈ X ⊂ Rd×N , Y0 = [ y01 , ... , y0N ] ∈ Y ⊂ Rk×N , and the training data points are drawn independently from an unknown source distribution , i.e. , x0i ∼ p0 ( x ) . Given a deep neural network fθ with learnable weights θ , we can solve for an optimal model using the standard empirical risk minimization ( ERM ) : θ̂0 = argminθ ∑ i L ( fθ ( x0i ) , y0i ) , where L ( · ) is a suitable loss function . If the training dataset is large enough and the network structure is complex enough , the ERM optimal model will generalize well on unseen data points , drawn from p0 ( x ) [ 39 ] . The challenge in CL is that the input distribution may be non-stationary . Hence , the testing samples may be drawn from drifted distributions . The resulting distributional gap will lead to poor generalization during the testing time . Our goal is to update the source-trained model fθ̂0 continually using solely unlabeled data to avoid poor generalization without forgetting past experiences . To model this process , we consider a set of sequentially arriving target domains T t , t = 1 . . . T , with unlabeled datasets DtT = ( Xt ) , where Xt ∈ Rd×Mt , xti ∼ pt ( x ) , and ∀t1 , t2 : pt1 ̸= pt2 ( see Figure 1 ) . Since these domains are unlabeled using ERM is implausible . As stated , common UDA methods can not address CL settings . To address challenges of “ domain shift ” and “ catastrophic forgetting ” , we consider that the base network fθ ( · ) can be decomposed into a deep encoder ϕv ( · ) : X → Z ⊂ Rp and a classifier subnetwork hw ( · ) : Z → Y , i.e. , fθ = hw ◦ ϕv , where θ = ( w , v ) . Our method is based on consolidating the internal distribution that is formed in the embedding space Z . We assume that as the result of initial training on the source domain , the source classes become separable in Z . If at each time-steps , we update the model such that the internal distribution remains stable , i.e. , the distance between the distributions ϕ ( p0 ( x0 ) ) and ϕ ( pt ( xt ) ) is minimized , then the model continues to generalize well on the target domains , despite initially being trained with the source domain labeled dataset . This strategy has been used extensively by the existing UDA algorithms but we are constrained with the accessibility of DS in CL , i.e. , the term ϕ ( p0 ( x0 ) ) can not be computed . 4 Proposed Solution Our solution is based on consolidating the intermediate distribution that is learned in the discriminative embedding space to retain model generalizability . Upon learning the initial source domain , the encoder is trained to map the input source distribution into a multi-modal distribution pJ ( z ) in the embedding space . Each mode in this distribution corresponds to one of the input classes and the training data points that belong to a particular class , are mapped to the same cluster ( see Figure 1 , middle ) . The internally learned multimodal distribution is represented empirically by the source domain data representations { ( ϕv ( x0i ) , y0i ) } Ni=1 . We use a GMM p0J ( z ) with k component as the parametric model to estimate the internal distribution : p0J ( z ) = k∑ j=1 α0jN ( z|µ0j , Σ0j ) , ( 1 ) where α0j denotes the mixture weights , and µ 0 j and Σ0j denote the mean and co-variance matrices for the components . Computing these parameters is generally a challenging iterative procedure . But since the labels are accessible , we can compute the parameters for each mode independently . Let S0j denote the support set for the mode j , i.e. , S0j = { ( x0i , y0i ) ∈ DS | argmaxy0i = j } . The MAP estimates for the GMM parameters would be : α̂0j = |S0j | N , µ̂j = ∑ ( x0i , y 0 i ) ∈S 0 j 1 |S0j | ϕv ( x 0 i ) , Σ̂ 0 j = ∑ ( x0i , y 0 i ) ∈S 0 j 1 |S0j | ( ϕv ( x 0 i ) − µ̂0j ) ⊤ ( ϕv ( x 0 i ) − µ̂0j ) . ( 2 ) We can update the estimates in Eq . 2 for t > 1 , to be described later . Let p̂tJ ( z ) denote the estimated GMM at the time-step t. Our idea is to consolidate this distribution to retain model generalization . In the absence of source data , we adapt the model such that the encoder always aligns the target distributions with this distribution in the embedding space . We implement this process by drawing random samples from the GMM and build a labeled pseudo-dataset : D̂tP = ( Z t P , Ŷ t P ) , where ZtP = [ z t 1 , . . . , z t Np ] ∈ Rp×Np , ŶP = [ ŷp , t1 , ... , ŷ p , t Np ] ∈ Rk×Np , and zti ∼ p̂tJ ( z ) . We determine the labels according to the model predictions . We include only those samples for which the model prediction confidence-level is more than a preset threshold τ to exclude outlier samples . To retain the model generalization power , we solve the following problem at time t : min v , w N∑ i=1 L ( hw ( z p i ) , ŷ p , t i ) + λD ( ϕv ( pt ( Xt ) ) , p̂ t J ( Z t P ) ) , ( 3 ) where D ( · , · ) denotes a probability discrepancy measure and λ is a trade-off parameter . The first term enforces the classifier to preserve its generalizability on the internal distribution . The second term enforces alignment of the target domain distribution with the internally learned distribution in the embedding space . We use the SWD metric [ 27 ] for the term D ( · , · ) . Upon updating the model , we can also update the internal distribution p̂tJ ( z ) using the pseudo-dataset samples . Solving Eq . ( 3 ) would help the model to generalize well on the new domain T t. However , catastrophic forgetting will remain untackled because the encoder subnetwork is updated unconditionally to minimize the domain discrepancy term in Eq . 3.To tackle catastrophic forgetting , we rely on experience replay . When a task is learned , we select a small subset of the training data points as representative samples , to be stored in a memory buffer . These samples are replayed during model adaptation to mitigate catastrophic forgetting . Possible selection strategies includes mean of features ( MoF ) [ 40 ] , ring buffer [ 41 ] , and reservoir sampling [ 42 ] . Existence of the internal distribution makes MoF a natural choice . Upon model update at time-step t and then updating the GMM , we can compute the distance of representations for all the data points of T t from their corresponding cluster mean , i.e. , dtj , l = ∥µtj − ϕ ( xtl ) ∥22 , ∀xtl s.t . ŷtl = argmax fθ̂t ( x t l ) = j . Given a memory budget of Nb samples , we pick the Mb = Nb/k samples per class that have the closest distance to the mean to form the buffer-stored dataset Dtb = D t−1 b ∪ ( Xtb , Ŷ tb ) . These samples are the most informative samples to estimate the internal distributions . We update Eq . ( 3 ) to tackle catastrophic forgetting as follows : min v , w N∑ i=1 L ( hw ( z p i ) , ŷ p , t i ) + Nb∑ i=1 L ( hw ( ϕv ( x b i ) ) , ŷ b i ) + λD ( ϕv ( pt ( Xt ) ) , p̂ t J ( Z t P ) ) + λD ( ϕv ( pt ( X t b ) ) , p̂ t J ( Z t P ) ) . ( 4 ) Adding the second supervised term in Eq . ( 4 ) helps to mitigate catastrophic forgetting . The fourth term helps to consolidate the internal distribution across all the previous tasks . Our proposed solution for lifelong domain adaptation , named Lifelong Domain Adaptation Using Consolidated Internal Distribution ( LDAuCID ) , is presented and visualized in Algorithm 1 and Figure 1 , respectively . 5 Theoretical Analysis Algorithm 1 LDAuCID ( λ , τ , Nb ) 1 : Source Training : 2 : Input : source labeled dataset DS = ( X0 , Y0 ) 3 : θ̂0 = ( ŵ0 , v̂0 ) = argminθ ∑ i L ( fθ ( x 0 i ) , y 0 i ) 4 : Internal Distribution Estimation : 5 : Use Eq . ( 2 ) and estimate α0j , µ 0 j , and Σ 0 j 6 : Memory Buffer Initialization 7 : D0b = ( X0b , Ŷ 0b ) Pick the Nb/k samples with the least dtj , l = ∥µtj − ϕ ( xtl ) ∥22 , ŷ0 , ib = argmax f θ̂t ( x 0 , Nb b ) 8 : Continual Unsupervised Domain Adaptation : 9 : for t = 1 , . . . , T do 10 : Input : target unlabeled dataset DtT = ( Xt ) 11 : Pseudo-Dataset Generation : 12 : D̂tP = ( ZtP , Ŷ t P ) = 13 : ( [ zp , t1 , . . . , z p , t N ] , [ ŷ p , t 1 , . . . , ŷ p , t N ] ) , where : zp , ti ∼ p̂ t−1 J ( z ) , 1 ≤ i ≤ Np and ŷp , ti = argmaxj { hŵt ( z p , t i ) } if with confidence τ : maxj { hŵt ( z p , t i ) } > τ 14 : for itr = 1 , . . . , ITR do 15 : draw data batches from DtT and D̂P 16 : Update the model by solving Eq . ( 4 ) 17 : end for 18 : Internal Distribution Estimate Update : 19 : Use Eq . ( 2 ) similar to step 5 above . 20 : Memory Buffer Update 21 : Dtb = Dt−1b ∪ ( X t b , Ŷ t b ) , where ( X t b , Ŷ t b ) is computed similar to step 7 above . 22 : end for We provide an analysis within a standard PAClearning framework [ 39 ] . Consider the set of classifier sub-networks H = { hw ( · ) |hw ( · ) : Z → Rk , v ∈ RV } as the hypothesis class within PAC-learning . Let e0 denote the expected error on the source domain , et denote the expected error on the target domains , and eJt denote the expected error on the pseudo-dataset for a given h ∈ H. Also , let p̂0 ( x ) = 1N ∑N n=1 δ ( ϕv ( x s n ) ) and p̂t ( x ) = 1 Mt ∑Mt m=1 δ ( ϕv ( x t m ) ) denote the empirical source and target distributions in the embedding space . We provide the following theorem . Theorem 1 : Consider LDAuCID algorithm at learning time-step t = T . Then for all the previously learned tasks t < T , the following holds : et ≤ eJT−1 +W ( ϕ ( p̂t ) , p̂tJ ) + T−2∑ s=t W ( p̂sJ , p̂ s+1 J ) + e ( w∗ ) + √ ( 2 log ( 1 ξ ) /ζ ) ( √ 1 Mt + √ 1 Np + √ 1 Nb ) , ( 5 ) where e ( w∗ ) denotes expected error for the best joint-trained optimal model in the hypothesis space , i.e. , the model trained as w∗ = argminw ec ( w ) = argminw { et ( h ) + etJ ( h ) } , W ( · , · ) denotes the WD distance , and ζ and ξ are two constants . Proof : Included in the Appendix . Theorem 1 explains why LDAuCID algorithm is effective . Major terms in the right-hand side of Eq . 5 , as an upperbound for the expected error for each task ( domain ) , are continually minimized by LDAuCID . The first term is minimized because the internal distribution random samples are used to minimize the empirical error term as the first term of Eq . ( 4 ) . The second term is minimized as the third terms of Eq . ( 4 ) when the task distribution is aligned with the empirical internal distribution in the embedding space at time t. The third term which is a summation which models the effect of continual learning . The sth term in this sum is minimized when the task T s is learned and we deliberately update the internal distribution . This additive term grows as more tasks are learned after learning a particular task which can potentially make the upperbound looser , leading to forgetting effects . The fourth term is a constant small term if the tasks are related , i.e. , share the same classes . LDAuCID does not minimize this terms . It just suggests that the model should work well for a joint-training scenario in order to perform well in a sequential learning setting . The last term is a constant term which is negligible if we have sufficient numbers of source and target training data points . It also states that storing more samples in the memory buffer leads to better performance , as expected . We conclude if the upperbound in Eq . ( 5 ) is sufficiently tight , the domains are relevant , and the hypothesis space is a suitable space to learn the tasks , then adapting the model using LDAuCID can both tackle catastrophic forgetting and also improve model generalization on the target domains .

Summary: This paper proposes a problem formulation at the intersection between continual learning (CL) and unsupervised domain adaptation (UDA). It introduces a method to perform UDA throughout the lifespan of a model, mitigating catastrophic forgetting of previous information. The proposed method is validated on benchmarks proposed ad-hoc for this problem, as well as on standard UDA benchmarks - showing that is can compete with state-of-the-art approaches even in standard settings.