A Prototype-Oriented Framework for Unsupervised Domain Adaptation

1 Introduction . In many real-world applications , such as healthcare and autonomous driving , data labeling can be expensive and time-consuming . To make predictions on a new unlabeled dataset , one may naively use an existing supervised model trained on a large labeled dataset . However , even subtle changes in the data-collection conditions , such as lighting or background for natural images , can cause a model ’ s performance to degrade drastically [ 1 ] . This shift in the input data distribution is referred to in the literature as covariate shift [ 2 ] . By leveraging the labeled samples from the source domain and unlabeled samples from the target domain , unsupervised domain adaptation aims to overcome this issue , making the learned model generalize well in the target domain [ 3 ] . Ben-David et al . [ 4 , 5 ] provide an H-divergence based theoretical upper bound on the target error . Ganin [ 6 ] popularizes learning an invariant representation between the source and target domains to minimize this divergence . Numerous prior methods [ 7–12 ] follow this trend , focusing on using the source and target samples for feature alignment in the latent space . While this approach can reduce the discrepancy between domains , directly using the source and target samples for feature alignment has the following problems . First , several commonly used methods that can be used to quantify the difference between two empirical distributions , such as maximum mean discrepancy ( MMD ) [ 13 ] and Wasserstein distance [ 14 , 15 ] , are sensitive to outlier samples in a mini-batch when used to match the source and target marginal distributions [ 16 , 17 ] . We attribute this problem to the sampling variability of both the source and target samples . Second , while we typically assume that the two domains share the same label space , we can not guarantee that the samples drawn from the source and target domains will cover the same set of classes in each mini-batch . Especially , if the label proportions shift between domains , learning domain invariant representation might not lead to improvements over using the source data alone to train the model [ 18 ] . If we pull the support of ∗ Equal contribution . Corresponding to : mingyuan.zhou @ mccombs.utexas.edu PyTorch code is available at https : //github.com/korawat-tanwisuth/Proto_DA 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the source and target feature representations from different classes closer together , the classifier will be more likely to misclassify those examples . Finally , aligning the target features to source features means that we need access to both the source and target data simultaneously . In applications such as personal healthcare , we may not have access to the source data directly during the adaptation stage ; instead , we may only be given access to the target data and the model trained on the source data . We propose an algorithm that constructs class prototypes to represent the source domain samples in the latent space . Using the prototypes instead of source features avoids the previously mentioned problems : 1 ) sampling variability in the source domain , 2 ) instance class-mismatching in a mini-batch , and 3 ) source-data privacy concerns . As we expect the classifier to make better predictions on the target data in regions where the source density is sufficiently high [ 19 ] , it is natural to consider encouraging the feature encoder to map the target data close to these prototypes . Motivated by the cluster assumption [ 20 ] ( decision boundaries should not cross high-density regions of the data ) , we provide a method to transport the target features to these class prototypes and vice versa . We further extend our bi-directional transport to address the potential shift in label proportions , a common problem that has been studied [ 21–25 ] but that has been often overlooked in prior works [ 6 , 7 , 26 ] . Compared to existing methods , the proposed one has several appealing aspects . First , it does not rely on adversarial training to achieve competitive performance , making the algorithm robust and converge much faster . Moreover , learnable prototypes not only avoid expensive computation but also bypass the need to directly access the source data . This attribute makes our algorithm applicable to the settings where preserving the source data privacy is a major concern . Unlike clustering-based approaches that typically require multiple forward passes before an update , our algorithm processes data in mini-batches for each update and is trained in an end-to-end manner . We highlight the main contributions of the paper as follows : 1 ) We utilize the linear classifier ’ s weights as class prototypes and propose a general probabilistic framework to align the target features to these prototypes . 2 ) We introduce the minimization of the expected cost of a probabilistic bi-directional transport for feature alignment , and illustrate its superior performance over related methods . 3 ) We test the proposed method under multiple challenging yet practical settings : single-source , multisource , class-imbalance , and source-private domain adaptation . In comparison to state-of-the-art algorithms , the proposed prototype-oriented method achieves highly competitive performance in domain adaptation , while requiring no additional model parameters and only having a moderate increase in computation over the source model alone . 2 Prototype-oriented conditional transport . In this section , we propose Prototype-oriented Conditional Transport ( PCT ) , a holistic method for domain adaption consisting of three parts : learning class prototypes , aligning the target features with learned prototypes using a probabilistic bi-directional transport framework of Zheng and Zhou [ 27 ] , and estimating the target class proportions . Our method does not introduce additional model parameters for aligning domain features , and the model can be learned in an end-to-end manner . We provide a motivating example of the application of our method on a synthetic dataset in Figure 2 . In domain adaptation , we are given a labeled dataset from the source domain , { ( xsi , ysi ) } ns i=1 ∼ Ds , and an unlabeled dataset from the target domain , { xtj } nt j=1 ∼ Dxt . We focus on the closed-category domain adaptation and assume that the source and target domains share the same label space , i.e. , ysi , y t j ∈ { 1 , 2 , . . . , K } , where K denotes the number of classes . The goal of domain adaptation is to learn a model with low risk on the target samples . The model typically consists of a feature encoder , Fθ : X → Rdf , parameterized by θ , and a linear classification layer Cµ : Rdf → RK , parameterized by µ . In prior works [ 6 , 7 , 26 ] , the feature encoder is a pre-trained neural network , and the classifier is a randomly initialized linear layer . To simplify the following notation , we denote fsi = Fθ ( x s i ) and f tj = Fθ ( x t j ) as the feature representations of the source data x s i and target data x t j , respectively . 2.1 Learning class prototypes . Most existing works [ 6 , 7 , 26 ] focus on aligning the source and target features in a latent space . By contrast , we propose to characterize the features of each class with a class prototype and align the target features with these class prototypes instead of the source features . This approach has several advantages . First , the feature alignment between two domains would be more robust to outliers in the source domain as we avoid using the source samples directly . Second , we do not need to worry about the missing classes in the sampled mini-batch in the source domain like we do when we align the features of source and target samples . Prototypes ensure that every class is represented for each training update . Last but not least , using the inferred prototypes instead of source features allows adapting to the target domain even without accessing the source data during the adaptation stage , which is an appealing trait when preserving the source data privacy is a concern ( see Table 6 ) . Previous methods [ 28–31 , 12 , 32 ] construct each class prototype as the average latent feature for that class extracted by the feature encoder , which is computationally expensive due to the forward passing of a large number of training samples . We propose to construct class prototypes in the same latent space but with learnable parameters : [ µ1 , µ2 , . . . , µK ] ∈ Rdf×K , where the dimension of each prototype , df , is the same as the hidden dimension after the feature encoder Fθ . This strategy has been successfully applied by Saito et al . [ 33 ] in a semi-supervised learning setting . We learn each class prototype in a way that encourages the prototype to be close to the source samples associated with that class in the feature space . In particular , given the prototypes and source samples xsi with features fsi and labels y s i , we use the cross-entropy loss to learn the prototypes : Lcls = E ( xsi , ysi ) ∼Ds [ ∑K k=1− log psik1 { ysi=k } ] , psik : = exp ( µTk f s i+bk ) ∑K k′=1 exp ( µ T k′f s i+bk′ ) , ( 1 ) where bk is a bias term and psik is the predictive probability for x s i to be classified to class k. We note that this way of learning the class prototypes is closely connected to learning the standard linear classification layer Cµ on source-only data with the cross-entropy loss . The neural network weights in the classification layer can be interpreted as the class prototypes . Therefore , compared with sourceonly approaches , constructing prototypes in this way introduce no additional parameters . As we show in Figure 4a , our method requires much fewer parameters than other domain-adaptation methods . Moreover , it requires much less computation than other prototype-based methods by avoiding the need to average the latent features for each class . 2.2 Bi-directional prototype-oriented conditional transport . In this section , we will discuss how we encourage the feature encoder to align the target data with class prototypes . Our approach is motivated by the cluster assumption , which has been widely adopted in both semi-supervised learning [ 20 , 34 , 35 ] and domain-adaptation literature [ 36 , 37 , 31 , 38 ] . The cluster assumption states that the input data distribution consists of separated clusters and that instances belonging to the same cluster tend to have the same class labels . This means that the decision boundaries should not cross data high-density regions . To achieve this goal , we minimize the expected pairwise cost between the target features and prototypes with respect to two differently constructed joint distributions . By minimizing the expected costs under these two different joint distributions , the target feature will be close to the prototypes , far from the decision boundaries . 2.2.1 Moving from target domain to class prototypes . To define the expected cost of moving from the target domain to the class prototypes , we first use the chain rule to factorize the joint distribution of the class prototype µk and target feature f t j as p ( f tj ) πθ ( µk |f t j ) , where drawing from the target feature distribution p ( f t j ) can be realized by selecting a random target sample xtj ∼ Dxt to obtain f t j = Fθ ( x t j ) . The conditional distribution , representing the probability of moving from target feature f tj to class prototype µk , is defined as πθ ( µk |f t j ) = p ( µk ) exp ( µ T k f t j ) ∑K k′=1 p ( µk′ ) exp ( µ T k′f t j ) , k ∈ { 1 , . . . , K } , ( 2 ) where , through the lens of Bayes ’ rule , p ( µk ) is the discrete prior distribution over the K classes for the target domain , and exp ( µTk f t j ) plays the role of an unnormalized likelihood term , measuring the similarity between class prototypes and target features . Intuitively , the target features are more likely to be moved to the prototypes which correspond to dominant classes in the target domain or which are closer to the target features ( or both ) . Note that in practice we often do not have access to the target class distribution p ( µk ) . We can use a uniform prior distribution for p ( µk ) . However , this could be sub-optimal , especially when classes are seriously imbalanced in the target domain . To address this issue , we propose a way to estimate { p ( µk ) } Kk=1 in Section 2.3 . We now define the expected cost of moving the target features to class prototypes as : Lt→µ = Extj∼Dxt Eµk∼πθ ( µk | ftj ) [ c ( µk , f t j ) ] = Extj∼Dxt [ ∑K k=1 c ( µk , f t j ) p ( µk ) exp ( µ T k f t j ) ∑K k′=1 p ( µk′ ) exp ( µ T k′f t j ) ] , ( 3 ) where c ( · , · ) , a point-to-point moving cost , is defined with the cosine dissimilarity as c ( µk , f t j ) = 1− µTk f t j ||µk||2||ftj ||2 . ( 4 ) We also consider other point-to-point costs and present the results in Section 4.2 . With Eq . ( 3 ) , it is straightforward to obtain an unbiased estimation of Lt→µ with a mini-batch from target domain Dxt . In this target-to-prototype direction , we are assigning each target sample to the prototypes according to their similarities and the class distribution . Intuitively , minimizing this expected moving cost encourages each target feature to get closer to neighboring class prototypes , reducing the violation of the cluster assumption . If we think of each prototype as the mode of the distribution of source features for a class , this loss encourages a mode-seeking behavior [ 27 ] . Still , this loss alone might lead to sub-optimal alignment . The feature encoder can map most of the target data to only a few prototypes . We connect this loss to entropy minimization to elucidate this point . Connection with entropy minimization . The expected cost of moving from the target domain to prototypes can be viewed as a generalization of entropy minimization [ 20 ] , an effective regularization in many prior domain-adaptation works [ 39 , 33 , 40 , 17 ] . If the point-to-point moving cost is defined as c ( µk , f t j ) = − log ptjk = − log exp ( µTk f t j ) ∑K k′=1 exp ( µ T k′f t j ) and the conditional probability is πθ ( µk |f t j ) = p t jk = exp ( µTk f t j ) ∑K k′=1 exp ( µ T k′f t j ) ( with a uniform prior ) , then the expected moving cost becomes : Lt→µ = −Extj∼Dxt [ ∑K k=1 p t jk log p t jk ] , which is equivalent to minimizing the entropy on the target samples . Entropy minimization alone also has a mode-seeking behavior and has the same tendency to drop some modes ( class prototypes ) . In other words , one trivial solution is to assign the same one-hot encoding to all the target samples [ 41 , 42 ] .

Summary: This paper proposes creating prototypes(p) for the source data and aligning these prototypes with the target(t) data via bidirectional conditional transport measure. They use last linear layer's weights as prototypes, so it saves cost, and their method do no require source data for adaptation. For the alignment, they define bidirectional transport, that has  prototypes to targets part and targets to prototypes part in the objective function. For the former, they add prior prototype distributions that comes from Bayes' rule and approximate via EM algorithm. The latter  corresponds to entropy minimization. They obtain state of art results in several domain adaptation scenarios.