A Prototype-Oriented Framework for Unsupervised Domain Adaptation
Year: 2021

Abstract:
Existing methods for unsupervised domain adaptation often rely on minimizing some statistical distance between the source and target samples in the latent space. To avoid the sampling variability, class imbalance, and data-privacy concerns that often plague these methods, we instead provide a memory and computationefficient probabilistic framework to extract class prototypes and align the target features with them. We demonstrate the general applicability of our method on a wide range of scenarios, including single-source, multi-source, class-imbalance, and source-private domain adaptation. Requiring no additional model parameters and having a moderate increase in computation over the source model alone, the proposed method achieves competitive performance with state-of-the-art methods.

heading: 1 Introduction
text: In many real-world applications, such as healthcare and autonomous driving, data labeling can be expensive and time-consuming. To make predictions on a new unlabeled dataset, one may naively use an existing supervised model trained on a large labeled dataset. However, even subtle changes in the data-collection conditions, such as lighting or background for natural images, can cause a model’s performance to degrade drastically [1]. This shift in the input data distribution is referred to in the literature as covariate shift [2]. By leveraging the labeled samples from the source domain and unlabeled samples from the target domain, unsupervised domain adaptation aims to overcome this issue, making the learned model generalize well in the target domain [3].
Ben-David et al. [4, 5] provide an H-divergence based theoretical upper bound on the target error. Ganin [6] popularizes learning an invariant representation between the source and target domains to minimize this divergence. Numerous prior methods [7–12] follow this trend, focusing on using the source and target samples for feature alignment in the latent space. While this approach can reduce the discrepancy between domains, directly using the source and target samples for feature alignment has the following problems. First, several commonly used methods that can be used to quantify the difference between two empirical distributions, such as maximum mean discrepancy (MMD) [13] and Wasserstein distance [14, 15], are sensitive to outlier samples in a mini-batch when used to match the source and target marginal distributions [16, 17]. We attribute this problem to the sampling variability of both the source and target samples. Second, while we typically assume that the two domains share the same label space, we cannot guarantee that the samples drawn from the source and target domains will cover the same set of classes in each mini-batch. Especially, if the label proportions shift between domains, learning domain invariant representation might not lead to improvements over using the source data alone to train the model [18]. If we pull the support of
∗ Equal contribution. Corresponding to: mingyuan.zhou@mccombs.utexas.edu PyTorch code is available at https://github.com/korawat-tanwisuth/Proto_DA
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
the source and target feature representations from different classes closer together, the classifier will be more likely to misclassify those examples. Finally, aligning the target features to source features means that we need access to both the source and target data simultaneously. In applications such as personal healthcare, we may not have access to the source data directly during the adaptation stage; instead, we may only be given access to the target data and the model trained on the source data.
We propose an algorithm that constructs class prototypes to represent the source domain samples in the latent space. Using the prototypes instead of source features avoids the previously mentioned problems: 1) sampling variability in the source domain, 2) instance class-mismatching in a mini-batch, and 3) source-data privacy concerns. As we expect the classifier to make better predictions on the target data in regions where the source density is sufficiently high [19], it is natural to consider encouraging the feature encoder to map the target data close to these prototypes. Motivated by the cluster assumption [20] (decision boundaries should not cross high-density regions of the data), we provide a method to transport the target features to these class prototypes and vice versa. We further extend our bi-directional transport to address the potential shift in label proportions, a common problem that has been studied [21–25] but that has been often overlooked in prior works [6, 7, 26].
Compared to existing methods, the proposed one has several appealing aspects. First, it does not rely on adversarial training to achieve competitive performance, making the algorithm robust and converge much faster. Moreover, learnable prototypes not only avoid expensive computation but also bypass the need to directly access the source data. This attribute makes our algorithm applicable to the settings where preserving the source data privacy is a major concern. Unlike clustering-based approaches that typically require multiple forward passes before an update, our algorithm processes data in mini-batches for each update and is trained in an end-to-end manner.
We highlight the main contributions of the paper as follows: 1) We utilize the linear classifier’s weights as class prototypes and propose a general probabilistic framework to align the target features to these prototypes. 2) We introduce the minimization of the expected cost of a probabilistic bi-directional transport for feature alignment, and illustrate its superior performance over related methods. 3) We test the proposed method under multiple challenging yet practical settings: single-source, multisource, class-imbalance, and source-private domain adaptation. In comparison to state-of-the-art algorithms, the proposed prototype-oriented method achieves highly competitive performance in domain adaptation, while requiring no additional model parameters and only having a moderate increase in computation over the source model alone.

heading: 2 Prototype-oriented conditional transport
text: In this section, we propose Prototype-oriented Conditional Transport (PCT), a holistic method for domain adaption consisting of three parts: learning class prototypes, aligning the target features with learned prototypes using a probabilistic bi-directional transport framework of Zheng and Zhou
[27], and estimating the target class proportions. Our method does not introduce additional model parameters for aligning domain features, and the model can be learned in an end-to-end manner. We provide a motivating example of the application of our method on a synthetic dataset in Figure 2.
In domain adaptation, we are given a labeled dataset from the source domain, {(xsi , ysi )} ns i=1 ∼ Ds, and an unlabeled dataset from the target domain, {xtj} nt j=1 ∼ Dxt . We focus on the closed-category domain adaptation and assume that the source and target domains share the same label space, i.e., ysi , y t j ∈ {1, 2, . . . ,K}, where K denotes the number of classes. The goal of domain adaptation is to learn a model with low risk on the target samples. The model typically consists of a feature encoder, Fθ : X → Rdf , parameterized by θ, and a linear classification layer Cµ : Rdf → RK , parameterized by µ. In prior works [6, 7, 26], the feature encoder is a pre-trained neural network, and the classifier is a randomly initialized linear layer. To simplify the following notation, we denote fsi = Fθ(x s i ) and f tj = Fθ(x t j) as the feature representations of the source data x s i and target data x t j , respectively.

heading: 2.1 Learning class prototypes
text: Most existing works [6, 7, 26] focus on aligning the source and target features in a latent space. By contrast, we propose to characterize the features of each class with a class prototype and align the target features with these class prototypes instead of the source features. This approach has several advantages. First, the feature alignment between two domains would be more robust to outliers in the source domain as we avoid using the source samples directly. Second, we do not need to worry about the missing classes in the sampled mini-batch in the source domain like we do when we align the features of source and target samples. Prototypes ensure that every class is represented for each training update. Last but not least, using the inferred prototypes instead of source features allows adapting to the target domain even without accessing the source data during the adaptation stage, which is an appealing trait when preserving the source data privacy is a concern (see Table 6).
Previous methods [28–31, 12, 32] construct each class prototype as the average latent feature for that class extracted by the feature encoder, which is computationally expensive due to the forward passing of a large number of training samples. We propose to construct class prototypes in the same latent space but with learnable parameters: [µ1,µ2, . . . ,µK ] ∈ Rdf×K , where the dimension of each prototype, df , is the same as the hidden dimension after the feature encoder Fθ. This strategy has been successfully applied by Saito et al. [33] in a semi-supervised learning setting. We learn each class prototype in a way that encourages the prototype to be close to the source samples associated with that class in the feature space. In particular, given the prototypes and source samples xsi with features fsi and labels y s i , we use the cross-entropy loss to learn the prototypes:
Lcls = E(xsi ,ysi )∼Ds [∑K k=1− log psik1{ysi=k} ] , psik := exp(µTk f s i+bk)∑K
k′=1 exp(µ T k′f s i+bk′ )
, (1)
where bk is a bias term and psik is the predictive probability for x s i to be classified to class k. We note that this way of learning the class prototypes is closely connected to learning the standard linear classification layer Cµ on source-only data with the cross-entropy loss. The neural network weights in the classification layer can be interpreted as the class prototypes. Therefore, compared with sourceonly approaches, constructing prototypes in this way introduce no additional parameters. As we show in Figure 4a, our method requires much fewer parameters than other domain-adaptation methods. Moreover, it requires much less computation than other prototype-based methods by avoiding the need to average the latent features for each class.

heading: 2.2 Bi-directional prototype-oriented conditional transport
text: In this section, we will discuss how we encourage the feature encoder to align the target data with class prototypes. Our approach is motivated by the cluster assumption, which has been widely adopted in both semi-supervised learning [20, 34, 35] and domain-adaptation literature [36, 37, 31, 38]. The cluster assumption states that the input data distribution consists of separated clusters and that instances belonging to the same cluster tend to have the same class labels. This means that the decision boundaries should not cross data high-density regions. To achieve this goal, we minimize the expected pairwise cost between the target features and prototypes with respect to two differently constructed joint distributions. By minimizing the expected costs under these two different joint distributions, the target feature will be close to the prototypes, far from the decision boundaries.

heading: 2.2.1 Moving from target domain to class prototypes
text: To define the expected cost of moving from the target domain to the class prototypes, we first use the chain rule to factorize the joint distribution of the class prototype µk and target feature f t j as p(f tj)πθ(µk |f t j), where drawing from the target feature distribution p(f t j) can be realized by selecting a random target sample xtj ∼ Dxt to obtain f t j = Fθ(x t j). The conditional distribution, representing the probability of moving from target feature f tj to class prototype µk, is defined as
πθ(µk |f t j) =
p(µk) exp(µ T k f t j)∑K
k′=1 p(µk′ ) exp(µ T k′f t j) , k ∈ {1, . . . ,K}, (2)
where, through the lens of Bayes’ rule, p(µk) is the discrete prior distribution over the K classes for the target domain, and exp(µTk f t j) plays the role of an unnormalized likelihood term, measuring the similarity between class prototypes and target features. Intuitively, the target features are more likely to be moved to the prototypes which correspond to dominant classes in the target domain or which are closer to the target features (or both). Note that in practice we often do not have access to the target class distribution p(µk). We can use a uniform prior distribution for p(µk). However, this could be sub-optimal, especially when classes are seriously imbalanced in the target domain. To address this issue, we propose a way to estimate {p(µk)}Kk=1 in Section 2.3. We now define the expected cost of moving the target features to class prototypes as:
Lt→µ = Extj∼Dxt Eµk∼πθ(µk | ftj) [ c(µk,f t j) ] = Extj∼Dxt [∑K k=1 c(µk,f t j) p(µk) exp(µ T k f t j)∑K
k′=1 p(µk′ ) exp(µ T k′f t j)
] , (3)
where c(·, ·), a point-to-point moving cost, is defined with the cosine dissimilarity as
c(µk,f t j) = 1−
µTk f t j
||µk||2||ftj ||2 . (4)
We also consider other point-to-point costs and present the results in Section 4.2. With Eq. (3), it is straightforward to obtain an unbiased estimation of Lt→µ with a mini-batch from target domain Dxt . In this target-to-prototype direction, we are assigning each target sample to the prototypes according to their similarities and the class distribution. Intuitively, minimizing this expected moving cost encourages each target feature to get closer to neighboring class prototypes, reducing the violation of the cluster assumption. If we think of each prototype as the mode of the distribution of source features for a class, this loss encourages a mode-seeking behavior [27]. Still, this loss alone might lead to sub-optimal alignment. The feature encoder can map most of the target data to only a few prototypes. We connect this loss to entropy minimization to elucidate this point.
Connection with entropy minimization. The expected cost of moving from the target domain to prototypes can be viewed as a generalization of entropy minimization [20], an effective regularization in many prior domain-adaptation works [39, 33, 40, 17]. If the point-to-point moving cost is
defined as c(µk,f t j) = − log ptjk = − log exp(µTk f t j)∑K
k′=1 exp(µ T k′f t j)
and the conditional probability is
πθ(µk |f t j) = p t jk = exp(µTk f t j)∑K
k′=1 exp(µ T k′f t j)
(with a uniform prior), then the expected moving cost becomes: Lt→µ = −Extj∼Dxt [∑K k=1 p t jk log p t jk ] , which is equivalent to minimizing the entropy on the target samples. Entropy minimization alone also has a mode-seeking behavior and has the same tendency to drop some modes (class prototypes). In other words, one trivial solution is to assign the same one-hot encoding to all the target samples [41, 42].

heading: 2.2.2 Moving from class prototypes to target domain
text: To ensure that each prototype has some target features located close by and avoid dropping class prototypes, we propose to add a cost of the opposite direction [27], i.e., moving from the prototypes to target features. Given a mini-batch, {xtj}Mj=1, of target samples of size M , denoting p̂(f
t) =∑M j=1 1 M δftj as the empirical distribution of the target features in this mini-batch, the probabilities of moving from a prototype µk to the M target features is defined as a conditional distribution:
πθ(f t j |µk) =
p̂(ftj) exp(µ T k f t j)∑M
j′=1 p̂(f t j′ ) exp(µ T k f t j′ )
= exp(µTk f t j)∑M
j′=1 exp(µ T k f t j′ ) , f tj ∈ {f t 1, . . . ,f t M}. (5)
As opposed to the probabilities of moving a target feature to different class prototypes πθ(µk |f t j), πθ(f t j |µk) normalizes the probabilities across the M target samples for each prototype, which ensures that each prototype will be assigned to some target features. Then, the expected cost of moving along this prototype-to-target direction is defined as:
Lµ→t = E{xtj}Mj=1∼Dxt Eµk∼p(µk)Eftj∼πθ(ftj |µk) [ c(µk,f t j) ]
= E{xtj}Mj=1∼Dxt [∑K k=1 p(µk) ∑M j=1 c(µk,f t j) exp(µTk f t j)∑M
j′=1 exp(µ T k f t j′ )
] , (6)
which can be estimated by drawing a mini-batch of M target samples.
Finally, combining the classification loss in Eq. (1), target-to-prototype moving cost in Eq. (3), and prototype-to-target moving cost in Eq. (6), our loss is expressed as
Lcls + Lt→µ + Lµ→t. (7)
Note that we treat µ as fixed in both Lt→µ and Lµ→t. This strategy allows us to apply our method in the source-data-private setting where we only have access to the source model. We also find empirically that this leads to more stable training.

heading: 2.3 Learning class proportions in the target domain
text: We propose to infer the class proportions {p(µk)}Kk=1 in the target domain by maximizing the loglikelihood of the unlabeled target data while fixing the class prototypes µ. Directly optimizing the marginal likelihood is intractable, so we use the EM algorithm [43–45] to derive the following iterative updates (see the derivation in Appendix B). We first initialize with a uniform prior: p(µk)
0 = 1K , and obtain new estimates at each update step l (starting from 0):
p(µk) l+1 = 1M ∑M j=1 π l θ(µk |f t j), where π l θ(µk |f t j) =
p(µk) l exp(µTk f t j)∑K
k′=1 p(µk′ ) l exp(µT k′f t j) . (8)
Intuitively, the average predicted probabilities over the target examples for each class are used to estimate the target proportions, with p(µk)
l+1 shown above providing an estimate based on a single mini-batch of M target samples. To estimate it based on the full dataset, we iteratively update it with p(µk)
l+1 ← (1− βl)p(µk)l + βlp(µk)l+1, where we follow the decaying schedule of the learning rate of the other parameters to set βl = β0(1 + γl)−α, in which γ = 0.0002, α = 0.75. The inintial value β0 is a hyper-parameter that can be set as either 0 to indicate a uniform prior, or a small value, such as 0.001, to allow the class proportions to be inferred from the data.

heading: 3 Related work
text: Feature distribution alignment. Most works on domain adaptation with deep learning focus on feature alignment to align either the marginal distributions [46, 6, 8, 47, 48] or the joint distributions [7, 26] of the deep features of neural networks. Early works use adversarial-based objectives, which are equivalent to minimizing the Jensen–Shannon (JS) divergence [49]. When the source and target domains have non-overlapping supports, the JS divergence fails to supply useful gradients [50, 51]. To remedy this issue, researchers propose using the Wasserstein distance, which arises from the optimal transport problem [52]. Courty et al. [9] develop Joint Distribution Optimal Transport (JDOT), defining the transport cost to be the joint cost in the data and prediction spaces. Several works [10, 12] extend this framework for deep neural networks. However, solving for the optimal couplings without
any relaxation is a linear programming problem, which has a complexity of O(M3 logM) [53]. Computing the transport probabilities in our algorithm has a complexity of O(dfMK), which is the same complexity as computing the predictive probabilities that need to be computed by most algorithms. In this category, all the works focus on aligning the target features with source features, whereas we align the target features with prototypes.
Prototype-based alignment. We make two distinctions to existing works in this area. First, prior domain-adaptation works for classification [29–31] and segmentation [54, 55] utilize prototypes for pseudo-label assignments. By contrast, we use prototypes to behave as representative samples of the source features to define the loss. Second, all of these works use some form of average latent features to construct class prototypes, which is computationally expensive. We instead adopt a parametric approach to learn the prototypes, avoiding that costly computation.
Learning under shifted class proportions. Although a shift in class proportions between two domains is a common problem in many applications, it is still largely under-explored. Over the years, researchers have viewed the question through different lenses: applying kernel distribution embedding [56, 57, 21], using an EM update [43, 58], placing a meta-prior over the target proportions [59], and casting the problem under causal and anti-causal learning [21–25]. Recently, Tachet des Combes et al. [18] propose aligning the target feature distribution with a re-weighted version of the source feature distribution. While this method achieves consistent improvements over feature-alignment algorithms, it still relies on learning domain-invariant representations. As we have discussed, this approach suffers from the problems of sampling variability and class-mismatching in a mini-batch, whereas the proposed method uses class prototypes and proportion estimation to avoid these issues.
Source-private adaptation. Finally, the proposed method can also be applied to a source-private setting where without seeing the raw source data, we only have access to the source model and target data while adapting to the target domain [31, 47, 60–63]. Liang et al. [31] introduce a clustering-based approach to generate pseudo-labels for the target data. That approach requires constructing class centers using a weighted average of the latent features. Different from that work, our prototypes behave as class centers and are more amenable to mini-batch stochastic gradient based training.

heading: 4 Experiments
text: In this section, we evaluate our method under four practical settings: single-source, multi-source, class-imbalance, and source-private domain adaptation. We present the setup, the results, and the analysis of the results in the upcoming sections.
Datasets. We use the following three datasets of varying sizes in our experiment: 1) Office-31 [3] consists of 4652 images coming from three domains: Amazon (A), Webcam (W), and DSLR (D). The total number of categories is 31. 2) Office-Home [64], a more challenging dataset than Office-31, consists of 15,500 images over 65 classes and four domains: Artistic images (Ar), Clip art (Cl), Product images (Pr), and Real-world (Rw). 3) DomainNet [65] is a large-scale dataset for domain adaptation. It consists of about 569,010 images with 345 categories from six domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. We further perform experiments on Cross-Digits, ImageClef, Office-Caltech, and VisDA datasets and provide the details in Appendix E.2.
Implementation details. We implement our method on top of the open-source transfer learning library (MIT license) [66], adopting the default neural network architectures for both the feature encoder and linear classifier. For the feature encoder network, we utilize a pre-trained ResNet-50 in all experiments except for multi-source domain adaptation, where we use a pre-trained ResNet-101. We fine-tune the feature encoder and train the linear layers from random initialization. The linear layers have the learning rate of 0.01, 10 times that of the feature encoder. The learning rate follows the following schedule as ηiter = η0(1 + γiter)−α, where η0 is the initial learning rate. We set η0 to 0.01, γ to 0.0002, and α to 0.75. We utilize a mini-batch SGD with a momentum of 0.9. We set the batch size for the source data asN = 32 and that for the target data asM = 96. We use all the labeled source samples and unlabeled target samples [6, 7, 26]. We set β0 to 0 (a uniform prior) in all settings except for the sub-sampled target datasets. We perform a sensitivity analysis (see Appendix E) and set β0 empirically to 0.001 for the sub-sampled target version of Office-31 and 0.0001 for that of Office-Home. We report the average accuracy from three independent runs. All experiments are conducted using a single Nvidia Tesla V100 GPU except for the DomainNet experiment, where we use four V100 GPUs. More implementation details can be found in Appendix D.

heading: 4.1 Main results
text: Single-source setting. We perform single-source domain adaptation on the Office-31 and OfficeHome datasets. In each experiment, one domain serves as the source domain and another as the target domain. We consider all permutations, leading to 6 tasks for the Office-31 dataset and 12 tasks for the Office-Home dataset. We compare our algorithm with state-of-the-art algorithms for domain adaptations from three different categories: adversarial-based, divergence-based, and optimal transport-based. We report the results on Office-31 in Table 1. PCT significantly outperforms the baselines, especially on the more difficult transfer tasks (D→A and W→ A). Although MDD [67], the best baseline domain-adaptation method, uses a bigger classifier, PCT still has 1.1% higher average accuracy. In Figure 4a, we visualize the number of parameters versus the average accuracy on the Office-31 dataset. While PCT uses fewer parameters than most methods, it still achieves the highest average accuracy. We report the average accuracies on the Office-Home dataset in Table 2. PCT outperforms baseline methods on 10 of the 12 transfer tasks, yielding 3.7% improvement on the average accuracy over MDD. The results in this setting demonstrate that aligning the target features with prototypes is more effective than directly aligning them with the source features.
PCT (Ours) 57.1 ±0.3 78.3 ±1.2 81.4 ±0.4 67.6 ±0.3 77.0 ±1.2 76.5 ±0.5 68.0 ±0.5 55.0 ±0.6 81.3 ±0.2 74.7 ±0.5 60.0 ±0.5 85.3 ±0.3 71.8
Multi-source setting. In this setting, we evaluate our method using Office-Home and DomainNet datasets [65]. For each task, we select one domain as the target domain and use the remaining five domains as the source. We use the same data splitting scheme as the original paper [65]. We compare against source-combined and multi-source algorithms introduced in Venkat et al. [70] for Office-Home and in Peng et al. [65] for DomainNet. Multi-source algorithms use domain labels and a classifier for each source domain, whereas source-combined algorithms combine all the source domains into a single source domain and perform single-source adaptation. We adopt a single classifier and do not use domain labels. Thus, PCT falls under the source-combined category. We report the results in Tables 3 and 4. While PCT is not designed specifically for multi-source domain adaptation, our method still outperforms those multi-source algorithms in both datasets. Since there are more variations of the data in the source domain in this setting, the increase in performance gain verifies our intuition that prototypes help mitigate the problem of sampling variability.
Sub-sampled setting. In many cases, the label proportions could significantly change from one dataset to another, resulting in a decrease in a model’s performance. To test our algorithm under this setting, we follow the experimental protocol in Tachet des Combes et al. [18]. We keep only thirty percent of the first bK/2c classes to simulate class imbalance. We directly take their results for the sub-sampled source data and perform additional experiments using the same sub-sampling scheme on the target data. The baselines in this setting are standard domain adaptation methods (DAN, JAN, and CDAN) and their importance weighted versions introduced in Tachet des Combes et al. [18]. We present the results in Table 5. On the sub-sampled source data, PCT with uniform prior outperforms the second-best method (IWCDAN) by 4% on Office-31 and 6.6% on Office-Home. Learning the prior distribution on the target domain does not improve the result, as this setting does not have a serious imbalance issue in the target domain. On the sub-sampled target data, PCT with a uniform prior already outperforms the baselines, 1.9% and 5.2% higher average accuracy than IWCDAN’s on Office-31 and Office-Home, respectively. Using a learnable prior further improves upon using a uniform prior by 1.0% on Office-31 and by 0.4% on Office-Home. The improvements confirm our intuition that prototypes help with the class imbalance in both the source and target domain while estimating the target proportion further boosts the performance in the target domain sub-sampled setting. We visualize the estimated proportions on the target data in Figure 3, verifying that the proportions are inferred correctly.
Source-data-private setting. In many practical applications, practitioners might not directly have access to the source data in the adaptation stage. Instead, a trained model on the source data is provided. In this setting, the goal is to adapt to the target domain while only operating on the given model. We compare our method with Source Hypothesis Transfer (SHOT) [31], a state-ofthe-art method proposed specifically for this setting. SHOT contains two losses: an information maximization (IM) loss and a pseudo-labeling loss. We follow their experimental protocol by first training the model on the source data alone. During the adaptation stage, we only use the target data to perform model adaptation. We use the transport losses to update the feature encoder while fixing the prototypes. We report the results in Table 6. From the standard setting where we have access to source data in Table 1, the average accuracy drops by 1.6% for Office-31 and by 0.8% for Office-Home. On the Office-31 dataset, our bi-directional loss outperforms the IM loss by 1.1% and the pseudo label loss by 0.8%. While the average accuracy for our method is 0.2% lower than both of their losses combined, the p-value for the independent two-sample t-test on the accuracies of different runs is 0.32, which is not statistically significant. On the Office-Home dataset, our approach performs 1.9% and 0.5% better than the pseudo label and IM losses, respectively. While their combined loss achieves 0.8% accuracy higher than that of our method, the pseudo-labeling loss in SHOT requires
constructing class centers, which does not scale well with large datasets. Our approach uses the classifier’s weights as class prototypes to avoid this issue.

heading: 4.2 Analysis of results
text: Ablation study. To examine the effect of each component in our framework, we perform ablation studies and present the results in Table 7. 1) Alignment strategy. Next, we present the result using optimal transport as the alignment strategy. We consider two variants of Prototype-oriented Optimal Transport (POT): exact linear program (POT) and Sinkhorn relaxation (POT-Sinkhorn). In each variant, we solve for the optimal couplings using the optimal transport formulation. After obtaining the transport probabilities, we update the feature encoder using the obtained probabilities as the weights for the transport cost. We can see that POT performs 1.7% better than DeepJDOT in Table 1, showing the effectiveness of using prototypes to define the transport costs with the target features. Still, both versions of POT underperforms PCT by 2.1% and 1.7%, respectively. 2) Effect of each loss in PCT. We examine the effect of each loss on the average test accuracy on the Office-31 dataset. We remove each transport loss while keeping the cross-entropy loss. The bi-directional loss leads to the best accuracy, while the drop in accuracy is more significant if we remove Lµ→t. This result is not surprising because, without Lµ→t, the model can map target data to only a few prototypes, leading to a degenerate solution. 3) Gradient stopping. We also show the algorithm’s performance without stopping the gradient of µ in the transport loss. PCT gains an additional 1.0% in accuracy with the gradient stopping strategy. The performance gain is consistent with the finding in recent work by Chen and He [77], where the authors apply the gradient stopping strategy to avoid degenerate solutions in contrastive learning. 4) Cost function. Finally, we explore the cost function inspired by the radial basis kernel. We can see that the cosine distance in PCT gives 3.1% higher average accuracy. In short, the choice of the probabilistic bi-directional transport framework, gradient-stopping strategy, and point-to-point cost function all contribute to the success of the proposed PCT method.
Table 7: Average accuracy (%) of PCT on Office-31 under different variants (ResNet-50).
POT POT-Sinkhorn PCT w/o (Lt→µ) PCT w/o (Lµ→t) w/o stop-grad c(µk,f t j) = exp(−µTk f t j) PCT (Ours) 87.9± 0.8 88.3± 0.9 88.6± 0.2 84.3± 0.9 89.0± 0.3 86.9± 0.4 90.0± 0.5
(a) (b) (c)
Figure 4: (a) Analysis of parameter efficiency, (b) comparison of convergence, and (c) a t-SNE visualization of the output of the feature encoder trained with PCT on the task A→W. In plot (c), prototypes (?), source features (·), and target features (×) are tightly clustered together for each class.
Convergence comparison. We plot test accuracy versus iteration number on the task (A→W) in Figure 4b to compare the convergence rate of different algorithms. We also visualize test accuracy versus convergence time in minutes in Appendix E. In both plots, PCT quickly converges within the first one thousand iterations, and the test accuracy does not fluctuate much thereafter. This phenomenon is not surprising since we use prototypes instead of the source features to align with the target features. We expect the prototypes to behave as representative samples of the source features, making the model converge quickly and stably.
Visualization. We visualize in Figure 4c the t-SNE plot of the source and target features as well as the prototypes for the task A→W. Figure 4c shows that both the source (dots ·) and target (crosses ×) features are close to the prototypes (black stars ?), indicating that our algorithm is learning meaningful prototypes and successfully align the target features with the prototypes.

heading: 5 Conclusion
text: We offer a holistic framework for unsupervised domain adaptation through the lens of a probabilistic bi-directional transport between the target features and class prototypes. With extensive experiments under various application scenarios of unsupervised domain adaptation, we show that the proposed prototype-oriented alignment method works well on multiple datasets, is robust against class imbalance, and can perform domain adaptation with no direct access to the source data. Without adding additional model parameters, our memory and computation-efficient algorithm achieves competitive performance with state-of-the-art methods on several widely used benchmarks.

heading: Acknowledgments
text: We thank Camillia Smith Barnes and Georgii Riabov for helpful discussions and feedback on the paper. K. Tanwisuth, X. Fan, H. Zheng, S. Zhang, and M. Zhou acknowledge the support of Grant IIS-1812699 from the U.S. National Science Foundation, the APX 2019 project sponsored by the Office of the Vice President for Research at The University of Texas at Austin, the support of a gift fund from ByteDance Inc., and the Texas Advanced Computing Center (TACC) for providing HPC resources that have contributed to the research results reported within this paper.

Review summaries:
In this paper, a UDA algorithm is proposed. The core idea is based on benefiting from the class prototypes in the source domain to align two domains in a shared embedding space, rather than using the source sample features. The class prototypes are learned on the source domain using a cross-entropy loss to relax computational load. Upon learning the prototypes, a transport-based loss function is used to align the target domain features with the learned prototypes. Experiments on several UDA scenarios using standard benchmark datasets are provided to demonstrate that the method is effective.

This paper proposes creating prototypes(p) for the source data and aligning these prototypes with the target(t) data via bidirectional conditional transport measure. They use last linear layer's weights as prototypes, so it saves cost, and their method do no require source data for adaptation. For the alignment, they define bidirectional transport, that has  prototypes to targets part and targets to prototypes part in the objective function. For the former, they add prior prototype distributions that comes from Bayes' rule and approximate via EM algorithm. The latter  corresponds to entropy minimization. They obtain state of art results in several domain adaptation scenarios.

This paper presents a prototype-based framework for UDA. The core idea is to employ a probabilistic bi-directional transport framework that aligns target domain samples with the source domain prototypes, where the prototypes are modelled using the classifier weights. A key highlight of the method is that it is applicable in the absence of source domain data, which is highly useful in practice. Benchmarking on three datasets shows the method achieves excellent results.