A Prototype-Oriented Framework for Unsupervised Domain Adaptation
Authors: Korawat Tanwisuth
XINJIE FAN
Huangjie Zheng
Shujian Zhang
Hao Zhang
Bo Chen
Mingyuan Zhou
Venue: NeurIPS 2021 Poster (2021)
Keywords: domain adaptation
Bayesian methods
distribution matching
data privacy
class imbalance
computer vision
deep learning

Abstract:  Existing methods for unsupervised domain adaptation often rely on minimizing some statistical distance between the source and target samples in the latent space. To avoid the sampling variability, class imbalance, and data-privacy concerns that often plague these methods, we instead provide a memory and computation-efficient probabilistic framework to extract class prototypes and align the target features with them. We demonstrate the general applicability of our method on a wide range of scenarios, including single-source, multi-source, class-imbalance, and source-private domain adaptation. Requiring no additional model parameters and having a moderate increase in computation over the source model alone, the proposed method achieves competitive performance with state-of-the-art methods.

A Prototype-Oriented Framework for Unsupervised Domain Adaptation
Korawat Tanwisuth" "1 Xinjie Fan Huangjie Zheng" .Suujia Zhang',
Hao Zhang", Bo Chen Mingyuan Zhou'
'The University of Texas at Austin 2 Cornell University Xiiiian University
korawat tanvisuth@utexas edu, mingyuan zhou@mccombs utexas. edu
Abstract
Existing methods for unsupervised domain adaptation often rely on minimizing some statistical distance between the source and target samples in the latent space. To avoid the sampling variability, class imbalance, and data-privacy concerns that often plague these methods, we instead provide a memory and computation- efficient probabilistic framework to extract class prototypes and align the target features with them. We demonstrate the general applicability our method on a wide range of scenarios, including single-source. multi-source class- imbalance,, and cource-privatte domain adaptation. Requiring no additional model parameters and having moderate increase in computation over the source model alone, the proposed method achieves competitive performance with state-of-the-art methods.
Introduction
In many real-world applications, such as healthcare and autonomous driving, data labeling can be expensive and time-consuming. make predictions on a new unlaheled dataset, one may naively use existing supervised model trained on a large labeled dataset. However, in the data-collection conditions, such as lighting or background for natural images, can cause a model's 'ss i performance degrade drastically [1]. This shift in the input data distribution is referred to in the literature as covariate shift [2]. By leveraging the labeled samples from the source domain and unlabeled samples from the target domain, unsupervised domain adaptation aims to overcome this issue, making the learned model generalize well the target domain B
subtle changes
Ben-David et al. 4 provide an H--divergence based theoretical upper bound on the target error. Ganin 6 popularizes learning an invariant representation between the source and target domains to minimize this divergence. Numerous prior methods 12-22) follow this trend, focusing using the source and target samples for feature alignment in the latent space, While this approach can reduce the discrepancy between domains, directly using the source and target samples for feature alignment has the following problems. First, several commonly used methods that can used to quantify the difference between two empirical distributions, such maximum mean discrepancy (MMD) 13] and Wasserstein distance [4,, [3). are sensitive to outlier samples
mini-batch when used to match the source and target marginal distributions [[]] We attribute this problem to the sampling variability both the source and target samples. Second, while we typically assume that the two domains share the same label space, cannot guarantee that the samples drawn from the source and target domains will cover the same set of classes in each mini-batch. Especially, the label proportions shift between domains, learning domain invariant representation might not lead to improvements over using the source data alone to train the model [8]. If we pull the support of
Equal contribution. Corresponding PyTorch code is available at https: ://g:thb/bbit www/korwwatt htw.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Source Data
Source Features
Feature Encoder
Cross Entropy
Amazon
(Xs Yg)
P1
Shared
Weights
Target Data
Class Prototypes
Feature Encoder
Transport
Loss
DSUR
Target Features
X,
Figure 1: This figure exhibits diagram Prototype-oriee Conditional Transport (PCT), Unlike existing methods align the target and source features, our method aligns the target features with class prototypes. The gray arrow indicates that the gradients of the prototypes do not back-propagate through the transport loss.
the source and target feature representations from different classes closer together, the classifier will be more likely to misclassify those examples. Finally, aligning the target features to source features means that we need access to both the source and target data simultaneously. In applications such as personal healthcare, we may not have access to the source data directly during the adaptation stage; instead, we may only be given access to the target data and the model trained on the source data.
We propose algorithm that constructs class prototypes represent the source domain samples in the latent space. Using the prototypes instead of source features avoids the previously mentioned problems: sampling variability in the source domain, 2) instance glass-mismatchin a mini-batch, and 3) source-data privacy concerns. As we expect the classifier to make better predictions on the target data in regions where the source density is sufficiently high [9), is natural to consider encouraging the feature encoder to map the target data close to these prototypes. Motivated by the cluster assumption (20) (decision boundaries should not cross high-density regions of the data), we provide method to transport the target features to these class prototypes and vice versa. We further extend our bi-directional transport address the potential shift label proportions, a common problem that has been studied 21:25 but that has been often overlooked in prior works [6] 7 26
Compared existing methods, the proposed one has several appealing aspects. First, does rely on adversarial training to achieve competitive performance, making the algorithm robust and converge much faster. Moreover, learnable prototypes bypass the need to directly access the source data. This attribute makes our algorithm applicable to the settings where preserving the source data privacy is major concern. Unlike clustering based approaches that typically require multiple forward passes before an update, our algorithm processes data in mini-batches for each update and trained an end-to-end manner.
only avoid expensive computation but also
We highlight the main contributions of the paper as follows: 1) We utilize the linear classifier's 's weights class prototypes and propose a general probabilistic framework to align the target features to these prototypes. 2) We introduce the minimization of the expected cost of a probabilistic bi-directional transport for feature alignment, and illustrate its superior performance over related methods. 3) We test the proposed method under multiple challenging yet practical settings: single-source, multi- source, class-imbalance, and source-private domain adaptation. In comparison state-of-the-art method achieves highly competitive performance in
algorithms, the proposed prototype-oriente domain adaptation, while requiring no additional model parameters and only having moderate increase in computation over the source model alone.
Prototype-oriented conditional transport
In this section, propose Prototype-oriented Conditional Transport (PCT), a holistic method for domain adaption consisting of three parts: learning class prototypes, aligning the target features with learned prototypes using probabilistic bi-directional transport framework of Zheng and Zhou

---

No Adaptation
DANN
PCT
Visualization of different methods on a synthetic dataset, where darker points marked with "." and lighter points marked with "x" denote the source and target samples, respectively, and the red and green colors denote two different classes. For each method, the left plot shows the data space whereas the right plot exhibits the output of the feature encoder in the latent space. Letters and B correspond the two class prototypes in the latent space. When there is clear class imbalance, DANN, a representative algorithm whose strategy is to match the marginal feature distributions between the source and target, fails to adapt to the target domain.
Figure 2:
277. and estimating the target class proportions. Our method does not introduce additional model parameters for aligning domain features, and the model can be learned in an end-to-end manner. We provide a motivating example of the application of our method on a synthetic dataset in Figure 2
In domain adaptation, we are given labeled dataset from the source domain, { [(=],,(() vf) D,,,
and an unlabeled dataset from the target domain, { ~ DE. We focus on the closed-category domain adaptation and assume that the source and target domains share the same label space, i.c., y² E 21.2..... K}. where K denotes the number of classes. The goal of domain adaptation is to learn model with low risk on the target samples. The model typically consists of a feature encoder, Fo :p:40 x Rdd, parameterized by 0, and a linear classification layer Cp Rd + R parameterized by μ.. In prior works [6] 26), the feature encoder is pre- trained neural network, and the classifier a randomly initialized linear layer. To simplify the following notation, we denote f² = Fe (z:) and fj Fo (1x)) as the feature representations of the source data x; and target data r.j. respectively.
2.1 Learning class prototypes
Most existing works 6 7 26 focus on aligning the source and target features in a latent space. By contrast, we propose to characterize the features of each class with class prototype and align the target features with these class prototypes instead of the source features. This approach has several advantages. First, the feature alignment between two domains would be more robust to outliers in the source domain as we avoid using the source samples directly. Second, we do not need to worry about the missing classes in the sampled mini-batch in the source domain like we do when we align the features of source and target samples. Prototypes ensure that every class is represented for each training update. Last but not least, using the inferred prototypes instead of source features allows adapting to the target domain even without accessing the source data during the adaptation stage, which is an appealing trait when preserving the source data privacy is a concern (see Table (
Previous methods [18 12, 32) construct each class prototype as the average latent feature for that class extracted by the feature encoder, which computationally expensive due to the forward passing of a large number of training samples. We propose to construct class prototypes in the same latent space but with learnable parameters: [M,, Mi......... 1 E R where the dimension of each prototype, d,, is the same as the hidden dimension after the feature encoder Fg. This strategy has been successfully applied by Saito et al. 33 in semi-supervised learning setting. We learn each class prototype in a way that encourages the prototype to be close to the source samples associated with that class in the feature space. In particular, given the prototypes and source samples a with features j and labels yf,, we use the cross-entropy loss to learn the prototypes:
Lela = E EK
explug S +bb)
=== - log [x (x/-13) Pix := Σ (i+bb)
(1)
where by is a bias term and pi the predictive probability for क to be classified to class k. We note that this way of learning the class prototypes is closely connected to learning the standard linear classification layer C₁. on source-only data with the cross-entropy loss. The neural network weights in the classification layer can be interpreted as the class prototypes. Therefore, compared with source- only approaches, constructing prototypes in this way introduce no additional parameters. As we show in Figure 4a our method requires much fewer parameters than other domain -adatatnn methods. Moreover, it requires much less computation than other prototype-based methods by avoiding the need to average the latent features for each class.
3

---

2.2 Bi-directional prototype-oriented conditional transport
In this section, we will discuss how we encourage the feature encoder to align the target data with class prototypes. Our approach is motivated by the cluster assumption, which has been widely adopted
in both semi-supervised learning [20] 34
351 and domain -ddattatio literature [36 37 31 38)
The cluster assumption states that the input data distribution consists of separated clusters and that instances belonging to the same cluster tend to have the same class labels. This means that the decision boundaries should not cross data high-density regions. To achieve this goal, we minimize the expected pairwise cost between the target features and prototypes with respect two differently constructed joint distributions. By minimizing the expected costs under these two different joint distributions, the target feature will be close to the prototypes, far from the decision boundaries.
2.2.1 Moving from target domain to class prototypes
To define the expected cost of moving from the target domain to the class prototypes, we first use the chain rule factorize the joint distribution of the class prototype 24g and target feature f as (T f)),, where drawing from the target feature distribution p(f))) can be realized by selecting random target sample z Dj to obtain fJJ Fo(z)) The conditional distribution, representing the probability of moving from target feature f to class prototype Mg. is defined as
p(N)) expllt f.)
Te (et (f)
A E K].
(2)
where, through the lens of Bayes' rule, p(2) is the discrete prior distribution over the K classes for the target domain, and exp(u] fý) plays the role of an unnormalized likelihood term, measuring the similarity between class prototypes and target features. Intuitively, the target features are more likely to be moved to the prototypes which correspond to dominant classes in the target domain or which are closer to the target features (or both). Note that in practice we often do not have access to the target class distribution p- We can a uniform prior distribution for p(ug). However, this could be sub-optimal, especially when classes are seriously imbalanced in the target domain. To address this issue, we propose way estimate (p(m)) K in Section 2.3
We now define the expected cost of moving the target features to class prototypes as:
Crmp Dr E m0 lsp [e(mg }3]] = En
E Et c(
(3)
EB PlM expla
where c(s,) ),, point-to-point moving cost, is defined with the cosine dissimilarity as
c(M f = pe
(4)
We also consider other point-to-point costs and present the results in Section 4.2 straightforward to obtain an unbiased estimation of Lz--, with a mini-batch from target domain Di.
With Eq. B it is
In this target-to-prototppe direction, we are assigning each target sample to the prototypes according to their similarities and the class distribution. Intuitively, minimizing this expected moving cost encourages each target feature to get closer to neighboring class prototypes, reducing the violation of the cluster assumption. If we think of each prototype the mode of the distribution of source features for a class, this loss encourages a mode-seeking behavior (27). Still, this loss alone might lead to sub-optimal alignment. The feature encoder can map most of the target data only a few prototypes. We connect this loss to entropy minimization to elucidate this point.
Connection with entropy minimization. The expected cost of moving from the target domain to prototypes can be viewed as a generalization of entropy minimization (20), an effective regularization in many prior
17]. If the point-to-point moving cost is explpt fi) and the conditional probability is works (39
33 40
defined as c(f2:: f/)
logg's
loog Σ (xp(
exp(n f3) (with uniform prior), then the expected moving cost (H 1ff) = p/y ER explun
becomes: Le-t = - -D [>* p/s log pog which is equivalent to minimizing the entropy on the target samples. Entropy minimization alone also has a mode-seeking behavior and has the same tendency to drop some modes (class prototypes). other words, one trivial solution to assign the same one-hot encoding to all the target samples 42

---

2.2.2 Moving from class prototypes to target domain
To ensure that each prototype has some target features located close by and avoid dropping class prototypes, we propose to add cost of the opposite direction [27], i.c., moving from the prototypes to target features. Given a mini-batch, { }= of target samples of size M, denoting p(f')
Σn J as the empirical distribution of the target features in this mini-batch, the probabilities of moving from prototype 2& to the M target features is defined as a conditional distribution:
((S)) exp( 1 To(f; (4)) Σy P(S explp (.) Σy= esplnE f} E (f f'):
expl J
(5)
As opposed the probabilities of moving target feature to different class prototypes no (14 fJ), (fū m₂) normalizes the probabilities across the M target samples for each prototype, which ensures that each prototype will be assigned to some target features. Then, the expected cost of moving along this prototype-to-targee direction is defined as:
C E B[a 10 NDF 16 p pplp )
(mx) [c( fj]
(m in
Σ p(m)) ( dH fJ) Ey ppplu
exp
(6)
which can be estimated by drawing a mini-batch of M target samples.
Finally, combining the classification loss in Eq. ( turget-to-proootppe moving cost in Eq. (3) and prototype-te tocirppeeotttee tottret target moving cost in Eq. (6) our loss is expressed as
Cela + Cr- + Cp-
Note that we treat z as fixed in both Li-, and Ly-st. This strategy allows us to apply our method in the source-data-private setting where we only have access to the source model. We also find empirically that this leads to more stable training.
2.3 Learning class proportions in the target domain
We propose to infer the class proportions {p( in the target domain by maximizing the log- likelihood of the unlabeled target data while fixing the class prototypes μ.. Directly optimizing the marginal likelihood intractable, we use the EM algorithm [43 45] derive the following iterative
updates (see the derivation in Appendix B). We first initialize with uniform prior: p(Mx)* K.
and obtain new estimates at each update step (starting from 0):
plaa) exple { p(p)) |+1 1 M j=1 To (142 fJ), where T/p (F)) E. p(My) exp(
(8)
Intuitively, the average predicted probabilities over the target examples for each class are used to estimate the target proportions, with ((1))(+1 shown above providing an estimate based on single mini-batch of M target samples. To estimate it based on the full dataset, we iteratively update it with p(u))*+ + (1 (')(()) +
where we follow the decaying schedule of the learning rate of the other parameters set f8l (1 71)-a in which y 0.0002, 0.75. The inintial value 30 is a pprramerrre that can be set as either ( to indicate a uniform prior, or a small value, such 0.001, to allow the class proportions be inferred from the data.
3 Related work
Feature distribution alignment. Most works on domain adaptation with deep learning focus on feature alignment to align either the marginal distributions (46) 6 8
47 4S or the joint distributions 222 of the deep features of neural networks. Early works use adversarial-be -based objectives, which are equivalent minimizing the Jensen-Shannow Shannon (JS) divergence [4). When the source and target domains have non-overlapping supports, the IS divergence fails to supply useful gradients [10] To remedy this issue, researchers propose using the Wasserstein distance, which arises from the optimal transport problem [52]. Courty et al. [9 develop Joint Distribution Optimal Transport (JDOT), defining the transport cost to the joint cost in the data and prediction spaces. Several works (10 12 extend this framework for deep neural networks. However, solving for the optimal couplings without
5

---

any relaxation is a linear programming problem, which has complexity of O(M log M) [33] Computing the transport probabilities in our algorithm has complexity of (d, which is the same complexity as computing the predictive probabilities that need to be computed by most algorithms. In this category, all the works focus on aligning the target features with source features, whereas we align the target features with prototypes.
Prototype-basse based alignment. We make two distinctions to existing works in this area. First, prior works for classification [2 and segmentation [54 55 utilize prototypes for pseudo-label assignments. By contrast, we use prototypes to behave as representative samples of the source features to define the loss. Second, all of these works use some form of average latent features to construct class prototypes, which computationally expensive. We instead adopt a parametric approach to learn the prototypes, avoiding that costly computation.
Learning under shifted class proportions. Although a shift in class proportions between two domains is a common problem in many applications, it is still largely under-explored. Over the years, researchers have viewed the question through different lenses: applying kernel distribution embedding 55 21), using an EM update [43 (58), placing a meta-prior over the target proportions [59], and casting the problem under causal and anti-causal learning (21:25) Recently, Tachet des Combes et al. [8] propose aligning the target feature distribution with a re-weighted version of the source feature distribution. While this method achieves consistent improvements over atature-aligmment algorithms, it still relies on learning approach suffers from the problems of sampling variability and maassmmismtthiing whereas the proposed method uses class prototypes and proportion estimation to avoid these issues.
representations. As we have discussed, this
mini-batch,
Source-private adaptation. Finally, the proposed method can also be applied to source- source-private private setting where without seeing the raw source data, we only have access to the source model and target data while adapting to the target domain [3] 47 63 Liang al. [31 introduce clustering-basee approach to generate pseudo-labels for the target data. That approach requires constructing class centers using& weighted average of the latent features. Different from that work, our prototypes
behave as class centers and are more amenable to mini-batch stochastic gradient based training.
4 Experiments
In this section, we evaluate our method under four practical settings: single-source, multi-source, class-imbalance, and source-private domain adaptation. We present the setup, the results, and the analysis of the results in the upcoming sections.
Datasets. We use the following three datasets of varying sizes in our experiment: 1) Office-31 3
consists of 4652 images coming from three domains: Amazon (A), Webcam (W), and DSLR (D). The total number of categories is 31. 2) Office-Home [64), a more challenging dataset than Office-31,, consists of 15,500 images over 65 classes and four domains: Artistic images (Ar), Clip art (CI), Product images (Pr), and world (Rw). 3) DomainNer [65] is large-scale dataset for domain adaptation. It consists of about 569.010 images with 345 categories from six domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. We further perform experiments on Cross-Digits, ImageClef, Office-Caltech, and VisDA datasets and provide the details Appendix E.2
Implementation details, We implement our method on top of the soen-source transfer learning library (MIT license) (66), adopting the default neural network architectures for both the feature encoder and linear classifier. For the feature encoder network, we utilize a pre-trained ResNet-50 in all experiments except for multi-source domain adaptation, where we use pre-trained 101. We fine-tune the feature encoder and train the linear layers from random initialization. The linear layers have the learning rate of 0.01, 10 times that of the feature encoder. The learning rate follows the following schedule as 7her = 7o + yiter) -a where Too is the initial learning rate. We set no to 0.01, 7 0.0002, and a to 0. 7.75. We utilize a mini-batch SGD with a momentum of 0.9. We set the batch size for the source data as N = 32 and that for the target data as M = 96. We use all the labeled source samples and unlabeled target samples 26). We set By to 0 (a uniform prior) in all settings except for the sub-sampled target datasets. We perform sensitivity analysis (see Appendix E)) and set 30 empirically to 0.001 for the sub-sampled target version of Office-31 1 and 0.0001 for that of Office-Homee We report the average accuracy from three independent runs. All experiments are conducted using a single Nvidia Tesla V100 GPU except for the DomainNet experiment, where we use four V100 GPUs More implementation details can be found in Appendix D

---

4.1 Main results
Single-source setting, We perform single-soume domain adaptation on the Office-31 and Office- Home datasets. In each experiment, one domain serves as the source domain and another as the target domain. We consider all permutations, leading to 6 tasks for the Office-31 dataset and 12 tasks for the Office-Home dataset. We compare our algorithm with state-of-the-art algorithms for domain adaptations from three different categories: adversarial-based, divergence-based, and optimal transport-based. We report the results Office-31 in Table [ PCT significantly outperforms the baselines, especially on the more difficult transfer tasks (D~> and W - A). Although MDD [67]. the best baseline domain-adaptation method, uses a bigger classifier, PCT still has 1.1% higher average accuracy. Figure 4a we visualize the number of parameters versus the average accuracy on the Office-31 dataset. While PCT uses fewer parameters than most methods, it still achieves the highest average accuracy. We report the average accuracies on the Office-Horie
dataset in Table 2 PCT outperforms baseline methods 10 of the 12 transfer tasks, yielding 3. 7% improvement the average accuracy over MDD. The results in this setting demonstrate that aligning the target features with prototypes is more effective than directly aligning them with the source features.
Table 1; Accuracy (%) on Office-31 for unsupervised domain adaptation (ResNet-50).
Category
Method
A W
D- W
W +D D
A D
D - A
W A
Avg
ResNet-50 68
68.4 0.2
96.7 + 0.1
99.3 + 0.1
68.9 + 0.2
62.5 + 0.3
60.7 + 0.3
76.1
Adversarial
DANN 6
0.4
96.9 0.2
99.1 0.1
68.2 0.4
6.5
82.2
ADDA
86.2 + 0.5
96.2 + 0.3
98.4 0.3
77.8 + 0.3
+.4
68.9
82.9
CDAN
91.1 0.1
98.6
100.0
92.9
0.3
69.3 0.3
87.7
MDD
67
94.5 0.3
98.4 0.1
100.0 100.0=00 0.0
93.5
74.6 0.3
72.2 0.1
88.9
Divergence
JAN
7
85.4
0.3
97.4 0.2
99.8 0.2
84.7 0.3
68.6 0.3
70.0 0.4
84.3
TPN
29
91.2 + 0.3
97.7 + 0.2
99.5 0.1
89.9 + 0.2
70.5 0.2
73.5 + 0.1
87.1
OT
DeT
10
88.9 + 0.3
98.5 + 0.1
99.6 0.2
88.2 + 0.1
72.1 1.11+00
70.1 + 4
86.2
ETD 69
92.1
100.0
100.0
88.0
71.0
67.8
86.2
PCT (Ours)
94.6 + 0.5
98.7 + 0.4
99.9 0.1 93.8 1.8
77.2 + 0.5 76.0 + 0.9
90.0
Table 2: Accuracy (%) on Office- for unsupervised domain adaptation (ResNet-50).
Method
Ar- Cl Ar Pt Ar + Rw
CI
Ar
CI
- Pr
CI
- Rw
Pr
- Ar Pt - CI
Pt
-
Rw Rw + Ar Rw
CI Rw
Pr
Avg
ResNet-50
68
34.9
50.0
58.0
37.4
41.9
46.2
38.5
31.2
60.4
41.2
59.9
46.1
DANN
45.6
59.3
70.1
47.0
58.5
60.9
46.1
43.7
68.5
63.2
51.8
76.8
$7.6
CDAN
20
50.7
70.6
76.0
57.6
70.0
70.0
57.4
50,9
77.3
70.9
56.7
81.6
65.8
MDD
54.9
73.7
53.6
78.1
60.2
82.3
JAN
45.9
61.2
68.9
50.4
39.7
61.0
45.8
43.4
70,3
63.9
52.4
76.8
58.3
TPN 29
51.2
71.2
76.0
65.1
72.9
72.8
55.4
48,9
76.5
70.9
53.4
80.4
66.2
DeE
[0
48.2
69.2
74.5
69.1
71.1
56.3
46.0
76.5
68.0
52.7
80.9
64.1
ETD [69]
51.3
71.9
85.7
57.6
69.2
73.7
57.8
51.2
79.3
70.2
57.5
82.1
67.3
(Ous)
57.1
78.3
41.
81.4
67.6
#0.8
77.0
21.2
76.5
=0.5
55.0
81.3 50.2
#0.3
#0.5
85.3 71.8
Multi-source setting. In this setting, we evaluate our method using Office-Home and DomainNet datasets [65]. For each task, we select one domain as the target domain and use the remaining five domains the source. We use the same data splitting scheme the original paper (65]. We compare against source-combingt and multi-source algorithms introduced in Venkat et al. (70] for Office-Home and in Peng et al. [65] for DomainNet. Multi-source algorithms use domain labels and a classifier for each source domain, whereas source-combined algorithms combine all the source domains into single source domain and perform single-source adaptation. We adopt single classifier and do not use domain labels. Thus, PCT falls under the source-combined category. We and 4 While PCT is not designed specifically for multi-source domain
report the results in Tables 3 adaptation, our method still outperforms those multi-source algorithms in both datasets. Since there are more variations of the data in the source domain in this setting, the increase in performance gain verifies our intuition that prototypes help mitigate the problem of sampling variability.
Table 3: Accuracy (%) on Office-Home for ResNet50- based MSDA methods.
Category
Models
R Ar
R CI
Pt
R Rw
Avg
Source-
DAN S
79.0
82.5
72.4
combined
D.CORAL f
58.1
58.6
79.5
82.7
72.2
RevGra M
68.4
59.1
79,5
82.7
72.4
81.8
SImpAl
70
70.8
56.3
80.2
81.5
72.2
PCT (O (Quns)
76.3 + 0.5
64.1 + 0.4
84.9 + 0.8
84.3 + 0.5
77.4
7

---

Table 4: Accuracy (%c
on DomainNet for ResNet101-baase MSDA
methods.
Category
Models
R
Clipart
Infogruph
R
Painting
Quickdraw
Sketch
Ang
DCTIN 73
48.6 0.7
23.5
0.6
48.8 0.6
0:2
33.500..
47.3 0.5
88. 0.6
Mulri-sourae
SDA
57.2
+-4+-1 1.2
51.6+0.4
6.2++0.3
61.6+0.9
49.6 + 0.6
41.5
M SDA-
65
58.6 0.5
26.0 0.9
52.3 0.6
6.3 6.3
62.7 = 0.5
49.5 4 0.8
42.8 0.8
ML-MSDA
61.4 0.8
26.2 4
19.1 0.3
57.0 1.0
50.3 0.7
44.3 4: 0.8
ResNet-
241
0.5
38.1
0.4
33.7
0.5
DAN 8
15.4 0.5
12.9099
1.
48.6 0.7
310 0.5
32.1 0.6
RTN 75
44.2
12.6.:07
35.3
14.
48.4 0.7
31.1 0.7
Source-
JAN
01.1+08
35.
12.1+0077
32.3+ 0.6
combined
DANN
45.5 0.6
13.1
37.0=0.7
1322088
0.7
318
32.6 0.7
ADDDA +7
0.8
114
36.7 0.5
1..7
49.1 08
33.5
32.22 0.6
SE 38
2.7
0.9
12
T.1
02.
9.1 0.5
16.1 0.4
54.3
45.7
58.4
40.5 0.6
38.5
PCT (Ours)
67.2
26.1
$5.0 1.00000 0.2
16.2
67.1 0.2
53.7
47.6 0.1
Sub-sampled setting, In many cases, the label proportions could significantly change from one dataset another, resulting decrease in a model's performance. To test our algorithm under this setting, we follow the experimental protocol in Tachet des Combes et al. (18). We keep only thirty percent of the first LK/2 classes simulate class imbalance. We directly take their results for the sub- sub-sampled source data and perform additional experiments using the same sub- sampling scheme on the target data. The baselines in this setting are standard domain adaptation methods (DAN, JAN, and CDAN) and their importance weighted versions introduced in Tachet des Combes et al. [8]. We present the results Table 5 On the sub-sampled source data, PCT with uniform prior outperforms the becond-best method (ICDDAN) by 4% on Office-31 and 6.6% on Office- Home. Learning the prior distribution on the target domain does not improve the result, as this setting does not have a serious imbalance issue in the target domain. On the sub-sampled target data, PCT with uniform prior already outperforms the baselines, 1.9% and 5.2% higher average accuracy than IWCDAN's on Office-31 and Coffic-HHoee respectively. Using a learnable prior further improves upon using uniform prior by 1.0% on Office-31 and by 0.4% on Office-Home. The improvements confirm our intuition that prototypes help with the class imbalance in both the source and target domain while estimating the target proportion further boosts the performance in the target domain sub-sampled setting. We visualize the estimated proportions on the target data in Figure verifying that the proportions are inferred correctly.
Table Average accuracy (%) on sub-sampled version Office-31 and Office-Home (ResNet-50).
Method
sub-S O-31
sub-T O-31
sub-S O-H
sub-T O-H
ResNet-50 50
68
75.7
76.1
51.4
58.2
DANN 3
76.2
75.9
51.8
58.3
JAN 7
78.2
78.1
53.9
61.4
CDAN 26
81.6
83.0
56.3
63.1
IWDAN 18
82.6
79.2
57.6
58.6
IWJAN 18
82.6
82.8
55,9
62.0
IWCDAN 18
83.9
83.5
61.2
64.6
(Ours)
87.9 = 0.4
85.4
67.8 + 0.3
69.8
#0.2
Leammaleee (Ours)
87.9 0.4
86.4 1 0.2
67.8 1 0.3
70.2 1 0.2
dource-data-private setting. In many practical applications, practitioners might not directly have access to the source data in the adaptation stage. Instead, trained model on the source data is provided. In this setting, the goal to adapt to the target domain while only operating on the given model. We compare our method with Source Hypothesis Transfer (SHOT) [31] a state-of- the-art method proposed specifically for this setting. SHOT contains two losses: information maximization (IM) loss and pseudo-labeling loss. We follow their experimental protocol by first training the model on the source data alone. During the adaptation stage, we only use the target data to perform model adaptation. We use the transport losses to update the feature encoder while fixing the prototypes. We report the results in Table From the standard setting where we have access to source data in Table [] the average accuracy drops by 1.6% for Office-31 and by 0.8% for Office-Home On the Office-3I dataset, our bi-directional loss outperforms the IM loss by 1.1% and the pseudo label loss by 0.8%. While the average accuracy for our method is 0.2% lower than both of their losses combined, the p-value for the independent two-sample /-test on the accuracies of different runs is 0.32, which is not statistically significant. On the Office-Homm dataset, our approach performs 1.9% and 0.5% better than the pseudo label and IM losses, respectively. While their combined loss achieves 0.8% accuracy higher than that of our method, the pscudo-labeling loss in SHOT requires

---

Uniform Propertion
0.08
True Pruportion
Eshmated
Class Proportions
0.07
Leabbiee Pnoon 0.16
0.05
0.04
0.03
0.02
0.01
0.00
Back Pack
Elke
Headphones
Koyboard
Laptop Computer
Letter Tray
Mobile Phone
Monitor
Mouse
Mug
Paper Noteboook
Pen
Bliee Helmmt
Phome
Printer
Projector:
Punchers
Ruler
Scissors
Speaker
Stapler
Tape Dispenser
Bookcase
Trash Can
Bottle
Calculator
Desk Chair
Desk Lamp
Desktop computer
Fille Cabinet
Ring
Class labels
Figure 3: Visualization of the estimated target proportions versus true class proportions for the task A - sD on the Office-31 dataset. The dotted line represents uniform proportion. It is clear that each orange bar (the learned proportions) close its adjacent blue bar (the true proportion). To quantify this observation, we measure the L1 loss between the true and learned proportions. The estimated proportions achieve lower 1.1 loss than the uniform distribution (0.16 vs 0.58), illustrating the effectiveness of our estimation strategy.
Table 6: Average Accuracy (%) on the private Office-31 and Office-Home (ResNet-50).
Source Model Only
SHOT-Pseudo Label [T
SHOT-IM (3]
SHOT [T
PCT (Ours)
Office-31 Hofece-Home 60.2
79.3
87.6 +0.5
87.3 +0.5
88.6 + 0.4
88.4 +0.6
69.1 10.6
70.5
71.8 +: 0.4
71.0 10.6
constructing class centers, which does not scale well with large datasets. Our approach uses the classifier's 's weights as class prototypes to avoid this issue.
4.2 Analysis of results
Ablation study. To examine the effect of each component our framework, we perform ablation studies and present the results in Table 1) Aligument strategy. Next, present the result using optimal transport as the aligmment strategy. We consider two variants of Prototype-oriented Optimal Transport (POT): exact linear program (POT) and Sinkhorn relaxation (POT- -rOT-Sinkhom). In each variant, we solve for the optimal couplings using the optimal transport formulation. After obtaining update the feature encoder using the obtained probabilities as the
the transport probabilities, weights for the transport cost. We can see that POT performs 1.7% better than DeepJDOT Table [ showing the effectiveness of using prototypes define the transport costs with the target features. Still, both versions of POT underperforms PCT by 2.1% and 1.7%, respectively. 2) Effect of each loss in PCT. We examine the effect of each loss on the average test accuracy the Office-31 dataset. We remove each transport loss while keeping the cross-entropy loss. The bi-directional loss leads the best accuracy, while the drop in accuracy is more significant we remove Za- This result is not surprising because, without C,-+t. the model can map target data to only a few prototypes, leading to a degenerate solution. 3) Gradient stopping. We also show the algorithm's 's performance without stopping the gradient of ęs in the transport loss. PCT gains an additional 1.0% in accuracy with the gradient stopping strategy. The performance gain is consistent with the finding in recent work by Chen and He [77], where the authors apply the gradient stopping strategy to avoid degenerate solutions in contrastive learning. 4) Cost function. Finally, we explore the cost function inspired by the radial basis kernel. We can see that the cosine distance in PCT gives 3. 3.1% higher average accuracy. In short, the choice of the probabilistic bi-directional transport framework, gradient-stopping strategy, and point-to- toint cost function all contribute to the success of the proposed PCT method.

---

Table 7: Average aceuracy (%) of PCT on Office-31 under different variants (ResNet-50).
POT
POT--Sinkhom
PCT wio (Cc 1) PCT w/o (Cu xt)
w/n stop grap-grad o(pee f2) = exp(-pl
PCT (Ours)
87.9 + 0.8
88.3 0.9
88.6 + 0.2
84.3 + 0.9
89.0 = 0.3
86.9
90.0 + 0.5
MDD
PCT
PM
Source Only
Accurary
Test Error
60
JANN
20-
Source Only
1000
2000
3000
4000
3000
Number of Patarmeters (in Millions)
Ieeration Number
(a)
(b)
Figure 4: (a) Analysis of parameter efficiency, (b) comparison of convergence, and (c) t-SNE visualization of the output of the feature encoder trained with PCT on the task A -> W. In plot (c), prototypes (+), source features (+), and target features (x) are tightly clustered together for each class.
Convergence comparison. We plot test accuracy versus iteration number on the task (A - W) in Figure[bbj compare the convergence versus convergence time in minutes in Appendix E In both plots, PCT quickly converges within first one thousand iterations, and the test accuracy does not fluctuate much thereafter. This phenomenon is not surprising since we use prototypes instead of the source features to align with the target features. We expect the prototypes to behave as representative samples of the source features, making the model converge quickly and stably.
of different algorithms. We also visualize test accuracy
Visualization. We visualize in Figure 4c the t-SNE plot of the source and target features as well as the prototypes for the task A + W. Figure 4c shows that both the source (dots -) and target (crosses
x) features are close to the prototypes (black stars *), indicating that our algorithm learning meaningful prototypes and successfully align the target features with the prototypes.
Conclusion
We offer holistic framework for unsupervised domain adaptation through the lens of a probabilistic bi-directional transport between the target features and class prototypes. With extensive experiments under various application scenarios of unsupervised domain adaptation, show that the proposed prototype-orientee alignment method works well on multiple datasets, is robust against class imbal- ance, and can perform domain adaptation with direct access the source data. Without adding additional model parameters, our memory and computation-effiiien algorithm achieves competitive performance with state-of-the-an methods several widely used benchmarks.
Acknowledgments
We thank Camillia Smith Barnes and Georgii Riabov for helpful discussions and feedback on the paper. K. Tanwisuth, X. Fan, H. Zheng, S. Zhang, and M. Zhou acknowledge the support of Grant 181-19 from the U.S. National Science Foundation, the APX 2019 project sponsored by the Office of the Vice President for Research The University of Texas at Austin, the support of gift fund from ByteDance Inc., and the Texas Advanced Computing Center (TACC) for providing HPC resources that have contributed to the research results reported within this paper.
10

---

References
[1] Ali Farhadi and Mostafa Kamali Tabrizi. Learning to recognize activities from the wrong view point. In European conference computer vision, pages 154-166. Springer, 2008.
[2] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[3] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213-226 Springer, 2010.
[4] Shai Ben-David. John Blitzer, Koby Crammer, and Fermando Pereira. Analysis of representations for domain adaptation. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007.
[5] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. theory of learning from different domains. Machine Learning. 79:151-175, 2010.
[6] Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis R. Bach and David M. Blei, editors, ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 1180- 1889. JMLR.org, 2015.
[7] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2208-2217. PMLR, 06-11 Aug 2017.
[8] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 97-105, 2015.
[9] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamoniy. Joint distribution optimal transportation for domain adaptation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[10] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nico- las Courty. DeepJDOT: Deep joint distribution optimal transport for unsupervised domain adaptation. CORR, abs/1803. 10081, 2018
[11] Chen- Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced Wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10285-10295, 2019.
[12] Renjun Xu, Pelen Liu, Liyan Wang, Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[13] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[14] Leonid V Kantorovich. On the translocation of masses. Journal of Mathematical Sciences, 133 (4):1381-1382, 2006.
[15] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6):355-607, 2019.
[16) Matthieu Lerasle, Zoltán Szabó, Timothée Mathieu, and Guillaume Lecué. Monk outlier-robust mean embedding estimation by gecdian-of-meame In International Conference on Machine Learning, pages 3782-3793. PMLR, 2019.
[17] Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Robust optimal transport with applications in generative modeling and domain adaptation. Advances Neural Information Processing Systems, 2020.

---

[18] Remi Tachet des Combes, Han Zhao, Yi-Xia Wang, and Geoff Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 2020.
[19] Fredrik D. Johansson, David Sontag, and Rajesh Ranganath. Support and invertibiliity in domain- invariant representations. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pages 527-536, 16-18 Apr 2019.
[20] Yves Grandvalet, Yoshua Bengio, et al. Semi-supervised learning by entropy minimization, In CAP, pages 281-296, 2005.
[21] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pages 819-827, PMLR, 2013.
[22] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv: wtt.:/006.444 6471, 2012.
arXiv:1206.6471
[23] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International conference on machine learning, pages 2839-2848. PMLR, 2016.
[24] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-3130. PMLR, 2018.
[25] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint arxiv: 2019.
[26] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biande and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[27] Huangjie Zheng and Mingyuan Zhou. Comparing probability distributions with conditional transport. arXiv preprint arXiv:2012. 14100, 2020.
[28] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4080-4090, 2017.
Chong-Wah [29] Yingwei Pan, Ting Yao, Yehao Li, Yu Wang. Chong- Wah Ngo, and Tao Mei. Transferrable prototypical networks for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[30] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4893-4902, 2019.
[31] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. PMLR, 2020.
[32] Xiangyu Yue, Zangwei Zheng, Shanghang Zhang, Yang Gao, Trevor Darrell, Kurt Keutzer, and Alberto Sangiovanni Vincentelli. Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13834-13844, 2021.
[33] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 8050-8058, 2019.
12

---

[34] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training:
a regularization method for supervised and supervised learning. IEEE transactions on pattern analysis and machine infelligence, 11(8):1979-199)) 2018.
[35] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, arXiv preprint 161.026610 2016.
[36] Yuan Shi and Fei Sha. Information -tterrriicaa learning of discriminative clusters for unsuper- vised domain adaptation. arXiv preprint 12Xiv:/206.64 2012.
[37] Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv: /802 0873,, 2018.
[38] Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. arXiv preprint arXiv:/706.05208, 2017.
[39] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Ad- versarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2517-2526, 2019.
[40] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. arXiv preprint arXiv:2002.07955. 2020.
[41] Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. arXiv preprint arXiv: 1711, 10288, 2017.
[42] Xiaofu Wu, Quan Zhou, Zhen Yang. Chumming Zhao, Longin Jan Latecki, et al. Entropy mini- mization vs. diversity maximization for domain adaptation. arXiv preprint 200202000..01690, 2020.
[43] Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new priori probabilities: a simple procedure. Neural computation, 14(1):21-44 2002.
[44] Guoliang Kang, Liang Zheng, Yan Yan, and Yi Yang. Deep adversarial attention alignment for unsupervised domain adaptation: the benefit of target expectation maximization. In Proceedings of the European Conference on Computer Vision (ECCV), pages 401-416, 2018.
[45] Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias- corrected calibration hard-to-beat at label shift adaptation. In International Conference on Machine Learning, pages 232- PMLR, 2020.
[46] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint urXiv:1412.3474. 2014.
[47] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176. 2017.
[48] Shujian Zhang, Xinjie Fan, Huangjie Zheng, Korawat Tanwisuth, and Mingyuan Zhou. Align- ment attention by matching key and query distributions. In NeurIPS 2021: Neural Information Processing Systems, Dec. 2021. (the first two authors contributed equally).
[49] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 1rXiv:1406.2661, 2014.
[50] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pages 214-223. PMLR, 2017.
[51] Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Sliced Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.
[52] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science Business Media, 2008.
13

---

[53] Quentin Mérigot and Edouard Oudet. Discrete optimal transport: complexity, geometry and applications. Discrete & Computational Geometry, 55(2):263-255 2016.
[54] Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao. Category anchor- anchor-guided unsupervised domain adaptation for semantic segmentation. arXiv preprint arXiv: 1910. 13049, 2019.
[55] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. International conference on machine learning, pages 5423-5432. PMLR, 2018.
[56] Marthinus Christoffel Du Plessis and Masashi Sugiyama. Semi-supervised learning of class balance under class-prior change by distribution matching. Neural Networks, 50: 1 10-119, 2014.
[57] Tuan Duong Nguyen, Marthinus Christoffel, and Masashi Sugiyama. Continuous target shift adaptation in supervised learning. In Asian Conference on Machine Learning, pages 285-300. PMLR, 2016.
[58] Yee Seng Chan and Hwee Tou Ng. Word sense disambiguation with distribution estimation. In IJCAI, volume 5, pages 1010-5. Citeseer, 2005.
[59] Amos Storkey. When training and test sets are different: Characterizing learning transfer. Dataset shift in machine learning, 30:3-28, 2009.
[60] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9641-9650, 2020.
[61] Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 615-625, 2021.
[62] Jogendra Nath Kundu, Rahul Mysore Venkatesh, Naveen Venkat, Ambareesh Revanur, and R Venkatesh Babu. Class-incremental domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part XIII 16, pages 53-69. Springer, 2020.
[63] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, al. Universal source-free fruree-freetee domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4544-4553, 2020.
[64] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5027, 2017.
[65] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1406-1415, 2019.
[66] Mingsheng Long Junguang Jiang, Bo Fu. Transfer-leaming-libe htam://Teannffr-LLami---------
https: com/
2020.
[67] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pages 7404-74 -411.. PMLR, 2019.
[68] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[69] Mengxue Li, Yi-Ming Zhai, You-Wei Luo, Peng-Fei Ge, and Chuan-Xian Ren. Enhanced transport distance for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13936-13944, 2020.
14

---

[70] Naveen Venkat, Jogendra Nath Kundu, Durgesh Kumar Singh, Ambareesh Revanur, and Venkatesh Babu. Your classifier can secretly suffice multi-source domain adaptation. arXiv preprint arXiv:2103. 169, 2021.
[71] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 450. Springer, 2016.
[72] Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specine distribution and classifier for cross-domain classification from multiple sources. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5989-5996, 2019.
[73] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3964 3973, 2018.
[74] Zhenpeng Li, Zhen Zhao, Yuhong Guo, Haifeng Shen, and Jieping Ye. Mutual learning network for multi-source domain adaptation. arXiv preprint arXiv:2003. /2944, 2020.
[75] Mingsheng Long. Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. arXiv preprint (rXiv:1602.044333 2016.
[76] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. CoRR, abs/1712.022560, 2017.
[77] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[78] Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look at dataset bias. In Domain adaptation in computer vision applications, pages 37-55. Springer, 2017.
[79] Boqing Gong, Fei Sha, and Kristen Grauman. Overcoming dataset bias: An unsupervised domain adaptation approach. In NIPS Workshop on Large Scale Visual Recognition and Retrieval, volume 3. Citeseer, 2012.
[80] Paul S Bradley, Kristin P Bennett, and Ayhan Demiriz. Constrained K-means clustering. Microsoft Research, Redmond, 20(0):0, 2000.
[81] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXi::1911.05371, 2019.
15


Reviews:
[{"review_id": "zkDjXP3E4ck", "reviewer": "Reviewer_KNFV", "summary": "This paper proposes creating prototypes(p) for the source data and aligning these prototypes with the target(t) data via bidirectional conditional transport measure. They use last linear layer's weights as prototypes, so it saves cost, and their method do no require source data for adaptation. For the alignment, they define bidirectional transport, that has  prototypes to targets part and targets to prototypes part in the objective function. For the former, they add prior prototype distributions that comes from Bayes' rule and approximate via EM algorithm. The latter  corresponds to entropy minimization. They obtain state of art results in several domain adaptation scenarios.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n#### *Originality:*\n\n- Their way of using prototypes (using last layers' weights) and formulation of CE resembles supervised contrastive learning(SupCon)[1]. However, this is just one part of their model, and further formulating the alignment in a two way conditional distributional setting and estimating class priors together is novel as of my knowledge.  \n\n\n- Defining the pointwise cost between a prototype and target as doc product makes the whole t->p cost an entropy minimization term, this idea is simple, but the connection is a nice contribution.\n\n\n- The related work is almost sufficiently cited in my opinion, however a couple of citations might worth adding[2,3,4]\n\n\n#### *Quality:*\n\n- The claims are supported well in a couple of different ways: \n\n    - The visualizations over synthetic data (Figure 1, 2 and 3) are well designed and convincing. \n    - The experiments are extensive and apparently superior \n    - Table 6 is a good ablation study that decreases the doubt about the roles of each loss term or their direct alternatives.\n    - The connections to the clustering and optimality are simple but enough support.\n\n\n - Did you achieve the results on Figure 1b by training a simple case(like Figure 2) or is it just an abstract scenario in order to visualize your idea?\n\n\n- How would you extend your algorithm to the cases of unseen classes? Would it work by adding nodes and prototypical weights to the output layer?\n\n\n- The last layer is actually a direct representative of the source classes, is this allowed within the data-privacy practices?\n\n#### *Clarity:*\n\n- Overall, the paper is written clearly and the flow is good.\n\n\n- There are several improvements I can suggest, for example the authors mentioned advantages of their method(other than main contributions) throughout the paper(such as imbalance, missing class within a batch, parameter costs, outlier robustness etc.) these arguments could be given in a more compact way and in one place.\n\n#### *Significance:*\n\n- The authors achieves really good results on a variety of tasks (single source, multi source, easier and harder datasets), with an intuitive method, their implementation can draw interest. \n\n- Especially given that utilizing prototypes is attempted a fairly good amount of times in unsupervised domain adaptation, and mostly with an adversarial training support and the results are not as superior as the paper at all, again as of my knowledge. So I believe both their method and results will be useful for the field. \n\n \n\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention hardness of eliminating bias problem in domain adaptation problems and energy cost encountered in large deep learning methods.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "dNQLg2QOGM_", "reviewer": "Reviewer_Z3wr", "summary": "This paper presents a prototype-based framework for UDA. The core idea is to employ a probabilistic bi-directional transport framework that aligns target domain samples with the source domain prototypes, where the prototypes are modelled using the classifier weights. A key highlight of the method is that it is applicable in the absence of source domain data, which is highly useful in practice. Benchmarking on three datasets shows the method achieves excellent results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n========\n+ The paper is well organized and written. The approach is well motivated.\n+ The source-data-private setting is practical. I especially liked the discussion around the use of classifier weights as prototypes, as compared to taking feature mean.\n+ The results look promising. The analysis in Section 4 (and Appendix) is comprehensive. The visualizations in Appendix Section E are interesting.\n\nWeaknesses / Clarifications\n=====================\nI do not have significant criticism of the paper, however addressing the following could improve the submission:\n\n- Regarding fixed $\\mu$: Once the model is trained on the source domain, the prototypes are kept fixed. This would enforce the target domain to acquire the semantics learned by the source domain. I agree that this would lead to a more stable training, however, this restriction may not be suitable in category-shift scenarios (e.g., see [P2]). Could the authors comment on this aspect?\n\n- Comparison with prior works: There are several source-data free approaches in the literature [P1, P2] that have been missed in the related work. In particular, [P2] also uses a prototype based metadata to perform “source-free” adaptation, however recommends using separate latent spaces for source and target domain. IMO, these should be discussed in the related work since ideas such as source-data-private setting, and compact feature clusters are already described in these works.\n\n- L86: “missing classes”: It is unclear what is meant by the term “missing classes” here. Does it imply that the method is applicable under category-shift scenario as well (e.g. open-set domain adaptation).\n\n- Classical DA methods work under the theoretical framework of [5] where the adapted model’s generalization bound depends on three terms (source-domain error, divergence between the domains, a constant that depends on the chosen hypothesis space). Here however, in the source-private-data setting, the source domain error becomes immaterial since the feature extractor is updated solely based on the target domain. It would be good to see some theoretical discussion on why the proposed method works (since the adapted feature extractor would not be ideal for the source domain). An interesting analysis could be analyzing the performance of source domain data on the adapted model.\n\nReferences:\n\n[P1] Kundu et al., “Universal Source-Free Domain Adaptation”, CVPR 2020\n\n[P2] Kundu et al., “Class-Incremental Domain Adaptation”, ECCV 2020\n\nMinor Comments\n=============\n- Please fix typos (e.g. Fig. 2 caption, “imabalance”). Avoid single worded lines (e.g. Table 5 caption has the word Resnet50 consuming an entire line).\n- Fig. 1b is not very informative/novel. This could be removed and the space obtained could be used to plot visualizations shown in Appendix (esp. Class proportion estimation).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "9lgfyuA_1rq", "reviewer": "Reviewer_ZA7J", "summary": "In this paper, a UDA algorithm is proposed. The core idea is based on benefiting from the class prototypes in the source domain to align two domains in a shared embedding space, rather than using the source sample features. The class prototypes are learned on the source domain using a cross-entropy loss to relax computational load. Upon learning the prototypes, a transport-based loss function is used to align the target domain features with the learned prototypes. Experiments on several UDA scenarios using standard benchmark datasets are provided to demonstrate that the method is effective.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. UDA is a highly explored area and new contributions should introduce novel ideas that can address existing challenges or boost state-of-the-art performances significantly. More specifically, the idea of using prototypes for UDA has been explored extensively in the literature. Transport-based optimization also has been used extensively for this purpose. Source-free UDA algorithms based on using prototypes have been developed as well, e.g.,:\n\n- Universal domain adaptation through self-supervision, NeurIPS 2020\n\n- Unsupervised model adaptation for continual semantic segmentation, AAAI 2021\n\nIt is true the proposed algorithm is different in the idea implementation details but the broad solution is not novel due to the precedent. Performance results also do not demonstrate a significant boost over prior works. Finally, there is no theoretical analysis to improve our understanding about using a similar approach to address UDA. Hence, I think this work is not suitable for publication at NeurIPS.\n\n2. Recent UDA algorithms perform more extensive experiments to demonstrate the generalizability of the method and offer benefits over prior works. In this aspect, I think the experiments need significant improvement:\n\n- Results on some important benchmarks, including, ImageCLEF, cross-domain digits, Office-Caltech, etc, are missing.\n\n- Multi-source and source-private experiments are even less extensive in terms of both used datasets and comparison with prior works.\n\n- Comparison has been done against mostly methods that have not been developed in the past three years. Please check the literature to include the existing state-of-the-art performance. \n\n- An ablative study on Eq (7) is helpful to determine the relative importance of the proposed terms. \n\n-In lines 49-55, it is claimed that the proposed approach is advantageous compared to adversarial learning because of robustness and faster convergence properties. Experiments are based on using test error versus learning iteration to demonstrate this claim. But I think this is not fully convincing. Comparisons should be performed based on time because the algorithms are different in terms of computations at each iteration. In other words, the execution time per iteration may be very different among these methods. A second set of experiments to demonstrate robustness is going to be very helpful.\n\n\n-----------------------------------\nAfter more elaboration, I finalized my updated rating and decided to maintain it. I think this work is best suited for future venues after incorporating all the review feedback.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have explored these aspects in a short \"broader impact\" section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-5OIfYgqeAy", "reviewer": "Reviewer_JCfs", "summary": "This paper deals with the problem of unsupervised domain adaptation (UDA) based on a prototype-oriented framework which is suggested to solve the fundamental issues on UDA (e.g., sampling variability in the source domain, instance class-mismatching in batch, source data privacy). The main contribution suggested in this paper is utilizing the linear classifier's weights as class prototypes with bi-directional transport loss for feature alignment between source and target domains. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\n As suggested in the main contribution, the author claims the originality of using learning parameters for class prototypes and bidirectional prototype-oriented transport loss.\n\nWhen it comes to constructing class prototypes with learnable parameters, I doubt the originality since the below paper also utilizes the learnable parameters to construct the class prototype for labeled data.\n\n- Semi-supervised Domain Adaptation via Minimax Entropy, Saito et al.\n\nIt would be more convincing if the author can prove the originality and show the main difference from the above paper.\n\nQuality:\n\nThe motivation of using learnable parameters for class prototype and technical approach using prototype-oriented transport loss seems to be reasonable.  It is supported with various unsupervised domain adaptation settings (one source, multi-source, source-data-private). Furthermore, explicit comparisons on the number of parameters and converging speed prove its effectiveness on practical domain adaptation. \n\nHowever, I have one concern related to the quality of this paper.\nComparison with proposed baselines seems to be insufficient. Specifically, the proposed method needs to be compared with other prototype-based alignment methods. \n\n- Contrastive Adaptation Network for Unsupervised Domain Adaptation, Kang et al.\n- Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation, Yue et al.\n\nClarity:\n\nIt is easy to follow and well-written. The paper contains well-supported theoretical analysis and experiments.\n\nSignificance:\n\nSuggesting the framework that requires no additional model parameters and converges fast seems to be applied in real-world domain adaptation problems.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am curious what kind of limitations that this paper could have when compared with other prototype-based alignment papers.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "More than one day"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your submission to NeurIPS.

This paper had somewhat mixed reviews, with 3 mostly positive and 1 more negative reviewer.  During the rebuttal phase, the negative reviewer maintained his/her position that the paper was not ready for publication, so we did not fully reach consensus on the paper.  However, I appreciated the responses that the authors provided to all the reviews, particularly to the more negative reviewer.  There are some valid concerns here---particularly with regard to the experimental comparisons---but overall I am positive about this paper and am happy to recommend it for acceptance at the conference.  Please try to incorporate as much of the rebuttal material as you can in the final paper, as it will strengthen the paper, as well as making sure to address as many of the reviewer criticisms as possible.
Decision: Accept (Poster)