A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Authors: Tiago Salvador
Kilian FATRAS
Ioannis Mitliagkas
Adam M Oberman
Venue: Accepted by TMLR ()

Abstract: Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In the case of an extreme label shift scenario between the source and target domains, where we have extra source classes not present in the target domain, the UDA problem becomes a harder problem called Partial Domain Adaptation (PDA). While different methods have been developed to solve the PDA problem, most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training. These strategies violate the main assumption in PDA: only unlabeled target domain samples are available. In addition, there are also experimental inconsistencies between developed methods - different architectures, hyper-parameter tuning, number of runs - yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods under different model selection strategies and a consistent evaluation protocol. We evaluate 6 state-of-the-art PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. Our two main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.

Published in Transactions on Machine Learning Research (05/2023)
A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Tiago Salvador!
tiago, hag..-/lldnnaasssuuuoooeomellliu
Mila Quebec AI Institute, McGill University
Kilian Fatras"
falian..atrrs@mmill...o quebec
Mila Quebec AI Institute, McGill University
Ioannis Mitliagkas
ioaniis Omnila quebec
Mila Quebec AI Institute, Université de Montréal, Canada CIFAR AT Chair
Adam Oberman
adam, www.mmobbrmonn@mil...ooom
Mila Quebec AI Institute, McGill University, Canada CIFAR AI Chair
Reviewed OpenReview: https:
https://openreeveau openreveeww net/ pet/formm? id= XcVeIBXeRt
Abstract
Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images lever- aging source labeled ones. In the case of an extreme label shift scenario between the source and target domains, where we have extra source classes not present in the target domain, the UDA problem becomes a harder problem called Partial Domain Adaptation (PDA). While different methods have been developed to solve the PDA problem, most successful algorithms model selection strategies that rely target labels to find the best and/or models along training. These strategies violate the main assumption in PDA: only unlabeled target domain samples are available. In addition, there are also experimental in- consistencies between developed methods different architectures, number of runs yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods under different model selection stralegies and con- sistent evaluation protocol. We evaluate state-of-the-art PDA algorithms on 2 different real-world datasets using different model selection strategies. Our main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.
prree-parrmeter tuning,
1 Introduction
Domain adaptation. Deep neural networks are highly successful in image recognition for in-distribution samples He et al. 2016) with this success being intrinsically tied to the large number of labeled training data. However, they tend to not generalize as well on images with different backgrounds or colors not seen during training. These differences constitute some examples of what known in the literature as covariate shift. Unfortunately, enriching the training set with new samples from different domains is challenging as labeling data is both an expensive and time-consuming task. Thus, rescarchers have focusod on unsupervisod domain adaptation (UDA) where we have access to unlabelled samples from different domain, known the target domain. The purpose of UDA is to classify these unlabeled samples by leveraging the knowledge given by the labeled samples from the source domain (Pan & Yang. 2010 Patel et al. 2015). In the standard
ihhsse authors contributed equally to this work

---

Published in Transactions on Machine Learning Research (05/2023)
UDA problem, the source and target domains are assumed to share the same classes. In this paper, we consider a more challenging variant of the problem called partial domain adaplalion (PDA): :: the classes in the target domain Yi form a subset of the classes in the source domain V, (Cao et al. 2018 i.e., 3/ C y The number of target classes is unknown as we do not have access to the labels. The extra source classes, not present the larget domain, make the PDA problem more difficult: simply aligning the source and target domains results in matching target samples to source samples whose classes are not present in the target domain.
Realistic evaluations. Most recent PDA methods report an increase of the target accuracy up to 15 percentage points on average when compared to the bascline approach that is trained only on source samples. While these successes constitute important breakthroughs in the DA research literature, target labels are used for model selection, violating the main UDA assumption. In their absence, the effectiveness of PDA methods remains unclear and model selection still constitutes an open problem as we show this work. Moreover, the hyper-parameter tuning is either unknown or lacks details and sometimes requires labelod target data, which makes it challenging to apply PDA methods to new datasets. Recent. work has highlighted the importance of model selection in the presence of domain shift. Gulrajani & Lopez- Paz (2021) showed that when evaluating domain generalization (DG) algorithms, whose goal is to generalize to a completely unseen domain, in a consistent and realistic setting no method outperforms the baseline ERM method by more than 1 percentage point. They argue that DG methods without a model selection strategyy remain incomplete and should therefore be specified as part of the method. A similar recommendation was done by Saito et al. (2021) for domain adaptation.
PDA methods have been designed using target labels at test time to select the best models. Related work (Saito et al., 2021 You et al. 2019: model selection strategies for domain adaptation claimed to select the best models without using target labels. However, a realistic empirical study of these strategies in PDA is still lacking, In this work, we conduct extensive experiments to study the impact of model selection strategies on the performance of partial domain adaptation methods. We evaluate 6 different PDA methods over 7 different model selection stralegies, 4 of which do not use larget labels, and 2 different datasets under the same experimental protocol for fair comparison. We list below our major findings:
For given method, the difference of accuracy attained by model selected without target labels and a model selected with target labels can reach 30 percentage points (See Table for a summary of results).
Only 1 pair of PDA methods and target label-free model selection strategies achieve comparable accuracies to when target labels are used, while still improving over a source only baseline.
Random seed plays an important role in the selection of hyper-parameters. Selected parameters are not stable across different seeds and the standard deviation between accuracies on the same task can be up to 8.4% even when relying on target labels for model selection.
Under a more realistic scenario where some target labels are available, 100 random samples is enough see only drop of 1 percentage point in accuracy (when compared to using all target samples). However, using only one labeled target sample per class leads to significant drop in performance.
Related work. Concurrent work Musgrave et al. 2022) also conducted a study of model selection methods for unsupervised domain adaptation. Similarly to our work, the authors study several model selection techniques and several domain adaptation methods, but their focus in the standard domain adaptation setting, while we focus on PDA. In Section 4 we discuss our slightly different methodologies and in Section we compare our respective findings. Other domain adaptation variants could have been considered like the open-set domain adaptation (Busto & Gall, 2017). However, these other variants require specific methods and it would have been computationally expensive to make an extensive study of each variant in a single work. We thus leave other domain adaptation variants for future work.
Outline. In Scction 2 we provide an overview and a mathematical description of the different model selection strategies considered in this work. Then in Section 3 we discuss the PDA methods that we consider in this work. In Section 4 we describe the training procedures, hyper-parameter tuning and evaluation protocols
2

---

Published in Transactions on Machine Learning Research (05/2023)
DATASET
Model Selection
ONLY
PADA
SAPN
BA3US
AR
JUMBOT
MPOT
Worst (w/o largel labels)
59.55 (-2.31)
52.72
(-11.00)
61.37 (-1.93)
62.25
(-13.73)
64.32 (-8.4)
61.28 (-15.87)
46.92 (-30.38)
OFFICE- HOME
Best (w/o target labels)
00.73 (-1.14)
63.08 (-0.61)
62.59 (-0.71)
75.37 (-0.61)
70.58 (-2.16)
71.61 (-2.51)
66.21 (-11.07)
Avg (w/o target labels)
60.22 + 0.43
59.5 +
62.02
69.9 5.1
67.7 = 2.8
67.8 5.8
59.7 + 7.6
ORACLE
61.87
63.72
63.30
75.98
72.73
77.15
77.31
Worst (w/o target labels)
55.02 (-4.16)
32
(-22.26)
42.83
(-19.81)
51.07
(-16.60)
55.69 (-18.15)
59 86 (-24.15)
61.62 (-25.33)
Best [w/o target labels) Aug (w/o target labels)
55.21 (-1.24)
56.83 (2.26)
58.62 (-102)
65.58 (-2.09)
67.20 (-6.65)
77.69 (-6.31)
78.10 (-8.54)
VISDA
$5.1 +
45.1 + 8.8
51.1 7.4
37.5 5.3
63.5 = 4.6
65.2 7.3
71.2 6.26
ORACLE
39.48
54.57
62.64
67.67
73.85
84.01
86.95
Task accuracy average computed over three different seeds (2020, 2021, 2022) on Partial OFFICE- HOME and Partial-VISDA. For each dataset and PDA method, we display the results of the worst and best performing model selection that do not use target labels as well as their average and the ORACLE model selection strategy. All results can be found in Table
Table 1:
used to evaluate all methods fairly. In Section 5 we discuss the results of the different benchmarked methods and the performance of the different model selection strategies. Finally Section G we give some rocommendations for future work in partial domain adaptation.
2 Model Selection Strategies: An Overview
In UDA, the goal is to classify unlabeled data from a target domain loveraging labeled data from a source domain (Pan & Yang, 2010). Formally, let D. (resp. Dj) be the labeled (resp. unlabeled) source (resp, target) dataset which is composed of n, (resp. n4) i.d random labeled (resp. unlabeled) vectors in Rd drawn from distribution p (resp. q), i.e., D. = vf) X E E (resp. D. = {[x]]]=_ E Bd). The goal is to find fa. typically a deep neural network, that minimizes the target risk (f( = P ((xx))<l & y]- In the standard domain adaptation, both domains share the same label space y. Vd, while in PDA the target classes form a subset of the source classes V C Va. To classify the samples from the two domains, we rely a feature extractor and a classifier. Formally, let go: X' + Hd be feature extractor and fo : Rd + y be a classifier, where d is the dimension of fcature space, Σ is the simplex and C' be the number of classes. The maps go and fo are usually neural networks parametrized by 9 and é.
As discussed in introduction, model selection (choosing hyper-parameters. training checkpoints, neural net- work architectures) is a crucial part of training neural networks. In the supervised learning setting, a validation set is used to estimate the model's accuracy. However, in UDA such approach is not possible as we have unlabeled target samples. Several strategies have been designed to address this issue. Below, we discuss the ones used in this work.
Source Accuracy (S-ACC). Ganin & Lempitsky (2015) used the accuracy estimated on a small validation set from the source domain to perform the model selection. While the source and target accuracies are related, there are no theoretical guarantees. Youu et al. (209 showed that when the domain gap is large this approach fails to select competitive models.
Deep Embedded Validation (DEV). Sugiyama et al. (2077 and Long et al. (001)) perform model selection through Smpoortancc-Wiighhe Cross- Validation (IWCV). Under the assumption that the source and target domain follow a covariate shift, the target risk can be estimated from the source risk through importance weights that give increased importance to source samples that are closer to target samples. Formally, the
importance weights w(x) correspond to the ratio of the larget and source densities, i.e., w(x) p(x)
g(x) and
are found by estimating each distribution using Gaussian kernels. Recently, You et al.] (2019) proposed an improved variant, Deep Embedded Validation (DEV), that controls the variance of the estimator, through method called control variate, and estimates the importance weights with a discriminative model that distinguish source samples from target samples leading more stable and effective method. The latter is based on the fact that
1
(xxx
1 + w(x)
3

---

Published in Transactions on Machine Learning Research (05/2023)
where & is an optimal source-target discriminator, meaning that if is drawn from p. xb(x) and xx(x) 0 if x is drawn from q.
Entropy (ENT). While minimizing the entropy of the target samples has been used in domain adaptation to improve accuracy by promoting tighter clusters, Morerio et al. (2018) showed that it can also be used for model selection. Formally, we want the classifier før and feature extractor go- that have the lowest entropy i.e.; Ei H (fo (9o(x())), where denotes the entropy. The intuition that a lower entropy model corresponds to a highly confident model with discriminative targel features and therefore reliable predictions.
Soft Neighborhood Density (SND). Saito et al. (021 argue that a good UDA model will have a cluster structure where nearby target samples are in the same class. They claim that entropy is not able to capture this property and propose the Soft Neighborhood Density (SND) score to address it. Formally, they compute the similarity between extracted features from the target data Sij= - (ge (x)), go (x))). Then, the SND score is defined as the entropy of the following probability distribution Pt. = (exp(Sia/+t/(22 exp(Sia)/7). The selected model ge the model with the highest SND score.
Target Accuracy (ORACLE). We consider well the target accuracy on all target samples. While we emphasize once again its use is not realistic in unsupervised domain adaptation (hence why we will refer to it as ORACLE), it has nonetheless been used to report the best accuracy achieved by the model along training in several previous works (Cao et al. 2018 Xu et al.) 2019 Jian et al., 2020 Gu et al., 2021 Nguyen et al. 2022). Here, we use it as an upper bound for all the other model selection strategies and to check the reproducibility of previous works.
Small Labeled Target Set (1-SHOT and 100-RND). For real-world applications in an industry setting, it is unlikely that model will be deployed without estimation of its performance on the target domain. Therefore, one can imagine situation where PDA method is used and small set of target samples is available. Thus, will compute the larget accuracy with 1 labeled sample per class (1-SHOT) and 100 random labeled target samples (100-RND) as model selection strategies. One could argue that the 100 random samples could have been used in the training with semi-supervised domain adaptation mothods. However, note that we do not know how many classes we have on the larget domain so it is hard to form a split when we have uncertainty of classes. For instance, 100-RND represents possibly less than 2 samples per class for one of our real-world dataset, as we do not know the number of classes, making a potential split between a train and valiclation target sets not possible.
3 Partial Domain Adaptation Methods
In this section, we give a brief description of the PDA methods considered in our study. They can be grouped into two families: adversarial training and divergence minimization.
Adversarial training. To solve the UDA problem, Ganin et al. (2016) aligned the source and target domains with the help of a domain discriminator trained adversarially to be able to distinguish the samples from the two domains. In addition to go and for we also consider an adversarial classifier De Z + {0,1} 1} parameterized by Ç. The maps are trained in a adversarial manner where Dr tries to distinguish the source data from the target data, while go is trained to confuse the domain discriminator Dç:
5-,0- = arg min arg max - Σ log (De +
Σ j=1
log (1- (19())))))
e C
However, when naively applied to the PDA setting, this strategy aligns target samples to source sample whose classes are not in the target domain and as a result the model performs worse than a model trained only on source data. To remedy this problem, Cao al. (2018) proposed PADA that introduces a PDA specific solution to adversarial domain adaptation: the contribution of the source-only class samples to the training of both the source classifier and the domain adversarial network is decreased. This is achieved through class weights that are calculated by simply averaging the classifier prediction all target samples. As the
4

---

Published in Transactions on Machine Learning Research (05/2023)
PDA Methods
PADA, SAFN, BA3US, AR, JUMBOT, MPOT
Model Selection Strategies)
S-ACC, ENT, DEV, SND, 1-SHOT, 100-RND, ORACLE
Architecture
ResNet50 backbone @ linear bottleneck @ linear classífication head
Experimental protocol 3 seeds on the 12 tasks of OFFICE-HOME: and 2 tasks of VIsDA
Table 2:
Summary of our considered methods, model selection strategies, architecture and datascts, where @ denotes the operation of adding layers.
Architecture
Runs per tasks
Model Selection
Method
(bottleneck)
Prrper-Paraneter
Along Training
PADA
Linear
1
IWCV (lacks details)
ORACLE
SAFN
NineLr
3
Unknown
ORACLE
BA3US
Linear
3
Unknown
ORACLE
AR
Din-Lincar
1
IWCV (lacks details)
ORACLE
JUMBOT
Linear
1
ORACLE
FINAL
MPOT
Linear
3
Unknown
ORACLE
Table 3: Summary the experimental protocol used for SOTA partial domain adaptation methods. We
refer Appendix A.1 for additional details.
source-only classes should not be predicted the target domain, they should have lower weights. More recently, Jian et al. 20200 proposed BA3US which augments the target mini-batel with source samples to transform the PDA problem into vanilla. DA problem. In addition, an adaptive weighted entropy objective is used to encourage incorrect classes have uniform and low prediction scores.
Divergence minimization. Another standard direction to align the source and target distributions in the feature space of a neural network is to minimize a given divergence between distributions of domains. The purpose of the minimization divergence approach is to find the optimal parameters 0* which minimnizes the divergence L:
argming L(g 984q).
where d8# denotes the pushforward operator of the map go- Xu el. al. 2019 empirically found that target samples have low feature compared to source samples. Based this insight, they proposed SAFN which progressively adapts the feature norms of the two domains by minimizing the Maximum Mean Feature Norm Discrepancy as the divergence L (Gretton et al.] 2012). Other approaches consider the optimal transport (OT) cost as the divergence L (Buushan Damodaran et al. 2018) with mini-batches (Peyré & Cuturi 2019 Fatras et al. 2020 2021b). For the PDA problem in specific, (Fatras et al., 2021a) developed JUMBOT, a mini-batch unbalanced optimal transport that learns joint distribution of the embedded samples and labels.
The use of unbalanced OT is critical for the PDA problem it allows to transport only the source samples whose classes are in the target domain. Based this work, (Nguyen et al. 2022 investigated the partial OT variant (Chapel et al. 2020), particular case of unbalanced OT, proposing M-POТ. Finally, another line of work is to use the Sanllorovich-uueenster similarly to WGAN (Arjovsky et al.) 2017). This is precisely the work of Gu et al. (2021) that proposed, AR. addition, source samples are reweighted in order to not transport the source-only class samples. The Kantorovich- Rubeishteeeeeeeeiin duality relies on one Lipschitz function which approximated using adversarial training like the PDA methods described above.
duality of optimal transport to perform the alignment
4 Experimental Protocol
In this scction, we discuss our choices regarding the training details, datasets and neural network architecture. We then discuss the hyper-parameter tuning used in this work. We summarize the PDA methods, model selection strategies and experimental protocol used in this work in Table 2 The main differences in the experimental protocol of the different published state-of-the-an (SOTA) methods is summarized in Table 3
5

---

Published in Transactions on Machine Learning Research (05/2023)
METHON
S. ONLY
PADA
SAFN
BA3US
AR
JUMBOT
MPOT
Reported
61.35
62.06
71.84
75.98
77.11
75.47
77.98
Reimplementation
61.87
63.72
74.72
75.98
76.00
77.15
77.31
Ours
61.87
63.72
63.30
75.98
72.73
77.15
77.31
Table 4: Comparison between reported accuracies on partial OFFICE-HOME from published methods with our reimplementation using the ORACLE model selection strategy. Ours denotes our reimplementation where all methods have the same bottleneck architectures discussed in Section 4
To perform our experiments we developed a PyTorch (Paszke et al. 2019 framework: BenchmarkPDA. We
make it available for researchers to use and contribute with new algorithms and model selection strategies:
httpss///giauub.ccm//bbrrrat
cow
It is the standard in the litcrature when proposing a new method to report directly the results of its competitors from the original papers (Cao et al., 2018 Xu et al.) 2019 Jian et al., 2020 Gu et al., 2021 Nguyen et al. 2022). As a result some methods differ for instance in the neural network architecture implementation (AR (Gu et al., 2021 SAFN (Xu et al. 2019)) or evaluation protocol JUMBOT (Fatras al., 2021a) with other methods. These changes often contribute to an increased performance of the newly proposed method leaving previous methods at disad disadvantage. Therefore we chose to implement all meth with the same commonly used neural network architecture, optimizer, learning rate schedule and evaluation protocol. We discuss the details below.
4.1 Methods, Datasets, Training and Evaluation Details
Methods. We implemented 6 PDA methods by adapting the code from the Official GitHub repositories of each method: Source Only, PADA (Cao et al. 2018), SAFN (Xu et al. 2019). BA3US (Jian et al.) 2020), AR (Gu et al. 2021), JUMBOT (atrra et al.) 2021a), MPOT (Nguyen et al. 2022). We provide the links to the different official repositories in Appendix A.1 comparison with previous reported results can be found in Table I1 and we postpone the discussion to Section
Datascts. We consider two standard real-world datasets used in DA. Our first datasct is OFFICE-HOME (Venkateswara el al. 2017). It is diflicult dataset for unsupervised domain adaplation (UDA), it has 15,500 images from four different domains: Art (A), Clipart (C), Product (P) and Real-World (R). For each domain, the dataset contains images of 65 object categories that are common in office and home scenarios. For the partial OFFICE-HOME setting, we follow Cao et al. (2018) and select the first 25 categories (in alphabetic order) in each domain as a partial target domain. We evaluate all methods on all 12 possible tasks, where by task we mean training a classifier using one domain as source data and another different domain as target data. For example, the AC lask is the partial domain adapíation scenario where the Art domain is used source and the Clipart domain is used as target. VISDA (Peng et al. 2017) is large-scale dataset for UDA. It has 152,397 synthetic images and 55,388 real-world images, where 12 object categorics are shared by these two domains. For the partial VIsDA setting, we follow [Cao et al. (0018) and select the first 6 categories, taken in alphabetic order, in each domain as a partial target domain. We evaluate the models in the two possible scenarios. We highlight that we are the first to investigate the performance of JUMBOT and MPOT on partial VISDA.
Model Selection Strategies We consider the 7 different strategies for model selection described in Section S-ACC, DEV, ENT, SND, ORACLE, 1-SHOT, 100-RND. We use them both for hyper-parameter tuning well selecting the best model along training. Since S-ACC, DEV and SND require source validation set, we divide the source samples into a training subset. (80%) and validation subset. (20%). Regardless of the model selection strategy used, all methods are trained using the source training subset. This is in contrast with previous work that uses all source samples, but necessary to ensure a fair comparison of the model selection strategies. We refer to Appendix A.2 for additional details.

---

Published in Transactions on Machine Learning Research (05/2023)
BA3US
JUMBOT
MPOT
SAFN
Dataset
Variant
ENT
DEV
SND
ENT
DEV
SND
ENT
DEV
SND
ENT
DEV
SND
Naive
52.60
63.10
44.48
52.30
26.75
17.67
49.01
16.72
30.63
32.12
49.67
5.01
OFFICE-HOME
Heurissic
58.45
63.10
60.96
56.24
45.79
55.16
49.01
45.61
30.63
46.27
49.67
49.67
Naive
39.06
36.99
1.14
35.89
54.53
11.99
75.04
55.33
36.11
52.82
53.26
0.83
VISDA
Heuristic
67.50 34.94
78
47.23
54.53
66.42
75.04
55.33
85.36
52.82
53.26
52.82
Table 5: Comparison between the naive model selection strategy and our heuristic approach. Accuracy on AC task for OPFICE-HOME and SR task for VISDA. Best results in bold.
Architecture. Our network is composed of a feature extractor with a linear classification layer on top of it. The feature extractor is ResNet50 (He et al. 2016), pre-trained on ImageNet (Deng et al. 2009), with its last linear layer removed and replaced by a linear bottleneck layer of dimension 256. This architecture is used by almost all methods with the exception of SAFN and AR which use different architectures. However we believe that it is important to use the same architecture for all methods to understand their own performance. While it is possible that we underestimate the performance of SAFN and AR as a result, it is also possible that other methods benefit from these different architectures. We leave this question for future work.
Optimizer. We use the SGD Robbins & Monro 1951) algorithm with momentum of 0.9, a weight decay of 54 and Nesterov acceleration. As the bottleneck and classifier layers are randomly initialized, we set their learning rates to be 10 times that of the pre-trained ResNet50 backbone. We schedule the learning rate Xo
with a strategy similar to the one in [Ganin et al. 2016); X = (I+#a) w. where i is the current iteration, Xo = 0.001, 7 = 0.001, v = 0.75. While this schedule is slightly different than the one reported in previous work, is the one implemented in the different official code implementations. We elaborate in the Appendix A.3 on the differences and provide additional details. Finally, as for the mini-batch size, JUMBOT and M- POT were designed with a stratified sampling, i,e., balanced source mini-batch with the same number of samples per class. This strategy reduces the optimal (ransport matching between largel. samples and source samples from source-only classes, which in turn leads to less target samples being classified as belonging to source-only classes. On the other hand, it was shown that for some methods (c.g. BA3US) using a larger mini-batch, than what was reported, leads to a decreased performance (Tatras el al.) 2021a). As a result, we used the default mini-batch strategies for each method. JUMBOT and M-POT use stratified mini-batches of size 65 for OFFICE-HONN and 36 for VISDA. All other methods use a random uniform sampling strategy with a mini-batch size of 36.
For the hyper-parameters chosen with each model selection strategy, run the methods for each tasks 3 times, each with a different seed (2020, 2021, 2022). We tried to control for the randomness across methods by setting the seeds at the beginning of training. Interestingly, as we discuss in more detail in Section 5 some het demonstrated a non-negligible variance across the different seeds showing that some
Evaluation Protocol.
and methods are not robust to randomness.
4.2 Plyper-Prammnn Tuning
perform random searches with the same number of runs for each method. In contrast, we perform hyper-paramett grid searches for each method. As a result, the hyper-parameter tuning budgets differs across the methods depending on the number of hyper-paramctors and the chosen grid. While one can argue this Icads to an unfair comparison of the methods, in practice in most real-world applications one will be interested in using the best method that our approach will precisely capture.
Previous works GGulrajani & Lopez-Paz 2021 Musgrave et al. 2021 2022)
The hyper- aaper-parametor tuning needs to be performed for cach tasks of cach datasct, but that would require a significant computational resources without a clear added benefit. Instead for each dataset, we first perform the pyper-parameter tuning on trained models over a single task: A2C for OFFICE- HOME and S2R for VISDA. Then in a second time, we use the selected hyper-paramcters from the first phase on the remaining tasks. This same strategy was adopted in Fatras et al., 2021a) and the parre-parameters were found to generalize to the remaining tasks in the dataset. We conjecture that this may be due to the fact that information
7

---

Published in Transactions on Machine Learning Research (05/2023)
regarding the number of target only classes is implicitly hidden in the hyper-parameters. See Appendix A.4 for more details regarding the hyper-parameters.
In practice during the first phase, we set the number of training iterations to 5000 steps for OFFICE-IIOME and 10000 steps for VISDA. Then, we apply the different model selection strategies to the different trained models and select the hyper-parameters that optimize the model selection strategies. As we are looking for the best hyper-paaameeers on a single task that hopefully generalize the remaining tasks, we do not consider the end of training as a hyper-parameter during this phase. In the second phase, we use the chosen Syper-parameters and run the methods all tasks. We use the model selection strategies to select the best model for each task along training, effectively considering the 'end of training' as a hyper-parameter.
With the wrong choice of hyper-parameters, we can end up with a degenerate model that predicts the same label for all examples with high confidence. This model will have good DEV, SND and ENT scores (being highly confident mcans it has low entropy) and therefore is considered a 'good' model according to these criterias when in fact it's quite the contrary.
search for JUMBOT, M-POT and BA3US were unsuccessful with the optimization reaching its end without the model being trained at all. The extreme case is a degenerate model that predicts the same label for all examples with high confidence. This poses a challenge to DEV, SND and ENT. A highly confident model has low entropy and therefore is considered a 'good' model according to ENT. A similar reasoning can be made for DEV and SND. This failure mode is accounted for in (Saito et al. 2021). ) Following their rocommendations, for JUMBOT, M-POT and BA3US, before applying the model selection strategy, we discard models whose source domain accuracy is below a certain threshold thr, which is set with the hcuristic as thr 0.9 Acc. Here Acc denotes the source domain accuracy of the Sourco-Only model. In our experiments, this leads to select models whom the source accuracy is al least of Ihr 69.01% for the A2C tasks on OFFICE-HOME and thr = 89.83% for the S2R tasks on VISDA. We choose this heuristic because the ablation study of some mothods showed that doing the adaptation decreased slightly the source accuracy (Bhushan Damodaran et al., 2018). Table 5 shows that our heuristic leads to improved results.
Several runs in our hyper-parameer
Lastly, when choosing the hyper-parameten parameters, we only consider the model the end of training, discarding the intermediate chockpoint models in order to select hyper-paramoters which do not lead to overfitting the end of training and better generalize to the other tasks. Following the above protocol, for each dataset we trained 468 models in total in order to find the best hyper-parameters. Then, to obtain the results with our neural network architecture on all tasks of each dataset, we trained an additional 1224 models for OFFICE-HOME and 156 models for VISDA. We additionally trained 231 models with the different neural network architectures for AR and SAFN. In total, 2547 models were trained to make this study and we present the different results in the next section.
5 Partial domain adaptation experiments
We start the results section by discussing the differences between our reproduced results and the published results from the different PDA methods. Then, we compare the performance of the different model selection strategies. Finally, we discuss the sensitivity of methods to the random seed.
5.1 Reproducibility Of Previous Results
We start by ensuring that our reimplementation of PDA methods was done correctly by comparing our reproduced results with previously reported results in Table [1] As such the model selection strategy used is ORACLE. On OFFICE-HOME, both PADA and JUMBOT achieved higher average tasks accuracy (1.6 and 1.7 percentage points, respectively) in our reimplementation, while for BA3US and MPOT we recover the reported accuracy in their respective papers. However, we saw a decrease in performance for both SAFN and AR of roughly 8 and 5 percentage points respectively. This is to be expected due to the differences in the neural network architectures. While we use a linear bottleneck layer, SAFN uses a nonlinear bottleneck layer. As for AR, they make two significant changes: the lincar classification head is replaced by spherical logistic regression (SLR) layer (Gu al., (2020) and the features are normalized (the 2-norm is set to dataset that requires tuning) before feeding them to the classification
dependent value, another
8

---

Published in Transactions on Machine Learning Research (05/2023)
DATASET
METHOD
S-ACC
ENT
DEV
SND
1-SHOT
100-RND
ORACLE
ONLY
60.38+00.5
60.73+0.2
00.
09.55+00.3
58.92+0.4
60.34+0.4
61.87+0.3
PADA
63.08++0.3
59.74+0.5
52.72-2.8
62.36+004
62.00+0.5
63.22+00.1
63.72+0.3
SAFN
62.09:10.
61.37+0.3
62.03=0.4
62.5910.1
49.30=0.7
62.36+0.2
63.30+0.2
OFFICE-HOME
BA3US
1..
73.36:10.6
62.25-7.1
75.37:10.8
65.5647.6
75.19-40.4
75.98.10.3
AR
65.68+0.3
70.
64.32=0.9
70.25#0.2
70.56+0.7
70.34=0.2
72.73+0.3
JUMBOT
62.8910.2
74.61110.8
61.28=0.1
72.2910.2
74.95-10.1
75.7410.33
77.1510.4
MPOT
06.24110.1
64.4610.1
61.37=0.2
46.92:10.4
68.2810.2
03.06:0.3
77.31
8. ONLY
25.15412.44
55.24433.
5..07=1.2
55.02:12.9
25.72:12.2
58.16440.6
59.48:10.4
PADA
47.4814.88
32.32.4.99
43.43.5.3
56.8311.0
53.15.12.9
54..8.12.77
54.5711..6
SAFN
58.20+1.7
42.83+6.3
58.62=1.3
44.8248.8
56.89+2.11
59.09+2.8
62.64+1.5
VISDA
BA3US
55.1013.7
65.58114
58.40=1.4
51.0711..
64.7711.4
67.44+1.2
67.6711.3
AR 66.68+1.0
61.27+3.6
67.20+1.5
55.69+0099
70.29+1.7
72.60+0.8
73.85+00.9
JUMBOT
60.6310.7
62.42+2.4
59.86=0.6
77.69:4.2
78.34+1.9
83.49=1.9
84.01#1.9
MPOT
70.02+2.0
74.64+4.4
61.62=1.3
78.40#3.9
70.96+3.7
86.69#5.1
86.95+5.0
Table 6: Task accuracy average over seeds 2020, 2021, 2022 on Partial OFFICE-HOME and Partial VISDA for the PDA methods and model selection stralegy. For each method, we highlight the best and worst label-free model selection strategies in green and red, respectively.
head. While we account for the first change by comparing to AR (w/ linear) results reported in (Gu ct al. 2021), in our neural network architecture we do not normalize the fcatures. These changes, nonlincar bottleneck layer for SAFN and feature normalization for AR, significantly boost the performance of both methods. Performing an ablation study of new architectures with respect to the original one, done in (Gu et al.) 2021), allows us to understand the architecture's influence on the method. We recommend authors to perform one as it also makes the method casier to reproduce results. When now comparing our reimplementation with the same neural network architectures, our SAFN reimplementation achieves higher average tasks accuracy by 3 percentage points, while our AR reimplementation is now only 1 percentage points below. The fact that AR reported results are from only one run, while ours are averaged across 3 distinct seeds, justifies the small remaining gap. Moreover, we report higher accuracy or on par on 4 of the 12 tasks. Given all the above and further discussion of the VISDA dataset results in Appendix [ our reimplementations are trustworthy and give validity to the results we discuss in the next sections.
5.2 Results for Model Selection Strategies
Model Selection Strategies (w/ w/o target labels) All average accuracies on the OFFICE-HOME and VISDA datasets can be found in Table 6 For all methods on OFFICE-HOME,
see that the results for model selection strategies which do not use target labels are below the results given by ORACLE. For some pairs, the drop of performance can be significant, leading some methods to perform on par with the s. ONLY method. That is the case on OFFICE-HOMR when DEV is paired with either BA3US, JUMBOT and MPOT. Even worse is MPOT with SND as the average accuracy is more than 10 percentage points below that of S. ONLY with any model selection strategy. Overall on OFFICE-HOME, except for MPOT, all methods when paired with either ENT or SND give results that are at most percentage points below compared to when paired with ORACLE.
A similar situation can be seen over the VISDA dataset where the accuracy without target labels can be down to 25 percentage points. Yet again, some model selection strategies can lead to scores even worse than S. ONLY. That is the case for PADA, SAFN and BA3US. Contrary to OFFICE-HOME, all model selection strategies without target labels lead to at least one method with results on par or worse in comparison to the S. ONLY method. More generally, no model selection strategy without target labels can lead to score on par to the ORACLE model selection strategy. Finally, PADA performs worse than S. ONLY for most model selection strategics, including the ones which use target labels. Perhaps little surprising, when combined with SND it achieved higher per run average than with ORACLE, although within the standard deviation. This is also consequence of the random seed dependence mentioned before on VISDA: as the hyper- hyper-paramee parameters were
9

---

Published in Transactions on Machine Learning Research (05/2023)
chosen by performing just one run, were simply "unlucky". In general, all of this confirms the standard assumption in the literature regarding the difficulty of the VISDA dataset.
These experiments also allow us to draw some conclusions regarding the robustness of methods to model We see that PADA and BA3US are not
selection strategies with respect to the number of robust for either one of the datasets, while SAPN is robust to the choice of model selection strategy for OFFICE-HOMM. These are the methods with fewest ayper-parameters ruling out the fact that less hyper- parameters leads to more robust methods and suggesting it is method and dataset specific. Furthermore regarding reliability, we find that the optimal transport based approaches (MPOT, JUMBOT) to be the most sensitive to model sclection strategics as they exhibit the largest performance gaps between the best and worse model selection strategies without target labels. It shows, that while they are able to achieve SOTA results with the ORACLE model selection strategy, that performance is highly tied to the paramete choice.
Model Selection Strategies (w/ target labels) We recall that the ORACLE model selection strategy uses all the target samples to compute the accuracy while 1-SHOT and 100-RND use only subsets: 1-SHOT has only one sample per class for a total of 25 and 6 on OFFICE-HOME and VISDA, respectively, while 100-RND has 100 random target samples. Our results show that using only 100 random target labeled samples is enough to reasonably approximate the target accuracy leading to only a small accuracy drop (one percentage point in almost all casos) for both datasets. Not surprisingly, the gap between the 1-SHOT and ORACLE model selection strategies is even bigger, leading in some instances to worse results than with a model selection strategy that uses no target labels. This poor performance of the 1-SHOT model selection strategy also highlights that semi-supervised domain adaptation (SSDA) methods are not & straighttorward alternative to the 100-RND model selection strategy. While one could argue that the target labels could be leveraged during training like in SSDA methods, one still needs labeled target data to perform model selection. However our results suggest that we would need at least 3 samples per class for SSDA methods. In addition, knowing that we have a certain number of labeled samples per class provides information regarding which classes are target only, one of the main assumptions in PDA. In that case, PDA methods could be tweaked. This warrants further study that we leave as future work. Finally, we have also investigated a smaller labeled larget set of 50 random samples (50-RND) instead of 100 random samples. The accuracies of methods using 50-RND were not as good as when using 100-RND. All results of pairs of methods and 50-RND can be found in Appendix B The smaller performance show that the size of the labeled target set is important element and we suggest to use at least 100 random samples.
Model Selection Strategies (w/o target labels) Among all 49 pairs of methods and model selection strategics, only the (BA3US, ENT) pair achioved average tasks accuracies within 3 percentage points of its ORACLE counterpart (i.e., (BA3US, ORACLe)), while improving over S. ONLY model. Our experiments show that there is no model selection strategy which performs well for all methods. That is why to deploy models in real-world scenario, we advisc to test selected models small labeled target set (i.e., 100-RND)) to assess the performance of the models as model selection strategies without target labels can perform poorly.
Our conclusion is that the model selection for PDA methods is still an open problem. We conjecture that it is also the case for domain adaptation as the considered metrics were developed first for this setting. For future proposed methods, researchers should specify not only which model selection strategy should be used, but also which hyper-paramett search grid should be considered, to deploy them in a real-world scenario.
Comparison with other model selection strategy studies Our study of model selection strategies is related to the Gulrajani & Paz (2021); Saito et al. (2021) : Musgrave et al. (2022). In their respective study, Gulrajani & Lopez-Paz (2021) for domain generalization and Saito et al. 2021 Musgrave et al. (2022) for unsupervised domain adaptation argue that the methods remain incomplete without model selection
strategies as the latter can have a big impact on their performance. Our findings are similar to theirs and we recommend that for each new partial domain adaptation method, its authors should recommend a target label-free model selection strategy.
10

---

Published in Transactions on Machine Learning Research (05/2023)
TASK
METHOD
S-ACC
ENT
DEV
SND
1-SHOT
100-RND
ORACLE
ONLY
46.96 1.5
48.17 3.9
49.00 1 0.9
48.17 + 3.9
49.43 =
50.01 1.6
51.86 1.4
PADA
44.56 + 5.9
40.83 11.3
11.04 4.3
56.14 9.7
52.94 + 4.3
49.34 + 8.4
49.34 +84
SAFN
52.04
29.86 16.7
28.46 2.9 16.5
49.97 = 3.3
47.83
56.88 2.1
S2R
BA3US
44.21
1.917
1.9
7..
66.79 1.5
71.45
71.77
AR
68.39 1.3
75.28 2.9
68.54
57.61 0.4
70.11 144
75.09 5.2
76.33 4.5
JUMBOT
55.23 + 2.3
56.25
54.35 2.0
75.23 + 8.4
89.94 81.27
90.55 1 0.5
MPOT
64.57 + 2.9
82.10 2.0
57.02 + 1.5
84.45 + 0.4
71.33 4.4
87.20 + 2.3
87.23 + 2.3
ONLY
63.34 3.4
62.32 2.7
61.13 1 3.3
61.88 1 2.3
66.30 3.9
2.0
67.11 111:200 2.1
PADA
50.39 +
23.80 1.6
45.82 + 9.2
57.53 10.3
53.36 1.7
59.43 5.8
59.81 + 6.2
SAPN
64.37
5.2
64.82 + 0.5
3.3
63.82 1
7..
68.40 1.40+11
R2S
BA3US
65.99 4.6
1.3
56.01 2.9
62.75 2.6
63.44
63.56 1
AR
64.97 +
53.26
65.86 3.5
53.78 2.1
70.46 + 4.7
70.11
71.36 + 5.5
JUMBOT
66.04 1.0
68.59 4.6
65.36 0.8
80.16 1.1
75.42
77.03
77.46 = 3.3
MPOT
75.47 + 3.8
67.18 9.1
66.21 + 1.2
72.36 + 7.4
86.18 70. 8.1
86.67 + 7.8
ONLY
55.15 + 2.4
55.24 +-14--..2
55.07 + 1.2
55.02 + 2.9
4 +
68.16 0.6
59.48 + 0.4
PADA
47.48 + 4.8
32.32 4.9
43.43 5.3
56.83 + 1.0
53.15 2.9
54.38
54.57 - 2.6
SAPN
58.20 1.7
42.83 6.3
58.62 + 1.3
44.82 8.8
56.89 2.1
59.09 2.8
62.64 1.5
Avg
BA3US
55.10
65.58 1.4
58.40 1.4
51.07 1 4.3
64.77 1.
67.44
67.67 + 1.3
AR
66.68 1.0
64.27
67.20 1.5
55.69 + 0.9
70.29 =
72.60 - 0.8
73.85 0.9
JUMBOT
60.63 4.7
62.42 2.4
59.86 0.6
77.69
4.2
78.34 1.9
83.49 1.9
84.01 1.9
MPOT
70.02 + 2.0
74.64
+ 4.4
61.62
1.3
78.40
+ 3.9
70.96 3.7
86.69 + 5.1
86.95 + 5.0
Table 7: Accuracy of different PDA methods based on different model selection strategics on the Partial VISDA tasks. Average done over three seeds (2020, 2021, 2022). For each method, we highlight the best and worst label-free model selection strategies in green and red, respectively.
5.3 Random Seed Dependence
Ideally, PDA methods should be robust to the choice of random seed. This of particular importance when performing typer-paramcter tuning since typically only one run per set of typer-parameters is done (that was the case in our work as well). We investigale this robustness by averaging all the results presented over three different seeds (2020, 2021 and 2022) and reporting the standard deviations. This is in contrast with previous work where only single run is reported (Fatras et al. 2021a Gu et al. 2021). Other works (Cao ct al. 2018 Xu et al. 2019 Jian et al., 2020) that report standard deviations do not specify if the random seed is different across runs. Results for all tasks on VISDA dataset are Table 7 and on OFFICE-HOMP in Appendix due to space constraints.
Our experiments show that some methods express a non-negligible instabilities over randomness with respect to any model selection methods. This is particularly true for BA3US when paired with DEV and 1-SHOT model selection strategies: there are several tasks where the standard deviation is above 10%. While in this case this instability may slem from the poor performance of the model selection strategies, it is also visible when ORACLE is the model selection strategy used. For instance, the M-POT has standard deviation of 3.3% on the AP tasks of OFFICE-HOME which corresponds to a variance of 11 11%. On VISDA this instability and seed dependence is even larger.
6 Conclusion
In this paper, we investigated how model selection strategies affect the performance PDA methods. We performed a quantitative study with six PDA methods and seven model selection strategies on two real-word datascts. Based on our findings, we provide the following recommendations:
i) Torget label samples should be used to test models before using them in reol-world scenorio. While this breaks the main PDA assumption, it is impossible to confidently deploy PDA models selected without the
11

---

Published in Transactions on Machine Learning Research (05/2023)
use of target labels. Indeed, model selection strategies without target labels lead to significant drop in performance in most cases in comparison to using small validation set. We argue that the cost of labeling it outweighs the uncertainty in current model selection strategies.
it) The robustness of new PDA method to randomness should be tested over at least three different seeds. We suggest to use the seeds (2020, 2021, 2022) to allow for a fair comparison with our results.
vit) An ablation study should be considered when novel architecture is proposed to quantify the associated increase of performance.
As our work focus on a quantitative study of model selection methods and reproducibility of statc-of-the-ar partial domain adaptation methods, we do not see any potential ethical concern. Future work will unvestigate new model selection strategies which can achieve similar results as model selection strategies which use label target samples.
Acknowledgments
This work was partially supported by NSERC Discovery grant (RGPIN-2019-065112) and a Samsung grant. Thanks also to CIFAR for their support through the Canada CIFAR AT Chairs program. Authors thank Christos Tsiiiottis and Chen Sun for early comments on the manuscript.
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasscrstein generative adversarial networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Leurning, volume 70 of Proceedings of Machine Learning Research, pp. 214-223, International Convention Centre, Sydney, Australia, 06 11 Aug 2017. PMLR. URL http://proceedingss mlr prt.:prre/////o//rrrrrss/// html
Bharath Bhushan Damodaran, Benjamin Kellenberger, Remi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In The European Conference on Computer Vision (ECCV), September 2018.
Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 754 763, 2017. doi: 10.1109/ICCV 2017............... 20177.88.
Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 135 150, 2018.
Lactitia Chapel, Mokhtar Z. Alaya, and Gilles Gasso. Partial optimal transport with applications positive- unlabeled learning, In Advances in Neural Information Processing Systems, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. In 2009 IEEE conference on compuler vision and pallern recognition, pp. 248 255. Ieee, 2009.
Kilian Fatras, Younes Zine, Rémi Flamary, Remi Gribonval, and Nicolas Courty. Learning with minibatch wasserstein asymptotic and gradient properties. In Silvia Chiappa and Roberto Calandra (eds.), Pro- ceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2131-2141, Online, 26-28 28 Aug 2020. PMLR. URL html
http://prroeeedings mlr. atress/v108/fatraaste
Kilian Fatras, Thibault Sejourne, Rémi Flamary, and Nicolas Courty. Unbalanced minibatch optimal trans- port; applications to domain adaptation. In Marina. Meila. and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3186 3197. PMLR, 18 24 Jul 2021a. URL httt:::/tpccceeiiiiiie httpreedddnnn......eees//seeeeee http///199fffatrrsas html
mlr
Kilian Fatras, Youns Zine, Szymon Majewski, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Mini- batch optimal transport distances; analysis and applications, 2021b.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Cham- bon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Téo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamoniy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python opti- mal transport. Journal of Machine Learning Research, 22(78):1 8, 2021. URL http://jmlm org org/papers/ v22/20-451. html
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei (ods.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1180 1189, Lille, France, 07 09 Jul 2015. PMLR. URL
hhaps://proceeedinss mlr. wal...preesv/33/ wtt.......s////3//an.......... gaainn5..tml
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. aamain-adversariio training of neural networks. The journal of machine learning research, 17(1):2096 2030, 2016.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. URL http://jmlr. org/
(nnpers/v13/geettonnl2 html
Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo-label loss. In IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2020.
Xiang Gu, Xi Yu, Yan Yang, Jian Sun, and Zongben Xu. Adversarial reweighting for partial domain
adaptation. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https: //openreview //oppareeieee..neffoormrttitttee wwt.://oooooriiidtsl..l.........
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain genoralization. In International Conference htt.pp/eeeeeeeeeeeeeeoooooaaaaaa
on Learning Representations, 2021. URL
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109 CVVP/.V1P.22220000
Liang Jian, Wang Yunbo, Hu Dapeng, He Ran, and Feng Jiashi. A balanced and uncertainty aware approach for partial domain adaptation. In European Conference on Computer Vision (ECCV), August 2020.
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael Jordan. Conditional adver- sarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https://proccedding www.iig.......s...e htt.rsss.cc/ppppr/////////e Paper. .dd.
Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsu- pervised deep domain adaptation. In International Conference on Learning Representations, 2018. URL
https://opeenreview. http:pppwererreeeeeeeeerrrraraae net//oorumkk
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Unsupervised domain adaptation: reality check. arXiv preprint arxi:: Arzio- 2111.1671,, 2021.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Benchmarking validation methods for unsupervised domain adaptation. arXiv preprint arXiv: Arrio Arriv-2208.071 2206007360, 2022.
Khai Nguyen, Dang Nguyen, Tung Pham, and Nhat Ho. Improving mini-batch optimal transport via partial transportation. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 22(10):1345-1359 2010. doi: 11109//KEE22009.199
13

---

Published in Transactions on Machine Learning Research (05/2023)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygekzier, d'Alché-Bue, E. Fox, and R. Garnett (eds.), Aduances in Neural Informa- tion Processing Systems 32, pp. 8024-8033 Curran Associates, Inc., 2019. URL http:ttt:/// /tatt:://pappre ceaapers.neuriie neurips. http::/pww...........a.aaaaaa/aa
learnpigeennnnn--nee library.brr.. -pdf
Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine, 32(3):53 69, 2015. doi: 10.1109/MSP:2011
2347059.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. CoRR, abs/1710 0bss/171000695 2017.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 2019.
Robbins and S. Monro. A stochastic approximation method. Annals of Mathematicnl Statistics, 22: 400 407, 1951.
Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. arXiv preprint arXiv:2108. 10860, 2021.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Rober Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, ((35):985-1000) 2007. URL http:
wwtg:/appers/////suulaaaaaa html
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervisod domain adaptation. In (IEEE) Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature approach for unsupervised domain adaptation. In The IEEE International Conference Computer Vision (ICCV), October 2019.
Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7124-7133. PMLR, 09-15 Jun 2019. http:httt:://// htttpp::/prroceeddngg..... mlr. wtt.r/////////e.. httasa/v7///// v97//ou19.. html
14

---

Published in Transactions on Machine Learning Research (05/2023)
A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Supplementary material
Outline. The supplementary material of this paper is organized as follows:
In Section
A
we give more details experimental protocol.
In Section
B we provide additional results from our experiments.
A Additional details on Experimental Protocol
A.1
Implementations in BenchmarkPDA
In order to reimplement the different PDA methods, we adapted the code from the official repository asso- ciated with each of the paper. We list them in Table g
Method
Code Repository
PADA
SAFN
BA3US AR JUMBOT M-POT
http:///gtthbb..com//mi/////a//
Table 8: Office Github code repositories for the PDA methods considered in this work.
One of our main claims regarding provious work is the use of target labels to choose the best model along training. This can be easily verified by inspecting the code. For PADA il can be seen on line 240 of the script "train pada.py", for BA3US in line 116 for the script haun_partiallpys for M-POT it can be seen line 164 of the file "run mOT.ppy" for SAFN it can be seen in the "eval.py" file and finally for AR in line 149 of the script "train.py".
For JUMBOT and M-POT which are based on optimal transport, we used the optimal transport solvers from FFlamary et al. 2021).
A.2 Model Selection
DEV requires learning a discriminative model to distinguish source samples from target samples. Its neural network architecture must be specified as well the training details. You et al. (2019) (DEV) use a multilayer perceptron, while Saito et al. (202T) (SND) use Support Vector Machine in their reimplementation of DEV. We empirically observed the latter to yield more stable weights and that was the one we usod. In order to train the SVM discriminator, following (Saito et al. 2002), we take 3000 feature embeddings from source samples used in training and 3000 random feature embeddings from target samples, both chosen randomly, We do 80/20 split into training and test data. The SVM is trained with linear kernel for maximum of 4000 iterations. Of 5 different SVM models trained with decay values spaced evenly on log space between 10-2 and 10ah the one that leads to the highest accuracy (in distinguishing source from target features) on the test data split is the chosen one.
As for SND, it also requires specifying a temperature for temperature scaling component of the strategy. We used the default value of 0.05 that is suggested in (Saito al. 2021
15

---

Published in Transactions on Machine Learning Research (05/2023)
Finally, mention that the samples used for 100-RND were randomly selected and their list is made available together with the code. As for the samples used for 1-SHOT, they are the same as the ones used in semi-supervised domain adaptation.
A.3 Optimizer
In general, all methods claim to adopt Nesterov's acceleration method as the optimization method with a momentum of 0.9 and setting the weight decay set to 5 10-4. The learning rate follows the annealing strategy as Ganin et al. (2016):
Up = 1 (1 + a.,
where p is the training progress linearly changing from 0 to 1, Mo 0.01 and a 10 and B = 0.75.
However, inspecting the Official code repo for each PDA method, the actual learning schedule is given by
Mi [0 + a
where is the iteration number in the training procedure, μ4 0.01 and a 0.001 and 0.75. Only when the total number of iterations is 10000 do the learning rate schedules match. Tn this work, we followed the latter since it is the one indeed used. For OFFICE-HOME, all methods are trained for 5000 iterations, while for VISDA they are trained for 10000 iterations, with the exception of the S. ONLY which is trained for 1000 iterations on OFFICE-HOMB and 5000 iterations on VISDA.
A.4 Hyper- Parameers
Tn Table g we report the values used for each hyper-parameter in our grid search. We report in Table 10 the prraeeparameters chosen by each model selection strategy for each method on both datasets. In addition, for the reproducibility of AR with the proposed architecture in Gu et al. (2021).ar a feature normalization layer is added in the bottleneck which requires specifying r, the value to which the 2-norm is set. This hyper- parameter therefore included in the hyper-paree syper-parammter grid scarch with the possible values of (5, 10, 20} which are the different values used in the experiments in (Gu al. 2021
B Additional Discussion of Results
In this section, we provide additional results that we could not add to the main paper due to the space constraints.
We start by ensuring that reimplementation of PDA methods was done correctly by comparing our reproduced results with previously reported results in Table
In Table 12 we show the accuracy per tasks on OFFICE- HOME averaged over three different seods (2020, 2021, 2022) for all pairs of methods and model selection strategies.
In Table 13 we compare previously reported results with ours on VISDA. While proposed methods reported rosults OFFICE-HOME, only PADA and AR results reported in the original papers for VISDA. Gu et al. 2022 AR) also report results for BA3US. Analysing the results, we see a 9 percentage point decrease in average tasks accuracy for PADA, but our experiments show that there is a significant seed dependence which we discuss in detail below. This is particularly important since Cao et al. (2018) (PADA) report results from a single run. Comparing our best seeds for PADA on the SR and RS tasks, we achieve 58.01% and 67.9% accuracy versus a reported 53.53% and 76.5%. Moreover, we point out that the official code repository for PADA does not include the details reproduce the VISDA experiments, so it possible that minor tweaks (e.g learning rate) are necessary. As for BA3US, our results are within the standard deviation being better the SR tasks and worse the RS tasks. Finally as for AR the results on OFFICE-HOME show, can be explained by the differences in the neural network architecture.
decrease in performance which,
Finally in Table 14 we show all the average tasks accuracies from all pairs of methods and model selection strategies on the OFFICE-HOME and VISDA datasets including the 50-RND model selection strategy.
16

---

Published in Transactions on Machine Learning Research (05/2023)
Method
HP
Values
PADA
A
[0.1, 0.5, 5.0,
Aurce Aent
[0.1, 5, 10) [0.01, 0.05, 0.1, 0.5,
BA3US
[0.005, 0.01, 0.05, 0.1,
SAFN
A
[0.01, 0.1,1.0)
Po
[2.5, 5.0, 7.5, 10.0]
Aup
[5.0, 10.0]
AR
Ajou
Aup
Aent
[0.01, 0.1,
T
[0.001, 0.01, 0.1]
!/1
[0.00001, 0.0001, 0.001, 0.01, 0.1]
JUMBOT
7/2
[0.1, 0.5, 1.J
7/3
[5, 10, 20]
[0.5, 1.0, 1.5)
771
[0.0001, 0.001, 0.01, 0.0] 10.1, 1.0, 5.0, 10.0] [0.1, 0.2, 0.4]
MPOT
1/2
m
Table 9: Hyper-Parace Parameter values for each PDA method considered in the grid search.
17

---

Published in Transactions on Machine Learning Research (05/2023)
Method
Datasct
HP
ORACLE
1-SHOT 50-RND 100-RND S-ACC
ENT
DEV
SND
OFFICE-HOME
X
0.5
0.1
0.1
0.5
0.1
1.0
5.0
0.5
PADA
VISDA
X
0.5
1.0
10.0
0.5
1.0
0.5
5.0
0.1
X
0.005
0.1
0.005
0.01
0.005
0.01
0.005
0.005
OFFICE-HOME
Ar
0.1
0.01
0.01
0.01
0.01
0.1
0.1
0.1
SAFN
0.005
0.005
0.05
0.05
0.005
0.05
0.005
0.05
VISDA
Ar
0.1
0.01
0.01
0.01
0.01
0.01
0.01
0.01
Awcee
5.0
10.0
5.0
5.0
5.0
0.1
10.0
1.0
OFFICE-HOME
Xent
0.05
0.05
0.01
0.05
0.01
0.1
0.05
0.01
BA3US
Awce
1.0
1.0
0.1
1.0
5.0
1.0
5.0
5.0
VISDA
Aent
0.5
0.5
0.5
0.5
0.05
0.5
0.05
1.0
Po
2.5
2.5
5.0
5.0
2.5
5.0
7.5
10.0
Aup
5.0
5.0
10.0
5.0
5.0
10.0
10.0
10.0
OFFICE-OMME
Alow
-5.0
-5.0
10.00
-5.0
-5.0
-10.0
-10.0
-10.0
Xent
0.1
0.1
1.0
1.0
0.01
1.0
0.01
1.0
AR
Po
2.5
2.5
2.5
2.5
2.5
7.5
2.5
10.0
Aup
10.0
10.0
10.0
10.0
5.0
10.0
10.0
10.0
VISDA
Alow
-10.0
-10.0
-10.0
-10.0
-5.0
-10.0
-10.0
-10.0
Aent
0.1
0.1
0.1
0.1
0.01
0.1
0.01
0.01
0.01
0.01
0.01
0.001
0.1
0.01
0.01
0.001
7]1
0.0001
0.0001
0.001
0.0001
0.01
1e-05
0.01
1e-05
OFFICE-HOME
f)2
0.5
1.0
0.5
0.1
0.1
0.5
1.0
1.0
7/3
10.0
5.0
5.0
5.0
5.0
20.0
10.0
5.0
JUMBOT
0.01
0.01
0.01
0.01
0.001
0.01
0.001
0.01
7]1
0.001
0.001
0.001
0.001
0.01
1c-05
0.01
0.0001
VISDA
7)2
1.0
1.0
0.5
1.0
0.1
0.5
1.0
1.0
7/3
5.0
5.0
5.0
5.0
10.0
5.0
20.0
5.0
0.5
0.5
1.0
0.5
1.0
1.5
1.0
1.5
7)1
0.01
0.01
0.01
0.01
0.001
0.0001
1.0
0.01
OFFICE-HOME
7/2
10.0
1.0
1.0
1.0
1.0
10.0
0.1
1.0
m
0.3
0.1
0.1
0.2
0.3
0.4
0.2
0.4
MPOT
0.5
0.5
0.5
0.5
1.0
1.0
1.0
0.5
7)1
0.01
0.001
0.01
0.01
0.001
0.0001
0.0001
0.01
VISDA
?/2
1.0
1.0
1.0
1.0
1.0
10.0
1.0
10.0
m
0.3
0.1
0.3
0.3
0.2
0.4
0.2
0.3
Table 10: Hyper-parameters selected for the different. methods for each model selection strategy on both HPFICE-HOME and VISDA.
18

---

Published in Transactions on Machine Learning Research (05/2023)
METHOD
A2C
A2P
A2R
C2A
C2P
C2R
P2A
P2C
P2R
R2A
R2C
R2P
Avg
S. ONLYf
46.33
67.51
75.87
59.14
59.94
62.73
58.22
41.79
74.88
67.40
48.18
74.17
61.35
8. ONLY (Ours)
45.43
68.91
79.53
55.59
57.42
65.23
59.32
40.80
75.80
69.88
47.20
77.31
61.87
PADA
51.95
67.00
78.74 74
52.16
53.78
59.03
52.61
43.22
78.79
73. 73
56.60
77.
62.06
PADA (Ours)
50.53
67.45
80.14
57.30
54.47
64.55
61.07
40.94
79.55
73.09
54.63
80.93
63.72
SAFNI*
58.93
76.25
81.42
70.43
72.97
77.78
72.36
55.34
80.40
75.81
60.42
79.92
71.84
8AFN (Ours)
59.98
79.85
85. 18
72.02
73.73
78.54
76.09
59.32
83.25
80.04
64.20
84.44
74.72
SAFN (Ours)
49.57
68.55
78.26
57.91
59.29
66.81
59.87
45.29
75.98
69.08
51.68
77.29
63.30
BABUS
60.62
83.16
88.39
71.75
72.79
83.40
75.45
61.59
86.53
79.25
62. 80
86.05
75.98
BA3US (Ours)
63.26
82.75
80.16
69.91
71.93
77.58
75.73
59.94
86.89
80.93
66.
86.93
75.98
AR
62.13
79.22
89.12
73.92
75.57
84.37
78.42
61.91
87.85
82.19 19
65.37
85.27
77.11
AR (Ous)
62.75
81.55
89.07
71.63
73.41
82.94
75.88
61.03
85.70
79.86
62.93
85.30
76.00
AR (Ours)
57.33
79.61
86.31
69.45
71.88
79.94
70.28
53.51
83.78
77.26
59.68
83.72
72.73
JUMBOT!
62.70
77.50
84.40
76.00
73.30
80.50
74.70
60.80
85.10
80.20
66.50 50
83.90
75.47
JUMBOT (Ours)
61.87
78.19
88.
77.69
76.75
84.15
76.83
63.72
84.80
81.79
64.70
87.17
77.15
MPOTT MPOT (Ours)
64.60
80.62
87.17
76.43
77.61
83.58
77.07
63.74
87.63
81.42
68.50
55
87.38
77.98
64.48
80.88
86.78
76.22
77.95
82.59
75.18
64.60
84.87
80.59
67.04
86.52
77.31
Table 11: our implementation using the ORACLE model selection stratogy. denotes different bottleneck architectures.
Comparison between reported (f) accuracies partial OFFICE-HOME from published methods with
19

---

Published in Transactions on Machine Learning Research (05/2023)
Txck]
| Marmoon
1X07
DWV
5000
1-s00r
GRACKE
8. Dea
44.00
=1
1.7
45.17
4:
41.74
1.8
42.33
6: 1.3
61:4=:1 1.7
$1.28
1.6
45.03
PAD4
50.
2.85
16
2.9
61.70 10 1.3
50.43
6: 0.5
52
02
30.41
0.5
59.50
830%
17:36
+10
0.1
87:00
2.0
08.19
111-4-000 0.6
19.37
0.3
31.00
3.7
47.58
0.5
19:57
0.03
AC
BA3U5
54.99
m
4.7
20
0.9
41.07
4: 18.5
02.21
0.0
44.50
21.0
02.03
2.0
51
m1 1.2
54.00
41.20
3.6
64.07
=: 1.6
50:00 2.
54.00
2.0
07.
1U30002
10.07
01.2
37
020
0.0
46.31
0.1
56.00
2.8
60
01.07
+ 0.9
01.5T=11
MPST
53.07
0.3
700 14
40.07
0.7
19
61.70
1.2
01.08
07.71
07.51
18:99
06.50
65.01 1.4
1.6
05.59 1.0
71.34
50.05
82.70 0.9
0.7
78.05
70.20
T1.00
78.58
79.01 1.6
5. 0852
78.97 0:1
70.24
09.29
77.35
79.50 0.5
PARA
70.79
68.00
70.72
78.05
80.14 1.4
SAEX
77-02
77.73
77.43
81
02.92
02
51.00
06.3 79.75
10.20-
05.47
88.29
72.01
23
82.97
81.91
86.51
77.14 0.3
$5.24
70.42
84.70 2.1
50
HPST
78.04
82.00
86.78 0.5
5. 0852
22.66
= 11.91
$4.21 2.1
01.42 2.7
61.76
9.7
50.47 2.4
03.76 1.1
25.00:000
PADA
58.00
= 1.4
50,07
+ 2.7
35.00
+ 10.1
=7 1.0
51.07
5.0
36,00
2 1.5
07.80
1.5
5AF%
57.85
1 0.0
56.54.1
2.2
56.75
0.5
57.91
0.3
18
2.1
56.42
1.0
57.01
1.5
BA515
61.08
2: 6.2
58.96
1.6
00.70:
3.2
68.50
2 0.0
66
1.4
05.16
1.2
09.01
1 0.2
68.21
8: 1.6
$4.86
2.0
02.72
1.0
64.82
1.6
58.99
02
04.55
2.4
09:66
0.5
1030000
60.09
=: 0.1
76.07
06.81
0.1
T1.81
1.3
74
0.0
24.00 0.4
77.00
0.1
MPOT
61 00.00-
0.5
50.58
0..
56.01
0 1.0
10.11
2.4
00:21 0.4
72.08
1.0
76.22 0.1
51.81
0.1
55.50
0.0
$1.55
1.2
53.45 2.1
53.24
33.57
2.2
57.02
PADA
56.13
8:
1.4
17.28
0.1
26.21
1. 20.5
52.10
1.7
56.25
53.90
16
51.17
3 1.7
57.89
8: 0.7
50.07
0.7
68.17
1.2
58.17
1.2
25
0.7
08.19
59.29
0.5
BASUS
67.13
82 8.0
71.07
0.3
50.08
4: 10.5
71.45
3.5
00.78 2 15.5
71.66
71.96
1.6
00.51
3.1
16.10-1
01.85
03.00-12 3.0
08.25
09:00
3.7
0.0
50.59
1:3
71.85
36.36
11.84
75
76.00
76.75
0.
MPOT
64.18
15.00:1
57.06
58.06
05.85
09:77
77.06
1.5
00:12
05.51
72.05-
77.58
58.77 1 0.51
56.95
57.91
55.37
56.13
58.50
59:52
0.7
00:33
2.1
56.00-44
07.91 3.7
00:88 3.0
58
00.27 2.7
01.07
+ 3.0
58.95
0.7
16.
39.08 0.5
12:33
39:00
61
59-87
0.0
RASU
08:00
5:0
16
01:00
T0.19
08.38
75.15
75
1.3
63:39
5.1
57.58
01.05
05.00
10.7
06.33
70.26
1.0
JUMB00:
44.24
=
72
9.4
38.00:
mu:25 0.5
75:81=1..
MPNT
64.13
mm 0.3
0.10 56.25-44
07.64 0.8
13.74 2
1.0 70
72.64 6
75.38-1..
20.25
= 0.35
35.75
25.00
00 0.9
37.10 8 1.0
37
1.0
J9:12
PADA
13.00
1:0
01 30
00.00-
0.6
294000
+ 2.0
13
1.0
80.00
12.19
- 0.0
39.58
20
43.00
1.1
13.90-
0.5
77
2.
43.14
1.7
45.29 0.
PC
BARUS
55.92
= 1.1
57.91
+ 2.
06.74
1.3
10.94
1.7
07 1.1
25.17 1.0
0.
18.35 =
52
24 1.0
41.72
0.7
51.25
1.6
5198
1.8
20.55 1.4
01.77
0
JUMB007
13.00
= 0.0
50.16
0.9
41.00
0.5
50.00
4.9
0.5
39/12
0.
MPOT
50.87
=: 1.1
00.77
2.6
43.00
2 0.6
00:00-0
3.0
50033 07
37.07
01.00
0.0
75.05
0.5
71:91
T1.00
71
70.06 1.3
75,30 1.2
70.70
79.56
76.46
75.08
0.6
70.13 4.7
$5.00
75.21
85.92
86,00 0.5
78102
52.05
76.20
50000
78
81.78 1.0
74
74.97
51
81.00 1.3
0812
68.00 0.0
20.24:1
09.23
08.05:10
03.58 0.9
PADA
60.27
50.35
TA.09
18:25:1
71,00 1.5
SAUX
67.02
0.0
08.11
56
03.00 0.6
22
80.00 0.8
70.00
70.31
T5.02
70.19
74.00
17.20
0.6
70.19
07.45
81.21
0.8
81.70
0.8
HP0T
07.68
0.6
70.02
80,00
0.6
5. 0832
46.03
=: 1.01
45.01
41.50 2: 1.0
43.75 2 0.00
06.15 2.0
43.00 01 0.2
47.20
0.9
PADA
59.00
=: 1.1
40.73
2 8.5
20.07
= 21.0
45.77
1.6
54.25
1.6
33.20
2.2
= 0.9
8AFM
10.73
0.1
18.76.
01
50.55
0.6
19.50
1.0
57.55
0.
50.65
0.5
51.88
0.6
BC
BASUS
51.81
2 0.5
65.65
27
65.30
3.5
66.35
1.0
88.10
0.
06.55
1.5
05.77
1.5
52.02
=: 1.0
55.04
1.2
49.61
0.6
b0.02
1.8
2.1
05.42
1.6
69.68
1.1
81.12
$1.81
45.12:00 0.5
69
1.7
$1.00 2.2
04.84
1.0
64.70
1.1
50.99
2: 1.5
$7.53 22 0.6
49.12 21 0.5
20.34
2: 1.2
1.7 $7.09
04.64
2
0.1
07.04
2 0.6
76.31
0.7
76-17
0.8
75.00
1.5
15.84
13.33 2.
75.98
25
77.51
PADA
78.85
0.3
16.00:1
1.7
01.07
1 11.3
90.92 0.4
80.13
80.8
066
78.23
2 0.5
76.23
1: 0.
25.65
0.5
0.5
60
76.41
0.6
77.29
0.5
BASUS
$1.85
22 4.1
$4.84
2
25.00
59.35
0.5
70.20
85.00 2:
86.88
0.2
77.66
2: 2.6
53.06 2:
73.64
0.4
$3.40
20 0.0
83 1.0
82,50 2 0.4
83.72
0.6
77.12
1.3
76.01
98.15
57
87.61
87.17
1.7
MPOT
77.61 0.3
15.17
75.59
63.14
77.05
62.00
86.52
1.2
62.09
0.2
17.51
Table 12: Average accuracy of different PDA methods based on different model selection strategies on the 12 tasks of Partial OFFICE-HOME. Average is done over three seeds (2020, 2021, 2022). For each method, we highlight the best and worst label-free model selection strategies in green and red, respectively.
20

---

Published in Transactions on Machine Learning Research (05/2023)
ALGORITHM
S2R
R25
Avg
S. ONLY
45.26
64.28
54.77
S. ONLY (Ours)
51.86
67.11
59.48
PADA 53.53
76.50
65.02
PADA (Ours)
49.34
59.81
54.57
SAFNÍ
67.65
SAFN (Ours)
56.88
68.
62.64
BA3UST
69.86
67.56
68.71
BA3US (Ours)
71.77
63.56
67.67
ARt AR (Ours)
85.30
74.82
80.06
76.33
71.36
73.85
JUMBOT 90.55
JUMBOT (Ours)
77.46
84.01
MPOT
MPOT (Ours) 87.23
86.67
86.95
Table 13: Comparison between reported (i) accuracies on partial VISDA from published methods with our implementation using the ORACLE model selection strategy. * denotes different bottleneck architectures.
DATASET
МETHOD
S-ACC
ENT
DEV
SND
1-SHOT
50-RND
100-RND
ORACLE
8. ONLY
60.38+0.5
60.73+0.2
00.22+0.3
59.55++0.3
58.92+0.4
60.28+0.4
60.34-0.4 0.4
61.87+0.3
PADA
63.08+0.3
59.74+0.5
52.72+-2.8
62.36+0.4
62.00+0.5
63.82+0.4
63.22-0.1
63.72+003
6..09+002
61.37+0.3
62.03+00.4
62.59+0.1
49.30++0.7
62.00+0.2
62.36-0.2
63.30++0.2
OFFICE-HOME
BA3US
68.32-1.1
73.36-1066
62.25.17.1
75.37-10.8
65.5647.6
73.22-10.3
75.19=0.4
75.98
AR JUMBOT
65.68-0.3
70.5840.4
64.32.0.9
70.2510.2
70.56:0.7
70.26:10.2
70.34=0.2
72.73-1..3
62.89-0.2
74.61400.8
61.28:10.1
72.2910.2
74.95.10.1
64.95:10.3
75.74=0.3
77.1510.4
MPOT
66.24-0.1
64.4640.11
61.37:10.2
46.9240.4
68.2840.2
69.90=10.5
73.06=0.3
77.31+0.5
8. ONLY 35.15+2.44
55.24433.2
55.07+1.2
55.02+-299
55.72+2.2
57.90++11
58.16-0.6
59.48+0.4
47.4824.8
32.3244.9
43.4315.3
56.8311.0
53.1512..
55.6742.5
54.38-2.7
54.57-2.6
SAFN
58.20-1.7
42.83-1633
58.6241.3
44.8248.8
56.8942.1
57.90433.3
59.09-2.8
62.6441.5
VISDA
BAšUS
35.10-3.7
63.584144
58.4011.4
51.07-44.3
64.774144
66.66:12.4
67.44-1.2
67.6711.3
AR 66.68+1.0
64.27-33.6
67.20
55.6910.9
70.2941.7
71.9140.3
72.60=0.8
73.8540.9
JUMBOT
6..63+0.7
62.42++2.4
59.86++0.6
77.69++4.2
78.34+1.9
82.85+2.9
83.49-1.9
84.01+1.9
MPOT
70.02+2..0
74.64+44.4
61.62-+1.3
78.40+-3.9
70.9643.77
86.65+t5.1
86.69-5.1
86.95+5.00
Table 14: Task accuracy average over seeds 2020, 2021, 2022 Partial OFFICE-HOME and Partial VISDA for the PDA methods and model selection strategies. For each method, we highlight the best and worst label-free model selection strategies in green and red, respectively.
21


Reviews:
[{"review_id": "Kfy7ywQ4pm", "reviewer": "Reviewer_6NSE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There are some minor issues that should be addressed for better readability and soundness. \n\nIt’s misleading to claim that PDA methods decrease up to 30% percentages according to Table 1. Please revise. \n\nAlso in Table 1, besides of reporting the worst and best results on different random seed, please also report the average results with variances. \n\nIn the first paragraph, domain shift is clearly not only limited to different backgrounds or colours. Therefore, ‘such shift is referred to as domain shift in the literature’ is not accurate. \n\nat the end of first paragraph, ‘outlier source only label’ may not be accurate. I understand because source only samples may be regarded as outliers w.r.t target distribution. Please rewrite it to avoid confusion as normally outlier is used w.r.t training set. \n\nIn table 2 row Architecture, the caption should clarify what does the symbol between networks mean. A table should explain itself.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a comprehensive investigation to existing partial domain adaptation methods with various model selection strategies. It’s interesting to know that existing PDA methods failed at an UDA setting while with 100 labelled test samples can ensure an effective PDA.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WDnh7rQZ2h", "reviewer": "Reviewer_xx5V", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Readability: The manuscript is a good read - the writing is simple and clear and describes a spectrum of latest works in Partial DA and model selection strategies in Sec. 2 & 3.\n\n* Significance: This study is of practical significance since model selection in Unsupervised DA is an important problem, and not commonly addressed by prior works. This work will motivate the study of more accurate model selection strategies that do not use expensive target annotations.\n\n* Reproducibility: The BenchmarkPDA codebase is very useful for future research in PartialDA and encourages fair evaluation and reproducibility.\n\n* Insights: Currently, the main goal is to reimplement, benchmark, and provide empirical results on the stability and reproducibility of the existing Partial DA methods using various model selection techniques. Though this has limited contributions in terms of algorithmic novelty, it is an important investigation which I believe has value to the progress of the field.\n\nWeaknesses & scope for improvement:\n\n* Motivation for PDA: The motivation to restrict the study to only Partial DA methods is unclear. As such the presented hypotheses could very well be applicable to other scenarios such as Open-set [P1], or Universal [P2] DA. Could the authors clarify why Partial DA was specifically chosen?\n\n* Missing meta-insights: It would be great to understand if certain methods are more robust than others, and why. Here are some interesting questions which could lead to useful insights: \"What contributes to the stability of  a method - is it due to less number of hyperparameters, or data-augmentation, or is it simply dataset and task dependent?\", \"Are there certain kinds of Partial DA approaches that are more reliable?\", \"Can we calibrate the trade-off between the amount of labeled examples required, and the reliability of model selection?\", \"What should a researcher keep in mind while developing a new model selection technique?\". Building a narrative that answers meta-questions like these could strengthen the work.\n\n* Minor comments:\n\n  1) The second bullet point in Page 2 (“Only 1 pair…”) is not revisited later in the paper (did I miss it?). Also, point (iii) in Conclusion (“An ablation study…”) is a generic recommendation which is not talked about in the main text. The manuscript can be improved by providing a more comprehensive discussion on these points.\n\n  2) Please fix typos:\n\n    - Table 7 best and worst models are incorrectly highlighted: For S2R method SAFN, SND produces the worst model (not ENT), and for the method AR, ENT selects the best model.\n\n    - Supplementary Table 6: In Office-Home dataset, for JUMBOT, the 50-RND selects worse models (10% worse) than 1-SHOT and 100-RND which looks like a typo. Please double check the same.\n\n  3) \"7 different state-of-the-art Partial DA algorithms\": In my opinion, the source-only model should not be considered as a partial DA algorithm. Thus there are six PDA algorithms that are studied in this work (PADA, SAFN, BA3US, AR, JUMBOT, MPOT). I would suggest the authors to make appropriate changes to reflect the same.\n\nReferences:\n\n[P1] Cao et al., \"Separate to Adapt: Open Set Domain Adaptation via Progressive Separation\", CVPR 2019\n\n[P2] Yu et al., \"Universal Domain Adaptation\", CVPR 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies an important topic of model selection in Partial Domain Adaptation. In the current practice, there is no single established model selection technique and therefore each method adopts its own technique (often using target labels which violates the label-free assumption in Unsupervised Domain Adaptation).\nThe authors elucidate this challenge and develop a benchmarking framework - BenchmarkPDA with a focus on reproducibility and fair evaluation under a consistent setting. A rigorous analysis of several label-based (1-shot, 50-RND, 100-RND, ORACLE) and label-free (S-ACC, ENT, DEV, SND) model selection techniques is presented across various Partial DA methods, hyperparameters and seeds.\nThe key insight is that each DA method underperforms using label-free model selection techniques and labeled target data is necessary to select a model close to the oracle performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I found the paper interesting and addressing an important topic. This work can be further improved (please refer to the points in the Weaknesses section).", "broader_impact_concerns": "This work investigates the reproducibility and stability of the existing partial domain adaptation methods and therefore does not impose any significant ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FcOnoAW4td", "reviewer": "Reviewer_fsHm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper studies an important topic which often receives too little attention in papers presenting new methods for problem settings that involve generalization outside of the training distribution without labeled examples. As the authors point out, the unavailability of labeled examples from the target distribution also implies the unavailability of a validation set for model selection, therefore model selection becomes an open problem in this space too.\n- Thorough empirical investigation of a large number of methods and model selection approaches on two different datasets, with design choices made to enable fair comparisons and reproducible results\n- Interesting findings and recommendations for the (P)DA community\n- The paper is nicely written and easy to follow (see below for some small exceptions)\n\nWeaknesses\n- Unclear (missing motivation) why the scope is limited to PDA in particular instead of DA more generally or other variants of DA\n- Missing formal problem description of DPA\n- Insufficient description of previous methods (both in DA/PDA methods as well as previous approaches to model selection considered)\n- Insufficient discussion of some design choices, as well as on the relationship between the findings of this work and findings of related papers (see below)\n- Some places where clarity can be improved (see below)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper performs a thorough experimental evaluation of methods for Partial Domain Adaptation (PDA) (a variant of Domain Adaptation (DA) where the target domain contains a strict subset of the classes appearing in the source domain, making the problem harder compared to standard DA). In particular, their evaluation is realistic in the sense that they relax the optimistic assumption made in previous work that a labeled set of examples are available from the target dataset for model selection / validation (as this assumption violates the definition of DA’s problem setup, that assumes only unlabeled examples from the target domain). They consider cases where a small(er) number of labeled examples are available from the target domain for validation (which might be viable in practice), as well as several more ‘legal’ variants for model selection without a labeled validation set, that were proposed in some previous work. Their thorough analysis surfaces interesting findings about the generalizability of model selection approaches, the degree of optimism that results from using an ‘oracle’ for model selection (that uses labeled target examples), the reliance of results on the random seed, and other findings in the context of specific PDA methods evaluated and their reliance on hyperparameters like architectural choices. Finally, the authors developed and released a framework in pytorch, BenchmarkPDA, for facilitating research in this direction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add a formal problem description of the problem setting of PDA\n- Add motivation for why the scope is limited to PDA in this work: why PDA in particular instead of other variants of the problem, and why not DA more generally\n- Add motivations for why the particular PDA methods were chosen for evaluation.\n- It would be great to also add more detail about the PDA methods and the model selection methods evaluated coming from previous literature. While the high-level descriptions provided are nice, more details into each approach would be useful to help the reader engage better with the space of solutions which in turn would aid in understanding and interpreting the findings from this evaluation\n- Add discussion on whether the findings from this work are analogous to or contradict findings in previous studies of similar spirit (e.g. Gulrajani and Paz for domain generalization). It sounds like that study and problem setting is quite related. Though I understand domain generalization does not even assume unlabelled examples from the target domain, some model selection approaches can be applied to both, like using a subset of the source for validation and the oracle. How do results differ w.r.t these, for example, across the two studies?\n- Define ‘negative transfer’, as the term appears several times in the paper, e.g. in the context of explaining why PDA is harder than DA. I’ve seen this term used before in transfer learning / continual learning to describe the issue arising when solving an earlier problem makes a later problem harder to solve. But here, there is no sequential task solving, if I understand correctly: there is just 1 training phase where the model is trained on labeled source data and unlabeled target data simultaneously (though, it would be nice to have a formal problem description as mentioned above, to clarify this point).\n- In Section 4.2, the authors mention ‘... for each task of each dataset’. The term ‘task’ hadn’t been used up to this point (unless I missed it?) and is not clear what it means here (I can somewhat infer from context, but should be defined)\n- In the same section,  the authors say that the unsuccessful runs of the hyperparameter search pose a challenge for DEV, SND and ENT, but it is not clear why that is. Looking at the description of these methods in Section 2, it seems that there are some missing pieces in terms of understanding why this is an issue. Please elaborate.\n- Regarding hyperparameters, the authors say ‘we only consider the model at the end of training’. This confused me for two reasons: first, isn’t the ‘end of training’ also a hyperparameter in and of itself (i.e the number of steps)? If not, then what is the stopping criterion? Second, I’m a bit confused here because, earlier (Section 4.1, in the ‘Model Selection Strategies’ paragraph), the authors said ‘we use them [model selection approaches] both for hyper-parameter tuning as well selecting the best model along training’. This seems to contradict the statement of only considering the model at the end of training? It would be great to elaborate on what exactly is the procedure for choosing hyperparameters and a checkpoint\n- In Section 4.2, the authors also say: ‘Then, to obtain the results with our neural network architecture on all tasks of each dataset, we trained an additional …’. It wasn’t clear to me what this additional training phase is for. I thought that after model selection, the chosen model (i.e. set of hyperparameters and checkpoint) would be directly deployed on the test set? The ‘with our neural network architecture’ part makes it sound like a different architecture was used for tuning rather than testing? But that doesn’t sound right, so I think I’m misunderstanding the setup. Please explain.\n- Section 5 about reproducibility: by reading the discussion here I realized that the reproduced numbers are sometimes obtained with a different architecture compared to the original numbers compared against? I wouldn’t really call this a reproducibility trial then, as the architecture is an important component which is changed. For the sake of verifying reproducibility / correctness of implementation, it would make more sense to use the same setup (including the same architecture) as the one that was used to obtain the results we are trying to reproduce.\n- On a similar vein, it sounds like the authors have changed the architectures of some of the compared approaches, in order to have the same architecture used for all. While I do agree that it’s nice to eliminate that factor of variation, in order to directly compare methods to each other, it might be that the performance of some is underestimated due to it being strongly dependent on a particular architecture from which it has now been decoupled? To what extent do the authors think it’s a possibility? Perhaps for the purposes of this paper this is OK but it would be great to add some discussion about the pros and cons of this design choice.  \n- RE: PADA combined with SND ‘performs better than the oracle on average, although within the standard deviation’: my understanding is that for assessing statistical significance, one needs to look at overlap of confidence intervals, not standard deviations. I find the wording here confusing.\n- Table 7: state what is the relationship between this and Table 6 (e.g. in the caption). Currently the captions of these two tables are similar and it takes some scrolling up and down to find the differences.\n- Section 5.3, when discussing random seeds, please state what exactly these seeds control (e.g. is it just the order of mini batches, or something more? Is this method-specific?)\n- Minor: fix inconsistent notation, e.g. dev (in section 2) versus DEV elsewhere in the paper; analogously for ent, snd, etc\n\nAdditional comments and discussion\n- Table 4 is hard to read as it contains a large number of figures. Maybe a better presentation of these results would be through a (set of) barplots, where each one shows the difference between the published results and reproduced results?\n- Regarding experimental setup, as explained in Section 4.1 for the datasets, can the particular selection of the first X categories to serve as the partial target domain bias the results? That is, conceivably the PDA problem becomes easier or harder depending on which categories are chosen to be source-only (excluded from the target). Is there a rationale for choosing the first X? Has previous literature studied this?\n", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is more a survey/benchmark paper, so the comprehensiveness and elaboration are the crucial factors impacting the paper quality. In a nutshell, this paper has great work in achieving these. The outcome of this paper is a solid BenchmarkPDA for reproducibility, and insights about the robustness and stability of well-established partial domain adaptation methods, with particular insights into the validation and parameter tuning strategies. Three reviewers assessed this paper and the revisions, with active interactions with the authors through OpenReview. Authors managed to address all comments reasonably well, which leads to unanimous acceptance recommendations from the reviewers. The Associate Editor agrees that this is a good paper that is meaningful for the community, and thus accepts the paper as is.
Decision: Accept (poster)