A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Authors: Tiago Salvador
Kilian FATRAS
Ioannis Mitliagkas
Adam M Oberman
Venue: Submitted to ICLR 2023 (2023)
Keywords: Domain adaptation
benchmark
reproducible research
model selection

Abstract: Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain. Most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training. However, these strategies violate the main assumption in PDA: only unlabeled target domain samples are available. Moreover, there are also inconsistencies in the experimental settings - architecture, hyper-parameter tuning, number of runs - yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods with the different model selection strategies under a consistent evaluation protocol. We evaluate 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. Our two main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.

Under review as a conference paper at ICLR 2023
A REPRODUCIBLE AND REALISTIC EVALUATION OF PARTIAL DOMAIN ADAPTATION METHODS
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised Domain Adaptation (UDA) aims classifying unlabeled target im- ages leveraging source labeled ones. In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain. Most successful algorithms use model selection strategies that rely on target labels to find the best lypper-parametes and/or models along training. However, these strategies violate the main assumption in PDA: only unlabeled target domain samples are available. Moreover, there are also inconsisteneies the experimental settings architecture, hyper-parameter tuming, number of runs yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods with the different model selection strategies under consistent evaluation protocol. We evaluate 7 representative PDA algorithms on 2 different real-world datasets using different model selection strategies. Our main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points: (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.
INTRODUCTION
Domain adaptation. Deep neural networks highly successful in image recognition for in- distribution samples (He et al., 2016) with this success being intrinsically tied to the large number of labeled training data. However, they tend not generalize as well on images with different back- ground or colors not seen during training. Such shift in the samples is referred to as domain shift in the literature. Unfortunately, enriching the training set with new samples from different domains is challenging as labeling data is both an expensive and cone-conuuming task. Thus, researchers have focused on unsupervised domain adaptation (UDA) where we have access to unlabelled samples from different domain, known as the target domain. The purpose of UDA is to classify these un- labeled samples by leveraging the knowledge given by the labeled samples from the source domain (Pan & Yang, 2010; Patel et al., 2015). In the standard UDA problem, the source and target domains are assumed to share the same classes. In this paper. we consider a more challenging variant of the problem called partial domain adaptation (PDA): the classes in the target domain Y form subset of the classes in the source domain y/s (Cao et al., 201 2018), ie.., Vt C ).. The number of target classes unknown as we do not have access the labels. The extra source classes, not present in the target domain, make the PDA problem more difficult: simply aligning the source and target domains forces a negative transfer where target samples are matched outlier source-only labels.
Realistic evaluations. Most recent PDA methods report an increase of the target accuracy up to 15 percentage points on average when compared to the baseline approach that uses only source domain samples. While these successes constitute important breakthroughs in the DA research literature, target labels are used for model selection, violating the main UDA assumption. In their absence, the effectiveness of PDA methods remains unclear and model selection constitutes yet to be solved problem as we show in this work. Moreover, the hyper-parameter tuning either unknown or lacks details and sometimes requires labeled target data, which makes it challenging to apply PDA methods to new datasets. Recent work has highlighted the importance of model selection in the presence of domain shift. Guirajani & Lopez-Paz (2021) showed that when evaluating domain generalization (DG) algorithms, whose goal is to generalize to a completely unseen domain, in consistent and realistic setting method outperforms the baseline ERM method by more than

---

Under review as a conference paper at ICLR 2023
DATASET
Model Solection
8. ONLY
PADA
SAPN
BASUS
AR
JUMBOT
MPOT
Wonst (wlo larget labels)
59.55 (-2.31)
52.72 (-11.00)
61.37 (-1.93)
62.25
(-13.73)
(-8.42)
61,28 (-15.87)
46.92 (-30.38)
OFFICE- HOME
Best (w/o target lahels)
60.73 (.1.1)
63.08 (-0.64)
(.0.7))
75.37 (.0.6l)
70.58 (-2.16)
74.61 (-2.54)
66.24 (-11.7))
GRACLE
61.87
63.72
75.98
77.15
77.31
Wonst (wio larget labels) Best (wil target labelis)
55.02 (-1.66
32.32
(-22.26)
42.83 (-19.11)
51.07 (-16.60)
55.69 (-1.15)
59.86 (-21.15)
61.62
(-25.33)
(-4.24)
56.83
(:4.02)
(.2.09)
67.20
(-6.31)
78.40 (.8.44)
GRACIE
39.48
34,37
62.64
67.67
73.85
84.01
86.95
Table 1: Task accuracy average computed over three different seeds (2020, 2021, 2022) Partial OFFICE-HOME and Partial-VISDA. For each dataset and PDA method, we display the results of the worst and best performing model selection that do not use target labels as well as the OR model selection strategy. All results can be found in Table 6.
percentage point. They argue that DG methods without model selection strategy remain incomplete and should therefore be specified as part of the method. A similar recommendation was done by Saito et al. (2021) for domain adaptation.
PDA methods have been designed using target labels at test time to select the best models. Paral- lel work (Saito et al., 2021; You et al., 2019) on model selection strategies for domain adaptation claimed to select the best models without using target labels. However, a realistic empirical study of these strategies in PDA is still lacking. In this work, conduct extensive experiments to study the impact of model selection strategies on the performance of partial domain adaptation methods. We evaluate 7 different PDA methods over 7 different model selection strategies, 4 of which do not use target labels, and different datasets under the same experimental protocol for a fair comparison. We list below our major findings:
The accuracy attained by models selected without target labels can decrease up to 30 percentage points compared to the one reported using target labels (See Table 1 for a summary of results).
Only pair of PDA methods and target label-free model selection strategies achieve comparable accuracies to when target labels are used, while still improving over source only baseline.
Random seed plays an important role in the selection of hyper-parameters, Selected parameters are not stable across different seeds and the standard deviation between accuracies on the same task can be up to 8.4% even when relying target labels for model selection.
Under a more realistic scenario where some target labels are available, 100 random samples is enough to see only drop of 1 percentage point in accuracy (when compared to using all target samples). However, the extreme case of using only one labeled target sample per class leads to significant drop in performance.
Outline. In Section 2, we provide an overview of the different model selection strategies considered in this work. Then in Section 3, we discuss the PDA methods that we consider. In Section 4 we describe the training procedures, hyper-parameter tuning and evaluation protocols used to evaluate all methods fairly. In Section 5, we discuss the results of the different benchmarked methods and the performance of the different model selection strategies. Finally in Section 6, we give some recommendations for future work in partial domain adaptation.
2 MODEL SELECTION STRATEGIES: AN OVERVIEW
Model selection (choosing hyper- paper-parameters, training checkpoints, neural network architectures) is a crucial part of training neural networks. In the supervised learning setting, a validation set is used estimate the model's accuracy. However, in UDA such approach is not possible we have unlabeled target samples. Several strategies have been designed to address this issue. Below, we discuss the ones used in this work.
Source Accuracy (S-ACC). Ganin & Lempitsky (2015) used the accuracy estimated on a small validation set from the source domain to perform the model selection. While the source and target accuracies are related, there are no theoretical guarantees. You et al. (2019) showed that when the domain gap large this approach fails to select competitive models.
Deep Embedded Validation (DEV). Sugiyama et al. (2007) and Long et al. (2018) perform model selection through Importance- enportance-Weighted Weighted Cross- Validation (IWCV). Under the assumption that the
2

---

Under review as a conference paper at ICLR 2023
source and target domain follow covariate shift, the target risk can be estimated from the source risk through importance weights that give increased importance to source samples that are closer to target samples. These importance weights correspond to the ratio of the target and source densities and are estimated using Gaussian kernels. Recently, You et al. (2019) proposed an improved variant, Deep Emhedded Validation (DEV), that controls the variance of the estimator and estimates the importance weights with discriminative model that distinguish source samples from target samples leading to a more stable and effective method.
Entropy (ENT). While minimizing the entropy of the target samples has been used in domain adaptation to improve accuracy by promoting tighter clusters, Morerio al. (2018) showed that can also be used for model selection. The intuition is that lower entropy model corresponds to a highly confident model with discriminative target features and therefore reliable predictions.
Soft Neighborhood Density (SND). Saito et al. (2021) argue that a good UDA model will have a cluster structure where nearby target samples are in the same class. They claim that entropy is not able to capture this property and propose the Soft Neighborhood Density (SND) score to address it.
Target Accuracy (ORACLE). We consider as well the target accuracy on all target samples. While we emphasize once again its use is not realistic in unsupervised domain adaptation (hence why we will refer to as OR ACI..,,, it has nonetheless been used to report the best accuracy achieved by the model along training in several previous works (Cao et al., 2018; Xu et al., 2019; Jian et al., 2020; Gu et al., 2021; Nguyen et al., 2022). Here, we use it as an upper bound for all the other model selection strategies and to check the reproducibility of previous works.
Small Labeled Target Set (1-SHOT and 100-RND). For real-world applications in an industry setting, it is unlikely that model will be deployed without the very least of an estimate of its performance for which target labels are required. Therefore, one can imagine a situation where a PDA method is used and small set of target samples is available. Thus, we will compute the target accuracy with labeled sample per class (I-SHOT) and 100 random labeled target samples (100- RND) model selection strategies. One could argue that the 100 random samples could have been used in the training with semi-supervised domain adaptation methods. However, note that we do not know how many classes we have on the target domain it is hard to form a split when we have uncertainty of classes. For instance, 100-RND represents possibly less than 2 samples per class for one of our real-world dataset, as we do not know the number of classes, making a potential split between train and validation target sets not possible.
3 PARTIAL DOMAIN ADAPTATION METHODS
In this section, we give brief description of the PDA methods considered in our study, They can be grouped into two families: adversarial training and divergence minimization.
Adversarial training. To solve the UDA problem, Ganin et al. (2016) aligned the source and target domains with the help of a domain discriminator trained adversarially to be able to distinguish the samples from the two domains. However, when applied to the PDA problem this strategy leads to negative transfer and the model performs worse than a model trained only on source data. Cao et al. (2018) proposed PADA that introduces PDA specific solution to adversarial domain adaptation: the contribution of the source-only class samples to the training of both the source classifier and the domain adversarial network is decreased. This is achieved through class weights that calculated by simply averaging the classifier prediction on all target samples. As the source-only classes should not be predicted in the target domain, they should have lower weights. More recently, Jian et al. (2020) proposed BA3US which augments the target mini-batch with source samples to transform the PDA problem into vanilla DA problem. In addition, an adaptive weighted complement entropy objective is used to encourage incorrect classes to have uniform and low prediction scores.
Divergence minimization. Another standard direction to align the source and target distributions in the feature space of a neural network is to minimize a given divergence between distributions of domains. Xu et al. (2019) empirically found that target samples have low feature norm compared to source samples. Based on this insight, they proposed SAFN which progressively adapts the feature norms of the two domains by minimizing the Maximum Mean Feature Norm Discrepancy (Gretton et al., 2012). Other approaches are based on optimal transport (OT) (Bhushan Damodaran et al., 2018) with bani-bacches (Peyré & Cuturi, 2019; Fatras et al., 2020; 2021b). For the PDA problem
3

---

Under review as a conference paper at ICLR 2023
PDA Methods
PADA, SAFN, BA3US AR, JUMBOT, MPOT
Model Selection Strategies
S-ACC, ENT, DEV, SND, 1-SHOT, 100-RND, ORACLE
Architecture
ResNet50 backbone @ linear bottleneck + linear classification head
Experimental protocol
3 seeds on the 12 tasks of OFFICE-IIOME and 2 tasks of VISDA
Table 2: Summary of our considered methods, model selection strategies, architecture and datasets.
Architecture (bottleneck)
Runs
Model Selection
Method
per task
Hyper-Parameeers
Along Training
PADA
Linear
1
IWCV (lacks details)
ORACLE
SAFN
Non-Linear
3
Unknown
ORACLE
BA3US
Linear
3
Unknown
ORACLE
AR
Non-Linear
1
IWCV (lacks details)
ORACLE
JUMBOT
Linear
ORACLE
FINAL
MPOT
Linear
3
Unknown
ORACLE
Table 3: Summary of the experimental protocol used for SOTA partial domain adaptation methods. We refer Appendix A.1 for additional details.
in specific, (Fatras et al., 2021a) a) developed JUMBOT, a mini-batch unbalanced optimal transport that learns joint distribution of the embedded samples and labels. The use of unbalanced OT is critical for the PDA problem as it allows to transport only portion of the mass limiting the negative transfer between distributions. Based on this work, (Nguyen al., 2022) investigated the partial OT variant (Chapel et al., 2020), a particular case of unbalanced OT, proposing M-POT. Finally, another line of work use the Senttrovvcc-Ruubeneee similarly to WGAN (Arjovsky et al., 2017). This is precisely the work of Gu et al. (2021) that proposed, AR. In addition, source samples are reweighted in order to reduce the negative transfer from the source-only class samples. The Kantorovich Saatorovich-Rubensteein duality relies on one Lipschitz function which is approximated using adversarial training like the PDA methods described above.
duality of optimal transport perform the alignment
EXPERIMENTAL PROTOCOL
In this section, we discuss our choices regarding the training details, datasets and neural network architecture. We then discuss the hyper-parameter tuning used in this work. We summarize the PDA methods, model selection strategies and experimental protocol used in this work in Table 2. The main differences in the experimental protocol of the different published state-of-the the-t-rrrtttttt (SOTA) methods is summarized in Table 3. To perform our experiments we developed a PyTorch (Paszke al.. 2019) framework: BenchmarkPDA. We make it available for other researchers to and contribute with new algorithms and model selection strategies:
https: /a//nnnonnoouuuummmm 4open wwappen.sciencee http:eee.....nceee///eneanaaaaaa
7F73
It is the standard in the literature when proposing new method to report directly the results of its competitors from the original papers (Cao et al., 2018; Xu et al., 2019; Jian et al., 2020; Gu et al., 2021: Nguyen al., 2022). As a result some methods differ for instance in the neural network architecture implementation (AR (Gu et al., 2021), SAFN (Xu et al., 2019)) or evaluation protocol JUMBOT (Fatras al., 2021a) with other methods. These changes often contribute to an increased performance of the newly proposed method leaving previous methods at disadvantage. Therefore we chose to implement all methods with the same commonly used neural network architecture, optimizer, learning rate schedule and evaluation protocol. We discuss the details below.
4.1 METHODS, DATASETS, TRAINING AND EVALCATION DETAILS
Methods. We implemented 7 PDA methods by adapting the code from the Official GitHub reposi- tories of each method: Source Only, PADA (Cao et al., 2018), SAFN (Xu et al., 2019), BABUS (Jian et al., 2020), AR (Gu et al., 2021), JUMBOT (Fatras et al., 2021a), MPOT (Nguyen et al., 2022). We
4

---

Under review as a conference paper at ICLR 2023
METHOD
A2C
A2P
A2R
C2A
C2P
C2R
P2A
P2C
P2R
R2A
R2C
R2P
Avg
S. ONLY
46.33
67.51
75.87
59.14
$9,94
62.73
58.
41.79
74.88
67.40
48.18
74.17
61.35
ONLY (Ours)
45.43
68.91
79.53
55.59
57.42
65.23
59.32
40.80
75.80
69.88
47.20
77.31
61.87
PADA
51.95
67.00
78.74
52.16
53.78
59.03
$2.61
43.22
78.79
73.73
56.60
77.09
62.06
PADA (Ours)
50.53
67.45
80.14
57.30
54.47
64.55
61.07
40.94
79.55
73.09
54.63
80.93
63.72
SAENT*
58.93
76.25
81.42
70.43
72.97
77.78
72.36
55.34
80.40
75.81
60.42
79.92
71.84
SAFN* (Ours)
59.98
79.85
85.18
72.02
73.73
78.54
76.09
59.32
83.25
80.04
64.20
84.44
74.72
SAFN (Ours)
49.57
68.55
78.26
57.91
59.29
66.81
59.87
45.29
75.98
69.08
51.68
77.29
63.30
BABUST
60.62
83.16
88.39
71.75
72.79
83,40
75.45
61.59
86.53
79.25
62.80
86.05
75.98
BABUS (Ours)
63.26
82.75
89.16
69.91
71.93
77.58
75.73
59.94
86.89
80.93
66.77
86.93
75.98
ART-
62.13
79.22
89.12
73.92
75.57
84.37
78.42
61.91
87.85
82.19
65.37
85.27
77.11
AR* (Ours)
62.75
81.55
89.07
71.63
73.41
82.94
75.88
61.03
85.70
79.86
62.93
85.30
76.00
AR (Ours)
57.33
79.61
86.31
69.45
71.88
79.94
70.28
$3.57
83.78
77.26
59.68
83.72
72.73
JUMBOT!
62.70
77.50
84,40
76.00
73.30
80.50
74.70
60.80
85.10
80.20
66.50
83,90
75.47
JUMBOT (Ours)
61.87
78.19|
88. 11
77.69
76.75
84.15
76.83
63.72
84,80
81.79
64.70
87.17
77.15
MPOT!
64.60
80.62
87.
76.43
77.61
83,58
77.07
63.74
87.63
81.42
68.
87
77.98 77.31
MPOT (Ours)
64.48
80.88
86.78
76.22
77.95
82.59
75.18
64.60
84.87
80.59
67.04
86.52
Table 4:
Comparison between reported (f) accuracies on partial OFFICE-HOME from published
methods with our implementation using the ORACLE model selection strategy. denotes differ- ent bottleneck architectures.
provide the links to the different official repositories in Appendix A. A comparison with previous reported results can be found in Table 4 and we postpone the discussion to Section 5.
Datasets. We consider two standard real-world datasets used in DA. Our first dataset is OFFICE- HOME (Venkateswara et al., 2017). It is a difficult dataset for unsupervised domain adaptation (UDA), it has 15,500 images from four different domains: Art (A), Clipart (C), Product (P) and Real- World (R). For each domain, the dataset contains images of 65 object categories that are common in office and home scenarios. For the partial OFFICE-HOME setting, we follow Cao et al. (2018) and select the first 25 categories (in alphabetic order) in each domain as a partial target domain. We evaluate all methods in all 12 adaptation scenarios. VISDA (Peng et al., 2017) is large- dataset for UDA. It has 152,397 synthetic images as source domain and 55,388 real-world images as target domain, where 12 object categories are shared by these two domains. For the partial VisDA setting, we follow Cao et al. (201 8) and select the first 6 categories, taken in alphabetic order, in each domain as a partial target domain. We evaluate the models in the two possible scenarios. We highlight that we are the first investigate the performance of JUMBOT and MPOT on partial VISDA.
Model Selection Strategies We consider the 7 different strategies for model selection described in Section 2: S-ACC, DEV, ENT, SND, ORACLE, 1-SHOO, 100-RND. We use them both for hyper- parameter tuning as well selecting the best model along training. Since S-ACC, DEV and SND require source validation set, we divide the source samples into training subset (80%) and validation subset (20%). Regardless of the model selection strategy used, all methods are trained using the source training subset. This is in contrast with previous work that uses all source samples, but necessary to ensure a fair comparison of the model selection strategies. We refer to Appendix A.2 for additional details.
Architecture. Our network is composed of a feature extractor with a linear classification layer on top of it. The feature extractor is ResNet50 (He et al., 2016), pre-trained on ImageNet (Deng et al., 2009), with its last linear layer removed and replaced by a linear bottleneck layer of dimension 256.
O Optimizer. We use the SGD (Robbins & Monro, 1951) algorithm with momentum of 0.9, a weight decay of 54 and Nesterov acceleration. As the bottleneck and classifier layers are randomly initialized, we set their learning rates to be 10 times that of the pre-trained ResNet50 backbone. We schedule the learning rate with strategy similar to the one in (Ganin et al., 2016): Xp = [I+aa)-v, O.75. While this schedule is slightly
where i is the current iteration, Xo 0.001, 7 0.001, different than the one reported in previous work, it is the one implemented in the different official code implementations. We elaborate in the Appendix A.3 on the differences and provide additional details. Finally, as for the mini-batch size. JUMBOT and M-POT were designed with a stratified
5

---

Under review as a conference paper at ICLR 2023
Dataset
Variant
BA3US
JUMBOT
MPOT
SAFN
ENT
DEV
SND
ENT
DEV
SND
ENT
DEV
SND
ENT
DEV
SND
Naive
52.60
63.10
44,48
$2.30
26.75
17.67
49.01
16.72
30,63
32.12
49.67
5,01
OFFICE-HOME
Heuristic
58.45
63.10
60.96
56.24
45.79
55.16
49.01
45.61
30.63
46.27
49.67
49.67
Naive
39.06
36.99
1.14
35.89
54.53
11.99
75.04
55.33
36.11
52.82
53.26
0.83
VISDA
Heuristic
67.50 34.94
38.76
47.23
54.53
66.42
75.04
55.33
85.36
52.82
53.26 52.82
Table 5: Comparison between the naive model selection strategy and our heuristic approach. Accu- racy on AC task for OFFICE-HOME and SR task for VISDA. Best results in bold.
sampling, i.e., a balanced source mini-batch with the same number of samples per class. This allows to reduce the negative transfer between domains and is crucial to their success. On the other hand, was shown that for some methods (e.g. BA3US) using larger mini-batch, than what was reported, leads to a decreased performance (Fatras et al., 2021a). As a result, we used the default mini-batch strategies for each method. JUMBOT and M-POT use stratified mini-batches of size 65 for OFFICE- HOME and 36 for VISDA. All other methods use a standard random uniform sampling strategy with a mini-batch size of 36.
For the hyper- paramettrsttes chosen with each model selection strategy, we run the methods for each task times, each with different seed (2020, 2021, 2022). We tried to control for the randomness across methods by setting the seeds at the beginning of training. Interestingly, as we discuss in more detail Section 5, some methods demonstrated a non-negligible variance across the different seeds showing that some hyper-parameters and methods are not robust to randomness.
Evaluation Protocol.
4.2
TUNING
Previous works (Gulrajani & Lopez-Paz, 2021; Musgrave et al., 2021; 2022) perform random searches with the same number of runs for each method. In contrast, we perform hyper-parameter grid searches for each method. As a result, the prrer-parameter tuning budgets differs across the methods depending on the number of hyper- soper-parameters and the chosen grid. While one can argue this leads to an unfair comparison of the methods, in practice in most real-world applications one will be interested in using the best method and our approach will capture precisely that.
The hyper-panne parameter tuning needs to be performed for each task of each dataset, but that would require a significant computational resources without a clear added benefit. Instead for each dataset, we perform the hyper-parameter tuning This same strategy was adopted in (Fatras et al., 2021a) and the hypee-parameters were found to generalize to the remaining tasks in the dataset. We conjecture that this may be due to the fact that information regarding the number of target only classes is implicitly hidden in the papper-prramterss.. See Appendix A.4 for more details regarding the hyper-parameters.
single task: A2C for OFFICE-HOME and S2R for VISDA.
Several runs in our syper-parameter search for JUMBOT, M-POT and BA3US were unsuccessful with the optimization reaching its end without the model being trained at all. This poses a challenge to DEV, SND and ENT and its one of the failures modes accounted for in (Saito et al., 2021). Follow- ing their recommendations, for JUMBOT, M-POT and BA3US, before applying the model selection strategy, we discard models whose source domain accuracy is below a certain threshold thr, which set with the heuristic as thr 0.9 Acc. Here Ace denotes the source domain accuracy of the Source-Only model. In our experiments, this leads to select models whom the source accuracy is at least of thr = 69.099999 % for the A2C task on OFFICE-HOME and thr = 59.83% for the S2R task on VISDA. We choose this heuristic because the ablation study of some methods showed that doing the adaptation decreased slightly the source accuracy (Bhushan Damodaran et al., 2018). Table 5 shows that our heuristic leads to improved results.
Lastly, when choosing the hyper-parameters we only consider the model at the end of training, discarding the intermediate checkpoint models in order to select hyper-parameen lead to overfitting at the end of training and better generalize to the other tasks. Following the above protocol, for each dataset we trained 468 models in total in order find the best bypee-parameters. Then, to obtain the results with our neural network architecture on all tasks of each dataset, we trained an additional /224 models for OFFICE-HOWE and /56 models for VISDA. We additionally
which do not

---

Under review as a conference paper at ICLR 2023
DATASET
METHOD
S-ACC
ENT
DEV
SND
1-SHOT
100-RND
ORACLE
S. ONLY
60,38=0,5
60.73=0.2
00.22-0..3
59.55=0.33
58.9240.4
60.34
61,87-0.3
PADA
63.0810.3
59.74.0.55
52.72.1..8
62.36.10.4
62.00.10.5
63.22.10.1
63.72.10.3
SAFN
62.09+0.2
61.37=0033
62.03+0.4
62.59:+0.1
49.30++0.7
62.36++0.2
63.30++0.2
OFRICEHHME
RA3LS
68.32++1.
73.36+0.6
62.25+7.1
75.37+0.8
67.
75.19+0.4
05.
65.68=0.3
70.58=0.4
64.32+0.9
70.25=0022
70.56+#0.7
70.34:0.2
72.73+0.3
JUMBOT
62.89+0.2
74.61=0.8
61.28+00.1
72.29+002
74.95+0.1
75.74+0.3
77.15+0.4
MPOT
66.2440.1
64.46=0.1
61.37:0.22
46.92:00.4
68.28:00.2
73.06100.3
77.31
S. ONLY
55.15+2.4
55,24
55.07=1.2
55.02:12.9
55.72::2.2
58.16:10.6
59.48:10.4
PADA
47,48.14.8
32.32.149
53.43..5.3
56.83.11.0
53.15.12.9
54.38.12.7
54.57.12.6
SAFN
58.20+1.7
42.83=6633
58.62+1.3
44.82+88.8
56.89:+2.1
59.09+2288
62.64+1.5
RA3US
55.10+3.7
65.58+1.4
18,40+1.4
51.07+4.33
1.
67.44+1.2
67.67+1.
66.68+10
64.27=3.6
67.20=1.5
55.69=0.9
70.29:1.77
72.60:
73.85=0.9
JUMBOT
60.63+0.7
62.42-2.4
59.86=0.6
77.69+4.2
78.34+1.9
13,49+119
84.01=1.9
MPOT
70.02
74.6414.4
61.62.1..3
78.4013.9
86.6915.11 70.9611..7
86.9515.0
Table 6: Task accuracy average over seeds 2020, 2021, 2022 on Partial OFFICE-HOME and Partial VISDA for the PDA methods and model selection strategy. For each method, we highlight the best and worst label-free model selection strategies in green and red, respectively.
trained 231 models with the different neural network architectures for AR and SAFN. In total, 2547 models were trained make this study and we present the different results in the next section.
PARTIAL DOMAIN ADAPTATION EXPERIMENTS
We start the results section by discussing the differences between our reproduced results and the pub- lished results from the different PDA methods. Then, we compare the performance of the different model selection strategies. Finally, we discuss the sensitivity of methods to the random seed.
5.1 REPRODUCTBIILIT OF PREVIOUS RESULTS
We start by ensuring that our reimplementation of PDA methods done correctly by comparing our reproduced results with previously reported results in Table 4. As such the model selection strategy used is ORACLE. On OFFICE-HOME, both PADA and JUMBOT achieved higher average task accuracy (1.6 and 1.7 percentage points, respectively) in our reimplementation, while for BA3US and MPOT we recover the reported accuracy in their respective papers. However, we saw decrease in performance for both SAFN and AR of roughly 8 and 5 percentage points respectively. This is to be expected due to the differences in the neural network architectures. While we use linear bottleneck layer, SAFN uses a nonlinear bottleneck layer. As for AR, they make two significant changes: the linear classification head is replaced by a spherical logistic regression (SLR) layer (Gu et al., 2020) and the features are normalized (the 2-norm is set to dataset dependent value, another hyper- parameter that requires tuning) before feeding them to the classification head. While we account for the first change by comparing AR (w/ linear) results reported in (Gu et al., 2021), in neural network architecture we do not normalize the features. These changes, nonlinear bottleneck layer for SAFN and feature normalization for AR, significantly boost the performance of both methods. When now comparing our reimplementation with the same neural network architectures, our SAFN reimplementation achieves a higher average task accuracy by 3 percentage points, while our AR reimplementation is now only percentage points below. The fact that AR reported results are from only one run, while ours are averaged across distinct seeds, justifies the small remaining gap. Moreover, we report higher accuracy or on par on 4 tasks of the 12 tasks. Given all the above and further discussion of the VISDA dataset results in Appendix B, our reimplementations are trustworthy and give validity to the results we discuss in the next sections.
5.2 RESULTS FOR MODEL SELECTION STRATEGIES
Model Selection Strategies (w/ vs w/o target labels) All average accuracies on the OFFICE-HOME and VISDA datasets be found in Table 6. For all methods on OFFICE-HOME, we see that the results for model selection strategies which do not use target labels are below the results given by ORACLE. For some pairs, the drop of performance can be significant, leading some methods to

---

Under review as a conference paper at ICLR 2023
perform par with the S. ONLY method. That is the case on OFFICE HOME when DEV is paired with either BA3US, JUMBOT and MPOT. Even worse is MPOT with SND as the average accuracy is more than 10 percentage points below that of S. ONLY with any model selection strategy. Overall on OFFICE-HOME, except for MPOT, all methods when paired with either ENT or SND give results that are at most 2 percentage points below compared to when paired with ORACLE.
A similar situation can be seen over the VISDA dataset where the accuracy without target labels can be down 25 percentage points. Yet again, some model selection strategies can lead scores even worse than S. ONLY. That is the case for PADA, SAFN and BA3US. Contrary to OFFICE-IIOME, all model selection strategies without target labels lead to at least one method with results on par or worse in comparison to the S. ONLY method. Overall, no model selection strategy without target labels can lead to score on par to the ORACLE model selection strategy. Finally, PADA performs worse than S. ONLY for most model selection strategies, including the ones which use target labels. However, when combined with SND it performs better than with ORACLE on average, although still within the standard deviation. This is consequence of the random seed dependence mentioned before on VISDA: as the hypper-paaameters were chosen by performing just one run, we were simply "unlucky". In general, all of this confirms the standard assumption in the literature regarding the difficulty of the VISDA dataset.
Model Selection Strategies (w/ target labels) We recall that the ORACLE model selection strategy uses all the target samples to compute the accuracy while I-SHOT and 100-RND use only subsets: 1- and VISDA, respectively,
SHOT has only one sample per class for total of 25 and 6 on OFFICE-HOME while (00-RND has 100 random target samples. Our results show that using only 100 random target labeled samples is enough to reasonably approximate the target accuracy leading to only a small accuracy drop (one percentage point in almost all cases) for both datasets. Not surprisingly, the gap between the 1-SHOT and ORACLE model selection strategies is even bigger. leading in some instances to worse results than with a model selection strategy that uses target labels. This poor performance of the I-SHOT model selection strategy also highlights that semi-supervised domain adaptation (SSDA) methods are not straightforward alternative to the 100-RND model selection strategy. While one could argue that the target labels could be leveraged during training like in SSDA methods, one still needs labeled target data to perform model selection. However our results suggest that we would need at least 3 samples per class for SSDA methods. In addition, knowing that we have a certain number of labeled samples per class provides information regarding which classes are target only, one of the main assumptions in PDA. In that case, PDA methods could be tweaked. This warrants further study that we leave as future work. Finally, we have also investigated a smaller labeled target set of 50 random samples (50-RND) instead of 100 random samples. The accuracies of methods using 50-RND were not as good as when using 100-RND. All results of pairs of methods and 50-RND can be found in Appendix B. The smaller performance show that the size of the labeled target set is an important element and we suggest to use at least 100 random samples.
Model Selection Strategies (w/o target labels) Only the (BA3US, ENT) pair achieved average task accuracies within 3 percentage points of its ORACLE counterpart (i.e., (BA3US, ORACLE)), while still improving over S. ONLY model. Our experiments show that there is no model selection strategy which performs well for all methods. That is why to deploy models in a real-world scenario, we advise to test selected models on a small labeled target set (i.e., 100-RND)) to assess the performance of the models as model selection strategies without target labels can perform poorly.
Our conclusion is that the model selection for PDA methods is still an open problem. We conjecture that it is also the case for domain adaptation as the considered metrics were developed first for this setting. For future proposed methods, researchers should specify not only which model selection strategy should be used, but also which spprr-pareetter search grid should be considered, in order to deploy them in a real-world scenario.
5.3 RANDOM SEED DEPENDENCE
Ideally, PDA methods should be robust to the choice of random seed. This is of particular importance when performing tlype--pramett done (that was the case in our work as well). We investigate this robustness by averaging all the results presented over three different seeds (2020, 2021 and 2022) and reporting the standard
tuning since typically only one run per set of haper-parameters
8

---

Under review as a conference paper at ICLR 2023
TASK
METHOD
S-ACC
ENT
DEV
SND
1-SHOT
100-RND
ORACLE
S. ONLY
46.96 + 1.5
48.17
+ 3.9
49.00
0.9
48.17 + 3.9
49.43 + 0.8
50.01
+ 1.6
$1.86 + 1.4
PADA
44.56
5.9
40.83
1 11.3
41.04
1.3
56.14.99 9.7
52.94 1.3
49.34
18.
49.34
1 8,4
SAFN
52.04
29.86
16.7
22
28.46 16.5
3.3
47.83
56.88 + 2.1
S2R
BA3US
44.21
3.0
71.17 1.9
1.9 48.78
46.12 7.8
1.5 66.79
71.45 0.8
71.77
1.1
AR
68.39
1.3
75.28
1.
57.61
0.4
70.11
5.2
76.33 - 4.5
JUMBOT
55.23
1 2.3
56.25 2.1
54.35
2.0
75.23
81.27
6.9 71.33
89.94
1 1.1
90.55
1 0.5
MPOT
64.57 2.9
8..10
57.02 :
1.5
84,45 0.4
87.20
+ 2.3
87.23
+ 2.3
S. ONLY
63.34 + 3.4
62.32 + 2.7
61.13
3.3
61.88 + 2.3
62.00 + 3.9
66.30 + 2.0
67.11 + 2.1
PADA
50.39 1: 3.8
23.80
1.6
45.82
1.2
$7.53 1: 10.3
53.36 1.7
59.43 5.8
59.81
1 6.2
SAFN
64.37 0.7
55.80 5.2
64.82 0.5
3,.
1.
68.40
1.2
R2S
BA3US
65.99 = 4.6
59.99 1.3
68.01
+ 19
36.01:22
63.44 22.75
1.9
63.56 + 1.8
AR
64.97 0.8
53.26 9.7
3.5
2.1
70.11 7..46:4.7
71.36 5.5
JUMBOT
66.04 1 1.0
68.59 4.6
65.36 1 0.8
80.16 1.1
75.42
77.03
2.7
77.46
1 3,3
МPOТ
75.47 = 3.8
67.18 9.1 9.1
66.21 + 1.2
72.36 7.4
3.1
86.18
+ 8.1
86.67 + 7.8
S. ONLY
55.15 + 2.4
55.24 3.2
55.07 + 1.2
55.02 + 2.9
2.2
58.16
0.6
59.48 + 0.4
PADA
47.48 4.8
32.32 4.9
5.3
56,83 1.0
23.15
54.38
2.7
54.57 2.6
SAFN
58.20 1.7
42.83 6.3
58.62
1.3
8.8
59.09 2.1
228
62.64
Avg
BA3US
55.10 1 3.7
65.58 1
58,40
1:001000 1.4
51.07
4.3
64.77
67.44.10
67.67 1 1.3
AR
66.68
64.27 3.6
67.20
1.5
55.69
0.9
70.29 1.7
72.60
+
73.85
0.9
JUMBOT
60.63 0.7
62.42 244
59.86
0.6
77.69
1 4.2
78.34
1 1.9
83.49
1 1.9
84.01
1 1.9
MPOT
70.02 = 2.0
74.64
4.4
61.62 + 1.3
78.40 + 3.9
70.96 + 3.7
86.69
5.1
86.95 + 5.0
Table 7: Accuracy of different PDA methods based on different model selection strategies on the 2 Partial VISDA tasks. Average is done over three seeds (2020, 2021, 2022). For each method, we highlight the best and worst label-free model selection strategies in green and red, respectively.
deviations. This is in contrast with previous work where only single run is reported (Fatras et al., 2021a; Gu et al., 2021). Other works (Cao et al., 2018; Xu et al., 2019; Jian et al., 2020) that report standard deviations do not specify if the random seed is different across runs. Results for all tasks on VISDA datasct are in Table 7 and on OFFICE-HOME in Appendix B due to space constraints.
Our experiments show that some methods express non-negligible instabilities over randomness with respect to any model selection methods. This is particularly true for BA3US when paired with DEV and 1-SHOT as model selection strategies: there several tasks where the standard deviation is above 10%. While in this case this instability may stem from the poor performance of the model selection strategies, it is also visible when OR ACLE is the model selection strategy used. For instance, the M-POT has a standard deviation of 3.3% on the AP task of OFFICE-HOME which corresponds to a variance of 11%. On VISDA this instability and seed dependence is even larger.
CONCLUSION
In this paper, we investigated how model selection strategies affect the performance PDA meth- ods. We performed a quantitative study with seven PDA methods and seven model selection strate- gies on two real-word datasets. Based on our findings, we provide the following recommendations:
i) Target label samples should be used to test models before using them in real-world scenario. While this breaks the main PDA assumption, it is impossible to confidently deploy PDA models selected without the use of target labels. Indeed, model selection strategies without target labels lead to significant drop in performance in most cases in comparison to using small validation set. We argue that the cost of labeling it outweighs the uncertainty in current model selection strategies.
ti) The robustness of new PDA method to randomness should be tested over at least three different seeds. We suggest to use the seeds (2020, 2021, 2022) to allow for a fair comparison with our results.
iil An ablation study should be considered when a novel architecture is proposed to quantify the associated increase of performance.
As our work focus on a quantitative study of model selection methods and reproducibility of state- of-the-art partial domain adaptation methods, we do not see any potential ethical concern. Future work will investigate new model selection strategies which can achieve similar results as model selection strategies which use label target samples.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214-223, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:// proceedings www.mlr...... www.s.....s//iio....e a jovskyy1 html.
Bharath Bhushan Damodaran, Benjamin Kellenberger, Remi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In The European Conference on Computer Vision (ECCV), September 2018.
Zhangjie Cao,, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 135-150, 2018.
Laetitia Chapel, Mokhtar Z. Alaya, and Gilles Gasso. Partial optimal transport with applications on positive-unlabeled learning. In Advances Neural Information Processing Systems, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-25822..2 lece, 2009.
Kilian Fatras, Younes Zine, Rémi Flamary, Remi Gribonval, and Nicolas Courty. Learning with minibatch wasserstein asymptotic and gradient properties. In Silvia Chiappa and Roberto Calandra (eds.). Proceedings of the Twenty Third International Conference on Artificial Intel- ligence and Statisties, volume 108 of Proceedings of Machine Learning Research, pp. 2131- 2141, Online, 26-28 Aug 2020. PMLR. htL:tt/:// /ttoc/oooeeiieoee ngs. mlr press/v108/ fatras20a html.
Utt.htp::/procced
Kilian Fatras, Thibault Sejourne, Rémi Flamary, and Nicolas Courty. Unbalanced minibatch op- timal transport; applications to domain adaptation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pp. 3186-3197. PMLR, 18-24 Jul 2021a. URL mwr. www.mm.t........mmmmm.mmmmm.. peress/vi39// wat./aaaaaaaaa.aaaaaaaaaaaalaaaa
Kilian Fatras. Younes Zine, Szymon Majewski, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Minibatch optimal transport distances; analysis and applications, 2021b.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanis- las Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78): 1-8, 2021. URL http: jmlr org/ papppspppp//sae v22 20- 451 .tml.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei (cds.). Proceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1180-1189, Lille, France, 07-09 Jul 2015. PMLR. URL wwwphhtss////ceeeeeee ganin15. html.
ml mir.press/w37/
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1):009-22)) 2030, 2016.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. URL http:/ / jmlr ort paperssppr/a/aaae vv 3/ gretton12a. html.
Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo-label loss. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
10

---

Under review as a conference paper at ICLR 2023
Xiang Gu, Xi Yu, Yan Yang, Jian Sun, and Zongben Xu. Adversarial reweighting for partial domain adaptation. In Thirty-Fifh Conference on Neural Information Processing Systems, 2021. URL
https: https://ppeeeeeveeeeeee net/ fo
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021, URL https: /wwp///pweeeweeeeeeeeeeeee net forummrr id=1odKexDoRtt.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 770-778, 2016.
doi: 10. N0.1199/CVR.2201.990
Liang Jian, Wang Yunbo, Hu Dapeng, He Ran, and Feng Jiashi. balanced and uncertainty-aware approach for partial domain adaptation. In European Conference Computer Vision (ECCV), August 2020.
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018 URL htwps: http::::p///cooeiigggs//. www.iinnnnnnneeereeeeeeeeeeereie wttppppppe//1///iL.... (ew.b/15333355319883383
wwwpeaddddd-pppp..aeeeeoo pdf.
Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. In International Conference on Learning Representations, 2018. URL https: wttpee/eee..eeeeeeeeeeeeeeeeeeee /ttee//ffoormmm-dreeeeeeee.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Unsupervised domain adaptation: A reality check. arXiv preprint arXiv: Arxir-2111.1500 2021.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Benchmarking validation methods for unsu- pervised domain adaptation. arXiv preprint arXiv: Arxiv-2208.077 07360, 2022.
Khai Nguyen, Dang Nguyen, Tung Pham, and Nhat Ho. Improving mini-batch optimal transport via partial transportation. In Proceedings of the 39th International Conference Machine Learning, 2022.
Simno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 22(10))1345-1333 2010. doi: 10.1109/TKDE.2000.199
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp- 8035. Curran Associates, Inc., 2019. URL http:/ /wtppraapeeeee/////eeeeeeeeeeee. neuri iss. cc cc/paper/ 9015 1:15--ytorch--- an iom-mmmprrtivee style teeeehhe hwih--iighhhheeee performance deep- learning library. pdf.
Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine, 32(3):53 69, 2015. doi: 10. 1109/MSP:2014.34477
Xingchao Peng, Ben Usman, Necla Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. CoRR, abs/1710.06924, 2017.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends@ in Machine Learning. 2019.
H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statisties, 22:400-407, 1951.
11

---

Under review as a conference paper at ICLR 2023
Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune the right way: Unsupervised validation of domain adaptation via soft neighborhood density. arXiv preprint arXiv:2108 ar0hv:2008.00800 2021.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(35):985-1005, 2007. URL http: jmlr. org/ ptggpapers//// / sugiyamas a .tml.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In (IEEE) Conference Computer Vision and Puttern Recognition (CVPR), 2017.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive fea- ture norm approach for unsupervised domain adaptation. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selec- tion in deep unsupervised domain adaptation. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learming Research, pp. 7124-7133, PMLR, 09-15 Jun 2019. URL https:/ /proceedings..... www.ss...s/...................e. peress/99) v97 / youulllll/ 9a html.
12

---

Under review as a conference paper at ICLR 2023
A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Supplementary material
Outline. The supplementary material of this paper organized as follows:
In Section A, we give more details on our experimental protocol.
In Section we provide additional results from our experiments.
A ADDITIONAL DETAILS ON EXPERIMENTAL PROTOCOL
A.1 IMPLEMENTATIONS IN BENCHMARKPDA
In order to reimplement the different PDA methods, we adapted the code from the official repository associated with each of the paper. We list them in Table 8.
Method
Code Repository
PADA
htspa://gtthub.com wttp://com///////tteee
SAFN
com/ happ:///gttibb.eee/temeebeabbeee
BA3US
AR
JUMBOT
M-POT
Table 8: Office Github code repositories for the PDA methods considered in this work.
One of our main claims regarding previous work is the use of target labels to choose the best model along training. This be easily verified by inspecting the code. For PADA it can be on line 240 of the script "train pada.py". for BA3US in line 116 for the script "run partpallptpp'" for M-POT it can be seen line 164 of the file "run mOT.p"".pp" for SAFN it can be seen in the "eval.py" file and finally for AR in line 149 of the script "train.py".
For JUMBOT and M-POT which are based on optimal transport, we used the optimal transport solvers from (Flamary et al., 2021).
A.2 MODEL SELECTION
DEV requires learning a discriminative model to distinguish source samples from target samples. Its neural network architecture must be specified as well the training details. You et al. (2019) (DEV) use a multilayer perceptron, while Saito et al. (2021) (SND) use a Support Vector Machine in their reimplementation of DEV. We empirically observed the latter to yield more stable weights and so that was the one we used. In order to train the SVM discriminator, following (Saito et al., 2021), take 3000 feature embeddings from source samples used in training and 3000 random feature embeddings from target samples, both chosen randomly. We do a 80/20 split into training and test data. The SVM is trained with linear kernel for maximum of 4000 iterations. Of different SVM models trained with decay values spaced evenly on log space between 10-2 and 10 the one that leads the highest accuracy (in distinguishing source from target features) on the test data split is the chosen one.
As for SND, it also requires specifying temperature for temperature scaling component of the strategy. We used the default value of 0.05 that is suggested in (Saito et 2021).
Finally, we mention that the samples used for 100-RND were randomly selected and their list is made available together with the code. As for the samples used for 1-SHOT, they are the same as the ones used in semi-supervised domain adaptation.
13

---

Under review as a conference paper at ICLR 2023
Method
HP
Values
PADA
À
00.1, 0.5, 1.0, 5.0, 10.0]
Amor Aent
001, 0.5, 1,5, 10) [0.01, 0.05, 0.1, 0.5, 1J
BA3US
X
[0.005, 0.01, 0.05, 0.1, 0.5]
SAFN
A
[0.01, 1.0,
Po
[2.5, 5.0, 7.5, 10.0)
Aup
[5.0, 10.0] Aup [0.01, 1.0)
AR
Asow
Xent
[0.001, 0.01, 0.1]
[0.00001, 0.0001, 0.001, 0.01, 0.1]
JUMBOT
72
[0.1, 0.5, 1.]
7Js
[5, 10, 20]
E
[0.5, 1.0, 1.5]
1)1
[0.0001, 0.001, 0.01, 1.0, 5.0, 10.0) [0.1, 0.2, 0.4]
MPOT
7)2
m
Table 9: Srrper-Parameter values for each PDA method considered in the grid search.
A.3 OPTIMIZER
In general, all methods claim to adopt Nesterov's acceleration method as the optimization method with a momentum of 0.9 and setting the weight decay set to x 10 The learning rate follows the annealing strategy as in Ganin et al. (2016):
Up (1 op))
where p is the training progress linearly changing from 0 to 1, Ho 0.01 and 10 and 3 = 0.75.
However, inspecting the Official code repo for each PDA method, the actual learning schedule is given by
(1 (1 + ai)
where i is the iteration number in the training procedure, Only when the total number of iterations is 10000 do the learning rate schedules match. this work, we followed the latter since it is the one indeed used. For OFFICE-HOME all methods are trained for 5000 iterations, while for VISDA they are trained for 10000 iterations, with the exception of the s. ONLY which is trained for 1000 iterations on OFFICE-HOME and 5000 iterations on VISDA.
0.01 and a 0.001 and 3 = 0.75.
A.4 SUPER-PARAMETERR
In Table 9, we report the values used for each hyper-paramette in our grid search. We report in Table 10 the hyper-paramettes chosen by each model selection strategy for each method both datasets. In addition, for the reproducibility of AR with the proposed architecture in Gu et al. (2021).a a feature normalization layer is added in the bottleneck which requires specifying r, the value which the 2-norm is set. This hyper-parameter is therefore included in the hyper-paramete grid search with the possible values of {5. 10, 20) which are the different values used the experiments in (Gu al., 2021).
B ADDITIONAL DISCUSSION OF RESULTS
In this section, we provide additional results that we could not add to the main paper due to the space constraints.
14

---

Under review as a conference paper at ICLR 2023
Method
Dataset
HP [ORACLE]
1-SHOT ( 50-RND 100-RND) S-ACC]
ENT
SND
OFFICE-HOME
0.5
0.1
0.1
0.5
0.1
1.0
5.0
0.5
PADA
VISDA
À
0,5
1.0
10.0
0.5
1.0
0.5
5.0
0.1
0.005
0.005
0.01
0.005
0.01
0.005
0.005
OFFICE-HOMEE
Ar
0.1
0.01
0.01
0.01
0.01
0.1
0.1
0.1
SAFN
0.005
0.005
0.05
0.05
0.005
0.05
0.005
0.05
VISDA
Ar
0.1
0.01
0.01
0.01
0.01
0.01
0.01
0.01
Awoe
5.0
10.0
5.0
5.0
5.0
0.1
10.0
1.0
OFFICE-HOME
Aent
0.05
0.05
0.01
0.05
0.01
0.1
0.05
0.01
BA3US
Awce
1.0
1.0
0.1
1.0
5.0
1.0
5.0
5.0
VISDA
Aent
0.5
0.5
0.5
0.05
0.5
0.05
Po
2.5
2.5
5.0
5.0
2.5
5.0
7.5
10.0
Aup
5.0
5.0
10.0
5.0
5.0
10.0
10.0
10.0
OFFICE-HOMME
Atow
-5.0
-5.0
-1.00
-5.0
-5.0
-10.0
-10.0
-10.0
Aent
0.1
0.1
1.0
1.0
0.01
1.0
0.01
1.0
AR
Po
2.5
2.5
2.5
2.5
2.5
7.5
2.5
10.0
Aup
10.0
10.0
10.0
10.0
5.0
10.0
10.0
10.0
VISDA
Alow
-10.0
-10.0
-10.0
-10.0
-5.0
10.0
-10.0
-10.0
Aent
0.1
0.1
0.1
0.1
0.01
0.1
0.01
0.01
0.01
0.01
0.01
0.001
0.1
0.01
0.01
0.001
7
0.0001
0.0001
0.001
0.0001
0.01
le-05
0.01
le-05
OHFICE-HME
f)2
0.5
1.0
0.5
0.1
0.1
0.5
1.0
1.0
10.0
5.0
5.0
5.0
5.0
20.0
10.0
5.0
JUMBOT
0.01
0.01
0.01
0.01
0.001
0.01
0.001
0.01
T
0.001
0.001
0.001
0.001
0.01
le-05
0.01
0.0001
VISDA
7/2
1.0
1.0
0.5
1.0
0.1
0.5
1.0
1.0
7/3
5.0
5.0
5.0
5.0
10.0
5.0
20.0
5.0
0.5
0.5
1.0
0.5
1.0
1.5
1.0
1.5
Th
0.01
0.01
0.01
0.01
0.001
0.0001
1.0
0.01
OFFICE-HOME
f)2
10.0
1.0
1.0
1.0
10.0
0.3
0.1
0.1
0.2
0.3
0.4
0.2
0.4
MPOT
0.5
0.5
0.5
0.5
1.0
1.0
1.0
0.5
Th
0.01
0.001
0.01
0.01
0.001
0.0001
0.0001
0.01
VISDA
Y)2
1.0
1.0
1.0
1.0
10.0
1.0
10.0
0.3
0.1
0.3
0.3
0.2
0.4
0.2
0.3
Table Hyper-parameters selected for the different methods for each model selection strategy on both OFFICE-HOME and VISDA.
In Table 11, we show the accuracy per task on OFFICE-HIMME averaged over three different seeds (2020, 2021, 2022) for all pairs of methods and model selection strategies.
In Table 12, we compare previously reported results with ours on VISDA. While proposed methods reported results on OFFICE-HOME, only PADA and AR results are reported in the original papers for VISDA. Gu et al. (2021) AR) also report results for BA3US. Analysing the results, we see a percentage point decrease in average task accuracy for PADA, is significant seed dependence which we discuss in detail below. This is particularly important since Cao et al. (2018) (PADA) report results from a single run. Comparing our best seeds for PADA the SR and RS tasks, achieve 58.01 % and 67.9% accuracy versus reported 53.53% and 76.5%. Moreover, we point out that the official code repository for PADA does not include details to reproduce the VISDA experiments, it is possible that minor tweaks (e.g learning rate) are necessary. As for BA3US, our results are within the standard deviation being better on the SR task and worse the RS task. Finally for on OFFICE- HOME show, can be explained by the differences in the neural network architecture.
our experiments show that there
decrease in performance which, as the results
15

---

Under review as a conference paper at ICLR 2023
Mirrac| M
P2C
R2C
R2P
Avg
PADA
50.15 =
65.33
16.23
58,00
56.13
0.5
20.33
40 30
13
5.70
= 94
335
58.95 = 13
14.8 155-::5
63,03
54Th
65:22
71.02 0.
57.85 0.6
57.55 0.1
0650 00
58
42
86
04
3100
10
40.75 0.1
76.25 2.6
62.09
2:0
5-4CC
BA305
50.10
72.79
17.41
63.23
06.54
05.08 11
10.07
07:15
114
17.14
00.00 11
63
10
13
72.15
00.89 10
5007
53.07
78.50
61.02
17
70.40
53,99
77.01
06.07
02.00
76.05
15.07
17.38
10-74 105
540%
42.00 3/0
65.23
71:23
56.54
50.20 0.0
0.1
70.25 2:67
51.31 2.03
6NT
59.30
75.38
96,03
65.95
11.00
2.0
10.55
57
75.06 03:05:13
05.34
1.4 75.87:1
74.85
60.15
06.35 11
71.11 108
500m
07.91
75:48-11
00.54
1..%%
73
05-45-10
44.70
19:07
7107
52.77 2-
5.47%
18:17-000
10 0.5
17:43-1
16.75 0.3
16.17
50 0.0
75.05 0-0
60.03 0.4
D1V
41.57
50105
60000
56:54
58.39
66.11
66.33 53
16.47
16.33 16.36-000
0.5
76:00 0-01
5000
66.07
16.06
50.41
47.05
75 -04
09:91
10:13
11.15
23.48
1.00
13.94
10.33 L.3
19.57
69.15
17.06
57.01 0.
7o 0.5
03.55 03.59 0
180
02:21
01.29
10.00
1:00
75.37
00.001
14.20
5000
33.96
49:75-2
51.39
9.2.4
14:5000
66.02
41361
03.02
11:08
20.43
55.34
73.35
31.00
49:73
4.3
03.02
45.83
45.37
63.52
57.55
07.00
07
50007
94305
14.00 01.0
51.30-4
10.47
59.78
03
79 20
65:55-766
70.30
76.29
13.43
14.95 L11
53397
09:76
19034
00.34 114
00.45
10.31
10.000
57.30
77.05
18:26 112
11.25
09.76
13:00
00:31
5AIN
41.58
67.53
11.01
56.47
58.19
50.85 0.3
7o.41 0.53
62.35:001 0.
100.00 RM0
BA305
62.53 3.6
92.00
58.28
11.55
06.57
63.66
75.19
00:07
10:01
76.56
76.40
13.00
59.02
87.64 1.6
75.24 11.
61.34
1.2
75.50 50
83.00
01.5
72.45
1.0
00.77
75.41
57.67
22.00 1:5
13.05-1
13
5.0N2Y
45.43 0/3
63.01
79.53
55.59 0.07
57.42
0.0
59.32
40.00
003
15:20
09:05
2:00
77.31 0.1
51.87
2:03
03.07
5741 1.3
10.30 0.
63.13 112
ORACLE
BAJUS
63.26 1/0
81.75
0.5
58.08
00:31 200
11.33
75.75
59144
56.09
0.3
2000 10
06.17 = 15
50.05 2.2
75:05:201 2.03
71.38
0.
12:73
S00
61.08
1.2
80308
5.1
66.78
76 01
17.45
07.50
64,0
1.4
00.00
07.04
96
00.50
17.31 1.5
Table 11: Average accuracy of different PDA methods based on different model selection strategies on the 12 tasks of Partial OFFICE-HOME. Average is done over three seeds (2020, 2021, 2022). Best results in bold.
Finally in Table 13, we show all the average task accuracies from all pairs of methods and model selection strategies on the OFFICE-HOME and VISDA datasets including the 50-RND model selection strategy.
16

---

Under review as a conference paper at ICLR 2023
ALGORITHM
S2R
R2S
Avg
S. ONLY!
45.26
64.28
54.77
S. ONLY (Ours)
51.86
67.11
59,48
PADA
53.53
76.50
65.02
PADA (Ours)
49.34
59.81
54.57
SAFNT
67.65
SAFN (Ours)
56.88
68.40
62.64
BA3UST
69.86
67.56
68.71
BABUS (Ours)
71.77
63.56
67.67
ART AR (Ours)
85.30
74.82
80.06
76.33
71.36
73.85
JUMBOT JUMBOT (Ours)
90.55
77.46
84.01
MPOT
MPOT (Ours)
87.23
86.67
86.95
Table 12: Comparison between reported (f) accuracies on partial VISDA from published methods with our implementation using the ORACLE model selection strategy. denotes different bottleneck architectures.
DATASET
METROD
S-ACC
ENT
DEV
SND
1-SHOT
50-RND
100-xND
ORACLE
S. ONLY
60.38=0.5
60.73=0.2
60.22+0.3
59.55=0.3
58.92-0.4
60.28-1044
60.3440.4|
61.87-00.3
PADA
63.08:10.3
59.74
52.7242.8
62.36=0.4
62.0010.5
63.82:10.4
63.22410.1
63.72:0..3
SAFN
62.09:102
61.37
62.0310.4
62.5910.1
49.3010.7
62.0010.2
62.36.10.2
63.30:10.2
OPRCE-HOME
BA3US
68.32-1.11
73.3620.6
62.2547.1
75.
63.5617.6
73.22:10.3
75.19.10.4
75 75.98.10.3
AR
70.58.0.4
64.32...9
70.2510.2
70.56.10.7
70.26:10.2
70.34110.
72.73:10.3
JUMBOT
62.89-40.2
74.61 0.8
61.28+0.11
72.29-000
0.1
64.95
75.74-10.3
77.1540.4
MPOT
66.24:10.1
64.4620.1
61.37:0.2
46.92.104
68.28:10.2
69.90.0055
73.06.10.3
77.31.10.5
S. ONLY
$5.15++2.44
3.2
55.07 1.2
55.02+-2.3
55.72+-2.2
57.90+1.1
58.16
59.48
PADA
47.48:14.8
32.32-1.9
43.431.5.3
16.831100
53.19::2.99
55.67-12.5
54.38.12.77
$4.57:12.6
SAFN
58.20 1.7
42.83
58.62
4..82
1.11
57.9011.3
59.0912.8
62.64:11..5
VISDA
BAS
10:33.73.
65.58-11.4 1.4
58.40- 1.4
51.07-4.33
64.774-14
66.66-12.44
67.44-12
67.67-+1.3
6..68=1100
64, 144.7733.6
67.20=
55.69=009
7..
71.9110.3
72.60+0.8
73.85=00.9
JUMBOT
60.63-+0.7
62.42442.4
59.86+00.6
77.69+4.2
78.34 1.9
82.85+22.9
83.49+1..9
84.0141999
MPOT
70.02-42.0
74.64=4.4
61.62+1.3
38.40
70.96+3..7
86.65-5..1
86.69+5.1
86.95-5.00
Table 13: Task accuracy average for the different PDA methods and model selection strategy pairs on Partial Office-Home and Partial VisDA. The average is computed over three difference seeds (2020, 2021, 2022).
17


Reviews:
[{"review_id": "vGFW_Owd2S3", "reviewer": "Reviewer_ZmSX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Existing algorithms for the problem of PDA(Partial Domain Adaptation) are considered and evaluated. While most previous successes have relied on target labels during training(model selection strategy), the availability of target domain labels is unrealistic. \n\nWith a more realistic evaluation of the PDA methods, the authors claim that performance drops are significant without target labels. The finding urges the community to rethink the evaluation protocol for practical PDA methods. ", "summary_of_the_review": "While the paper is somewhat interesting, it does not suggest any interesting algorithm or an exciting finding. Most experiments are straightforward and there are not many insights the readers can learn from. \nThe paper is not clear and is difficult to comprehend. \nFor the aforementioned reasons(novelty, technical contribution, and clarity), I recommend rejecting it.", "strength_and_weaknesses": "As the study of PDA methods is relatively new to the community, the benchmark and evaluation protocol for the field still needs a careful study/investigation. The submitted paper has dealt with the problem of interest comprehensively and stated the PDA model selection checkpoints. This is an interesting and significant technical challenge that matters to the community. \n\nThe benchmark and relevant code are open-sourced for reproducibility. \n\nWhile the paper is well-written overall, some sentences are difficult to follow. Some examples are as follows: \np.2 protocal -> protocol, methods pairs -> method pairs, 100 random samples is -> 100 random samples are \np.3 that distinguish -> that distinguishes, a high confident model -> a highly confident model\np.4  transform ... in a vanilla -> transform ... into a vanilla, found than target -> found that target.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KfFelao4lzM", "reviewer": "Reviewer_PkyR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper raises an important question about reproducibility of results for partial domain adaptation. The authors show that the stopping criterion is key to the success of partial domain adaptation algorithms in the literature, and there can be massive variations for various stopping criterions commonly used in the literature. Some algorithms even produce wide variations in performance for different random seeds. ", "summary_of_the_review": "The paper raises an interesting question, presents various experiments to showcase the problems with the current methods. However, a simple strong baseline and a suggestion for a robust experimental protocol is missing, which should be the take-away from such works. ", "strength_and_weaknesses": "Strength\n- The paper have done considerable amount of experiments along various dimensions\n- The paper is well-written and easy to follow\n\nWeakness\n- While the paper does a lot of experiments, the paper does not establish a strong baseline. The main takeaway from the paper is that partial domain adaptation works in literature produce high variations in performance to stopping criterions and random seeds. \nThe expected takeaway from such papers should be rather a simple strong baseline which can be concluded from the experiments under a certain fair experimental protocol, which works well across stopping criterions and random seeds. This would then encourage researchers to follow that experimental protocol and report their results and compare against that baseline. \n\nThe paper is overall in a good direction and I highly encourage the authors to consider this to refine their work. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9AXxe-Lw-R", "reviewer": "Reviewer_unvz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a benchmark for partial domain adaptation (PDA) to evaluate the model selection strategies for different PDA approaches on two different real-world datasets. The experimental results show that 1) Target labels are critical for model selection strategies; 2) only one method and model selection pair perform well on both datasets; 3) The results of PDA methods are influenced by the rand seeds. The authors also give some recommendations for future works.", "summary_of_the_review": "The authors study an important issue in partial domain adaptation and conduct extensive experiments to benchmark different PDA methods and model selection strategies. Based on these experimental results, this paper shows some findings. However, this paper does not present solutions for this issue, and the findings are also obvious.", "strength_and_weaknesses": "Strength:\n1.\tThis paper studies an important domain adaptation problem, particularly for partial domain adaptation. Using the target label to select models violates the domain adaptation assumption. But there is no suitable and effective metric for model selection. This is why researchers still use target labels for model selection.\n2.\tThis paper evaluates 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. This benchmark will foster the research community.\n3.\tBased on the extensive experiments, the authors give some conclusions and suggestions for future works.\n\nWeakness:\n1.\tThis work raises an essential issue in partial domain adaptation and evaluates many PDA algorithms and model selection strategies. However, it does not present any solution to this problem.\n2.\tThe findings of the experiments are a bit trivial. No target label for model selection strategies will hurt the performance, and the random seed would influence the performance. They are commonsense in domain adaptation, even in deep learning.\n3.\tThe writing needs to improve. The Tables are referenced but always placed on different pages. For example, Table 2 is referred in page 4 but placed on page 3, making it hard to read. The paper also has many typos, e.g., ‘that’ instead of ‘than’ in section 3.\n4.\tMany abbreviations lack definition and cause confusion. ‘AR’ in Table 5 stands for domain adaptation tasks and algorithms.\n5.\tIn section 4.2, heuristic strategies for Hyper-parameter turning is not clearly described. And the author said, “we only consider the model at the end of training”, but should we use the model selection strategies?\n6.\tIn section 5.2, part of Model Selection Strategies, the authors give a conclusion that seems to be wrong “only the JUMBOT and SND pair performed reasonably well with respect to the JUMBOT and ORACLE pair on both datasets.” In Table 6, the JUMBOT and SND pair performs worse than the JUMBOT and ORACLE pair by a large margin. For instance, on OFFICE-HOME, the JUMBOT and SND pair reaches 72.29 accuracies, while the JUMBOT and ORACLE pair achieves 77.15 accuracies.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject