Can We Evaluate Domain Adaptation Models Without Target-Domain Labels?
Authors: Jianfei Yang
Hanjie Qian
Yuecong Xu
Kai Wang
Lihua Xie
Venue: ICLR 2024 poster (2024)
Keywords: domain adaptation; feature transferability;
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A metric to evaluate domain adaptation models in an unsupervised manner, contributing to model selection and tuning

Abstract: Unsupervised domain adaptation (UDA) involves adapting a model trained on a label-rich source domain to an unlabeled target domain. However, in real-world scenarios, the absence of target-domain labels makes it challenging to evaluate the performance of UDA models. Furthermore, prevailing UDA methods relying on adversarial training and self-training could lead to model degeneration and negative transfer, further exacerbating the evaluation problem. In this paper, we propose a novel metric called the Transfer Score to address these issues. The proposed metric enables the unsupervised evaluation of UDA models by assessing the spatial uniformity of the classifier via model parameters, as well as the transferability and discriminability of deep representations. Based on the metric, we achieve three novel objectives without target-domain labels: (1) selecting the best UDA method from a range of available options, (2) optimizing hyperparameters of UDA models to prevent model degeneration, and (3) identifying which checkpoint of UDA model performs optimally. Our work bridges the gap between data-level UDA research and practical UDA scenarios, enabling a realistic assessment of UDA model performance. We validate the effectiveness of our metric through extensive empirical studies on UDA datasets of different scales and imbalanced distributions. The results demonstrate that our metric robustly achieves the aforementioned goals.

Published as a conference paper at ICLR 2024
CAN WE EVALUATE DOMAIN ADAPTATION MODELS WITHOUT BARGET-DOMAIN LABELS?
Jianfei Yang'; Hanjie Qian'; Yuecong Xu², Kai Wang', Lihua Xie'.
Nanyang Technological University National University of Singapore
ABSTRACT
Unsupervised domain adaptation (UDA) involves adapting model trained on label-rich source domain to an unlabeled target domain. However, in real-world scenarios, the absence of target-domain labels makes it challenging evaluate the performance of UDA models. Furthermore, prevailing UDA methods relying on adversarial training and self- self-training could lead model degeneration and negative transfer, further exacerbating the evaluation problem. In this paper, we propose a novel metric called the Transfer Score address these issues. The proposed metric enables the unsupervised evaluation of UDA models by assessing the spatial uniformity of the classifier via model parameters, as well as the transferability and discriminability of deep representations. Based on the metric, we achieve three novel objectives without target-domain labels: (1) selecting the best UDA method from range of available options, (2) optimizing hyperparameters of UDA models to prevent model degeneration, and (3) identifying which checkpoint of UDA model performs optimally. Our work bridges the gap between data-level UDA research and practical UDA scenarios, enabling a realistic assessment of UDA model performance. We validate the effectiveness of our metric through extensive empirical studies on UDA datasets of different scales and imbalanced distributions. The results demonstrate that our metric robustly achieves the aforementioned three goals.
INTRODUCTION
Deep neural networks have made significant progress in wide range of machine learning tasks. However, training deep models typically requires large amounts of labeled data, which can be costly or difficult obtain in some cases. To overcome this challenge, unsupervised domain adaptation (UDA) has emerged as technique for transferring knowledge from a labeled source domain to an unlabeled target domain. For example, in autonomous driving, UDA enables deep segmentation model trained on data from normal weather conditions to adapt to rainy, hazy, and snowy weather conditions where the data distribution changes dramatically (Liu al., 2020).
Despite the improvement of performance realized by UDA models, the current evaluation process relies on target-domain labels that are usually unavailable in real-world scenarios. As a result, it can be difficult to determine the effectiveness of UDA methods and how well they perform in the target domain. Moreover, since adversarial training is widely used in domain adaptation (Wang & Deng, be prone unstable training processes and even negative transfer
2018), many UDA models the hyperparameters are not well selected (Wang et al., 2019a), which can further undermine the viability of UDA models in practice. Therefore, there is pressing need to develop an unsupervised evaluation method for UDA models.
To evaluate UDA models in an unsupervised manner, contemplate whether the domain discrepancy metric could be a good indicator of model performance. The maximum mean discrepancy (MMD) commonly used to measure the statistical difference in the Hilbert space between the source and target domains (Gretton et al., 2012). Proxy A-distance (PAD) is established on the statistical learning theory of UDA (Ben-David et al., 2010) and measures distribution discrepancy by training a binary
"Equal contribution jjanfec.yanng wwwng@@ttu.d..g.gc hanjic001 @ntu.cdu.sg).
Codes available at https: // seepyssal thu.. fo/Traas fersccreeeeerrreeeeeeee

---

Published as a conference paper at ICLR 2024
-0.2
1.6
Transfer SCOTE
ACC (96
-MDD
ACC (96)
0.7
ACC (%)
80
1.4
1.2
75
75
-2
-1.3
1.0
70
Correlation: 0.75
70
Correla tio::
-1.6
70
Correlation:
-0.8
20
60
60 Epoch
80
20
40
80
80
20
Epoch
Epoch
Figure 1: A preliminary experiment of a classic UDA method (DANN (Guan Liu, 2021)) on Office-31 31 A-W task. The correlation between existing domain discrepancy metries (MMD and Proxy A-distance) and the target-domain accuracy is not clear while our proposed Transfer Score could clearly indicate the target-domain accuracy with high correlation value (Pearson, 1895).
classifier on the two domains (Ben-David al., 2010). However, as shown Fig. 1, our preliminary results suggest that these metrics fail to provide an accurate evaluation of UDA model performance in the target domain, and they cannot indicate the negative transfer phenomenon (Wang et al., 2019a).
In this paper, we propose an unsupervised metric for UDA model evaluation and selection, addressing three crucial challenges in real-world UDA scenarios: (1) selecting the best UDA model from a range of UDA method candidates; (2) adjusting hyperparameters to prevent the negative transfer and achieve enhanced performance; and (3) identifying the optimal checkpoint (epoch) of model parameters to avoid overfitring. To this end, the Transfer Score is proposed to accurately indicate the UDA model's performance in the target domain. The TS metric evaluates UDA models from two perspectives. First, it evaluates the spatial uniformity of the model directly from model parameters to determine whether the classifier is biased or overfitted for all classes. Second, it evaluates the transferability and discriminability of deep representations after UDA by calculating the clustering tendency and the mutual information. We conduct extensive experiments on public datasets (Office-31, Office-Home, VisDA-17, and DomainNet) using five categories of UDA methods, and our results demonstrate that the proposed metric effectively resolves the aforementioned challenges. As far as we know, TS is the first work to study and achieve simultaneous unsupervised UDA model comparison and selection.
2 RELATED WORK
2.1 UNSUPERVISED DOMAIN ADAPTATION
Current UDA methods aim to extract common features across labeled source domains and unlabeled target domains, improving the transferability of representations and robustness of models while mitigating the burden of manual labeling (Long et al., 2016). Most UDA methods could be generally divided into two categories: i) adversarial-based methods (Ganin & Lempitsky, 2015; Jiang et al., 2020; Xu et al., 2021; Yang et al., 2020a; 2021b), where domain invariint features are extracted by an adversarial training where domain discriminator is trained against feature extractor (Huang et al., 2011; Ganin & Lempitsky, 2015); and ii) metric-based methods (Zhang et al., 2019; Yang et al., 2021a; Sun et al., 2016), which mitigate domain shifts across domains by applying metric learning approaches, minimizing metrics such MMD (Gretton et al., 2012; Long et al., 2015; 2016; Xu et al., 2022), and PAD (Han et al., 2022; Xie et al., 2022). So far, there have emerged various streams of UDA approaches including reconstruction-ben based methods (Yang et al., 2020b; Ych et al., 2021; Li et al., 2022), norm-based methods (Xu et al., 2019), and reweighing- methods (Jin et al., 2020; Wang et al., 2022; Xu al., 2023).
2.2 MODEL EVALUATION AND VALIDATION OF UDA
The current evaluation process of UDA methods not feasible real-world applications since it relies on target-domain labels that are unavailable in real-world scenarios. There exists various domain discrepancy metrics such as MMD (Gretton et al., 2012) and PAD (Ben-David et al., 2010) that measure the discrepancies between source and target distribution. However, they only represent the cross-domain distribution shift under a certain feature space that cannot be directly related to model performance and used for model evaluation and selection.
2

---

Published as a conference paper at ICLR 2024
Few researchers pay attention to UDA model evaluation without target-domain annotations. In particular, Stable Transfer (Agostinelli et al., 2022) aims to analyze the different transferability metrics (Tran et al., 2019; Nguyen et al., 2020; You et al., 2021; 2022; Ma et al., 2021) for model selection in transfer learning under a pre-training and fine-tuning mechanism where both the source similar and labeled. Yet, the analysis does not apply to the task of UDA where
and target datasets there is a significant domain shift while the target domain is unlabeled. Unsupervised validation methods aim to choose a better model by cross-validation or hyperparameter tuning. DEV (You et al., 2019) firstly estimates the target risk based on labeled validation sets and massive iterations, which still assumes the viability of partial target-domain labels and takes up huge computation costs. There are other methods focusing unsupervised validation including entropy-basee method (Morerio et al., 2017) 7), geometry- -oanetr--based method (Saito et al., 2021), and out-o--distribution method (Garg et al., 2022). They only focus model selection but have not explored how to choose better UDA model from various candidates. Whereas, our transfer score can perform UDA method comparison and model selection simultancously without the cumbersome iterative process.
3 PRELIMINARY
3.1 UNSUPERVISED DOMAIN ADAPTATION
In unsupervised domain adaptation, we assume that model is learned from a labeled source domain Ds ([=;.y4)><eNN)) and an unlabeled target domain Dr = (a/) The label space Y is a finite set (Y = =,,..... K) shared between both domains. Assume that the source domain and the target domain have different data distributions. covariate shift) (Ben-David et al., 2010) between Ds and Dr. The objective of UDA is to learn model ((.) = h g where denotes the classifier and g denotes the feature extractor, which can predict the label y given the target-domain input z!.
other words, there exists a domain shift (i.e.
3.2 CHALLENGE: CAN WE EVALUATE UDA MODELS WITHOUT SARGET-DOMANE LABELS?
Despite the significant progress made in UDA approaches (Wang & Deng, 2018), most existing methods still require access to target-domain labels for model selection and evaluation. This limitation poses challenge in real-world scenarios where obtaining target-domain labels is often infeasible. This issue hinders the practical applicability of UDA methods. Moreover, current UDA approaches heavily rely on adversarial training (Ganin et al., 2016) and self-training techniques (i.e., pseudo labels) (Kumar et al., 2020; Cao et al., 2023) that often yield unstable adaptation outcomes, further impeding the effectiveness of UDA in practical settings. Thus, there is pressing need to develop methods that enable unsupervised model evaluation in UDA without relying on target-domain labels.
To address this issue, one may leverage metrics that measure the distribution discrepancy between the source and target domains to indicate model performance, as these metrics represent feature transferability (Pan et al., 2011). We consider two common metrics for UDA evaluation: maximum mean discrepancy (MMD) (Gretton et al., 2007) and proxy A-distance (PAD) (Ben-David et al., 2010). MMD is statistical test that determines whether two distributions p and q are the same. MMD estimated by MMD? (p. q) [op E,, where (6-) maps the input to another space and H. denotes the Reproducing Kernel Hilbert Space (RKHS). PAD is a measure of domain disparity between two domains established by the statistical learning theory (Ben-David et al., 2010). PAD is defined d, 2(1 2e) where e is the error of a domain classifier, e.g. SVM. We performed a preliminary experiment using MMD and PAD to indicate the model's performance. As depicted in Fig. 1, we observed a negative correlation between the target-domain accuracy and both MMD (correlation value of 0.75) and PAD (corrllatio value of 0.65). However, neither of these metrics provides a clear indication of the target-domain accuracy that could help model selection. Consequently, they insufficient for evaluating and selecting UDA models in real-world scenarios.
This paper introduces a novel metric called the Transfer Score, which has a strong correlation with target-domain accuracy. It serves three primary objectives in real-world UDA scenarios: (1) selecting the most appropriate UDA method from a range of available options, (2) optimizing hyperparameters to achieve enhanced performance, and (3) identifying the epoch at which the adapted model performs optimally. As illustrated in Fig. 1, our proposed metric exhibits a significantly higher correlation value of 0.97 with target-domain accuracy, showcasing its efficacy in real-world UDA scenarios.
3

---

Published as a conference paper at ICLR 2024
4 AN UNSUPERVISED METRIC: TRANSFER SCORE
As an unsupervised metric for UDA evaluation, the Transfer Score (TS) relies on the evaluation of model parameters and feature representations, which readily available in real-world UDA scenarios.
4.1 MEASURING UNIFORMITY FOR CLASSIFIER BIAS
Model parameters encapsulate the intrinsic characteristics of a deep model, as they are independent of the data. However, understanding the feature space solely based on these parameters is challenging, especially for complex feature extractors such as CNN (LeCun et al., 1998) and Transformer (Vaswani et al., 2017). Therefore, we defer the evaluation of the feature space to data-driven metric, discussed in Section 4.2. In this section, we focus on the transferability of the classifier via model parameters. In cross-domaile scenarios, biased predictions when applied to the target domain. This bias stems from an over-emphasis on certain classes in the source domain due their larger number of samples (Jamal et al., 2020). Consequently, we hypothesize that classifier with superior transferability should divide the feature space evenly and generate class-balanced prediction, rather than disproportionately emphasizing specific classes. Prior theoretical research has also demonstrated that evenly partitioning the feature space leads to improved model generalization ability (Wang al., 2020).
observe that a classifier trained the source domain often exhibits
Now, let's consider how to measure the uniformity of the feature space divided by classifier. We propose that the uniformity reflected by the consistency of the angles between the decision hyperplanes of the classifier. Let h(()) E Raxk denote K K-way classifier comprising K vectors [w1, W/2 Wk) which maps a d-dimensional feature to a prediction vector. When the feature space evenly partitioned by the classifier, the angles between any two vectors among the K vectors of the classifier are equal. We denote the ideal angle as 0g and define the angle matrix of the classifier as Σn IR where each entry Ozy is the angle between w, and wy- The uniformly distributed angle matrix is defined as E. IR K where each entry corresponds to the ideal angle 0x. Notably, the diagonal entries of both Σn and En are all set to
Definition The uniformity of fu(-) the square of the Frobenius norm of the difference matrix between Σn and Σ.:
K
K (0ij - 0x)
u = 2 1E En En
K(K -
j=1
where || || is the Frobenius norm of mutrix.
Intuitively, this metric can be interpreted as the mean squared error between all the cross-hyperplane angles and the ideal angle. The smaller value indicates more transferable classifier. We further provide a closed-form formula for computing the ideal angle 0x, which is proven in the Appendix. Theorem 1. When K S d d 1, the ideal angle Ox can calculated by
0x arccos (- K- (xx-))
1
2
In practical UDA applications, the feature dimension d is typically greater than the number of classes (Saenko et al., 2010; Venkateswara et al., 2017; Peng et al., 2019). Consequently, the uniformity metric 2 can be easily computed with Eq.
4.2 MEASURING FEATURE TRANSFERABILIT AND DISCRIMINABILITY
In addition to evaluating the transferability of the classifier, we also assess the transferability and discriminability of the feature space, they directly reflect the target-domain performance of UDA (Chen et al., 2019). As evaluating the feature space based on model parameters is challenging, propose to resort data-driven metrics: Hopkins statistic and mutual information.
Firstly, we propose to leverage the Hopkins statistic (Banerjee Dave, 2004) as a metric to measure the clustering tendency of the feature representation in the target domain. The Hopkins statistic,
4

---

Published as a conference paper at ICLR 2024
belonging to the family of sparse sampling tests, assesses whether the data is uniformly and randomly distributed and measures the clarity of the clusters. For a good UDA model, the feature space should exhibit distinct clusters for each class, indicating better transferability and discriminability (Deng et al., 2019; Li et al., 2021). Conversely, if the samples are randomly and uniformly distributed, achieving high classification accuracy becomes challenging. Therefore, we assume that the target- domain accuracy should be correlated with the Hopkins statistic. To compute the Hopkins statistic, we start by generating random sample set I comprising m < N, data points, sampled without replacement, from the feature embeddings of the target domain samples g(æ). Additionally, we generate a set U of m uniformly and randomly distributed data points. Next, we define two distance measures: ug which represents the distance of samples in U' from their nearest neighbors in R, and wi, which represents the distance of samples in R from their nearest neighbors in R. The Hopkins statistic is then defined as follows:
(3)
775
m 1==1
1=
where d denotes the dimension of the feature space.
The Hopkins statistic evaluates the distribution of samples within the feature space generated by the extractor g(-). However, it does not provide insights into how the classifier h(-) behaves within this feature space. As result, even in scenarios where all samples form single cluster or the classifier boundary intersects a densely populated region of samples, the Hopkins statistic can still yield a high value. To address this limitation, we propose the utilization of mutual information between the input and prediction in the target domain. By incorporating mutual information, we can discern the prediction confidence and diversity (class balance) of the UDA model, thereby reflecting the transferability of features in the target domain. The mutual information M is defined as:
M = (Exe,,,
- EgED, H (h(g(z))),
where H-) denotes the information entropy. As the mutual information value measures how well the model adheres to the cluster assumption. it serves as regularizer for domain adaptation in various works such as DIRT-T (Shu et al., 201 8), DINE (Liang et al., 2022), BETA (Yang et al., 2022a), and semi-supervised learning approaches (Grandvalet & Bengio, 2005).
4.3 TRANSFER SCORE
Consolidating the uniformity 4 which evaluates the classifier Iv(-) and the feature transferability metrics H, M which assess the feature space generated by the feature extractor g()), we introduce the Transfer Score to evaluate the target-domain model o (-) = f o g:
Definition 2. The Transfer Score is given by
M In K
T= u H
(5)
where K is the number of classes for the normalization purpose.
We aim to use a larger transfer score indicate better transferability. To this end, as the greater uniformity indicates a larger bias and lower transferability, we use the negative uniformity. In contrast, we use the Hopkins statistic and the absolute value of mutual information as they are positively correlated to the transferability. The mutual information is especially normalized because it does not have a fixed range as the uniformity and Hopkins statistic does. Our TS serves two purposes for UDA: (1) comparing different UDA methods to select the most suitable one, and (2) assisting in hyperparameter tuning.
Saturation Level of UDA Training. It has been observed that UDA does not consistently lead to improvement for deep models (Wang et al., 2019b). Due to potential overfitting, the highest target-domain accuracy often achieved during the training process, but the current UDA methods directly use the last-epoch model. To determine the optimal epoch for model selection after UDA, we introduce the saturation level of UDA training, represented by the coefficient of variation of the TS. Definition 3. Denote Tm as the Transfer Score at epoch m. The saturation level is defined as
Sm Fm
(6)
pm
where a. and μ are the standard deviation and mean within a sliding window T, respectively.
5

---

Published as a conference paper at ICLR 2024
When the saturation level of the TS falls below predefined threshold Ç. it indicates that the TS has reached a point of saturation. Beyond this threshold, training the model further could potentially result in decline in performance. Therefore, to determine the optimal checkpoint, we select the epoch with the highest TS within that time window. This approach does not necessarily select the best-performing model but effectively enables mitigate the risk of performance degradation caused by continued training and overfitting.
EMPIRICAL STUDIES
5.1
SETUP
Dataset. We employ four datasets in our studies for different purposes. Office-31 (Saenko et al., 2010) is the most common benchmark for UDA including three domains (Amazon, Webcam, DSLR) in 31 categories. Office-Home (Venkateswara et al., 2017) is composed of four domains (Art. Clipart, Product, Real World) in 65 categories with distant domain shifts. VisDA-17 (Peng et al., 2017) is synthetic-to-real object recognition dataset including a source domain with 152k synthetic images and target domain with 55k real images from Microsoft COCO. DomainNet (Peng et al., 2019 is the largest DA dataset containing 345 classes in 6 domains. We adopt two imbalanced domains, Clipart (c) and Painting (p).
Baseline. To thoroughly evaluate the effectiveness and robustness of our metric across different UDA methods, we select classic UDA baseline methods including five types: adversarial UDA method (DANN (Ganin et al., 2016), CDAN (Long et al., 2018), MDD (Zhang et al., 2019)), moment matching method (DAN (Long et al., 2017), CAN (Kang et al., 2019)), norm-based method (SAFN (Xu et al., 2019)), self-training method (FixMatch (Sohn et al., 2020), SHOT (Liang et al., 2021), CST (Liu et al., 2021), 1), AaD (Yang et al., 2022b)) and reweighing-baze
method (MCC (Jin et al., 2020)). For comparison, choose recent works on unsupervised validation of UDA and out-o--distributton (OOD) model evaluation methods as our comparative baselines: C-Entropy (Morerio et al., 2018) based on entropy, SND (Saito al., 2021) based neighborhood structure, ATC (Garg al., 2022) for OOD evaluation, and DEV (You et al., 2019).
Implementation Details. For the Office-31 31 and Office-Home datasets, employ ResNet-50, while ResNet- 101 is used for VisDA-17 and DomainNet. The hyperparameters, training epochs, learning rates, and optimizers are according the default configurations provided in their original papers. We set the hyperparameters Office-31 I dataset, which performs well across all other datasets. Each experiment is repeated three times, and the reported results are the mean values with standard deviations. All the figures report the mean results except Fig. 4 which visualizes specific training procedures. We have included a detailed implementation of empirical studies in the supplementary materials.
= 3 and Ç = 0.01 based on simple validation conducted on the
5.2 TASK 1: SELECTING A BETTER UDA METHOD
We assess the capability of TS comparing and selecting UDA methods. this end, we train UDA baseline models Office-Home and calculate the TS at the last training epoch. The results of TS are visualized with the target-domain accuracy in Fig. 2. It is shown that the highest TS accurately indicates the best UDA method for four tasks, and the TS even reflects the tendency of
ACC
Mathod
Mathod
Malaod
Mathed
(a) Ar-
(b) CI
(c) Prw
(d) Rw Ar
Figure 2: The relationship of accuracy and TS among different methods on four tasks of Office-Home. The proposed TS can accurately indicate the target-domain performance. (FM denotes Fix FixMathh.)

---

Published as a conference paper at ICLR 2024
1.0
16
Dr
Trantster
Trantt
Somecor only
ACC
ACC
ACC
(a) DANND D >A (a: loss weight) (b) SAFN D D-+A (Ar: scalar)
(c) MDD A A- (y: margin)
Figure The TS and the target-domain accuracy under different hyperparameter settings. The models with better hyperparameters reflected by higher TS.
the target-domain accuracy across different UDA methods. Thus, our metric proves to be effective in selecting a good UDA method from various UDA candidates, but it may not necessarily choose the absolute best-perforning model if the performance difference is very small. For Task 1, though existing unsupervised validation methods (e.g.. SND (Saito al., 2021) and C-Entropy (Morerio et al., 2018)) do not consider UDA model comparison, their scores can be tested for Task 1. The results are discussed in the appendix, which shows that existing approaches cannot achieve Task
5.3 TASK 2: HYPERPARAMETER TUNING FOR UDA
After selecting UDA model, it is crucial to perform hyperparameter tuning inappropriate hyperparameters can lead to decreased performance and even negative transfer (Wang et al., 2019h). We evaluate TS via three methods with their hyperparameters: DANN with the weight adversarial loss SAFN with the residual scalar of feature-norm enlargement Ar, and MDD with the margin Their target-domain accuracies and TS on Office-31 are shown Fig. 3, where models with higher TS show significant improvement after adaptation. it is observed that unsuitable hyperparameters can result in performance degradation, sometimes even worse than the source-only model. Overall, the TS metric proves to be valuable in guiding hyperparameter tuning, allowing us to avoid unfavorable outcomes caused by inappropriate hyperparameter choices.
5.4 TASK 3: CHOOSING A GOOD CHECKPOINT AFTER TRAINING
The selection of an appropriate checkpoint (epoch) is crucial UDA, as UDA models often tend to overfit the target domain during training. In Fig. 4, it is observed that all the baseline UDA methods experience a decline performance after epochs of training. Notably, SAFN on DomainNet (c-p) exhibits a significant drop in accuracy of over 17.0%. We utilize the saturation level of TS identify a good model checkpoint (marked as star) within the window size (in red). Detailed results on UDA methods listed in Tab. 1. Compared to the last epoch (i.e., empirical choice), our method works well on most UDA baseline methods, demonstrating robust strategy to choose a reliable checkpoint while overcoming the negative transfer due to the overfitting issue.
We compare our method with the recent works on model evaluation for UDA and -iitt---ittiitttion tasks in Tab. 2. We find that our method outperforms all other methods. C-Entropy cannot choose a better model checkpoint on many tasks, since the entropy only reflects the prediction confidence, which has been enriched by structure for UDA model evaluation, but it has a very high computational complexity. It is noteworthy that all other methods cannot achieve the goal of task and 2.
perspectives in method. The SND leverages neighborhood
Table 1: The model accuracy (%) of the last epoch and the epoch chosen by our method.
Dataset
VisDA-17 Ours
DomainNet (c-pp)
DomainNet (p-tc)
Method
Last
Imp.
Last
Ours
Imp.
Last
Ours
Imp. 1
DAN
66.9+0.4
68.34.0.3
+1.4
35.6++0.5
36.8+0.3
+1.2
45.6++0.
45.6++0.5
DANN
72.910.5
73.810.3
40.9
36.0:10.5
37.910.3
34.510.3
40.210.4
AFN
$8.8::0.6
74:0..5
+15.2
29.6:15.8
41.0:10.5
+11.4
39.1
45.91:0.
+6.8
CDAN
76.21007
76.410.6
40.2
39.5:0.2
39.8:1022
+0.3
44.110.4
44.810.3
+0.7
MDD
71443300
7..51:..
43.1
42,9H0.2
42.5:10.1
-0.4
48.01:.3
48.210.4
40.2
MCC
76.4+0.7
79.5+00.
+3.1
32,9+0.8
37,3+0.1
+4.4
44.6+-0.3
44.4-0.
+0.2
FixMatch
49.24:0.9
66.61.022
+17.4
40.1110.3
41.5:10.2
+1.4
52.740.3
53.210.5
+0.5

---

Published as a conference paper at ICLR 2024
Sonter Levves
ACC 090
ACC
Sofurestoon
8
(a) CDAN (VisDA-17) 17)
(b) DANN (VisDA-17)
MDD (VisDA-Il 17)
162
273
Suttu eitton Level
ACC (96)
J0CK
Saturatton
Epoch
Epocch
Epoch
(d) SAFN (DomainNet)
(c) DANN (DomainNet)
(f) MCC (DomainNet)
Figure 4: The choice of model checkpoint after UDA training for different methods.
Table Comparison of different methods on Office-31 (W->A), Office-Home (Rw Ar) and VisDA-17.
Table 3: The ablation study on three UDA methods.
CDAN
MCC
M
MCC
CST
AaD
Task
Publication
W-A
Rw-1Ar
VisDA-17
W-+A
Rw-tAr
VisDA-17 17
35.8
81.8
84.9
Soureo-Uily DEV
65.6
35.8
82.2
84.9
JCML-19
66.4
63.5
72.6
67.6
63.1
72.3
35.9
82.7
$4.9
C- C-Earrapy
ICLR-18
63.8
61.7
69.9
72.4
66.9
68.9
<
36.0
83.8
85.3
SND
ICCV:21
67.0
70.8
70.3
67.4
68.8
73.0
73.5
73.9
73.0
36.5
83.3
85.6
Ours
73.3
74,9
76.4
74.5
73.3
79.5
37.3
83.8
85.9
5.5 ANALYTICS
Ablation Study of Metrics. assess the robustness of each metric in the TS, we performed checkpoint selection experiment on MCC and DomainNet (c->p) using three independent metrics: uniformity, Hopkins statistic, and mutual information. As summarized Tab. 3, the uniformity, Hopkins statistic, and mutual information achieve the accuracies of 35.8%, 35.8%, and 35.9% respectively, slightly lower than the accuracy obtained using TS (37.3%). Two more experiments are provided VisDA-17 for CST and AaD.
We explore the viability of our method on a large-scale long- tailed imbalanced dataset, DomainNet. As shown in Fig. 5(a), the label distributions of these two domains are long-tailed and shifted. In Fig. 5(b), it is shown that TS can accurately indicate the performance of various UDA methods, successfully achieving task As illustrated in Tab. 1 and Fig. 4, TS can stably improve UDA methods by choosing a better model checkpoint on DomainNet. We further explore the model uniformity on the imbalanced dataset. Fig. 6(a) and Fig. 6(b), the increased uniformity reflects decreasing accuracy during training. indicating the model becomes more biased, which explains why these UDA methods perform worse on the imbalanced dataset.
Evaluation on Imbalanced Dataset.
Hyperparameter Sensitivity. We study the sensitivity of r using DAN and MCC on the DomainNet, varying 5 within the range of [2. 7]. The results, depicted in Fig. 6(c), indicate that the best perfor- mance is achieved when the window size is set to 3. This finding suggests that considering relatively short range TS values has been sufficient for UDA checkpoint selection. Our method also includes another aaper-parameter
controls the variation of accuracy within the sliding window when the saturation point chosen. conducted experiments on all four datasets and found that when the UDA models converge, such variations always do not exceed 1.0%. Thus, we just need to set Ç to be sufficiently small, Le., 0.01.
t-SNE Visualization and Uniformity of Classifier. To gain more intuitive understanding of TS. we visualize target-domain features using t-SNE (Van der Maaten & Hinton, 2008) and the classifier parameters using unit circle in Fig. where each radius line in the unit circle corresponds
8

---

Published as a conference paper at ICLR 2024
1.40
Cppart Pointing
52.5
1020
Number of insoacce
820
47.5
1.35
45.0
-130
42.5
40.0
37.5-
230-
35.0
MDD
MCC
DAN DANN CDAN
FM
Calogotias
24
Method
(a) Long-tailed label distribution of DomainNet
(b) UDA Method on c-+p
Figure 5: Empirical studies on the imbalanced dataset DomainNet.
ACC
N000
Epoch
Epoch
Window Size
(a) MCC c->
(b) CDAN p-+
(c) Sensitivity of 7 on c->p
Figure 6: The uniformity and sensitivity study on the imbalanced dataset DomainNet.
classifier vector w; after dimension reduction. The visualization includes the source-only model (S.O.), DAN, and MCC, with the accuracy ranking of S.O.<DAN<MCC. is found that higher Hopkins statistic value accurately captures the better clustering tendency, and the classifiers with better uniformity perform better. In cases where the Hopkins statistic of S.O. and DAN are similar (0.85 vs. 0.88), the difference in mutual information (0.57 vs. 0.70) provides justifications for the better performance of DAN.
(a) Source-only
(b) DAN
(c) MCC
Figure 7: The t-SNE visualization and the uniformity of classifier on VisDA2017. From left to right, H 0.85 + 0.88 0.93, M 0.57 0.70 - 0.89, 0.1 + 0.09 0.05. Greater H. M indicates better clustering tendency, while less Z/ indicates better uniformity.
CONCLUSION
UDA tackles the negative effect of data shift in machine learning. Previous UDA methods all rely on target-domain labels for model selection and tuning, which is not realistic in practice. This paper presents a solution to the evaluation challenge of UDA models in scenarios where target-domain labels unavailable. To address this, we introduce novel metric called the transfer score, which evaluates the uniformity of classifier as well as the transferability and discriminability of features, represented by the Hopkins statistic and mutual information, respectively. Through extensive empirical analysis on four public UDA datasets, we demonstrate the efficacy of the TS in UDA model comparison, hyperparameter tuning, and checkpoint selection.

---

Published as a conference paper at ICLR 2024
This work is supported by the NTU Presidential Postdoctoral Fellowship. "Adaptive Multimodal Learning for Robust Sensing and Recognition in Smart Cities" project fund, at Nanyang Technological University, Singapore. This research is jointly supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG2-PhD- 2021-08- 100-008008).
Acknowledgements.
REFERENCES
Andrea Agostinelli, Michal Pándy, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. How stable are transferability metrics evaluations? In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pp. 303-321. Springer, 2022.
Amit Banerjee and Rajesh N Dave. Validating clusters using the hopkins statistic. 2004 IEEE International conference on fuzzy systems (IEEE Cat. No. 04CH37542), volume 1, pp. 149-153. 153. IEEE, 2004.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1-2)1151-155, 2010.
Haozhi Cao, Yuecong Xu, Jianfei Yang. Pengyu Yin, Shenghai Yuan, and Lihua Xie. Multi-modal continual test-time adaptation for 3d semantic segmentation. arXiv preprint arXiv:2333.100 10457, 2023.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In International conference on machine learning, pp. 1081-1090. PMLR, 2019.
Zhijie Deng. Yucen Luo, and Jun Zhu. Cluster alignment with a teacher for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9944 9953, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 1 17(1):2096 2030, 2016.
Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint arXiv:2201.04234, 2022.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pp. 529 0066,,66650 2005.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bemhard Schölkopf, and Alex Smola. A kernl method for the two-sample-problmm. In Advances in Newral Information Processing Systems, pp. 513-520, 2007.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf. and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Hao Guan and Mingxia Liu. Domain adaptation for medical image analysis: survey. IEEE Transactions Biomedical Engineering, 2021.
Zhongyi Han, Haoliang Sun, and Yilong Yin. Learning transferable parameters for unsupervised domain adaptation. IEEE Transactions on Image Processing, 31:6424-6439, 2022.
Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and Doug Tygar. Ad- versarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence, pp. 43-58, 2011.
10

---

Published as a conference paper at ICLR 2024
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang. Liqiang Wang, and Boqing Gong. Rethinking balss-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7610-7619, 2020.
Xiang Jiang, Qicheng Lao, Stan Matwin, and Mohammad Havaci. Implicit class-conditioned domain alignment for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 4816-4827. PMLR, 2020.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In European Conference Computer Vision, pp. 464-480 Springer, 2020.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4893-4902, 2019.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Learning, pp. 5468-5479. PMLR, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86( (6(11))227-22324, 1998.
Jichang Li, Guanbin Li, Yemin Shi, and Yizhou Yu. Cross-domain adaptive clustering for semi- supervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2505-2514, 14, 2021.
Yun Li, Zhe Liu, Lina Yao, Jessica JM Monaghan, and David McAlpine. Disentangled and side-aware unsupervised domain adaptation for cross-datasct subjective tinnitus diagnosis. IEEE Journal of Biomedical and Health Informatics, 2022.
Jian Liang. Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Jian Liang, Dapeng Hu, Jiashi Feng, and Ran He. Dine: Domain adaptation from single and multiple black-box predictors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. Advances in Neural Information Processing Systems, 34, 2021.
Ziwei Liu, Zhongqi Miao, Xingang Pan, Xiaohang Zhan, Dahua Lin, Stella Yu, and Boqing Gong. Open compound domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12415,, 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 136-144, Red Hook, NY, USA, 2016. Curran Associates Inc.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning (ICMI), 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, 2018.
Xinhong Ma. Junyu Gao, and Changsheng Xu. Active universal domain adaptation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 1968-8977 2021.
11

---

Published as a conference paper at ICLR 2024
Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entroopy correlation alignment for unsupervised deep domain adaptation. arXiv preprint arXiv:1711.10288, 2017.
Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. International Conference on Leurning Representations, 2018. URL https: / openreview. net/ BerrummiddJJWWehhhgh.
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pp. 7394-7305. PMLR, 2020.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-210 2011.
Karl Pearson. Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352) 1895.
Xingehao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:/710.0692 2017.
Xingehao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference computer vision, pp. 213-226. Springer, 2010.
Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9184-9193, 2021.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach unsupervised domain adaptation. In Proc. 6th International Conference on Learning Representations, 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-608, 2020.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Anh Tran, Cuong V Nguyen, and Tal Hassner. Transfcrability and hardness of supervised classifi- cation tasks. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp- 1395-1405. 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proc. CVPR, pp. 5018-5027, 2017.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Nearocomputing, 312: 135-153, 2018.
Xiyu Wang, Yuecong Xu, Kezhi Mao, and Jianfei Yang. Calibrating class weights with multi-modal information for partial video domain adaptation. In the 30th ACM International Conference on Multimedia, 2022.
12

---

Published as a conference paper at ICLR 2024
Zhennan Wang, Canqun Xiang, Wenbin Zou, and Chen Xu. Mma regularization: Decorrelating weights of neural networks by maximizing the minimal angles. Advances in Neural Information Processing Systems, 33:19099-19110, 2020.
Zirui Wang, Zihang Dai, Barnaáás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1 11293-11 1302, 2019a
Zirui Wang, Zihang Dai, Barnabas Poczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b.
Binhui Xie, Shuang Li, Fangrui Lv, Chi Harold Liu, Guoren Wang, and Dapeng Wu. A collaborative aligmment framework of transferable knowledge extraction for unsupervised domain adaptation. IEEE Transactions on Knowledge and Data Engineering, 2022.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 1426-1435, 2019.
Yuecong Xu, Jianfci Yang. Haozhi Cao, Zhenghua Chen, Li, and Kezhi Mao. Partial video domain adaptation with partial adversarial temporal attentive network. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9332-9341, 2021.
Yuecong Xu, Haozhi Cao, Kezhi Mao, Zhenghua Chen, Lihua Xie, and Jianfei Yang. Aligning correlation information for domain adaptation in action recognition. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Yuecong Xu, Jianfei Yang, Haozhi Cao, Keyu Wu, Min Wu, Zhengguo Li, and Zhenghua Chen. Multi-source video domain adaptation with temporal attentive moment alignment network. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
Jianfei Yang, Han Zou, Yuxun Zhou, Zhaoyang Zeng, and Lihua Xie. Mind the discriminability: Asymmetric adversarial domain adaptation. European Conference on Computer Vision, pp. 589-606. Springer, 2020a.
Jianfei Yang, Jiangang Yang. Shizheng Wang, Shuxin Cao, Han Zou, and Lihua Xie. Advancing imbalanced domain adaptation: Cluster-level discrepancy minimization with a comprehensive benchmark. IEEE Transactions on Cybermetics, 2021a
Jianfei Yang, Han Zou, Yuxun Zhou, and Lihua Xie. Robust adversarial discriminative domain adaptation for real-world cross-domane visual recognition. Neurocomputing, 433:28-36 36, 2021b.
Jianfei Yang, Xiangyu Peng, Kai Wang, Zheng Zhu, Jiashi Feng, Lihua Xie, and Yang You. Divide to adapt: Mitigating confirmation bias for domain adaptation of black-box predictors. arXiv preprint arXiv:2205. 144447. 2022a.
Jinyu Yang. Weizhi An, Sheng Wang, Xinliang Zhu, Chaochao Yan, and Junzhou Huang. Label- driven reconstruction for domain adaptation in semantic segmentation. In European Conference on Computer Vision, pp. 480-498. Springer, 2020b.
Shiqi Yang, Shangling Jui, Joost van de Weijer, al. Attracting and dispersing: A simple approach for source-free domain adaptation. Advances in Neural Information Processing Systems, 35: 5802-5815, 2022b.
Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
4085-4095, 2020.
Hao- Yeh, Baoyao Yang, Pong C Yuen, and Tatsuya Harada. Sofa: Source-data-fie feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 474 483, 2021.
13

---

Published as a conference paper at ICLR 2024
Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning, pP- 7133. PMLR, 2019.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of
pre-trained models for transfer learning. In International Conference on Machine Learning, pp. 12133-12143. PMLR, 2021.
Kaichao You, Yong Liu, Ziyang Zhang, Jianmin Wang, Michael Jordan, and Mingsheng Long. Ranking and tuning pre-trained models: new paradigm for exploiting model hubs. The Journal of Machine Learning Research, 23(1):9400-9446 2022.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference Machine Learning, pp. 7404-7413, 2019.
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 PROOF OF THEOREM 1
Suppose there are K unit vectors [W1, u2,,.., wx] in a d-dimensional space R and the angles between any two unit vectors are equal, denote the angle in this case as 0x- Assume that K I d d 1. By expanding the sum squares formula, we have
K (w W3}
K
|
(7)
The first term on the right-hand side greater than or equal to and we have
K
(wi, way) 2
=-K
(8)
ixj.==1
i=1
When the angle between any two vectors is equal to K, the equation holds true, which can be simplified to
1
(wi, wz) = K -1
(9)
Thus,
0K arccos
(10)
K -
Q.E.D.
A.2 MORE RESULTS OF EMPIRICAL STUDIES
Here we put more empirical results the proposed Transfer Score (TS) of unsupervised domain adaptation (UDA) on other datasets.
Score
for Score
AcC Oll
ACC cla
AGC ONI
Fondfor
Wethod
Wethod
Welhod
Methoz
(a) A
(b) A-W
(c) D
(d) W-A
Fondfer Score
Transdur Scott
ACC
ACCTEM
ACCITEA (M4
(e) Ar-
(f) Rr
Cl-AAr
(h) CI->Rw
Transter Scond
Fanooler Score
ACC
AGC
Pr-+Ar
Pr-+C
(k) Rw-+C
Rw-Pr
Figure 8: The relationship of accuracy and TS among different methods on tasks of Office-31 and tasks of Office-Home.
15

---

Published as a conference paper at ICLR 2024
A.2.1 TASK 1 (UDA METHOD COMPARISON) ON OFFICE-31 31 AND HOME
In this section, add the experiments on all other tasks Office-31 and Office-Home in Fig. For Offce-31, we use all 4 tasks except the W-D and D-W as their transfer accuracy is over 99%. In most cases, the transfer score can help select the best-performing model across the candidates. Although in few cases. the difference in Transfer Scores does not perfectly align with the variations accuracy, such as MCC (Jin et al., 2020) and MDD (Zhang et al., 2019), higher accuracy generally corresponds to better Transfer Scores. In A->D, D-A, and W->, MCC achieves the highest Transfer Score and best performance.
Only in the A-W task, CDAN (Long et al., 2018) outperforms MCC by a margin of 0.9%, but its Transfer Score is relatively lower. However, the difference in TS between CDAN and MCC is very small, less than 0.3%, since they all perform well (>90%) in the A- task, which makes it difficult for the metric to reflect such a small accuracy gap. Overall, TS still effectively reflects the quality of the models.
A.2.2 TASK 2 (HYPER-PARAMETE) TUNING) ON VISDA-17 DATASET
Table 4: The relationship between TS and the target-domain accuracy using different hyper- aaper-parameters of MCC VisDA-17. The models with better prrame-rrrmeters are reflected by higher TS.
Temperature
ACC (%)
Transfer Score
0.1
55.5
1.179
71.7
1.801
76.5
1.819
9
56.1
1.706
27
51.0
1.014
We also investigate the impact of hyper pareeerrrreeterr in the MCC method. The temperature parameter in MCC is used for probability rescaling. In the original paper, the authors analyze the sensitivity of the hyper- ppper-parameter on the A-W task the Office-31 dataset with access to the target-domain label and conclude that the optimal value for temperature is 3. From Tab. 4, we can draw similar conclusions, which are obtained from experiments conducted on a much larger dataset, VisDA-17. 17. reasonable to believe that our proposed Transfer Score can help determine the approximate range of typer-parameters without accessing the target labels.
A.2.3 TASK 3 (EPOCH SELECTION) ON VISDA-17 AND DOMAINNET DATASET
We further test the proposed method of selecting a good model checkpoint based on TS after UDA training on more tasks. We choose two datasets for our study: VisDA-17 and DomainNet, which include three tasks: Synthetic to Real, Clipart to Painting (c-p)), and Real to Sketch (r+s). As shown in Fig. 9, for most methods, when the model overfits the target-domain, there is certain decrease in accuracy. Our method can effectively capture the checkpoint before overfitting occurs. For SAFN (Xu et al., 2019) and MCC, method is highly effective, providing improvement of 5%-51%99 17% compared to selecting the last epoch, thus avoiding overtraining. For DAN (Long et al., 2017) and DANN (Ganin al., 2016), the improvements 2%, possibly due to slower convergence or larger fluctuations in these models. Overall, our method effectively prevent overfitting and identify good checkpoint.
relatively smaller, ranging from 0.5% to
A.3 LIMITATION
A.3.1 LIMITATIONS IN TASK (UDA METHOD SELECTION)
Table 5: Component analysis of the transfer score on Office-Homme
Ar-CCI
CI-PP
Rw
Rw-tAr
Method
U
N
H
.M
M
1/
DANN
0,838
0.667
0.066
0.846
0.678
0.069
0.855
0.696
0.066
0.829
0.690
0.066
0.063
0.734
0.063
MDD
0.876
0.707
0.066
0.879
0.722
0.066
0.881
0.731
0.070
0.861
0.720
0.067
is found that some methods cannot be measured accurately since they add part of the criterion that directly relevant the TS (e.g., mutual information) into the training procedure. For example, as
16

---

Published as a conference paper at ICLR 2024
Saturoa Lavell
22
ACC
Satur Satur-ttion
ACC
Epoch
Epoch
Ezoch
(a) MCC (VisDA-17)
(b) MCC (DomainNet r->s)
SAFN (VisDA-17)
aiiionn Lavell
ACC.09)
Arc.
Saturalton
ACC
Sattur
Epoch
Tpoch
(d) SAFN (DomainNet c->p)
(e) SAFN (DomainNet r r-)s)
(1) DAN (VisDA-17)
Saturation Leavl
Saturration Larvll
Suturaton Looel
ACC.N))
ACC (80
ACC
000
(g) DAN (DomainNet c-t)
(h) DAN (DomainNet r->s)
DANN (DomainNet r->s)
Figure The choice of model checkpoint after UDA training for different methods.
shown in Tab. have observed that the Hopkins statistic value (Banerjee & Dave, 2004) for SAFN becomes unusually high after few epochs of training, indicating high degree of clustering features. This is because the regularization mechanism SAFN not only encourages enlargement of feature norms but also concentrates features around fixed value. The concentration of feature norms further enhances the tendency for clustering. However, this does not lead to a high accuracy, because such clustering tendency is class-agnostic. Therefore, if we add some regularization to directly restrain one component of the TS, the TS might be less effective.
A.3.2 LIMITATION TASK 3 (EPOCH SELECTION)
As shown in Fig. 10, the accuracy of MDD continuously increases without early stopping point. In this case, our solution cannot find one of the best checkpoints but can provide a relatively cost- effective point. We also observe that the saturation level becomes smoother than those in Fig. 9 that encounter negative transfer. In this manner, smoother convergence of saturation level might indicate robust training curve without negative transfer, which will be explored in future work.
A.4 IMPLEMENTATION DETAILS.
Table 6: Hyper-paramete settings for all the baseline methods.
Hyper-pan parameter
DAN
DANN
CDAN
SAFN
MDD
MCC
LR
0.003
0.01
0.01
0.001
0.004
0.005
Trade-off
1
0.1
1
Others
Ar:
Margin:
Temperature: 3
17

---

Published as a conference paper at ICLR 2024
Sutur At0m Lovel
182
Epoch
Epoch
(a) MDD (DomainNet c-p)
(b) MDD (DomainNet r-+s)
Figure 10: Failure cases of the epoch selection of the model (i.e., task3).
We list all the hyper- paprm-paraeeters of our baseline methods Tab. 6. We follow the original papers to set these hyper- ppprmeerrameters, except that obtain some better hyper-parameten in terms of performances which are listed in the table. Note that the LR indicates the starting learning rate and the decay strategy same those of the original papers.
A.5 CORRELATION BETWEEN TRANSFER SCORE AND ACCURACY
To prove the better correlation with the accuracy, we conducted an experiment by comparing our metric with two advanced unsupervised validation metries, C-Entropy and SND, on Office-31 and VisDA-17. The baseline models are DANN and MCC for Office-31 and VisDA-17, respectively. As shown Fig. 11 and Fig. 12, our approach shows the best correlation coefficients. is also observed that for VisDA-17, only our metric can reflect the overfitting issue while the other two metrics that keep increasing during training contradict with the decreasing accuracies.
Teans Fransfer Stoore
ACC (%0)
ACC (%))
ACC
Correlation: 0.74
Commeationn 0.97
Epoch
Epoch
Epoch
C-entropy
(b) SND
(c) Transfer Score
Figure 11: The correlation between method and existing transfer metrics. (DANN Office-31
W)
Trantfer Scurce
ACC (%)
East
SND
ACC.
Correlation: 9-85
Correlation: 0.57
Correiation:
Epach
Epach
Epach
(a) C-entropy
(b) SND
(c) Transfer Score
Figure 12: The correlation between our method and existing transfer metrics. (MCC on VisDA-17)
A.6 EPOCH SELECTION FOR MORE UDA METHODS
To further demonstrate the effectiveness of TS, we add four more UDA methods as baselines on VisDA-17. The results have been shown in Table 7. demonstrates that using transfer help select a better epoch, improving SHOT, CAN, CST, and AaD by 0.5%, 0.4%, 11 11.8%, and 2.5%, respectively.
18

---

Published as a conference paper at ICLR 2024
Table 7: The comparison for epoch selection VisDA-17.
Method
Last
Ours
Imp.
SHOT
77.5
78.0
0.5
CAN
86.4
86.8
0.4
CST
72.0
83.8
11.8
AaD
83.4
85.9
2.5
A.7 EFFECTIVENESS OF TS ON SEMANTIC SEGMENTATION
We conducted an experiment on semantic segmentation. We use a classic cross-domain segmentation approach FDA (Yang & Soatto, 2020) as baseline on an imbalanced dataset, GTA-5 Sity-aes We train the FDA using the default hyper-parameters and use our TS to choose the best epoch of model. The mloU results are shown in Table is shown that our metric can choose better epoch of the model with an average mloU of 39.7% while the last epoch of the model only achieves 37. 7%. This demonstrates that our metric still works effectively on the imbalanced cross-domain semantic segmentation task.
Table 8: Evaluation cross domaan segmentation (GTA-5- Cityscapes)
bding
pole
light
sign
vege
temain
sky.
person
mige
Niyclee
Avg
Vanilla Ouns
32.3
14.1
37.7
A.8 DIFFERENCE WITH SND
We summarize the differences between our method and SND Saito et al. (2021) regarding the task, method and baseline UDA methods. Our work can achieve the model comparison while SND cannot. The transfer score measures the transferability from 3 perspectives while SND measures it via single metric. the empirical study, we prove the effectiveness of transfer score using more UDA baselines.
Table 9: The differences between our metric and SND.
Transfer Score (Ours)
SND
Model comparison
Epoch selection
Task
flyper-parameter tuning
sepper-parametr tuning
Epoch selection
Method
Transfer score is a three-fold metric, includ- ing uniformity of model weights, the tual information of features, and the cluster- ing tendency.
SND uses a single metric, the den- sity of implicit local neighborhoods, which describes the clustering ten- dency.
11 UDA methods:
Adversarial: CDAN, DANN, MDD Moment matching: DAN, CAN
4 methods: CDAN
Adversarial: Reweighing-basee MCC Self-training: NC, PL.
Baseline UDA methods
Reweighing- MCC
Self Self-training: FixMatch, CST, SHOT, AaD Norm-based. SAFN
A.9 USING TS AS A UDA REGULARIZER
add the experiment to explore the effectiveness of the three components of the transfer score. The experiments conducted Office-31 (A-W)) and Office- Home (Ar-CCl) based on ResNet50. We add the three parts of transfer score as an independent regularizer on the target domain. As shown in the Table 10, it is shown that every component brings some improvement compared to the source-only
19

---

Published as a conference paper at ICLR 2024
model. The total transfer for A-W. W. The vamilla Hopkins statistic calculated at the batch level so we think it can be enhanced further learning objective for UDA. We think this is an explorable direction in the future work.
brings significant improvement by 24.1% for A-W and 12.6%
Table 10: Evaluation using TS as learning objective for UDA.
A
Ar-CC
Source-only
68.4
34.9
Uniformity
75.1
41.8
Hopkins statistic
75.6
41.8
Mutual information
92
45.6
Transfer Score (Total)
92.5
47.6
A.10 COMPARISON WITH SND AND C-ENTROPY ON TASK
Unsupervised model evaluation methods (c.g., SND Saito al. (2021) and C-Entropy Morerio et al. (2018)) aim to select model parameters with better hyper-ppaaaae parameters for a specific UDA method. Though these works do not consider Task I in their original papers, we find that their scores still be tested for Task To this end, we conduct the experiments and calculate the SND and C-entropy of 6 UDA methods on three datasets including Office-31 (D (D+AA), Office- Home (Ar-PPr), and VisDA-17. The results are shown in Fig. 13, where we mark the selected model for each metric with bold font. observed that the C-entropy succeeds selecting the best model (MCC) on Office-31 1 but fails in Office Home and 17,, while SND fails all three datasets. In comparison, our proposed metric consistently selects the best UDA method for all three transfer tasks, significantly outperforming existing unsupervised validation methods.
20

---

Published as a conference paper at ICLR 2024
78
Trans fer Score
ACC 0%0
EEnt
ACC
1.50
-26
PSC
ove
1.45
Method
Mothod
Mothed
(a) C-entropy
(b) SND
(c) Transfer Score
0.0
1.40
-12
ACC 1963
ACC
145
36
1.0
100
56
475
PG
400
Method
Method
Method
(d) C-entropy
(e) SND
(f) Transfer Score
00
72-
72
Traisfer Score
ACC 1960
ACC (96)
Method
Method
Method
C-entropy
(h) SND
Transfer Score
Figure 13: The comparison of UDA method comparison (Task 1). The three rows of figures performed on Office-31 (D (D>>)), Office-Home (Ar-PPr), and VisDA-17, respectively (top-down).
21


Reviews:
[{"review_id": "8LR7INFyLj", "reviewer": "Reviewer_hZ7S", "summary": "To evaluate the performance of the Unsupervised Domain Adaptation (UDA) model, this paper proposes a novel metric called “Transfer Score”. Based on this metric, we achieve three novel objectives without target-domain labels: (1) selecting the best UDA method (2) optimizing the hyperparameter of the UDA model (3) selecting the best checkpoint.", "questions": "I would like to see authors' response regarding the weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tEvaluate UDA models in an unsupervised manner is very important, but there is a lack of relevant research in the current community\n2.\tExperiments demonstrate the effectiveness of “Transfer Score” in method selection, hyperparameter tuning, and checkpoint selection. \n3.\tThe paper is well-written and easy to understand.", "weaknesses": "1.\t“Transfer Score” uses clustering and class balance as the measurement criteria. However, the assumption of clustering and class balance usually may not always hold. Acutally, in many real world tasks, class imbanlance often exists. For example, in cross-domain semantic segmentation, a severe category imbalance is often present [1], which limits the application of this metric.\n2.\tSome methods[2,3] directly adopt Eq. 1 and Eq. 4 as optimization objectives. For these methods, the transfer score may be invalid.\n3.\tThe author needs to verify the effectiveness of the “Transfer Score” on more advanced UDA methods[3,4]. \n\n[1] Liu Y, Deng J, Tao J, et al. Undoing the damage of label shift for cross-domain semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 7042-7052.\n[2] Liang J, Hu D, Wang Y, et al. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(11): 8602-8617.\n[3] Yang S, Jui S, van de Weijer J. Attracting and dispersing: A simple approach for source-free domain adaptation[J]. Advances in Neural Information Processing Systems, 2022, 35: 5802-5815.\n[4] Kang G, Jiang L, Yang Y, et al. Contrastive adaptation network for unsupervised domain adaptation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 4893-4902.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XX3o5D2Zae", "reviewer": "Reviewer_eAAs", "summary": "Targeting at evaluating UDA models without target-domain labels, this paper proposes a new metric called the Transfer Score by assessing the spatial uniformity of the classifier via model parameters as well as the transferability and discriminability of deep representations. Three types of experiments are conducted for model evaluation.", "questions": "Please refer to the Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The proposed metric is meaningful and intuitive.\n+ The proposed metric seems useful in various evaluation settings.", "weaknesses": "- As the authors stated, “prevailing UDA methods relying on adversarial training and self-training could lead to model degeneration and negative transfer”, How can we prove this viewpoint? Can the proposed metric be applied to these types of methods for performance improvement in regular UDA settings?\n- Although the proposed metric is intuitive, it is difficult to validate that it is definitely correct for evaluating a UDA model. In fact, various existing UDA approaches also adopt “Transfer Scores” for guiding the model learning process, such as [a]. A comparison between these metrics should be conducted.\n- In figure 1, the correlation analysis is conducted only in one UDA task (Office31 A->W) and one UDA model (DANN), more comprehensive analysis and more comparison (with more metrics) should be conducted.\n- In Definition 2, the design of the Score seems handcrafted. How can we ensure this formulation is optimal? Why use this design? The reasons should be analyzed.\n- Some SOTA UDA approaches are not employed for experimental analysis [b-c].\n- From Table 1, why are the improvements of the last four methods on the DomainNet (p->c) task so small?\n\n[a] Active universal domain adaptation, ICCV 2021;\n[b] Fixbi: Bridging domain spaces for unsupervised domain adaptation, CVPR 2021;\n[c] Cycle self-training for domain adaptation, NeurIPS 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Xa77TXH1g", "reviewer": "Reviewer_iLFW", "summary": "This paper proposes a method named Transfer Score, which measures the classifier bias and feature discriminability, for the unsupervised validation problem.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This idea is simple and easy to follow.", "weaknesses": "- The experiments are weak.\n  - Task1 and Task2 are actually unsupervised model evaluation problems, all the unsupervised validation methods such as SND can be employed directly, but the authors do not compare their method with these methods.\n  - In the experiments, only six UDA methods were evaluated. In task 2, there are only five candidate hyperparameters. The number is too small to illustrate the effectiveness of the proposed method. \n  - The UDA datasets employed in the experiments are not comprehensive. On office-home, only four tasks were selected, and the situation was similar for DomainNet and Office. How do you select these tasks?\n\n- The authors said, \"TS is the **first** metric to perform simultaneous model comparison and selection without target-domain labels\". As far as I know, the three tasks in experiments are actually unsupervised validation problems, and this is an existing research field. Besides, some out-of-distribution generalization prediction methods also solve the model selection problem without labels.\n  > K-Means Clustering Based Feature Consistency Alignment for Label-Free Model Evaluation. \\\n  > Predicting Out-of-Distribution Error with Confidence Optimal Transport.\\\n  > Leveraging Unlabeled Data to Predict Out-of-Distribution Performance.\\\n  > ...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zDEUGIMDhg", "reviewer": "Reviewer_VJy8", "summary": "This paper studies the performance evaluation problem for unsupervised domain adaptation methods. Authors propose a evaluation metric called the Transfer Score. The propose metric consides both spatial uniformity from the model parameters and the transferability and discriminability calculated from the target samples. The metric can help users to select UDA methods, hyperparameters and checkpoints. Authors provide rich experiments to evalutate the effectiveness of the proposed metric.", "questions": "1. What if the Transfer Score used as a optimization function for UDA task? I am curious about this.\n\n2. Mutual information in Eq.4 is used as a term in the Transfer Score, if some methods (e.g. SHOT) minimize this term during adaptation, will it directly cause mutual information term to become invalid?\n\n3. Because there is no source information used in Transfer Score, can authors provide the results on Source-Free Domain Adaptation methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed Transfer Score is novel and the calculation process is clear.\n\n2. The authors provide code in Supplementary Material, which makes the experiments convincing.\n\n3. The structure of the paper is well organized.", "weaknesses": "1. Experimental results of baseline methods (e.g., DEV, SND) should be provided in the experiment section, there are only Transfer Score results of many dataset in that section.\n\n2. Ablation study of Table 3 is weak, authors should provide the same one like Fig. 2 to evaluation the effectiveness of each term.\n\n3. The representation of Equation 1 is incorrect, especially i*(i-1) and the use of i.\n\n4. Authors should provide the complete results of all tasks from the different datasets * different UDA methods * baseline methods and proposed score, like Fig.3. The experimental results in the paper seem a bit sparse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)