Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation
Authors: Dapeng Hu
Jian Liang
Jun Hao Liew
Chuhui Xue
Song Bai
Xinchao Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Unsupervised Domain Adaptation; Model Selection; Hyperparameter Selection; Unsupervised Validation;
TL;DR: We introduce MixVal, a novel and straightforward target-only method for model selection in unsupervised domain adaptation (UDA).

Abstract: Unsupervised domain adaptation (UDA) has been widely applied in improving model generalization on unlabeled target data. However, accurately selecting the best UDA model for the target domain is challenging due to the absence of labeled target data and domain distribution shifts. Traditional model selection approaches involve training extra models with source data to estimate the target validation risk. Recent studies propose practical methods that are based on measuring various properties of model predictions on target data. Although effective for some UDA models, these methods often lack stability and may lead to poor selections for other UDA models.
In this paper, we present MixVal, an innovative model selection method that operates solely with unlabeled target data during inference. MixVal leverages mixed target samples with pseudo labels to directly probe the learned target structure by each UDA model. Specifically, MixVal employs two distinct types of probes: the intra-cluster mixed samples for evaluating neighborhood density and the inter-cluster mixed samples for investigating the classification boundary. With this comprehensive probing strategy, MixVal elegantly combines the strengths of two state-of-the-art model selection methods, Entropy and SND. We extensively evaluate MixVal on 11 UDA methods across 4 adaptation settings, including classification and segmentation tasks. Experimental results consistently demonstrate that MixVal achieves state-of-the-art performance and maintains exceptional stability in model selection. 
Code is available at \url{https://github.com/LHXXHB/MixVal}.

Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation
Dapeng Hul* Jian Liang Jun Hao Liew2 Chuhui Xue²
Song Bai²
Xinchao Wang" f
"ByteDance Inc. CCRPAC & MAIS, Institute of Automation, Chinese Academy of Sciences
National University of Singapore
Abstract
Unsupervised domain adaptation (UDA) has been widely applied in improving model generalization unlabeled target data. However, accurately selecting the best UDA model for the target domain is challenging due to the absence of labeled target data and domain distribution shifts. Traditional model selection approaches involve training extra models with source data to estimate the target validation risk. Recent studies propose practical methods that are based on measuring various properties of model predictions on target data. Although effective for some UDA models, these methods often lack stability and may lead to poor selections for other UDA models. In this paper, we present Mix Val, an innovative model selection method that operates solely with unlabeled target data during inference. Mix Val leverages mixed target samples with pseudo labels to directly probe the learned tar- get structure by each UDA model. Specifically, Mix Val employs two distinct types of probes: the intra-cluster mixed samples for evaluating neighborhood density and the inter-cluster mixed samples for investigating the classification boundary. With this comprehensive probing strategy, Mix Val elegantly combines the strengths of two state-of offte-of-thh-ar model selection methods, Entropy and SND. We extensively evaluate Mix Val on 11 UDA methods across 4 adaptation settings, including classi- fication and segmentation tasks. Experimental results consistently demonstrate that Val achieves state-of-the-a of-the-art performance and maintains exceptional stability in model selection. Code is available at https: happs:// //githubb com Hom//LXXXH//MMB /Mixlal.
Introduction
Despite the remarkable achievements of supervised learning in visual recognition tasks [1-4], deep neural networks face challenges when comes to generalizing to out-of-distribution data [5]. As for the obstacle of out-of-domain generalization, deep domain adaptation techniques [6] provide an effective solution by transferring knowledge from a label-rich source domain to a related, yet label-scarce, target domain. Unsupervised domain adaptation (UDA) has drawn significant interest in recent years within the field of computer vision, including image classification [7-9], semantic segmentation [10-12], and object detection [13, 14], thanks to its practical setup with completely unlabeled target data. As the UDA landscape continues to evolve, various UDA settings have been explored to consider real-world scenarios, such variations in category overlaps between domains and concerns related to source data privacy. These settings include closed-set UDA [7], partial-set UDA [15, 16], open-set UDA [17], open-partial-set UDA [18], and source-free UDA [19-21]. To ef- fectively address these UDA problems, researchers have developed various novel solutions, including cross-domain alignment techniques [7, 9, 22, 11, 12], target-domain regularization methods [23-25], and self-training strategies in the target domain [26, 27, 20].
"Contact: Frexhbb14ggaain com. Partial work was done during an internship at ByteDance with Song Bai. Connesponding authors: Jian Liang (liangjian92 gmai).com), and Xinchao Wang (xinchao @ edu.s).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Nevertheless, is worth emphasizing that hyperparameters, including those associated with both the -eetood-peecfic loss functions and the deep training pipeline, play a critical role in ensuring superior performance of UDA models during deployment. Notably, even the model trained by state-of-the- art UDA approach can underperform the baseline source-trained model without adaptation the hyperparameters are not properly set [28]. Surprisingly, the selection of hyperparameters has been largely overlooked in previous UDA studies, as highlighted by You et al. [18] and Saito et al. [29].
To further fill this research gap, this paper tackles the challenging problem of unsupervised model selection in domain adaptation. Figure ((a) illustrates the UDA validation pipeline. During the training stage, we employ different values of a hyperparameter 3, denoted as 32,32.35 Ps, to train corresponding UDA models denoted as M1. M2, M3. Subsequently, in the validation stage, our objective is to identify the model with the minimal risk on the target domain and determine its associated /3 value as the optimal choice for the UDA method. Dealing with model selection in UDA presents two primary challenges. On one hand, the absence of labeled target data makes conventional supervised target validation unfeasible. On the other hand, even if we had access labeled source data, relying solely on the source risk [7] cannot ensure accurate estimation of the target risk due to domain distribution shifts between domains [29]. Therefore, advanced validation approaches are necessary to ensure accurate model selection in UDA.
Existing validation solutions in UDA can be classified into two types. The first type consists of source- based methods, which include the vanilla baseline of using source risk directly for target validation, importance-Weighted
denoted as Source Val [7]. Additionally, Importance- Weighted Cross- Validation (IWCV)[30) and Deep Embedded Validation (DEV) [18] re-weight the source risk based on density similarity between domains. Reverse Validation (RV) [31] builds a symmetric UDA task to estimate target risk by reversing the source risk. However, source-based methods require training extra models involving source data and are vulnerable to severe distribution shifts between domains. On the other hand, the second type, target-only validation methods, is more straightforward and only utilizes unlabeled target samples. Entropy [30], based on the assumption of low-density separation [32], selects the model that produces target predictions with the lowest mean entropy. Soft Neighborhood Density (SND) [29], which relies on the assumption of neighborhood consistency, chooses the model with the highest consistency of predictions within the target neighborhood. While both Entropy and SND outperform source-based methods in certain UDA tasks, they have not fully explored the learned target structure for model selection, leading instability across various UDA methods.
In this paper, we introduce Mix Val, a novel target-only method for model selection in UDA. Mix Val leverages mixed target samples validation probes. To achieve this, Mix employs mixup [33] with unlabeled target samples and their predictions, generating pseudo-labeied mixed samples. These mixed samples are further categorized into two types, depending on whether mixup is performed intra-cluster or inter-cluster. Mix Val takes a novel strategy by evaluating inference-stage interpolation consistency for both types of mixed samples. Compared to Entropy and SND, Mix Val innovatively employs mixed samples for probing the target structure, rather than directly measuring certain properties of target predictions. Moreover, Mix Val elegantly combines the ad adaatagees of both Entropy and SND. On one hand, Mix employs intra-cluster mixed samples to evaluate neighborhood density, leveraging the neighborhood consistency assumption utilized in SND. On the other hand, it utilizes inter-cluster mixed samples to evaluate the classification boundary, considering the low- density separation assumption used in Entropy. The novel probing allows Mix Val to benefit from the strengths of both assumptions, making it a competitive UDA model selection method.
Our main contributions are highlighted threefold:
We study the significant yet under-explored model selection problem in UDA, providing comprehensive empirical evaluations of existing validation baselines.
We introduce Mix Val, a novel target-only validation method that directly probes the target structure using mixed samples. Mix Val combines the strengths of both SND and Entropy through novel consistency-
probing with inter-cluster and intra-cluster mixed samples.
We conduct extensive experiments to evaluate Mix Val's effectiveness in model selection across various UDA settings, including closed-set UDA, partial-set UDA, open-partial- set UDA, and source-free UDA. The results highlight Mix Val's exceptional stability and superior performance in UDA model selection compared to existing baselines.
'Throughout this paper, use model selection, validation, and hyperparameter selection interchangeably.

---

Table 1: Comparison of assumptions considered in validation methods using target data.
neighborhood consistency
prior of class diversity
low-density separation
Validation Method
Entropy [42]
InfoMax [28]
X
X
SND [29]
Corr-C C [43]
X
Mix Val
/
Related Work
2.1 Unsupervised Domain Adaptation
Domain adaptation aims to leverage labeled source domains to facilitate transductive learning in label-scarce target domain [34]. Unsupervised domain adaptation (UDA), which assumes that the target domain is entirely unlabeled, has gained significant attention due to its practical nature. Many traditional methods have been proposed to address UDA [35-38]. In recent years, deep learning-basse based UDA methods have witnessed significant progress, with domain adversarial learning emerging popular approach. Adversarial domain adaptation has been explored multiple levels, including input-level [39]. feature-level [7, 9, 15], prediction-level [11, 22], and entropy-level [12]. Additionally, there has been growing interest in adapting UDA models with a specific focus on the leverage of target data, leading to explorations in techniques such as target clustering [40, 27, 41] and target output regularization [24, 25]. Notably, Xu el al. [23] proposed feature-level regularization approach that diverges from the popular mainstream methods. These UDA methods primarily address conventional scenarios that require source data for adaptation. More recently, practical variant called source-free UDA has gained increasing attention [19, 20]. where adaptation relies solely on the source-trained trained model. To comprehensively compare different validation approaches, we conduct hyperparameter selection for various representative UDA methods across diverse UDA settings.
2.2 Model Selection in Unsupervised Domain Adaptation
Model selection in UDA poses significant challenge due to the absence of labeled target data and the presence of domain distribution shifts between domains. Some pioneering efforts have been made to address this issue. Ganin el al. [7] initially introduced the estimation of the unavailable target risk using source risk for target-domain validation (Source However, Source Val is susceptible severe domain shifts. Later, Ganin et al. [31] proposed Reverse Validation (RV), which considers a symmetric UDA problem from target to source and employs the reversed source risk for validation. Sugiyama et al. [30] introduced Importance- Weighted
Pross-Validation (IWCV), which estimates the target risk by re- re-weighting the source risk based on input-level domain similarity. You al. [18] further proposed Deep Embedded Validation (DEV), which considers feature-level similarity and controls variance in IWCV. While these source- source-based methods have proven effective, their practical applicability is limited by the need of extensive model training, access source data, and the challenge of dealing with severe domain shifts. Consequently, recent efforts have shifted towards target-only validation methods, which rely solely unlabeled target data. Morerio el al. [42] pioneered the use of mean entropy (Entropy) of target predictions for validation, considering the assumption of low-density separation [32]. Musgrave al. [28] introduced the Information Maximization (InfoMax) score, which further considers class diversity in addition to Entropy. Tu al. [43] introduced Corr-C, which prioritizes predictions with both large class diversity and high confidence. However, class diversity serves strict prior that is unsuitable for UDA scenarios with label shift [44]. Saito et al. [29] introduced Soft Neighborhood Density (SND), the state-of-the -tt art target-only validation method that prioritizes high neighborhood consistency. SND has demonstrated that target-only validation can outperform source-based methods, including IWCV and DEV. We also align with the research line of target-only validation, which offers simplicity and adaptability across various UDA settings. Table showcases a comparison of how our method Mix Val differs from other target-only validation methods in terms of the consideration of common assumptions.
3

---

2.3 Mixup
Mixup, a data augmentation technique originally proposed by Zhang el al. [33], is used during model training to improve generalization. Mixup has been extended different levels, such as feature-level [45], patch-level [46], and token-level [47]. In addition to supervised learning, mixup has been successfully applied in semi-supervised learning [48, 49] and domain adaptation [50-52]. Unlike existing studies that consider mixup as a small perturbation to regularize model training, we employ mixup to generate in-between probing samples for model evaluation.
Methodology
3.1 Problem Setting
Unsupervised domain adaptation (UDA). We begin by introducing the UDA problem with a K-way image classification task, easily extendable semantic segmentation tasks. In this setup, are provided with a labeled source domain D, = { N²) consisting of nts labeled images v, with their corresponding labels y/. and an unlabeled target domain D. = { containing n, unlabeled images Zt. Here, 2g and Ty represent image vectors, and y/s are one-hot ground truth labels. The objective of UDA is to learn a discriminative model M using D3 and D., which can accurately predict the target labels {vt) for the corresponding target images {ar 17 under covariate shift [30] or label shift [44] between domains. Each image vector E Rd in the input space is associated with a one-hot pscudo label encoding, denoted as ý E RK which is predicted by model M. We denote |C,| K the number of categories in the source domain and |G, as that in the target domain. Besides the vanilla UDA, which refers to closed-set UDA where both domains share the same label space (C., C,), we also investigate other challenging UDA settings to evaluate the versatility of model selection approaches. These include partial-set UDA [15], where the source domain contains more categories than the target domain (C, C)), and open-partial-set UDA [53], where both domains have overlapping but not identical labels (C, nc, + Ø, C. Z C. C. & C))
Traditional UDA settings [34, 6, 15, 53] typically require simultaneous access to source and target data for effective target adaptation. The general optimization objective can be represented as follows:
Luda = Cave (M: Pa)+ + &Cadapt (M; D..
Here, Care is the cross-entropy loss with labeled source data, and Cadapt denotes the adaptation loss that adapts the training of model M to unlabeled target data. The scalar coefficient | is a common type of hyperparameter, and 17 is hyperparameter specific to the adaptation loss. Additionally, there are other training-relan related hyperparameters, such as learning rate, iterations, and network architccture.
Recently, source-free free unsupervised domain adaptation (SFUDA) [19, 20] has gained significant attention in the research community. Unlike traditional UDA, SFUDA decouples the source supervised learning and unsupervised target adaptation into two sequential stages. The target adaptation stage in SFUDA is typically guided by the following objective function:
Latuda Ladapt (M; Ma, Dr,
Here, M, represents the source pre-trained model. During the target adaptation stage, a target model M is learned with access to the source model M, and unlabeled target data {=
Model selection. In UDA, the goal of model selection is to pinpoint the hyperparameter configuration that offers the best performance on unlabeled target data. For example, let's consider the hyperparam- eter 3, with set of possible values { 1, where m is the number of candidate values. We proceed to train m distinct models [M,) mm each employing different 3 value. Our aim is identify the model M that delivers the highest performance on D, among all candidate UDA models { [M, )m thus determining the optimal hyperparameter value. Figure 1(a) visually illustrates this process, using 3 as the value for m. Model selection in UDA confronts two key challenges. Firstly, the absence of labeled target data renders traditional supervised validation methods, such as using a hold-out dataset [54], unfeasible. Secondly, the presence of severe domain distribution shifts between domains poses risk when using source data for selecting the best model for the target data. Additionally, concerns about source data privacy can limit direct source data utilization.

---

Training Stage
Hyperparamecerr f
MixVal
Source Images (Labeled)
models
UDA Method
M₁
M2
Ma
Target Images
Validation Stage
(Unlabeled)
MixVal
Real target samples
Intra-cluster mixup
M2 is the best model
M2
P Inter-cluster mixup
target domain, select f2!
(a) Pipeline of validation in UDA with Mix
(b) Two types of probes
Figure I: Overview the proposed method (MixVal) for model selection in UDA.
3.2 MixVal: Mixed Samples as Probes for Unsupervised Validation
Motivation. While existing target-only model selection methods [42, 29] have demonstrated com- petitive performance, they sometimes result in extremely poor selections when applied to different UDA methods and datasets, even within the standard closed-set UDA scenario. This phenomenon is substantiated by comprehensive large-scale empirical study conducted by Musgrave el al. [28]. We highlight two common limitations in existing target-only validation methods that potentially hinder their ability to generalize across various UDA scenarios. First, these methods primarily rely on raw target predictions to calculate specific metrics for evaluating the learned target structure. However, it's important to note that these target data have already been utilized during the transductive model training. Second, while both the assumptions of low-density separation and neighhorhood consistency are effective for model selection, existing validation methods often design their metrics based solely of these assumptions, summarized in Table 1.
To address these limitations, we present Mix Val, target-only validation method that thoroughly evaluates the target structure through the use of mixed samples. We employ the mixup technique [33] to create novel in-between target samples. These mixed samples are then used for consistency evaluation probe the learned target structure. With two distinct types of mixed samples, ensure a comprehensive assessment of the target structure acquired by each candidate UDA model.
Generation of mixed samples. We first generate labeled set of mixed samples using mixup with unlabeled target data and inference-stage UDA models. Specifically, we create a mixed sample Fmix and label ymix by performing convex combination of pair target samples of, af and their corresponding pseudo labels of, jf which are predicted by the UDA model. The process of mixed sample generation is formulated as follows:
Tmix
xf + (1 X) xtf
Mmnix A * olt + (1-
where À is scalar used for interpolation, and o and of denote the one-hot pseudo label encodings. By performing mixup on random target samples for a single epoch in the inference stage, we generate set of mixed samples {[ vinx) ) p where n, represents the total number.
Interpolation consistency evaluation. Next, we leverage each candidate UDA model to infer labels for all mixed samples, resulting in predicted labels (úft ) Using both the mixed labels ( y and predicted labels, we assess the accuracy of mixed samples for each candidate model. This assessment yields the Interpolation Consistency Evaluation (ICE) score, defined as follows:
ICE = Cecuracy((y'iiz/) 1 ()
For all UDA models (M.) we apply the same mixup operation to generate identical mixed samples, but the mixed labels differ because model predictions for target samples vary among different models. subsequently calculate the ICE score for each candidate model M,, yielding set of ICE scores JM The candidate model with higher ICE score is deemed superior.
5

---

MixVal via two types of probes. Through meticulous analysis of the mixup operation applied to unlabeled target samples and their pseudo labels. we have identified two distinct categories of mixed samples, as illustrated in Figure ((b). The first type involves mixing target samples with the same pseudo label, referred to as intra-cluster mixup. When we evaluate the ICE score after performing intra-cluster mixup, we effectively measure the neighborhood density within each cluster. A higher ICE score indicates a higher level of neighborhood consistency [29]. Conversely, the second type involves mixing target samples with different pseudo labels, termed inter-cluster mixup. By evaluating the ICE score in this context, we can assess the classification margin between clusters. A higher ICE score in inter-cluster mixup signifies larger margin of the classification boundary, indicating low-density separation [42]. For comprehensive probing, Mix Val uses both types of mixed samples as probes. It separately ranks all candidate models in ascending order based each type of ICE score. Then, Mix Val takes the average rank from both rankings to select the candidate model with the highest average rank. Kindly refer to Appendix A for the pseudocode of Val.
Connection to interpolation consistency training. We provide a comprehensive comparison between our MixVal approach and interpolation consistency training (ICT), a commonly used technique in semi-supervised learning [48]. While our ICE score draws inspiration from ICT, there are three significant distinctions between Mix Val and ICT. ICT primarily employs inter-cluster mixup as a minor training perturbation to regularize model learning with unlabeled data. In contrast, Mix Val explicitly applies mixup to create in-between target samples, encompassing both inter- and intra-cluster scenarios, during the inference stage without requiring model re-training.
4 Experiments
4.1 Setup
Datasets. For image classification tasks, we consider 4 popular UDA benchmarks of varied scales. Office-31 [55] is a classic domain adaptation benchmark consisting of 31 object categories across domains: Amazon (A), DSLR (D), and Webcam (W). Office-Home [56] is a challenging benchmark with 65 different object categories in domains: Art (Ar), Clipart (CI), Product (Pr), and World (Re). VisDA [57] is a large-scale benchmark for the synthetic-to-re
object recognition task, featuring 12 categories. It consists of a training (T) domain with 152k synthetic images and a validation (V) domain with 55k realistic images. DomainNer [58] is recent large-scale benchmark comprising approximately 600k images across 345 categories in 6 distinct domains. For evaluation, we focus on subset of 126 classes with 7 tasks [59] from 4 domains: Real (R), Clipart (C), Painting (P), and Sketch (S). For semantic segmentation tasks, we use the synthetic GTAV [60] dataset as the source domain and the real-world Citysaaee [61] dataset as the target domain.
UDA methods. We use the validation baselines discussed in Section 2.2 to conduct model selection for various UDA methods, with a specific emphasis on the practical target-only baselines. For closed- set UDA, we consider ATDOC [27]. BNM [24]. CDAN [9], MCC [25], MDD [22], SAFN [23], DMRL [50], AdaptSeg [11], and AdvEnt [12]. For partial-set UDA, we consider PADA [15] and SAFN [23]. For source-free UDA, we consider SHOT [20]. For open-partial-se UDA, we consider
DANCE [41]. For ATDOC, BNM, CDAN, PADA, SAFN, SHOT, DMRL, and DANCE, we select the loss coefficient among 7 varied candidate values. For MDD, we validate the margin factor, while for MCC, we validate the temperature. For AdaptSeg and AdvEnt, validate both the loss coefficient and training iteration. For MCC and MDD, we also include a two-hyperparameter validation task, where the bottleneck dimension is considered as additional hyperparameter, with possible values. Kindly refer to Appendix B for details of hyperparameter settings.
Implementation details. We utilize the Transfer Learning Library' to train UDA models single RTX TITAN 16GB GPU. The batch size is set to 32, and the total number of iterations is set to 5,000. We save the final model as checkpoint. For VisDM and G7AVVCityscaees, DomainNer, we use ResNet-34 [3], and for the other benchmarks, we use ResNet-50 [3]. Regarding SND [29], we employ the official implementation. For basece methods, we split 80% of the source data as the training set and the remaining 20% as the validation set. In our Mix Val approach, we use a fixed value of X to ensure a fair comparison among candidate models. Specifically, we set A = 0.55 for all experiments, which allows us to probe with in-between samples.
use ResNet- 101 [3], for
hattps://githae wwtp://wwwwt..aeaeeeeeeeeeeeeeee Learning- Library

---

Table 2: Validation accuracy (%) of closed-set UDA Office- Hooee (Home). bold: Best value.
Method
ATDOC [2?]
BNM
CDAN
+A
-P
-Re
agg
-AAr
-CI
-+P
RRe
avg
-
+0
-PP
-Re
sgg
Source Val
66.63
52.54
78.37
76.61
68.59
62.44
50.71
17.55
74.76
66.37
55.00
12.65
69,50
68.81
58.99
IWCV [30]
67.97
51.03
78.31
79.26
69.89
66.56
18.16
74.09
73.28
65.52
61.31
11.24
67.17
71.93
60.41
DEV [18]
54.23
77.78
79.55
69.
65.76
73.92
77.59
67.23
57.04
68.76
76.91
67.49
RY [31]
68.68
56.13
78.93
79.64
70.85
68.25
56.75
78.08
78.67
70.44
67.66
56.74
76.01
77.68
69.52
Entropy [12]
63.67
55.83
76.54
78.36
68.60
66.28
51.19
74:15
77.61
68.14
67.06
57.55
76.37
77.45
69.76
InfoMax [28]
63.67
55.63
77.61
78.
68.82
66.28
54.49
74.15
77.64
68.14
67.66
57.56
76.37
77.45
69.76
SND [29]
55.63
76.54
77.54
68.34
51,19
74.15
77.64
68.14
67.94
57.56
76.96
77.68
Cor-C [13]
63.51
50.39
73.89
73. 83.88
65.12
58.10
45.37
68.97
70.59
60.76
53,84
11.21
61.96
67.65
56.91
MixVal
66.47
56.87
78.14
79. 20
70.17
67.36
56.18
10
78.12
69.44
67.71
$7.78
76.89
77.76
70.03
Worst
62.89
50.39
73.89
73.88
65,26
58.10
45.37
68.96
70.39
60.75
$3.80
11.21
61.78
67.65
56.86
68.97
58.35
80.27
80.58
12.04
68.93
57.51
78.43
79.57
71.11
68.19
57.90
77,44
78.19
70.43
Method
-Ar
-XC
MCC (25)
->rr
-RRe
avg
-Aa
-CCI
MDD (22)
--Pr
-<Re
avg
-AA
-CC
SAFN (23)
-Pr
-Re
uvg
Home
AVG
Source Val
66.37
56.33
79.35
30138
70.89
62:31
14.41
75.27
75.35
66.94
61.34
31.3 11:34
73.86
74.34
65.77
66.26
IWCV
68.69
58.93
80.37
50.08
72.02
64.20
56.50
73.78
74.28
67.19
64.31
22.36
72.31
74.29
65.82
66.81
DEV [18]
68.81
58.07
78.54
50.00
71.38
75.94
63.15
50.47
71.20
74.54
RV [31]
70.40
58.80
80.63
80 39
72.56
66.57
55.75
76.60
76.90
68-M0 946
64.31
50.13
78:27
74.93
65.78
69.68 68
Entropy [42]
69:29
59,33
50.63
S0.96
72.55
66.54
57.03
77.27
77.45
69.72
39,85
46.41
72.51
73.18
62.99
68.63
InfoMax [28]
66.58
58.48
79.12
71.25
73.77
68.84
SND (29]
69.05
55.61
79.72
79
70.87
51,34
38.01
77.61
68,46
58.86
57.90
46.41
67.04
68.
18
59.88
66.02
Cor-CC (43)
69.05
55.61
79.72
79.
70.87
47.79
31.69
63.40
60.63
50.88
62.
46.41
68.83
68.
61.52
61.06
Vix
69.79
59.24
80.47
80 74
72.56
65.73
58.01
77.36 36
76.91
69: 50
65.98
53.14
74.76
75.40
67.32
69.84
Wrest
62.72
76.19
67.93
31.79
37.00
45
68
26
Best
70.68
59.95
80.93
81.02
73.14
66.75
58.36
77.61
77.45
70.04
66.39
53.1
74.90
75.57
67.55
70.72
Table 3: Validation accuracy (%) of closed-set set UDA on Office-31 (Office) and VisDA.
Method
-AA
-11
ATDOC1221
WW
av8
1
-AA
->13
BNM [24]
-WW
T-VV
->A
-11
CDAN
-WW
zgg
T-vV
Souneeal
72.56
88.96
87.50
83.11
67.79
72.97
90.36
89.43
84.34
70.51
65.90
91.16
89.06
61.37
64.50
IWCV (30)
72.56
86.14
$6.54
$1.75
67.79
72.92
55.54
89.43
82.63
76.94
63.90
69.08
58.74
63.91
64.30
DEV [18]
72.56
86.14
86.54
81.25
70.34
72.92
85.54
89.43
82.63
76.94
63.90
91.16
58.30
81.12
64.50
RV [30]
74.93
K7:23
54.04
89.43
74.58
73.27
91.16
58.30
34.24
70.02
Earmpy [42]
87,80
$2.41
89.06
$3.95
IndoMax: [28]
73.29
87,50
70.52
$5.54
79.73
71.62
91.16
$3.69
SND (29)
73.29
92.37
87,50
$4.49
77.37
55.54
83.14
81.04
69:65
71.55
92.57
58.55
54.16
S8.46
Con-CC (43)
71.05
90.96
84 40
82.14
63.79
67.16
84.34
78.99
76.83
70.51
39.29
67.67
59.02
61,86
64.50
MisVall
73.61
90.96
86. 54
83.70
77.37
74.97
56 48
87.00
82.81
74.51
72.78
92.64
89.06
84.81
88.46
Woest
71.05
56.14
B1.10
50.53
62.85
67.16
811.11
75.99
76.83
23.00
38.29
67.57
57.11
61.02
64.50
Best
75,31
92.37
87.30
85.16
77.37
75.52
90.36
89:43
85,10
76.94
73.38
92.77
89.06
$5.07
50.46
Method
->A
-D
MCC (2:5
-WW
T-VVV
-1A
-D
MDD [22]
-WW
T-V
-1.
-11>
SAFF [25)
-W
Bvg
T-V
Office
AVG
WiDA
AVG
SourceVal
90:95
91.07
35.05
91.06
86.25
84.34
12.25
09.20
85.75
87.17
50.03
TWCV [30]
88:55
64.27
91.16
85.25
72.25
79.75
71.55
[18)
72.20
93.08
75.65
91.16
72.25
68.21
73.38
$2.36
71.97
RV [31]
73.97
89.06
93.00
85.37
92-57
84.61
77.23
69.69
90.83
87.17
52.23
66.33
83.90
75.62
Enmmpy [42]
73.93
90.56
93.46
85.00
82.22
76.31
92:57
90.82
85.57
78.95
68:23
91.57
85 66
81.87
70.20
83.53
72.17
TnfaMax [28]
73.93
89.16
88:55
83.88
81 48
76.50
92.57
90.82
86.63
78.95
68.23
91.57
N7.42
82.41
70.20
83.13
74.32
SND [29]
73.93
91.97
93.46
$6.45
89.35
76.50
92.17
90.82
85.50
78.95
68.23
89.90
85.66
$1.28
58.15
$3.99
72.32
Con-C (43)
23.93
91.37
93.46
86.25
69.35
74.25
91.57
85.66
83.83
72.25
09.39
86,75
80.38
73.51
02.52
78.24
07.82
Mix
74.09
91.77
93.31
86 36
81 48
75.97
91.77
91.74
86.49
76.95
69.61
89.90
86
87.
74.41
84.30
77.86
"Weest
86.75
77.13
15
74442
93.16
92.57
92.20
78.95
70.06
91.57
$3.02
Table 4: Validation accuracy (%) of closed-set UDA on DomainNet.
CDAN
BNM
[24]
ATDOC
[27]
Method
C
.P
R
-
avg
-> C
R
>>5
avg
-> C
>.
R
avp
Entropy [-1]
67.09
65.30
74,42
59.54
66.66
63.36
69,28
74.31
48.60
62.66
63.75
61.35
79.60
52.17
64,34
InfoMax [28)
67.09
74.42
66.66
67.05
64.28
55.67
65.33
63.75
61.85
79.60
52.17
64.34
SND [29]
67.09
64.68
74.42
59.34
66.38
56.56
54,50
74.31
42.37
56.93
63.75
61.85
79.60
47.00
63.05
Cor--C [43)
57.35
62,88
74.42
54.63
62.32
59.75
63.41
77.62
42.37
60.79
59.98
62.27
74.42
53.69
62.59
Mix
67.09
65.80
74.42
59.34
66.66
67.84
66.40
78.68
58.49
67.85
68.94
68.44
79.60
61.73
69.68
Worst
57.35
60.76
73.44
51.41
60. 74
55.79
54,50
74.31
42.37
56.74
59.98
61.85
74.42
47.00
60.81
Best
67.09
65,80
74.44
66.66
67.86
66.50
78.68
58.49
67.88
70.30
68.44
62.23
4.2 Results
We report Val's validation performance as averages from three random runs. In the tables, 'Best' signifies the optimal selection, and "Worst" represents the worst one. For brevity, we report average results for tasks with the same target domain. Kindly refer to Appendix D for detailed results.
Closed-set UDA. We begin by comparing the performance of validation baselines within the standard closed-set UDA scenario. Specifically, we report the results for both sasree-based and target-only validation methods across several well-established UDA benchmarks. Table 2 provides the results on the medium-scale benchmark Office-Homee Home, while Table 3 presents the results on the small-scale benchmark Office-31 31 and the large-scale benchmark VisDA. Furthermore, Table 4 presents the results
7

---

Table 5: Validation accuracy (%) of partial-set UDA Office-Homee Home.
PADA
SAFN
Method
++ Ar
CI
Pr
-3 Re
avg
- Ar
+ C1
- Pr
- Re
Source
41.90
71.89
58.87
66.82
54.71
74.41
76.45
68.
IWCV [30]
59.65
50.51
66.84
72.96
62.49
69.36
53.91
71.78
76.38
67.86
DEV [18]
49.29
70.46
64.76
69.36
54.94
73.95
76.06
RV [31]
57.79
40.87
63.87
70.83
58.34
68.98
52.74
72.83
77.14
67.92
Entropy [42]
60.08
46.51
53.16
62.47
55.56
71.75
55.62
76.36
76.59
70.08
InfoMax [28]
60.08
51.40
60.20
66.67
59.59
63.67
51.74
69.64
73.62
64.67
SND (29]
67.80
50.71
59.46
67.13
61.27
71.75
51.74
76.36
78.36
69.55
Corr Corr-Cle [43)
61.34
45.65
54,90
62.25
56,04
71.23
55.70
76.94
79.13
70.75
MixVal
67.68
51.01
72.94
78.64
67.57
71.70
57.91
77.08
78.94
71.41
Worst
56.29
39.76
50,49
59.31
46
62.48
49.91
68.50
73.62
63.63
Best
69.33
55.86
74.55
79.59
69.83
73.37
58.09
77.35
79.33
72.03
Table 6: HOS (%) of open-partiall-et UDA and accuracy (%) of source-free UDA.
Method
DANCE [41]
Home
[20]
Office
Vi+DA
-AAr
-+C1
-+rr
RRc
-+.A
-D
-W
T~
Entropy [42]
32.00
27.52
82.65
InfoMax [28]
32,00
39.48
27.52
38.01
34.25
71.67
90.76
88.68
83.70
82.65
SND 129]
15.05
4.33
23.75
16.79
14.98
71.67
90.76
88. 68.68
83.70
82.65
Cor-CC
29.60
4.33
23.75
16.79
18.62
71.58
90.76
90.19
84.18
82,65
Mix
71.54
52,90
78.61
65.01
67.01
72.04
92.37
92.32
85,58
83.12
Worst
15.05
4.33
15.17
16.79
12.84
71.56
90.76
88.68
83.67
80.57
Best
77.01
66.29
78.81
69.81
72.98
75.06
94.78
93.33
87.72
83.12
of practical target-only validation baselines on the large-scale benchmark DomainNet. The current ofate-of-the-art source-based baseline is DEV [18], and for target-only methods, SND [29] holds the state-of-thes position. Nonetheless, our observations differ from previous findings. When consider- ing the averaged validation accuracy across 6 popular UDA methods, we note that RV consistently outperforms other suurce-based validation methods across three benchmarks. Regarding target-only validation methods, entropy-based approaches like Entropy and InfoMax prove be competitive, particularly demonstrating a significant advantage over SND on the Office- Home benchmark. As expected, our Mix Val consistently attains the highest average accuracy across all four benchmarks, surpassing the vanilla Source Val baseline, while most existing validation methods may noticeably underperform in comparison this baseline. Notably, Mix Val demonstrates exceptional performance on large-scale benchmarks such as VisDA and DomainNet, approaching the upper bound of 'Best' and outperforming the second-best method by substantial margins. MixVal's consistent advantages in closed-set UDA highlight its effectiveness in this widely studied setting.
Partial-set UDA. Following [29]. assess validation performance in partial-set UDA, scenario with label shifts between domains. We validate two representative partial-set UDA methods, PADA and SAFN, and present the validation accuracy of Office- in Table 5. We observe that Mix Val consistently outperforms all other validation baselines, maintaining performance close to the 'Best'. However, notice some differences in comparison closed-set UDA. Specifically, SND achieves more stable validation performance than Entropy, because Entropy can be susceptible to structure collapse [29]. Second, methods emphasizing 'class diversity', such InfoMax and Corr-C, often yield lower results than VolrceVal. Additionally, RV consistently underperforms Source in scenarios without symmetric label distribution, making it unsuitable for UDA with label shifts.
open-partial-set UDA. We extend our evaluation UDA scenarios with label shifts, conducting validation for popular open- partial-set UDA method, DANCE. Table 6 presents the HOS [62. 63] results on Office-Home. Home. Mix Val consistently provides strong validation performance, nearing the 'Best', while other validation baselines tend to align with the 'Worst'.
Source-free UDA. Finally, we compare the target-only validation baselines in source-free UDA. We conduct validation for popular method, SHOT, and report the accuracy of Office-// and VisDA in Table We find that Mix Val consistently outperforms other baselines this practical setting.
4.3 Analysis
Influence of pseudo label quality. Because Mix Val utilizes pseudo labels of target data for the mixup operation, it is crucial to assess the impact of pscudo label quality. When examining UDA tasks with

---

SELA44
Cutta
Tushicc
Accuracy
SND
MixVal
48-
256
54.050.244
12
1
20
57.5
60.3
9
5
17
24
26
7
1024
56.
08.9999
59.8
(66.551.447.. 347.3
10
16
20
23
21
18
22
19
5
10
2048
59.455553335
15
25
27
1.0
1.5
2.0
2.5
3.0
3,5
4.0
1.0
1.5
2.0
2.5
3.0
3.5
4.0
1.0
1.5
2.0
2.5
3,0
3.5
4,0
CRAX WXDA
PARK Ontre
Figure 2: (a) presents the effect of different consistency evaluation techniques. (b) provides compar- isons between SND and our Mix Val on a validation task involving two hyperparameters: MCC on the Ar-Cl, where y represents the bottleneck dimension and 1) represents the temperature. The SND and Mix Val scores of the 28 candidate models are ranked in ascending order. To enhance clarity, we include the real target accuracy for each model in the left figure labeled as 'Accuracy'.
low-quality pseudo labels (around 50% accuracy), such as the results of '-> CI', it becomes evident that Mix Val consistently maintains stable validation performance.
Influence of consistency evaluation. Mix Val utilizes image-level mixup-based consistency evaluation for the target probing. To gain a deeper understanding of its role in MixVal, we substitute it with other consistency evaluation techniques, including instance-based augmentations such RandAug [64] and SSL Aug [65], as well as mixup at other levels, such as CutMix [46] and FeatMix [45]. The results in Figure 2(a) consistently demonstrate that image-level mixup outperforms other strategies.
Table 7: Two-hyperparameter validation accuracy (%) of closed-set UDA on Office- Home.
MDD [22]
MCC [25]
Method
Ar -+ CI
CI
- Pr
Pr +
Re +
avg
Ar -
CI + Pr
Pr -
Re
Rs - Ar
avg
Source Val
55.99
13.15
78.77
69.39
69.33
57.91
76.84
81.13
72.89
72.19
IWCV [30]
37.89
72,92
80.42
58.43
62.42
46.09
77.74
80.68
74.45
69.74
DEV (18]
$2.60
72.11
53.36
67.70
61.44
59.47
76.84
81.94
74.08
73.08
RV [31]
$7.59
72.25
80.83
70.79
70.37
59.13
76.84
82.03
71.98
72.50
Entrupy [42]
57.21
73.19
B0,U6
72.31
70.69
59.75
71.77
82.37
74.33
73.56
InfoMax [28]
57.59
72.92
80.06
72.31
70.72
59.70
78.73
82.58
70.33
72.84
SND [29]
38.10
56.45
70.03
65.10
57.42
53.49
74.97
77.25
74.12
69.96
Com-C [43]
30.17
44.74
57.15
50.76
45.71
44.90
56.75
74.32
67.61
60.90
Vix
55.99
72.63
80.27
72.12
70.25
60.08
78.52
81.95
74.43
73.75
Worst
30.17
39.81
53,36
50.76
43.53
43.02
56.75
73.47
67.24
60.12
Best
57.59
73.35
80,93
72.52
71.10
61.10
78.94
83.04
75.36
74.61
lww-hyyperparamete validation. We further evaluate all validation baselines in challenging tasks
that involve two hyperparameters. Quantitative comparisons in Table 7 consistently position Mix Val as one of the top-performance validators. Qualitative comparisons in Figure 2(b) clearly illustrate that Mix Val scores demonstrate a strong correlation with actual accuracy, whereas SND [29] scores exhibit noticeable deviation, resulting in selections far from optimal.
Table 8: Validation accuracy (%) of mixup-based closed-set UDA method DMRL [50].
DMRL
[50]
Home
DMRL [50]
Office
Method
Ar
+CI
PPr
RRRe
avg
+
+D
-W
avg
Entropy [42]
58.14
50.25
69.06
71.37
62.20
63.67
80.52
86.67
76.95
InfoMax [28]
58.14
50.75
69.06
71.37
62.33
63.67
80.52
86.67
76.95
SND [29]
57.74
49.96
69.28
71.42
62.10
61.84
84.14
86.67
77.55
Corr-C [433
58.29
49.46
68.67
71.73
62.04
60.23
77.51
81.13
72.95
Mix Val
59.13
50.41
69.28
72.07
62.72
64,89
82.93
86.67
78.16
Worst
57.71
48.99
67.78
70.72
61.30
60.23
76.31
81.13
72.55
Best
59.20
50.75
69.28
72.24
62.87
65.30
84.14
86.67
78.70
Validation of misup-based UDA method. The ICE score of Mix Val is based on the evaluation of mixup-basee based interpolation consistency during the inference stage. To assess Val's robustness, we apply it in the validation of the UDA method DMRL [50], which incorporates mixup-basee based consistency training within the target domain and across two domains at the same time. Mix Val is employed to determine the optimal consistency loss coefficient for DMRL, and the results are reported in Table Notably, Mix Val consistently outperforms most of the existing target-only validation baselines, even when subjected to the 'attack' posed by specialized UDA model training.

---

70-
65.
Method
BNM [24]
Accuncyyt
Method
AdaptSegtt ||
AdvEnt [12]
Entropy [42]
28.21
Source
39.52
39.08
InfoMax [28)
28.21
AccArg
Entropy [42]
39.47
38.41
SND [29]
52.42
Bat
Ranklater
SND [29]
40.69
40.02
Cor--C C [43]
28.21
Rasking
Renkinn
MixVal
42.20
40.02
Mix Val
54.78
Rank/vg Son
Worst
33.84
33.06
Worst
28.21
0.5
0.5
0.7
0.8
0.9
Best
42.20
41.78
Best
Figure 3: Analysis of probing.
Table 9: Segmentation mloU (%).
Table 10: ViT results.
Analysis of probing. In Mix we generate mixed samples by applying mixup with mix ratio À of 0.55 to both target samples and their hard pseudo labels. We then calculate ICE scores for both intra-cluster and inter-cluster mixed samples. The final score for model selection is determined by averaging ascending rankings of both types of ICE scores, Here, we present a comprehensive ablation study to delve into these design choices within VixVal. To do this, we conduct validation experiments for PADA on four partial-se set UDA tasks (Ar-CI,, CI-Prr, Pr-Re, Re->An and report the average accuracy results in Figure values) as the final score for model selection. 'RankAvg' denotes the use of the average ranking of ICE scores for selection. 'RankInter' relies solely on inter-cluster mixed samples for probing. 'RankIntra' employs only intra-cluster mixed samples for probing. 'Hard' and 'Soft' indicate the utilization of hard and soft pseudo labels, respectively. It's worth noting that for intra-cluster mixed samples, there is distinction between 'Hard' and 'Soft' labels, all pseudo labels involved are the same. The observations drawn from Figure 3 are as follows: (i) The mix ratio (A) influences the level of ambiguity in mixed samples. In Mix Val, À = 0.55 generates more ambiguous in in-between samples with greater probing capabilities than the easier samples created by = 0.9. Consequently, validation performance À 0.55 surpasses that at À 0.9. (it) In Mix Val, while soft pseudo labels may provide additional performance improvements for inter-cluster probing, the use of hard pseudo labels offers simplicity and stability advantages. (iii) The separate probing of each type of mixed samples demonstrates effectiveness, and the combination of intra-cluster and inter-cluster probing enhances Mix Val's stability. Kindly refer Appendix C for more discussions on probing.
'AccAvg' represents the direct average of ICE scores (i.e., accuracy
Extension to segmentation. We extend the use of Vix to the domain adaptive segmentation task on CTAV2Cityscapes following the same configuration SND [29]. Our model selection process involves 66 candidate models generated by varying two hyperparameters: the loss coefficient and training iteration. We save checkpoints every 2,000 iterations, starting from the 10,000th iteration to the 30,000th iteration. The results in Table demonstrate that Mix Val consistently delivers the top validation performance for both UDA methods, while Entropy consistently underperforms Source Val.
Influence of backbone. addition to ResNet backbones, we also utilize ViT-B [4] as the backbone for the validation of BNM on the task R-S. Accuracies reported in Table 10 highlight Mix Val's superior stability with VIT, especially in comparison to Entropy, InfoMax, and Corr-C
Limitations and impacts. While Mix Val excels in classification tasks, further research is needed to address its limitations. This includes establishing the theoretical foundations of probing data structure using pseudo-labeled mixed samples. Additionally, extending Mix Val to other machine learning tasks like object detection and regression presents challenges. Poor hyperparameter selections could potentially have negative societal impacts on the real-world deployment of UDA models. Nevertheless, have not observed such situations during our extensive evaluation of Mix
Conclusion
In summary, we introduce Mix novel and straightforward target-only method for model selection in unsupervised domain adaptation (UDA). Mix leverages inference-stage mixup to generate two distinct types of mixed samples, facilitating effective probing of the learned target structure while elegantly considering key assumptions used in previous approaches. Our extensive evaluations encompass diverse UDA methods and adaptation scenarios, consistently affirming the superior and reliable performance of Mix Val when compared to existing model selection methods.
10

---

Acknowledgement
This project is supported by the Singapore Ministry of Education Academic Research Fund project titled "Towards Robust Single Domain Generaliza-
Tier I (WBS: A-8001229-00-000,, tion Deep Learning". " Jian Liang is supported by the National Natural Science Foundation of China (Grant No. 62276256) and the Beijing Nova Program under Grant 201100002221000
References
[1] Russakovsky, O., J. Deng, H. Su, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
[2] Krizhevsky, A., I. Sutskever, G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 2012.
[3] He, K. X. Zhang. S. Ren, al. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 2016.
[4] Dosovitskiy, A., L. Beyer, A. Kolesnikov, al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Representations, 2021.
[5] Arjovsky, M., L.. Bottou, I, Guirajani, al. Invariant risk minimization. arXiv preprint arXiv: 190.002830 2019.
[6] Long, M., Y. Cao, J. Wang, al. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning. 2015.
Ganin, Y. V. Lempitskk, Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning. 2015.
[8] Saito, K., K. Watanabe, Y. Ushiku, al. Maximum classifier discrepancy for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition. 2018
Long, M., Z. Cao, Wang, et al. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems. 2018.
[10] Hoffiman. D. Wang. F. Yu, Fens in the wild: Pixel-level adversarial and constraint based adaptation. arXiv preprint arXiv: /622002699, 2016.
[11] Tsai, Y.H.,, W.-C. Hung, Schulter, al. Learning adapt structured output space for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[12] Vu, T.-H., Jain, Bucher, et al. Advent: Adversarial entropy minimization domain adaptation in semantic segmentation. In JEEE Conference on Computer Vision and Pattern Recognition. 2019.
[13] Chen, Y., W. Li, C. Sakaridis, al. Domain adaptive faster r-cnn for object detection in the wild. IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[14] Saito, K., Y. Ushiku, T. Harada, et al. Strong-weal distribution alignment for adaptive object detection. In IEEE Conference on Computer Vision and Pattern Recognition. 2019.
[15] Cao, Z., L. Ma. M. Long, et al. Partial adversarial domain adaptation. In European Conference on Computer Vision. 2018.
[16] Liang, J., Y. Wang, D. Hu, el al. A balanced and uncertainty-aware approach for partial domain adaptation. In European Conference on Computer Vision. 2020.
[17] Panareda Busto, J. Gall. Open set domain adaptation. In IEEE International Conference on Computer Vision. 2017.
[18] You, K., X. Wang, M. Long, et al. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning. 2019.
[19] Li, R., Q. Jiao, W. Cao, et al. Model adaptation: Unsupervised domain adaptation without source data. In IEEE Conference on Computer Vision and Pattern Recognition. 2020.
[20] Liang, J., D. Hu, J. Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference Machine Learning, 2020.

---

[21] Liang, J., R. He, T. Tan. A comprehensive survey test-time adaptation under distribution shifts. arXiv preprini arXiv:2303.15366 15361, 2023.
[22] Zhang, Y., T. Liu, M. Long, et al. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning. 2019.
[23] Xu, R., G. Li, Yang. et al. Larger norm more transferable: adaptive feature norm approach for unsupervised domain adaptation. In IEEE International Conference on Computer Vision. 2019.
[24] Cui, S., S. Wang, J. Zhuo, et al. Towards discriminability and diversity: Batch nuclear-nowd norm maximization under label insufficient situations. In IEEE Conference on Compider Vision and Pattern Recognition. 2020.
[25] Jin, Y., X. Wang, M. Long, et al. Minimum class confusion for versatile domain adaptation. In European Conference on Computer Vision. 2020.
[26] Zou, Y., Z. Yu, B. Kumar, et al. Unsupervised domain adaptation for semantic segmentation via class- balanced self-training. In European Conference on Computer Vision. 2018.
[27] Liang, D. Hu, Feng. Domain adaptation with auxiliary target domain classificr. In IEFE Conference on Computer Vision and Pattern Recognition. 2021.
[28] Musgrave, K., S. Belongie, S.-N. Lim. Benchmarking validation methods for unsupervised domain adaptation. arXiv preprint arXiv:2208.07.5600 2022.
[29] Saito, K., D. Kim, P. Teterwak, et al. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In IEEE International Conference Computer Vision. 2021.
[30] Sugiyama, M., M. Krauledat, K.-R. Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 2007.
[31] Ganin, Y., E. Ustinova, H. Ajakan, al. Domain aovearaal training of neural networks. Journal of Machine Learning Research, 2016.
[32] Grandvalet, Y., Y. Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems. 2004.
[33] Zhang, H., M. Cisse, Y. N. Dauphin, et al. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations. 2018.
[34] Pan, S. J., Q. Yang. A survey on transfer lcarning. IEEE Transactions on Knowledge and Data Engineering, 2009.
[35] Gong, B., Y. Shi, F. Sha, al. Geodesic flow kernel for unsupervised domain adaptation. In IEFE Conference on Computer Vision and Pattern Recognition. 2012.
[36] Fernando, B., A. Habrard, M. Scbban, al. Unsupervised visual domain adaptation using subspace alignment. JEEE International Conference on Computer Vision. 2013.
[37] Sun, B., J. Feng, K. Saenko. Return frustratingly easy domain adaptation. AAAI Conference on Artificial Inteligence. 2016.
[38] Pan, S. J., I. W. Tsang, J. T. Kwok, et al. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 2010.
[39] Hoffman, J., F. Tzeng, T. Park, et al. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference Machine Learning. 2018.
[40] Shu, R., H. H. Bui, H. Narui, al. A dirt-t approach unsupervised domain adaptation. arXiv preprint arXiv:1802.0 082008873,,000 2018
[41] Saito, K., D. Kim, Sclaroff, et al. Universal domain adaptation through self supervision. Advances in Neural Information Processing Systems. 2020.
[42] Morerio, P., J. Cavazza, V. Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. arXiv preprint /711: 2017.
[43] Tu. W., W. Deng, T. Gedeon, et al. Assessing model diitriidistiittionn generalization with softmax prediction probability baselines and correlation method, 2023,
12

---

[44] Azizzadenesheli, K., A. Liu, F. Yang, et al. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:1903.09734 2019.
2019.
[45] Verma, V., A. Lamb, C. Beckham, al. Manifold mixup: Better representations by interpolating hidden states. In International Conference Machine Learning. 2019.
[46] Yun, S., D. Han, S. J. Oh, al. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE International Conference Computer Vision. 2019.
[47] Chen, J.-N., S. Sun, J. He, et al. Transmix: Attend mix for vision transformers. In IEEE Conference on Computer Vision and Puttern Recognition. 2022.
[48] Verma, V., K. Kawaguchi, A. Lamb, et al. Interpolation consistency training for semi-supervised learning. IEEE Transactions on Neural Networks, 2022.
[49] Berthelot, D., N. Carlini, I, Goodfellow, al. Mixmatch: A holistic approach semi-supervised learning. In Advances in Neural Information Processing Systems. 2019.
[50] Wu, Y., D. Inkpen, A. El-Roby. Dual mixup regularized learning for adversarial domain adaptation. In European Conference on Computer Vision. 2020.
[51] Mao, X., Y. Ma, Z. Yang, al. Virtual mixup training for unsupervised domain adaptation. arXiv preprint arXiv: 1905.042 2019.
[52] Yan, S., H. Song. N. Li, et al. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:200J.0065 2020.
[53] You. K., M. Long. Z. Cao, al. Universal domain adaptation. In IEEE Conference Computer Vision and Pottern Recognition. 2019.
[54] Goodfellow, L, Y. Bengio, A. Courville. Deep learning. MIT press, 2016.
[55] Saenko, K., B. Kulis, M. Fritz, et al. Adapting visual category models new domains. In European Conference on Computer Vision. 2010.
[56] Venkateswara, H., J. Eusebio, S. Chakraborty, et al. Deep hashing network for unsupervised domain adaptation. In IEEE Conference Computer Vision and Pattern Recognition. 2017.
[57] Peng. X., B. Usman, N. Kaushik, et al. Visda: The visual domain adaptation challenge. arXiv preprint 1rXiv:1710.0692. 2017.
[58] Peng. X., Q. Bai, X. Xia, al. Moment matching for multi-souree domain adaptation. In IEEE International Conference on Computer Vision. 2019.
[59] Saito, K., D. Kim, S. Sclaroff. et al. Semi-supervised domain adaptation via minimax entropy. In IEEE International Conference Computer Vision. 2019.
[60] Richter, S. R., V. Vineet, S. Roth, et al. Playing for data: Ground truth from computer games. In European Conference on Computer Vision. 2016.
[61] Cordis, M., M. Omran, S. Ramos, et al. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference Computer Vision and Pattern Recognition. 2016.
[62] Fu, B., Z. Cao, M. Long, et al. Learning to detect open classes for universal domain adaptation. In European Conference on Computer Vision. 2020.
[63] Bucci, S., M. R. Loghmani, T. Tommasi. On the effectiveness of image rotation for open set domain adaptation. In European Conference Computer Vision. 2020.
[64] Cubuk, E. D., B. Zoph, J. Shlens, et al. Randaugment: Practical automated data augmentation with reduced search space. In IEEE Conference Computer Vision and Puttern Recognition. Workshops. 2020.
[65] Chen, X., H. Fan, R. Girshick, al. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020.
[66] Duda, R. O., P.P. E. Hart, et al. Pattern classification and scene analysis. Wiley New York, 1973.
13

---

A
Algorithm
The PyTorch-style pseudocode for our validation approach Mix is presented Algorithm 1.
Algorithm PyTorch-style pseudocode for Mix Val.
x: A batch of real target images with shuffled order.
lam: The mix ratio, fixed scalar value between 0.5 and 1.0. net: A trained UDA model in the evaluation mode. model_list: A list containing candidate UDA models.
Calculate ICE scores for mini-batch. def (ce_score lam, net):
# Random batch index.
rand_idx torch randpern (x. shape [0])
inputs_a
inputs_b = x[rand_ idx]
Obtain model predictions and hard pseudo labels.
pred_a net (etipuus_aa)
pl_a = (dd===))
pl_b pl_a(rand_idx)
mixup.
same_idx = (pl_a == pl_b). nonzero tus_tuple=True [0]
Inter-cluster mixup.
diff_ide (pl_a != pl_b). nonzero ( (as tutplelllureee (rnperooss..uupee-Trre [o]
Mixup with images and hard pseudo labels.
mix_inputs lam inputs_a (1 lam) inputs_b
if lam 0.5:
mix_labels pl_a
else:
mix_labele = pl_b
Obtain predictions for the mixed samples. mix_pred metim__innuuts
mix_pred_labels = mixxppred max =mixppoodnax(ddn11 (1]
Calculate ICE scores for tio-dimensional probing.
ice_same torch. sum ( [sane___ [same_idx]
== max_labbel
same idx htteedidxxhhppppppp [0]
ice_diff torch. sum (rrch..uuuii._prrddlabbee (dif___aaablllfff_dddd [diff_idx:
== mix_labels (diff_iiddiff
diff idx shape [0]
return ice_same, ice_diff
Perform model selection based ICE scores.
def (aoaellmodel1ll _llst, x, lam):
Calculate ICE scores for all candidate models.
ice_same_list = []
ice_diff = []
for net in model_list:
ice_same, ice_diff ice_score (x, lam, net)
_ce_same_list.cn append (ice_same)
ice_diff.llst..ppeee (ice_diff)
Calculate the average rank of types of ICE scores.
ice_same_list = barch.tenson (ice_ssieemmmmlssss
ice_diff_list
rerch.temsrricceddffff
ice_same_rank torch. atrcia.argrrtororrrrrrr... (ice_ttaeeeeeeeeeeeeeeee
ice_diff_rank = torch argsort
overage_rank (ice_same_ (torch arr.ohhc.rgotticcccdiirii (ice_diff_list) ice_diff_rannk / 2 Choose the model with the highest average rank.
return model_ list [torch argmax argmax (ageraaavrrgg
14

---

B Hyperparameter Settings
In our main experiments, we follow the methodology of previous model selection studies [18, 29). For simplicity, we tune a single hyperparameter for various UDA methods. In the case of two- hyperparameter validation experiments, in MCC [25] and MDD [22], also tune the bottleneck dimension, choosing from the candidate values {256, 512, 1024, 2048). This variation is due to observations that the bottleneck dimension varies across different datasets in the official code of these UDA methods. For the validation of semantic segmentation tasks, we also consider two hyperparameters with the training iteration as an additional hyperparameter selected from the set {10, 000, 12, 000, 14, 000, 16, 000, 18, 000, 20, 000, 22, 000, 24, 000, 26, 000, 28, 000, 30, 000), fol- lowing SND [29]. Detailed hyperparameter settings are available in Table 11.
Table 11: Summary of the considered UDA methods and their corresponding hyperparameters.
UDA method
UDA Type
Hyperparameter
Search Space
Default Value
closed-set self-training
loss coefficient
{0.02, 0.05, 0.1,
ATDOC [27]
0.2
0.2, 1.0, 2.0}
closed-set
loss coefficient
{0.02, 0.05,
BNM [24]
1.0
output regularization
0.2, 1.0,
closed-set
loss coefficient
{0.05, 0.1,
CDAN [9]
1.0
feature alignment
0.5. 0.5,1.0,20 2.0, 5.0}
MCC [25]
closed-set
lemperature
{1.0, 1.5, 2.0,
2.5
output regularization
3.0,3.0,3.5, 4.03
closed-set
margin factor
{0.5, 1.0, 2.0,
MDD [22]
4.0
output alignment
3.0, 4.0, 6.0)
closed/partial-
loss coefficient
{0.002, 0,005, 0.01,
SAPN [23]
0.05
feature regularization
0.02, 0.05, 0.1,
partial-set
loss coefficient
10.05,0.1,00
PADA [15]
1.0
feature alignment
0.5, 1.0, 2.0, 5.03
open- peen-partill-se
loss coefficient
{0.02, 0.05,
DANCE [41]
0.05
self-supervision
0.2, 2.03
source-free
loss coefficient
{0.03, 0.05, O.T,
SHOT [20]
0.3
hypothesis tramsfer
0.3, 0.5, 1.0, 3.0}
closed-set
loss coefficient
(0.1, 0.5, 1.0, 2.0, 5.0, 10.03
DMRL [50]
2.0
mixup training
closed-sct
loss coefficient
[0.0001, 0.0003, 0.001,
AdaptSeg [11]
0.0002
output alignment
0.003, 0.01, 0.03)
closed-set
loss coefficient
[0.0001, 0.0003, 0.001,
AdvEnt [12]
0.001
output alignment
0.003, 0.010 0.03)
C Analysis of Intra-Cluster and Inter-Clusted Probing
In Vil,al, we use mixed samples to directly probe the intra-cluster and inter-cluster structures within the target representations learned by UDA model. This strategy shares similar spirits with the well-known Linear Discriminant Analysis (LDA) [66], which optimizes an objective focused on minimizing intra-cluster variance while maximizing inter-cluster variance. Drawing inspiration from the LDA optimization, we explore a straightforward validation baseline that employs an LDA-like metric. In this context, let's denote the number of target samples as N, the number of categories as K, and the feature of a sample encoded by a UDA model as f. The formulation for the LDA score is presented in Equation below.
K
N S
Nm
K
1
= m=
Fm
= N
Σ m=1
N
N
N
t=1
i=1
(1)
N
K
D Dintra
Σ
F
Dinter
1
Σ
I
f
Fl1
LDA
Dinter
K
Dintra
In our implementation, we utilize the predicted logits as features and compute variance using the La distance. We assess the LDA baseline against all target-only validation methods in both closed-
15

---

set UDA (Table 12) and partial-set UDA (Table 13). In both UDA scenarios, LDA outperforms existing validation methods in terms of average validation accuracy. This highlights the advantage of considering two effective assumptions, in contrast to the single assumption used in SND and Entropy. It's worth noting that while L.A performs well, still falls short of Mix Val's performance, underscoring the benefits of Mix Val's direct probing approach using mixed samples compared to the indirect probing via measurement of raw predictions used by SND, Entropy, and LDA.
Table 12: Validation accuracy (%) of closed-set UDA on Office-Home (Home). bold: Best value.
ATDOC [27]
BNM [24]
CDAN [9]
Method
-Ar
-Pr
avg
-AAr
-1
-rr
-Re
avg
-hAr
avg
Source Val
66.63
52.54
78.57
76.51
68.59
62.44
50.74
77.53
74.76
66.37
35.00
42.65
69
68.81
58,99
Entropy [42]
54.49
74.15
77.64
68.14
67.66
57.56
76.37
77.45
69.76
InfoMax [28]
63.67
55.63
77.61
78.36
68.82
66.28
54,49
74.15
77.64
68.1
67.66
57.56
76.37
77.45
69.76
SND [29]
63.67
55.63
76,54
77.54
68.34
66.28
54,49
74.15
77.64
68.14
67.94
57.56
76.96
77.68
70.04
Corr-C [43]
63.51
50.39
73.89
73.88
65.42
58.10
45.37
68.97
70.59
60.76
53.84
41.21
64.96
67.65
56,91
LDA
63.67
55.63
76,54
77.54
68.34
66.28
54,49
74.15
77.64
68.14
67.66
57.56
76.37
77.45
69.76
Vix
66.47
56.87
78.14
79.20
70.17
67.36
56.18
76.10
78.12
69.44
67.71
57.78
76.89
77.76
70.03
Worst
62.9
30.39
73.89
73.88
65.26
58.10
45.37
68.96
70.39
60.75
53,80
41.21
64.78
67.63
56.86
Best
68.97
58.35
80.27
80.58
72.04
68.93
57,51
78.43
79.57
71.11
68.19
$7.90
77.44
78.19
70.43
Method
MCC (25)
MDD (22)
SAFN (23)
Home
-Ar
-+CI
-Prr
Re
avg
-AA
-C1
Prrr
-Re
avg
-AA
-CI
-Pr
-Re
avg
AVG
Source Val
66.57
56.53
79.35
80.30
70.89
02.55
34.43
75.27
75.55
66 94
63.54
51
73.66
74.54
65.77
66 26
Entropy [42]
69.29
59.33
80.63
80,96
72.55
66.54
57.63
77.27
77.45
69.72
59.85
46.41
72.51
73.18
62.99
68.63
InfoMax [28]
66.58
58.48
79.12
80.81
71.25
66.54
57.74
77.27
77.45
69.75
64.56
49.71
73.77
73.18
65.31
68. 84
SND [29]
69.05
55.61
79.72
79.10
70.87
51.34
38.01
77.61
68.46
58
$7.90
46.41
67.04
68.18
59.88
66.02
Corr-C (43)
69.05
55.61
79.72
79.10
70.87
47.79
31.69
63.40
60.63
50.88
62.66
46.41
68.83
68.18
61.52
61.06
72.73
66.38
57.63
77.61
77.45
64.56
49.91
72.51
71.55
VixVal
69.79
59.24
80.47
80.74
72.56
65.73
58.01
77.36
76.91
69.50
65.98
53.14
74.76
75.40
67.32
69.84
Worst
67.93
47.79
31.69
63.40
60.63
50.88
37.90
46.41
67.04
68.18
59.88
Best
70.68
59.95
80.93
81.02
73.14
66.75
58.36
77.61
77.45
70.04
66.59
53 14
74.90
75.57
67.55
70.72
Table 13: Validation accuracy (%) of partial-set UDA on Office-Homme
PADA [15]
SAPN CI
[23]
Home
Mathod
Ar
avg
Pr
Re
avg
AVG
Source
57.21
41.90
64.48
71.89
58.87
66.82
54.71
74.41
76.48
68.11
63.49
Eatropy [42]
60.08
46.51
53.16
62.47
55.56
71.75
55.62
76.36
76.59
70.08
62.82
InfoMax (28)
60.08
51.40
60.20
66.67
59.59
63.67
51.74
69.64
73.62
64.67
62.13
SND [29]
67.80
50.71
59.46
67.13
61.27
71.75
51.74
76.36
78.36
69.55
65.41
Coer: C [43]
61.34
45.65
54.90
62.25
56.04
71.23
55.70
76.94
79.13
70.75
63.40
LDA
64.52
46.51
69.47
72.67
63.29
71.75
$4.39
73.93
77.10
69.29
66.29
MixVal
67.68
51.01
72.94
78.64
67.57
71.70
57.91
77.08
78.94
71.41
69.49
Worst
56.29
39.76
50.49
59.31
51.46
62.48
49.91
68.50
73.62
63.63
57.55
Best
69.33
55.86
74.55
79.59
69.83
73.37
58.09
77.35
79.33
72.03
70.93
D Full Validation Results
For classification tasks in our evaluation, we utilize the HOS score (%) [62, 63) for open-partial-set UDA and accuracy (%) for all other UDA tasks. In the case of segmentation tasks, we measure mloU (%) [11, 12). To accommodate space constraints in this main text, we present the averaged validation results for UDA tasks that share the same target domain. For example, '- Ar' represents the averaged results of three tasks on the Office-Home benchmark, namely 'CI+ Ar', 'Pr- Ar', and 'Re- Ar'. Additionally, please note that the 'avg' row signifies the average of the rows preceding within each UDA method, while the 'AVG' row represents the average results of 'avg' rows across all considered UDA methods. For more detailed validation results for each specific UDA task, please refer to the corresponding tables, from Table 14 to Table 29.
16

---

Table 14: Accuracy (%) of ATDOC [27], a closed-set UDA method, on Office-Homer
Method
Ar Ar-4
Ar Pr
Ar
Re
C
Ar
C
Pr
CI
Re
Pr
Ar
Pr
-4
CI
Pr
- Re
Re
Ar
Re
CI
Re
Pr
avg
Source
45.06
IWCV [30]
55.88
76.57
78.88
66.25
74.50
78.33
65.60
48.06
80.58
72.06
58.14
83.87
69.89
DEV [18]
76.55
78.88
66.25
77.67
64.77
51.29
81.62
71.36
59.98
69.70
RV [31]
56.38
76.12
80.01
66.25
76,80
78.33
67.82
S5.62
80.58
71.98
36.40
83.87
70.85
Entropy
55.88
74.14
78.88
59.25
74.52
77.67
64.19
54.39
78.54
67.57
57.21
80.96
68.60
InfoMax [28]
55.88
78.88
77.74
77.67
64.19
54.39
78.54
67.57
56.61
80.96
68.
SND [29]
55.88
74.14
78.88
59.25
74.52
75.21
64.19
$4.39
78.54
67.57
56.61
80.96
68
[43]
76.06
MixVal
57.19
74.82
79.13
64.19
78.10
77.90
65.63
55.28
80.57
69.60
58.15
81.49
70.17
Worst
51.41
72.00
76.04
48.04
76.06
67.57
51.71
Best
58.01
77.31
81.04
66.91
79.48
78.52
67.94
57.07
82.
72.06
39.98
84.03
72.04
Table 15: Accuracy (%) of BNM [24), closed-set UDA method, on Office- Home.
Method
Pr
Re
avg
Source Val
56.93
77.00
77.74
57.64
73.33
69.36
36.45
42.38
77.19
73.22
52.90
82.26
66.37
IWCV [30]
46.46
77.00
79.30
63.86
61.34
62.54
63.95
42,38
78.01
71.85
55.65
83.92
68.52
DEV [18]
57.75
71.62
79.30
57.64
67.90
75,46
66.21
54.04
78.01
73.42
57.37
82.25
68.41
RY [31]
58.67
77.00
79.30
65.68
73.33
75.46
65.64
52.05
81.25
73.42
59.54
83.92
70.44
Eatrogy [42]
67.04
78.04
63.41
71.44
73,93
63.38
52.00
80.95
71.86
57.37
83.96
68.
InfoMax [28]
53.40
67.04
78.04
63.41
71.44
73.93
63.58
$2.69
80.95
71.86
57.37
83.96
68. 14
[2]]
Con-C [43]
46.46
67.06
74,82
49.73
61.34
62.54
56.45
42.38
74.41
68.11
47.26
78.51
60:76
Vix
56.30
72.90
78.75
64.66
71.44
74.52
64.32
$4.04
81.09
73.09
58.19
83,96
69.44
Worst
46.46
67.04
74,82
49.73
61.34
62.34
36.45
42.38
74.41
68.11
47.26
78.51
60.75
Best
58.67
77.00
80.61
67.16
74.16
76.75
66.21
54.04
81.36
73.42
59.82
84.12
71.11
Table 16: Accuracy (%) of CDAN [9], a closed-set UDA method, on Office-Home.
Method
A3 Ar -
Ar
Re
Ar
CI -
Re
Pr Ar
CI
Pr
Re
Ar
Re
CI
Re
avg
Source Val
43.41
62.51
75.51
43,96
61.59
57.70
53,75
37,53
73.22
67.28
47.11
84. 19
58.99
IWCV [30]
43.18
62.51
77.81
44.71
$4.61
56.14
65.14
37.53
81.85
74.08
43.00
84.39
60.41
DEV [18]
57.16
71.75
77.81
62.46
58.64
71.08
65.14
56.54
$1.85
74.08
57.43
78.89
67. .9.49
[3I]
57.16
71.75
77.78
72.92
54.50
Entrryy [42]
57.35
72.43
77.74
63.62
72.92
73,40
65.22
56,66
81.20
74.08
58.47
83.76
69.76
InfoMax [28)
57.55
72.43
65.27
56.66
$1.20
69.76
SND [29]
$7.85
72.43
77.78
64.61
73.73
73:40
65.14
$6.66
$1.85
74.08
58.47
84.73
70.04
Cor--C
43.14
63.05
73.61
43,96
54.58
56.12
51.75
37.50
73.22
65,80
43.00
77.25
56.91
57.55
73.71
77.75
63.95
73.19
74.06
65.23
56.59
81.47
73.95
$9.19
83.76
70.03
Worst
43.14
62.51
73.61
43.96
$4.58
56.12
51.63
37.50
73.22
65.80
43.00
77.25
56.86
Best
57.55
73.71
78.33
64.61
73.89
65.76
56.66
81.85
74.21
59.50
70.43
Table 17: Accuracy (%) of MCC [25], closed-set UDA method, on Office-Home.
Method
++
Pr
Re
CI
Ar
CI Pt
Re
Ar
CI
Pt Re
Re Ar
Re
C
Re
avg
SourceVal
5723
19
81.75
60.65
76.50
78.79
61.13
33.15
$2.17
74.91
39.20
83.96
70.89
TWCV [30]
60.02
78.15
81.34
68.73
78.51
77.85
64.15
$7.85
81.04
73.18
58.92
84.46
72.02
[18]
73.01
71.38
RV [31|
59.34
78.53
80.70
69.10
77.83
78.22
67.20
$7.85
82
74.91
39.20
85.54
72.56
Batropy
78.53
57.85
82.51
72.55
InfoMax [28]
60.02
74.66
81.75
64.98
78.24
78.49
64.15
$4.52
82, 19
70.62
60.89
84.46
71.25
SND [29]
53.56
77.43
79.46
67.28
76.48
76.80
65.06
54,34
$1.04
74,82
58.92
85.24
70.87
Cor-CC [43]
53.56
77.43
79.46
67.28
76.48
76,80
65.06
54,34
81.04
74,82
58.92
85.24
70.87
MixVal
59.08
77.81
81.61
68.40
78.45
78.40
67.56
57.65
82.21
73.41
61.00
85.15
72.56
73.01
53.15
79.35
67.78
67.93
Best
60.02
78.53
81.75
69.22
78.51
78.79
67.90
58.49
82.51
74.91
61.35
85.74
73.14
Table 18: Accuracy (%) of MDD [22], closed-set set UDA method, on Office Home.
Office-Home.
avg
SourceNal
54.85
73.35
77.05
58.76
69.95
72.13
60.05
$1.02
77.36
68.81
57.42
$2.50
66
IWCV [30]
DEV [18]
57.71
75.42
77.05
58.76
72.99
70.51
63.95
56.43
80.26
70.54
56.68
82.14
68.
58.05
75.42
76.59
63.54
69.95
73.74
63.95
51.02
$0.38
72.23
58.17
84.43
68.96
Emirogy [42]
57.73
74.54
78.12
64.07
72.99
73.74
63.95
55.85
$0.38
71.61
39.31
$4.28
69.72
InfoMax [28]
58455
74.54
78.22
64,07
72.99
73.74
63,95
55.85
80.38
71.61
59.31
$4.28
69.75
SND (29)
75.42
77.05
44.99
72.99
48.06
37.08
21.60
80.26
71.94
34.39
84.43
58.86
Con-CC [43)
39.08
59.74
69.61
44.99
54.58
48.06
37.08
21.60
64.22
61.31
34.39
75.87
50.88
MixVal
75.13
79.90
Worst
39.08
59.74
69.61
44.99
$4.58
48.06
37.08
21.60
6-1.22
61.31
34.39
75.67
50.88
Best
58.05
75.42
78.22
64.07
72.99
73.74
63.95
57.02
80.38
72.23
60.02
84.43
70.04
17

---

Table 19: Accuracy (%) of SAFN [23], closed-set UDA method, on Office- Office-Home.
Method
Ar CI
Ar-+Pp
Ar Re
Ar
Pr
CI
Re
Pr
-1 Ar
Pr
Pr
Re
Re
Ar
Re
C1
Re
Pr
avg
SourceVal
69.36
46.07
77.71
65.77
IWCV [30]
50.24
69.72
77.28
62.63
67.24
69.86
58.84
49.69
75.72
71.45
57.16
79.97
65.82
DEV [18]
51.07
69.72
76.64
59.66
67.24
71.26
58.84
49.69
75.72
70.95
50.65
76.64
64.35
RV [31]
51.07
71.41
76.64
62.63
68.44
70.44
58.84
44.49
77.71
71.45
54,82
81.46
65.78
Entropy [42]
45.93
69.72
75.49
55.29
67.22
68.35
34.26
43,30
75.69
70.00
49.39
80.60
62.99
InfoMax [28)
50.47
69.72
75.49
62.46
70.98
68.35
61.23
43.30
75.69
70.00
55.37
80.60
65.31
SND [29]
45.93
64.36
70.60
55.29
60.13
62,50
54,26
43,30
71.43
64.15
49.99
76.64
59.88
Corr-C [43]
MixVal
51.73
72.14
77.08
63.92
70.98
71,55
62.71
50.52
77.56
71.30
$7.16
81.17
67.32
Worst
71.43
Best
51.73
72.27
77.30
64.65
70,98
71,70
63.66
50.52
77.71
71.45
$7,16
81.46
67.55
Table 20: Accuracy (%) of closed-set UDA methods on Office-31.
ATDOC
BNM W
Method
A A-1 D
A
W
D
-+
A
W -
A
avg
D
A
A
W
ave
-
D
W
A
avg
SourceVal
88.96
87,80
73.65
71.46
80.47
90.36
89.43
73.13
72.70
81.41
91.16
89.06
66.33
61.46
77.00
IWCV [30]
86.14
86.54
73.65
71.46
79.45
89,43
73.13
72.70
80.20
69.08
58.74
66.33
61.46
63.90
DEV [18]
86.14
86.54
73.65
71.46
79.45
85.54
89.43
73.13
72.70
80.20
91.16
88, 30
66.33
61.46
76.81
91.16
Entropy [42]
86.14
87.80
73.87
72:70
80. 13
85.34
83.14
71.07
74.26
78. 50
91.16
89.06
72.88
70.36
80.87
InfoMax [28]
86.14
87.80
73.87
72.70
80.13
85.54
83.14
71:07
69.97
77.43
91.16
88.30
72.88
70.36
80.68
SND (29]
92.37
87,80
73.87
72.70
85.54
83.14
74.62
74.26
79.39
92.37
88.55
72.88
70.22
81.01
Com-C
90.96
84.40
71.88
70.22
79.37
84.34
78.99
67.80
66.52
74.41
67.67
59.62
$8.15
58.43
60.97
MixVal
75.64
92.64
89.06
Worst
86.14
84.40
71.88
70:22
16
84.34 34
78. 99
67.6 80
66.52
74.41
67.65 67.67
57.11
58.
$8.43
60.34
Best
92.37
87.80
75.04
75.58
82. 70
90.36
89.43
75.75
75.29
82.71
92.77
89.06
76.18
70.57
82.15
Table 21: Accuracy (%) of closed-set UDA methods on Office-31.
MCC [25]
MDD [22)
SAPN (23)
Method
A
W
D
A
W
A
W
W
A-
D
SourceVal
91.07
72.89
06
74.76
83.73
87.17
68.96
69
77.33
IWCV [30]
91.16
88.55
73.33
72.89
48
91.16
89.18
76.68
74.30
82
86.55
80.38
68.96
69.68
76.39
DEV [18]
73.33
72.06
76.68
74.62
86.55
80.38
68.96
75. 84.84
RV [3i]
89.06
93.08
74.42
73.52
82.52
92.57
86.79
73.94
74.97
82.07
90.83
87.17
68.26
68.62
78.85
Eatropy [42]
90.36
93,46
74,83
73.02
82.97
92.57
90.82
78,03
74.38
84.00
91.57
85,66
67.20
69.26
78.42
InfoMax [28]
74.16
73.70
92.57
78.03
74.97
84.10
91.57
67.20
SND [29]
91.97
93.46
74.83
73.02
83.32
92.17
90.82
78.03
74.97
84.00
89.96
85.66
67.20
69.26
78.02
Corr-C
83.17
85.66
74.58
86.75
80.38
69.68
75.98
MixVal
91.77
93.21
74.74
73.44
83.29
91.77
91.74
77.35
74.58
83.88 86
89.96
86.83
69.91
69.31
79.00
Worst
86.75
87.17
71.18
69.93
76
87.35
85.66
73.91
72.20
79.
83.73
80,38
67.09
45
74.66
83.57
92.57
92.20
78.03
91.57
70.43
Table 22: Accuracy (%) of CDAN [9], a closed-set UDA method, on DomainNet.
Method
C-+
P-+
P R
avg
Entropy [42]
58.04
64.78
74.42
69.39
68.65
60.63
62.94
65.55
InfoMax [28)
58.04
64.78
74.42
69.39
68.65
60.63
62.94
65.55
SND [29]
58.04
64.78
74.42
69.39
68.65
60.63
60.70
65.23
Corr-C [43]
58.04
$7.73
65.07
$1.23
60.70
60.60
MixVal
58.04
64.78
74.42
69.39
68.65
60.63
62.94
65.55
Worst
$1.39
$7.73
73.44
36.98
63.06
$1.23
58.46
58.93
58.04
64.78
74.44
69.39
68.65
60.63
62.94
65.55
Table 23: Accuracy (%) of BNM [24], a closed-se set UDA method, on DomainNet.
Method
C
P C
P
R
R R-
C
R -
P
R R
S S P
avg
Entropy [42]
56,42
61.57
74.31
65.15
65.15
40.95
63.42
61.00
InfoMax [28]
56.42
68.95
74.31
65.15
65.15
54.93
63.42
64.05
SND [29]
43.78
61.57
74.31
51.55
$4.40
40.95
54.59
54.45
Corr-C [43]
43.78
60.03
77.62
59.47
67.19
40.95
59.64
58.38
Val
58.48
69.63
78.68
67.59
58.50
65.20
66.30
Worst
43.78
60.03
74.31
51.55
$4.40
40.95
54.59
54.23
Best
58.48
69.63
78.68
66.10
67.79
58,50
65.20
66.34
18

---

Table 24: Accuracy (%) of ATDOC [27]. closed-set UDA method, on DomainNet.
Method
P- C
p
avg
Entropy [42]
46.43
65.98
79.60
61.52
64.24
57.92
59,46
62.16
InfoMax [28]
46.43
65.98
79.60
61.52
64.24
57.92
59.46
62.16
SND [29]
46.43
65.98
79.60
61.52
64.24
47.58
59.46
60.69
Corr-C
54.71
60.63
64.58
$2.66
60.90
MixVal
62.11
69.64
79,60
68.24
69.79
61.35
67.10
68.26
Worst
46.43
60.63
74.42
59.33
64.24
47.38
39.46
58.87
Best
63.12
71.14
80.38
69.45
69.79
61,35
67.10
68.90
Table 25: Accuracy (%) of PADA [15], a partial-set UDA method, on Office-Home.
Method
Ar CI
Ar Pr
Rc Ar
Ar
CI Pr
CI
Re
Pr - At
Pr -
CI
Pr - Rs
Re - Ar
Rc
C1
Rc
Pr
avg
SourceVal
45.03
IWCV [30]
55.58
63.10
84.54
51.42
61.29
53.01
57.02
35.16
81.34
70.52
60.78
74.12
62.49
DEV [181
54.81
78.15
78.02
58,13
61.29
90.14
67.86
35.16
83.21
74.66
57,91
77.76
64.76
[31]
43.22
65.10
81,89
42.70
48.74
52.79
57.21
35.16
77.80
73.46
44.24
77.76
Entropy [42]
40.12
40.11
55.94
52.43
37.25
50.14
57.30
47.22
81.34
70.52
52.18
82.13
55.56
InfoMax [28]
SND (29]
40.12
40.11
55.94
58.13
56.13
64.11
70.62
$1.22
81.34
74.66
61.78
82.13
61.27
Con-CC
MixVal
45.02
78.15
83,69
56,23
57.85
68.19
71.41
47.22
84.04
75.39
61.77
82.82
67.57
Worst
40.12
40.11
55394
41.11
37.25
50.14
56.93
34.87
71.34
70.52
44.30
74.12
51.46
Best
53.58
78.15
86.53
58.13
61.29
68.19
73.00
51.22
84.04
76.86
60.78
84.20
69.83
Table 26: Accuracy (%) of SAFN [23], partial set UDA method, on Office- Home.
Method
-
C1
Ar
+
Pr
Re
-
CI - Pt
-
Ro
-: Ar
-
CI
Pr ->
Re - Ar
Re
Re
avg
SourceNal
59,40
77.14
81.34
63.97
67.00
71.29
65.60
46.21
76.81
70.89
58.51
79.10
68.
IWCV [30]
53.49
73.65
DEV [18]
55.22
74.45
80.07
70.98
67.00
71.29
63.45
$1.70
76.81
73.65
57.91
$0.39
68.58
53.67
71.60
81.34
58
67.00
73.27
65.70
48.54
76.81
73.65
67.92
Entropy [42)
58.93
74,90
80.73
70.98
74.12
69.80
70.16
50.09
79.24
74.10
57.85
80.06
70.08
InfoMax [28]
51.82
67.62
76.97
64.65
65.77
69.80
$9.69
50,09
74.10
66.67
53.31
75.52
64.67
SND (29)
51.82
74,90
80.73
70,98
74.12
75.10
70.16
50.09
79.24
74.10
$3.31
80.06
69.55
Cor-C [43]
59.40
77.20
82,16
67.58
72.89
75.10
70.16
55,70
80.12
75.94
$2.00
80,73
70.75
73.71
55.34
$0.73
Worst
51.52
67.62
76.97
61.07
62,35
69.80
59.69
46.21
74
66.67
52.00
75.52
6.166.
Best
55.70
80.73
Table 27: HOS [62, 63] (%) of DANCE [41], an open-partial-se UDA method, Office-Home.
Method
Pr
Rc
avg
Entrogy [42]
38.29
26.08
36.51
32.92
17.10
32.19
37.69
46.40
45.53
25.39
33.75
39.37
34.27
InfoMax [28]
38:29
26.08
36.51
32.92
17.10
32.19
37.69
46.40
45.33
25.39
33.75
39.37
34.25
SND [29]
1.00
0.00
12.73
0.00
42.84
1.95
19.77
11.99
35.69
25.39
0.00
28.40
14.98
Corr-C [43]
1.00
0.00
12.73
0.00
42.84
1.95
19.77
11.99
35.69
69.02
0.00
28.40
18.62
Worst
1.00
0.00
12,73
10.00
17.10
1.95
19.77
11.99
35.09
25.39
0.00
28.40
12.84
72.87
81.84
Table 28:
Accuracy (%) of DMRL (50), closed-set UDA method, Office- Homee
Method
avg
Entropy [42)
47.03
62.09
73.42
$5.20
64.34
66.54
51.92
48.96
74.16
67.20
54.75
80.74
62.20
InfoMax [28]
48.55
62.09
73.42
55.29
64.34
66.54
51.92
48.96
74.16
67.20
54.75
80.74
62.33
SND (29]
47.84
62.76
73.86
54.92
64.34
66.65
51.87
48.96
73.74
66.42
53.08
80.74
62.10
Cor-CC [43)
47.7
62.76
72.53
55.29
64.34
68.12
53,11
47.56
74.55
66.46
53.08
78.91
62.04
4774
Worst
47003
61.73
72.14
54.84
62.90
66.15
51.87
46.35
73.67
66.42
53.08
78.71
61.30
Best
Table 29: Accuracy (%) of DMRL [50] and SHOT [20] Office-31.
DMRL
[50]
SHOT [20]
Method
A D
W D
W
A
W D
W
A
Entropy [22]
80.52
86.67
61.77
65.57
73.63
90.76
S8.68
71.21
72.13
80.70
InfoMaa [28]
80.52
86.67
61.77
65.57
73.63
90.76
88.6 38.68
71.21
72.13
80.70
SND [29]
84.14
86.67
61.77
61.91
73.62
90.76
88.68
71.21
72.13
80.70
Corr-C
77.51
81.13
60.28
60.17
69.77
90.76
90.19
71.21
71.96
81.03
Mix Val
82.93
86.67
63.40
66.38
74.85
92.37
92.32
71.21
72.88
82.20
Best
84.14
86.67
64.22
66.38
75.35
94,78
93.33
75.58
74.55
84.56
19


Reviews:
[{"review_id": "1KSmrooKKR", "reviewer": "Reviewer_G1xN", "summary": "This paper introduces a simple validation method for unsupervised domain adaption. This method, called MixVal, proposes a new evaluation (ICE) based on MIXUP to select the most appropriate model candidates, which is obtained by training with different hyperparameters, such as loss coefficient/temperature/margin factor. Compared with other target-only validation methods, the paper considered three different inductive biases: neighborhood consistency/low-density separation/no prior label distribution.", "questions": "What are the relationships between RankInter and RankIntra? Is that necessary for us to separate such two different types during inference? \n \nIn Figure 3, AccAvg has a similar trend as RankIntra. RankInter and RankIntra seem not to compensate. In other words, handling the inductive bias neighborhood consistency and low-density separation at the same time does not make sure higher performance.", "limitations": "Please see the weakness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is well-written and easy to follow. Specifically, Table 1 and Figure 1 clearly show the motivation and pipeline, respectively.\n\nExperiments over several datasets and settings are impressive which sufficiently show the generalization of the proposed method.", "weaknesses": "**1. Lack novelty.** Behind the mentioned contributions, there is only one proposed technical point, the proposed evaluation ICE with MIXUP, which just takes less than half a page. It could not be enough as a NeurIPS paper, especially lacking sufficient theoretical support. \n\n**2. Lack theoretical support.** The method assumes that the model with a higher ICE score can obtain a higher real target accuracy. This paper needs such theoretical support that there is a positive correlation between the ICE score and the discriminability of the candidate models. \n\nThis assumption could not be valid in real-world systems. Ideally, the best ICE score is 100%. In this case, the “model” could be a linear system, which is certainly impossible in the applications. This might demonstrate there are some theoretical gaps between the score and the goal of UDA.  \n\nThe authors also recognized this limitation and tried to give some empirical observations in Figure 2(b). However, without the second technical point, this limitation seems very serious. \n\nFigure 2(b) looks a little bit tricky, where the first figure is shown with accuracy, and the following subfigures are in the ranked ascending order. In fact, all subfigures should be shown with the same indicator, ranking all model candidates in ascending order. In this case, the empirical conclusion about the high correlation is not convincing. \n\n**3. Confusion about formulation.** The formulation of the mixed pseudo label is different from its pseudocode.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments. We've provided responses to each of your remaining questions below.\n\n> **Q1**:  Lack novelty. Behind the mentioned contributions, there is only one proposed technical point, the proposed evaluation ICE with MIXUP, which just takes less than half a page. It could not be enough as a NeurIPS paper, especially lacking sufficient theoretical support.\n\n**A1**:   We appreciate the reviewer's recognition of **MixVal's simplicity and the novelty introduced by our proposed ICE evaluation with Mixup**. However, we **respectfully differ** on the assertion that our submission **lacks novelty due to a single proposed technical point**. Firstly, we'd like to clarify that the **ICE with Mixup encompasses two novel technical aspects** introduced by our submission: **the validation of models via synthesized sample probing and the novel application of mixup during the inference stage** to generate in-between discriminative samples. The simplicity of ICE might overshadow these nontrivial innovations. \n\nBeyond the technical facets, our innovation extends to the **novel analysis of high-level inductive bias** inherent in existing target-only validation methods, succinctly summarized in Table 1. Furthermore, **extensive empirical comparisons spanning diverse domain adaptation scenarios** are presented in Tables 2-7.\n\nAs for the content coverage, our introduction of MixVal actually spans **more than one page, as shown in Section 3.2**. As for the assertion that a submission with less than half a page of technical content may not suffice for NeurIPS, we respectfully differ in opinion.\n\n > **Q2**:   Lack theoretical support. ... This paper needs such theoretical support that there is a positive correlation between the ICE score and the discriminability of the candidate models. This assumption could not be valid in real-world systems. ... This might demonstrate there are some theoretical gaps between the score and the goal of UDA.\n\n**A2**:   Although formal theoretical analysis is absent, our **extensive experiments and analysis strongly validate MixVal**. Across various UDA scenarios outlined in Tables 2-7 (including VisDA, DomainNet, source-free UDA, and open-set shifts), MixVal consistently outperforms existing validation methods. This compelling empirical evidence contests the notion that \"this assumption could not be valid in real-world systems.\" Our results affirm MixVal's effectiveness in model selection based on ICE scores, **dispelling any unknown but conjectured theoretical gaps.**\n\n > **Q3**: ... some empirical observations in Figure 2(b). ... Figure 2(b) looks a little bit tricky, In this case, the empirical conclusion about the high correlation is not convincing.\n\n**A3**: We'd like to clarify that **Figure 2(b) is informative with comprehensive data**. While we can infer the rank from specific accuracies in the first part of Figure 2(b), reversing the process is not feasible. To enhance clarity, we've also included a ranking visualization in **Figure 1 of the PDF. Our observation remains consistent: MixVal exhibits a stronger correlation than the state-of-the-art method SND.**\n\n > **Q4**: Confusion about formulation. The formulation of the mixed pseudo label is different from its pseudocode.\n\n**A4**: No inconsistency between method and pseudocode. **Mixup formulation is general, while pseudocode offers detailed steps.**\n\n > **Q5**: What are the relationships between RankInter and RankIntra? Is that necessary for us to separate such two different types during inference? In Figure 3, AccAvg has a similar trend as RankIntra. RankInter and RankIntra seem not to compensate. In other words, handling the inductive bias neighborhood consistency and low-density separation at the same time does not make sure higher performance.\n\n**A5**: To clarify, \"RankInter\" signifies MixVal with inter-cluster mixup only, and \"RankIntra\" corresponds to MixVal with intra-cluster mixup only. In our MixVal implementation, we **achieve both types of probing seamlessly in a single-time inference**, as outlined in our \"ice_score\" function detailed in Appendix A.\n\nRegarding Figure 3, it's important to note that AccAvg is not employed in our MixVal methodology. This is because the direct combination of ICE scores from the two probing strategies is unstable, given that intra-cluster probing tends to yield higher ICE scores than inter-cluster probing. The Figure illustrates that AccAvg performs less effectively than RankAvg due to the instability associated with accuracy averaging. In our submission, **we highlighted in L298 that MixVal's combination of both probing strategies bolsters the stability of model selection, substantiated by our experimental outcomes detailed in Tables 2-5. In support of this point, we've provided additional results in Table 1 of the attached PDF in the global rebuttal.**\n\nTwo conclusions can be drawn from these results: (i) The two-dimensional probing strategy enhances the stability of validation compared to individual probing types. (ii) MixVal distinguishes itself from and significantly surpasses the combination of SND and Entropy."}, {"review_id": "LBDsW9hEbJ", "reviewer": "Reviewer_4S59", "summary": "In this paper propose a novel target-only method is proposed to employ mixup to synthesize in-between target samples for validation. MixVal leverages mixed target samples to directly probe the learned target structure, benefiting from an combination of inductive biases considered in prior approaches. MixVal performs intra-cluster and inter-cluster mixup to explicitly capture both inductive biases of neighborhood consistency and low density separation.", "questions": "DANCE (NeurIPS 2020) is not the SOTA of Open-partial-set UDA. How about the performance of some later work as listed below?\n[1] Guangrui Li et al. Domain Consensus Clustering for Universal Domain Adaptation. CVPR2021\n[2] Kuniaki Saito et al. OVANet: One-vs-All Network for Universal Domain Adaptation. ICCV2021\n[3] Liang Chen et al. Evidential Neighborhood Contrastive Learning for Universal Domain Adaptation. AAAI2022\n[4] Liang Chen at al. Geometric Anchor Correspondence Mining with Uncertainty Modeling for Universal Domain Adaptation. CVPR2022", "limitations": "The authors addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work propose a novel solution named MixVal that eliminates the need for source data access and avoids the cumbersome model re-training. Through inter-cluster and intra-cluster mixup, MixVal combines two essential inductive biases utilized in prior validation methods.\n2. The experiment results achieve SOTA.", "weaknesses": "1. The presentation need to be improved.\n2. Using Mixup in domain adaptation is a common practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback on our paper, particularly regarding its soundness and state-of-the-art experimental results. We have addressed your concerns as follows:\n\n\n> **Q1**:  The presentation need to be improved.\n\n**A1**: Thanks for the constructive suggestion. We will revise our paper according to comments from all reviewers to improve our presentation.\n\n> **Q2**:  Using Mixup in domain adaptation is a common practice.\n\n**A2**: We acknowledge the widespread use of Mixup in various domain adaptation methods. **However, it's important to clarify that our implementation of mixup differs significantly from the conventional practice employed in most mixup-related works.** Numerous domain adaptation approaches [32, 49, 50], as well as semi-supervised learning methods [31, 33], typically apply mixup during the training stage with a mix ratio close to 1.0 ($\\lambda \\in \\text{Beta}(\\alpha, \\alpha), \\alpha \\in [0.1, 0.4]$) for regularization effect, and tend to suffer from performance degradation when the mix ratio is near 0.5 due to manifold intrusion [A, B].\nIn contrast, our MixVal harnesses mixup during the inference stage to explore the learned target structure. This is executed with a fixed mix ratio close to 0.5. \n\nWe've highlighted some representative methods employing mixup for domain adaptation [32, 49, 50] and semi-supervised learning [31, 33] in L32. Furthermore, we've conducted validation experiments, specifically with DMRL [32], presenting results in Table 8 and corresponding analysis in L286-L289. The outcomes from the DMRL experiments demonstrate that MixVal offers resilience against attacks that involve mixup during domain adaptation model training.\n\nWe'll also clarify the distinctions between these methods and our approach in our paper.\n\n> **Q3**:  DANCE (NeurIPS 2020) is not the SOTA of Open-partial-set UDA. How about the performance of some later work as listed below?  [1] Guangrui Li et al. Domain Consensus Clustering for Universal Domain Adaptation. CVPR2021 [2] Kuniaki Saito et al. OVANet: One-vs-All Network for Universal Domain Adaptation. ICCV2021 [3] Liang Chen et al. Evidential Neighborhood Contrastive Learning for Universal Domain Adaptation. AAAI2022 [4] Liang Chen at al. Geometric Anchor Correspondence Mining with Uncertainty Modeling for Universal Domain Adaptation. CVPR2022\n\n**A3**:  We greatly appreciate your valuable suggestion. We'll include all of these relevant works in the related section of the open-partial-set UDA. It's worth noting that our submission is the first to tackle the model selection problem in the context of open-partial-set UDA, showcasing the validation method's generalization ability. \n\nOur choice of DANCE as the method for open-partial-set UDA serves specific purposes. Notably, DANCE holds a notable status as a standard universal domain method and has also been adopted by SND [27] for closed-set validation experiments. However, we acknowledge that DANCE doesn't represent the state-of-the-art (SOTA) in open-partial-set UDA.\n\n**To further address the concern on this, we've included the results of OVANet, as recommended by the reviewer, and have presented results in Table 2 of the attached PDF in the global rebuttal.** Specifically, we conduct the hyperparameter validation for the entropy objective's loss coefficient, spanning a range of values such as $\\\\{0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0, 3.0, 10.0\\\\}$. OVANet sets the default value as 0.1 by conducting supervised validation on one UDA task. From the results, we observed that MixVal consistently outperforms other target-only validation methods in terms of validation performance. This experiment further highlights the robust generalization ability of MixVal.\n\n\n[A] MixUp as Locally Linear Out-Of-Manifold Regularization, AAAI 2019\n\n[B] On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks, NeurIPS 2019"}, {"review_id": "Qr68tiRxYH", "reviewer": "Reviewer_PTRC", "summary": "Validating hyperparameters in UDA is challenging due to the unlabeled target data. For it, the paper proposes MixVal, a novel target-only approach that utilizes mixup to synthesize target samples for validation. MixVal combines inductive biases from prior approaches through intra-cluster and inter-cluster mixup, achieving state-of-the-art performance across 11 UDA methods and 4 adaptation settings.", "questions": "Please provide an in-depth response regarding the weaknesses mentioned above.", "limitations": "This paper addresses the important research topic, model/hyperparameter validation in UDA. I think that this work can be meaningful to the UDA community. However, it is unclear whether the proposed method is the most optimal approach for validation and even as a work of using Mixup.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Validation is crucial in UDA to utilize it in real-world problems, and this paper proposes an effective method to tackle it.\n2. Experimentally, it is clearly robust and effective to model selection across multiple benchmarks.", "weaknesses": "1. I can't find methodological novelty. I think that the proposed method is a combination of Entropy, SND, and Mixup.\n2. There are several papers utilizing Mixup in UDA. It requires not simply mentioning those papers, but rather providing a detailed summary and the differences from them. (e.g., CoWA-JMDS [1])\n3. Since Mixup is used for intra-cluster and inter-cluster augmentation, it is possible to utilize other famous Mixup variants like Manifold Mixup [2] or CutMix [3]. However, there are no relevant experiments conducted in this regard.\n\n[1] Lee, Jonghyun, et al. \"Confidence score for source-free unsupervised domain adaptation.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Verma, Vikas, et al. \"Manifold mixup: Better representations by interpolating hidden states.\" International conference on machine learning. PMLR, 2019.\n\n[3] Yun, Sangdoo, et al. \"Cutmix: Regularization strategy to train strong classifiers with localizable features.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the constructive comments. We have addressed all of your concerns in a detailed manner as outlined below.\n\n> **Q1**:  I can't find methodological novelty. I think that the proposed method is a combination of Entropy, SND, and Mixup.\n\n**A1**:  We **respectfully disagree with this assertion**. Our MixVal methodology stands apart from both Entropy and SND. Unlike Entropy and SND, which directly measure overall raw predictions across all target samples, MixVal takes **a distinct approach by using mixed samples to actively probe the learned target structure**. Furthermore, MixVal introduces **a novel analysis of high-level inductive bias** that differentiates it from existing works in UDA model selection.\n\nRegarding the use of mixup, we acknowledge that it's a fundamental data augmentation technique widely applied in various tasks, as detailed in Section 3.2. **The inclusion of mixup does not necessarily undermine novelty, especially considering its extensive influence—cited over 7200 times.** We emphasize that **the standard mixup is typically applied during the training stage with a mix ratio close to 1.0** ($\\lambda \\in \\text{Beta}(\\alpha, \\alpha), \\alpha \\in [0.1, 0.4]$) for regularization effect, and tends to suffer from **performance degradation when the mix ratio is near 0.5 due to manifold intrusion [A, B]**.\nIn contrast, our **MixVal harnesses mixup during the inference stage to explore the learned target structure**. This is executed with a fixed mix ratio close to 0.5. **These differences significantly set MixVal apart from other techniques including Entropy, SND, and mixup.**\n\n**From the perspective of experimental performance**, we provide a direct comparison **between Entropy+SND and our MixVal in Table 1 of the attached PDF in the global rebuttal**. Our findings reveal that **the combination of Entropy and SND fails to** demonstrate superior performance over either individual method, while consistently, **MixVal significantly outperforms Entropy+SND** across various tasks. \n\n> **Q2**: There are several papers utilizing Mixup in UDA. It requires not simply mentioning those papers, but rather providing a detailed summary and the differences from them. (e.g., CoWA-JMDS [1])\n\n**A2**:  We appreciate your reference to CoWA-JMDS [1]; we will certainly integrate it into our discussion on related research. It's important to emphasize that **our submission markedly differs from this specific paper**. While the mentioned work contributes to enhancing model performance within UDA, our focus centers on the often-overlooked yet significant model selection challenge in UDA. We've already included references to numerous UDA and semi-supervised learning papers that employ Mixup in L132. Additionally, we have **conducted comprehensive experiments employing MixVal for model selection with the UDA method DMRL [32]**, providing validation results in Table 8 and thorough analysis in L286-L289. This experiment demonstrates that our MixVal approach remains effective for UDA methods utilizing mixup for model training.\n\n> **Q3**: Since Mixup is used for intra-cluster and inter-cluster augmentation, it is possible to utilize other famous Mixup variants like Manifold Mixup [2] or CutMix [3]. However, there are no relevant experiments conducted in this regard.\n\n**A3**:  We appreciate your insights. To clarify, **we've already conducted experiments wherein we substituted mixup with Manifold Mixup and CutMix**. We provide a comparison of the **results in Figure 2 (a) and detailed analysis in L274-L278**. Notably, our findings indicate that image-level mixup outperforms other consistency regularizations including Manifold Mixup and CutMix.\n\n[A] MixUp as Locally Linear Out-Of-Manifold Regularization, AAAI 2019\n\n[B] On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks, NeurIPS 2019"}, {"review_id": "2hQVLUCiWx", "reviewer": "Reviewer_zt6b", "summary": "This paper proposed a new framework to validate the domain-adapted model through Mixed samples. Leveraging mixed samples is not new, but application to the domain-adapted model validation is novel. The framework is very general, so it can be combined with any adaptation methods. Experiments are very extensive over various domain adaptation methods over different validation methods including SND and Corr-C, and the authors provide the supporting evidence as much as possible. In many cases, the proposed framework shows superior performance.", "questions": "1. L194-195: How can we calculate ICE score as an accuracy between $y^{i}$ and $\\hat{y}^{i}=(...,1/2,...,1/2,...)$ (for example) because $\\hat{y}$ is one-hot pseudo-label according to L189?\n2. L244: Any more justification on why $\\lambda=0.55$? How's the performance variation depending on the choice of $\\lambda$?", "limitations": "I'm on the positive side, but without looking at the codes and theoretical justifications, the reproducibility cannot be fully verified.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "In the domain adaptation scenario, the target labels are generally not available. The performance of the adapted model is not typically very robust, so we might want to try the adaptation several times, but we do not know which model is better. So a good model validation framework is necessary. Therefore, the problem is well-defined and well-motivated in practice.\n\nThe main strengths of this paper are 1) simplicity, 2) generalizability, and 3) superior performance.\nIn addition, the claimed arguments are well-supported by extensive experiments, including Segmentation and Backbone change L301-304.", "weaknesses": "The main weakness of this paper is (as the authors also mentioned) the lack of theoretical analysis. This paper would add more value if the authors provided meaningful analysis to support the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for both your recognition of our contributions and your valuable and comprehensive feedback. We've taken each of your concerns into account and have provided detailed responses below.\n\n> **Q1**:  The main weakness of this paper is (as the authors also mentioned) the lack of theoretical analysis. This paper would add more value if the authors provided meaningful analysis to support the framework.\n\n**A1**: Thank you for your constructive suggestion. We acknowledge rigorous theoretical analysis of our method could shed more light on the challenge of model selection in domain adaptation. We recognize this as an avenue for future work. Although our current submission lacks extensive theoretical analysis, we have diligently compensated by offering an in-depth empirical analysis of our validation method, MixVal. This analysis is further elaborated in Appendix B and is accompanied by new comparisons, including **a comparison with the combination of Entropy [44] and SND [27], as depicted in Table 1 of the attached PDF in the global rebuttal.**\n\n> **Q2**:  L194-195: How can we calculate ICE score as an accuracy between $y^i$ and $y^i = (...,1/2,...,1/2,...) $ (for example) because \n $\\hat{y}$ is one-hot pseudo-label according to L189?\n\n**A2**: The notation $\\hat{y}$ denotes the pseudo label predicted by the fixed UDA model. In particular, $\\hat{y}_t^i$ signifies the pseudo label for target sample $x_t^i$. During mixup, when we combine target samples $x_t^i$ and $x_t^j$ to create a mixed sample, we utilize the hard one-hot labels from $\\hat{y}_t^i$ and $\\hat{y}_t^j$ along with a mix ratio exceeding 0.5. This choice ensures that the interpolated label of the mixed sample **avoids the scenario of (...,1/2,...,1/2,...)**. When computing the ICE score for this mixed sample, we predict its hard pseudo label using the fixed UDA model and compare it against the interpolated label. This comparison determines whether the mixed sample is accurately or inaccurately predicted.\n\n> **Q3**:  L244: Any more justification on why $\\lambda=0.55$? How's the performance variation depending on the choice of $\\lambda$?\n\n**A3**: The mix ratio $\\lambda$ in mixup controls the level of ambiguity of mixed samples. A common heuristic is that a value close to 0.5 for $\\lambda$ promotes more ambiguous in-between samples, potentially possessing stronger discriminatory capabilities. In contrast, a value approaching 1.0 generates simpler samples with lower discriminatory potential. This principle is supported by the results from Figure 3, where performance at $\\lambda=0.9$ is notably suboptimal. Hence, we set $\\lambda=0.55$ for all our experiments to ensure stable validation performance.\n\n> **Q4**:   without looking at the codes and theoretical justifications, the reproducibility cannot be fully verified\n\n**A4**: Our MixVal approach is straightforward, requiring no extra model re-training or extensive hyperparameter tuning. We've included detailed PyTorch-style pseudocode in Appendix A, covering every algorithm detail and step. Furthermore, we'll release our full code, including model training and evaluation, to ensure robust reproducibility of our results."}]

Author rebuttal:
We sincerely appreciate the reviewers for their valuable time and feedback. Particularly, we are grateful for the following recognitions:

- Our work investigates a **crucial** [Reviewer PTRC] problem which is **well-defined and well-motivated in practice.** [Reviewer zt6b].
- Our method MixVal is **motivated** [Reviewer G1xN], **novel** [Reviewer 4S59], **simple** [Reviewer zt6b], **effective** [Reviewer PTRC, Reviewer zt6b], and **robust** [Reviewer G1xN, Reviewer PTRC, Reviewer zt6b].
- Our experiments are **impressive** [Reviewer G1xN], **extensive** [Reviewer zt6b] and **state-of-the-art** [Reviewer 4S59] to demonstrate the effectiveness of our method. 
- Our work has good soundness [Reviewer 4S59, Reviewer PTRC, Reviewer zt6b],  good presentation [Reviewer G1xN, Reviewer PTRC, Reviewer zt6b], and excellent contribution [Reviewer zt6b].

Meta-review: This paper introduces MixVal, a target-only framework that leverages Mixup to generate mixed samples for validation. While the utilization of mixed samples is not without precedent, its application to the validation of domain-adapted models is a distinctive contribution of this work. MixVal employs intra-cluster and inter-cluster Mixup to capture the inductive biases of neighborhood consistency and low-density separation. Extensive evaluations over various domain adaptation methods and different settings prove the effectiveness and generalization ability of the proposed method.
Decision: Accept (poster)