Title: Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Unsupervised Domain Adaptation; Model Selection; Hyperparameter Selection; Unsupervised Validation;
Abstract: Unsupervised domain adaptation (UDA) has been widely applied in improving model generalization on unlabeled target data. However, accurately selecting the best UDA model for the target domain is challenging due to the absence of labeled target data and domain distribution shifts. Traditional model selection approaches involve training extra models with source data to estimate the target validation risk. Recent studies propose practical methods that are based on measuring various properties of model predictions on target data. Although effective for some UDA models, these methods often lack stability and may lead to poor selections for other UDA models.
In this paper, we present MixVal, an innovative model selection method that operates solely with unlabeled target data during inference. MixVal leverages mixed target samples with pseudo labels to directly probe the learned target structure by each UDA model. Specifically, MixVal employs two distinct types of probes: the intra-cluster mixed samples for evaluating neighborhood density and the inter-cluster mixed samples for investigating the classification boundary. With this comprehensive probing strategy, MixVal elegantly combines the strengths of two state-of-the-art model selection methods, Entropy and SND. We extensively evaluate MixVal on 11 UDA methods across 4 adaptation settings, including classification and segmentation tasks. Experimental results consistently demonstrate that MixVal achieves state-of-the-art performance and maintains exceptional stability in model selection. 
Code is available at \url{https://github.com/LHXXHB/MixVal}.

Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation

Abstract

Unsupervised domain adaptation (UDA) has been widely applied in improving model generalization on unlabeled target data. However, accurately selecting the best UDA model for the target domain is challenging due to the absence of labeled target data and domain distribution shifts. Traditional model selection approaches involve training extra models with source data to estimate the target validation risk. Recent studies propose practical methods that are based on measuring various properties of model predictions on target data. Although effective for some UDA models, these methods often lack stability and may lead to poor selections for other UDA models. In this paper, we present MixVal, an innovative model selection method that operates solely with unlabeled target data during inference. MixVal leverages mixed target samples with pseudo labels to directly probe the learned target structure by each UDA model. Specifically, MixVal employs two distinct types of probes: the intra-cluster mixed samples for evaluating neighborhood density and the inter-cluster mixed samples for investigating the classification boundary. With this comprehensive probing strategy, MixVal elegantly combines the strengths of two state-of-the-art model selection methods, Entropy and SND. We extensively evaluate MixVal on 11 UDA methods across 4 adaptation settings, including classification and segmentation tasks. Experimental results consistently demonstrate that MixVal achieves state-of-the-art performance and maintains exceptional stability in model selection. Code is available at https://github.com/LHXXHB/MixVal .

1 Introduction

Despite the remarkable achievements of supervised learning in visual recognition tasks [1-4], deep neural networks face challenges when it comes to generalizing to out-of-distribution data [5]. As for the obstacle of out-of-domain generalization, deep domain adaptation techniques [6] provide an effective solution by transferring knowledge from a label-rich source domain to a related, yet label-scarce, target domain. Unsupervised domain adaptation (UDA) has drawn significant interest in recent years within the field of computer vision, including image classification [7-9], semantic segmentation [10-12], and object detection [13, 14], thanks to its practical setup with completely unlabeled target data. As the UDA landscape continues to evolve, various UDA settings have been explored to consider real-world scenarios, such as variations in category overlaps between domains and concerns related to source data privacy. These settings include closed-set UDA [7], partial-set UDA [15, 16], open-set UDA [17], open-partial-set UDA [18], and source-free UDA [19-21]. To effectively address these UDA problems, researchers have developed various novel solutions, including cross-domain alignment techniques [7, 9, 22, 11, 12], target-domain regularization methods [23-25], and self-training strategies in the target domain [26, 27, 20].

* Contact: lhxxhb15@gmail.com . Partial work was done during an internship at ByteDance with Song Bai. † Corresponding authors: Jian Liang (liangjian92@gmail.com), and Xinchao Wang (xinchao@nus.edu.sg).

†

Nevertheless, it is worth emphasizing that hyperparameters, including those associated with both the method-specific loss functions and the deep training pipeline, play a critical role in ensuring superior performance of UDA models during deployment. Notably, even the model trained by a state-of-theart UDA approach can underperform the baseline source-trained model without adaptation if the hyperparameters are not properly set [28]. Surprisingly, the selection of hyperparameters has been largely overlooked in previous UDA studies, as highlighted by You et al. [18] and Saito et al. [29].

To further fill this research gap, this paper tackles the challenging problem of unsupervised model selection * in domain adaptation. Figure 1(a) illustrates the UDA validation pipeline. During the training stage, we employ different values of a hyperparameter β , denoted as β 1 , β 2 , β 3 , to train corresponding UDA models denoted as M 1 , M 2 , M 3 . Subsequently, in the validation stage, our objective is to identify the model with the minimal risk on the target domain and determine its associated β value as the optimal choice for the UDA method. Dealing with model selection in UDA presents two primary challenges. On one hand, the absence of labeled target data makes conventional supervised target validation unfeasible. On the other hand, even if we had access to labeled source data, relying solely on the source risk [7] cannot ensure accurate estimation of the target risk due to domain distribution shifts between domains [29]. Therefore, advanced validation approaches are necessary to ensure accurate model selection in UDA.

Existing validation solutions in UDA can be classified into two types. The first type consists of sourcebased methods, which include the vanilla baseline of using source risk directly for target validation, denoted as SourceVal [7]. Additionally, Importance-Weighted Cross-Validation (IWCV)[30] and Deep Embedded Validation (DEV) [18] re-weight the source risk based on density similarity between domains. Reverse Validation (RV) [31] builds a symmetric UDA task to estimate target risk by reversing the source risk. However, source-based methods require training extra models involving source data and are vulnerable to severe distribution shifts between domains. On the other hand, the second type, target-only validation methods, is more straightforward and only utilizes unlabeled target samples. Entropy [30], based on the assumption of low-density separation [32], selects the model that produces target predictions with the lowest mean entropy. Soft Neighborhood Density (SND) [29], which relies on the assumption of neighborhood consistency , chooses the model with the highest consistency of predictions within the target neighborhood. While both Entropy and SND outperform source-based methods in certain UDA tasks, they have not fully explored the learned target structure for model selection, leading to instability across various UDA methods.

In this paper, we introduce MixVal, a novel target-only method for model selection in UDA. MixVal leverages mixed target samples as validation probes. To achieve this, MixVal employs mixup [33] with unlabeled target samples and their predictions, generating pseudo-labeled mixed samples. These mixed samples are further categorized into two types, depending on whether mixup is performed intra-cluster or inter-cluster. MixVal takes a novel strategy by evaluating inference-stage interpolation consistency for both types of mixed samples. Compared to Entropy and SND, MixVal innovatively employs mixed samples for probing the target structure, rather than directly measuring certain properties of target predictions. Moreover, MixVal elegantly combines the advantages of both Entropy and SND. On one hand, MixVal employs intra-cluster mixed samples to evaluate neighborhood density, leveraging the neighborhood consistency assumption utilized in SND. On the other hand, it utilizes inter-cluster mixed samples to evaluate the classification boundary, considering the lowdensity separation assumption used in Entropy. The novel probing allows MixVal to benefit from the strengths of both assumptions, making it a competitive UDA model selection method.

Our main contributions are highlighted as threefold:

We study the significant yet under-explored model selection problem in UDA, providing comprehensive empirical evaluations of existing validation baselines.

We introduce MixVal, a novel target-only validation method that directly probes the target structure using mixed samples. MixVal combines the strengths of both SND and Entropy through novel consistency-based probing with inter-cluster and intra-cluster mixed samples.

We conduct extensive experiments to evaluate MixVal's effectiveness in model selection across various UDA settings, including closed-set UDA, partial-set UDA, open-partialset UDA, and source-free UDA. The results highlight MixVal's exceptional stability and superior performance in UDA model selection compared to existing baselines.

2 Related Work

2.1 Unsupervised Domain Adaptation

Domain adaptation aims to leverage labeled source domains to facilitate transductive learning in a label-scarce target domain [34]. Unsupervised domain adaptation (UDA), which assumes that the target domain is entirely unlabeled, has gained significant attention due to its practical nature. Many traditional methods have been proposed to address UDA [35-38]. In recent years, deep learning-based UDA methods have witnessed significant progress, with domain adversarial learning emerging as a popular approach. Adversarial domain adaptation has been explored at multiple levels, including input-level [39], feature-level [7, 9, 15], prediction-level [11, 22], and entropy-level [12]. Additionally, there has been growing interest in adapting UDA models with a specific focus on the leverage of target data, leading to explorations in techniques such as target clustering [40, 27, 41] and target output regularization [24, 25]. Notably, Xu et al. [23] proposed a feature-level regularization approach that diverges from the popular mainstream methods. These UDA methods primarily address conventional scenarios that require source data for adaptation. More recently, a practical variant called source-free UDA has gained increasing attention [19, 20], where adaptation relies solely on the source-trained model. To comprehensively compare different validation approaches, we conduct hyperparameter selection for various representative UDA methods across diverse UDA settings.

2.2 Model Selection in Unsupervised Domain Adaptation

Model selection in UDA poses a significant challenge due to the absence of labeled target data and the presence of domain distribution shifts between domains. Some pioneering efforts have been made to address this issue. Ganin et al. [7] initially introduced the estimation of the unavailable target risk using source risk for target-domain validation (SourceVal). However, SourceVal is susceptible to severe domain shifts. Later, Ganin et al. [31] proposed Reverse Validation (RV), which considers a symmetric UDA problem from target to source and employs the reversed source risk for validation. Sugiyama et al. [30] introduced Importance-Weighted Cross-Validation (IWCV), which estimates the target risk by re-weighting the source risk based on input-level domain similarity. You et al. [18] further proposed Deep Embedded Validation (DEV), which considers feature-level similarity and controls variance in IWCV. While these source-based methods have proven effective, their practical applicability is limited by the need of extensive model training, access to source data, and the challenge of dealing with severe domain shifts. Consequently, recent efforts have shifted towards target-only validation methods, which rely solely on unlabeled target data. Morerio et al. [42] pioneered the use of mean entropy (Entropy) of target predictions for validation, considering the assumption of low-density separation [32]. Musgrave et al. [28] introduced the Information Maximization (InfoMax) score, which further considers class diversity in addition to Entropy. Tu et al. [43] introduced Corr-C, which prioritizes predictions with both large class diversity and high confidence. However, class diversity serves as a strict prior that is unsuitable for UDA scenarios with label shift [44]. Saito et al. [29] introduced Soft Neighborhood Density (SND), the state-of-the-art target-only validation method that prioritizes high neighborhood consistency . SND has demonstrated that target-only validation can outperform source-based methods, including IWCV and DEV. We also align with the research line of target-only validation, which offers simplicity and adaptability across various UDA settings. Table 1 showcases a comparison of how our method MixVal differs from other target-only validation methods in terms of the consideration of common assumptions.

2.3 Mixup

Mixup , a data augmentation technique originally proposed by Zhang et al. [33], is used during model training to improve generalization. Mixup has been extended to different levels, such as feature-level [45], patch-level [46], and token-level [47]. In addition to supervised learning, mixup has been successfully applied in semi-supervised learning [48, 49] and domain adaptation [50-52]. Unlike existing studies that consider mixup as a small perturbation to regularize model training, we employ mixup to generate in-between probing samples for model evaluation.

3 Methodology

3.1 Problem Setting

Unsupervised domain adaptation (UDA). We begin by introducing the UDA problem with a K -way image classification task, easily extendable to semantic segmentation tasks. In this setup, we are provided with a labeled source domain D s = { ( x i s , y i s ) } n s i =1 consisting of n s labeled images x s with their corresponding labels y s , and an unlabeled target domain D t = { x i t } n t i =1 containing n t unlabeled images x t . Here, x s and x t represent image vectors, and y s are one-hot ground truth labels. The objective of UDA is to learn a discriminative model M using D s and D t , which can accurately predict the target labels { y i t } n t i =1 for the corresponding target images { x i t } n t i =1 , under covariate shift [30] or label shift [44] between domains. Each image vector x ∈ R d in the input space is associated with a one-hot pseudo label encoding, denoted as ˆ y ∈ R K , which is predicted by model M . We denote |C s | = K as the number of categories in the source domain and |C t | as that in the target domain. Besides the vanilla UDA, which refers to closed-set UDA where both domains share the same label space ( C s = C t ), we also investigate other challenging UDA settings to evaluate the versatility of model selection approaches. These include partial-set UDA [15], where the source domain contains more categories than the target domain ( C s ⊃ C t ), and open-partial-set UDA [53], where both domains have overlapping but not identical labels ( C s ∩ C t = ∅ , C s ̸⊇ C t , C s ̸⊆ C t ).

̸

Traditional UDA settings [34, 6, 15, 53] typically require simultaneous access to source and target data for effective target adaptation. The general optimization objective can be represented as follows:

Here, L src is the cross-entropy loss with labeled source data, and L adapt denotes the adaptation loss that adapts the training of model M to unlabeled target data. The scalar coefficient β is a common type of hyperparameter, and η is a hyperparameter specific to the adaptation loss. Additionally, there are other training-related hyperparameters, such as learning rate, iterations, and network architecture.

Recently, source-free unsupervised domain adaptation (SFUDA) [19, 20] has gained significant attention in the research community. Unlike traditional UDA, SFUDA decouples the source supervised learning and unsupervised target adaptation into two sequential stages. The target adaptation stage in SFUDA is typically guided by the following objective function:

Here, M s represents the source pre-trained model. During the target adaptation stage, a target model M is learned with access to the source model M s and unlabeled target data { x i t } n t i =1 .

Model selection. In UDA, the goal of model selection is to pinpoint the hyperparameter configuration that offers the best performance on unlabeled target data. For example, let's consider the hyperparameter β , with a set of possible values { β i } i m =1 , where m is the number of candidate values. We proceed to train m distinct models { M i } i m =1 , each employing a different β value. Our aim is to identify the model M that delivers the highest performance on D t among all candidate UDA models { M i } i m =1 , thus determining the optimal hyperparameter value. Figure 1(a) visually illustrates this process, using 3 as the value for m . Model selection in UDA confronts two key challenges. Firstly, the absence of labeled target data renders traditional supervised validation methods, such as using a hold-out dataset [54], unfeasible. Secondly, the presence of severe domain distribution shifts between domains poses a risk when using source data for selecting the best model for the target data. Additionally, concerns about source data privacy can limit direct source data utilization.

Source Images

(Labeled)

Target Images

(Unlabeled)

Training Stage

Validation Stage

UDAMethod

𝛽

!

𝛽

"

𝛽

#

𝑀

!

𝑀

"

𝑀

#

Hyperparameter

𝛽

𝑀

"

𝑀

"

is the best model for

target domain, select

𝛽

"

!

UDAmodels

MixVal

…

…

MixVal

Real target samples

Intra-cluster mixup

Inter-cluster mixup

(a) Pipeline of validation in UDA with MixVal

(b) Two types of probes

3.2 MixVal: Mixed Samples as Probes for Unsupervised Validation

Motivation. While existing target-only model selection methods [42, 29] have demonstrated competitive performance, they sometimes result in extremely poor selections when applied to different UDA methods and datasets, even within the standard closed-set UDA scenario. This phenomenon is substantiated by a comprehensive large-scale empirical study conducted by Musgrave et al. [28]. We highlight two common limitations in existing target-only validation methods that potentially hinder their ability to generalize across various UDA scenarios. First, these methods primarily rely on raw target predictions to calculate specific metrics for evaluating the learned target structure. However, it's important to note that these target data have already been utilized during the transductive model training. Second, while both the assumptions of low-density separation and neighborhood consistency are effective for model selection, existing validation methods often design their metrics based solely on one of these assumptions, as summarized in Table 1.

To address these limitations, we present MixVal, a target-only validation method that thoroughly evaluates the target structure through the use of mixed samples. We employ the mixup technique [33] to create novel in-between target samples. These mixed samples are then used for consistency evaluation to probe the learned target structure. With two distinct types of mixed samples, we ensure a comprehensive assessment of the target structure acquired by each candidate UDA model.

Generation of mixed samples. We first generate a labeled set of mixed samples using mixup with unlabeled target data and inference-stage UDA models. Specifically, we create a mixed sample x mix and its label y mix by performing a convex combination of a pair of target samples x i t , x j t and their corresponding pseudo labels ˆ y i t , ˆ y j t which are predicted by the UDA model. The process of mixed sample generation is formulated as follows:

where λ is a scalar used for interpolation, and ˆ y i t and ˆ y j t denote the one-hot pseudo label encodings. By performing mixup on random target samples for a single epoch in the inference stage, we generate a set of mixed samples { ( x i mix , y i mix ) } n t i =1 , where n t represents the total number.

Interpolation consistency evaluation. Next, we leverage each candidate UDA model to infer labels for all mixed samples, resulting in predicted labels { ˆ y i mix } n t i =1 . Using both the mixed labels { y i mix } n t i =1 and predicted labels, we assess the accuracy of mixed samples for each candidate model. This assessment yields the Interpolation Consistency Evaluation (ICE) score, defined as follows:

For all UDA models { M i } i m =1 , we apply the same mixup operation to generate identical mixed samples, but the mixed labels differ because model predictions for target samples vary among different models. We subsequently calculate the ICE score for each candidate model M i , yielding a set of ICE scores { ICE i } m i=1 . The candidate model with a higher ICE score is deemed superior.

MixVal via two types of probes. Through a meticulous analysis of the mixup operation applied to unlabeled target samples and their pseudo labels, we have identified two distinct categories of mixed samples, as illustrated in Figure 1(b). The first type involves mixing target samples with the same pseudo label, referred to as intra-cluster mixup . When we evaluate the ICE score after performing intra-cluster mixup , we effectively measure the neighborhood density within each cluster. A higher ICE score indicates a higher level of neighborhood consistency [29]. Conversely, the second type involves mixing target samples with different pseudo labels, termed inter-cluster mixup . By evaluating the ICE score in this context, we can assess the classification margin between clusters. A higher ICE score in inter-cluster mixup signifies a larger margin of the classification boundary, indicating low-density separation [42]. For comprehensive probing, MixVal uses both types of mixed samples as probes. It separately ranks all candidate models in ascending order based on each type of ICE score. Then, MixVal takes the average rank from both rankings to select the candidate model with the highest average rank. Kindly refer to Appendix A for the pseudocode of MixVal.

Connection to interpolation consistency training. We provide a comprehensive comparison between our MixVal approach and interpolation consistency training (ICT), a commonly used technique in semi-supervised learning [48]. While our ICE score draws inspiration from ICT, there are three significant distinctions between MixVal and ICT. ICT primarily employs inter-cluster mixup as a minor training perturbation to regularize model learning with unlabeled data. In contrast, MixVal explicitly applies mixup to create in-between target samples, encompassing both inter-cluster and intra-cluster scenarios, during the inference stage without requiring model re-training.

4 Experiments

4.1 Setup

Datasets. For image classification tasks, we consider 4 popular UDA benchmarks of varied scales. Office-31 [55] is a classic domain adaptation benchmark consisting of 31 object categories across 3 domains: Amazon (A), DSLR (D), and Webcam (W). Office-Home [56] is a challenging benchmark with 65 different object categories in 4 domains: Art (Ar), Clipart (Cl), Product (Pr), and Real-World (Re). VisDA [57] is a large-scale benchmark for the synthetic-to-real object recognition task, featuring 12 categories. It consists of a training (T) domain with 152k synthetic images and a validation (V) domain with 55k realistic images. DomainNet [58] is a recent large-scale benchmark comprising approximately 600k images across 345 categories in 6 distinct domains. For evaluation, we focus on a subset of 126 classes with 7 tasks [59] from 4 domains: Real (R), Clipart (C), Painting (P), and Sketch (S). For semantic segmentation tasks, we use the synthetic GTAV [60] dataset as the source domain and the real-world Cityscapes [61] dataset as the target domain.

UDA methods. We use the validation baselines discussed in Section 2.2 to conduct model selection for various UDA methods, with a specific emphasis on the practical target-only baselines. For closedset UDA, we consider ATDOC [27], BNM [24], CDAN [9], MCC [25], MDD [22], SAFN [23], DMRL [50], AdaptSeg [11], and AdvEnt [12]. For partial-set UDA, we consider PADA [15] and SAFN [23]. For source-free UDA, we consider SHOT [20]. For open-partial-set UDA, we consider DANCE [41]. For ATDOC, BNM, CDAN, PADA, SAFN, SHOT, DMRL, and DANCE, we select the loss coefficient among 7 varied candidate values. For MDD, we validate the margin factor, while for MCC, we validate the temperature. For AdaptSeg and AdvEnt, we validate both the loss coefficient and training iteration. For MCC and MDD, we also include a two-hyperparameter validation task, where the bottleneck dimension is considered as an additional hyperparameter, with 4 possible values. Kindly refer to Appendix B for details of hyperparameter settings.

Implementation details. We utilize the Transfer Learning Library † to train UDA models on a single RTX TITAN 16GB GPU. The batch size is set to 32, and the total number of iterations is set to 5,000. We save the final model as a checkpoint. For VisDA and GTAV2Cityscapes , we use ResNet-101 [3], for DomainNet , we use ResNet-34 [3], and for the other benchmarks, we use ResNet-50 [3]. Regarding SND [29], we employ the official implementation. For source-based methods, we split 80% of the source data as the training set and the remaining 20% as the validation set. In our MixVal approach, we use a fixed value of λ to ensure a fair comparison among candidate models. Specifically, we set λ = 0 . 55 for all experiments, which allows us to probe with in-between samples.

4.2 Results

We report MixVal's validation performance as averages from three random runs. In the tables, 'Best' signifies the optimal selection, and 'Worst' represents the worst one. For brevity, we report average results for tasks with the same target domain. Kindly refer to Appendix D for detailed results.

Closed-set UDA. We begin by comparing the performance of validation baselines within the standard closed-set UDA scenario. Specifically, we report the results for both source-based and target-only validation methods across several well-established UDA benchmarks. Table 2 provides the results on the medium-scale benchmark Office-Home , while Table 3 presents the results on the small-scale benchmark Office-31 and the large-scale benchmark VisDA . Furthermore, Table 4 presents the results

of practical target-only validation baselines on the large-scale benchmark DomainNet . The current state-of-the-art source-based baseline is DEV [18], and for target-only methods, SND [29] holds the state-of-the-art position. Nonetheless, our observations differ from previous findings. When considering the averaged validation accuracy across 6 popular UDA methods, we note that RV consistently outperforms other source-based validation methods across three benchmarks. Regarding target-only validation methods, entropy-based approaches like Entropy and InfoMax prove to be competitive, particularly demonstrating a significant advantage over SND on the Office-Home benchmark. As expected, our MixVal consistently attains the highest average accuracy across all four benchmarks, surpassing the vanilla SourceVal baseline, while most existing validation methods may noticeably underperform in comparison to this baseline. Notably, MixVal demonstrates exceptional performance on large-scale benchmarks such as VisDA and DomainNet , approaching the upper bound of 'Best' and outperforming the second-best method by substantial margins. MixVal's consistent advantages in closed-set UDA highlight its effectiveness in this widely studied setting.

Partial-set UDA. Following [29], we assess validation performance in partial-set UDA, a scenario with label shifts between domains. We validate two representative partial-set UDA methods, PADA and SAFN, and present the validation accuracy of Office-Home in Table 5. We observe that MixVal consistently outperforms all other validation baselines, maintaining performance close to the 'Best'. However, we notice some differences in comparison to closed-set UDA. Specifically, SND achieves more stable validation performance than Entropy, because Entropy can be susceptible to structure collapse [29]. Second, methods emphasizing 'class diversity', such as InfoMax and Corr-C, often yield lower results than SourceVal. Additionally, RV consistently underperforms SourceVal in scenarios without symmetric label distribution, making it unsuitable for UDA with label shifts.

Open-partial-set UDA. We extend our evaluation of UDA scenarios with label shifts, conducting validation for a popular open-partial-set UDA method, DANCE. Table 6 presents the HOS [62, 63] results on Office-Home . MixVal consistently provides strong validation performance, nearing the 'Best', while other validation baselines tend to align with the 'Worst'.

Source-free UDA. Finally, we compare the target-only validation baselines in source-free UDA. We conduct validation for a popular method, SHOT, and report the accuracy of Office-31 and VisDA in Table 6. We find that MixVal consistently outperforms other baselines in this practical setting.

4.3 Analysis

Influence of pseudo label quality. Because MixVal utilizes pseudo labels of target data for the mixup operation, it is crucial to assess the impact of pseudo label quality. When examining UDA tasks with

CDAN-

VisDA

PADA-

Office

Task

50

60

70

80

90

Accuracy (%)

SSLAug

CutMix

FeatMix

Mixup

1.0

1.5

2.0

2.5

3.0

3.5

4.0

256

512

1024

2048

59.561.158.050.244.944.543.0

57.560.360.457.449.646.746.1

56.959.159.859.256.551.347.3

57.158.659.759.859.557.453.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

13

19

26

12

3

1

0

9

17

22

24

14

5

2

6

10

16

20

23

21

8

4

7

11

15

18

25

27

SND

1.0

1.5

2.0

2.5

3.0

3.5

4.0

13

23

11

6

3

1

0

17

24

26

15

7

4

2

9

18

22

25

19

8

5

12

14

16

21

27

20

10

MixVal

0

10

20

(a)

(b)

low-quality pseudo labels (around 50% accuracy), such as the results of ' → Cl', it becomes evident that MixVal consistently maintains stable validation performance.

Influence of consistency evaluation. MixVal utilizes image-level mixup -based consistency evaluation for the target probing. To gain a deeper understanding of its role in MixVal, we substitute it with other consistency evaluation techniques, including instance-based augmentations such as RandAug [64] and SSLAug [65], as well as mixup at other levels, such as CutMix [46] and FeatMix [45]. The results in Figure 2(a) consistently demonstrate that image-level mixup outperforms other strategies.

Two-hyperparameter validation. We further evaluate all validation baselines in challenging tasks that involve two hyperparameters. Quantitative comparisons in Table 7 consistently position MixVal as one of the top-performance validators. Qualitative comparisons in Figure 2(b) clearly illustrate that MixVal scores demonstrate a strong correlation with actual accuracy, whereas SND [29] scores exhibit noticeable deviation, resulting in selections far from optimal.

Validation of mixup -based UDA method. The ICE score of MixVal is based on the evaluation of mixup -based interpolation consistency during the inference stage. To assess MixVal's robustness, we apply it in the validation of the UDA method DMRL [50], which incorporates mixup -based consistency training within the target domain and across two domains at the same time. MixVal is employed to determine the optimal consistency loss coefficient for DMRL, and the results are reported in Table 8. Notably, MixVal consistently outperforms most of the existing target-only validation baselines, even when subjected to the 'attack' posed by specialized UDA model training.

0.5

0.6

0.7

0.8

0.9

58

62

66

70

Accuracy(%)

Worst

Best

RankAvg-Hard

AccAvg

RankInter

RankIntra

Analysis of probing. In MixVal, we generate mixed samples by applying mixup with a mix ratio λ of 0.55 to both target samples and their hard pseudo labels. We then calculate ICE scores for both intra-cluster and inter-cluster mixed samples. The final score for model selection is determined by averaging ascending rankings of both types of ICE scores. Here, we present a comprehensive ablation study to delve into these design choices within MixVal. To do this, we conduct validation experiments for PADA on four partial-set UDA tasks (Ar → Cl, Cl → Pr, Pr → Re, Re → Ar) and report the average accuracy results in Figure 3. 'AccAvg' represents the direct average of ICE scores (i.e., accuracy values) as the final score for model selection. 'RankAvg' denotes the use of the average ranking of ICE scores for selection. 'RankInter' relies solely on inter-cluster mixed samples for probing. 'RankIntra' employs only intra-cluster mixed samples for probing. 'Hard' and 'Soft' indicate the utilization of hard and soft pseudo labels, respectively. It's worth noting that for intra-cluster mixed samples, there is no distinction between 'Hard' and 'Soft' labels, as all pseudo labels involved are the same. The observations drawn from Figure 3 are as follows: ( i ) The mix ratio ( λ ) influences the level of ambiguity in mixed samples. In MixVal, λ = 0 . 55 generates more ambiguous in-between samples with greater probing capabilities than the easier samples created by λ = 0 . 9 . Consequently, validation performance at λ = 0 . 55 surpasses that at λ = 0 . 9 . ( ii ) In MixVal, while soft pseudo labels may provide additional performance improvements for inter-cluster probing, the use of hard pseudo labels offers simplicity and stability advantages. ( iii ) The separate probing of each type of mixed samples demonstrates effectiveness, and the combination of intra-cluster and inter-cluster probing enhances MixVal's stability. Kindly refer to Appendix C for more discussions on probing.

Extension to segmentation. We extend the use of MixVal to the domain adaptive segmentation task on GTAV2Cityscapes , following the same configuration as SND [29]. Our model selection process involves 66 candidate models generated by varying two hyperparameters: the loss coefficient and training iteration. We save checkpoints every 2,000 iterations, starting from the 10,000th iteration to the 30,000th iteration. The results in Table 9 demonstrate that MixVal consistently delivers the top validation performance for both UDA methods, while Entropy consistently underperforms SourceVal.

Influence of backbone. In addition to ResNet backbones, we also utilize ViT-B [4] as the backbone for the validation of BNM on the task R → S. Accuracies reported in Table 10 highlight MixVal's superior stability with ViT, especially in comparison to Entropy, InfoMax, and Corr-C.

Limitations and impacts. While MixVal excels in classification tasks, further research is needed to address its limitations. This includes establishing the theoretical foundations of probing data structure using pseudo-labeled mixed samples. Additionally, extending MixVal to other machine learning tasks like object detection and regression presents challenges. Poor hyperparameter selections could potentially have negative societal impacts on the real-world deployment of UDA models. Nevertheless, we have not observed such situations during our extensive evaluation of MixVal.

5 Conclusion

In summary, we introduce MixVal, a novel and straightforward target-only method for model selection in unsupervised domain adaptation (UDA). MixVal leverages inference-stage mixup to generate two distinct types of mixed samples, facilitating effective probing of the learned target structure while elegantly considering key assumptions used in previous approaches. Our extensive evaluations encompass diverse UDA methods and adaptation scenarios, consistently affirming the superior and reliable performance of MixVal when compared to existing model selection methods.

References

Russakovsky, O., J. Deng, H. Su, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 2015.

Krizhevsky, A., I. Sutskever, G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems . 2012.

He, K., X. Zhang, S. Ren, et al. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition . 2016.

Dosovitskiy, A., L. Beyer, A. Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations . 2021.

Arjovsky, M., L. Bottou, I. Gulrajani, et al. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Long, M., Y. Cao, J. Wang, et al. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning . 2015.

Ganin, Y., V. Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning . 2015.

Saito, K., K. Watanabe, Y. Ushiku, et al. Maximum classifier discrepancy for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition . 2018.

Long, M., Z. Cao, J. Wang, et al. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems . 2018.

Hoffman, J., D. Wang, F. Yu, et al. Fcns in the wild: Pixel-level adversarial and constraint-based adaptation. arXiv preprint arXiv:1612.02649 , 2016.

Tsai, Y.-H., W.-C. Hung, S. Schulter, et al. Learning to adapt structured output space for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition . 2018.

Vu, T.-H., H. Jain, M. Bucher, et al. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition . 2019.

Chen, Y., W. Li, C. Sakaridis, et al. Domain adaptive faster r-cnn for object detection in the wild. In IEEE Conference on Computer Vision and Pattern Recognition . 2018.

Saito, K., Y . Ushiku, T. Harada, et al. Strong-weak distribution alignment for adaptive object detection. In IEEE Conference on Computer Vision and Pattern Recognition . 2019.

Cao, Z., L. Ma, M. Long, et al. Partial adversarial domain adaptation. In European Conference on Computer Vision . 2018.

Liang, J., Y . Wang, D. Hu, et al. A balanced and uncertainty-aware approach for partial domain adaptation. In European Conference on Computer Vision . 2020.

Panareda Busto, P., J. Gall. Open set domain adaptation. In IEEE International Conference on Computer Vision . 2017.

You, K., X. Wang, M. Long, et al. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning . 2019.

Li, R., Q. Jiao, W. Cao, et al. Model adaptation: Unsupervised domain adaptation without source data. In IEEE Conference on Computer Vision and Pattern Recognition . 2020.

Liang, J., D. Hu, J. Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning . 2020.

Liang, J., R. He, T. Tan. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361 , 2023.

Zhang, Y., T. Liu, M. Long, et al. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning . 2019.

Xu, R., G. Li, J. Yang, et al. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In IEEE International Conference on Computer Vision . 2019.

Cui, S., S. Wang, J. Zhuo, et al. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In IEEE Conference on Computer Vision and Pattern Recognition . 2020.

Jin, Y., X. Wang, M. Long, et al. Minimum class confusion for versatile domain adaptation. In European Conference on Computer Vision . 2020.

Zou, Y., Z. Yu, B. Kumar, et al. Unsupervised domain adaptation for semantic segmentation via classbalanced self-training. In European Conference on Computer Vision . 2018.

Liang, J., D. Hu, J. Feng. Domain adaptation with auxiliary target domain-oriented classifier. In IEEE Conference on Computer Vision and Pattern Recognition . 2021.

Musgrave, K., S. Belongie, S.-N. Lim. Benchmarking validation methods for unsupervised domain adaptation. arXiv preprint arXiv:2208.07360 , 2022.

Saito, K., D. Kim, P. Teterwak, et al. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In IEEE International Conference on Computer Vision . 2021.

Sugiyama, M., M. Krauledat, K.-R. Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 2007.

Ganin, Y., E. Ustinova, H. Ajakan, et al. Domain-adversarial training of neural networks. Journal of Machine Learning Research , 2016.

Grandvalet, Y., Y. Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems . 2004.

Zhang, H., M. Cisse, Y. N. Dauphin, et al. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations . 2018.

Pan, S. J., Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering , 2009.

Gong, B., Y. Shi, F. Sha, et al. Geodesic flow kernel for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition . 2012.

Fernando, B., A. Habrard, M. Sebban, et al. Unsupervised visual domain adaptation using subspace alignment. In IEEE International Conference on Computer Vision . 2013.

Sun, B., J. Feng, K. Saenko. Return of frustratingly easy domain adaptation. In AAAI Conference on Artificial Intelligence . 2016.

Pan, S. J., I. W. Tsang, J. T. Kwok, et al. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks , 2010.

Hoffman, J., E. Tzeng, T. Park, et al. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning . 2018.

Shu, R., H. H. Bui, H. Narui, et al. A dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv:1802.08735 , 2018.

Saito, K., D. Kim, S. Sclaroff, et al. Universal domain adaptation through self supervision. In Advances in Neural Information Processing Systems . 2020.

Morerio, P., J. Cavazza, V. Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. arXiv preprint arXiv:1711.10288 , 2017.

Tu, W., W. Deng, T. Gedeon, et al. Assessing model out-of-distribution generalization with softmax prediction probability baselines and a correlation method, 2023.

Azizzadenesheli, K., A. Liu, F. Yang, et al. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:1903.09734 , 2019.

Verma, V., A. Lamb, C. Beckham, et al. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning . 2019.

Yun, S., D. Han, S. J. Oh, et al. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE International Conference on Computer Vision . 2019.

Chen, J.-N., S. Sun, J. He, et al. Transmix: Attend to mix for vision transformers. In IEEE Conference on Computer Vision and Pattern Recognition . 2022.

Verma, V., K. Kawaguchi, A. Lamb, et al. Interpolation consistency training for semi-supervised learning. IEEE Transactions on Neural Networks , 2022.

Berthelot, D., N. Carlini, I. Goodfellow, et al. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems . 2019.

Wu, Y., D. Inkpen, A. El-Roby. Dual mixup regularized learning for adversarial domain adaptation. In European Conference on Computer Vision . 2020.

Mao, X., Y. Ma, Z. Yang, et al. Virtual mixup training for unsupervised domain adaptation. arXiv preprint arXiv:1905.04215 , 2019.

Yan, S., H. Song, N. Li, et al. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677 , 2020.

You, K., M. Long, Z. Cao, et al. Universal domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition . 2019.

Goodfellow, I., Y. Bengio, A. Courville. Deep learning . MIT press, 2016.

Saenko, K., B. Kulis, M. Fritz, et al. Adapting visual category models to new domains. In European Conference on Computer Vision . 2010.

Venkateswara, H., J. Eusebio, S. Chakraborty, et al. Deep hashing network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition . 2017.

Peng, X., B. Usman, N. Kaushik, et al. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 , 2017.

Peng, X., Q. Bai, X. Xia, et al. Moment matching for multi-source domain adaptation. In IEEE International Conference on Computer Vision . 2019.

Saito, K., D. Kim, S. Sclaroff, et al. Semi-supervised domain adaptation via minimax entropy. In IEEE International Conference on Computer Vision . 2019.

Richter, S. R., V . Vineet, S. Roth, et al. Playing for data: Ground truth from computer games. In European Conference on Computer Vision . 2016.

Cordts, M., M. Omran, S. Ramos, et al. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition . 2016.

Fu, B., Z. Cao, M. Long, et al. Learning to detect open classes for universal domain adaptation. In European Conference on Computer Vision . 2020.

Bucci, S., M. R. Loghmani, T. Tommasi. On the effectiveness of image rotation for open set domain adaptation. In European Conference on Computer Vision . 2020.

Cubuk, E. D., B. Zoph, J. Shlens, et al. Randaugment: Practical automated data augmentation with a reduced search space. In IEEE Conference on Computer Vision and Pattern Recognition, Workshops . 2020.

Chen, X., H. Fan, R. Girshick, et al. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Duda, R. O., P. E. Hart, et al. Pattern classification and scene analysis . Wiley New York, 1973.

A Algorithm

The PyTorch-style pseudocode for our validation approach MixVal is presented in Algorithm 1.

Algorithm 1 PyTorch-style pseudocode for MixVal.

B Hyperparameter Settings

In our main experiments, we follow the methodology of previous model selection studies [18, 29]. For simplicity, we tune a single hyperparameter for various UDA methods. In the case of twohyperparameter validation experiments, as in MCC [25] and MDD [22], we also tune the bottleneck dimension, choosing from the candidate values { 256 , 512 , 1024 , 2048 } . This variation is due to observations that the bottleneck dimension varies across different datasets in the official code of these UDA methods. For the validation of semantic segmentation tasks, we also consider two hyperparameters with the training iteration as an additional hyperparameter selected from the set { 10 , 000 , 12 , 000 , 14 , 000 , 16 , 000 , 18 , 000 , 20 , 000 , 22 , 000 , 24 , 000 , 26 , 000 , 28 , 000 , 30 , 000 } , following SND [29]. Detailed hyperparameter settings are available in Table 11.

C Analysis of Intra-Cluster and Inter-Cluster Probing

In MixVal, we use mixed samples to directly probe the intra-cluster and inter-cluster structures within the target representations learned by a UDA model. This strategy shares similar spirits with the well-known Linear Discriminant Analysis (LDA) [66], which optimizes an objective focused on minimizing intra-cluster variance while maximizing inter-cluster variance. Drawing inspiration from the LDA optimization, we explore a straightforward validation baseline that employs an LDA-like metric. In this context, let's denote the number of target samples as N , the number of categories as K , and the feature of a sample encoded by a UDA model as f . The formulation for the LDA score is presented in Equation 1 below.

In our implementation, we utilize the predicted logits as features and compute variance using the L 2 distance. We assess the LDA baseline against all target-only validation methods in both closed-

set UDA (Table 12) and partial-set UDA (Table 13). In both UDA scenarios, LDA outperforms existing validation methods in terms of average validation accuracy. This highlights the advantage of considering two effective assumptions, in contrast to the single assumption used in SND and Entropy. It's worth noting that while LDA performs well, it still falls short of MixVal's performance, underscoring the benefits of MixVal's direct probing approach using mixed samples compared to the indirect probing via measurement of raw predictions used by SND, Entropy, and LDA.

D Full Validation Results

For classification tasks in our evaluation, we utilize the HOS score ( % ) [62, 63] for open-partial-set UDA and accuracy ( % ) for all other UDA tasks. In the case of segmentation tasks, we measure mIoU ( % ) [11, 12]. To accommodate space constraints in this main text, we present the averaged validation results for UDA tasks that share the same target domain. For example, ' → Ar' represents the averaged results of three tasks on the Office-Home benchmark, namely 'Cl → Ar', 'Pr → Ar', and 'Re → Ar'. Additionally, please note that the 'avg' row signifies the average of the rows preceding it within each UDA method, while the 'AVG' row represents the average results of 'avg' rows across all considered UDA methods. For more detailed validation results for each specific UDA task, please refer to the corresponding tables, from Table 14 to Table 29.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new framework to validate the domain-adapted model through Mixed samples. Leveraging mixed samples is not new, but application to the domain-adapted model validation is novel. The framework is very general, so it can be combined with any adaptation methods. Experiments are very extensive over various domain adaptation methods over different validation methods including SND and Corr-C, and the authors provide the supporting evidence as much as possible. In many cases, the proposed framework shows superior performance.\n\nSTRENGTHS:\nIn the domain adaptation scenario, the target labels are generally not available. The performance of the adapted model is not typically very robust, so we might want to try the adaptation several times, but we do not know which model is better. So a good model validation framework is necessary. Therefore, the problem is well-defined and well-motivated in practice.\n\nThe main strengths of this paper are 1) simplicity, 2) generalizability, and 3) superior performance.\nIn addition, the claimed arguments are well-supported by extensive experiments, including Segmentation and Backbone change L301-304.\n\nWEAKNESSES:\nThe main weakness of this paper is (as the authors also mentioned) the lack of theoretical analysis. This paper would add more value if the authors provided meaningful analysis to support the framework.\n\nQUESTIONS:\n1. L194-195: How can we calculate ICE score as an accuracy between $y^{i}$ and $\\hat{y}^{i}=(...,1/2,...,1/2,...)$ (for example) because $\\hat{y}$ is one-hot pseudo-label according to L189?\n2. L244: Any more justification on why $\\lambda=0.55$? How's the performance variation depending on the choice of $\\lambda$?\n\nLIMITATIONS:\nI'm on the positive side, but without looking at the codes and theoretical justifications, the reproducibility cannot be fully verified.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nValidating hyperparameters in UDA is challenging due to the unlabeled target data. For it, the paper proposes MixVal, a novel target-only approach that utilizes mixup to synthesize target samples for validation. MixVal combines inductive biases from prior approaches through intra-cluster and inter-cluster mixup, achieving state-of-the-art performance across 11 UDA methods and 4 adaptation settings.\n\nSTRENGTHS:\n1. Validation is crucial in UDA to utilize it in real-world problems, and this paper proposes an effective method to tackle it.\n2. Experimentally, it is clearly robust and effective to model selection across multiple benchmarks.\n\nWEAKNESSES:\n1. I can't find methodological novelty. I think that the proposed method is a combination of Entropy, SND, and Mixup.\n2. There are several papers utilizing Mixup in UDA. It requires not simply mentioning those papers, but rather providing a detailed summary and the differences from them. (e.g., CoWA-JMDS [1])\n3. Since Mixup is used for intra-cluster and inter-cluster augmentation, it is possible to utilize other famous Mixup variants like Manifold Mixup [2] or CutMix [3]. However, there are no relevant experiments conducted in this regard.\n\n[1] Lee, Jonghyun, et al. \"Confidence score for source-free unsupervised domain adaptation.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Verma, Vikas, et al. \"Manifold mixup: Better representations by interpolating hidden states.\" International conference on machine learning. PMLR, 2019.\n\n[3] Yun, Sangdoo, et al. \"Cutmix: Regularization strategy to train strong classifiers with localizable features.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.\n\nQUESTIONS:\nPlease provide an in-depth response regarding the weaknesses mentioned above.\n\nLIMITATIONS:\nThis paper addresses the important research topic, model/hyperparameter validation in UDA. I think that this work can be meaningful to the UDA community. However, it is unclear whether the proposed method is the most optimal approach for validation and even as a work of using Mixup.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
In this paper propose a novel target-only method is proposed to employ mixup to synthesize in-between target samples for validation. MixVal leverages mixed target samples to directly probe the learned target structure, benefiting from an combination of inductive biases considered in prior approaches. MixVal performs intra-cluster and inter-cluster mixup to explicitly capture both inductive biases of neighborhood consistency and low density separation.

STRENGTHS:
1. This work propose a novel solution named MixVal that eliminates the need for source data access and avoids the cumbersome model re-training. Through inter-cluster and intra-cluster mixup, MixVal combines two essential inductive biases utilized in prior validation methods.
2. The experiment results achieve SOTA.

WEAKNESSES:
1. The presentation need to be improved.
2. Using Mixup in domain adaptation is a common practice.

QUESTIONS:
DANCE (NeurIPS 2020) is not the SOTA of Open-partial-set UDA. How about the performance of some later work as listed below?
[1] Guangrui Li et al. Domain Consensus Clustering for Universal Domain Adaptation. CVPR2021
[2] Kuniaki Saito et al. OVANet: One-vs-All Network for Universal Domain Adaptation. ICCV2021
[3] Liang Chen et al. Evidential Neighborhood Contrastive Learning for Universal Domain Adaptation. AAAI2022
[4] Liang Chen at al. Geometric Anchor Correspondence Mining with Uncertainty Modeling for Universal Domain Adaptation. CVPR2022

LIMITATIONS:
The authors addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a simple validation method for unsupervised domain adaption. This method, called MixVal, proposes a new evaluation (ICE) based on MIXUP to select the most appropriate model candidates, which is obtained by training with different hyperparameters, such as loss coefficient/temperature/margin factor. Compared with other target-only validation methods, the paper considered three different inductive biases: neighborhood consistency/low-density separation/no prior label distribution.\n\nSTRENGTHS:\nThis paper is well-written and easy to follow. Specifically, Table 1 and Figure 1 clearly show the motivation and pipeline, respectively.\n\nExperiments over several datasets and settings are impressive which sufficiently show the generalization of the proposed method.\n\nWEAKNESSES:\n**1. Lack novelty.** Behind the mentioned contributions, there is only one proposed technical point, the proposed evaluation ICE with MIXUP, which just takes less than half a page. It could not be enough as a NeurIPS paper, especially lacking sufficient theoretical support. \n\n**2. Lack theoretical support.** The method assumes that the model with a higher ICE score can obtain a higher real target accuracy. This paper needs such theoretical support that there is a positive correlation between the ICE score and the discriminability of the candidate models. \n\nThis assumption could not be valid in real-world systems. Ideally, the best ICE score is 100%. In this case, the “model” could be a linear system, which is certainly impossible in the applications. This might demonstrate there are some theoretical gaps between the score and the goal of UDA.  \n\nThe authors also recognized this limitation and tried to give some empirical observations in Figure 2(b). However, without the second technical point, this limitation seems very serious. \n\nFigure 2(b) looks a little bit tricky, where the first figure is shown with accuracy, and the following subfigures are in the ranked ascending order. In fact, all subfigures should be shown with the same indicator, ranking all model candidates in ascending order. In this case, the empirical conclusion about the high correlation is not convincing. \n\n**3. Confusion about formulation.** The formulation of the mixed pseudo label is different from its pseudocode.\n\nQUESTIONS:\nWhat are the relationships between RankInter and RankIntra? Is that necessary for us to separate such two different types during inference? \n \nIn Figure 3, AccAvg has a similar trend as RankIntra. RankInter and RankIntra seem not to compensate. In other words, handling the inductive bias neighborhood consistency and low-density separation at the same time does not make sure higher performance.\n\nLIMITATIONS:\nPlease see the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-21"}