SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dapeng Hu $^{1*}$  Jian Liang $^{3\dagger}$  Jun Hao Liew $^{2}$  Chuhui Xue $^{2}$  Song Bai $^{2}$  Xinchao Wang $^{1\dagger}$
$^{1}$ National University of Singapore  $^{2}$ ByteDance Inc.
$^{3}$ CRIPAC & MAIS, Institute of Automation, Chinese Academy of Sciences
Unsupervised domain adaptation (UDA) has been widely applied in improving model generalization on unlabeled target data. However, accurately selecting the best UDA model for the target domain is challenging due to the absence of labeled target data and domain distribution shifts. Traditional model selection approaches involve training extra models with source data to estimate the target validation risk. Recent studies propose practical methods that are based on measuring various properties of model predictions on target data. Although effective for some UDA models, these methods often lack stability and may lead to poor selections for other UDA models. In this paper, we present MixVal, an innovative model selection method that operates solely with unlabeled target data during inference. MixVal leverages mixed target samples with pseudo labels to directly probe the learned target structure by each UDA model. Specifically, MixVal employs two distinct types of probes: the intra-cluster mixed samples for evaluating neighborhood density and the inter-cluster mixed samples for investigating the classification boundary. With this comprehensive probing strategy, MixVal elegantly combines the strengths of two state-of-the-art model selection methods, Entropy and SND. We extensively evaluate MixVal on 11 UDA methods across 4 adaptation settings, including classification and segmentation tasks. Experimental results consistently demonstrate that MixVal achieves state-of-the-art performance and maintains exceptional stability in model selection. Code is available at https://github.com/LHXXHB/MixVal.
Despite the remarkable achievements of supervised learning in visual recognition tasks [1-4], deep neural networks face challenges when it comes to generalizing to out-of-distribution data [5]. As for the obstacle of out-of-domain generalization, deep domain adaptation techniques [6] provide an effective solution by transferring knowledge from a label-rich source domain to a related, yet label-scarce, target domain. Unsupervised domain adaptation (UDA) has drawn significant interest in recent years within the field of computer vision, including image classification [7-9], semantic segmentation [10-12], and object detection [13, 14], thanks to its practical setup with completely unlabeled target data. As the UDA landscape continues to evolve, various UDA settings have been explored to consider real-world scenarios, such as variations in category overlaps between domains and concerns related to source data privacy. These settings include closed-set UDA [7], partial-set UDA [15, 16], open-set UDA [17], open-partial-set UDA [18], and source-free UDA [19-21]. To effectively address these UDA problems, researchers have developed various novel solutions, including cross-domain alignment techniques [7, 9, 22, 11, 12], target-domain regularization methods [23-25], and self-training strategies in the target domain [26, 27, 20].
*Contact: lhxxhb15@gmail.com. Partial work was done during an internship at ByteDance with Song Bai.
† Corresponding authors: Jian Liang (liangjian92@gmail.com), and Xinchao Wang (xinchao@nus.edu.sg).
Nevertheless, it is worth emphasizing that hyperparameters, including those associated with both the method-specific loss functions and the deep training pipeline, play a critical role in ensuring superior performance of UDA models during deployment. Notably, even the model trained by a state-of-the-art UDA approach can underperform the baseline source-trained model without adaptation if the hyperparameters are not properly set [28]. Surprisingly, the selection of hyperparameters has been largely overlooked in previous UDA studies, as highlighted by You et al. [18] and Saito et al. [29].
To further fill this research gap, this paper tackles the challenging problem of unsupervised model selection * in domain adaptation. Figure 1(a) illustrates the UDA validation pipeline. During the training stage, we employ different values of a hyperparameter  $\beta$ , denoted as  $\beta_{1}, \beta_{2}, \beta_{3}$ , to train corresponding UDA models denoted as  $M_{1}, M_{2}, M_{3}$ . Subsequently, in the validation stage, our objective is to identify the model with the minimal risk on the target domain and determine its associated  $\beta$  value as the optimal choice for the UDA method. Dealing with model selection in UDA presents two primary challenges. On one hand, the absence of labeled target data makes conventional supervised target validation unfeasible. On the other hand, even if we had access to labeled source data, relying solely on the source risk [7] cannot ensure accurate estimation of the target risk due to domain distribution shifts between domains [29]. Therefore, advanced validation approaches are necessary to ensure accurate model selection in UDA.
Existing validation solutions in UDA can be classified into two types. The first type consists of source-based methods, which include the vanilla baseline of using source risk directly for target validation, denoted as SourceVal [7]. Additionally, Importance-Weighted Cross-Validation (IWCV)[30] and Deep Embedded Validation (DEV) [18] re-weight the source risk based on density similarity between domains. Reverse Validation (RV) [31] builds a symmetric UDA task to estimate target risk by reversing the source risk. However, source-based methods require training extra models involving source data and are vulnerable to severe distribution shifts between domains. On the other hand, the second type, target-only validation methods, is more straightforward and only utilizes unlabeled target samples. Entropy [30], based on the assumption of low-density separation [32], selects the model that produces target predictions with the lowest mean entropy. Soft Neighborhood Density (SND) [29], which relies on the assumption of neighborhood consistency, chooses the model with the highest consistency of predictions within the target neighborhood. While both Entropy and SND outperform source-based methods in certain UDA tasks, they have not fully explored the learned target structure for model selection, leading to instability across various UDA methods.
In this paper, we introduce MixVal, a novel target-only method for model selection in UDA. MixVal leverages mixed target samples as validation probes. To achieve this, MixVal employs mixup [33] with unlabeled target samples and their predictions, generating pseudo-labeled mixed samples. These mixed samples are further categorized into two types, depending on whether mixup is performed intra-cluster or inter-cluster. MixVal takes a novel strategy by evaluating inference-stage interpolation consistency for both types of mixed samples. Compared to Entropy and SND, MixVal innovatively employs mixed samples for probing the target structure, rather than directly measuring certain properties of target predictions. Moreover, MixVal elegantly combines the advantages of both Entropy and SND. On one hand, MixVal employs intra-cluster mixed samples to evaluate neighborhood density, leveraging the neighborhood consistency assumption utilized in SND. On the other hand, it utilizes inter-cluster mixed samples to evaluate the classification boundary, considering the low-density separation assumption used in Entropy. The novel probing allows MixVal to benefit from the strengths of both assumptions, making it a competitive UDA model selection method.
Our main contributions are highlighted as threefold:
- We study the significant yet under-explored model selection problem in UDA, providing comprehensive empirical evaluations of existing validation baselines.
- We introduce MixVal, a novel target-only validation method that directly probes the target structure using mixed samples. MixVal combines the strengths of both SND and Entropy through novel consistency-based probing with inter-cluster and intra-cluster mixed samples.
- We conduct extensive experiments to evaluate MixVal's effectiveness in model selection across various UDA settings, including closed-set UDA, partial-set UDA, open-partial-set UDA, and source-free UDA. The results highlight MixVal's exceptional stability and superior performance in UDA model selection compared to existing baselines.
*Throughout this paper, we use model selection, validation, and hyperparameter selection interchangeably.
Table 1: Comparison of assumptions considered in validation methods using target data.
Validation Method	neighborhood consistency	no prior of class diversity	low-density separation
Entropy [42]	X	✓	✓
InfoMax [28]	X	X	✓
SND [29]	✓	✓	X
Corr-C [43]	X	X	✓
MixVal	✓	✓	✓
2.1 Unsupervised Domain Adaptation
Domain adaptation aims to leverage labeled source domains to facilitate transductive learning in a label-scarce target domain [34]. Unsupervised domain adaptation (UDA), which assumes that the target domain is entirely unlabeled, has gained significant attention due to its practical nature. Many traditional methods have been proposed to address UDA [35-38]. In recent years, deep learning-based UDA methods have witnessed significant progress, with domain adversarial learning emerging as a popular approach. Adversarial domain adaptation has been explored at multiple levels, including input-level [39], feature-level [7, 9, 15], prediction-level [11, 22], and entropy-level [12]. Additionally, there has been growing interest in adapting UDA models with a specific focus on the leverage of target data, leading to explorations in techniques such as target clustering [40, 27, 41] and target output regularization [24, 25]. Notably, Xu et al. [23] proposed a feature-level regularization approach that diverges from the popular mainstream methods. These UDA methods primarily address conventional scenarios that require source data for adaptation. More recently, a practical variant called source-free UDA has gained increasing attention [19, 20], where adaptation relies solely on the source-trained model. To comprehensively compare different validation approaches, we conduct hyperparameter selection for various representative UDA methods across diverse UDA settings.
2.2 Model Selection in Unsupervised Domain Adaptation
Model selection in UDA poses a significant challenge due to the absence of labeled target data and the presence of domain distribution shifts between domains. Some pioneering efforts have been made to address this issue. Ganin et al. [7] initially introduced the estimation of the unavailable target risk using source risk for target-domain validation (SourceVal). However, SourceVal is susceptible to severe domain shifts. Later, Ganin et al. [31] proposed Reverse Validation (RV), which considers a symmetric UDA problem from target to source and employs the reversed source risk for validation. Sugiyama et al. [30] introduced Importance-Weighted Cross-Validation (IWCV), which estimates the target risk by re-weighting the source risk based on input-level domain similarity. You et al. [18] further proposed Deep Embedded Validation (DEV), which considers feature-level similarity and controls variance in IWCV. While these source-based methods have proven effective, their practical applicability is limited by the need of extensive model training, access to source data, and the challenge of dealing with severe domain shifts. Consequently, recent efforts have shifted towards target-only validation methods, which rely solely on unlabeled target data. Morerio et al. [42] pioneered the use of mean entropy (Entropy) of target predictions for validation, considering the assumption of low-density separation [32]. Musgrave et al. [28] introduced the Information Maximization (InfoMax) score, which further considers class diversity in addition to Entropy. Tu et al. [43] introduced Corr-C, which prioritizes predictions with both large class diversity and high confidence. However, class diversity serves as a strict prior that is unsuitable for UDA scenarios with label shift [44]. Saito et al. [29] introduced Soft Neighborhood Density (SND), the state-of-the-art target-only validation method that prioritizes high neighborhood consistency. SND has demonstrated that target-only validation can outperform source-based methods, including IWCV and DEV. We also align with the research line of target-only validation, which offers simplicity and adaptability across various UDA settings. Table 1 showcases a comparison of how our method MixVal differs from other target-only validation methods in terms of the consideration of common assumptions.
2.3 Mixup
Mixup, a data augmentation technique originally proposed by Zhang et al. [33], is used during model training to improve generalization. Mixup has been extended to different levels, such as feature-level [45], patch-level [46], and token-level [47]. In addition to supervised learning, mixup has been successfully applied in semi-supervised learning [48, 49] and domain adaptation [50-52]. Unlike existing studies that consider mixup as a small perturbation to regularize model training, we employ mixup to generate in-between probing samples for model evaluation.
Unsupervised domain adaptation (UDA). We begin by introducing the UDA problem with a  $K$ -way image classification task, easily extendable to semantic segmentation tasks. In this setup, we are provided with a labeled source domain  $\mathcal{D}_{\mathrm{s}} = \{(x_{\mathrm{s}}^{i},y_{\mathrm{s}}^{i})\}_{i = 1}^{n_{\mathrm{s}}}$  consisting of  $n_{\mathrm{s}}$  labeled images  $x_{\mathrm{s}}$  with their corresponding labels  $y_{\mathrm{s}}$ , and an unlabeled target domain  $\mathcal{D}_{\mathrm{t}} = \{x_{\mathrm{t}}^{i}\}_{i = 1}^{n_{\mathrm{t}}}$  containing  $n_{\mathrm{t}}$  unlabeled images  $x_{\mathrm{t}}$ . Here,  $x_{\mathrm{s}}$  and  $x_{\mathrm{t}}$  represent image vectors, and  $y_{\mathrm{s}}$  are one-hot ground truth labels. The objective of UDA is to learn a discriminative model  $M$  using  $\mathcal{D}_{\mathrm{s}}$  and  $\mathcal{D}_{\mathrm{t}}$ , which can accurately predict the target labels  $\{y_{\mathrm{t}}^{i}\}_{i = 1}^{n_{\mathrm{t}}}$  for the corresponding target images  $\{x_{\mathrm{t}}^{i}\}_{i = 1}^{n_{\mathrm{t}}}$ , under covariate shift [30] or label shift [44] between domains. Each image vector  $x\in \mathcal{R}^d$  in the input space is associated with a one-hot pseudo label encoding, denoted as  $\hat{y}\in \mathcal{R}^K$ , which is predicted by model  $M$ . We denote  $|\mathcal{C}_{\mathrm{s}}| = K$  as the number of categories in the source domain and  $|\mathcal{C}_{\mathrm{t}}|$  as that in the target domain. Besides the vanilla UDA, which refers to closed-set UDA where both domains share the same label space  $(\mathcal{C}_{\mathrm{s}} = \mathcal{C}_{\mathrm{t}})$ , we also investigate other challenging UDA settings to evaluate the versatility of model selection approaches. These include partial-set UDA [15], where the source domain contains more categories than the target domain  $(\mathcal{C}_{\mathrm{s}}\supset \mathcal{C}_{\mathrm{t}})$ , and open-partial-set UDA [53], where both domains have overlapping but not identical labels  $(\mathcal{C}_{\mathrm{s}}\cap \mathcal{C}_{\mathrm{t}}\neq \emptyset ,\mathcal{C}_{\mathrm{s}}\nsubseteq \mathcal{C}_{\mathrm{t}},\mathcal{C}_{\mathrm{s}}\not\subseteq \mathcal{C}_{\mathrm{t}})$ .
Traditional UDA settings [34, 6, 15, 53] typically require simultaneous access to source and target data for effective target adaptation. The general optimization objective can be represented as follows:
$$
\mathcal {L} _ {\mathrm {u d a}} = \mathcal {L} _ {\mathrm {s r c}} (M; \mathcal {D} _ {\mathrm {s}}) + \beta \mathcal {L} _ {\mathrm {a d a p t}} (M; \mathcal {D} _ {\mathrm {s}}, \mathcal {D} _ {\mathrm {t}}, \eta).
$$
Here,  $\mathcal{L}_{\mathrm{src}}$  is the cross-entropy loss with labeled source data, and  $\mathcal{L}_{\mathrm{adapt}}$  denotes the adaptation loss that adapts the training of model  $M$  to unlabeled target data. The scalar coefficient  $\beta$  is a common type of hyperparameter, and  $\eta$  is a hyperparameter specific to the adaptation loss. Additionally, there are other training-related hyperparameters, such as learning rate, iterations, and network architecture.
Recently, source-free unsupervised domain adaptation (SFUDA) [19, 20] has gained significant attention in the research community. Unlike traditional UDA, SFUDA decouples the source supervised learning and unsupervised target adaptation into two sequential stages. The target adaptation stage in SFUDA is typically guided by the following objective function:
$$
\mathcal {L} _ {\mathrm {s f u d a}} = \mathcal {L} _ {\mathrm {a d a p t}} (M; M _ {\mathrm {s}}, \mathcal {D} _ {\mathrm {t}}, \eta).
$$
Here,  $M_{\mathrm{s}}$  represents the source pre-trained model. During the target adaptation stage, a target model  $M$  is learned with access to the source model  $M_{\mathrm{s}}$  and unlabeled target data  $\{x_{\mathrm{t}}^{i}\}_{i = 1}^{n_{\mathrm{t}}}$ .
Model selection. In UDA, the goal of model selection is to pinpoint the hyperparameter configuration that offers the best performance on unlabeled target data. For example, let's consider the hyperparameter  $\beta$ , with a set of possible values  $\{\beta_i\}_{i=1}^m$ , where  $m$  is the number of candidate values. We proceed to train  $m$  distinct models  $\{M_i\}_{i=1}^m$ , each employing a different  $\beta$  value. Our aim is to identify the model  $M$  that delivers the highest performance on  $\mathcal{D}_t$  among all candidate UDA models  $\{M_i\}_{i=1}^m$ , thus determining the optimal hyperparameter value. Figure 1(a) visually illustrates this process, using 3 as the value for  $m$ . Model selection in UDA confronts two key challenges. Firstly, the absence of labeled target data renders traditional supervised validation methods, such as using a hold-out dataset [54], unfeasible. Secondly, the presence of severe domain distribution shifts between domains poses a risk when using source data for selecting the best model for the target data. Additionally, concerns about source data privacy can limit direct source data utilization.
(a) Pipeline of validation in UDA with MixVal
(b) Two types of probes
Figure 1: Overview of the proposed method (MixVal) for model selection in UDA.
3.2 MixVal: Mixed Samples as Probes for Unsupervised Validation
Motivation. While existing target-only model selection methods [42, 29] have demonstrated competitive performance, they sometimes result in extremely poor selections when applied to different UDA methods and datasets, even within the standard closed-set UDA scenario. This phenomenon is substantiated by a comprehensive large-scale empirical study conducted by Musgrave et al. [28]. We highlight two common limitations in existing target-only validation methods that potentially hinder their ability to generalize across various UDA scenarios. First, these methods primarily rely on raw target predictions to calculate specific metrics for evaluating the learned target structure. However, it's important to note that these target data have already been utilized during the transductive model training. Second, while both the assumptions of low-density separation and neighborhood consistency are effective for model selection, existing validation methods often design their metrics based solely on one of these assumptions, as summarized in Table 1.
To address these limitations, we present MixVal, a target-only validation method that thoroughly evaluates the target structure through the use of mixed samples. We employ the mixup technique [33] to create novel in-between target samples. These mixed samples are then used for consistency evaluation to probe the learned target structure. With two distinct types of mixed samples, we ensure a comprehensive assessment of the target structure acquired by each candidate UDA model.
Generation of mixed samples. We first generate a labeled set of mixed samples using mixup with unlabeled target data and inference-stage UDA models. Specifically, we create a mixed sample  $x_{\mathrm{mix}}$  and its label  $y_{\mathrm{mix}}$  by performing a convex combination of a pair of target samples  $x_{\mathrm{t}}^{i}, x_{\mathrm{t}}^{j}$  and their corresponding pseudo labels  $\hat{y}_{\mathrm{t}}^{i}, \hat{y}_{\mathrm{t}}^{j}$  which are predicted by the UDA model. The process of mixed sample generation is formulated as follows:
$$
x _ {\mathrm {m i x}} = \lambda * x _ {\mathrm {t}} ^ {i} + (1 - \lambda) * x _ {\mathrm {t}} ^ {j}, \qquad y _ {\mathrm {m i x}} = \lambda * \hat {y} _ {\mathrm {t}} ^ {i} + (1 - \lambda) * \hat {y} _ {\mathrm {t}} ^ {j}.
$$
where  $\lambda$  is a scalar used for interpolation, and  $\hat{y}_{\mathrm{t}}^{i}$  and  $\hat{y}_{\mathrm{t}}^{j}$  denote the one-hot pseudo label encodings. By performing mixup on random target samples for a single epoch in the inference stage, we generate a set of mixed samples  $\{(x_{\mathrm{mix}}^i,y_{\mathrm{mix}}^i)\}_{i = 1}^{n_{\mathrm{t}}}$ , where  $n_t$  represents the total number.
Interpolation consistency evaluation. Next, we leverage each candidate UDA model to infer labels for all mixed samples, resulting in predicted labels  $\{\hat{y}_{\mathrm{mix}}^i\}_{i = 1}^{n_t}$ . Using both the mixed labels  $\{y_{\mathrm{mix}}^i\}_{i = 1}^{n_t}$  and predicted labels, we assess the accuracy of mixed samples for each candidate model. This assessment yields the Interpolation Consistency Evaluation (ICE) score, defined as follows:
$$
\operatorname {I C E} = \operatorname {A c c u r a c y} \left(\left\{y _ {\text {m i x}} ^ {i} \right\} _ {i = 1} ^ {n _ {t}}, \left\{\hat {y} _ {\text {m i x}} ^ {i} \right\} _ {i = 1} ^ {n _ {t}}\right).
$$
For all UDA models  $\{M_i\}_{i=1}^m$ , we apply the same mixup operation to generate identical mixed samples, but the mixed labels differ because model predictions for target samples vary among different models. We subsequently calculate the ICE score for each candidate model  $M_i$ , yielding a set of ICE scores  $\{\mathrm{ICE}_i\}_{i=1}^m$ . The candidate model with a higher ICE score is deemed superior.
MixVal via two types of probes. Through a meticulous analysis of the mixup operation applied to unlabeled target samples and their pseudo labels, we have identified two distinct categories of mixed samples, as illustrated in Figure 1(b). The first type involves mixing target samples with the same pseudo label, referred to as intra-cluster mixup. When we evaluate the ICE score after performing intra-cluster mixup, we effectively measure the neighborhood density within each cluster. A higher ICE score indicates a higher level of neighborhood consistency [29]. Conversely, the second type involves mixing target samples with different pseudo labels, termed inter-cluster mixup. By evaluating the ICE score in this context, we can assess the classification margin between clusters. A higher ICE score in inter-cluster mixup signifies a larger margin of the classification boundary, indicating low-density separation [42]. For comprehensive probing, MixVal uses both types of mixed samples as probes. It separately ranks all candidate models in ascending order based on each type of ICE score. Then, MixVal takes the average rank from both rankings to select the candidate model with the highest average rank. Kindly refer to Appendix A for the pseudocode of MixVal.
Connection to interpolation consistency training. We provide a comprehensive comparison between our MixVal approach and interpolation consistency training (ICT), a commonly used technique in semi-supervised learning [48]. While our ICE score draws inspiration from ICT, there are three significant distinctions between MixVal and ICT. ICT primarily employs inter-cluster mixup as a minor training perturbation to regularize model learning with unlabeled data. In contrast, MixVal explicitly applies mixup to create in-between target samples, encompassing both inter-cluster and intra-cluster scenarios, during the inference stage without requiring model re-training.