SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. For image classification tasks, we consider 4 popular UDA benchmarks of varied scales. Office-31 [55] is a classic domain adaptation benchmark consisting of 31 object categories across 3 domains: Amazon (A), DSLR (D), and Webcam (W). Office-Home [56] is a challenging benchmark with 65 different object categories in 4 domains: Art (Ar), Clipart (Cl), Product (Pr), and Real-World (Re). VisDA [57] is a large-scale benchmark for the synthetic-to-real object recognition task, featuring 12 categories. It consists of a training (T) domain with 152k synthetic images and a validation (V) domain with 55k realistic images. DomainNet [58] is a recent large-scale benchmark comprising approximately 600k images across 345 categories in 6 distinct domains. For evaluation, we focus on a subset of 126 classes with 7 tasks [59] from 4 domains: Real (R), Clipart (C), Painting (P), and Sketch (S). For semantic segmentation tasks, we use the synthetic GTAV [60] dataset as the source domain and the real-world Cityscapes [61] dataset as the target domain.
UDA methods. We use the validation baselines discussed in Section 2.2 to conduct model selection for various UDA methods, with a specific emphasis on the practical target-only baselines. For closed-set UDA, we consider ATDOC [27], BNM [24], CDAN [9], MCC [25], MDD [22], SAFN [23], DMRL [50], AdaptSeg [11], and AdvEnt [12]. For partial-set UDA, we consider PADA [15] and SAFN [23]. For source-free UDA, we consider SHOT [20]. For open-partial-set UDA, we consider DANCE [41]. For ATDOC, BNM, CDAN, PADA, SAFN, SHOT, DMRL, and DANCE, we select the loss coefficient among 7 varied candidate values. For MDD, we validate the margin factor, while for MCC, we validate the temperature. For AdaptSeg and AdvEnt, we validate both the loss coefficient and training iteration. For MCC and MDD, we also include a two-hyperparameter validation task, where the bottleneck dimension is considered as an additional hyperparameter, with 4 possible values. Kindly refer to Appendix B for details of hyperparameter settings.
Implementation details. We utilize the Transfer Learning Library $^{\dagger}$  to train UDA models on a single RTX TITAN 16GB GPU. The batch size is set to 32, and the total number of iterations is set to 5,000. We save the final model as a checkpoint. For VisDA and GTAV2Cityscapes, we use ResNet-101 [3], for DomainNet, we use ResNet-34 [3], and for the other benchmarks, we use ResNet-50 [3]. Regarding SND [29], we employ the official implementation. For source-based methods, we split  $80\%$  of the source data as the training set and the remaining  $20\%$  as the validation set. In our MixVal approach, we use a fixed value of  $\lambda$  to ensure a fair comparison among candidate models. Specifically, we set  $\lambda = 0.55$  for all experiments, which allows us to probe with in-between samples.
$\dagger$ https://github.com/thuml/Transfer-Learning-Library
Table 2: Validation accuracy (%) of closed-set UDA on Office-Home (Home). bold: Best value.
Method	ATDOC [27]	BNM [24]	CDAN [9]
→Ar	→Cl	→Pr	→Re	avg	→Ar	→Cl	→Pr	→Re	avg	→Ar	→Cl	→Pr	→Re	avg
SourceVal	66.63	52.54	78.57	76.61	68.59	62.44	50.74	77.53	74.76	66.37	55.00	42.65	69.50	68.81	58.99
IWCV [30]	67.97	54.03	78.31	79.26	69.89	66.56	48.16	74.09	73.28	65.52	61.31	41.24	67.17	71.93	60.41
DEV [18]	67.39	54.23	77.78	79.39	69.70	65.76	56.39	73.92	77.59	68.41	67.23	57.04	68.76	76.91	67.49
RV [31]	68.68	56.13	78.93	79.64	70.85	68.25	56.75	78.08	78.67	70.44	67.66	56.74	76.01	77.68	69.52
Entropy [42]	63.67	55.83	76.54	78.36	68.60	66.28	54.49	74.15	77.64	68.14	67.66	57.56	76.37	77.45	69.76
InfoMax [28]	63.67	55.63	77.61	78.36	68.82	66.28	54.49	74.15	77.64	68.14	67.66	57.56	76.37	77.45	69.76
SND [29]	63.67	55.63	76.54	77.54	68.34	66.28	54.49	74.15	77.64	68.14	67.94	57.56	76.96	77.68	70.04
Corr-C [43]	63.51	50.39	73.89	73.88	65.42	58.10	45.37	68.97	70.59	60.76	53.84	41.21	64.96	67.65	56.91
MixVal	66.47	56.87	78.14	79.20	70.17	67.36	56.18	76.10	78.12	69.44	67.71	57.78	76.89	77.76	70.03
Worst	62.89	50.39	73.89	73.88	65.26	58.10	45.37	68.96	70.59	60.75	53.80	41.21	64.78	67.65	56.86
Best	68.97	58.35	80.27	80.58	72.04	68.93	57.51	78.43	79.57	71.11	68.19	57.90	77.44	78.19	70.43
Method	MCC [25]	MDD [22]	SAFN [23]	Home
→Ar	→Cl	→Pr	→Re	avg	→Ar	→Cl	→Pr	→Re	avg	→Ar	→Cl	→Pr	→Re	avg
SourceVal	66.57	56.53	79.55	80.90	70.89	62.53	54.43	75.27	75.55	66.94	63.54	51.34	73.66	74.54	65.77
IWCV [30]	68.69	58.93	80.37	80.08	72.02	64.20	56.50	73.78	74.28	67.19	64.31	52.36	72.31	74.29	65.82
DEV [18]	68.81	58.07	78.54	80.10	71.38	64.42	56.94	76.85	75.94	68.54	63.15	50.47	71.20	74.54	64.84
RV [31]	70.40	58.80	80.63	80.39	72.56	66.57	55.75	76.60	76.90	68.96	64.31	50.13	73.77	74.93	65.78
Entropy [42]	69.29	59.33	80.63	80.96	72.55	66.54	57.63	77.27	77.45	69.72	59.85	46.41	72.51	73.18	62.99
InfoMax [28]	66.58	58.48	79.12	80.81	71.25	66.54	57.74	77.27	77.45	69.75	64.56	49.71	73.77	73.18	65.31
SND [29]	69.05	55.61	79.72	79.10	70.87	51.34	38.01	77.61	68.46	58.86	57.90	46.41	67.04	68.18	59.88
Corr-C [43]	69.05	55.61	79.72	79.10	70.87	47.79	31.69	63.40	60.63	50.88	62.66	46.41	68.83	68.18	61.52
MixVal	69.79	59.24	80.47	80.74	72.56	65.73	58.01	77.36	76.91	69.50	65.98	53.14	74.76	75.40	67.32
Worst	62.72	54.63	76.19	78.19	67.93	47.79	31.69	63.40	60.63	50.88	57.90	46.41	67.04	68.18	59.88
Best	70.68	59.95	80.93	81.02	73.14	66.75	58.36	77.61	77.45	70.04	66.59	53.14	74.90	75.57	67.55
Table 3: Validation accuracy (%) of closed-set UDA on Office-31 (Office) and VisDA.
Method	ATDOC [27]	BNM [24]	CDAN [9]
→A	→D	→W	avg	T→V	→A	→D	→W	avg	T→V	→A	→D	→W	avg	T→V
SourceVal	72.56	88.96	87.80	83.11	67.79	72.92	90.36	89.43	84.24	70.51	63.90	91.16	89.06	81.37	64.50
IWCV [30]	72.56	86.14	86.54	81.75	67.79	72.92	85.54	89.43	82.63	76.94	63.90	69.08	58.74	63.91	64.50
DEV [18]	72.56	86.14	86.54	81.75	70.34	72.92	85.54	89.43	82.63	76.94	63.90	91.16	88.30	81.12	64.50
RV [31]	74.93	89.96	87.23	84.04	77.37	70.71	88.55	89.43	82.90	74.58	73.27	91.16	88.30	84.24	76.02
Entropy [42]	73.29	86.14	87.80	82.41	62.85	72.67	85.54	83.14	80.45	58.36	71.62	91.16	89.06	83.95	80.46
InfoMax [28]	73.29	86.14	87.80	82.41	76.49	70.52	85.54	83.14	79.73	58.36	71.62	91.16	88.30	83.69	80.46
SND [29]	73.29	92.37	87.80	84.49	77.37	74.44	85.54	83.14	81.04	69.65	71.55	92.37	88.55	84.16	80.46
Corr-C [43]	71.05	90.96	84.40	82.14	67.79	67.16	84.34	78.99	76.83	70.51	58.29	67.67	59.62	61.86	64.50
MixVal	73.61	90.96	86.54	83.70	77.37	74.97	86.48	87.00	82.81	74.51	72.73	92.64	89.06	84.81	80.46
Worst	71.05	86.14	84.40	80.53	62.85	67.16	84.34	78.99	76.83	23.08	58.29	67.67	57.11	61.02	64.50
Best	75.31	92.37	87.80	85.16	77.37	75.52	90.36	89.43	85.10	76.94	73.38	92.77	89.06	85.07	80.46
Method	MCC [25]	MDD [22]	SAFN [23]	Office
→A	→D	→W	avg	T→V	→A	→D	→W	avg	T→V	→A	→D	→W	avg	T→V	AVG
SourceVal	73.11	90.96	91.07	85.05	80.46	75.72	91.06	86.23	84.34	72.25	69.20	83.73	87.17	80.03	70.71	83.02
IWCV [30]	73.11	91.16	88.55	84.27	81.48	75.49	91.16	89.18	85.28	72.25	69.32	86.55	80.38	78.75	66.33	79.43
DEV [18]	72.70	89.16	93.08	84.98	81.48	75.65	91.16	89.18	85.33	72.25	68.21	86.55	80.38	78.38	66.33	82.36
RV [31]	73.97	89.06	93.08	85.37	82.22	74.46	92.57	86.79	84.61	77.23	68.69	90.83	87.17	82.23	66.33	83.90
Entropy [42]	73.93	90.56	93.46	85.98	82.22	76.31	92.57	90.82	86.57	78.95	68.23	91.57	85.66	81.82	70.20	83.53
InfoMax [28]	73.93	89.16	88.55	83.88	81.48	76.50	92.57	90.82	86.63	78.95	68.23	91.57	87.42	82.41	70.20	83.13
SND [29]	73.93	91.97	93.46	86.45	69.35	76.50	92.17	90.82	86.50	78.95	68.23	89.96	85.66	81.28	58.15	83.99
Corr-C [43]	73.93	91.37	93.46	86.25	69.35	74.25	91.57	85.66	83.83	72.25	68.39	86.75	80.38	78.51	62.52	78.24
MixVal	74.09	91.77	93.21	86.36	81.48	75.97	91.77	91.74	86.49	78.95	69.61	89.96	86.83	82.13	74.41	84.39
Worst	70.56	86.75	87.17	81.49	69.35	73.06	87.35	85.66	82.02	72.25	67.27	83.73	80.38	77.13	58.15	76.50
Best	74.42	91.97	93.46	86.62	82.23	76.52	92.57	92.20	87.10	78.95	70.06	91.57	87.42	83.02	75.30	85.34
Table 4: Validation accuracy (%) of closed-set UDA on DomainNet.
Method	CDAN [9]	avg	BNM [24]	avg	ATDOC [27]	avg
→C	→P	→R	→S	→C	→P	→R	→S	→C	→P	→R	→S
Entropy [42]	67.09	65.80	74.42	59.34	66.66	63.36	64.28	74.31	48.69	62.66	63.75	61.85	79.60	52.17	64.34
InfoMax [28]	67.09	65.80	74.42	59.34	66.66	67.05	64.28	74.31	55.67	65.33	63.75	61.85	79.60	52.17	64.34
SND [29]	67.09	64.68	74.42	59.34	66.38	56.56	54.50	74.31	42.37	56.93	63.75	61.85	79.60	47.00	63.05
Corr-C [43]	57.35	62.88	74.42	54.63	62.32	59.75	63.41	77.62	42.37	60.79	59.98	62.27	74.42	53.69	62.59
MixVal	67.09	65.80	74.42	59.34	66.66	67.84	66.40	78.68	58.49	67.85	68.94	68.44	79.60	61.73	69.68
Worst	57.35	60.76	73.44	51.41	60.74	55.79	54.50	74.31	42.37	56.74	59.98	61.85	74.42	47.00	60.81
Best	67.09	65.80	74.44	59.34	66.66	67.86	66.50	78.68	58.49	67.88	70.30	68.44	80.38	62.23	70.34
We report MixVal's validation performance as averages from three random runs. In the tables, 'Best' signifies the optimal selection, and 'Worst' represents the worst one. For brevity, we report average results for tasks with the same target domain. Kindly refer to Appendix D for detailed results.
Closed-set UDA. We begin by comparing the performance of validation baselines within the standard closed-set UDA scenario. Specifically, we report the results for both source-based and target-only validation methods across several well-established UDA benchmarks. Table 2 provides the results on the medium-scale benchmark Office-Home, while Table 3 presents the results on the small-scale benchmark Office-31 and the large-scale benchmark VisDA. Furthermore, Table 4 presents the results
Table 5: Validation accuracy (%) of partial-set UDA on Office-Home.
Method	PADA [15]	avg	SAFN [23]	avg
→ Ar	→ Cl	→ Pr	→ Re	→ Ar	→ Cl	→ Pr	→ Re
SourceVal	57.21	41.90	64.48	71.89	58.87	66.82	54.71	74.41	76.48	68.11
IWCV [30]	59.65	50.51	66.84	72.96	62.49	69.36	53.91	71.78	76.38	67.86
DEV [18]	66.88	49.29	72.40	70.46	64.76	69.36	54.94	73.95	76.06	68.58
RV [31]	57.79	40.87	63.87	70.83	58.34	68.98	52.74	72.83	77.14	67.92
Entropy [42]	60.08	46.51	53.16	62.47	55.56	71.75	55.62	76.36	76.59	70.08
InfoMax [28]	60.08	51.40	60.20	66.67	59.59	63.67	51.74	69.64	73.62	64.67
SND [29]	67.80	50.71	59.46	67.13	61.27	71.75	51.74	76.36	78.36	69.55
Corr-C [43]	61.34	45.65	54.90	62.25	56.04	71.23	55.70	76.94	79.13	70.75
MixVal	67.68	51.01	72.94	78.64	67.57	71.70	57.91	77.08	78.94	71.41
Worst	56.29	39.76	50.49	59.31	51.46	62.48	49.91	68.50	73.62	63.63
Best	69.33	55.86	74.55	79.59	69.83	73.37	58.09	77.35	79.33	72.03
Table 6: HOS (%) of open-partial-set UDA and accuracy (%) of source-free UDA.
Method	DANCE [41]	Home avg	SHOT [20]	Office avg	VisDA T→V
→Ar	→Cl	→Pr	→Re	→A	→D	→W
Entropy [42]	32.00	39.48	27.52	38.08	34.27	71.67	90.76	88.68	83.70	82.65
InfoMax [28]	32.00	39.48	27.52	38.01	34.25	71.67	90.76	88.68	83.70	82.65
SND [29]	15.05	4.33	23.75	16.79	14.98	71.67	90.76	88.68	83.70	82.65
Corr-C [43]	29.60	4.33	23.75	16.79	18.62	71.58	90.76	90.19	84.18	82.65
MixVal	71.54	52.90	78.61	65.01	67.01	72.04	92.37	92.32	85.58	83.12
Worst	15.05	4.33	15.17	16.79	12.84	71.56	90.76	88.68	83.67	80.57
Best	77.01	66.29	78.81	69.81	72.98	75.06	94.78	93.33	87.72	83.12
of practical target-only validation baselines on the large-scale benchmark DomainNet. The current state-of-the-art source-based baseline is DEV [18], and for target-only methods, SND [29] holds the state-of-the-art position. Nonetheless, our observations differ from previous findings. When considering the averaged validation accuracy across 6 popular UDA methods, we note that RV consistently outperforms other source-based validation methods across three benchmarks. Regarding target-only validation methods, entropy-based approaches like Entropy and InfoMax prove to be competitive, particularly demonstrating a significant advantage over SND on the Office-Home benchmark. As expected, our MixVal consistently attains the highest average accuracy across all four benchmarks, surpassing the vanilla SourceVal baseline, while most existing validation methods may noticeably underperform in comparison to this baseline. Notably, MixVal demonstrates exceptional performance on large-scale benchmarks such as VisDA and DomainNet, approaching the upper bound of 'Best' and outperforming the second-best method by substantial margins. MixVal's consistent advantages in closed-set UDA highlight its effectiveness in this widely studied setting.
Partial-set UDA. Following [29], we assess validation performance in partial-set UDA, a scenario with label shifts between domains. We validate two representative partial-set UDA methods, PADA and SAFN, and present the validation accuracy of Office-Home in Table 5. We observe that MixVal consistently outperforms all other validation baselines, maintaining performance close to the 'Best'. However, we notice some differences in comparison to closed-set UDA. Specifically, SND achieves more stable validation performance than Entropy, because Entropy can be susceptible to structure collapse [29]. Second, methods emphasizing 'class diversity', such as InfoMax and Corr-C, often yield lower results than SourceVal. Additionally, RV consistently underperforms SourceVal in scenarios without symmetric label distribution, making it unsuitable for UDA with label shifts.
Open-partial-set UDA. We extend our evaluation of UDA scenarios with label shifts, conducting validation for a popular open-partial-set UDA method, DANCE. Table 6 presents the HOS [62, 63] results on Office-Home. MixVal consistently provides strong validation performance, nearing the 'Best', while other validation baselines tend to align with the 'Worst'.
Source-free UDA. Finally, we compare the target-only validation baselines in source-free UDA. We conduct validation for a popular method, SHOT, and report the accuracy of Office-31 and VisDA in Table 6. We find that MixVal consistently outperforms other baselines in this practical setting.
Influence of pseudo label quality. Because MixVal utilizes pseudo labels of target data for the mixup operation, it is crucial to assess the impact of pseudo label quality. When examining UDA tasks with
Figure 2: (a) presents the effect of different consistency evaluation techniques. (b) provides comparisons between SND and our MixVal on a validation task involving two hyperparameters: MCC on the  $\mathrm{Ar}\rightarrow \mathrm{Cl}$ , where  $\gamma$  represents the bottleneck dimension and  $\eta$  represents the temperature. The SND and MixVal scores of the 28 candidate models are ranked in ascending order. To enhance clarity, we include the real target accuracy for each model in the left figure labeled as 'Accuracy'.
low-quality pseudo labels (around  $50\%$  accuracy), such as the results of  $\rightarrow \mathrm{Cl}^{\prime}$ , it becomes evident that MixVal consistently maintains stable validation performance.
Influence of consistency evaluation. MixVal utilizes image-level mixup-based consistency evaluation for the target probing. To gain a deeper understanding of its role in MixVal, we substitute it with other consistency evaluation techniques, including instance-based augmentations such as RandAug [64] and SSLAug [65], as well as mixup at other levels, such as CutMix [46] and FeatMix [45]. The results in Figure 2(a) consistently demonstrate that image-level mixup outperforms other strategies.
Table 7: Two-hyperparameter validation accuracy (%) of closed-set UDA on Office-Home.
Method	MDD [22]	avg	MCC [25]
Ar → Cl	Cl → Pr	Pr → Re	Re → Ar	Ar → Cl	Cl → Pr	Pr → Re	Re → Ar	avg
SourceVal	55.99	73.15	78.77	69.39	69.33	57.91	76.84	81.13	72.89	72.19
IWCV [30]	37.89	72.92	80.42	58.43	62.42	46.09	77.74	80.68	74.45	69.74
DEV [18]	52.60	72.11	53.36	67.70	61.44	59.47	76.84	81.94	74.08	73.08
RV [31]	57.59	72.25	80.83	70.79	70.37	59.13	76.84	82.03	71.98	72.50
Entropy [42]	57.21	73.19	80.06	72.31	70.69	59.75	77.77	82.37	74.33	73.56
InfoMax [28]	57.59	72.92	80.06	72.31	70.72	59.70	78.73	82.58	70.33	72.84
SND [29]	38.10	56.45	70.03	65.10	57.42	53.49	74.97	77.25	74.12	69.96
Corr-C [43]	30.17	44.74	57.15	50.76	45.71	44.90	56.75	74.32	67.61	60.90
MixVal	55.99	72.63	80.27	72.12	70.25	60.08	78.52	81.95	74.43	73.75
Worst	30.17	39.81	53.36	50.76	43.53	43.02	56.75	73.47	67.24	60.12
Best	57.59	73.35	80.93	72.52	71.10	61.10	78.94	83.04	75.36	74.61
Two-hyperparameter validation. We further evaluate all validation baselines in challenging tasks that involve two hyperparameters. Quantitative comparisons in Table 7 consistently position MixVal as one of the top-performance validateators. Qualitative comparisons in Figure 2(b) clearly illustrate that MixVal scores demonstrate a strong correlation with actual accuracy, whereas SND [29] scores exhibit noticeable deviation, resulting in selections far from optimal.
Table 8: Validation accuracy (%) of a mixup-based closed-set UDA method DMRL [50].
Method	DMRL [50]	Home avg	DMRL [50]	Office avg
→Ar	→Cl	→Pr	→Re	→A	→D	→W
Entropy [42]	58.14	50.25	69.06	71.37	62.20	63.67	80.52	86.67	76.95
InfoMax [28]	58.14	50.75	69.06	71.37	62.33	63.67	80.52	86.67	76.95
SND [29]	57.74	49.96	69.28	71.42	62.10	61.84	84.14	86.67	77.55
Corr-C [43]	58.29	49.46	68.67	71.73	62.04	60.23	77.51	81.13	72.95
MixVal	59.13	50.41	69.28	72.07	62.72	64.89	82.93	86.67	78.16
Worst	57.71	48.99	67.78	70.72	61.30	60.23	76.31	81.13	72.55
Best	59.20	50.75	69.28	72.24	62.87	65.30	84.14	86.67	78.70
Validation of mixup-based UDA method. The ICE score of MixVal is based on the evaluation of mixup-based interpolation consistency during the inference stage. To assess MixVal's robustness, we apply it in the validation of the UDA method DMRL [50], which incorporates mixup-based consistency training within the target domain and across two domains at the same time. MixVal is employed to determine the optimal consistency loss coefficient for DMRL, and the results are reported in Table 8. Notably, MixVal consistently outperforms most of the existing target-only validation baselines, even when subjected to the 'attack' posed by specialized UDA model training.
Figure 3: Analysis of probing.
Method	AdaptSegt [11]	AdvEnt [12]
SourceVal	39.52	39.08
Entropy [42]	39.47	38.41
SND [29]	40.69	40.02
MixVal	42.20	40.02
Worst	33.84	33.06
Best	42.20	41.78
Table 9: Segmentation mIoU (%)
Method	BNM [24]
Entropy [42]	28.21
InfoMax [28]	28.21
SND [29]	52.42
Corr-C [43]	28.21
MixVal	54.78
Worst	28.21
Best	55.16
Table 10: ViT results.
Analysis of probing. In MixVal, we generate mixed samples by applying mixup with a mix ratio  $\lambda$  of 0.55 to both target samples and their hard pseudo labels. We then calculate ICE scores for both intra-cluster and inter-cluster mixed samples. The final score for model selection is determined by averaging ascending rankings of both types of ICE scores. Here, we present a comprehensive ablation study to delve into these design choices within MixVal. To do this, we conduct validation experiments for PADA on four partial-set UDA tasks (Ar→Cl, Cl→Pr, Pr→Re, Re→Ar) and report the average accuracy results in Figure 3. 'AccAvg' represents the direct average of ICE scores (i.e., accuracy values) as the final score for model selection. 'RankAvg' denotes the use of the average ranking of ICE scores for selection. 'RankInter' relies solely on inter-cluster mixed samples for probing. 'RankIntra' employs only intra-cluster mixed samples for probing. 'Hard' and 'Soft' indicate the utilization of hard and soft pseudo labels, respectively. It's worth noting that for intra-cluster mixed samples, there is no distinction between 'Hard' and 'Soft' labels, as all pseudo labels involved are the same. The observations drawn from Figure 3 are as follows: (i) The mix ratio  $(\lambda)$  influences the level of ambiguity in mixed samples. In MixVal,  $\lambda = 0.55$  generates more ambiguous in-between samples with greater probing capabilities than the easier samples created by  $\lambda = 0.9$ . Consequently, validation performance at  $\lambda = 0.55$  surpasses that at  $\lambda = 0.9$ . (ii) In MixVal, while soft pseudo labels may provide additional performance improvements for inter-cluster probing, the use of hard pseudo labels offers simplicity and stability advantages. (iii) The separate probing of each type of mixed samples demonstrates effectiveness, and the combination of intra-cluster and inter-cluster probing enhances MixVal's stability. Kindly refer to Appendix C for more discussions on probing.
Extension to segmentation. We extend the use of MixVal to the domain adaptive segmentation task on GTAV2Cityscapes, following the same configuration as SND [29]. Our model selection process involves 66 candidate models generated by varying two hyperparameters: the loss coefficient and training iteration. We save checkpoints every 2,000 iterations, starting from the 10,000th iteration to the 30,000th iteration. The results in Table 9 demonstrate that MixVal consistently delivers the top validation performance for both UDA methods, while Entropy consistently underperforms SourceVal.
Influence of backbone. In addition to ResNet backbones, we also utilize ViT-B [4] as the backbone for the validation of BNM on the task  $\mathbb{R}\rightarrow \mathbb{S}$ . Accuracies reported in Table 10 highlight MixVal's superior stability with ViT, especially in comparison to Entropy, InfoMax, and Corr-C.
Limitations and impacts. While MixVal excels in classification tasks, further research is needed to address its limitations. This includes establishing the theoretical foundations of probing data structure using pseudo-labeled mixed samples. Additionally, extending MixVal to other machine learning tasks like object detection and regression presents challenges. Poor hyperparameter selections could potentially have negative societal impacts on the real-world deployment of UDA models. Nevertheless, we have not observed such situations during our extensive evaluation of MixVal.
In summary, we introduce MixVal, a novel and straightforward target-only method for model selection in unsupervised domain adaptation (UDA). MixVal leverages inference-stage mixup to generate two distinct types of mixed samples, facilitating effective probing of the learned target structure while elegantly considering key assumptions used in previous approaches. Our extensive evaluations encompass diverse UDA methods and adaptation scenarios, consistently affirming the superior and reliable performance of MixVal when compared to existing model selection methods.
This project is supported by the Singapore Ministry of Education Academic Research Fund Tier 1 (WBS: A-8001229-00-00), a project titled "Towards Robust Single Domain Generalization in Deep Learning". Jian Liang is supported by the National Natural Science Foundation of China (Grant No. 62276256) and the Beijing Nova Program under Grant Z211100002121108.
The PyTorch-style pseudocode for our validation approach MixVal is presented in Algorithm 1.