SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison Methods. We compare our method with three groups of methods: (1) Supervised method as a baseline to show the effectiveness of training with unlabeled data.(2) Classical UDA methods including DANN [1] and CDAN [24]; and (3) Popular SSL methods including MixMatch [6], ReMixMatch[5], FixMatch [30], FM-Rot (joint training with rotation prediction task), FM-Pre (pre-trained by rotation prediction tasks), FreeMatch [41], SoftMatch [9], and Adamatch [7].
Evaluation Protocols. We evaluate the performance across labeled, unlabeled and unseen distributions to verify the general applicability of our framework. Therefore, we consider three evaluation protocols: (1) Label-Domain Evaluation (L): test samples are drawn from labeled distribution, (2)
Table 2: Comparison of accuracy (%) for Feature Distribution Mismatch SSL on CIFAR100.
Method	400 labeled	4000 labeled	10000 labeled
ratio 0.5	ratio 1.0	ratio 1.0	ratio 1.0
L	UL	US	L	UL	US	L	UL	US	L	UL	US
Supervised	10.6	8.9	6.9	10.6	8.9	6.9	48.0	17.0	24.0	61.6	20.8	30.8
DANN [1]	11.3	9.3	7.0	11.4	9.3	7.0	46.7	30.8	27.6	61.4	42.7	37.8
CDAN [24]	11.6	9.5	7.3	11.6	10.0	7.5	47.1	31.3	28.8	62.6	41.1	39.1
MixMatch [6]	10.7	3.6	5.4	15.4	2.5	6.1	46.4	10.9	21.1	60.3	31.4	37.6
AdaMatch [7]	6.8	4.6	1.3	6.0	4.5	2.1	19.1	6.4	1.8	26.7	9.0	1.6
ReMixMatch [5]	39.6	20.6	32.7	39.2	19.1	3.1	68.4	35.8	58.1	75.5	40.2	63.2
RM-SSFA (ours)	43.1	22.9	35.1	43.0	23.1	35.1	71.0	37.9	60.8	76.4	41.6	63.5
FixMatch [30]	25.8	4.7	16.5	15.7	3.5	8.5	53.0	16.0	39.0	65.3	33.0	52.8
FM-SSFA (ours)	37.0	23.2	25.8	25.7	22.2	22.5	60.2	52.5	51.2	69.1	57.8	58.0
FreeMatch [41]	35.0	1.4	21.0	17.4	2.2	7.2	55.2	16.3	41.4	67.0	23.9	53.4
FreeM-SSFA (ours)	37.6	25.1	31.4	27.5	18.7	21.9	62.0	54.2	53.0	70.2	61.8	60.0
SoftMatch [9]	35.5	2.9	24.1	19.4	4.6	12.8	58.3	29.7	48.9	68.3	34.7	56.3
SM-SSFA (ours)	40.8	29.5	34.8	31.1	22.4	25.2	62.6	54.4	53.8	70.3	61.8	59.9
UnLabel-Domain Evaluation (UL): test samples are drawn from unlabeled distribution, and (3) UnSeen-Domain Evaluation (US): test samples are drawn from unseen distribution.
Image Corruption. Our SSFA framework is a generic framework that can be easily integrated with existing pseudo-label-based SSL methods. In this work, we combine SSFA with four SSL methods: FixMatch, ReMixMatch, FreeMatch and SoftMatch, denoted by FM-SSFA, RM-SSFA, FreeMSSFA and SM-SSFA, respectively. Table 2 shows the compared results on the image corruption experiment with different numbers of labeled samples and ratio. We can observe that (1) the UDA methods, DANN and CDAN, exhibit very poor performance on labeled-domain evaluation (L). This is because UDA methods aim to adapt the model to the unlabeled domain, which sacrifices performance on the labeled domain. And the UDA methods are primarily designed to adapt to a single unlabeled distribution, making it unsuitable for FDM-SSL scenarios. In contrast, our methods largely outperform the two UDA methods on all evaluations. (2) The traditional SSL methods suffer from significant performance degradation in the FDM-SSL setting, particularly in unlabeled-domain evaluation (UL). And this degradation becomes more severe when the number of labeled data is small and the proportion of unlabeled data with corruption is high, due to the increased feature mismatch degree. Our methods largely outperform these SSL methods, validating the effectiveness of our self-supervised feature adaptation strategy for FDM-SSL. (3) SSFA largely improves the performance of original SSL methods on all three evaluations. The gains on unlabeled-domain evaluation (UL) indicate that SSFA can generate more accurate pseudo-labels for unlabeled samples and reduce confirmation bias. Moreover, the gains on unseen-domain evaluation (US) validate that SSFA can enhance the model's robustness and generalization.
Style Change. We conduct experiments on OFFICE-31 and OFFICE-HOME benchmarks to evaluate the impact of style change on SSL methods. Specifically, we evaluate SSL methods in two scenarios where unlabeled data is sampled from a single distribution and a mixed distribution. The results are summarized in Table 3. Compared to the supervised method, most SSL methods present significant performance degradation. One possible explanation is that style change causes an even greater feature distribution mismatch compared to image corruption, resulting in more erroneous predictions for unlabeled data. As shown in Table 3(a), our method shows consistent performance improvement over existing UDA and SSL methods on OFFICE-31. In some cases, such as "W/A", the task itself may be relatively simple, so using only label data can already achieve high accuracy. The results on OFFICE-HOME are summarized in Table 3(b). To demonstrate that our method can indeed improve the accuracy of pseudo-labels, we add an evaluation protocol UU, which indicates the pseudo-labels accuracy of selected unlabeled samples. As shown in Table 3(b), compared to existing UDA and SSL methods, our approach achieves the best performance on all setups and significantly improves the accuracy of pseudo-labels on unlabeled data. The results provide experimental evidence that supports the theoretical analysis in Section 4.4.
Table 3: Comparison of accuracy (%) for Feature Distribution Mismatch SSL on OFFICE-31 and OFFICE-HOME.
(a) Results on OFFICE-31
Method	Single Domain	Multiple Domains
A/D	A/W	D/A	W/A	A/DW	D/AW
L	UL	US	L	UL	US	L	UL	US	L	UL	US	L	UL	L	UL
supervised	65.9	57.1	52.0	65.9	47.8	61.0	93.0	56.8	84.9	93.6	56.2	92.4	68.0	51.9	93.3	60.7
DANN [1]	53.9	17.9	20.1	15.3	7.2	4.6	74.7	4.8	16.7	2.5	2.9	4.2	28.1	7.6	4.7	2.8
CDAN [24]	2.9	5.6	2.6	2.9	2.3	4.2	4.7	2.9	2.6	41.3	7.7	18.5	2.9	3.2	45.5	10.7
FixMatch [30]	58.4	46.9	53.3	56.3	9.5	19.3	82.8	4.7	50.6	78.3	9.4	62.6	55.0	48.8	80.2	40.3
FM-Rot	60.0	53.1	57.1	54.6	27.4	19.3	70.8	29.5	50.6	75.9	17.9	66.9	54.0	40.1	63.0	26.9
FM-SSFA	67.5	61.7	63.0	66.4	56.5	62.5	93.0	59.3	85.9	93.0	56.4	92.8	69.1	60.8	92.4	61.8
(b) Results on OFFICE-HOME
Method	A/CPR	C/APR	P/ACR	A/ACPR	C/ACPR	R/ACPR
L	UL	UU	L	UL	UU	L	UL	UU	L	UL	UU	L	UL	UU	L	UL	UU
supervised	48.4	39.9	-	40.9	31.2	-	68.4	36.5	-	49.3	43.0	-	41.0	36.3	-	61.6	46.8	-
DANN [1]	45.0	37.8	-	31.8	7.7	-	53.9	25.6	-	49.3	42.0	-	30.5	20.2	-	44.5	32.1	-
CDAN [24]	30.4	22.8	-	1.2	1.8	-	63.0	34.3	-	4.4	4.2	-	10.5	6.9	-	13.5	10.2	-
FixMatch [30]	27.2	17.7	19.3	28.1	19.6	19.4	55.6	24.9	30.6	32.4	23.0	31.0	36.9	30.6	36.0	42.2	31.5	35.4
FM-Pre	21.2	15.0	15.4	28.6	22.9	25.4	51.9	22.0	26.3	30.6	20.1	24.8	37.9	33.8	41.7	33.4	23.8	24.9
FM-Rot	1.7	2.1	0.0	33.4	20.1	0.7	43.4	14.6	17.2	25.6	17.7	21.3	37.3	31.0	33.2	45.6	34.7	42.3
FM-SSFA	50.7	44.0	69.0	45.1	37.6	66.2	70.6	41.9	70.5	55.0	45.5	73.2	44.7	41.7	66.0	64.8	52.7	80.3
5.3 Feature Visualization
Figure 3 visualizes the domain-level features generated by SSL models with/without SSFA respectively. In Figure 3 (a), the vanilla FixMatch model maps labeled and unlabeled samples to different clusters in the feature spaces without fusing samples from different domains. Conversely, Figure 3 (b) shows that our FM-SSFA model can effectively fuse these samples. We further compute the  $A$ -distance metric [4] to measure the distributional divergence between the labeled and unlabeled distributions. The  $A$ -distance of FM-SSFA (0.91) is lower than that of FixMatch (1.10), verifying the implicit distribution alignment of SSFA.
Additionally, Figure 4 visualizes the class-level features generated by SSL models with/without SSFA respectively. In Figure 4 (a), FixMatch fails to distinguish samples from different classes. In contrast, FM-SSFA can separate samples from different categories well, as shown in Figure 4 (b). This result indicates that SSFA can help to extract more discriminative features.
Figure 3: Visualization of domain-level features using different methods, where "label" represents the labeled data drawn from the labeled domain, and "unlabel0" to "unlabel9" represent the unlabeled data drawn from ten unlabeled domains respectively.
Figure 4: Visualization of class-level features using different methods.
Combined with different self-supervised tasks. Table 4 compares different self-supervised tasks combined with our SSFA framework on the image corruption experiment. We employ different self-supervised losses corresponding to the rotation prediction task, contrastive learning task, and
Table 4: Combined with different self-supervised tasks. "Rot", "SimClr", and "EM" represent the rotation prediction task, contrastive learning task [11] and entropy minimization task [38] respectively.
Method	400 labeled	4000 labeled	10000 labeled
ratio 0.5	ratio 1.0	ratio 1.0	ratio 1.0
L	UL	US	L	UL	US	L	UL	US	L	UL
FixMatch[30]	25.8	4.7	16.5	15.7	3.5	8.5	53.0	16.0	39.0	65.3	33.0
FM-SSFA (Rot)	37.0	23.2	25.8	25.7	22.2	22.5	60.2	52.5	51.2	69.1	57.8
FM-SSFA (SimClr)	22.8	13.7	17.4	19.2	13.6	14.1	55.3	43.8	45.1	66.2	53.6
FM-SSFA (EM)	22.9	14.5	17.2	18.0	15.9	14.6	55.2	44.2	44.7	66.7	52.5
entropy minimization task, which are the cross entropy loss, the contrastive loss from SimCLR and the entropy loss, respectively. As shown, different self-supervised tasks can bring significant performance gains. And the rotation prediction task leads to the largest improvements compared to the other two self-supervised tasks. This can be attributed to the more optimal parameters and the direct supervision signals in rotation prediction. In the experiments, we employ the rotation prediction task as the default self-supervised task.
The effectiveness of Feature Adaptation Module. To evaluate the effectiveness of the Feature Adaptation Module in the SSFA framework, we compare FM-SSFA with a baseline method FM-Rot, where we remove the Feature Adaptation Module and only add the self-supervised rotation prediction task. As shown in Table 3(a) and 3(b), FM-SSFA largely outperforms FM-Rot on OFFICE-31 and OFFICE-HOME, especially in the UL evaluation metric. The superiority of FM-SSFA highlights that the Feature Adaptation Module helps the model to adapt to unlabeled samples from different distributions, resulting in better generalization on unlabeled domain. Moreover, we can observe the FM-Rot is only marginally better than FixMatch and, in some cases, may even perform worse. These results suggest that simply integrating the self-supervised task into SSL methods only brings limited performance gains and may even be detrimental in some scenarios.
Small distribution shift between labeled and un
labeled data. To demonstrate the robustness of our proposed method, we evaluate SSFA on scenarios where there is a small distribution shift between labeled and unlabeled data. Specially, we conduct experiments on two setups: ratio=0.0 (degrade into traditional SSL) and ratio=0.1. Table 5 shows our method can still bring significant improvements over the baseline. This indicates that our proposed SSFA framework is robust to various SSL scenarios even under a low noise ratio.
Robustness to confident threshold  $\tau$ . Figure 5 illustrates the impact of confident threshold  $\tau$  for different SSL models on OFFICE-31. As shown in Figure 5, the vanilla FixMatch model is highly sensitive to the values of  $\tau$ . For example, when  $\tau$  is set to 0.95, FixMatch achieves relatively high performance with confident enough pseudo-labels. When  $\tau$  is lowered to 0.85, the performance of FixMatch on UL and UU degrades significantly, indicating that the model has deteriorated by too many wrong pseudo-labels. In contrast, FM-SSFA is more robust to different values of  $\tau$ .
Table 5: The performance of SSFA when the distribution shift between labeled and unlabeled data is small on CIFAR100.
Method	ratio 0.0	ratio 0.1
L/UL	US	L	UL US
FixMatch [30]	33.3	25.8	31.7	10.5 24.8
FM-SSFA	41.3	33.0	41.2	15.8 32.9
Figure 5: The impact of  $\tau$  for different SSL models on OFFICE-31 ("A/W" task).
The number of shared layers between auxiliary and main task. Table 6 analyzes the impact of different numbers of shared layers in the shared feature extractor of the auxiliary and main task. As shown, the performance difference is negligible between sharing 2 and 3 layers in the feature extractor, while a significant performance degradation happens if we share all layers (4 layers) of the feature extractor. We argue that this is because too many shared parameters of the feature extractor may lead to over-adaptation of the feature extractor and compromise of the main task, thus the predictions of
Table 6: The impact of shared layers between auxiliary and main task on OFFICE-HOME. "X layers" denotes the number of shared layers for the feature extractor.
Method	A/CPR	C/APR	P/ACR	R/ACP
L	UL	UU	L	UL	UU	L	UL	UU	L	UL	UU
FM-SSFA (2 layers)	48.9	39.9	70.8	46.0	39.5	69.0	70.3	42.0	67.3	61.0	40.8	67.0
FM-SSFA (3 layers)	47.5	39.8	71.0	45.4	39.0	77.6	72.3	45.4	74.4	63.4	43.9	76.8
FM-SSFA (4 layers)	44.4	39.1	57.1	1.3	2.4	0.0	57.9	33.4	59.5	40.3	28.7	54.3
pseudo-labels may be more erroneous. In the experiments, we set the number of shared layers to 2 by default.
In this paper, we focus on a realistic SSL setting, FDM-SSL, involving a mismatch between the labeled and unlabeled distributions, complex mixed unlabeled distributions and widely unknown test distributions. The primary challenge lies in the scarcity of labeled data and the potential presence of mixed distributions within the unlabeled data. To address this challenge, we propose a generalized framework SSFA, which introduces a self-supervised task to adapt the feature extractor to the unlabeled distribution. By incorporating this self-supervised adaptation, the model can improve the accuracy of pseudo-labels to alleviate confirmation bias, thereby enhancing the generalization and robustness of the SSL model under distribution mismatch.
Broader Impacts and Limitations. The new problem setting takes into account the model's generalization across different distributions, which is essential for expanding the application of SSL methods in real-world scenarios. Additionally, SSFA serves as a simple yet effective framework that can be seamlessly integrated with any pseudo-label-based SSL methods, enhancing the overall performance and adaptability of these models. However, the performance of SSFA is affected by the shared parameters between the main task and the auxiliary task. We hope that SSFA can attract more future attention to explore the effectiveness of feature adaptation with the self-supervised task.
This work is partially supported by National Key R&D Program of China no. 2021ZD0111901, National Natural Science Foundation of China (NSFC): 61976203, 62276246 and 62306301, and National Postdoctoral Program for Innovative Talents under Grant BX20220310.
[1] Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain-adversarial neural networks. arXiv, 2014.
[2] Gholamali Aminian, Mahed Abroshan, Mohammad Mahdi Khalili, Laura Toni, and Miguel Rodrigues. An information-theoretical approach to semi-supervised learning under covariateshift. In AISTATS, 2022.
[3] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In IJCNN, 2020.
[4] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NeurIPS, 2006.
[5] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR, 2020.
[6] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.
[7] David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alexey Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. In ICLR, 2022.
[8] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR, 2017.
[9] Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. In ICLR, 2023.
[10] Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. Adversarial-learned loss for domain adaptation. In AAAI, 2020.
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[12] Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, and Taylan Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. In ICML, 2022.
[13] Mohammad Zalbagi Darestani, Jiayu Liu, and Reinhard Heckel. Test-time training can close the natural distribution shift performance gap in deep learning based compressed sensing. In ICML, 2022.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[15] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros. Test-time training with masked autoencoders. In NeurIPS, 2022.
[16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17:2096-2030, 2016.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[19] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurIPS, 2021.
[20] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[21] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
[22] D.-H. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. In ICML, 2013.
[23] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
[24] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NeurIPS, 2018.
[25] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. In ICCV, 2013.
[26] Prabhu Teja S and François Fleuret. Test time adaptation through perturbation robustness. In NeurIPS, 2021.
[48] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[49] Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschlager, and Susanne Saminger-Platz. Central moment discrepancy (CMD) for domain-invariant representation learning. In ICLR, 2017.
[50] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In NeurIPS, 2021.
[51] Jing Zhang, Wanqing Li, Chang Tang, Philip Ogunbona, et al. Unsupervised domain expansion from multiple sources. arXiv, 2020.
[52] Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: test time robustness via adaptation and augmentation. In NeurIPS, 2022.
[53] Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi-supervised domain generalization with stochastic stylematch. International Journal of Computer Vision, 131:2377-2387, 2023.