SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Limitations of prior work analyzing self-training Prior research views self-training as consistency regularization, ensuring pseudolabels for original samples align with their augmentations [12, 87, 79]. This approach abstracts the role played by the optimization algorithm and instead evaluates the global minimizer of a population objective promoting pseudolabel consistency. It also relies on specific assumptions about class-conditional distributions to guarantee pseudolabel accuracy across domains. However, this framework doesn't address issues in iterative label propagation. For example, when augmentation distribution has long tails, the consistency of pseudolabels depends on the sampling frequency of "favorable" augmentations (for more discussion see App. F.2). Our analysis thus follows the iterative examination of self-training [17].
In this study, we highlight the synergistic behavior of self-training and contrastive pretraining under distribution shift. Shifts in distribution are commonplace in real-world applications of machine learning, and even under natural, non-adversarial distribution shifts, the performance of machine learning models often drops. By simply combining existing techniques in self-training and constractive learning, we find that we can improve accuracy by  $3 - 8\%$  rather than using either approach independently. Despite these significant improvements, we note that one limitation of this combined approach is that performing self-training sequentially after contrastive pretraining increases the computation cost for UDA. The potential for integrating these benefits into one unified training paradigm is yet unclear, presenting an interesting direction for future exploration.
Beyond this, we note that our theoretical framework primarily confines the analysis to training the backbone and linear network independently during the pretraining and fine-tuning/self-training phases. Although our empirical observations apply to deep networks with full fine-tuning, we leave a more rigorous theoretical study of full fine-tuning for future work. Our theory also relies on a covariate shift assumption (where we assume that label distribution also doesn't shift). Investigating the complementary nature of self-training and contrastive pretraining beyond the covariate shift assumption would be another interesting direction for future work.
SG acknowledges the JP Morgan AI Ph.D. Fellowship and Bloomberg Ph.D. Fellowship for their support.
[1] Alexandari, A., Kundaje, A., and Shrikumar, A. (2021). Adapting to label shift with bias-corrected calibration. In International Conference on Machine Learning (ICML).
[2] Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. (2019). A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229.
[3] Azizzadenesheli, K., Liu, A., Yang, F., and Anandkumar, A. (2019). Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations (ICLR).
[4] Bardes, A., Ponce, J., and LeCun, Y. (2021). Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906.
[5] Baricz, Á. (2008). Mills' ratio: Monotonicity patterns and functional inequalities. Journal of Mathematical Analysis and Applications, 340(2):1362-1370.
[6] Bekker, J. and Davis, J. (2020). Learning from positive and unlabeled data: a survey. Machine Learning.
[7] Ben-David, S., Lu, T., Luu, T., and Pál, D. (2010). Impossibility Theorems for Domain Adaptation. In International Conference on Artificial Intelligence and Statistics (AISTATS).
[8] Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., and Raffel, C. (2019). Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785.
[9] Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
[10] Blanchard, G., Lee, G., and Scott, C. (2011). Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24.
[11] Cabannes, V., Kiani, B. T., Balestriero, R., LeCun, Y., and Bietti, A. (2023). The ssl interplay: Augmentations, inductive bias, and generalization. arXiv preprint arXiv:2302.02774.
[12] Cai, T., Gao, R., Lee, J., and Lei, Q. (2021). A theory of label propagation for subpopulation shift. In International Conference on Machine Learning, pages 1170-1182. PMLR.
[13] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. (2020). Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924.
[14] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660.
[15] Chapelle, O., Scholkopf, B., and Zien, A. (2006). Semi-supervised learning. 2006. Cambridge, Massachusetts: The MIT Press View Article, 2.
[16] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020a). A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR.
[17] Chen, X., Chen, W., Chen, T., Yuan, Y., Gong, C., Chen, K., and Wang, Z. (2020b). Self-pu: Self boosted and calibrated positive-unlabeled training. In International Conference on Machine Learning, pages 1510-1519. PMLR.
[94] Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR).
[95] Zhang, J., Menon, A., Veit, A., Bhojanapalli, S., Kumar, S., and Sra, S. (2021). Coping with label shift via distributionally robust optimisation. In International Conference on Learning Representations (ICLR).
[96] Zhang, K., Schölkopf, B., Muandet, K., and Wang, Z. (2013). Domain Adaptation Under Target and Conditional Shift. In International Conference on Machine Learning (ICML).
[97] Zhang, R. (2019). Making convolutional networks shift-invariant again. In ICML.
[98] Zhang, W., Ouyang, W., Li, W., and Xu, D. (2018). Collaborative and adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition.
[99] Zhang, Y., Liu, T., Long, M., and Jordan, M. (2019). Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning. PMLR.
[100] Zhu, X. and Ghahramani, Z. (2003). Learning from labeled and unlabeled data with label propagation. CMU CALD tech report CMU-CALD-02-107, 2002.
Appendix Table of Contents
A Other Related Works 17
B More Details on Problem Setup 18
C Additional Experiments and Details 19
C.1 Additional setup and notation 19
C.2 Dataset details 19
C.3 Method details 20
C.4 Additional UDA experiments 22
C.5 Additional SSL experiments 23
C.6 Other experimental details 23
D Additional Results in Toy Setup 23
D.1 Detailed description of our simplified setup 23
D.2 Discussion on self-training and contrastive learning objectives 25
D.3 Additional empirical results in our simplified setup 26
D.4 Reconciling Practice: Experiments with deep networks in toy setup 27
E Formal Statements from Sec. 4 29
E.1 Analysis of ERM and ST: Formal Statement of Theorem 2 29
E.1.1 Proof of Proposition 3 34
E.1.2 Analysis with  $\rho >0$  inContrastive Pretraining Objective (3) 36
E.1.3 Proof of Theorem 5 38
E.1.4 Proof of Corollary 6 41
E.2 Analysis of STOC: Formal Statement of Theorem 7 42
E.3 Analysis for SSL 48
F Limitations of Prior Work 48
F.1Contrastive learning analysis 48
F.2 Self-training analysis 48
G Additional Lemmas 49
A Other Related Works
Unsupervised domain adaption. Without assumption on the nature of shift, UDA is underspecified [7]. This challenge has been addressed in various ways by researchers. One approach is to investigate additional structural assumptions under which UDA problems are well posed [77, 74]. Popular settings for which DA is well-posed include (i) covariate shift [96, 92, 19, 20, 35] where  $p(x)$  can change from source to target but  $p(y|x)$  remains invariant; and (ii) label shift [69, 53, 3, 1, 31, 95, 66, 30] where the label marginal  $p(y)$  can change but  $p(x|y)$  is shared across source and target. Principled methods with strong theoretical guarantees exist for adaptation under these settings when target
distribution's support is a subset of the source support. Other works [25, 6, 32, 28] extend the label shift setting to scenarios where previously unseen classes may appear in the target and  $p(x|y)$  remains invariant among seen classes. A complementary line of research focuses on constructing benchmarks to develop heuristics for incorporating the unlabeled target data, relying on benchmark datasets ostensibly representative of "real-world shifts" to adjudicate progress [72, 85, 70, 62, 63]. As a result, various benchmark-driven heuristics have been proposed [54, 55, 82, 81, 99, 98, 26, 79]. Our work engages with the latter, focusing on two popular methods: self-training and contrastive pretraining.
Domain generalization. In domain generalization, the model is given access to data from multiple different domains and the goal is to generalize to a previously unseen domain at test time [10, 59]. For a survey of different algorithms for domain generalization, we refer the reader to Gulrajani and Lopez-Paz [37]. A crucial distinction here is that unlike the domain generalization setting, in DA problems, we have access to unlabeled examples from the test domain.
Semi-supervised learning. To learn from a small amount of labeled supervision, semi-supervised learning methods leverage unlabeled data alongside to improve learning models. One of the seminal works in SSL is the pseudolabeling method [75], where a classifier is trained on the labeled data and then used to classify the unlabeled data, which are then added to the training set. The work of Zhu and Ghahramani [100] built on this by introducing graph-based methods, and the transductive SVMs [43] presented an SVM-based approach. More recent works have focused on deep learning techniques, and similar to UDA, self-training and contrastive pretraining have emerged as two prominent choices. We delve into these methods in greater detail in the following paragraphs. For a discussion on other SSL methods, we refer interested readers to [15, 84, 91].
Self-training. Two popular forms of self-training are pseudolabeling [52] and conditional entropy minimization [34], which have been observed to be closely connected [8, 52, 79, 78]. Motivated by its strong performance in SSL and UDA settings [79, 89, 30, 78], several theoretical works have made attempts to understand its behavior [50, 87, 17]. [87, 12] aims to understand the behavior of the global minimizer of self-training objective by studying input consistency regularization, which enforces stability of the prediction for different augmentations of the unlabeled data. Our analysis of self-training is motivated by the work of Chen et al. [17] which explores the iterative behavior of self-training to unlearn spurious features. The setting of spurious features is of particular interest, since prior works have specifically analyzed the failures of out-of-distribution generalization in the presence of spurious features [60, 71].
Contrastive learning. An alternate line of work that uses unlabeled data for learning representations in the pretraining stage is contrastive learning [36, 61, 13, 16, 88]. Given an augmentation distribution, the main goal of contrastive objectives is to map augmentations drawn from the same input (positive pairs) to similar features, and force apart features corresponding to augmentations of different inputs (negative pairs) [13, 14, 41]. Prior works [11, 44, 38] have also shown a close relationship between contrastive [16, 39] and non-contrastive objectives [4, 93]. Consequently, in our analysis pertaining to the toy setup we focus on the mathematically non-contrastive objective Barlow Twins [93]. Using this pretrained backbone (either as an initialization or as a fixed feature extractor) a downstream predictor is learned using labeled examples. Several works [39, 73, 38, 2, 44] have analyzed the in-distribution generalization of the downstream predictor via label consistency arguments on the graph of positive pairs (augmentation graph). In contrast, we study the impact of contrastive learning under distribution shifts in the UDA setup. Other works [76, 40] that examine contrastive learning for UDA also conform to the augmentation graph view point, making additional assumptions that guarantee linear transferability. In our simplified setup involving spurious correlations, these abstract assumptions break easily when the augmentations are of a generic nature, akin to practice. Finally, some empirical works [58, 57] have found self-supervised objectives like contrastive pretraining to reduce dependence on spurious correlations. Corroborating their findings, we extensively evaluate the complementary benefits of contrastive learning and self-training on real-world datasets. Finding differing results in SSL and UDA settings, we further examine their behavior theoretically in our toy setup.
B More Details on Problem Setup
In this section, we elaborate on our setup and methods studied in our work.
Unsupervised Domain Adaptation (UDA). We assume that we are given labeled data from the source distribution and unlabeled data from a shifted, target distribution, with the goal of performing well on target data. We assume that the source and target distributions have the same label marginals  $\mathrm{P}_{\mathrm{S}}(y) = \mathrm{P}_{\mathrm{T}}(y)$  (i.e., no label proportion shift) and the same Bayes optimal predictor, i.e.,  $\arg \max_{y} p_{\mathrm{S}}(y \mid x) = \arg \max_{y} p_{\mathrm{T}}(y \mid x)$ . Here, even with infinite labeled source data, the challenge lies in generalizing out-of-distribution. In experiments, we assume access to finite data but in theory, we assume population access to labeled source and unlabeled target.
Semi-Supervised Learning (SSL). Here, there is no distribution shift, i.e.,  $\mathrm{P_S} = \mathrm{P_T} = \mathrm{P_U}$ . We are given a small number of labeled examples and a comparatively large amount of unlabeled examples, both drawn from the same distribution. Without loss of generality, we denote this distribution with  $\mathrm{P_T}$ . The goal in SSL is to generalize in-distribution. The challenge is primarily due to limited access to labeled data. Here, in experiments, we assume limited access to labeled data but a comparatively larger amount of unlabeled in-distribution data. In theory, we assume population access to unlabeled data but limited labeled examples.
Methods. As discussed in the main paper, we compare four methods for learning with labeled and unlabeled data. Table 8 summarizes the main methods and key differences between those methods in UDA and SSL setup. For exact implementation in our experiments, we refer reader to App. C.3.
C Additional Experiments and Details
C.1 Additional setup and notation
Recall, our goal is to learn a predictor that maps inputs  $x \in \mathcal{X} \subseteq \mathbb{R}^d$  to outputs  $y \in \mathcal{Y}$ . We parameterize predictors  $f = h \circ \Phi : \mathbb{R}^d \mapsto \mathcal{Y}$ , where  $\Phi : \mathbb{R}^d \mapsto \mathbb{R}^k$  is a feature map and  $h \in \mathbb{R}^k$  is a classifier that maps the representation to the final scores or logits. With  $A : \mathcal{X} \to \mathcal{A}$ , we denote the augmentation function that takes in an input  $x$  and outputs an augmented view of the input  $A(x)$ . Unless specified otherwise, we perform full-finetuning in all of our experiments on real-world data. That is, we backpropagate gradients in both the linear head  $h$  and the backbone  $\phi$ . For UDA, we denote source labeled points as  $\{(x_i, y_i)\}_{i=1}^n$  and target unlabeled points as  $\{(x_i')\}_{i=1}^m$ . For SSL, we use the same notation for labeled and unlabeled in-distribution data.
C.2 Dataset details
For both UDA and SSL, we conduct experiments across eight benchmark datasets. Each of these datasets consists of domains, enabling us to construct source-target pairs for UDA. The adopted source and target domains are standard to previous studies [76, 30, 70]. Because the SSL setting lacks distribution shift, we do not need to worry about domain designations and default to using source alone. To simulate limited supervision in SSL, we sub-sample the original labeled training set to  $10\%$ . Below provide exact details about the datasets used in our benchmark study.
- CIFAR10 We use the original CIFAR10 dataset [48] as the source dataset. For target domains, we consider CINIC10 [22] which is a subset ofImagenet restricted to CIFAR10 classes and downsampled to  $32 \times 32$ .
- FMoW In order to consider distribution shifts faced in the wild, we consider FMoW-WILDs [47, 18] from WILDS benchmark, which contains satellite images taken in different geographical regions and at different times. We use the original train as source and OOD val and OOD test splits as target domains as they are collected over different time-period. Overall, we obtain 3 different domains (1 source and 2 targets).
- BREEDs We also consider BREEDs benchmark [72] in our setup to assess robustness to subpopulation shifts. BREEDs leverage class hierarchy in ImageNet [68] to re-purpose original classes to be the subpopulations and defines a classification task on superclasses. We consider distribution shift due to subpopulation shift which is induced by directly making the subpopulations present in the training and test distributions disjoint. BREEDs benchmark contains 4 datasets Entity-13, Entity-30, Living-17, and Non-living-26, each focusing on different subtrees and levels in the hierarchy. Overall, for each of the 4 BREEDs datasets (i.e., Entity-13, Entity-30, Living
Figure 4: Examples from all the domains in each dataset.
17, and Non-living-26), we obtain one different domain which we consider as target. We refer to source and target as follows: BREEDs sub-population 1, BREEDs sub-population 2.
- OfficeHome We use four domains (art, clipart, product and real) from OfficeHome dataset [85]. We use the product domain as source and the other domains as target.
- Visda We use three domains (train, val and test) from the Visda dataset [64, 63]. While 'train' domain contains synthetic renditions of the objects, 'val' and 'test' domains contain real world images. To avoid confusing, the domain names with their roles as splits, we rename them as 'synthetic', 'Real-1' and 'Real-2'. We use the synthetic (original train set) as the source domain and use the other domains as target.
We summarize the information about source and target domains in Table 3.
Train-test splits We partition each source and target dataset into  $80\%$  and  $20\%$  i.i.d. splits. We use  $80\%$  splits for training and  $20\%$  splits for evaluation (or validation). We throw away labels for the  $80\%$  target split and only use labels in the  $20\%$  target split for final evaluation. The rationale behind splitting the target data is to use a completely unseen batch of data for evaluation. This avoids evaluating on examples where a model potentially could have overfit. over-fitting to unlabeled examples for evaluation. In practice, if the aim is to make predictions on all the target data (i.e., transduction), we can simply use the (full) target set for training and evaluation.
Simulating SSL settings and limited supervision. For SSL settings, we choose the in-distribution domain as the source domain. To simulate limited supervision in SSL, we sub-sample the original labeled training set to  $10\%$  and use all the original dataset as unlabeled data. For evaluation, we further split the original holdout set into two partitions (one for validation and the other to report final accuracy numbers).
C.3 Method details
For implementation, we build on top of WILDs [70] and RLSbench [30] open source libraries.
Dataset	Source	Target
CIFAR10	CIFAR10v1	CINIC10
FMoW	FMoW (2002–’13)	FMoW (2013–’16), FMoW (2016–’18)
Entity13	Entity13 (sub-population 1)	Entity13 (sub-population 2)
Entity30	Entity30 (sub-population 1)	Entity30 (sub-population 2), Living17 (sub-population 1)
Living17	Living17 (sub-population 1)	Living17 (sub-population 2), Nonliving26 (sub-population 1)
Nonliving26	Nonliving26 (sub-population 1)	Nonliving26 (sub-population 2),
Officehome Product
Synthetic
(originally referred to as train)
Synthetic, Real-1 (originally referred to as val),
Real-2 (originally referred to as test)
Visda
Table 3: Details of source and target sets in each dataset considered in our testbed.
ERM (Source only) training. We consider Empirical Risk Minimization (ERM) on the labeled source data as a baseline. Since this simply ignores the unlabeled target data, we call this as source only training. As mentioned in the main paper, we perform source only training with data augmentations. Formally, we minimize the following ERM loss:
$$
L _ {\text {s o u r c e o n l y}} (f) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(f \left(A \left(x _ {i}\right), y _ {i}\right)\right), \tag {4}
$$
where  $A$  is the stochastic data augmentation operation and  $\ell$  is a loss function. For SSL, the ERM baseline only uses the small of labeled data available.
Contrastive Learning (CL). We perform contrastive pretraining on the unlabeled dataset to obtain the backbone  $\phi_{\mathrm{cl}}$ . And then we perform full fine-tuning with source labeled data by initializing the backbone with  $\phi_{\mathrm{cl}}$ . We use SwAV [13] for contrastive pretraining. The main idea behind SwAV is to train a model to identify different views of the same image as similar, while also ensuring that it finds different images to be distinct. This is accomplished through a swapped prediction mechanism, where the goal is to compute a code from an augmented version of the image and predict this code from other augmented versions of the same image. In particular, given two image features  $\phi(x_{a1}')$  and  $\phi(x_{a2}')$  from two different augmentations of the same image  $x'$ , i.e.,  $x_{a1}', x_{a2}' \sim A(x')$ , SwAV computes their codes  $z_{a1}$  and  $z_{a2}$  by matching the features to a set of  $K$  prototypes  $\{c_1, \dots, c_K\}$ . Then SwAV minimizes the following loss such that  $\phi(x_{a1}')$  can compute codes  $z_{a2}$  and  $\phi(x_{a2}')$  can compute codes  $z_{a1}$ :
$$
L _ {\mathrm {S w A V}} (\phi) = \sum_ {i = 1} ^ {m} \sum_ {x _ {i, a 1} ^ {\prime}, x _ {i, a 2} ^ {\prime} \sim A \left(x _ {i} ^ {\prime}\right)} \ell^ {\prime} \left(\phi \left(x _ {i, a 1} ^ {\prime}\right), z _ {i, a 2}\right) + \ell^ {\prime} \left(\phi \left(x _ {i, a 2} ^ {\prime}\right), z _ {i, a 1}\right), \tag {5}
$$
where  $\ell^{\prime}$  computes KL-divergence between codes computed with features (e.g.  $\phi(x_{a1})$ ) and the code computed by another view (e.g.  $z_{a2}$ ). For more details about the algorithm, we refer the reader to Caron et al. [13]. In all UDA settings, unless otherwise specified, we pool all the (unlabeled) data from the source and target to perform SwAV. For SSL, we leverage in-distribution unlabeled data.
We employ SimCLR [16] for the CIFAR10 dataset, aligning with previous studies that have utilized contrastive pretraining on the same dataset [51, 76]. The reason for this choice is that SwAV relies on augmentations that involve cropping images to a smaller resolution, making it more suitable for datasets with larger resolutions beyond  $32 \times 32$ .
Self-Training (ST). For self-training, we apply FixMatch [79], where the loss on labeled data and on pseudolabeled unlabeled data are minimized simultaneously. Sohn et al. [79] proposed FixMatch as a variant of the simpler Pseudo-label method [52]. This algorithm dynamically generates psuedolabels and overfits on them in each batch. FixMatch employs consistency regularization on the unlabeled data. In particular, while pseudolabels are generated on a weakly augmented view of