SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the unlabeled examples, the loss is computed with respect to predictions on a strongly augmented view. The intuition behind such an update is to encourage a model to make predictions on weakly augmented data consistent with the strongly augmented example. Moreover, FixMatch only overfits to the assigned labeled with weak augmentation if the confidence of the prediction with strong augmentation is greater than some threshold  $\tau$ . Refer to  $A_{\mathrm{weak}}$  as the weak-augmentation and  $A_{\mathrm{strong}}$  as the strong-augmentation function. Then, FixMatch uses the following loss function:
$$
\begin{array}{l} L _ {\text {F i x M a t c h}} (f) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(f \left(A _ {\text {s t r o n g}} \left(x _ {i}\right), y _ {i}\right)\right) \\ + \frac {\lambda}{m} \sum_ {i = 1} ^ {m} \ell \left(f \left(A _ {\text {s t r o n g}} \left(x _ {i} ^ {\prime}\right), \widetilde {y} _ {i}\right)\right) \cdot \mathbb {I} \left[ \max  _ {y} f _ {y} \left(A _ {\text {s t r o n g}} \left(x _ {i} ^ {\prime}\right)\right) \geqslant \tau \right], \\ \end{array}
$$
where  $\widetilde{y}_i = \arg \max_y f_y(T_{\mathrm{weak}}(x_i))$ . For UDA, our unlabeled data is the union of source and target unlabeled data. For SSL, we only leverage in-distribution unlabeled data.
We adapted our implementation from Sagawa et al. [70] which matches the implementation of Sohn et al. [79] except for one detail. While Sohn et al. [79] augments labeled examples with weak augmentation, Sagawa et al. [70] proposed to strongly augment the labeled source examples.
Self-Training Over Contrastive learning (STOC). Finally, rather than performing FixMatch from a randomly initialized backbone, we initialize FixMatch with a contrastive pretrained backbone.
C.4 Additional UDA experiments
Table 4: Results in the UDA setup. We report accuracy on target (OOD) data from which we only observe unlabeled examples during training. For benchmarks with multiple target distributions (e.g., OH, Visda), we report average accuracy on those targets.
Method	Living17	Nonliv26	Entity13	Entity30	FMoW (2 tgts)	Visda (2 tgts)	OH (3 tgts)	CIFAR→ CINIC
ERM	60.2±0.1	45.4±0.2	68.6±0.1	55.7±0.0	56.5±0.1	20.8±0.2	9.5±0.2	74.3±0.1
ST	71.1±0.2	56.8±0.1	78.0±0.3	66.7±0.1	56.9±0.4	39.1±0.1	11.1±0.1	78.3±0.3
CL	74.1±0.2	57.4±0.3	76.9±0.2	66.6±0.3	61.5±0.5	63.2±0.2	22.8±0.1	77.5±0.1
STOC (ours)	82.6±0.1	62.1±0.2	81.9±0.2	72.0±0.2	65.3±0.1	70.1±0.2	27.1±0.3	79.9±0.3
Table 5: Results in the UDA setup with source only contrastive pretraining. We report accuracy on target (OOD) data from which we only observe unlabeled examples during training. For benchmarks with multiple target distributions (e.g., OH, Visda), we report average accuracy on those targets.
Method	Living17	Nonliv26	Entity13	Entity30	FMoW(2 tgts)	Visda(2 tgts)	OH(3 tgts)	CIFAR→CINIC
CL (source only)	67.3±0.1	49.1±0.2	71.5±0.1	58.5±0.3	53.9±0.1	33.3±0.2	21.7±0.1	77.7±0.1
STOC (source only)	75.0±0.2	58.4±0.1	79.8±0.3	67.5±0.1	56.3±0.4	42.7±0.1	25.7±0.1	77.8±0.1
CL	74.1±0.2	57.4±0.3	76.9±0.2	66.6±0.3	61.5±0.5	63.2±0.2	22.8±0.1	77.5±0.1
STOC	82.6±0.1	62.1±0.2	81.9±0.2	72.0±0.2	65.3±0.1	70.1±0.2	27.1±0.3	79.9±0.3
C.5 Additional SSL experiments
Table 6: Results in the SSL setup. We report accuracy on hold-out ID data. Recall that SSL uses labeled and unlabeled data from the same distribution during training.
Method	Living17	Nonliv26	Entity13	Entity30	FMoW	Visda	OH	CIFAR
ERM	76.8±0.1	64.9±0.2	80.1±0.0	70.4±0.3	33.6±0.4	99.2±0.0	32.0±0.2	85.5±0.1
ST	85.4±0.1	75.7±0.2	85.4±0.2	77.3±0.1	33.6±0.3	99.2±0.1	32.0±0.1	93.1±0.1
CL	91.1±0.5	84.6±0.6	90.7±0.4	85.5±0.3	43.1±0.2	97.6±0.3	49.7±0.2	91.7±0.2
STOC (ours)	92.0±0.1	85.8±0.2	91.3±0.3	86.1±0.2	44.4±0.1	97.7±0.2	49.9±0.2	93.06±0.3
C.6 Other experimental details
Augmentations. For weak augmentation, we leverage random horizontal flips and random crops of pre-defined size. For SwAV, we also perform multicrop augmentation as proposed in Caron et al. [13]. For strong augmentation, we apply the following transformations sequentially: random horizontal flips, random crops of pre-defined size, augmentation with Cutout [24], and RandAugment [21]. For the exact implementation of RandAugment, we directly use the implementation of Sohn et al. [79]. Unless specified otherwise, for all methods, we default to using strong augmentation techniques.
Architectures. In our work, we experiment with Resnet18, Resnet50 [42] trained from scratch (i.e. random initialization). We do not consider off-the-shelf pretrained models (e.g., onImagenet [68]) to avoid confounding our conclusions about contrastive pretraining. However, we note that our results on most datasets tend to be comparable to and sometimes exceed those obtained with ImageNet pretrained models. For BREEDs datasets, we employ Resnet18 architecture. For other datasets, we train a Resnet50 architecture.
Except for Resnets on CIFAR dataset, we used the standard pytorch implementation [27]. For Resnet on Cifar, we refer to the implementation here: https://github.com/kuangliu/pytorch-cifar. For all the architectures, whenever applicable, we add antialIASing [97]. We use the official library released with the paper.
Hyperparameters. For all the methods, we fix the algorithm-specific hyperparameters to the original recommendations. For UDA, given that the setup precludes access to labeled data from the target distribution, we use source hold-out performance to pick the best hyperparameters. During pretraining, early stopping is done according to lower values of pretraining loss.
We tune the learning rate and  $\ell_2$  regularization parameter by fixing the batch size for each dataset that corresponds to the maximum we can fit to 15GB GPU memory. We default to using cosine learning rate schedule [56]. We set the number of epochs for training as per the suggestions of the authors of respective benchmarks. For SSL, we run both ERM and FixMatch for approximately 2000 epochs. Note that we define the number of epochs as a full pass over the labeled training source data. We summarize the learning rate, batch size, number of epochs, and  $\ell_2$  regularization parameter used in our study in Table 7.
Compute infrastructure. Our experiments were performed across a combination of Nvidia T4, A6000, and V100 GPUs.
D Additional Results in Toy Setup
In this section we will first give more details on our simplified setup that captures both contrastive pretraining and self-training in the same framework. Then, we provide some additional empirical results that are not captured theoretically but mimic behaviors observed in real world settings, highlighting the richness of our setup.
D.1 Detailed description of our simplified setup
In this subsection, we will first re-iterate the problem setup in Sec. 4 and provide some comparisons between our setup and those in closely related works. We will then describe the four methods: ERM,
Dataset	Batch size	l2 regularization set	Learning rate set
CIFAR10	200	{0.001, 0.0001, 10-5, 0.0}	{0.2, 0.1, 0.05, 0.01, 0.003, 0.001}
FMoW	64	{0.001, 0.0001, 10-5, 0.0}	{0.01, 0.003, 0.001, 0.0003, 0.0001}
Entity13	256	{0.001, 0.0001, 10-5, 0.0}	{0.4, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005}
Entity30	256	{0.001, 0.0001, 10-5, 0.0}	{0.4, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005}
Entity30	256	{0.001, 0.0001, 0-5, 0.0}	{0.4, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005}
Nonliving26	256	{0.001, 0.0001, 10-5, 0.0}	{0.4, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005}
Officehome	96	{0.001, 0.0001, 10-5, 0.0}	{0.01, 0.003, 0.001, 0.0003, 0.0001}
Visda	96	{0.001, 0.0001, 10-5, 0.0}	{0.03, 0.01, 0.003, 0.001, 0.0003}
Table 7: Details of the batch size, learning rate set and  ${\ell }_{2}$  regularization set considered in our testbed.
ST, CL, and STOC, providing details on the exact estimates returned by these algorithms in the SSL and UDA settings.
Data distribution. We consider binary classification and model the inputs as consisting of two kinds of features:  $x = [x_{\mathrm{in}}, x_{\mathrm{sp}}]$  where  $x_{\mathrm{in}} \in \mathbb{R}^{d_{\mathrm{in}}}$  is the invariant feature that is predictive of the label across both source  $\mathrm{P}_{\mathrm{S}}$  and target  $\mathrm{P}_{\mathrm{T}}$  and  $x_{\mathrm{sp}} \in \mathbb{R}^{d_{\mathrm{sp}}}$  is the spurious feature that is correlated with the label  $y$  only on the source domain  $\mathrm{P}_{\mathrm{S}}$  but uncorrelated with label  $y$  in  $\mathrm{P}_{\mathrm{T}}$ . Here,  $x_{\mathrm{in}} \in \mathbb{R}^{d_{\mathrm{in}}}$  determines the label using the ground truth classifier  $w^{\star} \sim \operatorname{Unif}(\mathbb{S}^{d_{\mathrm{in}} - 1})$ , and  $x_{\mathrm{sp}} \in \mathbb{R}^{d_{\mathrm{sp}}}$  is strongly correlated with the label on source but random noise on target. Formally, we sample  $y \sim \operatorname{Unif}\{-1,1\}$  and generate inputs  $x$  conditioned on  $y$  as follows
$$
\mathrm {P} _ {\mathrm {S}}: x _ {\mathrm {i n}} \sim \mathcal {N} (\gamma \cdot \mathrm {y} w ^ {\star}, \Sigma_ {\mathrm {i n}}) x _ {\mathrm {s p}} = \mathrm {y 1} _ {d _ {\mathrm {s p}}}
$$
$$
\mathrm {P} _ {\mathrm {T}}: x _ {\text {i n}} \sim \mathcal {N} (\gamma \cdot \mathrm {y} w ^ {\star}, \Sigma_ {\text {i n}}) x _ {\mathrm {s p}} \sim \mathcal {N} (\mathbf {0}, \Sigma_ {\mathrm {s p}}), \tag {6}
$$
where  $\gamma$  is the margin afforded by the invariant feature. We set covariance of the invariant features  $\Sigma_{\mathrm{in}} = \sigma_{\mathrm{in}}^2 \cdot (\mathbf{I}_{d_{\mathrm{in}}} - w^\star w^{\star \top})$  to capture structure in the invariant feature that the variance is less along the latent predictive direction  $w^{\star}$ . Note that the spurious feature is completely predictive of the label in the source data, and is distributed as spherical Gaussian in the target data with  $\Sigma_{\mathrm{sp}} = \sigma_{\mathrm{sp}}^2 \mathbf{I}_{d_{\mathrm{sp}}}$ .
Why is our simplified setup interesting? In our setup,  $x_{\mathrm{in}}$  is the hard to learn feature that generalizes from source to target. The hardness of learning this feature is determined by the value of the margin  $\gamma$  and how it compares with size of the spurious feature  $(\sqrt{d_{\mathrm{sp}}})$ . Since,  $\gamma / \sqrt{d_{\mathrm{sp}}}$  is small in our setup,  $x_{\mathrm{in}}$  is much harder to learn on source data (even with population access) compared to the spurious feature  $x_{\mathrm{sp}}$  which generalizes poorly from source to target. These two types of features have been captured in similar analysis on spurious correlations [71, 60] since it imitates pitfalls emanating from the presence of spurious features in real world datasets (e.g., the easy to learn background feature in image classification problems). While this setup is simple, it is also expressive enough to elucidate both self-training and contrastive learning behaviors we observe in real world settings. Specifically, it captures the separation results we observe in Sec. 3.
Differences of our setup with prior works. While our distribution shift settings bears the above similarities it also has important differences with works analyzing self-training and contrastive pretraining individually. Chen et al. [17] analyze the iterative nature of self-training algorithm, where the premise is that we are given a classifier that not only has good performance on source data but in addition does not rely too much on the spurious feature. Under the strong condition of small norms along the spurious feature, they show that self-training can provably unlearn this small dependence when the target data along the spurious feature is random noise. This assumption is clearly violated in setups where the spurious correlation is strong (as in our toy setup), i.e., the dependence on the spurious feature is rather large (much larger than that on the invariant feature) for any classifier that is trained directly on source data. Consequently, we show the need for "good" pretrained representations from contrastive pretraining over which if we train a linear predictor (using source labeled data), it will provably have a reduced "effective" dependence on the spurious feature.
Using an augmentation distribution similar to ours, Saunshi et al. [73] carried out contrastive pretraining analysis with the backbone belonging to a capacity constrained function class (similar analysis also in [40]). Our setup differs from this in two key ways: (i) we specifically consider a distribution
shift from source to target. Unlike their setting, it is not sufficient to make augmentations consistent with ground truth labels, since the predictor that uses just the spurious feature also assigns labels consistent with both ground truth predictions and augmentations on the source data; and (ii) our augmentation distribution assumes no knowledge of the invariant feature, which is why we augment all dimensions uniformly, as opposed to selectively augmenting a set of dimensions. In other words, we assume no knowledge of the structure of the optimal target predictor. For e.g., if we had knowledge of the spurious dimensions we could have just selectively augmented those. Assuming knowledge of these perfect augmentations is not ideal for two reasons: (a) it makes the problem so easy that just training an ERM model on source data with these augmentations would already yield a good target predictor (which rarely happens in practice); and (b) in real-world datasets perfect augmentations for the downstream task are not known. Hence, we stick to generic augmentations in our setup.
D.2 Discussion on self-training and contrastive learning objectives
Method	UDA Setup	SSL Setup
ERM:	h_erm = arg min_h E_P_S l(h(x), y)	h_erm = arg min_h 1/n ∑i=1n l(h(xi), yi)
{ (xi, yi) }^n i=1 ~ P_T^n
ST:	Starting from h_erm optimize over h (to get h_st):
EPT(x) l(h(x), sgn(h(x)))	Starting from h_erm optimize over h (to get h_st):
EPT(x) l(h(x), sgn(h(x)))
CL:	Φcl = arg min_φ L_cl(Φ)	Φcl = arg min_φ L_cl(Φ)
Use (PS(x) + PT(x))/2 for L_cl(Φ)	Use PT(x) for L_cl(Φ)
h_cl = arg min_h E_P_S l(h o Φ_cl(x), y)	h_cl = arg min_h 1/n ∑i=1n l(h o Φ_cl(x_i), y_i)
STOC:	Starting from h_cl optimize over h (to get h_stoc):
EPT(x) l(h o Φ_cl(x), sgn(h o Φ_cl(x)))	Starting from h_cl optimize over h (to get h_stoc):
EPT(x) l(h o Φ_cl(x), sgn(h o Φ_cl(x)))
Table 8: Description of methods for SSL vs. UDA: For each method we provide exact objectives used for experiments and analysis in the SSL and UDA setups (pertaining to Sec. 4).
In text we will describe our objectives and methods for the UDA setup. In Table 8 we constrain the differences in the methods and objectives for SSL and UDA setups. Recall from Section 2 that we learn linear classifiers  $h$  over features extractors  $\Phi$ . We consider linear feature extractor i.e.  $\Phi$  is a matrix in  $\mathbb{R}^{k\times d}$ . For mathematical convenience, we assume access to infinite unlabeled data and hence replace the empirical quantities over unlabeled data with their population counterpart. In the UDA setting, we further assume access to infinite labeled data from the source. Note that due to distribution shift between source and target, "ERM" on infinite labeled data from the source does not necessarily achieve optimal performance on the target. For binary classification, we assume that the linear layer  $h$  maps features to a scalar in  $\mathbb{R}$  such that the prediction is  $\mathrm{sgn}(h^\top \Phi x)$ . We use the exponential loss  $\ell(f(x), y) = \exp(-y f(x))$  as the classification loss.
Contrastive pretraining. We obtain  $\Phi_{\mathrm{cl}} \coloneqq \arg \min_{\Phi} \mathcal{L}_{\mathrm{cl}}(\Phi)$  by minimizing the Barlow Twins objective [93], which prior works have shown is also equivalent to spectral contrastive and non-contrastive objectives [33, 11]. In Sec. 4, we consider a constrained form of Barlow Twins in (3) which enforces representations of different augmentations  $a_1, a_2$  of the same input  $x$  to be close in representation space, while ensuring feature diversity by staying in the constraint set. We assume a strict constraint on regularization ( $\rho = 0$ ) for the theoretical arguments in the rest of the main paper. In App. E.1.2 we prove that all our claims hold for small  $\rho$  as well. In (7), we redefine the pretraining objective with a regularization term (instead of a constraint set) where  $\kappa$  controls the strength of the regularization term, with higher values of  $\kappa$  corresponding to stronger constraints on feature diversity. We then learn a linear classifier  $h_{\mathrm{cl}}$  over  $\Phi_{\mathrm{cl}}$  to minimize the exponential loss on labeled source data.
$$
\mathcal {L} _ {\mathrm {c l}} (\Phi) := \mathbb {E} _ {x \sim \mathrm {P} _ {\mathrm {U}}} \mathbb {E} _ {a _ {1}, a _ {2} \sim \mathrm {P} _ {\mathrm {A}} (\cdot | x)} \| \Phi (a _ {1}) - \Phi (a _ {2}) \| _ {2} ^ {2} + \kappa \cdot \| \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} [ \Phi (a) \Phi (a) ^ {\top} ] - \mathbf {I} _ {k} \| _ {F} ^ {2} \tag {7}
$$
Figure 5: Ablations on pretraining hyperparameters: In the UDA setup we plot the performance of CL and STOC as we vary two pretraining hyper-parameters: (left) the output dimension  $(k)$  of the feature extractor  $\Phi$ ; and (right) the strength  $(\kappa)$  of the regularizer in the Barlow Twins objective in (7). While ablating on  $k$  we fix  $\kappa = 0.5$ , and while ablating on  $\kappa$  we fix  $k = 10$ . Other problem parameters are taken from Example 1.
Augmentations. Data augmentations play a key role in contrastive pre-training (and also as we see later, state-of-the-art self-training variants like FixMatch). Given input  $x \in \mathcal{X}$ , let  $\mathrm{P_A}(a \mid x)$  denote the distribution over its augmentations, and  $\mathrm{P_A}$  denote the marginal distribution over all possible augmentations. We use the following simple augmentations where we scale the magnitude of each co-ordinate by a uniformly independent amount, i.e.,
$$
a \sim \mathrm {P} _ {\mathrm {A}} (\cdot \mid x) \equiv c \odot x \quad \text {w h e r e}, \quad c \sim \operatorname {U n i f} [ 0, 1 ] ^ {d}. \tag {8}
$$
The performance of different methods heavily depends on the assumptions we make on augmentations. We try to mirror practical settings where the augmentations are fairly "generic", not encoding any information about which features are invariant or spurious, and hence perturb all features symmetrically.
$$
\mathcal {L} _ {\mathrm {s t}} (h; \Phi) := \mathbb {E} _ {\mathrm {P} _ {\top} (x)} \ell \left(h ^ {\top} \Phi x, \operatorname {s g n} \left(h ^ {\top} \Phi (x)\right)\right) \quad \text {U p d a t e :} h ^ {t + 1} = \frac {h ^ {t} - \eta \nabla_ {h} \mathcal {L} _ {\mathrm {s t}} \left(h ^ {t} ; \Phi\right)}{\| h ^ {t} - \eta \nabla_ {h} \mathcal {L} _ {\mathrm {s t}} \left(h ^ {t} ; \Phi\right) \| _ {2}} \tag {9}
$$
For convenience, we keep the feature backbone  $\Phi$  fixed across the self-training iterations and only update the linear head on the pseudolabels.
STOC(Self-training after contrastive learning). Finally, we can combine the two unsupervised objectives where we do the self-training updates (2) with  $h_0 = h_{\mathrm{cl}}$  and  $\Phi_0 = \Phi_{\mathrm{cl}}$  starting with the contrastive learning model rather than just source-only ERM. Here, we only update  $h$  and fix  $\Phi_{\mathrm{cl}}$ .
D.3 Additional empirical results in our simplified setup
We conduct two ablations on the hyperparameters for contrastive pretraining. First, we vary the dimensionality  $k$  of the linear feature extractor  $\Phi \in \mathbb{R}^{k\times d}$ . Second, we vary the regularization strength  $\kappa$  that enforces feature diversity in the Barlow Twins objective (7). In Figure 5 we plot these ablations in the UDA setup.
Varying feature dimension. We find that CL recovers the full set of predictive features (i.e. both spurious and invariant) only when  $k$  is large enough (Figure 5(left)). Since the dimensionality of the true feature is 5 in our Example 1, reducing  $k$  below the true feature dimension hurts CL. Once  $k$  crosses a certain threshold, CL features completely capture the projection of the invariant feature  $w_{\mathrm{in}}$ . After this point, it amplifies the component along  $w_{\mathrm{in}}$ . It retains the amplification over the spurious feature  $w_{\mathrm{sp}}$  even as we increase  $k$ . This is confirmed by our finding that further increasing  $k$  does not hurt CL performance. This is also inline with our theoretical observations, where we find that for
Figure 6: Results with linear backbone: We plot the OOD accuracy for ERM, CL, ST and STOC in the UDA setup and ID accuracy in the SSL setup when the feature extractor  $\Phi$  is a linear network. Note, that the feature extractor is still fixed during CL and STOC.
suitable  $w^{\star}$ , the subspace spanned by  $w_{\mathrm{in}}$  and  $w_{\mathrm{sp}}$  are contained in a low rank space (as low as rank 2) of the contrastive representations (Theorem 5). Once CL has amplified the dependence along  $w_{\mathrm{in}}$  STOC improves over CL by unlearning any remaining dependence on the spurious  $w_{\mathrm{sp}}$ . The above arguments for the CL trend also explain why the performance of STOC continues to remain  $\approx 100\%$  as we vary  $k$ .
Varying regularization strength. In our main theoretical arguments we consider the constrained form of the Barlow Twins objective (3) with a strict constraint of  $\rho = 0$  (we relax this theoretically as well, see E.1.2). For our experiments, we optimize the regularized version of this objective (7), where the constraint term now appears as a regularizer which enforces feature diversity, i.e. the features learned through contrastive pretraining span orthogonal parts of the input space (as governed under the metric defined by augmentation covariance matrix  $\Sigma_A$ ). If  $\kappa$  is very low, then trivial solutions exist for the Barlow Twins objective. For e.g.,  $\phi \approx 0$  (zero vector) achieves very low invariance loss. When  $\kappa < 0.05$ , we find that CL recovers these trivial solutions (Figure 5(right)). Hence, both CL and STOC perform poorly. As we increase  $\kappa$  the performance of both CL and STOC improve, mainly because the features returned by  $\Phi_{\mathrm{cl}}$  now comprise of the predictive directions  $w_{\mathrm{in}}$  and  $w_{\mathrm{sp}}$ , as predictive by our theoretical arguments for  $\rho = 0$  (which corresponds to large  $\kappa$ ). On the other hand, when  $\kappa$  is too high optimization becomes hard since  $\kappa$  directly effects the Lipschitz constant of the loss function. Hence, the performance of CL drops by some value. Note that this does not effect the performance of STOC since CL continues to amplify  $w_{\mathrm{in}}$  over  $w_{\mathrm{sp}}$  even if it is returning suboptimal solutions with respect to the optimization loss of the pretraining objective.
D.4 Reconciling Practice: Experiments with deep networks in toy setup
In this section we delve into the details of Sec. 4.5, i.e., we analyze performance of different methods when we make some design choices that imitate practice. First, we look at experiments involving a deep non-linear backbone  $\Phi$ . Here, the non-linear  $\Phi$  is learned during contrastive pretraining and
Figure 7: Results with non-linear backbone: We plot the OOD accuracy for ERM, CL, ST and STOC in the UDA setup and ID accuracy in the SSL setup when the feature extractor  $\Phi$  is a nonlinear one-hidden layer network with ReLU activations. Note, that the feature extractor is still fixed during CL and STOC.
Figure 8: Finetuning the contrastive representations during STOC: We propagate gradients to the feature backbone  $\Phi$  when running STOC algorithm. Note that CL still fixes the contrastive representations when learning a fixed linear head over it. On the (left) we show results in UDA setup where we compare the performance of STOC with and without augmentations (along with other practical design choices like confidence thresholds and continuing to optimize source loss as done in FixMatch) when the feature backbone is non-linear. On the (right) we show results for STOC and CL in the SSL setup when the feature backbone is linear.