SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we prove our main theorem about the recovery of both spurious  $w_{\mathrm{sp}}$ , invariant  $w_{\mathrm{in}}$  features by the contrastive learning feature backbone, and also the amplification of the invariant over the spurious feature (where amplification is defined relatively with respect to what is observed in the data distribution alone). We begin by defining some quantities needed for analysis, that are fully determined by the choice of problem parameters for the model in (6).
From Section 4, we recall the definitions of  $w_{\mathrm{in}} \coloneqq [w^{\star},0,\dots ,0]$  and  $w_{\mathrm{sp}} \coloneqq [0,\dots 0,w^{\prime}]$  where  $w^{\prime} = \mathbf{1}_{d_{\mathrm{sp}}} / \sqrt{d_{\mathrm{sp}}}$ . Let us now define  $u_{1},u_{2}$  as the top two eigenvectors of  $\Sigma_A$  with eigenvalues  $\lambda_1,\lambda_2 > 0$ , (note that in our problem setup both  $\Sigma_A$  and  $\widetilde{\Sigma}$  are full rank positive definite matrices), and  $\tau \coloneqq \sqrt{\lambda_1 / \lambda_2}$ . Next we define  $\alpha$  as the angle between  $u_{1}$  and  $w_{\mathrm{in}}$ , i.e.,  $\cos (\alpha) = u_1^\top w_{\mathrm{in}}$ . Based on the definitions of  $\alpha$  and  $\tau$ , both of which are fully determined by the eigen decomposition of the post-augmentation feature covariance matrix  $\Sigma_A$ , we now restate Theorem 5:
Theorem 14 (Formal; CL recovers both invariant  $w_{\mathrm{in}}$  and spurious  $w_{\mathrm{sp}}$  but amplifies  $w_{\mathrm{in}}$ ). Under Assumption 4 ( $w^{\star} = \mathbf{1}_{d_{\mathrm{in}}} / \sqrt{d_{\mathrm{in}}}$ ), the CL solution  $\Phi_{\mathrm{cl}} = [\phi_1, \phi_2, \dots, \phi_k]$  satisfies  $\phi_j^\top w_{\mathrm{in}} = \phi_j^\top w_{\mathrm{sp}} = 0 \forall j \geqslant 3$ . For  $\tau, \alpha$  as defined above, the solution for  $\phi_1, \phi_2$  is:
$$
\left[ \begin{array}{c c} w ^ {\star} \cdot \cot (\alpha) / \tau , & w ^ {\star} \\ w ^ {\prime} \cdot 1 / \tau , & w ^ {\prime} \cdot \cot (\alpha) \end{array} \right] \enspace . \quad \left[ \begin{array}{c c} \cos \theta , & \sin \theta \\ \sin \theta , & - \cos \theta \end{array} \right],
$$
where  $0 \leqslant \alpha, \theta \leqslant \pi / 2$ . Let us redefine  $\phi_1 = c_1 w_{\mathrm{in}} + c_3 w_{\mathrm{sp}}$  and  $\phi_2 = c_2 w_{\mathrm{in}} + c_4 w_{\mathrm{sp}}$ .
For constants  $K_{1}, K_{2} > 0$ ,  $\gamma = K_{1}K_{2} / \sigma_{\mathrm{sp}}$ ,  $d_{\mathrm{sp}} = \sigma_{\mathrm{sp}}^{2} / K_{2}^{2}$ ,  $\forall \epsilon > 0$ ,  $\exists \sigma_{\mathrm{sp0}}$ , such that for  $\sigma_{\mathrm{sp}} \geqslant \sigma_{\mathrm{sp0}}$ :
$$
\frac {K _ {1} K _ {2} ^ {2} d _ {\mathrm {i n}}}{2 L \sigma_ {\mathrm {i n}} ^ {2} (d _ {\mathrm {i n}} - 1)} + \epsilon \geqslant \frac {c _ {1}}{c _ {3}} \geqslant \frac {K _ {1} K _ {2} ^ {2} d _ {\mathrm {i n}}}{2 L \sigma_ {\mathrm {i n}} ^ {2} (d _ {\mathrm {i n}} - 1)} - \epsilon
$$
$$
\frac {L \sqrt {d _ {\mathrm {s p}}}}{\gamma} + \epsilon \geqslant \left| \frac {c _ {2}}{c _ {4}} \right| \geqslant \frac {L \sqrt {d _ {\mathrm {s p}}}}{\gamma} - \epsilon ,
$$
where  $L = 1 + K_2^2$
Proof. We will first show that the only components of interest are  $\phi_1, \phi_2$ . Then, we will prove conditions on the amplification of  $w_{\mathrm{in}}$  over  $w_{\mathrm{sp}}$  in  $\phi_1, \phi_2$ . Following is the proof overview:
I. When  $w^{\star} = 1_{d_{\mathrm{in}}} / \sqrt{d_{\mathrm{in}}}$ , from the closed form expressions for  $\Sigma_A$  and  $\widetilde{\Sigma}$ , show that the solution returned by solving the Barlow Twins objective depends on  $w_{\mathrm{in}}$  and  $w_{\mathrm{sp}}$  only through the first two components  $\phi_1, \phi_2$ .
II. For the components  $\phi_1, \phi_2$ , we will show that the dependence along  $w_{\mathrm{in}}$  is amplified compared to  $w_{\mathrm{sp}}$  when the target data sufficiently denoises the spurious feature (i.e.,  $\sigma_{\mathrm{sp}}$  is sufficiently large).
Part-I:
We can divide the space  $\mathbb{R}^d$  into two subspaces that are perpendicular to each other. The first subspace is  $\mathcal{W} = \{b_1 \cdot w_{\mathrm{in}} + b_2 \cdot w_{\mathrm{sp}} : b_1, b_2 \in \mathbb{R}\}$ , i.e. the rank 2 subspace spanned by  $w_{\mathrm{in}}$  and  $w_{\mathrm{sp}}$ . The second subspace is  $\mathcal{W}_{\perp}$  where  $\mathcal{W}_{\perp} = \{u \in \mathbb{R}^d : u^\top w_{\mathrm{in}} = 0, u^\top w_{\mathrm{sp}} = 0\}$ . Then, from Lemma 23 we can conclude that the matrix  $\Sigma_A$  can be written as:
$$
\Sigma_ {A} = \Sigma_ {A _ {\mathcal {W}}} + \Sigma_ {A _ {\mathcal {W} _ {\perp}}}
$$
$$
\Sigma_ {A _ {\mathcal {W}}} = \frac {1}{4} \left[ \begin{array}{c c} \left(\gamma^ {2} \left(1 + 1 / 3 d _ {\mathrm {i n}}\right) + \sigma_ {\mathrm {i n}} ^ {2} / 3 \left(1 - 1 / d _ {\mathrm {i n}}\right)\right) \cdot w ^ {\star} w ^ {\star^ {\top}}, & \gamma \sqrt {d _ {\mathrm {s p}}} / 2 \cdot w ^ {\star} w ^ {\prime^ {\top}} \\ \gamma \sqrt {d _ {\mathrm {s p}}} / 2 \cdot w ^ {\prime} w ^ {\star^ {\top}}, & \left(d _ {\mathrm {s p}} / 2 + 4 / 3 \cdot \sigma_ {\mathrm {s p}} ^ {2} + 1 / 6\right) \cdot w ^ {\prime} w ^ {\prime^ {\top}} \end{array} \right], \tag {97}
$$
where  $\Sigma_{A_{\mathcal{W}_{\perp}}} \coloneqq \mathbb{E}_{a \sim \mathrm{P}_{\mathrm{A}}}\left[\Pi_{\mathcal{W}_{\perp}}(a)(\Pi_{\mathcal{W}_{\perp}}(a))^{\top}\right]$  is the covariance matrix in the null space of  $\mathcal{W}$ , and  $\Pi_{\mathcal{W}_{\perp}}(a)$  is the projection of augmentation  $a$  into the null space of  $\mathcal{W}$ , i.e. the covariance matrix in the space of non-predictive (noise) features. Similarly we can define:
$$
\tilde {\Sigma} = \tilde {\Sigma} _ {\mathcal {W}} + \tilde {\Sigma} _ {\mathcal {W} _ {\perp}}
$$
$$
\tilde {\Sigma} _ {\mathcal {W}} = \frac {1}{4} \left[ \begin{array}{c c} \gamma^ {2} \cdot w ^ {\star} w ^ {\star} ^ {\top}, & \gamma \sqrt {d _ {\mathrm {s p}}} / 2 \cdot w ^ {\star} w ^ {\prime} ^ {\top} \\ \gamma \sqrt {d _ {\mathrm {s p}}} / 2 \cdot w ^ {\prime} w ^ {\star} ^ {\top}, & \left(d _ {\mathrm {s p}} / 2 + \sigma_ {\mathrm {s p}} ^ {2} / 2\right) \cdot w ^ {\prime} w ^ {\prime} ^ {\top} \end{array} \right] \tag {98}
$$
Here again  $\widetilde{\Sigma}_{\mathcal{W}_{\perp}} := \mathbb{E}_{x \sim \mathrm{P}_U} \left[ \Pi_{\mathcal{W}_{\perp}} (\mathbb{E}_{c \sim \mathrm{Unif}[0,1]^d}(c \odot x)) (\Pi_{\mathcal{W}_{\perp}} (\mathbb{E}_{c \sim \mathrm{Unif}[0,1]^d}(c \odot x)))^\top \right]$  is the covariance matrix of mean augmentations after they are projected onto the null space of predictive features. The above decomposition also follows from result in Lemma 23.
From Proposition 3, the closed form expression for the solution returned by optimizing the Barlow Twins objective in (3) is  $U^{\top}\Sigma_{A}^{-1 / 2}$  where  $U$  are the top-k eigenvectors of:
$$
\sum_ {A} ^ {- 1 / 2} \cdot \tilde {\Sigma} \cdot \sum_ {A} ^ {- 1 / 2} \tag {99}
$$
When  $w^{\star} = \mathbf{1}_{d_{\mathrm{in}}} / \sqrt{d_{\mathrm{in}}}$ , then  $\Sigma_{A\mathcal{W}_{\perp}} = \widetilde{\Sigma}_{\mathcal{W}_{\perp}} + B$  where  $B$  is a diagonal matrix with diagonal given by  $\frac{1}{3} \cdot \mathrm{diag}(\widetilde{\Sigma}_{\mathcal{W}_{\perp}})$ . Further, since  $\mathrm{diag}(\widetilde{\Sigma}_{\mathcal{W}_{\perp}}) = p \cdot \mathbb{1}_d$  for some constant  $p > 0$ , the eigenvectors of  $\widetilde{\Sigma}_{\mathcal{W}_{\perp}}$  and  $\Sigma_{A\mathcal{W}_{\perp}}$  are exactly the same. Hence, when we consider the SVD of the expression  $\Sigma_A^{-1/2}\widetilde{\Sigma}\Sigma_A^{-1/2}$ , the matrices  $\Sigma_{A\mathcal{W}_{\perp}}$  and  $\widetilde{\Sigma}_{\mathcal{W}_{\perp}}$  have no effect on the SVD components that lie along the span of the predictive features. In fact, we only need to consider two rank 2 matrices (first terms in (98), (97)) and only do the SVD of  $\Sigma_{A\mathcal{W}}^{-1/2} \cdot \widetilde{\Sigma}_{\mathcal{W}} \cdot \Sigma_{A\mathcal{W}}^{-1/2}$ .
There are only two eigenvectors of  $\Sigma_{A_{\mathcal{W}}}^{-1/2} \cdot \widetilde{\Sigma}_{\mathcal{W}} \cdot \Sigma_{A_{\mathcal{W}}}^{-1/2}$ . We use  $\lambda_1, \lambda_2$  to denote the eigenvalues of  $\Sigma_{A_{\mathcal{W}}}$ , and  $[\cos(\alpha)w^{\star}, \sin(\alpha)w']^\top$ ,  $[\sin(\alpha)w^{\star}, -\cos(\alpha)w']^\top$  for the corresponding eigenvectors. Similarly, we use  $\widetilde{\lambda}_1, \widetilde{\lambda}_2$  to denote the eigenvalues of  $\widetilde{\Sigma}_{\mathcal{W}}$ , and  $[\cos(\beta)w^{\star}, \sin(\beta)w']^\top$ ,  $[\sin(\beta)w^{\star}, -\cos(\beta)w']^\top$  for the corresponding eigenvectors. Let  $\mathrm{SVD}_U(\cdot)$  denote the operation of obtaining the singular vectors of a matrix. Then, to compute the components of the final expression:  $\mathrm{SVD}_U(\Sigma_A^{-1/2}\widetilde{\Sigma}\Sigma_A^{-1/2})^\top\Sigma_A^{-1/2}$  that lies along the span of predictive features (in  $\mathcal{W}$ ), we need only look at the decomposition of the following matrix:
$$
\left[ \begin{array}{l l} \cos \theta , & \sin (\theta) \\ \sin \theta , - \cos (\theta) \end{array} \right] = \operatorname {S V D} _ {U} \left(\left[ \begin{array}{l l} 1 / \sqrt {\lambda_ {1}}, & 0 \\ 0, & 1 / \sqrt {\lambda_ {2}} \end{array} \right] \cdot \left[ \begin{array}{l l} \cos (\alpha - \beta), & \sin (\alpha - \beta) \\ \sin (\alpha - \beta), & - \cos (\alpha - \beta) \end{array} \right] \cdot \left[ \begin{array}{l l} \sqrt {\tilde {\lambda} _ {1}}, & 0 \\ 0, & \sqrt {\tilde {\lambda} _ {2}} \end{array} \right]\right) \tag {100}
$$
Based on the above definitions of  $\theta, \alpha, \lambda_1, \lambda_2$ , we can then formulate  $\phi_1$  and  $\phi_2$  in the following way:
$$
\left[ \phi_ {1}, \phi_ {2} \right] = \left[ \begin{array}{l l} w ^ {\star} \cdot \frac {\cos (\alpha)}{\sqrt {\lambda_ {1}}}, & w ^ {\star} \cdot \frac {\sin (\alpha)}{\sqrt {\lambda_ {2}}} \\ w ^ {\prime} \cdot \frac {\sin (\alpha)}{\sqrt {\lambda_ {1}}}, & w ^ {\prime} \frac {- \cos (\alpha)}{\sqrt {\lambda_ {2}}} \end{array} \right] \cdot \left[ \begin{array}{l l} \cos \theta , & \sin (\theta) \\ \sin \theta , & - \cos (\theta) \end{array} \right] \tag {101}
$$
To summarize, using arguments in Lemma 23 and the fact that  $w^{\star} = \mathbf{1}_{d_{\mathrm{in}}} / \sqrt{d_{\mathrm{in}}}$ , we can afford to focus on just two rank two matrices  $\Sigma_{A\mathcal{W}}$ ,  $\widetilde{\Sigma}_{\mathcal{W}}$  in the operation:  $\mathrm{SVD}_U(\Sigma_A^{-1/2})\widetilde{\Sigma}\Sigma_A^{-1/2}$ . The other singular vectors from the SVD only impact directions that span  $\mathcal{W}_{\perp}$ , and the singular vectors obtained by considering only the rank 2 matrices lie only in the space of  $\mathcal{W}$ .
Part-II:
From the previous part we obtained forms of  $\phi_1, \phi_2$  in terms of:  $\lambda_1, \lambda_2, \alpha, \theta$ , all of which are fully specified by the SVD of  $\Sigma_{A_{\mathcal{W}}}$  and  $\widetilde{\Sigma}_{\mathcal{W}}$ . If we define  $\tau := \frac{\sqrt{\lambda_1}}{\sqrt{\lambda_2}}$ , we can evaluate  $c_1, c_2, c_3, c_4$  as:
$$
c _ {1} = \frac {\cot (\alpha)}{\tau} + \tan (\theta) \tag {102}
$$
$$
c _ {2} = - 1 + \frac {\cot (\alpha) \tan (\theta)}{\tau} \tag {103}
$$
$$
c _ {3} = \frac {1}{\tau} - \cot (\alpha) \tan (\theta) \tag {104}
$$
$$
c _ {4} = \frac {\tan (\theta)}{\tau} + \cot (\alpha) \tag {105}
$$
Now, we are ready to begin proofs for our claims on the amplification factors, i.e. on the ratios  $c_1 / c_3$ ,  $|c_2 / c_4|$ .
We will first prove some limiting conditions for  $c_{1} / c_{3}$ , followed by those on  $|c_{2} / c_{4}|$ . For each of these conditions we will rely on the forms for  $c_{1}, c_{2}, c_{3}, c_{4}$  derived in the previous part, in terms of  $\alpha, \theta, \tau$  (where  $0 \leqslant \alpha, \theta \leqslant \pi / 2$ ). We will also rely on some lemmas that characterize the asymptotic behavior of  $\alpha, \theta$  and  $\tau$  as we increase  $\sigma_{\mathrm{sp}}$ . We defer the full proof of these helper lemmas to later sections.
Asymptotic behavior of  $c_{1} / c_{3}$
From Lemma 25 and Lemma 26, when  $\gamma = K_1 / \sqrt{z}$  and  $\sigma_{\mathrm{sp}} = K_2\sqrt{z}$ , then:
$$
\lim  _ {z \rightarrow \infty} \frac {c _ {1}}{c _ {3}} = \frac {\cot \alpha + \tau \tan \theta}{1 - \tau \cot \alpha \tan \theta} = \lim  _ {z \rightarrow \infty} \tau \tan \theta = \frac {K _ {1} K _ {2} ^ {2}}{\left(1 + K _ {2} ^ {2}\right) 2 \sigma_ {\mathrm {i n}} ^ {2} \left(1 - ^ {1} / d _ {\mathrm {i n}}\right)}, \tag {106}
$$
where we apply Moore-Osgood when applying limits on intermediate forms. We can do this since  $\tau$  tan  $\theta$  approaches a constant, and each of cot  $\alpha$ ,  $\tau$  and tan  $\theta$  are continuous and smooth functions of  $z$  (see Lemma 24).
Asymptotic behavior of  $\left| \frac{c_2}{c_4} \right|$ .
When we consider the limiting behavior of  $\frac{c_2}{c_4z}$ , as we increase  $z$  or equivalently  $\sigma_{\mathrm{sp}}$  when  $\gamma = K_1 / \sqrt{z}$  and  $\sigma_{\mathrm{sp}} = K_2\sqrt{z}$ , then we get:
$$
\lim  _ {z \rightarrow \infty} \left| \frac {c _ {2}}{c _ {4} z} \right| = \left| \frac {- 1 + \cot (\alpha) \tan (\theta)}{\frac {\tan (\theta) z}{\tau} + \cot (\alpha) z} \right|. \tag {107}
$$
From Lemma 26,  $\cot \alpha \tan \theta \rightarrow 0$ . Next, if we consider  $\lim_{z\to \infty}z\tan \theta /\tau = \lim_{z\to \infty}\tau \tan \theta \cdot z / \tau^2$ . For  $z / \tau^2$ , we invoke Lemma 28, which states that when  $\gamma = K_1 / \sqrt{z}$  and  $\sigma_{\mathrm{sp}} = K_2\sqrt{z}$ , then:
$$
\lim  _ {z \rightarrow \infty} \frac {z}{\tau^ {2}} = \frac {2 \sigma_ {\mathrm {i n}} ^ {2} / 3 \left(1 - 1 / d _ {\mathrm {i n}}\right)}{1 + 4 / 3 K _ {2} ^ {2}}. \tag {108}
$$
Further, in our bound on  $c_{1} / c_{3}$ , we derived that  $\tau \tan \theta \rightarrow K_{1}K_{2}^{2} / (1 + K_{2}^{2})2\sigma_{\mathrm{in}}^{2}(1 - 1 / d_{\mathrm{in}})$ . Once again using Moore-Osgood we can plug this along with (108) to get:
$$
\lim  _ {z \rightarrow \infty} \frac {\tan (\theta) z}{\tau} = \frac {K _ {1} K _ {2} ^ {2}}{\left(1 + K _ {2} ^ {2}\right)\left(3 + 4 K _ {2} ^ {2}\right)}. \tag {109}
$$
Finally, from Lemma 27, when  $\gamma = K_1 / \sqrt{z}$  and  $\sigma_{\mathrm{sp}} = K_2\sqrt{z}$ , then:
$$
\lim  _ {z \rightarrow \infty} \frac {z}{\tan \alpha} = \frac {K _ {1}}{\left(1 + ^ {4} / 3 K _ {2} ^ {2}\right)}. \tag {110}
$$
Plugging, 109 and 110 into 107 we get the following limit:
$$
\lim  _ {z \rightarrow \infty} \left| \frac {c _ {2}}{c _ {4} z} \right| = \frac {1 + K _ {2} ^ {2}}{K _ {1}}. \tag {111}
$$
Since  $z = K_{1}\sqrt{d_{\mathrm{sp}}}\big / \gamma$
$$
\lim  _ {z \rightarrow \infty} \left| \frac {c _ {2} \gamma}{c _ {4} K _ {1} \sqrt {d _ {\mathrm {s p}}}} \right| = \frac {1 + K _ {2} ^ {2}}{K _ {1}} \quad \Longrightarrow \quad \lim  _ {z \rightarrow \infty} \left| \frac {c _ {2} \gamma}{c _ {4} \sqrt {d _ {\mathrm {s p}}}} \right| = 1 + K _ {2} ^ {2} \tag {112}
$$
Since both  $c_{1} / c_{3}$  and  $|c_{2} / c_{4}|$  are continuous functions of  $z$ , with  $\lim \inf_{z\to \infty}$  and  $\lim \sup_{z\to \infty}$  converging to the limits in 106 and 107 for both quantities respectively, we conclude that  $\forall \epsilon >0$  there exists  $\sigma_{\mathrm{sp}_0}$  such that for all  $\sigma_{\mathrm{sp}}\geqslant \sigma_{\mathrm{sp}_0}$ , the following is true:
$$
\frac {K _ {1} K _ {2} ^ {2} d _ {\mathrm {i n}}}{2 L \sigma_ {\mathrm {i n}} ^ {2} \left(d _ {\mathrm {i n}} - 1\right)} + \epsilon \geqslant \frac {c _ {1}}{c _ {3}} \geqslant \frac {K _ {1} K _ {2} ^ {2} d _ {\mathrm {i n}}}{2 L \sigma_ {\mathrm {i n}} ^ {2} \left(d _ {\mathrm {i n}} - 1\right)} - \epsilon \tag {113}
$$
$$
\frac {\left(1 + K _ {2} ^ {2}\right) \sqrt {d _ {\mathrm {s p}}}}{\gamma} + \epsilon \geqslant \left| \frac {c _ {2}}{c _ {4}} \right| \geqslant \frac {\left(1 + K _ {2} ^ {2}\right) \sqrt {d _ {\mathrm {s p}}}}{\gamma} - \epsilon , \tag {114}
$$
This completes both Part-I and Part-II of the proof for Theorem 5.
E.1.4 Proof of Corollary 6
Corollary 15 (CL improves OOD error over ERM but is still imperfect). For  $\gamma$ ,  $\sigma_{\mathrm{sp}}$ ,  $d_{\mathrm{sp}}$  defined as in Theorem 5,  $\exists \sigma_{\mathrm{sp}_1}$  such that  $\forall \sigma_{\mathrm{sp}} \geqslant \sigma_{\mathrm{sp}_1}$ , the target accuracy of CL (linear predictor on  $\Phi_{\mathrm{cl}}$ ) is  $\geqslant 0.5$ $\operatorname{erfc}(-L' \cdot \gamma / \sqrt{2} \sigma_{\mathrm{sp}})$  and  $\leqslant 0.5$ $\operatorname{erfc}(-4L' \cdot \gamma / \sqrt{2} \sigma_{\mathrm{sp}})$ , where  $L' = K_2^2 K_1 / \sigma_{\mathrm{in}}^2 (1 - {}^1 / d_{\mathrm{in}})$ . When  $\sigma_{\mathrm{sp}_1} > \sigma_{\mathrm{in}} \sqrt{1 - 1 / d_{\mathrm{in}}}$ , the lower bound on accuracy is strictly better than ERM from scratch.
Proof. Recall from Theorem 14, all  $\phi_j$ , for  $j \geqslant 3$ , lie in the null space of  $w_{\mathrm{in}}$  and  $w_{\mathrm{sp}}$ . Since, the predictive features are strictly contained in the rank two space spanned by  $w_{\mathrm{in}}$  and  $w_{\mathrm{sp}}$ , without loss of generality we can restrict ourselves to the case where  $k = 2$ , and when doing training a head  $h = [h_1, h_2]^\top \in \mathbb{R}^2$  over contrastive pretrained representations using source labeled data, we get the following max margin solution:
$$
h _ {1} = c _ {1} \cdot \gamma + c _ {3} \cdot \sqrt {d _ {\mathrm {s p}}}
$$
$$
h _ {2} = c _ {2} \cdot \gamma + c _ {4} \cdot \sqrt {d _ {\mathrm {s p}}} \tag {115}
$$
Without loss of generality we can divide both  $h_1$  and  $h_2$  by  $h_1$  and get the final classifier to be  $\phi_1 + \frac{h_2}{h_1} \cdot \phi_2$ :
$$
\begin{array}{l} \left(c _ {1} w _ {\mathrm {i n}} + c _ {3} w _ {\mathrm {s p}}\right) + \frac {h _ {2}}{h _ {1}} \cdot \left(c _ {2} w _ {\mathrm {i n}} + c _ {4} w _ {\mathrm {s p}}\right) \\ = \left(c _ {1} w _ {\text {i n}} + c _ {3} w _ {\text {s p}}\right) + \frac {\left(c _ {2} \gamma + c _ {4} \sqrt {d _ {\mathrm {s p}}}\right)}{\left(c _ {1} \gamma + c _ {3} \sqrt {d _ {\mathrm {s p}}}\right)} \cdot \left(c _ {2} w _ {\text {i n}} + c _ {4} w _ {\text {s p}}\right) \tag {116} \\ \end{array}
$$
From Lemma 29, we can derive the target accuracy of the classifier  $h$  on top of CL representations to be the following:
$$
0. 5 \operatorname {e r f c} \left(- \frac {c _ {1} + \beta c _ {2}}{c _ {3} + \beta c _ {4}} \cdot \frac {\gamma}{\sqrt {2} \sigma_ {\mathrm {s p}}}\right) \tag {117}
$$
where  $\beta = (c_{2}\gamma + c_{4}\sqrt{d_{\mathrm{sp}}}) / (c_{1}\gamma + c_{3}\sqrt{d_{\mathrm{sp}}})$
Substituting  $\beta$  into the expression  $\frac{c_1 + \beta c_2}{c_3 + \beta c_4}$  we get:
$$
\frac {c _ {1} ^ {2} \gamma + c _ {1} c _ {3} \sqrt {d _ {\mathrm {s p}}} + c _ {2} ^ {2} \gamma + c _ {2} c _ {4} \sqrt {d _ {\mathrm {s p}}}}{c _ {1} c _ {3} \gamma + c _ {3} ^ {2} \sqrt {d _ {\mathrm {s p}}} + c _ {2} c _ {4} \gamma + c _ {4} ^ {2} \sqrt {d _ {\mathrm {s p}}}} \tag {118}
$$
We first substitute expressions for  $c_{1}, c_{2}, c_{3}, c_{4}$  from (102), (103), (104) and (105) in the above expression. Then for  $\gamma = K_{1} / \sqrt{z}$ ,  $\sigma_{\mathrm{sp}} = K_{2}\sqrt{z}$ , we substitute the expressions for  $\cot \alpha$ ,  $\tan \theta$ , and  $\tau = \lambda_1 / \lambda_2$  with their corresponding closed form expressions (as functions of  $z$ ) from Lemma 24. On the resulting expression we apply do repeated applications of L'Hôpital's rule to get the following result:
$$
\lim  _ {z \rightarrow \infty} \frac {c _ {1} ^ {2} \gamma + c _ {1} c _ {3} \sqrt {d _ {\mathrm {s p}}} + c _ {2} ^ {2} \gamma + c _ {2} c _ {4} \sqrt {d _ {\mathrm {s p}}}}{c _ {1} c _ {3} \gamma + c _ {3} ^ {2} \sqrt {d _ {\mathrm {s p}}} + c _ {2} c _ {4} \gamma + c _ {4} ^ {2} \sqrt {d _ {\mathrm {s p}}}} = \frac {2 K _ {2} ^ {2} K _ {1}}{\sigma_ {\mathrm {i n}} ^ {2} \left(1 - 1 / d _ {\mathrm {i n}}\right)} \tag {119}
$$
Based on  $\gamma, d_{\mathrm{sp}}, \sigma_{\mathrm{sp}}$  defined in Theorem 5, and (119) we can conclude that  $\exists \sigma_{\mathrm{sp}_1}$  such that for all  $\sigma_{\mathrm{sp}} \geqslant \sigma_{\mathrm{sp}_1}$ :
$$
\frac {4 K _ {2} ^ {2} K _ {1}}{\sigma_ {\mathrm {i n}} ^ {2} \left(1 - ^ {1} / d _ {\mathrm {i n}}\right)} \geqslant \frac {c _ {1} ^ {2} \gamma + c _ {1} c _ {3} \sqrt {d _ {\mathrm {s p}}} + c _ {2} ^ {2} \gamma + c _ {2} c _ {4} \sqrt {d _ {\mathrm {s p}}}}{c _ {1} c _ {3} \gamma + c _ {3} ^ {2} \sqrt {d _ {\mathrm {s p}}} + c _ {2} c _ {4} \gamma + c _ {4} ^ {2} \sqrt {d _ {\mathrm {s p}}}} \geqslant \frac {K _ {2} ^ {2} K _ {1}}{\sigma_ {\mathrm {i n}} ^ {2} \left(1 - ^ {1} / d _ {\mathrm {i n}}\right)} \tag {120}
$$
Finally, applying (120) to Lemma 29, we conclude the following: When  $\gamma = K_1K_2 / \sigma_{\mathrm{sp}}$ ,  $d_{\mathrm{sp}} = \sigma_{\mathrm{sp}}^2 /K_2^2$ , there exists  $\sigma_{\mathrm{sp1}}$ , such that for any  $\sigma_{\mathrm{sp}} \geqslant \sigma_{\mathrm{sp1}}$ , target accuracy of CL is at least 0.5 erfc  $\left(-L' \cdot \frac{\gamma}{\sqrt{2}\sigma_{\mathrm{sp}}}\right)$  and at most 0.5 erfc  $\left(-4L' \cdot \frac{\gamma}{\sqrt{2}\sigma_{\mathrm{sp}}}\right)$ , where  $L' = \frac{K_2^2K_1}{\sigma_{\mathrm{in}}^2(1 - 1 / d_{\mathrm{in}})}$ .
Comparison with ERM. Recall from Theorem 8 the performance of ERM classifier (trained from scratch) is 0.5  $\operatorname{erfc}\left(-\gamma^{2} / \sqrt{2d_{\mathrm{sp}}}\sigma_{\mathrm{sp}}\right)$ . The lower bound on the performance of classifier over CL representations is strictly better than ERM when:
$$
\begin{array}{l} \frac {\gamma}{\sqrt {d _ {\mathrm {s p}}}} <   L ^ {\prime} \\ \Longleftarrow \frac {K _ {2} ^ {2} K _ {1}}{\sigma_ {\mathrm {i n}} ^ {2} (1 - ^ {1} / d _ {\mathrm {i n}})} > \frac {\gamma}{\sqrt {d _ {\mathrm {s p}}}} \Longleftarrow \frac {K _ {2} ^ {2} K _ {1}}{\sigma_ {\mathrm {i n}} ^ {2} (1 - ^ {1} / d _ {\mathrm {i n}})} > \frac {K _ {1} K _ {2} ^ {2}}{\sigma_ {\mathrm {s p}} ^ {2}} \\ \Longleftarrow \sigma_ {\mathrm {s p}} > \sigma_ {\mathrm {i n}} \sqrt {1 - ^ {1} / d _ {\mathrm {i n}}} \Longleftarrow \sigma_ {\mathrm {s p} 1} > \sigma_ {\mathrm {i n}} \sqrt {1 - ^ {1} / d _ {\mathrm {i n}}}. \\ \end{array}
$$
This completes our proof of Corollary 6.
E.2 Analysis of STOC: Formal Statement of Theorem 7
Recall ERM solution over contrastive pretraining. We showed that without loss of generality when  $k$  (the output dimensionality of  $\Phi$ ) is greater than 2, we can restrict  $k$  to 2 and the  $\Phi$  can be denoted as  $[\phi_1,\phi_2]^\top$  where  $\phi_{1} = c_{1}w^{\star} + c_{3}w_{\mathrm{sp}}$  and  $\phi_{2} = c_{2}w^{\star} + c_{4}w_{\mathrm{sp}}$ . The ERM solution of the linear head is then given by  $h_1,h_2\in \mathbb{R}$ :
$$
h _ {1} = c _ {1} \cdot \gamma + c _ {3} \cdot \sqrt {d _ {\mathrm {s p}}}, \text {a n d} h _ {2} = c _ {2} \cdot \gamma + c _ {4} \cdot \sqrt {d _ {\mathrm {s p}}}. \tag {121}
$$
STOC performs self-training of the linear head over the CL solution. Before introducing the result, we need some additional notation. Let  $h^t$  denote the solution of the linear head at iterate  $t$ . Without loss of generality, assume that the coefficients in  $\phi_1 = c_1 w_{\mathrm{in}} + c_3 w_{\mathrm{sp}}$  and  $\phi_2 = c_2 w_{\mathrm{in}} + c_4 w_{\mathrm{sp}}$  are such that  $c_2$  is positive and  $c_1, c_3$ , and  $c_4$  are negative. Moreover, for simplicity of exposition, assume that  $|c_4| > |c_3|$ .
Theorem 16. Under the conditions of Corollary 15 and when  $\frac{\gamma^2}{\sigma_{\mathrm{sp}}} \geqslant \left[\frac{-c_3 - c_4}{(c_2 + c_1) \cdot |c_1|}\right] \vee \left[\frac{c_4}{c_1 \cdot c_2}\right]$ , the target accuracy of  $ST$  over  $CL$  is lower bounded by  $0.5 \cdot \operatorname{erfc} \left(-\left|c^2 / c_4\right| \cdot \gamma / (\sqrt{2}\sigma_2)\right) \geqslant 0.5 \cdot \operatorname{erfc} \left(-L \cdot \sqrt{d_{\mathrm{sp}}} / (\sqrt{2}\sigma_{\mathrm{sp}})\right)$  with  $L \geqslant 1$ .
Before proving Theorem 16, we first connect the condition  $\frac{\gamma^2}{\sigma_{\mathrm{sp}}} \geqslant \left[\frac{-c_3 - c_4}{(c_2 + c_1) \cdot |c_1|}\right] \vee \left[\frac{c_4}{c_1 \cdot c_2}\right]$  with the result obtained with contrastive learning.