SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Longleftrightarrow \left[ 2 r (\sigma_ {t}) + 2 \cdot R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \right] \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}} \leqslant 2 r ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {1}{\sigma_ {t}}\right) - R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t} ^ {2}}\right) \tag {44}
$$
$$
\Longleftrightarrow \left[ r (\sigma_ {t}) + R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {1}{\sigma_ {t}} \right] \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}} \leqslant r (\sigma_ {t}) - \frac {1}{\sigma_ {t}} - \frac {R ^ {\prime \prime}}{2} \left(\frac {\mu_ {t}}{\sigma_ {t} ^ {2}}\right) \tag {45}
$$
$$
\Longleftrightarrow \left[ R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \right] \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}} + \frac {R ^ {\prime \prime}}{2} \left(\frac {\mu_ {t}}{\sigma_ {t} ^ {2}}\right) \leqslant \left(\mathrm {r} (\sigma_ {t}) - \frac {1}{\sigma_ {t}}\right) \cdot \left(1 - \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}}\right) \tag {46}
$$
$$
\Longleftrightarrow \left[ R ^ {\prime \prime} \left(\frac {\mu_ {t} ^ {2}}{\sigma_ {t}}\right) \right] \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}} + \frac {R ^ {\prime \prime}}{2} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \left(\sigma_ {t} r (\sigma_ {t}) - 1\right) \cdot \left(1 - \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}}\right) \tag {47}
$$
Now, we will argue the monotonicity of LHS and RHS in (47). Observe that LHS is increasing in  $\mu_t$  and decreasing in  $\sigma_t$  and RHS is decreasing in  $\sigma_t$  as  $(\sigma_t \mathrm{r}(\sigma_t) - 1)$  is increasing (and the multiplier is negative). Moreover, if (47) holds true for maximum value of RHS and minimum of LHS, then we would have (36). Thus substituting  $\mu_t = \gamma$  and  $\sigma_t = \sigma_0$  in LHS and  $\sigma_t = \sigma_{\mathrm{sp}}$  in RHS, we get:
$$
\left[ R ^ {\prime \prime} \left(\frac {\gamma^ {2}}{\sigma_ {0}}\right) \right] \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}} + \frac {R ^ {\prime \prime}}{2} \left(\frac {\gamma}{\sigma_ {0}}\right) \leqslant \left(\sigma_ {\mathrm {s p}} \mathrm {r} \left(\sigma_ {\mathrm {s p}}\right) - 1\right) \cdot \left(1 - \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}}\right) \tag {48}
$$
$$
\Longleftrightarrow R ^ {\prime \prime} \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\sigma_ {0}} + \frac {R ^ {\prime \prime}}{2} \left(\frac {\gamma}{\sigma_ {0}}\right) \leqslant \left(\sigma_ {\mathrm {s p}} \mathrm {r} \left(\sigma_ {\mathrm {s p}}\right) - 1\right) \cdot \left(1 - \frac {\sigma_ {\mathrm {s p}} ^ {2}}{\gamma^ {2}}\right) \tag {49}
$$
Taking  $\gamma \leqslant \frac{1}{2\sqrt{\sigma_{\mathrm{sp}}}}$  and substituting  $R'' = \mathbf{r}''(\sigma_0)$ :
$$
(5 / 4) \cdot \mathrm {r} ^ {\prime \prime} (\sigma_ {0}) \cdot \sigma_ {\mathrm {s p}} \leqslant \left(\sigma_ {\mathrm {s p}} \mathrm {r} \left(\sigma_ {\mathrm {s p}}\right) - 1\right) \cdot \left(1 - 4 \cdot \sigma_ {\mathrm {s p}} ^ {3}\right) \tag {51}
$$
Analytically solving the above expression, we get that (51) is satisfied for all values of  $\sigma_{\mathrm{sp}} \geqslant 1$  when  $d_{\mathrm{sp}} \geqslant 1$ . For example, the expression in (51) is also satisfied for the problem parameter used in the running example of the main paper.
As a remark, we note that in the proof of Lemma 10, the conditions derived are loose because of the relaxations made to simply the proof. In principle, the proof (and hence the conditions) can be tightened by carefully propagating second-order terms (which depend on  $\sigma_t$ ) in (40).
Lemma 11 (Sufficiency conditions for ST). Define  $\alpha_{1}$  and  $\alpha_{2}$  as in (20) and (21) respectively. If  $\sigma_{\mathrm{sp}} \leqslant \gamma$ , then we have for all  $t$ :
$$
\alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2} \cdot \mu_ {t}}{\gamma^ {2}} \geqslant \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right). \tag {52}
$$
Proof. We upper bound and lower bound  $\alpha_{1}$  and  $\alpha_{2}$  by using the properties of  $\mathfrak{r}(\cdot)$ . Recall:
$$
\alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \left[ r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) - r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) \right]. \tag {53}
$$
$$
\alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \left[ r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) + r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) - \frac {2}{\sigma_ {t}} \right]. \tag {54}
$$
We now use Taylor's expansion on  $\mathrm{r}(\cdot)$  and we get:
$$
\mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) + \mathrm {r} ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {55}
$$
$$
\left. \right. \mathrm {r} \left(\sigma_ {t}\right) - \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) + \mathrm {r} ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \leqslant \mathrm {r} \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t}\right) - \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) + R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {56}
$$
where  $R'' = \mathrm{r}''(\sigma_0)$ . This is because  $\mathbf{r}''(\cdot)$  takes positive values and is a decreasing function in  $\sigma_t$  (refer to Lemma 21). We now lower bound  $\alpha_{1}(\mu_{t},\sigma_{t})$  and upper bound  $\alpha_{2}(\mu_{t},\sigma_{t})$ :
$$
\frac {\alpha_ {1} \left(\mu_ {t} , \sigma_ {t}\right)}{\sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right)} \leqslant 2 r ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \tag {57}
$$
$$
\frac {\alpha_ {2} \left(\mu_ {t} , \sigma_ {t}\right)}{\sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right)} \geqslant 2 \mathrm {r} (\sigma_ {t}) + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \tag {58}
$$
Substituting the lower bound and upper bound in (52) gives us the following as stricter a sufficient condition (i.e., (59) implies (52)):
$$
\left[ 2 \mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \right] \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2} \cdot \mu_ {t}}{\gamma^ {2}} \geqslant 2 \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \tag {59}
$$
$$
\Longleftrightarrow \left[ 2 r \left(\sigma_ {t}\right) + r ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \right] \geqslant 2 r ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \cdot \frac {\gamma^ {2}}{\sigma_ {\mathrm {s p}} ^ {2} \cdot \mu_ {t}} \tag {60}
$$
$$
\Longleftrightarrow 2 \mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} - 2 \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \cdot \frac {\gamma^ {2}}{\sigma_ {\mathrm {s p}} ^ {2} \cdot \mu_ {t}} \geqslant 0 \tag {61}
$$
$$
\Longleftrightarrow 2 \mathrm {r} (\sigma_ {t}) \cdot \sigma_ {t} + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} - 2 - 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \frac {\gamma^ {2}}{\sigma_ {\mathrm {s p}} ^ {2}} \geqslant 0 \tag {62}
$$
$$
\Longleftrightarrow 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} - 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \frac {\gamma^ {2}}{\sigma_ {\mathrm {s p}} ^ {2}} \geqslant 0 \tag {63}
$$
$$
\Longleftrightarrow \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} + 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \left[ 1 - \frac {\gamma^ {2}}{\sigma_ {\mathrm {s p}} ^ {2}} \right] \geqslant 0 \tag {64}
$$
Hence, when  $\left[1 - \frac{\gamma^2}{\sigma_{\mathrm{sp}}^2}\right] \leqslant 0$ , we have condition in (64) hold true as  $\mathbf{r}'(\sigma_t)$  is always negative. Hence, the condition  $\gamma \geqslant \sigma_{\mathrm{sp}}$  gives us the necessary condition.
E.1.1 Proof of Proposition 3
For convenience, we first restate the Proposition 3 which gives us a closed form solution for (3) when  $\rho = 0$ . Then, we provide the proof, focusing first on the case of  $k = 1$ , and then showing that extension to  $k > 1$  is straightforward and renders the final form in the proposition that follows.
Proposition 12 (Barlow Twins solution). The solution for (3) is  $U_k^\top \Sigma_{\mathsf{A}}^{-1/2}$  where  $U_k$  are the top  $k$  eigenvectors of  $\Sigma_{\mathsf{A}}^{-1/2} \widetilde{\Sigma} \Sigma_{\mathsf{A}}^{-1/2}$ . Here,  $\Sigma_{\mathsf{A}} := \mathbb{E}_{a \sim \mathrm{P}_{\mathsf{A}}} [aa^\top]$  is the covariance over augmentations, and  $\widetilde{\Sigma} := \mathbb{E}_{x \sim \mathrm{P}_{\mathsf{U}}} [\widetilde{a}(x)\widetilde{a}(x)^\top]$  is the covariance matrix of mean augmentations  $\widetilde{a}(x) := \mathbb{E}_{\mathrm{P}_{\mathsf{A}}(a|x)}[a]$ .
Proof. We will use  $\phi(x)$  to denote  $\phi^\top x$  where  $\phi \in \mathbb{R}^d$ . Throughout the proof, we use  $a$  to denote augmentation and  $x$  to denote the input. We will use  $\mathrm{P_A}(a \mid x)$  as the probability measure over the space of augmentations  $\mathcal{A}$ , given some input  $x \in \mathcal{X}$  (with corresponding density)  $p_{\mathrm{A}}(\cdot \mid x)$ . Next, we use  $p_{\mathrm{A}}(\cdot)$  to denote the density associate with the marginal probability measure over augmentations:  $\mathrm{P_A} = \int_{\mathcal{X}} \mathrm{P_A}(a \mid x) \, \mathrm{dP_U}$ . Finally, the joint distribution over positive pairs  $A_+(a_1, a_2) = \int_{\mathcal{X}} \mathrm{P_A}(a_1 \mid x) \mathrm{P_A}(a_2 \mid x) \, \mathrm{dP_U}$ , gives us the positive pair graph over augmentations.
Before we solve the optimization problem in (3) for  $\Phi \in \mathbb{R}^{k\times d}$  for any general  $k$ , let us first consider the case where  $k = 1$ , i.e. we only want to find a single linear projection  $\phi$ . The constraint  $\rho = 0$ , transfers onto  $\phi$  in the following way:
$$
\mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} [ \phi (a) ^ {2} ] = 1 \quad \equiv \quad \phi^ {\top} \Sigma_ {A} \phi = 1 \tag {65}
$$
Under the above constraint we want to minimize the invariance loss, which according to Lemma 22 is given by  $2\cdot \int_{\mathcal{A}}\phi (a)L(\phi)(a)\mathrm{d}\mathrm{P}_{\mathsf{A}}$  , where  $L(\phi)(\cdot)$  is the following linear operator.
$$
L (\phi) (a) = \phi (a) - \int_ {\mathcal {A}} \frac {A _ {+} \left(a , a ^ {\prime}\right)}{p _ {\mathrm {A}} (a)} \cdot \phi \left(a ^ {\prime}\right) \mathrm {d} a ^ {\prime}. \tag {66}
$$
Based on the definition of the operator, we can reformulate the constrained optimization for contrastive pretraining as:
$$
\underset {\phi : \phi^ {\top} \Sigma_ {A} \phi = 1} {\arg \min } \int_ {\mathcal {A}} \phi (a) \cdot L (\phi) (a) \mathrm {d P} _ {\mathbf {A}} \tag {67}
$$
$$
\begin{array}{l} \Rightarrow \underset {\phi : \phi^ {\top} \Sigma_ {A} \phi = 1} {\arg \min } \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} [ \phi (a) ^ {2} ] - \int_ {A} \int_ {A} \phi (a) \cdot \phi \left(a ^ {\prime}\right) \cdot A _ {+} \left(a, a ^ {\prime}\right) d a d a ^ {\prime} (68) \\ \Rightarrow \underset {\phi : \phi^ {\top} \Sigma_ {A} \phi = 1} {\arg \min } \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} [ \phi (a) ^ {2} ] - \int_ {\mathcal {X}} \int_ {\mathcal {A}} \int_ {\mathcal {A}} p _ {\mathrm {A}} (a | x) p _ {\mathrm {A}} \left(a ^ {\prime} \mid x\right) \cdot \phi (a) \phi \left(a ^ {\prime}\right) \mathrm {d P} _ {\cup} (69) \\ \Longrightarrow \underset {\phi : \phi^ {\top} \Sigma_ {A} \phi = 1} {\arg \min } \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} [ \phi (a) ^ {2} ] - \int_ {\mathcal {X}} [ \widetilde {\phi} (x) ] ^ {2} \mathrm {d P} _ {\cup}, (70) \\ \end{array}
$$
where  $\widetilde{\phi} (x) = \mathbb{E}_{a\sim \mathrm{P}_{\mathsf{A}}(\cdot |x)}\phi (x) = \mathbb{E}_{c\sim \mathrm{Unif}[0,1]^d}\big[\phi^\top (c\odot x)\big]$ . Note that,
$$
\begin{array}{l} \tilde {\phi} (x) ^ {2} = \left(\mathbb {E} _ {c \sim \operatorname {U n i f} [ 0, 1 ] ^ {d}} \left[ \phi^ {\top} (c \odot x) \right]\right) ^ {2} (71) \\ = \phi^ {\top} \left(\mathbb {E} _ {c \sim \operatorname {U n i f} [ 0, 1 ] ^ {d}} [ c \odot x ]\right) \left(\mathbb {E} _ {c \sim \operatorname {U n i f} [ 0, 1 ] ^ {d}} [ c \odot x ]\right) ^ {\top} \phi (72) \\ \Rightarrow \int_ {\mathcal {X}} \left[ \tilde {\phi} (x) \right] ^ {2} \mathrm {d P} _ {\cup} = \phi^ {\top} \tilde {\Sigma} \phi (73) \\ \end{array}
$$
Further, since  $\mathbb{E}_{a\sim \mathrm{P}_A}[\phi (a)^2 ] = \phi^\top \Sigma \phi$  we can now rewrite our main optimization problem for  $k = 1$  as:
$$
\begin{array}{l} \underset {\phi : \phi^ {\top} \Sigma_ {A} \phi = 1} {\arg \min } \quad \phi^ {\top} \Sigma_ {A} \phi - \phi^ {\top} \tilde {\Sigma} \phi (74) \\ = \underset {\phi : \phi^ {\top} \Sigma_ {A} \phi = 1} {\arg \max } \phi^ {\top} \tilde {\Sigma} \phi (75) \\ \end{array}
$$
Recall that in our setup both  $\widetilde{\Sigma}$  and  $\Sigma_{A}$  are positive definite and invertible matrices. To solve the above problem, let's consider a re-parameterization:  $\phi^{\prime} = \Sigma_{A}^{1 / 2}\phi$ , thus  $\phi^{\top}\Sigma_{A}\phi = 1$ , is equivalent to the constraint  $\| \phi^{\prime}\|_{2}^{2} = 1$ . Based on this re-parameterization we are now solving:
$$
\underset {\| \phi^ {\prime} \| _ {2} ^ {2} = 1} {\arg \max } \quad \phi^ {\prime} ^ {\top} \Sigma_ {A} ^ {- 1 / 2} \cdot \widetilde {\Sigma} \cdot \Sigma_ {A} ^ {- 1 / 2} \phi^ {\prime}, \tag {76}
$$
which is nothing but the top eigenvector for  $\Sigma_A^{-1/2} \cdot \widetilde{\Sigma} \cdot \Sigma_A^{-1/2}$ .
Now, to extend the above argument from  $k = 1$  to  $k > 1$ , we need to care of one additional form of constraint in the form of feature diversity:  $\phi_i^\top \Sigma_A \phi_j = 0$  when  $i \neq j$ . But, we can easily redo the reformulations above and arrive at the following optimization problem:
$$
\begin{array}{l l} \arg \max  _ {\left\| \phi_ {i} ^ {\prime} \right\| _ {2} ^ {2} = 1, \forall i} & \left[ \phi_ {1} ^ {\prime}, \phi_ {2} ^ {\prime}, \dots , \phi_ {k} ^ {\prime} \right] ^ {\top} \Sigma_ {A} ^ {- 1 / 2} \cdot \tilde {\Sigma} \cdot \Sigma_ {A} ^ {- 1 / 2} \left[ \phi_ {1} ^ {\prime}, \phi_ {2} ^ {\prime}, \dots , \phi_ {k} ^ {\prime} \right], \\ \phi_ {i} ^ {\prime \top} \phi_ {j} ^ {\prime} = 0, \forall i \neq j \end{array} \tag {77}
$$
where  $\phi_i' = \Sigma_A^{1/2} \phi_i$ . The above is nothing but the top  $k$  eigenvectors for the matrix  $\Sigma_A^{-1/2} \cdot \widetilde{\Sigma} \cdot \Sigma_A^{-1/2}$ . This completes the proof of Proposition 12.
E.1.2 Analysis with  $\rho > 0$  in Contrastive Pretraining Objective (3)
In (3) we considered the strict version of the optimization problem where  $\rho = 0$ . Here, we will consider the following optimization problem that we optimize for our experiments in the simplified setup:
$$
\mathcal {L} _ {\mathrm {c l}} (\Phi , \kappa) := \mathbb {E} _ {x \sim \mathrm {P} _ {\mathrm {U}}} \mathbb {E} _ {a _ {1}, a _ {2} \sim \mathrm {P} _ {\mathrm {A}} (\cdot | x)} \| \Phi (a _ {1}) - \Phi (a _ {2}) \| _ {2} ^ {2} + \kappa \cdot \| \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} [ \Phi (a) \Phi (a) ^ {\top} ] - \mathbf {I} _ {k} \| _ {F} ^ {2}, \tag {78}
$$
where  $\kappa > 0$  is some finite constant (note that every  $\rho$  corresponds to some  $\kappa$  and particularly  $\rho = 0$ , corresponds to  $\kappa = \infty$ ). Let  $\Phi^{\star}$  be the solution for (3) with  $\rho = 0$ , i.e. the solution described in Proposition 3. Now, we will show that in practice we can provably recover something close to  $\Phi^{\star}$  when  $\kappa$  is large enough.
Theorem 13 (Solution for (78) is approximately equal to  $\Phi^{\star}$ ). If  $\widehat{\Phi}$  is some solution that achieves low values of the objective  $\mathcal{L}_{\mathrm{cl}}(\Phi, \kappa)$  in (78), i.e.,  $\mathcal{L}_{\mathrm{cl}}(\widehat{\Phi}, \kappa) \leqslant \epsilon$ , then there exists a matrix  $W \in \mathbb{R}^{k \times k}$  such that:
$$
\mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \| W \cdot \Phi^ {\star} (a) - \widehat {\Phi} (a) \| _ {2} ^ {2} \leqslant \frac {k \epsilon}{2 \gamma_ {k + 1}},
$$
$$
w h e r e, \gamma_ {k + 1} \geqslant \frac {2 \gamma_ {1} ^ {2}}{k \epsilon} \cdot \left(1 - \sqrt {\frac {\epsilon}{\kappa}}\right) - \frac {\gamma_ {1}}{k},
$$
where  $\gamma_{k + 1}$  is the  $k + 1^{th}$  eigenvalue for  $\mathbf{I}_d - \Sigma_A^{-1 / 2}\tilde{\Sigma}\Sigma_A^{-1 / 2}$ . Here,  $\lambda_1\leqslant \lambda_2\leqslant \ldots \leqslant \lambda_d$
Proof. Since we know that  $\mathcal{L}_{\mathrm{cl}}(\widehat{\Phi},\kappa)\leqslant \epsilon$ , we can individually bound the invariance loss and the regularization term:
$$
\mathbb {E} _ {x \sim \mathrm {P} _ {\mathrm {U}}} \mathbb {E} _ {a _ {1}, a _ {2} \sim \mathrm {P} _ {\mathrm {A}} (\cdot | x)} \| \hat {\Phi} (a _ {1}) - \hat {\Phi} (a _ {2}) \| _ {2} ^ {2} \leqslant \epsilon \tag {79}
$$
$$
\left\| \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \hat {\Phi} (a) \hat {\Phi} (a) ^ {\top} \right] - \mathbf {I} _ {k} \right\| _ {F} ^ {2} \leqslant \frac {\epsilon}{\kappa} \tag {80}
$$
$$
\forall i \in [ k ]: 1 - \sqrt {\frac {\epsilon}{\kappa}} \leqslant \widehat {\phi} _ {i} ^ {\top} \Sigma_ {A} \widehat {\phi} _ {i} \leqslant 1 + \sqrt {\frac {\epsilon}{\kappa}} \tag {81}
$$
$$
\forall i \in [ k ]: \mathbb {E} _ {x \sim \mathrm {P} _ {\mathrm {U}}} \mathbb {E} _ {a _ {1}, a _ {2} \sim \mathrm {P} _ {\mathrm {A}} (\cdot | x)} \left(\hat {\phi} _ {i} ^ {\top} a _ {1} - \hat {\phi} _ {i} ^ {\top} a _ {2}\right) ^ {2} \leqslant \epsilon \tag {82}
$$
Let  $\phi_1^\star, \phi_2^\star, \phi_3^\star, \ldots, \phi_d^\star$  be the solution returned by the analytical solution for  $\rho = 0$ , i.e. the solution in Proposition 3. Now, since  $\Phi^\star$  would span  $\mathbb{R}^d$  when  $\Sigma_A$  is full rank, we can denote:
$$
\widehat {\phi} _ {i} = \sum_ {j = 1} ^ {d} \eta_ {i} ^ {(j)} \phi_ {j} ^ {\star} \tag {83}
$$
Now from Lemma 22, the invariance loss for  $\widehat{\phi}_i$  can be written using the operator  $L(\phi)(a) = \phi(a) - \int_{\mathcal{A}} \frac{A_+(a,a')}{p_A(a)} \phi(a') \, \mathrm{d}a'$ :
$$
\begin{array}{l} \operatorname {I n v a r i a n c e} \left. \operatorname {L o s s} \left(\widehat {\phi_ {i}}\right) := \mathbb {E} _ {x \sim \mathrm {P} _ {\mathrm {U}}} \mathbb {E} _ {a _ {1}, a _ {2} \sim \mathrm {P} _ {\mathrm {A}} (\cdot | x)} \left(\widehat {\phi_ {i} ^ {\top}} a _ {1} - \widehat {\phi_ {i} ^ {\top}} a _ {2}\right) ^ {2} \right. (84) \\ = 2 \cdot \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \widehat {\phi} _ {i} (a) L \left(\widehat {\phi} _ {i}\right) (a) \right] (85) \\ = 2 \cdot \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \left(\sum_ {j = 1} ^ {d} \eta_ {i} ^ {(j)} \phi_ {i} ^ {\star}\right) L \left(\sum_ {j = 1} ^ {d} \eta_ {i} ^ {(j)} \phi_ {j} ^ {\star}\right) (a) \right] (86) \\ = 2 \cdot \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \left(\sum_ {j = 1} ^ {d} \eta_ {i} ^ {(j)} \phi_ {j} ^ {\star}\right) \left(\sum_ {j = 1} ^ {d} \eta_ {i} ^ {(j)} L \left(\phi_ {j} ^ {\star}\right) (a)\right) \right] (87) \\ = 2 \cdot \sum_ {j = 1} ^ {d} \left(\eta_ {i} ^ {(j)}\right) ^ {2} \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \phi_ {j} ^ {\star} (a) L \left(\phi_ {j} ^ {\star}\right) (a) \right] (88) \\ + 2 \cdot \sum_ {m = 1, n = 1, m \neq n} ^ {d} \eta_ {i} ^ {(m)} \eta_ {i} ^ {(n)} \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \phi_ {m} ^ {\star} (a) L \left(\phi_ {n} ^ {\star}\right) (a) \right] (89) \\ \end{array}
$$
Since,  $\phi_i^\star (\cdot)$  are eigenfunctions of the operator  $L$  [38], we can conclude that:
$$
\sum_ {m = 1, n = 1, m \neq n} ^ {d} \eta_ {i} ^ {(m)} \eta_ {i} ^ {(n)} \mathbb {E} _ {a \sim \mathrm {P _ {A}}} [ \phi_ {m} ^ {\star} (a) L (\phi_ {n} ^ {\star}) (a) ] = 0,
$$
and if  $\gamma_1 \leqslant \gamma_2 \leqslant \gamma_3 \ldots \leqslant \gamma_d$  are the eigenvalues for  $\phi_1^\star, \phi_2^\star, \phi_3^\star, \ldots, \phi_d^\star$  under the decomposition of  $L(\phi)(\cdot)$  then:
$$
\mathbb {E} _ {x \sim \mathrm {P} _ {\mathrm {U}}} \mathbb {E} _ {a _ {1}, a _ {2} \sim \mathrm {P} _ {\mathrm {A}} (\cdot | x)} \left(\widehat {\phi_ {i} ^ {\top}} a _ {1} - \widehat {\phi_ {i} ^ {\top}} a _ {2}\right) ^ {2} = 2 \cdot \sum_ {j = 1} ^ {d} \gamma_ {j} \left(\eta_ {i} ^ {(j)}\right) ^ {2} \tag {90}
$$
Recall, we are also aware of a condition on the regularization term:  $1 - \sqrt{\frac{\epsilon}{\kappa}} \leqslant \widehat{\phi}_i^\top \Sigma_A \widehat{\phi}_i \leqslant 1 + \sqrt{\frac{\epsilon}{\kappa}}$ .
$$
\begin{array}{l} \widehat {\phi} _ {i} ^ {\top} \Sigma_ {A} \widehat {\phi} _ {i} = \left(\sum_ {j = 1} ^ {d} \eta_ {i} ^ {(j)} \phi_ {j} ^ {\star}\right) ^ {\top} \Sigma_ {A} \left(\sum_ {j = 1} ^ {d} \eta_ {i} ^ {(j)} \phi_ {j} ^ {\star}\right) = \sum_ {j = 1} ^ {d} \left(\eta_ {i} ^ {(j)}\right) ^ {2} (91) \\ \Rightarrow 1 - \sqrt {\frac {\epsilon}{\kappa}} \leqslant \sum_ {j = 1} ^ {d} \left(\eta_ {i} ^ {(j)}\right) ^ {2} \leqslant 1 + \sqrt {\frac {\epsilon}{\kappa}} \forall i. (92) \\ \end{array}
$$
In order to show that the projection of  $\widehat{\phi}_i$  on  $\Phi^*$  is significant, we need to argue that the term  $\sum_{j=k+1}^{d}\left(\eta_i^{(j)}\right)^2$  is small. The argument for this begins with the condition on invariance loss, and the fact that  $\gamma_1 \leqslant \gamma_2 \leqslant \ldots \leqslant \gamma_k \leqslant \gamma_{k+1} \leqslant \ldots \leqslant \gamma_d$ :
$$
\begin{array}{l} \frac {\epsilon}{2} \geqslant \sum_ {j = k + 1} ^ {d} \left(\eta_ {i} ^ {(j)}\right) ^ {2} \gamma_ {j} \geqslant \gamma_ {k + 1} \cdot \left(\sum_ {j = k + 1} ^ {d} \left(\eta_ {i} ^ {(j)}\right) ^ {2}\right) (93) \\ \Longrightarrow \sum_ {j = k + 1} ^ {d} \left(\eta_ {i} ^ {(j)}\right) ^ {2} \leqslant \frac {\epsilon}{2 \gamma_ {k + 1}} (94) \\ \end{array}
$$
Extending the above result  $\forall i$  by simply adding the bounds completes the claim of our first result in Theorem 13. Next, we will lower bound the eigenvalue  $\gamma_{k + 1}$ . Recall that,  $\sum_{j = 1}^{k}\left(\eta_i^{(j)}\right)^2\geqslant 1 - \sqrt{\frac{\epsilon}{\kappa}} -\frac{\epsilon}{2\gamma_{k + 1}}$ . Thus,
$$
\gamma_ {1} \cdot \left(1 - \sqrt {\frac {\epsilon}{\kappa}} - \frac {\epsilon}{2 \gamma_ {k + 1}}\right) \leqslant \sum_ {j = 1} ^ {k} \gamma_ {j} \left(\eta_ {i} ^ {(j)}\right) ^ {2} \leqslant k \gamma_ {k + 1} \cdot \frac {\epsilon}{2 \gamma_ {1}} \tag {95}
$$
We assume that all eigenvalues are strictly positive, which is true under our augmentation distribution. Given,  $\gamma_{k + 1}\geqslant \gamma_1$ , we can rearrange the above to get:
$$
\gamma_ {k + 1} \geqslant \frac {2 \gamma_ {1} ^ {2}}{k \epsilon} \cdot \left(1 - \sqrt {\frac {\epsilon}{\kappa}}\right) - \frac {\gamma_ {1}}{k} \tag {96}
$$
This completes the claim of our second result in Theorem 13.
E.1.3 Proof of Theorem 5