SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left[ 2 \mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \right] \cdot \frac {\sigma_ {\mathrm {s p}} \cdot \left(- c _ {4} - c _ {3}\right)}{\gamma \cdot \left(c _ {2} + c _ {1}\right)} \geqslant 2 \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \tag {174}
$$
$$
\Longleftrightarrow \left[ 2 r (\sigma_ {t}) + r ^ {\prime \prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \right] \geqslant 2 r ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \cdot \frac {\gamma \cdot (c _ {2} + c _ {1})}{\sigma_ {\mathrm {s p}} \cdot (- c _ {4} - c _ {3})} \tag {175}
$$
$$
\Longleftrightarrow 2 \mathrm {r} (\sigma_ {t}) + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} - 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \cdot \frac {\gamma \cdot \left(c _ {2} + c _ {1}\right)}{\sigma_ {\mathrm {s p}} \cdot \left(- c _ {4} - c _ {3}\right)} \geqslant 0 \tag {176}
$$
$$
\Longleftrightarrow 2 \mathrm {r} (\sigma_ {t}) \cdot \sigma_ {t} + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} - 2 - 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \mu_ {t} \cdot \frac {\gamma \cdot \left(c _ {2} + c _ {1}\right)}{\sigma_ {\mathrm {s p}} \cdot \left(- c _ {4} - c _ {3}\right)} \geqslant 0 \tag {177}
$$
$$
\Longleftrightarrow 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} - 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \mu_ {t} \cdot \frac {\gamma \cdot \left(c _ {2} + c _ {1}\right)}{\sigma_ {\mathrm {s p}} \cdot \left(- c _ {4} - c _ {3}\right)} \geqslant 0 \tag {178}
$$
$$
\Longleftrightarrow \mathrm {r} ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} + 2 \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left[ 1 - \mu_ {t} \cdot \frac {\gamma \cdot \left(c _ {2} + c _ {1}\right)}{\sigma_ {\mathrm {s p}} \cdot \left(- c _ {4} - c _ {3}\right)} \right] \geqslant 0 \tag {179}
$$
Thus, if we have  $\mu_t \geqslant \frac{\sigma_{\mathrm{sp}} \cdot (-c_4 - c_3)}{\gamma \cdot (c_2 + c_1)}$ , then (174) holds true which in-turn implies  $|\delta_2| \geqslant \delta_1$ . Plugging in  $\mu_t \geqslant \mu_0$ , we get the required condition.
E.3 Analysis for SSL
For SSL analysis, we argue that the projection learned by contrastive pretraining can significantly improve the generalization of the linear head learned on top, leaving little to no room for improvement for self-training. Our analysis leverages the margin-based bound for linear models from Kakade et al. [45]. Before introducing the result, we present some additional notation. Let  $\mathrm{Err}_D(w)$  denote 0-1 error of a classifier on a distribution  $D$ . Define 0-1 error with margin  $\gamma$  as  $\widehat{\operatorname{Err}}^{\gamma}(w) = \sum_{i=1}^{n} \frac{\mathbb{I}[y_i w^\top x_i \leqslant \gamma]}{n}$ .
Theorem 19 (Corollary 6 in Kakade et al. [45]). For all classifiers  $w$  and margin  $\gamma$ , we have with probability at least  $1 - \delta$ :
$$
\operatorname {E r r} _ {T} (w) \leqslant \widehat {\operatorname {E r r}} ^ {\gamma} (w) + 4 \frac {B}{\gamma} \sqrt {\frac {1}{n}} + \sqrt {\frac {\log (1 / \delta)}{n}} + \sqrt {\frac {\log \left(\log_ {2} (4 B / \gamma)\right)}{n}}, \tag {180}
$$
where  $B$  is an upper bound on the  $\ell_2$  norm of the input points  $x$ .
When  $\widetilde{\mathrm{Err}}^{\gamma}(w)$  is close to zero, the denominating term in RHS of (180) is  $4\frac{B}{\gamma}\sqrt{\frac{1}{n}}$ . With Proposition 3, CL solution  $\phi$  obtained on the target domain alone (for SSL setup) is  $w_{\mathrm{in}}$  when  $k = 1$ . For larger  $k$ 's the CL solution is dominated by the  $\phi_1 = w_{\mathrm{in}}$ . Thus, SSL mainly reduces the B on the projected data by reducing the dependency from order  $\sqrt{d}$  to 1 where 1 is the dimensionality of the output of  $\phi$  without altering the margin. Thus, we get a tighter upper bound for linear probing performed on top CL features when compared with linear probing done on inputs directly.
Intuitively, since the target data has only one predictive feature (along  $w_{\mathrm{in}}$ ), CL directly recovers this predictive feature as it is the predominant direction that minimizes invariance loss.
F Limitations of Prior Work
F.1 Contrastive learning analysis
Prior works that analyze contrastive learning show that minimizers of the CL objective recover clusters in the augmentation graph, which weights pairs of augmentations with their probability of being sampled as a positive pair [39, 11, 73, 44]. When there is no distribution shift in the downstream task, assumptions made on the graph in the form of consistency of augmentations with downstream labels, is sufficient to ensure that a linear probed head has good ID generalization. Under distribution shift, these assumptions are not sufficient and stronger ones are needed. E.g., some works assume that same-domain/class examples are weighted higher than cross-class cross-domain pairs [40, 76].
Using notation defined in [76], the assumption on the augmentation graph requires cross-class and same-domain weights  $(\beta)$  to be higher than cross-class and cross-domain weights  $(\gamma)$ . It is unclear if examples from different classes in the same domain will be "connected" if strong spurious features exist in the source domain and augmentations fail to mask them completely (e.g., image background may not be completely masked by augmentations but it maybe perfectly predictive of the label on source domain). In such cases, the linear predictor learnt over CL would fail to generalize OOD. In our toy setup as well, the connectivity assumption fails since on source  $x_{\mathrm{sp}}$  is perfectly predictive of the label and the augmentations are imperfect, i.e., augmentations do not mask  $x_{\mathrm{sp}}$  and examples of different classes do not overlap in source (i.e.,  $\beta = 0$ ). On the other hand, since  $x_{\mathrm{sp}}$  is now random on target, augmentations of different classes may overlap, i.e.,  $\gamma > 0$ , thus breaking the connectivity assumption. This is also highlighted in our empirical findings of CL furnishing representations that do not fully enable linear transferability from source to target (see Sec. 5). These empirical findings also call into question existing assumptions on data augmentations, highlighting that perfect linear transferability may not typically hold in practice. It is in this setting that we believe self-training can improve over contrastive learning by unlearning source-only features and improving linear transferability.
F.2 Self-training analysis
Some prior works on self-training view it as consistency regularization that constrain pseudolabels of original samples to be consistent with all their augmentations [12, 87, 79]. This framework
abstracts the role played by the optimization algorithm and instead evaluates the global minimizer of a population objective that enforces consistency of pseudolabels. In addition, certain expansion assumptions on class-conditional distributions are needed to ensure that pseudolabels have good accuracy on source and target domains. This framework does not account for challenges involved in propagating labels iteratively. For e.g., when augmentation distribution has long tails, the consistency of pseudolabels depends on the sampling frequency of "favorable" augmentations. As an illustration, consider our augmentation distribution in the toy setup in Sec. 4. If it were not uniform over dimensions, but instead something that was highly skewed, then a large number of augmentations need to be sampled for every data point to propagate pseudolabels successfully from source labeled samples to target unlabeled samples during self-training. This might hurt the performance of ST when we are optimizing for only finitely many iterations and over finitely many datapoints. This is why in our analysis we instead adopt the iterative analysis of self-training [17].
G Additional Lemmas
In this section we define some additional lemmas that we use in our theoretical analysis in E.
Lemma 20 (Upper bound and lower bounds on erfc; Kschischang [49]). Define  $\operatorname{erfc}(x) = \frac{2}{\sqrt{\pi}} \cdot \int_{x}^{\infty} \exp(-z^{2}) \cdot dz$ . Then we have:
$$
{\frac {2}{\sqrt {\pi}}} \cdot {\frac {\exp (- x ^ {2})}{x + \sqrt {x ^ {2} + 2}}} <   \operatorname {e r f c} (x) \leqslant {\frac {2}{\sqrt {\pi}}} \cdot {\frac {\exp (- x ^ {2})}{x + \sqrt {x ^ {2} + 4 / \pi}}}
$$
Lemma 21 (Properties of Mill's ratio [5]). Define the Mill's ratio as  $\mathrm{r}(x) = \exp \left(x^2 / 2\right) \cdot \operatorname{erfc}\left(x / \sqrt{2}\right) \cdot \sqrt{\pi / 2}$ . Then following assertions are true: (i)  $\mathrm{r}(x)$  is a strictly decreasing log-convex function; (ii)  $\mathrm{r}'(x) = x \cdot \mathrm{r}(x) - 1$  is an increasing function with  $\mathrm{r}'(x) < 0$  for all  $x$ ; (iii)  $\mathrm{r}''(x) = \mathrm{r}(x) + x^2 \cdot \mathrm{r}(x) - x$  is a decreasing function with  $\mathrm{r}''(x) > 0$  for all  $x$ ; (iv)  $x^2 \cdot \mathrm{r}'(x)$  is a decreasing function of  $x$ .
Lemma 22 (invariance loss as product with operator  $L$ ). The invariance loss for some  $\phi \in \mathbb{R}^d$  is given as:  $2 \cdot \int_{\mathcal{A}} \phi(a) \cdot L(\phi)(a) \, \mathrm{dP}_{\mathcal{A}}$  where the operator  $L$  is defined as:
$$
L (\phi) (a) = \phi (a) - \int_ {\mathcal {A}} \frac {A _ {+} (a , a ^ {\prime})}{p _ {\mathsf {A}} (a)} \cdot \phi (a ^ {\prime}) \mathrm {d} a ^ {\prime}
$$
Proof. The invariance loss for  $\phi$  is given by:
$$
\begin{array}{l} \mathbb {E} _ {x \sim \mathrm {P} _ {\mathsf {U}}} \mathbb {E} _ {a _ {1}, a _ {2} \sim \mathrm {P} _ {\mathsf {A}} (\cdot | x)} (a _ {1} ^ {\top} \phi - a _ {2} ^ {\top} \phi) ^ {2} = 2 \mathbb {E} _ {x \sim \mathrm {P} _ {\mathsf {U}}} \mathbb {E} _ {a \sim \mathrm {P} _ {\mathsf {A}} (\cdot | x)} \left[ \phi (a) ^ {2} \right] \\ - 2 \mathbb {E} _ {a _ {1}, a _ {2} \sim A _ {+} (\cdot , \cdot)} [ \phi (a _ {1}) \phi (a _ {2}) ] \tag {181} \\ \end{array}
$$
$$
\begin{array}{l} = 2 \cdot \int_ {\mathcal {A}} \phi (a) ^ {2} \mathrm {d P} _ {\mathcal {A}} - 2 \cdot \int_ {\mathcal {A}} \phi (a) \left(\int_ {\mathcal {A}} \frac {A _ {+} (a , a _ {2})}{p _ {\mathcal {A}} (a)} \cdot \phi (a _ {2}) \mathrm {d} a _ {2}\right) \mathrm {d P} _ {\mathcal {A}} (182) \\ = 2 \cdot \int_ {A} \phi (a) \cdot L (\phi) (a) d P _ {A} (183) \\ \end{array}
$$
Lemma 23. If  $\mathcal{W}$  is the space spanned by  $w_{\mathrm{in}}$  and  $w_{\mathrm{sp}}$ , and  $\mathcal{W}_{\perp}$  is the null space for  $\mathcal{W}$ , then for any  $u \in \mathcal{W}$  and any  $v \in \mathcal{W}_{\perp}$ , the covariance along these directions  $\mathbb{E}_{a \sim \mathrm{P}_{\mathsf{A}}} [a^{\top} uv^{\top} a] = 0$ .
Proof: We can write the covariance over augmentations after we break down the augmentation  $a$  into two projections:  $a = \Pi_{\mathcal{W}}(a) + \Pi_{\mathcal{W}_{\perp}}(a)$
$$
\begin{array}{l} \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ a ^ {\top} u v ^ {\top} a \right] = \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \left(u ^ {\top} \left(\Pi_ {\mathcal {W}} (a) + \Pi_ {\mathcal {W} _ {\perp}} (a)\right)\right) \left(v ^ {\top} \left(\Pi_ {\mathcal {W}} (a) + \Pi_ {\mathcal {W} _ {\perp}} (a)\right)\right) \right] (184) \\ = \mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \left(u ^ {\top} \Pi_ {\mathcal {W}} (a)\right) \left(v ^ {\top} \Pi_ {\mathcal {W} _ {\perp}} (a)\right) \right] (185) \\ = u ^ {\top} \left(\mathbb {E} _ {a \sim \mathrm {P} _ {\mathrm {A}}} \left[ \Pi_ {\mathcal {W}} (a) \Pi_ {\mathcal {W} _ {\perp}} (a) ^ {\top} \right]\right) v = 0 (186) \\ \end{array}
$$
where the last inequality follows from the fact that  $\mathbb{E}_{a\sim \mathrm{P}_{\mathsf{A}}}\left[\Pi_{\mathcal{W}}(a)\Pi_{\mathcal{W}_{\perp}}(a)^{\top}\right] = \mathbb{E}_{a\sim \mathrm{P}_{\mathsf{A}}}\left[\Pi_{\mathcal{W}}(a)\right]\mathbb{E}_{a\sim \mathrm{P}_{\mathsf{A}}}\left[\Pi_{\mathcal{W}_{\perp}}(a)\right]^{\top}$ , since the noise in the null space of  $\mathcal{W}$  is drawn independent of the component along  $\mathcal{W}$ , and furthermore the individual expectations evaluate to zero.
Lemma 24 (closed-form expressions for eigenvalues and eigenvectors of  $\Sigma_A, \tilde{\Sigma}$ ). For a  $2 \times 2$  real symmetric matrix  $\left[ \begin{array}{ll}a, & b\\ c, & d \end{array} \right]$  the eigenvalues  $\lambda_1, \lambda_2$  are given by the following expressions:
$$
\lambda_ {1} = \frac {(a + b + \delta)}{2}, \lambda_ {2} = \frac {(a + b - \delta)}{2},
$$
where  $\delta = \sqrt{4c^2 + (a - b)^2}$ . Further, the eigenvectors are given by  $U = \begin{bmatrix} \cos (\theta), & \sin (\theta) \\ \sin (\theta), & -\cos (\theta) \end{bmatrix}$ , where:
$$
\tan (\theta) = \frac {b - a + \delta}{2 c}.
$$
For full proof of these statements see [23]. Here, we will use these statements to arrive at closed form expressions for the eigenvalues and eigenvectors of  $\Sigma_A$ ,  $\tilde{\Sigma}$ .
Proof. We can now substitute the above formulae with  $a, b, c, d$  taken from the expressions of  $\Sigma_A$  and  $\widetilde{\Sigma}$ , to get the following values:  $\lambda_1, \lambda_2$  are the eigenvalues of  $\Sigma_A$ , with  $\alpha$  determining the corresponding eigenvectors  $[\cos(\alpha), \sin(\alpha)]$ ,  $[\sin(\alpha), -\cos(\alpha)]$ ; and  $\widetilde{\lambda}_1, \widetilde{\lambda}_2$  are the eigenvalues of  $\widetilde{\Sigma}$ , with  $\beta$  determining the corresponding eigenvectors:  $[\cos(\beta), \sin(\beta)]$ ,  $[\sin(\beta), -\cos(\beta)]$ .
$$
\begin{array}{l} \lambda_ {1} = \frac {1}{8} \left(\gamma^ {2} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right) + \frac {d _ {\mathrm {s p}}}{2} + \frac {2 \sigma_ {\mathrm {s p}} ^ {2}}{3} + \frac {1}{6} \right. \\ + \sqrt {\gamma^ {2} d _ {\mathrm {s p}} + \left(\left(\gamma^ {2} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right)\right) - \left(\frac {d _ {\mathrm {s p}}}{2} + \frac {2 \sigma_ {\mathrm {s p}} ^ {2}}{3} + \frac {1}{6}\right)\right) ^ {2}}) \tag {187} \\ \end{array}
$$
$$
\begin{array}{l} \lambda_ {2} = \frac {1}{8} \left(\gamma^ {2} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right) + \frac {d _ {\mathrm {s p}}}{2} + \frac {2 \sigma_ {\mathrm {s p}} ^ {2}}{3} + \frac {1}{6} \right. \\ - \sqrt {\gamma^ {2} d _ {\mathrm {s p}} + \left(\left(\gamma^ {2} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right)\right) - \left(\frac {d _ {\mathrm {s p}}}{2} + \frac {2 \sigma_ {\mathrm {s p}} ^ {2}}{3} + \frac {1}{6}\right)\right) ^ {2}}) \tag {188} \\ \end{array}
$$
$$
\widetilde {\lambda} _ {1} = \frac {1}{8} \left(\gamma^ {2} + \frac {d _ {\mathrm {s p}}}{2} + \frac {\sigma_ {\mathrm {s p}} ^ {2}}{2} + \sqrt {\gamma^ {2} d _ {\mathrm {s p}} + \left(\gamma^ {2} - \left(\frac {d _ {\mathrm {s p}}}{2} + \frac {\sigma_ {\mathrm {s p}} ^ {2}}{2}\right)\right) ^ {2}}\right) \tag {189}
$$
$$
\widetilde {\lambda} _ {2} = \frac {1}{8} \left(\gamma^ {2} + \frac {d _ {\mathrm {s p}}}{2} + \frac {\sigma_ {\mathrm {s p}} ^ {2}}{2} - \sqrt {\gamma^ {2} d _ {\mathrm {s p}} + \left(\gamma^ {2} - \left(\frac {d _ {\mathrm {s p}}}{2} + \frac {\sigma_ {\mathrm {s p}} ^ {2}}{2}\right)\right) ^ {2}}\right) \tag {190}
$$
$$
\begin{array}{l} \tan (\alpha) = \frac {1}{\gamma \sqrt {d _ {\mathrm {s p}}}} \left(\frac {d _ {\mathrm {s p}}}{2} + \frac {2 \sigma_ {\mathrm {s p}} ^ {2}}{3} + \frac {1}{6} - \left(\gamma^ {2} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right)\right)\right) \\ + \sqrt {\gamma^ {2} d _ {\mathrm {s p}} + \left(\left(\gamma^ {2} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right)\right) - \left(\frac {d _ {\mathrm {s p}}}{2} + \frac {2 \sigma_ {\mathrm {s p}} ^ {2}}{3} + \frac {1}{6}\right)\right) ^ {2}}) \tag {191} \\ \end{array}
$$
$$
\tan (\beta) = \frac {1}{\gamma \sqrt {d _ {\mathrm {s p}}}} \left(\frac {d _ {\mathrm {s p}}}{2} + \frac {\sigma_ {\mathrm {s p}} ^ {2}}{2} - \gamma^ {2} + \sqrt {\gamma^ {2} d _ {\mathrm {s p}} + \left(\gamma^ {2} - \left(\frac {d _ {\mathrm {s p}}}{2} + \frac {\sigma_ {\mathrm {s p}} ^ {2}}{2}\right)\right) ^ {2}}\right) \tag {192}
$$
Consider the subclass of problem parameters,  $d_{\mathrm{sp}} = z$ ,  $\gamma = K_1 / \sqrt{z}$  and  $\sigma_{\mathrm{sp}} = K_2\sqrt{z}$  for fixed constants  $K_{1},K_{2} > 0$  and some variable  $z > 0$ , which we can vary to give us different problem instances for our toy model in (6).
$$
\begin{array}{l} \lambda_ {1} = \frac {1}{8} \left(\frac {K _ {1} ^ {2}}{z} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right) + \frac {z}{2} + \frac {2 K _ {2} ^ {2} z}{3} + \frac {1}{6} \right. \\ + \sqrt {K _ {1} ^ {2} + \left(\left(\frac {K _ {1} ^ {2}}{z} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right)\right) - \left(\frac {z}{2} + \frac {2 K _ {2} ^ {2} z}{3} + \frac {1}{6}\right)\right) ^ {2}}\left. \right) \tag {193} \\ \end{array}
$$
$$
\begin{array}{l} \lambda_ {2} = \frac {1}{8} \left(\frac {K _ {1} ^ {2}}{z} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right) + \frac {z}{2} + \frac {2 K _ {2} ^ {2} z}{3} + \frac {1}{6} \right. \\ - \sqrt {K _ {1} ^ {2} + \left(\left(\frac {K _ {1} ^ {2}}{z} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right)\right) - \left(\frac {z}{2} + \frac {2 K _ {2} ^ {2} z}{3} + \frac {1}{6}\right)\right) ^ {2}}\left. \right) \tag {194} \\ \end{array}
$$
$$
\widetilde {\lambda} _ {1} = \frac {1}{8} \left(\frac {K _ {1} ^ {2}}{z} + \frac {z}{2} + \frac {K _ {2} ^ {2} z}{2} + \sqrt {K _ {1} ^ {2} + \left(\frac {K _ {1} ^ {2}}{z} - \left(\frac {z}{2} + \frac {K _ {2} ^ {2} z}{2}\right)\right) ^ {2}}\right) \tag {195}
$$
$$
\widetilde {\lambda} _ {2} = \frac {1}{8} \left(\frac {K _ {1} ^ {2}}{z} + \frac {z}{2} + \frac {K _ {2} ^ {2} z}{2} - \sqrt {K _ {1} ^ {2} + \left(\frac {K _ {1} ^ {2}}{z} - \left(\frac {z}{2} + \frac {K _ {2} ^ {2} z}{2}\right)\right) ^ {2}}\right) \tag {196}
$$
$$
\begin{array}{l} \tan (\alpha) = \frac {1}{K _ {1}} \left(\frac {z}{2} + \frac {2 K _ {2} ^ {2} z}{3} + \frac {1}{6} - \left(\frac {K _ {1} ^ {2}}{z} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right)\right) \right. \\ + \sqrt {K _ {1} ^ {2} + \left(\frac {K _ {1} ^ {2}}{z} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right) - \left(\frac {z}{2} + \frac {2 K _ {2} ^ {2} z}{3} + \frac {1}{6}\right)\right) ^ {2}}) \tag {197} \\ \end{array}
$$
$$
\tan (\beta) = \frac {1}{K _ {1}} \left(\frac {z}{2} + \frac {K _ {2} ^ {2} z}{2} - \frac {K _ {1} ^ {2}}{z} + \sqrt {K _ {1} ^ {2} + \left(\frac {K _ {1} ^ {2}}{z} - \left(\frac {z}{2} + \frac {K _ {2} ^ {2} z}{2}\right)\right) ^ {2}}\right) \tag {198}
$$
From Stewart [80], we can use the closed form expression for the singular vectors of a  $2 \times 2$  full rank asymmetric matrix  $\left[ \begin{array}{ll}a, & b\\ c, & d \end{array} \right]$ . The singular vectors are given by
$$
\left[ \begin{array}{c c} \cos \theta , & \sin \theta \\ \sin \theta , & - \cos \theta \end{array} \right],
$$
where,  $\tan (2\theta)$  is given by:
$$
\tan (2 \theta) = \frac {2 a c + 2 b d}{a ^ {2} + b ^ {2} - c ^ {2} - d ^ {2}}.
$$
Now, substituting the values in the expression from (100), we get singular vectors of the above form where  $\theta \in [0,\pi /2]$  satisfies:
$$
\theta = \frac {1}{2} \tan^ {- 1} \left(\frac {2 \tan (\beta - \alpha) ; \cdot \left(\widetilde {\lambda} _ {1} - \widetilde {\lambda} _ {2}\right) \cdot \sqrt {\lambda_ {1} \lambda_ {2}}}{\left(\lambda_ {2} \widetilde {\lambda} _ {1} - \lambda_ {1} \widetilde {\lambda} _ {2}\right) - \left(\lambda_ {1} \widetilde {\lambda} _ {1} - \lambda_ {2} \widetilde {\lambda} _ {2}\right) \cdot \tan^ {2} (\alpha - \beta)}\right) \tag {199}
$$
Lemma 25 (asymptotic behavior of  $\tau$  tan  $\theta$ ). For  $\gamma = K_1 / \sqrt{z}$ ,  $\sigma_{\mathrm{sp}} = K_2\sqrt{z}$
$$
\lim  _ {z \rightarrow \infty} \tau \tan \theta = \frac {K _ {1} K _ {2} ^ {2}}{\left(1 + K _ {2} ^ {2}\right) 2 \sigma_ {\mathrm {i n}} ^ {2} \left(1 - 1 / d _ {\mathrm {i n}}\right)}
$$
Proof. In order to determine the asymptotic nature of  $\tan (\theta)$  as  $z\to \infty$  , we take the limit of a slightly different term first, since we have the closed form expression of  $\tan (2\theta)$
$$
\begin{array}{l} \lim  _ {z \rightarrow \infty} \tau \tan (2 \theta) = \sqrt {\frac {\lambda_ {1}}{\lambda_ {2}}} \cdot \frac {2 \tan (\alpha - \beta) \cdot \left(\tilde {\lambda} _ {1} / \tilde {\lambda} _ {2} - 1\right)}{\left(\tilde {\lambda} _ {1} / \tilde {\lambda} _ {2} - \lambda_ {1} / \lambda_ {2}\right) - \left(\lambda_ {1} \tilde {\lambda} _ {1} / \lambda_ {2} \tilde {\lambda} _ {2} - 1\right) \cdot \tan^ {2} (\alpha - \beta)} \\ = 2 \tan (\alpha - \beta) \cdot \frac {\widetilde {\lambda_ {1}} / \widetilde {\lambda_ {2}} - 1}{\widetilde {\lambda_ {1}} / \widetilde {\lambda_ {2}} \cdot \lambda_ {2} / \lambda_ {1} - 1}, \\ \end{array}
$$
since it is easy to see that  $\lim_{z\to \infty}\tan^2 (\alpha -\beta)\cdot \left(\frac{\lambda_1\widetilde{\lambda_1}}{\lambda_2\widetilde{\lambda_2}} -1\right) = 0.$
If we use  $\tan (\alpha -\beta) = \frac{\tan\alpha - \tan\beta}{1 + \tan\alpha\tan\beta}$ , and substitute the functions of  $z$ , for all the quantities in the above expression using Lemma 24, we derive:  $\lim_{z\to \infty}\tau \tan 2\theta = {}^{2}K_{1}K_{2}^{2} / (1 + K_{2}^{2})2\sigma_{\mathrm{in}}^{2}(1^{-1} / d_{\mathrm{in}})$ .
Since  $\tau \to \infty$ ,  $\tan(2\theta) \to 0$ , and further from Taylor approximation of  $\tan(2\theta)$ ,  $\tan(2\theta) \to 2\theta$ . We can use this to derive the limit for  $\tau$ $\tan\theta$ , which would just be  $1/2 \cdot 2K_1K_2^2/(1 + K_2^2)2\sigma_{\mathrm{in}}^2(1 - 1/d_{\mathrm{in}}) = K_1K_2^2/(1 + K_2^2)2\sigma_{\mathrm{in}}^2(1 - 1/d_{\mathrm{in}})$ .
Lemma 26 (asymptotic behaviors of cot  $\alpha$ , tan  $\theta$ ). For  $\gamma = K_1 / \sqrt{z}$ ,  $\sigma_{\mathrm{sp}} = K_2\sqrt{z}$  following the expressions in Lemma 24,
$$
\lim  _ {z \to \infty} \cot \alpha = 0, \qquad \lim  _ {z \to \infty} \tan \theta = 0.
$$
Proof. For  $\tan \theta$ , since  $\tau \to \infty$ , and  $\tau$ $\tan \theta$  approaches a constant (from Lemma 25), we conclude  $\lim_{z\to \infty}\tan \theta = 0$ . For  $\cot \alpha$ ,
$$
\lim  _ {z \rightarrow \infty} \frac {z}{2} + \frac {2 K _ {2} ^ {2} z}{3} + \frac {1}{6} - \left(\frac {K _ {1} ^ {2}}{z} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right)\right) = \infty ,
$$
$$
\lim  _ {z \rightarrow \infty} \sqrt {K _ {1} ^ {2} + \left(\frac {K _ {1} ^ {2}}{z} \left(1 + \frac {1}{3 d _ {\mathrm {i n}}}\right) + \frac {\sigma_ {\mathrm {i n}} ^ {2}}{3} \left(1 - \frac {1}{d _ {\mathrm {i n}}}\right) - \left(\frac {z}{2} + \frac {2 K _ {2} ^ {2} z}{3} + \frac {1}{6}\right)\right) ^ {2}} = \infty .
$$
Thus,  $\cot \alpha \to 0$
Lemma 27 (asymptotic behavior of  $z$  cot  $\alpha$ ). For  $\gamma = K_1 / \sqrt{z}$ ,  $\sigma_{\mathrm{sp}} = K_2\sqrt{z}$  following the expressions in Lemma 24,