SlowGuess/NeurIPS_2023_OCR huggingface dataset:
fixed for CL and STOC. Then, we investigate trends when we continue to propagate gradients onto  $\Phi$  during STOC (we call this full-finetuning). Unlike previous cases, this allows features to be updated.
Results with non-linear feature extractor  $\Phi$ . In Fig. 7 we plot the performance of the four methods when we use a non-linear feature extractor during contrastive pretraining. This feature extractor is a one-hidden layer neural network (hidden dimension is 500) with ReLU activations. We find that the trends observed with linear backbones in Fig. 6 are also replicated with the nonlinear one. Specifically, we note that STOC improves over CL under distribution shifts, whereas CL is already close to optimal when there are no distribution shifts. We also see that CL and ST individually are subpar. In SSL, we see a huge drop in the performance of ST (over ERM) mainly because we only fit on pseudolabels during ST. This is different from practice where we continue to optimize loss on labeled data points while fitting the pseudolabels. Consequently, when we continue to optimize performance on source labeled data the performance of ST in SSL setup is improves from  $51.1\% \rightarrow 72.6\%$ .
Results with full fine-tuning. Up till this point, we have only considered the case (for both SSL and UDA) where we fix the contrastive learned features when running CL and STOC, i.e., we only optimized the linear head  $h$ . Now, we shall consider the setting where gradients are propagated to  $\Phi$  during STOC. Note that we still fix the representations for training the linear head during CL. Results for this setting are in Figure 8. We show two interesting trends that imitate real world behaviors.
STOC benefits from augmentations during full-finetuning: In the UDA setup we find that ST while updating  $\Phi_{\mathrm{cl}}$  can hurt due to overfitting issues when training with the finite sample of labeled and unlabeled data (drop by  $>7\%$  over CL). This is due to overfitting on confident but incorrect pseudolabels on target data. This can exacerbate components along spurious feature  $w_{\mathrm{sp}}$  from source. One reasoning behind this is that deep neural networks can perfectly memorize them on finite unlabeled target data [94]. Heuristics typically used in practice (e.g. in FixMatch [79]) help avoid overfitting on incorrect pseudolabels: (i) confidence thresholding; to pick confident pseudolabel examples; (ii) pseudolabel a different augmented input than the one on which the self-training loss is optimized; and (iii) optimize source loss with labeled data simultaneously when fitting pseudolabels. Intuitively, thresholding introduces a curriculum where we only learn confident examples in the beginning whose pseudolabels are mainly determined by component along the invariant feature  $w_{\mathrm{in}}$ . Augmentations prevent the neural network from memorizing incorrect pseudolabels and optimizing source loss prevents forgetting of features learned during CL. When we implement these during full-finetuning in STOC we see that STOC now improves over CL (by  $>20\%$ ).
Can we improve contrastive pretraining features during STOC? We find that self-training can also improve features learned during contrastive pretraining when we update the full backbone during STOC (see Figure 8(right)). Specifically, in the SSL setup we find that STOC can now improve substantially over CL. Recall, that when we fixed  $\Phi_{\mathrm{cl}}$  this was not possible (see E.3 and Fig. 2(b)). This is mainly because STOC can now improve performance beyond just recovering the generalization gap for the linear head (which is typically small). This feature improvement is observed even when we fully finetune a linear feature extractor. Similar trends are also observed with the non-linear backbone. But, it becomes harder to identify a good stopping criterion for CL training. Thus, it remains unclear if STOC and CL have complementary benefits for feature learning in UDA or SSL settings. Investigating this is an interesting avenue for future work.
E Formal Statements from Sec. 4
Recall from Section 2 that we learn linear classifiers  $h$  over features extractors  $\Phi$ . We consider linear feature extractor i.e.  $\Phi$  is a matrix in  $\mathbb{R}^{d \times k}$  and the linear layer  $h: \mathbb{R}^k \to \mathbb{R}$  with a prediction as  $\operatorname{sgn}(h^\top \Phi x)$ . We use the exponential loss  $\ell(f(x), y) = \exp(-y f(x))$ .
E.1 Analysis of ERM and ST: Formal Statement of Theorem 2
For ERM and ST, we train both  $h$  and  $\Phi$ . This is equivalent to  $\Phi = I_{d\times d}$  being identity and training a linear head  $h$ . Recall that the ERM classifier is obtained by minimizing the population loss on labeled source data:
$$
h _ {\mathrm {E R M}} = \underset {h} {\arg \min } \mathbb {E} _ {(x, y) \sim \mathrm {P} _ {\mathrm {S}}} [ \ell (x, y) ]. \tag {10}
$$
We split Theorem 2 into Theorem 8 and Theorem 9. Before we characterize the ERM solution, we recall some additional notation. Define  $w_{\mathrm{in}} = [w^{\star},0,\dots,0]^\top$ , and  $w_{\mathrm{sp}} = [0,\dots,0,\mathbf{1}_{d_{\mathrm{sp}}} / \sqrt{d_{\mathrm{sp}}}]^\top$ . The following proposition characterizes  $h_{\mathrm{ERM}}$  and 0-1 error of the classifier on target:
Theorem 8 (ERM classifier and its error on target). ERM classifier obtained as in (10) is given by
$$
\frac {h _ {E R M}}{\| h _ {E R M} \| _ {2}} = \frac {\gamma \cdot w _ {\mathrm {i n}} + \sqrt {d _ {\mathrm {s p}}} \cdot w _ {\mathrm {s p}}}{\sqrt {\gamma^ {2} + d _ {\mathrm {s p}}}}.
$$
The target accuracy of  $h_{ERM}$  is given by 0.5 · erf(c  $(-\gamma^2 / (\sqrt{2d_{\mathrm{sp}}} \cdot \sigma_{\mathrm{sp}}))$ ).
Proof. To prove this theorem, we first derive a closed-form expression for the ERM classifier and then use Lemma 29 to derive its 0-1 error on target. For Gaussian data with the same covariance matrices for class conditional  $\mathrm{P_S}(x|y = 1)$  and  $\mathrm{P_S}(x|y = 0)$ , Bayes decision rule is given by the Fisher's linear discriminant direction (Chapter 4; Bishop [9]):
$$
h (x) = \left\{ \begin{array}{l l} 1, & \text {i f} h ^ {\top} x > 0 \\ 0, & \text {o t h e r w i s e} \end{array} \right.
$$
where  $h = 2 \cdot \gamma (w_{\mathrm{in}}) + 2 \cdot \sqrt{d_{\mathrm{sp}}} (w_{\mathrm{sp}})$ . Plugging  $h$  in Lemma 29 we get the desired result.
ST performs ERM in the first stage using labeled data from the source, and then subsequently updates the head  $h$  by iteratively generating pseudolabels on the unlabeled target:
$$
\mathcal {L} _ {\mathrm {s t}} (h) := \mathbb {E} _ {\mathrm {P} _ {\top} (x)} \ell \left(h ^ {\top} x, \operatorname {s g n} \left(h ^ {\top} x\right)\right). \tag {11}
$$
Starting with  $h_{\mathrm{ST}}^{0} = h_{\mathrm{ERM}} / \| h_{\mathrm{ERM}}\|_{2}$  (the classifier obtained with ERM) we perform the following iterative procedure for self-training:
$$
h _ {\mathrm {S T}} ^ {t + 1} = \frac {h _ {\mathrm {S T}} ^ {t} - \eta \nabla_ {h} \mathcal {L} _ {\mathrm {s t}} \left(h _ {\mathrm {S T}} ^ {t}\right)}{\left\| h _ {\mathrm {S T}} ^ {t} - \eta \nabla_ {h} \mathcal {L} _ {\mathrm {s t}} \left(h _ {\mathrm {S T}} ^ {t}\right) \right\| _ {2}} \tag {12}
$$
Next, we characterize ST solution:
Theorem 9 (ST classifier and its error on target). Starting with ERM solution, ST will lead to:
(i) (Necessary condition)  $h_{ST}^{t} = w_{\mathrm{sp}}$  as  $t \to \infty$ , such that the target accuracy is  $50\%$  for all  $\sigma_{\mathrm{sp}} \geqslant 1$  and  $\gamma \leqslant \frac{1}{2\sqrt{\sigma_{\mathrm{sp}}}}$ .
(ii) (Sufficient condition)  $h_{ST}^{t} = w_{\mathrm{in}}$  as  $t \to \infty$ , such that the target accuracy is  $100\%$  when the problem parameters  $\gamma, \sigma_{\mathrm{sp}}$  satisfy:  $\gamma \geqslant \sigma_{\mathrm{sp}}$ .
Proof. The proof can be divided into two parts: (i) deriving closed-form expressions for updates on  $h_{\mathrm{ST}}^t$  in terms of  $h_{\mathrm{ST}}^{t - 1}$  and (ii) obtaining conditions under which the component along  $w_{\mathrm{in}}$  monotonically increases or decreases with  $t$  after re-normalizing the norm of updated  $h$ . For notation convenience, we denote  $h_{\mathrm{ST}}$  with  $h$  in the rest of the proof.
Part-1. First, the loss of self-training with classifier  $h := [h_{\mathrm{in}}, h_{\mathrm{sp}}]$  where  $h_{\mathrm{in}} \in \mathbb{R}^{d_{\mathrm{in}}}$  and  $h_{\mathrm{sp}} \in \mathbb{R}^{d_{\mathrm{sp}}}$  is given by:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {s t}} (h) = \mathbb {E} _ {\mathrm {P} _ {\mathrm {T}} (x)} \left[ \ell \left(h ^ {\top} x, \operatorname {s g n} \left(h ^ {\top} x\right)\right) \right] (13) \\ = \mathbb {E} _ {\mathrm {P} _ {\top} (x)} \left[ \exp \left(- \operatorname {s i g n} \left(h ^ {\top} x\right) \cdot \left(h ^ {\top} x\right)\right) \right] (14) \\ = \mathbb {E} _ {\mathrm {P} _ {\top} (x)} \left[ \exp \left(- | h ^ {\top} x |\right) \right] (15) \\ = \mathbb {E} _ {\mathrm {P} _ {\top} (x)} \left[ \exp \left(- \left| h _ {\text {i n}} ^ {\top} x _ {\text {i n}} + h _ {\mathrm {s p}} ^ {\top} x _ {\mathrm {s p}} \right|\right) \right] (16) \\ = \mathbb {E} _ {y \sim U \{- 1, 1 \}, z \sim \mathcal {N} (0, 1)} \left[ \exp \left(- | \gamma \cdot y \cdot h _ {\mathrm {i n}} ^ {\top} w ^ {\star} \right. \right. \\ \left. + \left[ \sigma_ {\mathrm {i n}} \left(\left\| h _ {\mathrm {i n}} \right\| _ {2} ^ {2} - \left(h _ {\mathrm {i n}} ^ {T} w ^ {\star}\right) ^ {2}\right) + \sigma_ {\mathrm {s p}} \cdot \left\| h _ {\mathrm {s p}} \right\| _ {2} \right] \cdot z \Bigg |\right) \Bigg ]. (17) \\ = \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ \exp \left(- \left| \gamma \cdot h _ {\mathrm {i n}} ^ {\top} w ^ {\star} + \left[ \sigma_ {\mathrm {i n}} \left(\left\| h _ {\mathrm {i n}} \right\| _ {2} ^ {2} - \left(h _ {\mathrm {i n}} ^ {T} w ^ {\star}\right) ^ {2}\right) + \sigma_ {\mathrm {s p}} \cdot \left\| h _ {\mathrm {s p}} \right\| _ {2} \right] \cdot z \right|\right) \right], (18) \\ \end{array}
$$
where (16) to (17) is implied by simply replacing the definition of target distribution and (17) to (18) is implied by the symmetry of the function with respect to  $y$  and  $-y$  due to the symmetry of the absolute function and Gaussian distribution. For a classifier  $h^t$ , we denote  $\mu_t = \gamma \cdot h_{\mathrm{in}}^T w^\star$  and  $\sigma_t = \left[\sigma_{\mathrm{in}}\left(\| h_{\mathrm{in}}^t\|_2^2 - \left(h_{\mathrm{in}}^T w^\star\right)^2\right) + \sigma_{\mathrm{sp}} \cdot \| h_{\mathrm{sp}}^t\|_2\right]$ . With this notation, we can re-write the loss in (18) as  $\mathcal{L}_{\mathrm{st}}(h^t) = \mathbb{E}_{z \sim \mathcal{N}(0, \sigma_t^2)}\left[\exp \left(-|\mu_t + z|\right)\right]$ .
Now we derive a closed-form expression of  $\mathcal{L}_{\mathrm{st}}(h^t)$  in Lemma 30:
$$
\mathcal {L} _ {\mathrm {s t}} \left(h ^ {t}\right) = \frac {1}{2} \left(\exp \left(\frac {\sigma_ {t} ^ {2}}{2} - \mu_ {t}\right) \cdot \operatorname {e r f c} \left(- \frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) + \exp \left(\frac {\sigma_ {t} ^ {2}}{2} + \mu_ {t}\right) \cdot \operatorname {e r f c} \left(\frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right)\right). \tag {19}
$$
Define the Mill's ratio as  $\mathrm{r}(x) = \exp \left(x^2 / 2\right) \cdot \operatorname{erfc} \left(x / \sqrt{2}\right) \cdot \sqrt{\pi / 2}$  as in Baricz [5]. We will frequently use standard properties of the Mill's ratio. We list them in Lemma 21 for completeness. Define:
$$
\begin{array}{l} \alpha_ {1} (\mu_ {t}, \sigma_ {t}) = - \exp \left(\frac {\sigma_ {t} ^ {2}}{2} - \mu_ {t}\right) \cdot \mathrm {e r f c} \left(- \frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) + \exp \left(\frac {\sigma_ {t} ^ {2}}{2} + \mu_ {t}\right) \cdot \mathrm {e r f c} \left(\frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right), \\ = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \left[ r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) - r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) \right] \tag {20} \\ \end{array}
$$
$$
\begin{array}{l} \alpha_ {2} (\mu_ {t}, \sigma_ {t}) = \exp \left(\frac {\sigma_ {t} ^ {2}}{2} - \mu_ {t}\right) \cdot \mathrm {e r f c} \left(- \frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) + \exp \left(\frac {\sigma_ {t} ^ {2}}{2} + \mu_ {t}\right) \cdot \mathrm {e r f c} \left(\frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) \\ - \frac {2 \sqrt {2}}{\sigma_ {t} \sqrt {\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \\ = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \left[ r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) + r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) - \frac {2}{\sigma_ {t}} \right]. \tag {21} \\ \end{array}
$$
Let  $\widetilde{h}^{t + 1}$  denote the un-normalized gradient descent update at iterate  $t + 1$ . We have:
$$
\widetilde {h} ^ {t + 1} = h ^ {t} - \eta \cdot \frac {\partial \mathcal {L} _ {\mathrm {s t}} \left(h ^ {t}\right)}{\partial h}. \tag {22}
$$
Now we will individually argue about the update of  $\widetilde{h}^{t + 1}$  along the first  $d_{\mathrm{in}}$  dimensions and the last  $d_{\mathrm{sp}}$  dimensions. First, we have:
$$
\begin{array}{l} \widetilde {h} _ {\mathrm {i n}} ^ {t + 1} = h _ {\mathrm {i n}} ^ {t} - \eta \cdot \frac {\partial \mathcal {L} _ {\mathrm {s t}} (h ^ {t})}{\partial h _ {\mathrm {i n}}} \\ = h _ {\mathrm {i n}} ^ {t} - \frac {\eta}{2} \left(- \exp \left(\frac {\sigma_ {t} ^ {2}}{2} - \mu_ {t}\right) \cdot \operatorname {e r f c} \left(- \frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) \right. \\ \left. + \exp \left(\frac {\sigma_ {t} ^ {2}}{2} + \mu_ {t}\right) \cdot \operatorname {e r f c} \left(\frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right)\right) \cdot \gamma \cdot w ^ {\star} \\ - \frac {\eta}{2} \left(\exp \left(\frac {\sigma_ {t} ^ {2}}{2} - \mu_ {t}\right) \cdot \operatorname {e r f c} \left(- \frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) \right. \\ + \exp \left(\frac {\sigma_ {t} ^ {2}}{2} + \mu_ {t}\right) \cdot \operatorname {e r f c} \left(\frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) \\ - \frac {2 \sqrt {2}}{\sigma_ {t} \sqrt {\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \cdot (2 h _ {\mathrm {i n}} ^ {t} - 2 (h _ {\mathrm {i n}} ^ {t} \mathbin {\text {\lower 1 . 0 p t}} w ^ {\star}) w ^ {\star}) \cdot \sigma_ {\mathrm {i n}} ^ {2} \\ = h _ {\text {i n}} ^ {t} - \frac {\eta}{2} \cdot \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \gamma \cdot w ^ {\star} - \frac {\eta}{2} \cdot \alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \left(2 h _ {\text {i n}} ^ {t} - 2 \left(h _ {\text {i n}} ^ {t} \top w ^ {\star}\right) w ^ {\star}\right) \cdot \sigma_ {\text {i n}} ^ {2}. \tag {23} \\ \end{array}
$$
Notice that the update of  $h_{\mathrm{in}}^{t + 1}$  is split into two components, one along  $w^{\star}$  and the other along the orthogonal component  $2h_{\mathrm{in}}^{t} - 2(h_{\mathrm{in}}^{t}\top w^{\star})w^{\star}$ . We will now argue that since at initialization, the component along  $(I - w^{\star}w^{\star \top})$  is zero then it will remain zero. In particular, we have:
$$
h _ {\text {i n}} ^ {0} ^ {\top} \left(I - w ^ {\star} w ^ {\star \top}\right) \propto w ^ {\star \top} \left(I - w ^ {\star} w ^ {\star \top}\right) = 0. \tag {24}
$$
With (23), we can argue that if  $(I - w^{\star}w^{\star \top})h_{\mathrm{in}}^{t} = 0$ , then  $(I - w^{\star}w^{\star \top})\widetilde{h}_{\mathrm{inv}}^{t + 1} = 0$  implying that  $(I - w^{\star}w^{\star \top})\widetilde{h}_{\mathrm{in}}^{t} = 0$  for all  $t > 0$ . Hence, we have:
$$
\begin{array}{l} \widetilde {h} _ {\mathrm {i n v}} ^ {t + 1} = h _ {\mathrm {i n}} ^ {t} - \eta \cdot \frac {\partial \mathcal {L} _ {\mathrm {s t}} (h ^ {t})}{\partial h _ {\mathrm {i n}}} \\ = h _ {\text {i n}} ^ {t} - \frac {\eta}{2} \cdot \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \gamma \cdot w ^ {\star}. \tag {25} \\ \end{array}
$$
Second, we have the update  $\widetilde{h}_{\mathrm{sp}}^{t + 1}$  given by:
$$
\begin{array}{l} \widetilde {h} _ {\mathrm {s p}} ^ {t + 1} = h _ {\mathrm {s p}} ^ {t} - \eta \cdot \frac {\partial \mathcal {L} _ {\mathrm {s t}} (h ^ {t})}{\partial h _ {\mathrm {s p}}} \\ = h _ {\mathrm {s p}} ^ {t} - \frac {\eta}{2} \left(\exp \left(\frac {\sigma_ {t} ^ {2}}{2} - \mu_ {t}\right) \cdot \operatorname {e r f c} \left(- \frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) \right. \\ + \exp \left(\frac {\sigma_ {t} ^ {2}}{2} + \mu_ {t}\right) \cdot \operatorname {e r f c} \left(\frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) - \frac {2 \sqrt {2}}{\sigma_ {t} \sqrt {\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right)) \cdot h _ {\mathrm {s p}} ^ {t} \cdot \sigma_ {\mathrm {s p}} ^ {2} \\ = h _ {\mathrm {s p}} ^ {t} - \frac {\eta}{2} \cdot \alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) \cdot h _ {\mathrm {s p}} ^ {t} \cdot \sigma_ {\mathrm {s p}} ^ {2}. \tag {26} \\ \end{array}
$$
Re-writing the expressions (25) and (26) for the update of  $\widetilde{h}^{t + 1}$ , we have:
$$
\widetilde {h} _ {\text {i n}} ^ {t + 1} = h _ {\text {i n}} ^ {t} \left(1 - \frac {\eta}{2} \cdot \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \gamma^ {2} / \mu_ {t}\right). \tag {27}
$$
$$
\widetilde {h} _ {\mathrm {s p}} ^ {t + 1} = h _ {\mathrm {s p}} ^ {t} \left(1 - \frac {\eta}{2} \cdot \alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \sigma_ {\mathrm {s p}} ^ {2}\right). \tag {28}
$$
Here, we replace  $h_{\mathrm{sp}}^t = \mu_t \cdot w^\star / \gamma$  in (25) to get (27). Updates in (27) and (28) show that  $\widetilde{h}_{\mathrm{inv}}^{t+1}$  remains in the direction of  $h_{\mathrm{in}}^t$  and  $\widetilde{h}_{\mathrm{sp}}^{t+1}$  remains in the direction of  $h_{\mathrm{sp}}^t$ .
Part-2. Now we will derive conditions under which  $h_{\mathrm{in}}^t$  and  $h_{\mathrm{sp}}^t$  will show monotonic behavior for necessary and sufficient conditions. We will first argue the condition under which ST will provably fail and converge to a classifier with a random target performance. For this, at every  $t$ , if we have:
$$
\frac {\left\| \widetilde {h} _ {\mathrm {s p}} ^ {t + 1} \right\| _ {2}}{\left\| \widetilde {h} ^ {t + 1} \right\| _ {2}} > \left\| h _ {\mathrm {s p}} ^ {t} \right\| _ {2}, \tag {29}
$$
then we can argue that as  $t \to \infty$ , we have  $\left\| h_{\mathrm{sp}}^t \right\|_2 = 1$  and hence, the ST classifier will have random target performance. Thus, we will focus on conditions, under which the norm on  $\left\| h_{\mathrm{sp}}^t \right\|_2$  increases with  $t$ . Re-writing (29), we have:
$$
\left\| \widetilde {h} _ {\mathrm {s p}} ^ {t + 1} \right\| _ {2} > \left\| \widetilde {h} ^ {t + 1} \right\| _ {2} \cdot \left\| h _ {\mathrm {s p}} ^ {t} \right\| _ {2} \tag {30}
$$
$$
\left\| \tilde {h} _ {\mathrm {s p}} ^ {t + 1} \right\| _ {2} > \left(\left\| \tilde {h} _ {\mathrm {s p}} ^ {t + 1} \right\| _ {2} + \left\| \tilde {h} _ {\mathrm {i n}} ^ {t + 1} \right\| _ {2}\right) \cdot \left\| h _ {\mathrm {s p}} ^ {t} \right\| _ {2} \tag {31}
$$
$$
\left\| \widetilde {h} _ {\mathrm {s p}} ^ {t + 1} \right\| _ {2} \cdot \left(1 - \left\| h _ {\mathrm {s p}} ^ {t} \right\| _ {2}\right) > \left\| \widetilde {h} _ {\mathrm {i n}} ^ {t + 1} \right\| _ {2} \cdot \left\| h _ {\mathrm {s p}} ^ {t} \right\| _ {2} \tag {32}
$$
$$
\frac {\left\| \tilde {h} _ {\mathrm {s p}} ^ {t + 1} \right\| _ {2}}{\left\| h _ {\mathrm {s p}} ^ {t} \right\| _ {2}} > \frac {\left\| \tilde {h} _ {\mathrm {i n}} ^ {t + 1} \right\| _ {2}}{\left\| h _ {\mathrm {i n}} ^ {t} \right\| _ {2}}. \tag {33}
$$
Plugging in (27) and (28) into (33), we get:
$$
\left| 1 - \frac {\eta}{2} \cdot \alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \sigma_ {\mathrm {s p}} ^ {2} \right| > \left| 1 - \frac {\eta}{2} \cdot \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \gamma^ {2} / \mu_ {t} \right|. \tag {34}
$$
For small enough  $\eta$ , we have the necessary condition for the failure of ST as:
$$
\alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \sigma_ {\mathrm {s p}} ^ {2} <   \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \gamma^ {2} / \mu_ {t}. \tag {35}
$$
Now we show in Lemma 11 and Lemma 10 that if the conditions assumed in the theorem continue to hold, then we can success and failure respectively.
Lemma 10 (Necessary conditions for ST). Define  $\alpha_{1}$  and  $\alpha_{2}$  as in (20) and (21) respectively. If  $\sigma_{\mathrm{sp}} \geqslant 1$  and  $\gamma \leqslant \frac{1}{2\sqrt{\sigma_{\mathrm{sp}}}}$ , then we have for all  $t$ :
$$
\alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2} \cdot \mu_ {t}}{\gamma^ {2}} \leqslant \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right). \tag {36}
$$
Proof. We upper bound and lower bound  $\alpha_{1}$  and  $\alpha_{2}$  by using the properties of  $\mathbf{r}(\cdot)$ . Recall:
$$
\alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \left[ r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) - r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) \right]. \tag {37}
$$
$$
\alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \left[ r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) + r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) - \frac {2}{\sigma_ {t}} \right]. \tag {38}
$$
We now use Taylor's expansion on  $\mathbf{r}(\cdot)$  and we get:
$$
\mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) + R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {39}
$$
and similarly, we get:
$$
\mathrm {r} \left(\sigma_ {t}\right) - \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t}\right) - \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) + R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {40}
$$
where  $R'' = \mathrm{r}''(\sigma_0)$ . This is because  $\mathrm{r}''(\cdot)$  takes positive values and is a decreasing function in  $\sigma_t$  (refer to Lemma 21). We now lower bound  $\alpha_{1}(\mu_{t},\sigma_{t})$  and upper bound  $\alpha_{2}(\mu_{t},\sigma_{t})$ :
$$
\frac {\alpha_ {1} \left(\mu_ {t} , \sigma_ {t}\right)}{\sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right)} \geqslant 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) - R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {41}
$$
$$
\frac {\alpha_ {2} \left(\mu_ {t} , \sigma_ {t}\right)}{\sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right)} \leqslant 2 \mathrm {r} (\sigma_ {t}) + 2 \cdot R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {42}
$$
Substituting the lower bound and upper bound in (36) gives us the following as stricter a necessary condition (i.e., (43) implies (36)):
$$
\left[ 2 \mathrm {r} (\sigma_ {t}) + 2 \cdot R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \right] \cdot \frac {\sigma_ {\mathrm {s p}} ^ {2} \cdot \mu_ {t}}{\gamma^ {2}} \leqslant 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) - R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {43}
$$