SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \boldsymbol {a} _ {i} ^ {\prime \top} \operatorname {d i a g} (\boldsymbol {s} _ {i}) \boldsymbol {\gamma} - \boldsymbol {a} _ {i} ^ {\prime \top} \boldsymbol {s} _ {i} \boldsymbol {s} _ {i} ^ {\top} \boldsymbol {\gamma} _ {i} - \sum_ {t \in \mathcal {T} _ {i}} \left(\boldsymbol {a} _ {i 1} ^ {\prime} - \boldsymbol {a} _ {i t} ^ {\prime}\right) \boldsymbol {s} _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right) \right| \leq 2 \Gamma A \left(\left(1 - \boldsymbol {s} _ {i 1}\right) ^ {2} + Q _ {i}\right). \tag {45}
$$
Plugging in  $\pmb{a}_i, \bar{\pmb{a}}_i$  in the bound above and assuming  $\pi \leq 1$  (w.l.o.g.), (44) is implied by the following stronger inequality
$$
\begin{array}{l} - \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {i} ^ {\prime} \cdot \left(6 \Gamma A ((1 - s _ {i 1}) ^ {2} + Q _ {i}) + \sum_ {t \in \mathcal {T} _ {i}} (\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}) s _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t})\right) \\ \leq - \frac {1 + \pi}{n} \sum_ {i = 1} ^ {n} \ell_ {i} ^ {\prime} \cdot \sum_ {t \in \mathcal {T} _ {i}} (\bar {\boldsymbol {a}} _ {i 1} - \bar {\boldsymbol {a}} _ {i t}) s _ {i t} (\gamma_ {i 1} - \gamma_ {i t}) \\ = - \frac {1 + \pi}{n} \sum_ {i = 1} ^ {n} \ell_ {i} ^ {\prime} \cdot \sum_ {t \in T _ {i}} s _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}). \\ \end{array}
$$
First, we claim that  $0.5\pi \sum_{t\in \mathcal{T}_i}s_{it}(\pmb {\gamma}_{it} - \pmb {\gamma}_{it})\geq 6\Gamma A((1 - s_{i1})^2 +Q_i)$  for all  $i\in [n]$ . The proof of this claim directly follows the earlier argument, namely, following (37), (38) and (39) which leads to the choice
$$
R \geq \frac {\operatorname* {m a x} (2 , \delta^ {- 1})}{\Theta} \log \left(\frac {C _ {0} \cdot T \Gamma A}{\pi \gamma_ {\min } ^ {\text {g a p}}}\right), \tag {46}
$$
for some constant  $C_0 > 0$ . Here, we choose sufficiently large  $C_0 \geq 64\pi$  to ensure  $R = R_{\pi} \geq \bar{R}_{\mu}$ .
Following this control over the perturbation term  $6\Gamma A((1 - s_{i1})^2 +Q_i)$ , to conclude with the result, what remains is proving the comparison
$$
- \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {i} ^ {\prime} \cdot \sum_ {t \in \mathcal {T} _ {i}} \left(\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}\right) \boldsymbol {s} _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right) \leq - \frac {1 + 0 . 5 \pi}{n} \sum_ {i = 1} ^ {n} \ell_ {i} ^ {\prime} \cdot \sum_ {t \in \mathcal {T} _ {i}} \boldsymbol {s} _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right). \tag {47}
$$
To proceed, we split the problem into two scenarios.
Scenario 1:  $\| \bar{\pmb{p}} - \pmb{p}^{mm} \| \leq \epsilon = \frac{\pi}{4A\Theta}$  for some  $\epsilon > 0$ . In this scenario, for any token, we find that
$$
\left| \boldsymbol {a} _ {i t} - \bar {\boldsymbol {a}} _ {t} \right| = \left| \boldsymbol {k} _ {i t} ^ {\top} (\bar {\boldsymbol {p}} - \boldsymbol {p} ^ {m m}) \right| \leq A \Theta \epsilon = \pi / 4.
$$
$$
\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t} \leq \bar {\boldsymbol {a}} _ {i 1} - \bar {\boldsymbol {a}} _ {i t} + 2 A \Theta \epsilon = 1 + 0. 5 \pi .
$$
Similarly,  $\pmb{a}_{i1} - \pmb{a}_{it} \geq 1 - 0.5\pi \geq 0.5$ . Since all terms  $\pmb{a}_{i1} - \pmb{a}_{it}, \pmb{s}_{it}, \pmb{\gamma}_{i1} - \pmb{\gamma}_{it}$  in (47) are nonnegative and  $(\pmb{a}_{i1} - \pmb{a}_{it})\pmb{s}_{it}(\pmb{\gamma}_{i1} - \pmb{\gamma}_{it}) \leq (1 + 0.5\pi)\pmb{s}_{it}(\pmb{\gamma}_{i1} - \pmb{\gamma}_{it})$ , above implies the desired result (47).
**Scenario 2:**  $\| \bar{\pmb{p}} - \pmb{p}^{mm} \| \geq \epsilon = \frac{\pi}{4A\Theta}$ . Since  $\bar{\pmb{p}}$  is not (locally) max-margin, in this scenario, for some  $i \in [n]$ ,  $\nu = \nu(\epsilon) > 0$ , and  $\tau \in \mathcal{T}_i$ , we have that
$$
\bar {\boldsymbol {p}} ^ {\top} \left(\boldsymbol {k} _ {i 1} - \boldsymbol {k} _ {i \tau}\right) = \boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i \tau} \leq 1 - 2 \nu .
$$
Here  $\tau = \arg \max_{t\in \mathcal{T}_i}\bar{\pmb{p}}^\top \pmb{k}_{it}$  denotes the nearest point to  $\pmb{k}_{i1}$  (along the  $\bar{\pmb{p}}$  direction). Note that a non-neighbor  $t\in \bar{\mathcal{T}}_i$  cannot be nearest because  $\bar{\pmb{p}}\in \mathrm{cone}_{\mu}(\pmb{p}^{mm})$  and (33) holds. Recall that  $s_i = \mathbb{S}(\bar{R}\pmb {a}_i)$  where  $\bar{R} = \| p\| \Theta \geq R\Theta$ . To proceed, let  $\underline{\boldsymbol{a}}_i\coloneqq \min_{t\in \mathcal{T}_i}\pmb {a}_{i1} - \pmb {a}_{it}$ ,
$$
\mathcal {I} := \left\{i \in [ n ]: \underline {{a}} _ {i} \leq 1 - 2 \nu \right\}, \quad [ n ] - \mathcal {I} := \left\{i \in [ n ]: 1 - 2 \nu <   \underline {{a}} _ {i} \right\}.
$$
For all  $i\in [n] - \mathcal{I}$
$$
\begin{array}{l} \sum_ {t \in \mathcal {T} _ {i}} \left(\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}\right) s _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right) - (1 + 0. 5 \pi) \sum_ {t \in \mathcal {T} _ {i}} s _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right) \\ \leq (2 A - (1 + 0. 5 \pi)) \Gamma \sum_ {t \in \mathcal {T} _ {i}, a _ {i 1} - a _ {i t} \geq 1 + \frac {\pi}{2}} s _ {i t} \tag {48} \\ \leq (2 A - (1 + 0. 5 \pi)) \Gamma T e ^ {- \hat {R} (1 + \frac {\pi}{2})} \\ \leq 2 A \Gamma T e ^ {- \tilde {R} \left(1 + \frac {\pi}{2}\right)}. \\ \end{array}
$$
For all  $i \in \mathcal{I}$ , split the tokens into two groups: Let  $\mathcal{N}_i$  be the group of tokens obeying  $\pmb{a}_{i1} - \pmb{a}_{it} \leq 1 - \nu$  and  $\mathcal{T}_i - \mathcal{N}_i$  be the rest of the neighbors. Observe that
$$
\frac {\sum_ {t \in \mathcal {T} _ {i} - \mathcal {N} _ {i}} \boldsymbol {s} _ {i t}}{\sum_ {t \in \mathcal {T} _ {i}} \boldsymbol {s} _ {i t}} \leq T \frac {e ^ {\nu \bar {R}}}{e ^ {2 \nu \bar {R}}} = T e ^ {- \bar {R} \nu}.
$$
Thus, using  $|\pmb{a}_{i1} - \pmb{a}_{it}| \leq 2A$  and recalling the definition of  $\gamma_{\mathrm{min}}^{\mathrm{gap}}$ , observe that
$$
\sum_ {t \in \mathcal {T} _ {i} - \mathcal {N} _ {i}} (\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}) \boldsymbol {s} _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) \leq \frac {2 \Gamma A T e ^ {- \bar {R} v}}{\gamma_ {\min } ^ {\text {g a p}}} \sum_ {t \in \mathcal {T} _ {i}} \boldsymbol {s} _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}).
$$
$$
\begin{array}{l} \sum_ {t \in \mathcal {T} _ {i}} (\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}) s _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) = \sum_ {t \in \mathcal {N} _ {i}} (\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}) s _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) + \sum_ {t \in \mathcal {T} _ {i} - \mathcal {N} _ {i}} (\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}) s _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) \\ \leq \sum_ {t \in \mathcal {N} _ {i}} (1 - v) s _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) + \frac {2 \Gamma A T e ^ {- \bar {R} v}}{\gamma_ {\min} ^ {\text {g a p}}} \sum_ {t \in \mathcal {T} _ {i}} s _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) \\ \leq \left(1 - \nu + \frac {2 \Gamma A T e ^ {- \bar {R} \nu}}{\gamma_ {\min} ^ {\text {g a p}}}\right) \sum_ {t \in \mathcal {T} _ {i}} s _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) \\ \leq \left(1 + \frac {2 \Gamma A T e ^ {- \bar {R} v}}{\gamma_ {\min} ^ {\text {g a p}}}\right) \sum_ {t \in \mathcal {T} _ {i}} s _ {i t} (\gamma_ {i 1} - \gamma_ {i t}). \\ \end{array}
$$
Hence, choosing
$$
R \geq \frac {1}{\nu \Theta} \log \left(\frac {8 \Gamma A T}{\gamma_ {\min } ^ {\text {g a p}} \pi}\right) \tag {49}
$$
results in that
$$
\begin{array}{l} \sum_ {t \in \mathcal {T} _ {i}} (\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}) \boldsymbol {s} _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) - (1 + \frac {\pi}{2}) \sum_ {t \in \mathcal {T} _ {i}} \boldsymbol {s} _ {i t} (\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}) \\ \leq \left(\frac {2 \Gamma A T e ^ {- \bar {R} \nu}}{\gamma_ {\min } ^ {\text {g a p}}} - \frac {\pi}{2}\right) \sum_ {t \in \mathcal {T} _ {i}} s _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right) \tag {50} \\ \leq - \frac {\pi}{4} \sum_ {t \in \mathcal {T} _ {i}} s _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right) \\ \leq - \frac {\pi}{4 T} \gamma_ {\min } ^ {g a p} e ^ {- \bar {R} (1 - 2 v)}. \\ \end{array}
$$
Here, the last inequality follows from the fact that  $\sum_{t\in \mathcal{T}_i}s_{it}\geq \max_{t\in \mathcal{T}_i}s_{it}\geq \frac{e^{-R(1 - 2\nu)}}{\sum_{t = 1}^T e^{-\bar{R}(a_{i1} - a_{it})}}\geq e^{-\bar{R}(1 - 2\nu)} / T$
From Assumption A, we have  $c_{\mathrm{min}} \leq -\ell' \leq c_{\mathrm{max}}$  for some positive constants  $c_{\mathrm{min}}$  and  $c_{\mathrm{max}}$ . It follows from (48) and (50) that
$$
\begin{array}{l} - \frac {1}{n} \sum_ {i} ^ {n} \ell_ {i} ^ {\prime} \cdot \left(\sum_ {t \in \mathcal {T} _ {i}} \left(\boldsymbol {a} _ {i 1} - \boldsymbol {a} _ {i t}\right) \boldsymbol {s} _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right) - \sum_ {t \in \mathcal {T} _ {i}} (1 + 0. 5 \pi) \boldsymbol {s} _ {i t} \left(\boldsymbol {\gamma} _ {i 1} - \boldsymbol {\gamma} _ {i t}\right)\right) \\ \leq c _ {\max } 2 A \Gamma T \Gamma e ^ {- \bar {R} (1 + \frac {\pi}{2})} - \frac {c _ {\min }}{n T} \cdot \frac {\pi \gamma_ {\min } ^ {\text {g a p}}}{4} e ^ {- \bar {R} (1 - 2 v)} \\ \leq 0. \\ \end{array}
$$
Combing with (49), this is guaranteed by choosing
$$
R \geq \max  \left\{\frac {1}{\nu \Theta} \log \left(\frac {8 \Gamma A T}{\gamma_ {\operatorname* {m i n}} ^ {\text {g a p}} \pi}\right), \frac {1}{(2 \nu + \pi / 2) \Theta} \log \left(\frac {8 n \Gamma A T ^ {2} c _ {\max }}{c _ {\min } \gamma_ {\min } ^ {\text {g a p}} \pi}\right) \right\},
$$
where  $\nu = \nu (\frac{\pi}{4A\Theta})$  depends only on  $\pi$  and global problem variables.
Combining this with the prior  $R$  choice (46) (by taking maximum), we conclude with the statement.
Proof. This proof is a direct corollary of Lemma 14 which itself is a special case of the nonlinear head Theorem 8. Let us verify that  $f(\pmb{X}) = \pmb{\nu}^{\top}\pmb{X}^{\top}\mathbb{S}(\pmb{X}\pmb{p})$  satisfies the assumptions of Lemma 14 where we replace the nonlinear head with linear  $\pmb{\nu}$ . To see this, set the optimal sets to be the singletons  $O_{i} = \{\mathrm{opt}_{i}\}$ . Given  $(X_{i},Y_{i})$ , let  $s_i = \mathbb{S}(K_i\pmb{p})$  and  $q_{i} = q_{i}^{p} = \sum_{t\neq \mathrm{opt}_{i}}s_{it}$ . Recalling score definition  $\gamma_{i} = Y_{i}\cdot X_{i}\nu$  and setting  $\nu_{i}\coloneqq \gamma_{i\mathrm{opt}_{i}}$  and  $Z_{i}\coloneqq \sum_{t\neq \mathrm{opt}_{i}}\gamma_{it}s_{it}$ , a particular prediction can be written as
$$
\begin{array}{l} Y _ {i} \cdot \boldsymbol {v} ^ {\top} \boldsymbol {X} _ {i} ^ {\top} \mathbb {S} (\boldsymbol {X} _ {i} \boldsymbol {p}) = \gamma_ {i} ^ {\top} s _ {i} = \gamma_ {i o p t _ {i}} (1 - q _ {i}) + \sum_ {i \neq o p t _ {i}} \gamma_ {i t} s _ {i t} \\ = \nu_ {i} (1 - q _ {i}) + Z _ {i}. \\ \end{array}
$$
To proceed, we demonstrate the choices for  $C,\epsilon >0$  . Let  $C\coloneqq -\min_{i\in [n],t\in [T]}\gamma_{it}\wedge 0$  and  $q_{\mathrm{max}} = \max_{i\in [n]}q_i$  . Note that  $Z_{i}\geq \sum_{t\neq \mathrm{opt}_{i}}\gamma_{it}s_{it}\geq q_{i}\gamma_{\mathrm{min}}\geq -Cq_{\mathrm{max}}$  . Now, using strict score optimality of opt's for all  $i\in [n]$  , we set
$$
\epsilon := 1 - \sup  _ {i \in [ n ]} \frac {\sum_ {t \neq \operatorname {o p t} _ {i}} \gamma_ {i t} s _ {i t}}{\nu_ {i} q _ {i}} \geq 1 - \sup  _ {i \in [ n ]} \frac {\sup  _ {t \neq \operatorname {o p t} _ {i}} \gamma_ {i t}}{\gamma_ {i \operatorname {o p t} _ {i}}} > 0.
$$
We conclude by observing  $Z_{i} \leq \nu_{i} q_{i} \frac{\sum_{i \neq \mathrm{opt}_{i}} \gamma_{it} s_{it}}{\nu_{i} q_{i}} \leq \nu_{i} q_{i} \epsilon$  as desired.
Proof. We first show that  $\lim_{t\to \infty}\| \pmb {p}(t)\| = \infty$  . From Lemma 4, we have
$$
\left\langle \nabla \mathcal {L} (\boldsymbol {p}), \boldsymbol {p} ^ {m m \star} \right\rangle = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell^ {\prime} \left(Y _ {i} \cdot \boldsymbol {v} ^ {\top} X _ {i} ^ {\top} \mathbb {S} \left(\boldsymbol {K} _ {i} \boldsymbol {p}\right)\right) \cdot \left\langle \boldsymbol {K} _ {i} \boldsymbol {p} ^ {m m \star}, \mathbb {S} ^ {\prime} \left(\boldsymbol {a} _ {i}\right) \boldsymbol {\gamma} _ {i} \right\rangle ,
$$
where  $\gamma_{i} = Y_{i}\cdot X_{i}\nu$  and  $a_{i} = K_{i}p$
It follows from Lemma 4 that  $\langle \nabla \mathcal{L}(\pmb{p}), \pmb{p}^{mm\star} \rangle < 0$  for all  $\pmb{p} \in \mathbb{R}^d$ . Hence, for any finite  $\pmb{p}$ ,  $\langle \nabla \mathcal{L}(\pmb{p}), \pmb{p}^{mm\star} \rangle$  cannot be equal to zero, as a sum of negative terms. Therefore, there are no finite critical points  $\pmb{p}$ , for which  $\nabla \mathcal{L}(\pmb{p}) = 0$ . On the other hand, Lemma 6 states  $\nabla \mathcal{L}(\pmb{p}(t)) \to 0$  which implies that  $\| \pmb{p}(t) \| \to \infty$ .
Next, we provide the directional convergence for the setting  $n = 1$ . Let us consider an arbitrary value of  $\epsilon \in (0,1)$  and set  $\pi = \epsilon / (1 - \epsilon)$ . As  $\lim_{t\to \infty}\| p(t)\| = \infty$ , we can select a specific  $t_\epsilon$  such that for all  $t \geq t_\epsilon$ , it holds that  $\| p(t)\| \geq R_\epsilon \lor 1/2$  for any choice of  $R_\epsilon$ . To proceed, we choose  $R_\epsilon$  based on Lemma 5 so that for any  $t \geq t_\epsilon$ , we have that
$$
\left\langle - \nabla \mathcal {L} (\boldsymbol {p} (t)), \frac {\boldsymbol {p} ^ {m m \star}}{\| \boldsymbol {p} ^ {m m \star} \|} \right\rangle \geq (1 - \epsilon) \left\langle - \nabla \mathcal {L} (\boldsymbol {p} (t)), \frac {\boldsymbol {p} (t)}{\| \boldsymbol {p} (t) \|} \right\rangle .
$$
Multiplying both sides by the stepsize  $\eta$  and using the gradient descent update, we get
$$
\begin{array}{l} \left\langle \boldsymbol {p} (t + 1) - \boldsymbol {p} (t), \frac {\boldsymbol {p} ^ {m m \star}}{\| \boldsymbol {p} ^ {m m \star} \|} \right\rangle \geq (1 - \epsilon) \left\langle \boldsymbol {p} (t + 1) - \boldsymbol {p} (t), \frac {\boldsymbol {p} (t)}{\| \boldsymbol {p} (t) \|} \right\rangle \\ = \frac {(1 - \epsilon)}{2 \| \boldsymbol {p} (t) \|} \left(\| \boldsymbol {p} (t + 1) \| ^ {2} - \| \boldsymbol {p} (t) \| ^ {2} - \| \boldsymbol {p} (t + 1) - \boldsymbol {p} (t) \| ^ {2}\right) \\ \geq (1 - \epsilon) \left(\frac {1}{2 \| \boldsymbol {p} (t) \|} \left(\| \boldsymbol {p} (t + 1) \| ^ {2} - \| \boldsymbol {p} (t) \| ^ {2}\right) - \| \boldsymbol {p} (t + 1) - \boldsymbol {p} (t) \| ^ {2}\right) \tag {51} \\ \geq (1 - \epsilon) \left(\| \boldsymbol {p} (t + 1) \| - \| \boldsymbol {p} (t) \| - \| \boldsymbol {p} (t + 1) - \boldsymbol {p} (t) \| ^ {2}\right) \\ \geq (1 - \epsilon) \left(\| \boldsymbol {p} (t + 1) \| - \| \boldsymbol {p} (t) \| - 2 \eta (\mathcal {L} (\boldsymbol {p} (t)) - \mathcal {L} (\boldsymbol {p} (t + 1)))\right). \\ \end{array}
$$
Here, the second inequality is obtained from  $\| \pmb{p}(t) \| \geq 1/2$ ; the third inequality follows since for any  $a, b > 0$ , we have  $(a^2 - b^2) / (2b) - (a - b) \geq 0$ ; and the last inequality uses Lemma 6.
Summing the above inequality over  $t \geq t_{\epsilon}$  gives
$$
\left\langle \frac {\boldsymbol {p} (t)}{\| \boldsymbol {p} (t) \|}, \frac {\boldsymbol {p} ^ {m m \star}}{\| \boldsymbol {p} ^ {m m \star} \|} \right\rangle \geq 1 - \epsilon + \frac {C (\epsilon , \eta)}{\| \boldsymbol {p} (t) \|},
$$
for some finite constant  $C(\epsilon, \eta)$  defined as
$$
C (\epsilon , \eta) := \left\langle \boldsymbol {p} \left(t _ {\epsilon}\right), \frac {\boldsymbol {p} ^ {m m \star}}{\| \boldsymbol {p} ^ {m m \star} \|} \right\rangle - (1 - \epsilon) \| \boldsymbol {p} \left(t _ {\epsilon}\right) \| - 2 \eta (1 - \epsilon) \left(\mathcal {L} \left(\boldsymbol {p} \left(t _ {\epsilon}\right)\right) - \mathcal {L} ^ {*}\right), \tag {52}
$$
where  $\mathcal{L}^*$  denotes the minimum objective.
Since  $\| p(t)\| \to \infty$ , we get
$$
\lim  _ {t \rightarrow \infty} \inf  _ {\epsilon} \left\langle \frac {\boldsymbol {p} (t)}{\| \boldsymbol {p} (t) \|}, \frac {\boldsymbol {p} ^ {m m \star}}{\| \boldsymbol {p} ^ {m m \star} \|} \right\rangle \geq 1 - \epsilon .
$$
Given that we can choose any value of  $\epsilon \in (0,1)$ , we have  $\pmb{p}(t) / \| \pmb{p}(t) \| \to \pmb{p}^{mm\star} / \| \pmb{p}^{mm\star} \|$ .
Proof. Following the proof of Lemma 7, let  $(\mathcal{T}_i)_{i=1}^n$  denote the sets of SVM-neighbors as defined in Definition 2. We define  $\bar{\mathcal{T}}_i = [T] - \mathcal{T}_i - \{\alpha_i\}$  as the tokens that are non-SVM neighbors. Additionally, let  $\mu$  be defined as in (32). Let us denote the initialization lower bound as  $R_\mu^0 \coloneqq R$ , where  $R$  is given in the Theorem 3's statement. Consider an arbitrary value of  $\epsilon \in (0, \mu/2)$  and let  $1/(1 + \pi) = 1 - \epsilon$ .
We additionally denote  $R_{\epsilon} \gets R_{\pi} \lor 1/2$  where  $R_{\pi}$  was defined in Lemma 7(L3.). At initialization  $p(0)$ , we set  $\epsilon = \mu / 2$  to obtain  $R_{\mu}^{0} = R_{\mu / 2}$  and provide the proof in four steps:
Step 1: There are no stationary points within  $C_{\mu, R_\mu^0}(\pmb{p}^{mm})$ . We begin by proving that there are no stationary points within  $C_{\mu, R_\mu^0}(\pmb{p}^{mm})$ . Then, since  $R_\mu^0 \geq \bar{R}_\mu$  per Lemma 7, we can apply (L2.) to find that: For all  $\pmb{q}, \pmb{p} \in \mathrm{cone}_{\mu}(\pmb{p}^{mm})$  with  $\pmb{q} \neq 0$  and  $\| \pmb{p} \| \geq R_\mu^0$ , we have that  $-\pmb{q}^\top \nabla \mathcal{L}(\pmb{p})$  is strictly positive.
Step 2: It follows from Lemma 7(L3.) that, for all  $\epsilon \in (0,\mu /2)$ , all  $p\in C_{\mu ,R_\epsilon}(p^{mm})$  satisfy
$$
\left\langle - \nabla \mathcal {L} (\boldsymbol {p}), \frac {\boldsymbol {p} ^ {m m}}{\| \boldsymbol {p} ^ {m m} \|} \right\rangle \geq (1 - \epsilon) \left\langle - \nabla \mathcal {L} (\boldsymbol {p}), \frac {\boldsymbol {p}}{\| \boldsymbol {p} \|} \right\rangle . \tag {53}
$$
The argument above applies to a general  $\epsilon \in (0,\mu /2)$ . However, at initialization  $\pmb{p}(0)$ , we set  $\epsilon = \mu /2$  to obtain our earlier  $R_{\mu}^{0}$  choice. To proceed, for any  $\epsilon \in (0,\mu /2)$ , we will show that after gradient descent enters the conic set  $C_{\mu ,R_{\epsilon}}(\pmb{p}^{mm})$  for the first time, it will never leave the set. Let  $t_\epsilon$  be the first time gradient descent enters  $C_{\mu ,R_{\epsilon}}(\pmb{p}^{mm})$ . In Step 4, we will prove that such  $t_\epsilon$  is guaranteed to exist. Additionally, for  $\epsilon \gets \mu /2$ , note that  $t_\epsilon = 0$  i.e. the point of initialization.
Step 3: Updates remain inside the cone  $C_{\mu, R_{\epsilon}}(\pmb{p}^{mn})$ . By leveraging the results from Step 1 and Step 2, we demonstrate that the gradient iterates, with an appropriate constant step size, starting from  $\pmb{p}(t_{\epsilon}) \in C_{\mu, R_{\epsilon}}(\pmb{p}^{mn})$ , remain within this cone.
We proceed by induction. Suppose that the claim holds up to iteration  $t \geq t_{\epsilon}$ . This implies that  $\pmb{p}(t) \in C_{\mu, R_{\epsilon}}(\pmb{p}^{mm})$ . Hence, recalling cone definition, for  $\mu > 0$  and  $R_{\epsilon}$ , we have  $\left\langle \frac{\pmb{p}(t)}{\|\pmb{p}(t)\|}, \frac{\pmb{p}^{mm}}{\|\pmb{p}^{mm}\|} \right\rangle \geq 1 - \mu$  and  $\| \pmb{p}(t) \| \geq R_{\epsilon}$ . Let
$$
\rho (t) := - \frac {1}{1 - \epsilon} \left\langle \nabla \mathcal {L} (\pmb {p} (t)), \frac {\pmb {p} ^ {m m}}{\| \pmb {p} ^ {m m} \|} \right\rangle .
$$
Note that  $\rho(t) > 0$  due to Step 1. This together with the gradient descent update rule gives
$$
\begin{array}{l} \left\langle \frac {\boldsymbol {p} (t + 1)}{\| \boldsymbol {p} (t) \|}, \frac {\boldsymbol {p} ^ {m m}}{\| \boldsymbol {p} ^ {m m} \|} \right\rangle = \left\langle \frac {\boldsymbol {p} (t)}{\| \boldsymbol {p} (t) \|} - \frac {\eta}{\| \boldsymbol {p} (t) \|} \nabla \mathcal {L} (\boldsymbol {p} (t)), \frac {\boldsymbol {p} ^ {m m}}{\| \boldsymbol {p} ^ {m m} \|} \right\rangle \\ \geq 1 - \mu - \frac {\eta}{\| \boldsymbol {p} (t) \|} \left\langle \nabla \mathcal {L} (\boldsymbol {p} (t)), \frac {\boldsymbol {p} ^ {m m}}{\| \boldsymbol {p} ^ {m m} \|} \right\rangle \tag {54a} \\ = 1 - \mu + \frac {\eta \rho (t) (1 - \epsilon)}{\| \boldsymbol {p} (t) \|}. \\ \end{array}
$$
Note that from Lemma 7, we have  $\langle \nabla \mathcal{L}(\pmb {p}(t)),\pmb {p}(t)\rangle < 0$  which implies that  $\| \pmb {p}(t + 1)\| \geq \| \pmb {p}(t)\|$ . This together with  $R_{\epsilon}$  definition and  $\| \pmb {p}(t)\| \geq 1 / 2$  implies that
$$
\begin{array}{l} \| \boldsymbol {p} (t + 1) \| \leq \frac {1}{2 \| \boldsymbol {p} (t) \|} \left(\| \boldsymbol {p} (t + 1) \| ^ {2} + \| \boldsymbol {p} (t) \| ^ {2}\right) \\ = \frac {1}{2 \| \boldsymbol {p} (t) \|} \left(2 \| \boldsymbol {p} (t) \| ^ {2} - 2 \eta \langle \nabla \mathcal {L} (\boldsymbol {p} (t)), \boldsymbol {p} (t) \rangle + \eta^ {2} \| \nabla \mathcal {L} (\boldsymbol {p} (t)) \| ^ {2}\right) \tag {54b} \\ \leq \| \boldsymbol {p} (t) \| - \frac {\eta}{\| \boldsymbol {p} (t) \|} \left\langle \nabla \mathcal {L} (\boldsymbol {p} (t)), \boldsymbol {p} (t) \right\rangle + \eta^ {2} \| \nabla \mathcal {L} (\boldsymbol {p} (t)) \| ^ {2}. \\ \end{array}
$$
Hence, using (53)
$$
\begin{array}{l} \frac {\| \boldsymbol {p} (t + 1) \|}{\| \boldsymbol {p} (t) \|} \leq 1 - \frac {\eta}{\| \boldsymbol {p} (t) \|} \left\langle \nabla \mathcal {L} (\boldsymbol {p} (t)), \frac {\boldsymbol {p} (t)}{\| \boldsymbol {p} (t) \|} \right\rangle + \eta^ {2} \frac {\| \nabla \mathcal {L} (\boldsymbol {p} (t)) \| ^ {2}}{\| \boldsymbol {p} (t) \|} \\ \leq 1 - \frac {\eta}{(1 - \epsilon) \| \boldsymbol {p} (t) \|} \left\langle \nabla \mathcal {L} (\boldsymbol {p} (t)), \frac {\boldsymbol {p} ^ {m m}}{\| \boldsymbol {p} ^ {m m} \|} \right\rangle + \eta^ {2} \frac {\| \nabla \mathcal {L} (\boldsymbol {p} (t)) \| ^ {2}}{\| \boldsymbol {p} (t) \|} \tag {54c} \\ = 1 + \frac {\eta \rho (t)}{\| \boldsymbol {p} (t) \|} + \frac {\eta^ {2} \| \nabla \mathcal {L} (\boldsymbol {p} (t)) \| ^ {2}}{\| \boldsymbol {p} (t) \|} =: C _ {1} (\rho (t), \eta). \\ \end{array}
$$
Here, the second inequality follows from (53).
Now, it follows from (54a) and (54c) that
$$
\begin{array}{l} \left\langle \frac {\boldsymbol {p} (t + 1)}{\| \boldsymbol {p} (t + 1) \|}, \frac {\boldsymbol {p} ^ {m m}}{\| \boldsymbol {p} ^ {m m} \|} \right\rangle \geq \frac {1}{C _ {1} (\rho (t) , \eta)} \left(1 - \mu + \frac {\eta \rho (t) (1 - \epsilon)}{\| \boldsymbol {p} (t) \|}\right) \\ = 1 - \mu + \frac {1}{C _ {1} (\rho (t) , \eta)} \left((1 - \mu) (1 - C _ {1} (\rho (t), \eta)) + \frac {\eta \rho (t) (1 - \epsilon)}{\| \boldsymbol {p} (t) \|}\right) \\ = 1 - \mu + \frac {\eta}{C _ {1} (\rho (t) , \eta)} \left((\mu - 1) \left(\frac {\rho (t)}{\| \boldsymbol {p} (t) \|} + \frac {\eta \| \nabla \mathcal {L} (\boldsymbol {p} (t)) \| ^ {2}}{\| \boldsymbol {p} (t) \|}\right) + \frac {\rho (t) (1 - \epsilon)}{\| \boldsymbol {p} (t) \|}\right) \tag {55} \\ = 1 - \mu + \frac {\eta}{C _ {1} (\rho (t) , \eta)} \left(\frac {\rho (t) (\mu - \epsilon)}{\| \boldsymbol {p} (t) \|} - \eta (1 - \mu) \frac {\| \nabla \mathcal {L} (\boldsymbol {p} (t)) \| ^ {2}}{\| \boldsymbol {p} (t) \|}\right) \\ \geq 1 - \mu , \\ \end{array}
$$
where the last inequality uses our choice of stepsize  $\eta \leq 1 / L_{p}$  in Theorem 3's statement. Specifically, we need  $\eta$  to be small to ensure the last inequality. We will guarantee this by choosing a proper  $R_{\epsilon}$  in Lemma 7. Specifically, Lemma 7 leaves the choice of  $C_0$  in  $R_{\epsilon}$  lower bound of (46) open (it can always be chosen larger). Here, by choosing  $C_0 \gtrsim 1 / L_p$  will ensure  $\eta \leq 1 / L_p$  works well.
To proceed, we have that