SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on this lemma, what we need is constructing a perturbation to modify  $\pmb{q}$ 's directional neighbors and make sure all of them have strictly better scores than their associated selected-tokens. Note that, once we construct a new candidate (say  $\pmb{q}_0 = \pmb{q} + \pmb{\mathrm{perturbation}}$ ), all sufficiently large scalings of  $\pmb{q}_0$  will achieve  $\mathcal{L}(R \cdot q_0) < \mathcal{L}_{\star}$ . Thus, we can find a strictly better solution than  $\mathcal{L}_{\star}$  for any norm lower bound  $R_0$  – which is enforced within the definition of  $C_{\epsilon}$ .
Lemma 10 There are at most  $d$  directional neighbors i.e.  $|\mathcal{M}_q| \leq d$ .
Proof. Directional neighbors are indices  $(i,t)$  obeying the inequality
$$
\left(\boldsymbol {k} _ {i \alpha_ {i}} - \boldsymbol {k} _ {i t}\right) ^ {\top} \boldsymbol {q} = \Gamma_ {\boldsymbol {q}}.
$$
Declare  $\pmb{D}$  to be the matrix with rows obtained by these key differences  $\pmb{k}_{it} - \pmb{k}_{i\alpha_i}$ .  $\pmb{D} \in \mathbb{R}^{M \times d}$  where  $M = |\mathcal{M}_q|$ . We then obtain  $\pmb{D}\pmb{q} = -\Gamma_q\mathbf{1}_M$  where  $\mathbf{1}_M$  is the all ones vector. If  $M > d$ , the equality  $\pmb{D}\pmb{q} = -\Gamma_q\mathbf{1}_M$  cannot be satisfied because  $\mathbf{1}_M$  is not in the range space of  $\pmb{D}$  by our assumption of general key embedding positions.
To proceed,  $|\mathcal{M}_q| \leq d$  and let  $D$  be as defined in the lemma above.  $D$  is also full-rank by our assumption of general key positions. We use  $D$  to construct a perturbation as follows. Let  $\mathcal{M}_q^+ \subset \mathcal{M}_q$  be the set of directional neighbors with strictly higher scores than their associated selected-tokens. In other words, all  $(j,\beta) \in \mathcal{M}_q^+$  obeys
$$
\gamma_ {j \beta} > \gamma_ {j \alpha_ {j}}.
$$
Define the score difference  $\kappa = \min_{(j,\beta)\in \mathcal{M}_q^+}\gamma_{j\beta} - \gamma_{j\alpha_j} > 0$ . We know  $\kappa >0$  because  $\alpha$  is not neighbor-optimal. Finally, define the indicator vector of  $\mathbf{1}_{+}$  with same dimension as cardinality  $|\mathcal{M}_q|$ .  $\mathbf{1}_{+}$  is 1 for the rows of  $\pmb{D}$  corresponding to  $\mathcal{M}_q^+$  and is 0 otherwise. Finally, set the perturbation as
$$
\boldsymbol {q} ^ {\perp} = \boldsymbol {D} ^ {\dagger} \mathbf {1} _ {+}.
$$
where we used the full-rankness of  $D$  during pseudo-inversion. To proceed, for a small  $\epsilon_0 > 0$ , consider the candidate direction  $\pmb{q}_0 = \pmb{q} + \epsilon_0 \pmb{q}^\perp$ . We pick  $\epsilon_0 = O(\epsilon)$  sufficiently small to ensure  $\pmb{q}_0 \in C_\epsilon$ . To finalize, let us consider the margins of the tokens within  $\pmb{q}_0$ . Similar to Lemma 9, set  $\delta_q = \min_{i \in [n], t \neq \alpha_i, (i,t) \notin \mathcal{M}_q} \Gamma_q^{it} - \Gamma_q > 0$ . Let  $\bar{\epsilon}_0 = \| \pmb{q}_0 \| - 1 = \| \pmb{q} + \epsilon_0 \pmb{q}^\perp \| - 1$ . Using definition of  $\pmb{q}^\perp$ , we have that
- For  $(i,t)\in \mathcal{M}_q^+$ , we achieve a margin of
$$
\left(\boldsymbol {k} _ {i \alpha_ {i}} - \boldsymbol {k} _ {i t}\right) ^ {\top} \left(\boldsymbol {q} + \epsilon_ {0} \boldsymbol {q} ^ {\perp}\right) / \left(1 + \bar {\epsilon} _ {0}\right) = \frac {\Gamma_ {\boldsymbol {q}} - \epsilon_ {0}}{1 + \bar {\epsilon} _ {0}}.
$$
For  $(i,t)\in \mathcal{M}_{q}^{+}$  , we achieve a margin of  $\frac{\Gamma_q}{1 + \bar{\epsilon}_0}$
- For  $(i, t) \notin \mathcal{M}_q$ , setting  $K = \| q^\perp \| \cdot \sup_{i, t, \tau} \| k_{i\alpha_i} - k_{it} \|$ , we achieve a margin of at most
$$
\left(\boldsymbol {k} _ {i \alpha_ {i}} - \boldsymbol {k} _ {i t}\right) ^ {\top} \left(\boldsymbol {q} + \epsilon_ {0} \boldsymbol {q} ^ {\perp}\right) / \left(1 + \bar {\epsilon} _ {0}\right) \geq \frac {\Gamma_ {\boldsymbol {q}} + \delta_ {\boldsymbol {q}} - \epsilon_ {0} K}{1 + \bar {\epsilon} _ {0}}.
$$
In short, since  $\bar{\epsilon}_0 = O(\epsilon_0)$ , setting  $\epsilon_0$  sufficiently small guarantees that  $\mathcal{M}_q^+$  is the set of directional neighbors of  $q_0$ . Since  $\mathcal{M}_q^+$  has strictly higher scores than their associated selected-tokens, applying Lemma 9 on  $q_0$  shows that,  $\mathcal{L}(R \cdot q_0) < \mathcal{L}_{\star}$  for sufficiently large  $R$  implying  $\| \bar{p}(R) \| < \infty$ .
- (B)  $q$  selects multiple tokens for some inputs  $i \in [n]$ : In this setting, we will again construct a perturbation to create a scenario where  $q_0 = q + \epsilon q^\perp$  selects a single token for each input  $i \in [n]$ . We will then employ margin analysis (first statement of Lemma 8) to conclude that  $q_0$  outperforms a  $\delta \ll \epsilon$  neighborhood of  $q$ .
Let  $\mathcal{I} \subset [n]$  be the set of inputs for which  $\mathbf{q}$  selects multiple tokens. Specifically, for each  $i \in \mathcal{I}$ , there is  $\mathcal{T}_i \subset [T]$  such that  $|\mathcal{T}_i| \geq 2$  and for any  $i \in \mathcal{I}$  and  $\theta \in \mathcal{T}_i$ ,
$$
\boldsymbol {k} _ {i \theta} ^ {\top} \boldsymbol {q} = \arg \max  _ {t \in [ T ]} \boldsymbol {k} _ {i t} ^ {\top} \boldsymbol {q}.
$$
From these multiply-selected token indices let us select the highest score one, namely,  $\beta_{i} = \arg \max_{\theta \in \mathcal{T}_{i}}\pmb{\gamma}_{it\theta}$  for  $i\in \mathcal{I}$ . Now, define the unique optimal tokens for each input as  $\alpha \in \mathbb{R}^n$  where  $\alpha_{i} := \beta_{i}$  for  $i\in \mathcal{I}$  and  $\alpha_{i} = \arg \max_{t\in [T]}\pmb{k}_{it}^{\top}\pmb{q}$  for  $i\notin \mathcal{I}$ . Define  $\mathcal{L}_{\star} = \frac{1}{n}\sum_{i = 1}^{n}\ell (\pmb{\gamma}_{i\alpha_i})$  as earlier.
Secondly, we construct a perturbation  $\pmb{q}^{\perp}$  to show that  $\pmb{q}_0 = \pmb{q} + \epsilon_0\pmb{q}^{\perp}$  can select tokens  $\alpha$  asymptotically. To see this, define the matrix  $D$  where each (unique) row is given by  $\pmb{k}_{i\alpha_i} - \pmb{k}_{i\theta}$  where  $\theta \in \mathcal{T}_i, \theta \neq \alpha_i, i \in I$ . Now note that,  $(\pmb{k}_{i\alpha_i} - \pmb{k}_{i\theta})^\top \pmb{q} = 0$  for all  $\theta \in \mathcal{T}_i, \theta \neq \alpha_i, i \in I$ . Since keys are in general positions, this implies that  $D$  has at most  $d - 1$  rows and, thus, its rows are linearly independent. Consequently, choose  $\pmb{q}^{\perp} = D^{\dagger}\mathbf{1}$  where  $\dagger$  denotes pseudo-inverse and  $\mathbf{1}$  is the all ones vector. Also let  $\Gamma_q$  be the margin of directional margin of  $q$  that is
$$
\Gamma_ {\boldsymbol {q}} = \min  _ {i \in [ n ], t \notin \mathcal {T} _ {i}} \left(\boldsymbol {k} _ {i \alpha_ {i}} - \boldsymbol {k} _ {i t}\right) ^ {\top} \boldsymbol {q}.
$$
With this choice and setting  $K = \sup_{i,t,\tau} \| \pmb{k}_{i\alpha_i} - \pmb{k}_{it} \|$ , we have that
- For  $\theta \in \mathcal{T}_i, \theta \neq \alpha_i$ :  $(\pmb{k}_{i\alpha_i} - \pmb{k}_{i\theta})^\top \pmb{q}_0 = (\pmb{k}_{i\alpha_i} - \pmb{k}_{i\theta})^\top \pmb{q}^\perp = \epsilon_0$
- For all other  $(i, t)$  with  $t \neq \alpha_{i}$ :  $(\pmb{k}_{i\alpha_{i}} - \pmb{k}_{it})^{\top} \pmb{q}_{0} \geq \Gamma_{\pmb{q}} - K \| \pmb{q}^{\perp} \| \epsilon_{0}$ .
Choosing  $\epsilon_0 < \Gamma_q / (1 + K\| q^\perp \|)$ , together, these imply that,
-  $\alpha = (\alpha_{i})_{i=1}^{n}$  is the selected-token of  $q_{0}$ ,
-  $q_{0}$  achieves a directional margin of
$$
\Gamma_ {\boldsymbol {q} _ {0}} = \frac {\epsilon_ {0}}{\| \boldsymbol {q} _ {0} \|} \geq \frac {\epsilon_ {0}}{1 + \epsilon_ {0} \| \boldsymbol {q} ^ {\perp} \|},
$$
-  $\pmb{q}_0$  is neighbor-optimal because directional neighbors are  $\theta \in \mathcal{T}_i, \theta \neq \alpha_i$  and  $\gamma_{i\theta} < \gamma_{i\alpha_i}$ .
Note that, these conditions lay the groundwork for applying the first statement of Lemma 8 with  $\pmb{p} \gets \pmb{q}_0$ . We next explore the optimal directions within  $\mathcal{N}_{\delta}$  and show that  $\pmb{q}_0$  strictly outperform them in terms of training loss.
To proceed, given small  $\delta \ll \epsilon_0$ , let us study  $\mathcal{L}_R = \min_{p\in \mathcal{N}_{\delta},R\leq \| p\| < \infty}\mathcal{L}(p)$ . Here, recall that  $\pmb{p}$  has a  $\delta$ -small directional perturbation around  $\pmb{q}$  which can modify the token selections by breaking the ties between the multiply-selected token indices by  $\pmb{q}$ . However, thanks to the distinct token score assumption, for large  $R\leq \| p\|$ , the optimal  $\pmb{p}\in \mathcal{N}_{\delta}$  is guaranteed to (uniquely) select indices  $\alpha_{i}\in \mathcal{T}_{i}$ . Because all other  $\pmb{p}$  directions - which, asymptotically, either select other tokens  $\theta \in \mathcal{T}_i,\theta \neq \alpha_i$  or split the probabilities equally across a subset of  $\mathcal{T}_i$  - achieve a larger loss. For instance,  $\pmb{q}$  will split the probabilities equally across  $\mathcal{T}_i$  to achieve an asymptotic loss of
$$
\mathcal {L} _ {\star} ^ {q} := \lim  _ {R \to \infty} \mathcal {L} (R \cdot q) = \frac {1}{n} \sum_ {i \notin I} \ell (\boldsymbol {\gamma} _ {i \alpha_ {i}}) + \frac {1}{n} \sum_ {i \in I} \ell \left(\frac {1}{| \mathcal {T} _ {i} |} \sum_ {\theta \in \mathcal {T} _ {i}} \boldsymbol {\gamma} _ {i \theta}\right)
$$
Thus,  $\mathcal{L}_{\star}^{q} > \mathcal{L}_{\star}$  because  $\ell \left(\frac{1}{|\mathcal{T}_i|}\sum_{\theta \in \mathcal{T}_i}\gamma_{i\theta}\right) > \ell (\gamma_{i\alpha_i}) = \ell (\gamma_{i\alpha_i})$  where  $\alpha_{i}$  has the highest score i.e.  $\pmb{\gamma}_{i\alpha_i}>$ $\frac{1}{|\mathcal{T}_i|}\sum_{\theta \in \mathcal{T}_i}\pmb{\gamma}_{i\theta}$ . Set  $\tilde{\pmb{p}} (R) = \arg \min_{\pmb {p}\in N_{\delta},\| \pmb {p}\| \leq R}\mathcal{L}(\pmb {p})$ . Consequently, there are two scenarios are:
-  $\lim_{R\to \infty}\| \tilde{p} (R)\|$  is finite. This already proves the statement of the theorem as  $\tilde{p} (R) / R\rightarrow 0$  within  $\delta < \epsilon$  neighborhood of  $\pmb{q}$ .
- For sufficiently large  $R$ , the selected-tokens of  $\tilde{p}(R)$  are  $\alpha = (\alpha_i)_{i=1}^n$ .
Proceeding with the second (remaining scenario), we study the directional margin of  $\tilde{\pmb{p}}(R)$ . More broadly, for any  $\pmb{p} \in \mathcal{N}_{\delta}$  and  $\bar{\pmb{p}} = \pmb{p} / \| \pmb{p} \|$  with selected-tokens  $\alpha$ , using the fact that  $\| \bar{\pmb{p}} - \pmb{q} \| \leq \delta \ll \epsilon_0$ , we can bound the directional margin as
For  $\theta \in \mathcal{T}_i,\theta \neq \alpha_i\colon (\pmb {k}_{i\alpha_i} - \pmb {k}_{i\theta})^\top \bar{\pmb{p}} = (\pmb {k}_{i\alpha_i} - \pmb {k}_{i\theta})^\top (\bar{\pmb{p}} -\pmb {q})\leq K\delta .$
- For all other  $(i, t)$  with  $t \neq \alpha_{i}$ :  $(\pmb{k}_{i\alpha_{i}} - \pmb{k}_{it})^{\top} \bar{\pmb{p}} \geq \Gamma_{\pmb{q}} - K\delta$ .
This means that, any such  $\bar{\pmb{p}}\in \mathcal{N}_{\delta}$  achieves a directional margin of at most
$$
\Gamma_ {\bar {p}} \leq K \delta .
$$
Applying Lemma 8 and setting  $\delta = O(\epsilon_0)$ , this implies that for
$$
R \gtrsim \frac {1}{\Gamma_ {q _ {0}} - \Gamma_ {\bar {p}}} = \frac {1}{\frac {\epsilon_ {0}}{1 + \epsilon_ {0} \| q ^ {\perp} \|} - K \delta} = O (\frac {1}{\epsilon_ {0}}),
$$
we have that  $\mathcal{L}(R\cdot q_0) < \min_{\| \pmb {p}\| = R,\pmb {p}\in \mathcal{N}_{\delta}}\mathcal{L}(\pmb {p})$  . Since this holds for all  $R$  (68) holds (similar to Case (A1)) and we conclude that whenever  $\| \bar{\pmb{p}} (R)\| \to \infty$  , it doesn't directionally converge within  $\mathcal{N}_{\delta}$  (i.e.  $\delta >0$  neighborhood of  $\pmb{q}$  ) proving the advertised result.
B.6 Proof of Lemma 2
We prove a slightly general restatement where we require  $\nu \in \mathrm{range}(\mathbf{W}^{\top})$  - instead of full-rank  $\pmb{W}$
Lemma 11 Suppose for all  $i \in [n]$  and  $t \neq \mathsf{opt}_i$ ,  $Y_i = 1$  and  $\gamma_{it} < \gamma_{i\mathrm{opt}_i}$ . Also suppose  $\pmb{\nu} \in \mathrm{range}(\pmb{W}^{\top})$ . Then,  $\pmb{p}^{mm\star}$  exists - i.e. (ATT-SVM) is feasible for optimal indices  $\alpha_i \gets \mathsf{opt}_i$ .
Proof. To establish the existence of  $p^{mm\star}$ , we only need to find a direction that demonstrates the feasibility of (ATT-SVM), i.e. we need to find  $p$  that satisfies the margin constraints. To begin, let's define the minimum score difference:
$$
\underline {{\gamma}} = \min  _ {i \in [ n ], t \neq \operatorname {o p t} _ {i}} \gamma_ {i \operatorname {o p t} _ {i}} - \gamma_ {i t}.
$$
We then set  $\pmb{p} = \underline{\gamma}^{-1}(\pmb{W}^{\top})^{\dagger}\pmb{v}$  where  $\dagger$  denotes pseudo-inverse. By assumption  $\pmb{W}^{\top}\pmb{p} = \underline{\gamma}^{-1}\pmb{v}$ . To conclude, observe that  $\pmb{p}$  is a feasible solution since  $\pmb{k}_{it} = \pmb{W}\pmb{x}_{it}$  and for all  $i\in [n]$  and  $t\neq \mathrm{opt}_i$ , we have that
$$
\begin{array}{l} \left(\boldsymbol {k} _ {i \text {o p t} _ {i}} - \boldsymbol {k} _ {i t}\right) ^ {\top} \boldsymbol {p} = \left(\boldsymbol {x} _ {i \text {o p t} _ {i}} - \boldsymbol {x} _ {i t}\right) ^ {\top} \boldsymbol {W} ^ {\top} \boldsymbol {p} = \underline {{\gamma}} ^ {- 1} \left(\boldsymbol {x} _ {i \text {o p t} _ {i}} - \boldsymbol {x} _ {i t}\right) ^ {\top} \boldsymbol {W} ^ {\top} \left(\boldsymbol {W} ^ {\top}\right) ^ {\dagger} \boldsymbol {v} \\ = \underline {{\gamma}} ^ {- 1} \left(\boldsymbol {x} _ {i \operatorname {o p t} _ {i}} - \boldsymbol {x} _ {i t}\right) ^ {\top} \boldsymbol {v} \geq 1, \\ \end{array}
$$
which together with the constraints in (ATT-SVM) completes the proof.
C Addendum to Section 3
C.1 Proof of Theorem 5
Proof. Suppose the claim is incorrect and either  $\pmb{p}_R / R$  or  $\pmb{v}_r / r$  fails to converge as  $R, r$  grows. Set  $\Xi = 1 / \| \pmb{p}^{mm} \|$ ,  $\Gamma = 1 / \| \pmb{v}^{mm} \|$ ,  $\tilde{\pmb{p}}^{mm} = R \Xi \pmb{p}^{mm}$  and  $\tilde{\pmb{v}}^{mm} = r \Gamma \pmb{v}^{mm}$ . The proof strategy is obtaining a contradiction by proving that  $(\tilde{\pmb{v}}^{mm}, \tilde{\pmb{p}}^{mm})$  is a strictly better solution compared to  $(\pmb{v}_r, \pmb{p}_R)$  for large  $R, r$ . Without losing generality, we will set  $\alpha_i = 1$  for all  $i \in [n]$  as the problem is invariant to tokens' permutation. Define  $q_i^p = 1 - s_{i1}^p$  to be the amount of non-optimality (cumulative probability of non-first tokens) where  $s_i^p = \mathbb{S}(\pmb{K}_i\pmb{p})$  is the softmax probabilities.
- Case 1:  $p_R / R$  does not converge. Under this scenario there exists  $\delta, \gamma = \gamma(\delta) > 0$  such that we can find arbitrarily large  $R$  with  $\| p_R / R - \tilde{p}^{mm} / R \| \geq \delta$  and margin induced by  $p_R / R$  is at most  $\Xi(1 - \gamma)$  (from strong convexity of (ATT-SVM)). Following  $q_i^p$  definition above, set  $\hat{q}_{\max} = \sup_{i \in [n]} q_i^{p_R}$  to be
worst non-optimality in  $\pmb{p}_R$  and  $q_{\mathrm{max}}^{\star} = \sup_{i\in [n]}q_i^{\tilde{\pmb{p}}^{mm}}$  to be the same for  $\tilde{\pmb{p}}^{mm}$ . Repeating the identical argument in Theorem 8 (specifically (84)), we can bound the non-optimality amount  $q_{i}^{\tilde{p}^{mm}}$  of  $\tilde{\pmb{p}}^{mm}$  as
$$
q _ {i} ^ {\tilde {\boldsymbol {p}} ^ {m m}} = \frac {\sum_ {t \neq \alpha_ {i}} \exp \left(\boldsymbol {k} _ {i t} ^ {\top} \tilde {\boldsymbol {p}} ^ {m m}\right)}{\sum_ {t \in [ T ]} \exp \left(\boldsymbol {k} _ {i t} ^ {\top} \tilde {\boldsymbol {p}} ^ {m m}\right)} \leq \frac {\sum_ {t \neq \alpha_ {i}} \exp \left(\boldsymbol {k} _ {i t} ^ {\top} \tilde {\boldsymbol {p}} ^ {m m}\right)}{\exp \left(\boldsymbol {k} _ {i \alpha_ {i}} ^ {\top} \tilde {\boldsymbol {p}} ^ {m m}\right)} \leq T \exp (- R \Xi). \tag {71}
$$
Thus,  $q_{\max}^{\star} = \max_{i \in [n]} q_i^{\tilde{p}^{mm}} \leq T \exp(-R\Xi)$ . Next without losing generality, assume first margin constraint is  $\gamma$ -violated by  $p_R$  and  $\min_{t \neq \alpha_1} (k_{1\alpha_1} - k_{1t})^\top p_R \leq \Xi R(1 - \gamma)$ . Denoting the amount of non-optimality of the first input as  $q_1^{p_R}$ , we find
$$
q _ {1} ^ {\boldsymbol {p} _ {R}} = \frac {\sum_ {t \neq \alpha_ {1}} \exp \left(\boldsymbol {k} _ {1 t} ^ {\top} \boldsymbol {p} _ {R}\right)}{\sum_ {t \in [ T ]} \exp \left(\boldsymbol {k} _ {1 t} ^ {\top} \boldsymbol {p} _ {R}\right)} \geq \frac {1}{T} \frac {\sum_ {t \neq \alpha_ {1}} \exp \left(\boldsymbol {k} _ {1 t} ^ {\top} \boldsymbol {p} _ {R}\right)}{\exp \left(\boldsymbol {k} _ {1 \alpha_ {1}} ^ {\top} \boldsymbol {p} _ {R}\right)} \geq T ^ {- 1} \exp (- (1 - \gamma) R \Xi). \tag {72}
$$
We similarly have  $q_{\mathrm{max}}^{\star} \geq T^{-1} \exp(-R\Xi)$  to find that
$$
\log (\hat {q} _ {\max }) \geq - (1 - \gamma) \Xi R - \log T,
$$
$$
- \Xi R - \log T \leq \log \left(q _ {\max } ^ {\star}\right) \leq - \Xi R + \log T. \tag {73}
$$
In words,  $\tilde{\pmb{p}}^{mm}$  contains exponentially less non-optimality compared to  $\pmb{p}_R$  as  $R$  grows. The remainder of the proof differs from Theorem 8 as we need to upper/ lower bound the logistic loss of  $(\tilde{\nu}^{mm},\tilde{\pmb{p}}^{mm})$  and  $(\nu_{r},\pmb{p}_{R})$  respectively to conclude with the contradiction.
First, let us upper bound the logistic loss of  $(\tilde{\pmb{v}}^{mm},\tilde{\pmb{p}}^{mm})$ . Set  $\boldsymbol {r}_i = X_i^\top \mathbb{S}(\pmb {K}_i\tilde{\pmb{p}}^{mm})$ . Observe that if  $\| \pmb {r}_i - \pmb {x}_{i1}\| \leq \epsilon_i$ , we have that  $\pmb{v}^{mm}$  satisfies the SVM constraints on  $\boldsymbol {r}_i$  with  $Y_{i}\cdot \boldsymbol{r}_{i}^{\top}\boldsymbol{v}^{mm}\geq 1-$ $\epsilon_i / \Gamma$ . Consequently, setting  $\epsilon_{\mathrm{max}} = \sup_{i\in [n]}\epsilon_{i}$ ,  $\pmb{v}^{mm}$  achieves a label-margin of  $\Gamma -\epsilon_{\mathrm{max}}$  on the dataset  $(Y_{i},\boldsymbol {r}_{i})_{i\in [n]}$ . With this, we upper bound the logistic loss of  $(\tilde{\pmb{v}}^{mm},\tilde{\pmb{p}}^{mm})$  as follows. Let  $M = \sup_{i\in [n],t,\tau \in [T]}\| x_{it} - x_{i\tau}\|$ . Let us recall the fact (73) that worst-case perturbation is
$$
\epsilon_ {\max } \leq M \exp (- \Xi R + \log T) = M T \exp (- \Xi R).
$$
$$
\begin{array}{l} \mathcal {L} \left(\tilde {\boldsymbol {v}} ^ {m m}, \tilde {\boldsymbol {p}} ^ {m m}\right) \leq \max  _ {i \in [ n ]} \log \left(1 + \exp \left(- Y _ {i} \boldsymbol {r} _ {i} ^ {\top} \tilde {\boldsymbol {v}} ^ {m m}\right)\right). \\ \leq \max  _ {i \in [ n ]} \exp (- Y _ {i} \boldsymbol {r} _ {i} ^ {\top} \tilde {\boldsymbol {v}} ^ {m m}) \\ \leq \exp (- r \Gamma + r \epsilon_ {\max }) \\ \leq e ^ {r M T \exp (- \Xi R)} e ^ {- r \Gamma}. \tag {74} \\ \end{array}
$$
Conversely, we obtain a lower bound for  $(\pmb{v}_r,\pmb{p}_R)$ . Set  $\pmb{r}_i = \pmb{X}_i^\top \mathbb{S}(\pmb{K}_i\pmb{p}_R)$ . Using Assumption C, we find that solving (SVM) on  $(Y_{i},\pmb{r}_{i})_{i\in [n]}$  achieves at most  $\Gamma -\nu e^{-(1 - \gamma)\Xi R} / T$  margin. Consequently, we have
$$
\begin{array}{l} \mathcal {L} \left(\boldsymbol {v} _ {r}, \boldsymbol {p} _ {R}\right) \geq \frac {1}{n} \max  _ {i \in [ n ]} \log \left(1 + \exp \left(- Y _ {i} \boldsymbol {r} _ {i} ^ {\top} \boldsymbol {v} _ {r}\right)\right) \\ \geq \frac {1}{2 n} \max  _ {i \in [ n ]} \exp (- Y _ {i} \boldsymbol {r} _ {i} ^ {\top} \boldsymbol {v} _ {r}) \wedge \log 2 \\ \geq \frac {1}{2 n} \exp (- r (\Gamma - v e ^ {- (1 - \gamma) \Xi R} / T)) \wedge \log 2 \\ \geq \frac {1}{2 n} e ^ {r (\nu / T) \exp (- (1 - \gamma) \Xi R)} e ^ {- r \Gamma} \wedge \log 2. \tag {75} \\ \end{array}
$$
Observe that, this lower bound dominates the previous upper bound when  $R$  is large, namely, when (ignoring the multiplier  $1 / 2n$  for brevity)
$$
(\nu / T) e ^ {- (1 - \gamma) \Xi R} \geq M T e ^ {- \Xi R} \Longleftrightarrow R \geq R _ {0} := \frac {1}{\gamma \Xi} \log \left(\frac {M T ^ {2}}{\nu}\right).
$$
Thus, we indeed obtain the desired contradiction since such large  $R$  is guaranteed to exist when  $p_R / R \to p^{mm}$ .
- Case 2:  $\nu_r / r$  does not converge. This is the simpler scenario: There exists  $\delta > 0$  such that we can find arbitrarily large  $r$  obeying  $\| \pmb{v}_r / r - \pmb{v}^{mm} / \| \pmb{v}^{mm} \| \| \geq \delta$ . If  $\| \pmb{p}_R / R - \Xi \pmb{p}^{mm} \| \to 0$ , then "Case
1" applies. Otherwise, we have  $\| \pmb{p}_R / R - \Xi \pmb{p}^{mm}\| \to 0$ , thus we can assume  $\| \pmb{p}_R / R - \Xi \pmb{p}^{mm}\| \leq \epsilon$  for arbitrary choice of  $\epsilon > 0$ .
On the other hand, due to the strong convexity of (SVM), for some  $\gamma \coloneqq \gamma (\delta) > 0$ ,  $\nu_{r}$  achieves a margin of at most  $(1 - \gamma)\Gamma r$  on the dataset  $(Y_{i},x_{i1})_{i\in [n]}$ . Additionally, since  $\| \pmb{p}_R / R - \Xi \pmb{p}^{mm}\| \leq \epsilon$ ,  $\pmb{p}_R$  strictly separates all optimal tokens (for small enough  $\epsilon >0$ ) and  $\hat{q}_{\mathrm{max}}\coloneqq f(\epsilon)\to 0$  as  $R\to \infty$ . Consequently, setting  $\boldsymbol {r}_i = \boldsymbol {X}_i^\top \mathbb{S}(\boldsymbol {K}_i\boldsymbol {p}_R)$ , for sufficiently large  $R > 0$  setting  $M = \sup_{i\in [n],t\in [T]}\| x_{it}\|$ , we have that
$$
\begin{array}{l} \min  _ {i \in [ n ]} Y _ {i} \boldsymbol {v} _ {r} ^ {\top} \boldsymbol {r} _ {i} \leq \min  _ {i \in [ n ]} Y _ {i} \boldsymbol {v} _ {r} ^ {\top} \boldsymbol {x} _ {i 1} + \sup  _ {i \in [ n ]} | \boldsymbol {v} _ {r} ^ {\top} (\boldsymbol {r} _ {i} - \boldsymbol {x} _ {i 1}) | \\ \leq (1 - \gamma) \Gamma r + M f (\epsilon) r \\ \leq (1 - \gamma / 2) \Gamma r. \tag {76} \\ \end{array}
$$
This in turn implies that logistic loss is lower bounded by (following (75)),
$$
\mathcal {L} \left(\boldsymbol {v} _ {r}, \boldsymbol {p} _ {R}\right) \geq \frac {1}{2 n} e ^ {\gamma \Gamma r / 2} e ^ {- \Gamma r} \wedge \log 2.
$$
Going back to (74), this exponentially dominates the upper bound of  $(\tilde{p}^{mm},\tilde{\nu}^{mm})$  whenever  $rMT\exp (-\Xi R) < r\gamma \Gamma /2$ , (that is, whenever  $R,r$  are sufficiently large), again concluding the proof.
C.2 Proof of Theorem 6
We will prove this result in two steps. Our first claim restricts the optimization to the particular quadrant induced by  $\min_{t\neq \alpha_i}(\pmb {k}_{i\alpha_i} - \pmb {k}_{it})^\top \pmb {p}_R\geq 0$  under the theorem's condition  $\mathbb{S}(\pmb {K}_i\pmb {p}_R)_{\alpha_i}\to 1$
Lemma 12 Suppose  $\mathbb{S}(K_i p_R)_{\alpha_i} \to 1$ . Then, there exists  $R_0$  such that for all  $R \geq R_0$ , we have that,
$$
\min  _ {t \neq \alpha_ {i}} \left(\boldsymbol {k} _ {i \alpha_ {i}} - \boldsymbol {k} _ {i t}\right) ^ {\top} \boldsymbol {p} _ {R} \geq 0, \quad \text {f o r a l l} \quad i \in [ n ]. \tag {77}
$$
Proof. Suppose the claim does not hold. Set  $s_i^R = \mathbb{S}(K_i p_R)$ . Fix  $R_0$  such that  $s_{i\alpha_i}^R \geq 0.9$  for all  $R \geq R_0$ . On the other hand, there exists arbitrarily large  $R$  for which  $(k_{i\alpha_i} - k_{it})^\top p_R < 0$  for some  $t \neq \alpha_i \in [T]$  and  $i \in [n]$ . At this  $(R,i,t)$  choices, we have that  $s_{it}^R \geq s_{i\alpha_i}^R$ . Since  $s_{it}^R + s_{i\alpha_i}^R \leq 1$ , we find  $s_{i\alpha_i}^R < 0.5$  which contradicts with  $s_{i\alpha_i}^R \geq 0.9$ .
Let  $Q$  be the set of  $\pmb{p}$  satisfying the quadrant constraint (77) - i.e. indices  $(\alpha_{i})_{i = 1}^{n}$  are selected. Let  $\pmb{h}_R$  be the solution of regularization path of  $(\nu ,\pmb {p})$  subject to the constraint  $\pmb {p}\in Q$  . From Lemma 12, we know that, for some  $R_0$  and all  $R\geq R_0$ $\pmb {h}_R = \pmb {p}_R$  . Thus, if the limit exists, we have that  $\lim_{R\to \infty}\pmb {h}_R / R = \lim_{R\to \infty}\pmb {p}_R / R.$
To proceed, we will prove that  $\lim_{R\to \infty}\pmb{h}_R / R$  exists and is equal to  $\pmb{p}^{\text{relax}} / \| \pmb{p}^{\text{relax}}\|$  and simultaneously establish  $\nu_r / r\rightarrow \nu^{mm} / \| \nu^{mm}\|$ .
Lemma 13  $\lim_{R\to \infty}\pmb {h}_R / R = \pmb{p}^{\text{relax}} / \| \pmb{p}^{\text{relax}}\|$  and  $\lim_{r\to \infty}\nu_r / r = \nu^{mm} / \| \nu^{mm}\|$ .
Proof. The proof will be similar to that of Theorem 5. As usual, we aim to show that SVM-solutions constitute the most competitive direction. Set  $\Xi = 1 / \| p^{relax}\|$ .