Benign Overfitting in Single-Head Attention

## Abstract

Abstract The phenomenon of benign overfitting , where a trained neural network perfectly fits noisy training data but still achieves near-optimal test performance, has been extensively studied in recent years for linear models and fully-connected/convolutional networks. In this work, we study benign overfitting in a single-head softmax attention model, which is the fundamental building block of Transformers. We prove that under appropriate conditions, the model exhibits benign overfitting in a classification setting already after two steps of gradient descent. Moreover, we show conditions where a minimum-norm/maximum-margin interpolator exhibits benign overfitting. We study how the overfitting behavior depends on the signal-to-noise ratio (SNR) of the data distribution, namely, the ratio between norms of signal and noise tokens, and prove that a sufficiently large SNR is both necessary and sufficient for benign overfitting.

## 1 Introduction

Neural networks often exhibit a remarkable phenomenon, known as *benign overfitting*, where they achieve a perfect fit to noisy training examples and still generalize well to unseen data [, ]. This phenomenon contradicts classical wisdom in machine learning, and has become a central research question in the theory of deep learning.
Existing works on benign overfitting study under what conditions the phenomenon occurs in different architectures. These works focus on linear models, and on shallow fully-connected and convolutional neural networks.

In recent years, Transformers [] have emerged as a leading neural network architecture, with impactful applications across a wide range of domains such as natural language processing and computer vision. The fundamental building block of Transformers is the attention mechanism, which allows them to process sequences and focus different parts of the input. Despite the central role of the attention mechanism, we currently do not understand their overfitting behavior and the conditions under which they exhibit benign overfitting.

In this work, we show the first benign-overfitting results for the attention mechanism. We consider classification with a single-head softmax attention model, and study the conditions that allow for benign overfitting.
In our results, the data distribution consists of two tokens: a *signal token*, which can be used for correctly classifying clean test examples, and a *noise token*, which is independent of the label but can be used for interpolating (i.e., perfectly fitting) noisy training examples. We study the singnal-to-noise ratio (SNR), namely, the expected ratio between the norms of signal and noise tokens, that allows for benign overfitting.

Below we summarize our main contributions:

- •
In Theorem [4](#Thmtheorem4) ([Section 3](#S3)) we show that under appropriate conditions, gradient descent with the logistic loss exhibits benign overfitting already after two iterations. This result holds when the SNR is $\Theta(1/\sqrt{n})$, where $n$ is the number of training samples.
- •
We then turn to consider other natural learning rules, which allow for benign overfitting under a weaker requirement on the SNR. In Theorems [6](#Thmtheorem6) and [8](#Thmtheorem8) ([Section 4](#S4)), we prove that minimum-norm (i.e., maximum-margin) interpolators exhibit benign overfitting when the SNR is $\Omega(1/\sqrt{n})$ without requiring an upper bound on the SNR.
- •
In Theorem [10](#Thmtheorem10) ([Section 4](#S4)), we prove that the above requirement on the SNR is tight. Namely, if the SNR is smaller than it, then the min-norm interpolator exhibits harmful overfitting, where it fits the training data but has poor generalization performance.
- •
In Section [6](#S6), we complement our theoretical results with an empirical study. We show that a sufficiently large SNR and input dimension are necessary to achieve benign overfitting.

The paper is structured as follows. In Section [2](#S2), we provide some preliminaries and define the data distribution and the single-head attention model. In Sections [3](#S3) and [4](#S4) we state our main results on benign overfitting with gradient descent and with min-norm interpolators. In Section [5](#S5) we discuss the main proof ideas, with all formal proofs deferred to the appendix. Finally, in Section [6](#S6) we show empirical results.

### Related Work

Optimization in Transformers. [] provided a theoretical analysis of training a shallow Vision Transformer (ViT) for a classification task. They showed that the sample complexity required to achieve a zero generalization error is correlated with the inverse of the fraction of label-relevant tokens, the token noise level, and the initial model error.
[] showed that optimizing the attention layer via gradient descent leads to convergence to an SVM solution, where the implicit bias of the attention mechanism depends on whether the parameters are represented as a product of key-query matrices or directly as a combined matrix, with different norm-minimization objectives in each case.
[] provided a regularization path analysis and prove that the attention weights converge in a direction to a max-margin solution that separates locally optimal tokens from non-optimal. They also showed that running gradient descent, with a specific initialization direction and without optimizing the attention head, converges in a direction to the same max-margin solution.
[] expanded on their findings by identifying non-trivial data settings for which the convergence of GD is provably global, i.e., without requiring assumptions about the initialization direction. They also provided convergence rate bounds and analysis for optimizing both the attention weights and the attention head, although they did not consider the case of noisy data labels, as we do in our work. Another line of work looks at the learning dynamics of single-layer linear attention models trained on linear regression tasks [, , ].
Additional works that consider optimization dynamics in Transformers include [, ].

Benign overfitting.
A significant body of research has explored why neural networks (NNs) that perfectly interpolate the training data can still generalize well [, ]. This has sparked substantial interest in studying overfitting and generalization in NNs trained to fit datasets with noisy labels.
The literature on benign overfitting is broad and cannot be reasonably covered here. We refer the reader to the surveys [, ].
Most relevant to our work are [, , ] that studied benign overfitting in convolutional neural networks. Their data distribution resembles ours, as we discuss in Section [2.1](#S2.SS1). Benign overffiting in fully-connected two-layer neural network classification was studied in [, , , , , , ] for various activation functions, data distributions and loss functions (both the logistic and the hinge losses).

## 2 Preliminaries

Notations. We use bold-face letters to denote vectors and matrices, and let $[m]$ be shorthand for $\{1,2,\dots,m\}$. Given a vector $\bm{x}$, we denote by $x_{j}$ its $j$-th coordinate. Let $\bm{I}_{d}$ be the $d\times d$ identity matrix, and let ${\bm{0}}_{d}$ (or just ${\bm{0}}$, if $d$ is clear from the context) denote the zero vector in $\mathbb{R}^{d}$. We let $\left\|\cdot\right\|$ denote the Euclidean norm. We denote a multivariate Gaussian
distribution with mean vector $\bm{\mu}$ and covariance matrix ${\bm{\Sigma}}$ by $N(\bm{\mu},{\bm{\Sigma}})$.
We use standard big-Oh notation, with $\Theta(\cdot),\Omega(\cdot),O(\cdot)$ hiding universal constants and $\tilde{\Theta}(\cdot),\tilde{\Omega}(\cdot),\tilde{O}(\cdot)$ hiding constants and factors that are polylogarithmic in the problem parameters. We use $\mathbb{I}(\cdot)$ to denote the indicator variable of an event. For a finite set $\mathcal{A}$, denote the uniform distribution over $\mathcal{A}$ by $\mathsf{Unif}(\mathcal{A})$ and let $|\mathcal{A}|$ be its cardinality.

### 2.1 Data Generation Setting

In this work we focus on the following data distribution:

###### Definition 1 .

Let $\bm{\mu}_{1},\bm{\mu}_{2}\in\mathbb{R}^{d}$ such that $\left\|\bm{\mu}_{1}\right\|=\left\|\bm{\mu}_{2}\right\|=\rho$
for some $\rho>0$
and $\langle\bm{\mu}_{1},\bm{\mu}_{2}\rangle=0$, be two fixed orthogonal vectors representing the signal contained in each data point. Define $\mathcal{D}_{\text{clean}}$ as the distribution over $\mathbb{R}^{2\times d}\times\{\pm 1\}$ of labelled data such that a data point $(\bm{X},\widetilde{y})$ is generated by the following procedure:

- 1.
Sample the label $\widetilde{y}\sim\mathsf{Unif}\{\pm 1\}$.
- 2.
Generate
a vector $\bm{u}$, which represents the signal,
as follows:
If $\widetilde{y}=+1$, set $\bm{u}=\bm{\mu}_{1}$; and if $\widetilde{y}=-1$, set $\bm{u}=\bm{\mu}_{2}$.
- 3.
Generate a vector $\bm{\xi}$, which represents the noise, from the Gaussian distribution $\bm{\xi}\sim\mathcal{N}({\bm{0}},\bm{I}_{d}-\bm{\mu}_{1}\bm{\mu}_{1}^{\top}/\rho^{2}-\bm{\mu}_{2}\bm{\mu}_{2}^{\top}/\rho^{2})$.
- 4.
Denote $\bm{X}=(\bm{x}^{(1)},\bm{x}^{(2)})^{\top}$. Select $k\sim\mathsf{Unif}\{1,2\}$ and set $\bm{x}^{(k)}=\bm{u}$. Set the other token $\bm{x}^{(3-k)}=\bm{\xi}$.

To study the overfitting behavior we also need to introduce label-flipping noise:

###### Definition 2 .

Let $\eta\in[0,1/2)$ be the label flipping probability.
We define $\mathcal{D}$ as the distribution over $\mathbb{R}^{2\times d}\times\{\pm 1\}$ which is the $\eta$-label-flipped version of $\mathcal{D}_{\text{clean}}$. Namely, to generate $(\bm{X},y)\sim\mathcal{D}$, first generate $(\bm{X},\widetilde{y})\sim\mathcal{D}_{\text{clean}}$, then let $y=\widetilde{y}$ with probability $1-\eta$ and $y=-\widetilde{y}$ with probability $\eta$.

Our data
distribution resembles the distributions considered by [, , ]. They proved benign overfitting in two-layer convolutional neural networks, and in their setting each data point consists of two patches $\bm{x}^{(1)},\bm{x}^{(2)}$ (rather than two tokens in our setting).
Since our single-head attention model is invariant to the order of the tokens, we assume without loss of generality throughout this work that $\bm{x}^{(1)}$ is the signal token and $\bm{x}^{(2)}$ is the noise token in all data points.
Note that the noise token $\bm{x}^{(2)}=\bm{\xi}$ is independent of the label, and that it is generated from $\mathcal{N}({\bm{0}},\bm{I}_{d}-\bm{\mu}_{1}\bm{\mu}_{1}^{\top}/\rho^{2}-\bm{\mu}_{2}\bm{\mu}_{2}^{\top}/\rho^{2})$, ensuring that it is orthogonal to the signal vector. Note that when the dimension $d$ is large, $\|\bm{\xi}\|\approx\sqrt{d-2}\approx\sqrt{d}$ by standard concentration bounds. Therefore, we denote the signal-to-noise ratio (SNR) as $\mathrm{SNR}=\|\bm{\mu}\|/\sqrt{d}=\rho/\sqrt{d}$.

We consider a training dataset $\{(\bm{X}_{i},y_{i})\}^{n}_{i=1}$ of $n$ samples generated i.i.d. from the distribution $\mathcal{D}$.
Denote the index set of data whose labels are not flipped by $\mathcal{C}=\{i:\widetilde{y}_{i}=y_{i}\}$ (“clean examples”), and the index set of data whose labels are flipped by $\mathcal{N}=\{i:\widetilde{y}_{i}=-y_{i}\}$ (“noisy examples”). For indices in $\mathcal{C}$, we further denote
$\mathcal{C}_{1}:=\mathcal{C}\cap\{i:\bm{x}_{i}^{(1)}=\bm{\mu}_{1}\},\mathcal{C}_{2}:=\mathcal{C}\cap\{i:\bm{x}_{i}^{(1)}=\bm{\mu}_{2}\},$
and define the subsets $\mathcal{N}_{1}$, $\mathcal{N}_{2}$ of $\mathcal{N}$ analogously.

###### Definition 1 .

###### Definition 2 .

### 2.2 Single-Head Attention Model

We consider the following single-head attention model:

$$ $f(\bm{X};\bm{W},\bm{p})=\bm{v}^{\top}\bm{X}^{\top}\mathbb{S}(\bm{X}\bm{W}\bm{q})~{},$ $$

where $\mathbb{S}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ is the softmax function, the key-query matrix $\bm{W}\in\mathbb{R}^{d\times d}$ and the linear head vector $\bm{v}\in\mathbb{R}^{d}$ are the trainable parameters, and the query vector $\bm{q}\in\mathbb{R}^{d}$ is an arbitrary fixed unit vector.
We follow [] and assume
that $\bm{q}=(1,0,\ldots,0)^{\top}$, obtaining the following model:

$$ $\displaystyle f(\bm{X};\bm{p},\bm{v})=\bm{v}^{\top}\bm{X}^{\top}\mathbb{S}(\bm{X}\bm{p}),$ (2.1) $$

Here the trained parameters are $\bm{p},\bm{v}\in\mathbb{R}^{d}$. Thus, instead of the key-query matrix $\bm{W}$ we have a vector $\bm{p}$ that controls the attention.
Throughout this paper we will use the model ([2.1](#S2.E1)).
We denote the output of the softmax layer $\mathbb{S}(\bm{X}_{i}\bm{p})$ by $\bm{s}_{i}=(s_{i,1},s_{i,2})^{\top}$, and
denote the output of the attention layer $\bm{X}_{i}^{\top}\bm{s}_{i}$ by $\bm{r}_{i}=s_{i,1}\bm{\mu}_{i}+s_{i,2}\bm{\xi}_{i}$, where $0\leq s_{i,1},s_{i,2}\leq 1$, $s_{i,1}+s_{i,2}=1$ are the attention on two tokens of the $i$-th sample.

## 3 Benign Overfitting with Gradient Descent

In this section, we study the joint optimization of the head $\bm{v}$ and attention weights $\bm{p}$ using the logistic loss function. We show that the model exhibits benign overfitting after just two iterations of gradient descent (GD).

Formally, for a training dataset $\{(\bm{X}_{i},y_{i})\}^{n}_{i=1}$ we define the empirical risk as

$$ $\displaystyle\mathcal{L}(\bm{v},\bm{p})=\frac{1}{n}\sum_{i=1}^{n}\ell(y_{i}\cdot f(\bm{X}_{i};\bm{p},\bm{v})),$ $$

where $\ell(z)=\log(1+\exp(-z))$ is the logistic loss function, and $f$ is the model from Eq. ([2.1](#S2.E1)). We consider GD optimization. Starting from $\bm{p}_{0}={\bm{0}}$ and $\bm{v}_{0}={\bm{0}}$, we have

$$ $\displaystyle\bm{v}_{t+1}=\bm{v}_{t}-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{t},\bm{p}_{t})\quad\quad\text{and}\quad\quad\bm{p}_{t+1}=\bm{p}_{t}-\mathbf{\beta}\nabla_{\bm{p}}\mathcal{L}(\bm{v}_{t},\bm{p}_{t}),$ $$

where $\mathbf{\beta}$ is the step size.
When we discuss some fixed $t$, we sometimes write in the subscript “$t=\cdot$”, e.g., $\bm{p}_{t=2}$ instead of $\bm{p}_{2}$.
We make the following assumptions:

###### Assumption 3 (Assumptions for GD with SNR = $\Theta(1/\sqrt{n})$ ) .

Let $\delta\in(0,0.5)$ be a desired probability of failure.
For universal constants $C_{\rho}\geq 6,C_{\beta}\geq 16$, as well as a sufficiently large universal constant $C$ that may depend on $C_{\rho}$ and $C_{\beta}$, the following conditions hold:

- 1.
Number of samples $n$ is sufficiently large: $n\geq C\log(1/\delta)$.
- 2.
Dimension $d$ is sufficiently large: $d\geq Cn^{2}\log(n/\delta)$.
- 3.
Signal strength satisfies $\rho=C_{\rho}\cdot\sqrt{d/n}$
- 4.
Label flipping rate satisfies $\eta\leq 1/C$.
- 5.
Step size satisfies $\beta=C_{\beta}\cdot(n/d)$.
- 6.
Initialization at zero: $\left\|\bm{v}_{0}\right\|=\left\|\bm{p}_{0}\right\|=0$.

Item [1](#S3.I1.i1)
is required to estimate the number of clean examples compared to noisy examples. The assumption of high dimensionality (Item [2](#S3.I1.i2)) is important
for enabling benign overfitting (see empirical results in Section [6](#S6)), and implies that noise tokens from different training samples are nearly-orthogonal.
This assumption appears in many prior works on benign overfitting in neural network classification (e.g., [, , , , , , , ]).
Item [3](#S3.I1.i3) states that the signal-to-noise ratio (SNR) is $\tfrac{\rho}{\sqrt{d}}=\Theta(1/\sqrt{n})$. In Section [5](#S5) we will discuss how the SNR affects the dynamics of GD.
Interestingly, SNR of $\Theta(1/\sqrt{n})$ matches the lower bound of the required SNR that allows for benign overfitting with the min-norm (i.e. max-margin) learning rule that we will study in Section [4](#S4).
Item [4](#S3.I1.i4) ensures the flipping rate is small enough to allow the model to learn the signal token.
Item [5](#S3.I1.i5), namely, using a step size of $\Theta(n/d)$, is required to achieve benign overfitting after two iterations; with a smaller step size, the model will need more iterations to fit the noisy samples, which we will demonstrate empirically in Section [6](#S6).

We now state our main result on benign overfitting with GD:

###### Theorem 4 .

Suppose that Assumption [3](#Thmtheorem3) holds. Then, with probability at least $1-\delta$ over the training dataset, after two iterations of GD we have:

- •
Higher softmax probability for optimal tokens:
$\displaystyle s_{i,1}^{t=2}>1/2,\;\forall i\in\mathcal{C}\quad\quad\text{and}\quad\quad s_{i,2}^{t=2}\geq 1-1/c_{\rho}^{2},\;\forall i\in\mathcal{N}$
where
$s_{i,j}^{t}$ is the softmax probability of the $j^{\text{th}}$ token in the $i^{\text{th}}$ sample at time $t$.
- •
The classifier $\bm{X}\mapsto\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2}))$ correctly classifies all training data points:
$y_{i}=\mathrm{sign}(f(\bm{X}_{i};\bm{v}_{t=2},\bm{p}_{t=2})),\ \forall i\in[n].$
- •
The classifier $\bm{X}\mapsto\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2})$ generalizes well:
$\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}(y\neq\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2})))\leq\eta+\exp(-d/C_{1}n^{2}),$
where $C_{1}:=C_{1}(c_{\rho},c_{\mathbf{\beta}})$ is a constant.

We can also conclude that for the clean-labeled distribution $\mathcal{D}_{\text{clean}}$ we have

$$ $\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}_{\text{clean}}}(y\neq\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2})))\leq\exp(-d/C_{1}n^{2}),$ $$

which approaches zero as $d$ grows (see Assumption [3](#Thmtheorem3), item [2](#S3.I1.i2)).

[Theorem 4](#Thmtheorem4) shows that after two iterations of GD, the attention focuses on the signal tokens for clean examples, and on the noise tokens for noisy examples. The model uses the noise tokens for interpolating noisy training examples, while still achieving good generalization performance using the signal token.

###### Assumption 3 (Assumptions for GD with SNR = Θ ​ ( 1 / n ) Θ 1 𝑛 \Theta(1/\sqrt{n}) ) .

###### Theorem 4 .

## 4 Benign Overfitting of Max-Margin Solution

In the previous section we showed that GD exhibits benign overfitting in a setting where the SNR is $\Theta(1/\sqrt{n})$.
We now turn to study the overfitting behavior of single-head attention models, when using another learning rule, which returns solutions that interpolate the training data with large margin while keeping the parameters norms small. As we will show, such a learning rule allows us to obtain benign overfitting under a weaker requirement on the SNR, namely, the SNR is $\Omega(1/\sqrt{n})$ without requiring an upper bound on it.

We note that learning rules that return min-norm (or max-margin) solutions are considered natural, and hence understanding properties of min-norm interpolators has attracted much interest in recent years, even in settings where the implicit bias of GD does not necessarily lead to a min-norm solution (see, e.g., [, , , , , ]). More directly related to our work, min-norm interpolation with Transformers has been studied in [, ], and
benign/tempered overfitting in min-norm univariate neural network interpolators has been studied in [].

We first consider the following learning rule:

$$ $(\bm{v}_{(r,R)},\bm{p}_{(r,R)})=\mathop{\mathrm{argmax}}\limits_{\|\bm{v}\|\leq r,\|\bm{p}\|\leq R}\min_{i\in[n]}y_{i}\cdot f(\bm{X}_{i};\bm{p},\bm{v})~{},$ (4.1) $$

where $f$ is the model from ([2.1](#S2.E1)). The learning rule returns a solution that maximizes the margin $\min_{i\in[n]}y_{i}\cdot f(\bm{X}_{i};\bm{p},\bm{v})$ under a restriction on the parameter norms.
We make the following assumption:

###### Assumption 5 (Assumptions for max-margin with SNR $=\Omega(1/\sqrt{n})$ ) .

Let $\delta\in(0,0.5)$ be a desired probability of failure. There exists a sufficiently large constant $C$ such that the following hold:

- 1.
Dimension $d$ is sufficiently large: $d\geq Cn^{2}\log(n/\delta)$.
- 2.
Number of samples $n$ is sufficiently large: $n\geq C\log(1/\delta)$.
- 3.
Signal strength: $\rho\geq C\sqrt{d/n}$.
- 4.
Label flipping rate: $0\leq\eta\leq 1/C$.
- 5.
Norm constraint of $\bm{p}$ satisfies: $R\geq C\sqrt{\eta n/d+1/\rho^{2}}\log(\rho n)$.

Items [1](#S4.I1.i1), [2](#S4.I1.i2) and [4](#S4.I1.i4) are similar to Assumption [3](#Thmtheorem3).
Item [3](#S4.I1.i3)
requires $\mathrm{SNR}\geq\Omega(1/\sqrt{n})$, which is a weaker requirement than the $\Theta(1/\sqrt{n})$ requirement in Assumption [3](#Thmtheorem3). We will show later a lower bound on the required SNR for benign overfitting, implying that the $\Omega(1/\sqrt{n})$ bound is tight.
Item [5](#S4.I1.i5) provides the lower bound for the norm constraint of $\bm{p}$ so that the model can allocate enough attention on signal token to achieve benign overfitting. Note that the norm constraint $r$ for $\bm{v}$ can take any positive value. Intuitively, since the model is linear in $\bm{v}$, once $\bm{p}$ is properly learned, $\bm{v}$ can achieve accurate classification even with a small norm.

With these assumptions in place, we give our result on benign overfitting with the learning rule ([4.1](#S4.E1)).

###### Theorem 6 .

Suppose that Assumption [5](#Thmtheorem5) holds,
and consider the classifier $\bm{X}\rightarrow\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)}))$, where $(\bm{v}_{(r,R)},\bm{p}_{(r,R)})$ is the solution to Problem ([4.1](#S4.E1)).
Then,
with probability at least $1-\delta$ over the training dataset, we have:

- •
The classifier $\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)}))$ correctly classifies all training data points:
$y_{i}=\mathrm{sign}(f(\bm{X}_{i};\bm{p}_{(r,R)},\bm{v}_{(r,R)})),\ \forall i\in[n].$
- •
The classifier $\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)}))$ generalizes well on test data:
$\displaystyle\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}$
$\displaystyle(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)})))$
$\displaystyle\leq\eta+\exp(-\Omega(d/n^{2}))+\exp\Big{(}-\Omega\big{(}\frac{(1-\zeta)}{\sqrt{\eta n/d+1/\rho^{2}}}-\frac{\log(d)}{R}\big{)}^{2}\Big{)},$
where $\zeta=\Theta(\sqrt{\eta n/d+1/\rho^{2}}\log(\rho n)/R)$.

###### Remark 7 .

To see why Theorem [6](#Thmtheorem6) implies benign overfitting, consider the limit $R\rightarrow\infty$. Then,
the upper bound for test error
becomes $\eta+\exp(-\Omega(d/n^{2}))+\exp(-\Theta((1/\rho^{2}+\eta n/d)^{-1}))$, which can be arbitrarily close to $\eta$ if $d$ is large (see Assumption [5](#Thmtheorem5), item [1](#S4.I1.i1)).

Next, we consider the following learning rule, which explicitly requires to minimize the parameters norms while allowing interpolation with margin at least $\gamma$:

$$ $\displaystyle\left(\bm{v}_{\gamma},\bm{p}_{\gamma}\right)=\mathop{\mathrm{argmin}}_{\left\|\bm{p}\right\|^{2}+\left\|\bm{v}\right\|^{2}}\text{ s.t. }\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{p},\bm{v})\geq\gamma~{},$ (4.2) $$

where $f$ is the model from Eq. ([2.1](#S2.E1)).
We show that under Assumption [5](#Thmtheorem5), the solution $\left(\bm{v}_{\gamma},\bm{p}_{\gamma}\right)$ exhibits benign overfitting for large enough $\gamma$ and $d$:

###### Theorem 8 .

Suppose that Assumption [5](#Thmtheorem5) (items [1](#S4.I1.i1) through [4](#S4.I1.i4)) holds,
and consider the classifier $\bm{X}\rightarrow\mathrm{sign}(f(\bm{X};\bm{p}_{\gamma},\bm{v}_{\gamma}))$, where $(\bm{v}_{\gamma},\bm{p}_{\gamma})$ is a solution of Problem ([4.2](#S4.E2)).
Then there exists $\gamma_{0}$ such that for any $\gamma\geq\gamma_{0}$ , with probability at least $1-\delta$ over the training dataset, we have:

- •
The classifier $\mathrm{sign}(f(\bm{X};\bm{p}_{\gamma},\bm{v}_{\gamma}))$ correctly classifies all training data points:
$y_{i}=\mathrm{sign}(f(\bm{X}_{i};\bm{p}_{\gamma},\bm{v}_{\gamma})),\ \forall i\in[n].$
- •
The classifier $\mathrm{sign}(f(\bm{X};\bm{p}_{\gamma},\bm{v}_{\gamma}))$ generalizes well on test data:
$\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{\gamma},\bm{v}_{\gamma})))\leq\eta+\exp(-\Omega(d/n^{2}))+\exp(-\Theta((1/\rho^{2}+\eta n/d)^{-1})).$

Thus, for large enough $\gamma$, the theorem implies that the trained model interpolates the training data, and the test error approaches $\eta$ as $d\to\infty$.

Note that Theorems [6](#Thmtheorem6) and [8](#Thmtheorem8) hold only when $\mathrm{SNR}=\Omega(1/\sqrt{n})$. This
raises the question: what is the overfitting behavior of min-norm interpolators when the SNR is smaller? We now
consider the case where $\rho\leq\sqrt{1/Cn}$ for some sufficiently large universal constant $C$. We will show that in this case, although the model can correctly classify all training samples, the test error of learning rule ([4.1](#S4.E1)) is at least a universal constant, indicating that benign overfitting does not happen.
Formally, we make the following assumptions:

###### Assumption 9 (Assumptions for max-margin with SNR $\leq O(1/\sqrt{n})$ ) .

Let $\delta\in(0.0.5)$ be a desired probability of failure. There exists a sufficiently large constant $C$ such that the following hold:

- 1.
Dimension $d$ is sufficiently large: $d\geq Cn^{2}\log(n/\delta)$
- 2.
Number of samples $n$ is sufficiently large: $n\geq C\log(1/\delta)$.
- 3.
Signal strength: $\rho\leq\sqrt{d/Cn}$.
- 4.
Label flipping rate is a constant $\eta\in(0,1/2)$.
- 5.
The norm of $\bm{p}$ should be sufficiently large: $R\geq C\sqrt{\frac{n}{d}}\log\big{(}\frac{n\rho}{d}\big{)}$.

Compared with Assumption [5](#Thmtheorem5), the main difference is in the second item that $\mathrm{SNR}\leq O(1/\sqrt{n})$. Additionally, the condition on $\eta$ is relaxed, as in our analysis clean and noisy samples can be treated equivalently when the norm of the signal token is sufficiently small.
With these assumptions in place, we can state the following theorem which characterizes the training error and test error of the single-head attention model when the SNR is small:

###### Theorem 10 .

Suppose that Assumption [9](#Thmtheorem9) holds,
and consider the classifier $\bm{X}\rightarrow\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)}))$, where $(\bm{v}_{(r,R)},\bm{p}_{(r,R)})$ is a solution of Problem ([4.1](#S4.E1)).
Then, with probability at least $1-\delta$ over the training data, we have:

- •
The classifier $\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)}))$ correctly classifies all training data points:
$y_{i}=\mathrm{sign}(f(\bm{X}_{i};\bm{p}_{(r,R)},\bm{v}_{(r,R)})),\ \forall i\in[n].$
- •
The classifier $\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)}))$ does not generalize well on test data:
$\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}_{\text{clean}}}(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)})))\geq\frac{1}{16}.$

###### Assumption 5 (Assumptions for max-margin with SNR = Ω ​ ( 1 / n ) absent Ω 1 𝑛 =\Omega(1/\sqrt{n}) ) .

###### Theorem 6 .

###### Remark 7 .

###### Theorem 8 .

###### Assumption 9 (Assumptions for max-margin with SNR ≤ O ​ ( 1 / n ) absent 𝑂 1 𝑛 \leq O(1/\sqrt{n}) ) .

###### Theorem 10 .

## 5 Proof ideas

In this section we briefly discuss the main proof ideas. The formal proofs are deferred to the appendix.

### 5.1 Proof ideas for Section 3

In this subsection we discuss the main proof idea of Theorem [4](#Thmtheorem4).
Since the initialization is at zero, $\bm{v}_{t}$ is a linear combination of the training data tokens. Therefore, we can express $\bm{v}_{t=1}$ as $\lambda_{1}^{t=1}\bm{\mu}_{1}+\lambda_{2}^{t=1}\bm{\mu}_{2}+\sum_{i=1}^{n}y_{i}\theta_{i}^{t=1}\bm{\xi}_{i}$, where $\lambda_{1}^{t=1}>0,\lambda_{2}^{t=1}<0$. Note that $\lambda_{1}^{t}>0,\lambda_{2}^{t}<0$ holds since $|\mathcal{C}|>|\mathcal{N}|$. We begin by analyzing the first step of GD. Specifically, we show that after one step, the coefficients of $\bm{v}_{t=1}$ can be estimated as $|\lambda_{k}^{t=1}|\approx\frac{\mathbf{\beta}}{8}(1-2\eta),k\in[2]$ and $\theta_{i}^{t=1}=\frac{\beta}{4n},i\in[n]$. Moreover, we have $\bm{p}_{t=1}=0$, and hence for a training sample $(\bm{X}_{j}=(\bm{\mu}_{k},\bm{\xi}_{j}),y_{j})$, the margin is:

$$ $\displaystyle y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ $\displaystyle=\frac{1}{2}y_{j}\bm{v}_{t=1}^{\top}(\bm{x}_{j}^{(1)}+\bm{x}_{j}^{(2)})\approx\frac{1}{2}y_{j}\lambda_{k}^{t=1}\left\|\bm{\mu}_{k}\right\|^{2}+\frac{1}{2}\theta_{j}^{t=1}\left\|\bm{\xi}_{j}\right\|^{2},$ $$

where in the last approximate equality we use the high dimensional setting (i.e. by item [2](#S3.I1.i2) in our assumption $d\gg n^{2}\log(n)$) to neglect the $\sum_{i\in[n]:i\neq j}y_{i}y_{j}\theta_{j}^{t=1}\bm{\xi}_{i}^{\top}\bm{\xi}_{j}$ term, since it is much smaller (in absolute value) than the other terms.
Indeed, we have w.h.p. that
$|\bm{\xi}_{i}^{\top}\bm{\xi}_{j}|\leq\sqrt{d}\log(n)$, $\left\|\bm{\xi}_{j}\right\|^{2}\approx d$ and recall that $\left\|\bm{\mu}_{k}\right\|^{2}=C_{\rho}^{2}(d/n)$ (item [3](#S3.I1.i3) in our assumption). Therefore, for a clean sample $j\in\mathcal{C}$, the margin is $y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})\approx\frac{\mathbf{\beta}(1-2\eta)}{16}\frac{dC_{\rho}^{2}}{n}+\frac{\mathbf{\beta}}{8n}d>0,$
for large enough $C_{\rho}$. On the other hand, for a noisy sample $j\in\mathcal{N}$, we have $y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})\approx-\frac{\mathbf{\beta}(1-2\eta)}{16}\frac{dC_{\rho}^{2}}{n}+\frac{\mathbf{\beta}}{8n}d<0$.
This implies that the classifier $\mathrm{sign}(f(\bm{X};\bm{v}_{t=1},\bm{p}_{t=1})$ does not correctly classify noisy training samples, but still correctly classifies clean training samples. Together with $\bm{p}_{t=1}={\bm{0}}$, the classifier $\mathrm{sign}(f(\bm{X};\bm{v}_{t=1},\bm{p}_{t=1})$ will also correctly classify, with high probability, a clean test sample.

Moreover, since the loss function $\ell$ is decreasing, the loss of noisy samples, denoted $\ell_{t=1,j},j\in\mathcal{N}$, dominates the loss of clean samples $\ell_{t=1,i},i\in\mathcal{C}$. This implies that after two iterations, the coefficients $|\theta_{j}^{t=2}|,j\in\mathcal{N}$, of the second (noise) tokens in $\bm{v}_{t=2}$, corresponding to noisy samples, grow faster than the coefficients $|\lambda_{i}^{t=2}|$ of the first (signal) tokens.
This property is important to allow for interpolation of noisy examples.
We also show that
$\bm{p}_{t=2}$
focuses on optimal tokens,
namely, on the
noise token
for noisy samples (i.e. $s_{i,2}^{t=2}\geq 1-1/c_{\rho}^{2},\forall i\in\mathcal{N}$), and on the signal token for
clean training and test samples.
Using this property we conclude that the model parameterized by $(\bm{v}_{t=2},\bm{p}_{t=2})$ exhibits benign overfitting.

###### Remark 11 .

Note that our proof implies the following behavior of GD.
After the first iteration, the model correctly classifies only the clean training samples, resulting in an expected training accuracy of $1-\eta$. Additionally, the model successfully classifies a clean test sample w.h.p., leading to the same expected test accuracy. After the second iteration, the model interpolates the training data, achieving
a
training accuracy of $1$. This is shown empirically in Figure [1](#S6.F1).
When using a smaller step size, we empirically observe a similar trend: after the first iteration, the model learns the signal tokens, and with more iterations, it captures the noisy tokens of the noisy samples and fits the entire dataset. This behavior
is shown
in Figure [2](#S6.F2).

###### Remark 11 .

### 5.2 Proof ideas for Section 4

Here we provide the proof sketch for Theorem [6](#Thmtheorem6). There are mainly two parts in our proof:

- •
First we determine the convergence behavior of $\bm{p}$ and $\bm{v}$ when the norm constraint $R$ is sufficiently large.
- •
Using properties derived from this convergence, we can analyze the training and test errors.

The first part of the proof builds upon techniques from [], which shows that jointly solving for $\bm{v}$ and $\bm{p}$ leads to convergence to their respective max-margin solutions. While their approach focuses on the asymptotic case where $R,r\rightarrow\infty$ under specific conditions on the training data, our work extends these techniques to the signal-noise data model and provides non-asymptotic results.

To begin, consider the output of the attention layer $\bm{r}_{i}=\bm{X}_{i}^{\top}\mathbb{S}(\bm{X}_{i}\bm{p})$ which is a combination of signal and noise tokens. This can be considered as a “token selection” based on softmax probabilities. Since $\{\bm{r}_{i}\}_{i\in[n]}$ determines the model’s output, we prove that only by selecting signal tokens for clean samples and noise tokens for noisy
samples can we reach the maximum margin when performing SVM on $(\bm{r}_{i},y_{i})_{i\in[n]}$ and we refer to this as *optimal tokens*.

###### Definition 12 (Optimal Token) .

We define the *optimal token* for sample $(\bm{X}_{i},y_{i})$ as

$$ $\displaystyle\bm{r}_{i}^{\star}$ $\displaystyle=\bm{x}_{i}^{(1)}=\bm{\mu}_{k},\ i\in\mathcal{C}_{k},k\in\{1,2\}\quad\quad\quad\text{and}\quad\quad\quad\bm{r}_{i}^{\star}=\bm{x}_{i}^{(2)}=\bm{\xi}_{i},\ i\in\mathcal{N}$ (5.1) $$

Based on this optimal token selection, we define the corresponding max-margin solution for $\bm{p}$ and $\bm{v}$, denoted by $\bm{p}_{mm}$ and $\bm{v}_{mm}$. We first define $\bm{p}_{mm}$ as follows:

###### Definition 13 (p-SVM) .

$$ $\displaystyle\bm{p}_{mm}=\mathop{\mathrm{argmin}}\limits_{\bm{p}\in\mathbb{R}^{d}}\|\bm{p}\|\;\;\;\;\text{subject to:}$ $\displaystyle\bm{p}^{\top}(\bm{\mu}_{k}-$ $\displaystyle\bm{\xi}_{i})\geq 1,i\in\mathcal{C}_{k}\;\;\;\text{and}\;\;\;\bm{p}^{\top}(\bm{\xi}_{i}-\bm{\mu}_{i})\geq 1,i\in\mathcal{N}$ $$

for all $k\in\{1,2\},i\in[n]$.
Let $\Xi:=1/\|\bm{p}_{mm}\|$ be the margin induced by $\bm{p}_{mm}$.

Then for a given $\bm{p}$, we define $\bm{v}(\bm{p})$ as the standard max-margin classifier on $(\bm{r}_{i},y_{i})_{i\in[n]}$
and $\bm{v}_{mm}$ as the standard max-margin classifier on $(\bm{r}^{\star}_{i},y_{i})_{i\in[n]}$ which represents the limiting case when $\bm{p}=\bm{p}_{mm}$ and $R\rightarrow+\infty$.

###### Definition 14 (v-SVM) .

$$ $\bm{v}(\bm{p}):=\mathop{\mathrm{argmin}}_{\bm{v}\in\mathbb{R}^{d}}\|\bm{v}\|\text{ s.t. }y_{i}\cdot\bm{v}^{\top}\bm{r}_{i}\geq 1,\quad\text{ for all }i\in[n].$ (5.2) $$

$\Gamma(\bm{p}):=1/\|\bm{v}(\bm{p})\|$
is the label margin induced by $\bm{v}(\bm{p})$.
When $\bm{r}_{i}=\bm{r}_{i}^{\star},i\in[n]$, we define

$$ $\bm{v}_{mm}:=\mathop{\mathrm{argmin}}_{\bm{v}\in\mathbb{R}^{d}}\|\bm{v}\|\text{ s.t. }y_{i}\cdot\bm{v}^{\top}\bm{r}_{i}^{\star}\geq 1,\quad\text{ for all }i\in[n].$ (5.3) $$

$\Gamma:=1/\|\bm{v}_{mm}\|$
is the label margin induced by $\bm{v}_{mm}$.

To show the optimality of this token selection, we prove that any other token selection that incorporates other tokens in $\bm{r}_{i}$ will strictly reduce the label margin. This is formalized in the following proposition:

###### Proposition 15 (optimal token condition) .

Suppose that Assumption [5](#Thmtheorem5) holds, with probability at least $1-\delta$ over the training dataset, for all $\bm{p}$,
the token selection under $\bm{p}$ results in a label margin (Def. [14](#Thmtheorem14)) of at most $\Gamma-\frac{C}{\|\bm{v}_{mm}\|^{3}n\rho^{2}}\cdot\max\limits_{i\in[n]}(1-s_{i\alpha_{i}})$
where $\alpha_{i}=\mathbb{I}(i\in\mathcal{C})+2\mathbb{I}(i\in\mathcal{N})$ and $C>0$ is some constant.

Then, it is natural to make a conjecture that when jointly optimizing $\bm{p}$ and $\bm{v}$ for ([4.1](#S4.E1)), they will converge to their respective max-margin solutions $\bm{p}_{mm}$ and $\bm{v}_{mm}$ as $R,r\rightarrow\infty$. We verify and formalize it in the following theorem.

###### Theorem 16 .

Suppose that Assumption [5](#Thmtheorem5) holds, with probability at least $1-\delta$ on the training dataset, we have

- •
The margin induced by $\bm{p}_{(r,R)}/R$ in p-SVM is at least $(1-\zeta)\Xi$, where
$\zeta=\frac{\log(4\sqrt{\rho^{2}+(1+\kappa)d}\|\bm{v}_{mm}\|^{3}d\rho^{2})}{R\Xi}.$
- •
The label margin induced by $\bm{v}_{(r,R)}/r$ in v-SVM is at least $(1-\gamma)\Gamma$, where $\gamma=\frac{2\sqrt{\rho^{2}+(1+\kappa)d}}{\Gamma\exp((1-\zeta)R\Xi)}$.

Here, $(\zeta,\gamma)$ quantify the difference between $(\bm{p}_{(r,R)},\bm{v}_{(r,R)})$ and $(\bm{p}_{mm},\bm{v}_{mm})$. As $R\rightarrow\infty$, both $\zeta$ and $\gamma$ converge to 0. Thus, for sufficiently large $R$, we conclude that $\bm{p}_{(r,R)}^{\top}(\bm{\mu}_{k}-\bm{\xi}_{i})$
becomes large for $i\in\mathcal{C}_{k}$. This ensures that $\bm{p}_{(r,R)}$ captures sufficient information about signal tokens, which enhances the accuracy of test sample predictions. Specifically, the attention weight on a signal token is lower bounded by
$0.5(1-\zeta)R\Xi\leq\langle\bm{p}_{(r,R)},\bm{\mu}_{j}\rangle$.
Since the signal token remains invariant between training and test data, we can estimate the attention layer’s output for a new test sample $(\bm{X},y)$.

###### Lemma 17 .

Suppose that Assumption [5](#Thmtheorem5) holds, with probability at least $1-\delta$ on the training dataset, for a given test sample $(\bm{X},y)$ with $\bm{X}=(\bm{\mu}^{\star},\bm{\xi}^{\star})$, where the signal $\bm{\mu}^{\star}$ can be $\bm{\mu}_{1}$ or $\bm{\mu}_{2}$, we have with probability at least $1-\exp\big{(}-\frac{1}{2}(\frac{1}{2}(1-\zeta)\Xi-K/R)^{2}\big{)}$ that
$\langle\bm{p}_{(r,R)},\bm{\mu}^{\star}\rangle-\langle\bm{p}_{(r,R)},\bm{\xi}^{\star}\rangle\geq K$,
for $K\leq\frac{1}{2}(1-\zeta)R\Xi$. Here $\zeta,\Xi$ follow the definitions in Theorem [16](#Thmtheorem16).

Therefore, if $K$ is large, which is equivalent to $R$ is large, the attention weight on the signal token is much greater than the noise token. As a result, the signal token $\bm{\mu}^{\star}$ will dominate the attention layer’s output, i.e. $\bm{r}^{\star}\rightarrow\bm{\mu}^{\star}$.

Finally, from Theorem [16](#Thmtheorem16), $\bm{v}_{(r,R)}$ converges to $\bm{v}_{mm}$, ensuring that it can make accurate predictions on $(\bm{\mu}_{k},y)$ if $(\bm{\mu}_{k},y)$ comes from the clean set. Thus, w.h.p. the learning of signal token $y\cdot\langle\bm{v}_{(r,R)},\bm{\mu}^{\star}\rangle$ is large enough to eliminate the randomness introduced by the noise token (denoted by $\Delta(\bm{\xi}^{\star})$ here) and the model will make accurate prediction with high probability:
$y\cdot f(\bm{p}_{(r,R)},\bm{v}_{(r,R)};\bm{X})\approx y\cdot\bm{v}_{(r,R)}^{\top}\bm{\mu}^{\star}-\Delta(\bm{\xi}^{\star})\geq 0$.

###### Definition 12 (Optimal Token) .

###### Definition 13 (p-SVM) .

###### Definition 14 (v-SVM) .

###### Proposition 15 (optimal token condition) .

###### Theorem 16 .

###### Lemma 17 .

## 6 Experiments

Figure: Figure 1: The left panel shows the train and test accuracies during training. It shows that benign overfitting occurs after $2$ iterations. After the first iteration, the model correctly classifies the clean training examples, but not the noisy ones. In the right panel, we show the softmax probability of the signal token for clean and noisy samples (average of the softmax probabilities $s_{j,1}^{t}$ over $\mathcal{C}$ and $\mathcal{N}$ respectively). We see that after $2$ iterations, the attention focuses on signal tokens for clean examples, and on noise tokens for noisy examples. Parameters: $n=200,d=40000,\beta=0.025,\rho=30,\eta=0.05,\text{test sample size}=2000$.
Refer to caption: /html/2410.07746/assets/x1.png

We complement our theoretical results with an empirical study on benign overfitting in single-head softmax attention.
We trained single-head softmax attention models (Eq. ([2.1](#S2.E1))) on data generated as specified in Section [2.1](#S2.SS1) using GD with a fixed step size and the logistic loss function. In all figures, the x-axis corresponds to the time and has a log scale. We add $1$ to the time so that the initialization $t=0$ can be shown in the log scale (i.e. iteration $10^{0}$ is the initialization).

In Figure [1](#S6.F1), we consider a setting similar to Theorem [4](#Thmtheorem4), and demonstrate that benign overfitting occurs after two iterations, and that the behavior of GD aligns with our discussion in Remark [11](#Thmtheorem11). We also plot how the softmax probabilities evolve during training, and see after two iterations a behavior similar to the first item of Theorem [4](#Thmtheorem4). In Figure [2](#S6.F2), we consider a similar setting, but with a smaller step size. Here, benign overfitting occurs after about $150$ iterations.

In Figure [3(a)](#S6.F3.sf1), we explore the behavior of GD with different SNR levels. When the SNR is too small the model exhibits catastrophic overfitting, namely, it fits the training data but has trivial generalization performance. When the SNR is sufficiently large we observe benign overfitting.
In Figure [3(b)](#S6.F3.sf2), we investigate the overfitting behavior with different dimensions $d$. If $d$ is sufficiently large we observe benign overfitting. If it is very small we are not able to overfit, namely, the training accuracy does not reach $1$. For intermediate values of $d$ we observe harmful overfitting.
Thus, we see that high dimensionality is crucial for benign overfitting.
Interestingly, we can see that achieving benign overfitting is possible even when $d\ll n^{2}$, suggesting that our assumption on $d$ in the theoretical results might not be tight.

Figure: Figure 2: The left panel shows train and test accuracies during training with a small step size. The clean training samples are correctly classified already after one iteration, but in contrast to Theorem [4](#Thmtheorem4) and Figure [1](#S6.F1), benign overfitting occurs after about $150$ iterations. In the right panel we see that the attention starts separating signal and noise tokens shortly before benign overfitting occurs. Parameters: $n=200,d=40000,\beta=0.0001,\rho=30,\eta=0.05,\text{test sample size}=2000$.
Refer to caption: /html/2410.07746/assets/x3.png

Figure: ((a)) Accuracy - different SNR’s
Refer to caption: /html/2410.07746/assets/x5.png

## 7 Conclusion

This paper took an initial step in establishing the benign overfitting phenomenon in a single-head softmax attention model.
Our results open up several future directions, including analyzing gradient descent for more than 2 steps, more complex data distributions containing more than 2 tokens and varying sequence length, and the self-attention architecture.

### Acknowledgements

WH was supported by the Google Research Scholar program.
GV was supported by research grants from the Center for New Scientists at the Weizmann Institute of Science, and the Shimon and Golde Picker – Weizmann Annual Grant.

## Appendix A Appendix

\localtableofcontents

###### Remark 18 .

Throughout our proofs, we assume without loss of generality that $\bm{\mu}_{1}=(\rho,0,0,...,0)^{\top}$, $\bm{\mu}_{2}=(0,\rho,0,...,0)^{\top}$ and $\bm{\xi}_{i}=(0,0,\bm{\xi}^{\top})$ for $\bm{\xi}\sim\mathcal{N}({\bm{0}},\bm{I}_{d-2})$. Indeed, since $\bm{\mu}_{1}$ and $\bm{\mu}_{2}$ are orthogonal, we can find orthogonal matrix $\bm{A}\in\mathbb{R}^{d\times d}$ such that $\bm{A}\bm{\mu}_{1}=(\rho,0,0,...,0)^{\top},\bm{A}\bm{\mu}_{2}=(0,\rho,0,...,0)^{\top}$ and $\bm{A}\bm{\xi}_{i}\sim\mathcal{N}(\bm{A}{\bm{0}},\bm{A}(\bm{I}_{d}-\bm{\mu}_{1}\bm{\mu}_{1}^{\top}/\rho^{2}-\bm{\mu}_{2}\bm{\mu}_{2}^{\top}/\rho^{2})\bm{A}^{\top})$, which mean that $\bm{A}\bm{\xi}_{i}=(0,0,\bm{\xi}^{\top})$ for $\bm{\xi}\sim\mathcal{N}({\bm{0}},\bm{I}_{d-2})$.
We emphasize that an orthogonal
transformation does not affect our results.

###### Remark 18 .

### A.1 Proofs for Sec. 3

#### A.1.1 Notations for Sec. 3 .

Given $a,b,c\in\mathbb{R}$, we denote by $c(a\pm b)$ the close segment $[c(a-b),c(a+b)]$. Given vector $\bm{x}$, we denote by $\bm{x}[i]$ the $i^{\text{th}}$ coordinate of $\bm{x}$, and $\bm{x}[i:j]$ denotes the subvector containing the elements from the $i^{\text{th}}$ to the $j^{\text{th}}$, inclusive.
We also list some key notations used in this section for convenience.

**Table 1: Usefull notation.**
| $\bm{x}_{i,j}$ | $j^{\text{th}}$ token in the $i^{\text{th}}$ sample |
| --- | --- |
| $\bm{\gamma}_{i,j}^{t}$ | $y_{i}\bm{v}_{t}^{\top}\bm{x}_{i,j}$ i.e. $j^{\text{th}}$ token score in time $t$ |
| $\alpha_{i,j}^{t}$ | softmax probability of the $j^{\text{th}}$ token in the $i^{\text{th}}$ sample in time $t$ |
| $\ell_{t,i}$ | $\ell(\bm{X}_{i};\bm{v}_{t},\bm{p}_{t})$ |

We remind that $\mathcal{C},\mathcal{N}\subseteq[n]$ denotes the indices of clean and noisy training examples, and $\mathcal{C}_{k},\mathcal{N}_{k}$ denotes the clean and noisy examples from cluster $k\in\{1,2\}$. For example if $i\in\mathcal{C}_{1}$, then $x_{i,1}=\mu_{1}$ and $y_{1}=1$, and for $j\in\mathcal{N}_{1}$ we have that $x_{j,1}=\mu_{1}$ and $y_{1}=-1$. Let $\mathbb{S}^{\prime}(\bm{v}):=\nabla\mathbb{S}(\bm{v})=\mathrm{diag}(\mathbb{S}(\bm{v}))-\mathbb{S}(\bm{v})\mathbb{S}(\bm{v})^{\top}$ denote the Jacobian of the softmax function $\mathbb{S}(\bm{v})$ at $\bm{v}\in\mathbb{R}^{d}$.

#### A.1.2 Additional Lemmas & Definitions for Sec 3 .

The following equations will be useful throughout the proof:

$$ $\displaystyle\nabla_{\bm{v}}\mathcal{L}(\bm{v},\bm{p})=\frac{1}{n}\sum_{i=1}^{n}\ell_{i}^{\prime}\cdot y_{i}\bm{X}_{i}^{\top}\mathbb{S}(X_{i}\bm{p})$ (A.1) $\displaystyle\nabla_{\bm{p}}\mathcal{L}(\bm{v},\bm{p})=\frac{1}{n}\sum_{i=1}^{n}\ell_{i}^{\prime}\cdot\bm{X}_{i}^{\top}\mathbb{S}^{\prime}(X_{i}\bm{p})\bm{\gamma}_{i},\text{ \ where \ }\bm{\gamma}_{i}=y_{i}\bm{v}^{\top}\bm{X}_{i}$ (A.2) $\displaystyle\ell^{\prime}(x)=-1/(1+\exp(x))$ (A.3) $\displaystyle\mathbb{S}^{\prime}(\bm{v})=\mathrm{diag}(\mathbb{S}(\bm{v}))-\mathbb{S}(\bm{v})\mathbb{S}(\bm{v})^{\top}$ (A.4) $$

###### Definition 19 (Good Training Set) .

We say that a training set $(\bm{X}_{1},\dots,\bm{X}_{n})$ is *good* if

- •
$\|\bm{\xi}_{i}\|_{2}^{2}\in(1\pm o_{n}(1))d$, for all $i\in[n]$.
- •
$|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|\leq\sqrt{d\log(12n^{2}/\delta)}$, for any $i,j\in[n]$.
- •
$|\mathcal{N}_{k}|\in\frac{n}{2}(\eta\pm o_{n}(1))$ and $|\mathcal{C}_{k}|=\frac{n}{2}(1-\eta\pm o_{n}(1))$, for $k\in\{1,2\}$.

###### Definition 20 (Good Test Sample) .

We say that a test sample $(\bm{X}=(\bm{x}_{1},\bm{x}_{2}),y)$ is *good* w.r.t. a training set $(\bm{X}_{1},\dots,\bm{X}_{n})$ and constant $C_{1}$ if

$$ $\displaystyle|\langle\bm{x}_{i,2},\bm{x}_{2}\rangle|\leq\frac{d}{C_{1}n},\ \ \ \forall i\in[n]$ $$

Next we write Lemma [59](#Thmtheorem59) slightly different, and also add a formal proof for completeness:

###### Lemma 21 .

Let $\delta>0$ and $C>0$. Suppose that Assumption  [23](#Thmtheorem23) (item [1](#A1.I2.i1)) holds with constant $C$, then with probability at least $1-\delta/2$ we have that

$$ $\displaystyle\left|\mathcal{C}_{k}\right|\in\frac{n}{2}(1-\eta\pm\sqrt{2/C}),\ \ \ \left|\mathcal{N}_{k}\right|\in\frac{n}{2}(\eta\pm\sqrt{2/C}),\ \ \ \forall k\in\{1,2\}.$ $$

Moreover,
we have

$$ $\displaystyle\left|\mathcal{C}_{k}\right|\in\frac{n}{2}(1-\eta\pm o_{n}(1)),\ \ \ \left|\mathcal{N}_{k}\right|\in\frac{n}{2}(\eta\pm o_{n}(1)),\ \ \ \forall k\in\{1,2\}~{}.$ $$

###### Proof.

By Hoeffding’s inequality,

$$ $\displaystyle\mathbb{P}\left(\left||\mathcal{C}_{j}|-\frac{n}{2}(1-\eta)\right|\geq\sqrt{n\log(16/\delta)/2}\right)\leq\delta/8,$ $$

which means that with probability at least $1-\delta/8$ we have that $|\mathcal{C}_{j}|\in\frac{n}{2}(1-\eta\pm c_{n})$, where $c_{n}=\sqrt{2n\log(16/\delta)}/n$. Hence, if $n\geq C\log(16/\delta)$, then $c_{n}=\sqrt{2\log(16/\delta)}/\sqrt{n}\leq\sqrt{2/C}$. Similarly, we can estimate $|\mathcal{N}_{k}|$ for $k\in\{1,2\}$, and by union bound, the result follows.
∎

###### Lemma 22 .

Let $\bm{z},\bm{\gamma},\bm{p}\in\mathbb{R}^{2}$ and let $\bm{\alpha}=\mathbb{S}(\bm{p})$, then

$$ $\displaystyle\bm{z}^{T}\mathbb{S}^{\prime}(\bm{p})\bm{\gamma}=(\gamma_{1}-\gamma_{2})(1-\alpha_{1})\alpha_{1}(z_{1}-z_{2})$ $$

###### Proof.

Observe that $\alpha_{1}+\alpha_{2}=1$. Therefore,

$$ $\displaystyle\bm{z}^{T}\mathbb{S}^{\prime}(\bm{p})\bm{\gamma}$ $\displaystyle=\bm{z}^{T}\mathrm{diag}(\bm{\alpha})\bm{\gamma}-\bm{z}^{T}\bm{\alpha}\bm{\alpha}^{\top}\bm{\gamma}=\sum_{i=1}^{2}z_{i}\alpha_{i}\gamma_{i}-\sum_{i=1}^{2}\alpha_{i}z_{i}\sum_{i=1}^{2}\alpha_{i}\gamma_{i}$ $\displaystyle=z_{1}\alpha_{1}\gamma_{1}+z_{2}\alpha_{2}\gamma_{2}-(\alpha_{1}z_{1}+\alpha_{2}z_{2})(\alpha_{1}\gamma_{1}+\alpha_{2}\gamma_{2})$ $\displaystyle=\left(\gamma_{2}-(\alpha_{1}\gamma_{1}+\alpha_{2}\gamma_{2})\right)\alpha_{2}z_{2}+\left(\gamma_{1}-(\alpha_{1}\gamma_{1}+\alpha_{2}\gamma_{2})\right)\alpha_{1}z_{1}$ $\displaystyle=\left(\alpha_{1}\gamma_{2}-\alpha_{1}\gamma_{1}\right)\alpha_{2}z_{2}+\left(\alpha_{2}\gamma_{1}-\alpha_{2}\gamma_{2}\right)\alpha_{1}z_{1}$ $\displaystyle=-\alpha_{1}\left(\gamma_{1}-\gamma_{2}\right)\alpha_{2}z_{2}+\alpha_{2}\left(\gamma_{1}-\gamma_{2}\right)\alpha_{1}z_{1}$ $\displaystyle=\alpha_{1}\left(\gamma_{1}-\gamma_{2}\right)\alpha_{2}(z_{1}-z_{2})$ $$

∎

Lemma [22](#Thmtheorem22) allows us to analyze $\nabla_{\bm{p}}\mathcal{L}$ as a function of the score gap.

###### Definition 19 (Good Training Set) .

###### Definition 20 (Good Test Sample) .

###### Lemma 21 .

###### Proof.

###### Lemma 22 .

###### Proof.

#### A.1.3 Proof of Thm. 4

###### Proof of Thm. 4 .

To simplify the proof, we will use the following assumption, which is slightly weaker than Assumption [3](#Thmtheorem3):

###### Assumption 23 (Assumptions for GD with SNR = $\Theta(1/\sqrt{n})$ ) .

Let $\delta>0$ be a desired probability of failure.
For constants $c_{\rho}\geq 6,c_{\mathbf{\beta}}\geq 16c_{\rho}\log(c_{\rho}^{2})$, there exists some large enough constant $C=C(c_{\mathbf{\beta}})$, such that the following hold:

- 1.
Number of samples $n$ should be sufficiently large: $n\geq C\log(16/\delta)$
- 2.
Dimension $d$ should be sufficiently large: $d\geq Cn^{2}\log(12n^{2}/\delta)$.
- 3.
Signal strength is: $\rho=c_{\rho}\sqrt{d/n}$
- 4.
Label flipping rate $\eta$: $\eta\leq 1/C$.
- 5.
The step size $\beta$ satisfies: $\beta=(c_{\mathbf{\beta}}\cdot n)/(c_{\rho}^{2}\cdot d)$.
- 6.
Initialization at zero: $\left\|\bm{v}_{0}\right\|=\left\|\bm{p}_{0}\right\|=0$.

Apart from slight adjustments to the constants within the logarithm at items [1](#A1.I2.i1) and [2](#A1.I2.i2) (which can be absorbed into $C$), the only changes are $c_{\mathbf{\beta}}\geq 16c_{\rho}\log(c_{\rho}^{2})$ (instead of $C_{\beta}\geq 16$) and $\beta=(c_{\mathbf{\beta}}\cdot n)/(c_{\rho}^{2}\cdot d)$ (instead of $\beta=C_{\beta}\cdot(n/d)$). Indeed, given $C_{\beta}\geq 16,C_{\rho}\geq 6$ and $\beta=C_{\beta}\cdot(n/d)$ which satisfy Assumption [3](#Thmtheorem3), define $c_{\rho}:=C_{\rho}\geq 6,c_{\mathbf{\beta}}:=C_{\beta}c_{\rho}^{2}\geq 16c_{\rho}\log(c_{\rho}^{2})$, which holds for any $c_{\rho}\geq 6$. We also have that $\beta=C_{\beta}\cdot(n/d)=(c_{\mathbf{\beta}}/c_{\rho}^{2})\cdot(n/d)$ , i.e., $\beta,c_{\rho},c_{\mathbf{\beta}}$ satisfy Assumption [23](#Thmtheorem23).

Next, under Assumption [23](#Thmtheorem23), we argue that with probability at least $1-\delta$ the training set is good (Def. [19](#Thmtheorem19)) i.e.:

- •
$|\mathcal{C}_{k}|\in\frac{n}{2}(\eta\pm o_{n}(1))$ and $\mathcal{N}_{k}\in\frac{n}{2}(1-\eta\pm o_{n}(1))$, for $k\in\{1,2\}$.
- •
$\|\bm{\xi}_{i}\|_{2}^{2}\in(1\pm o_{n}(1))d$, for any $i\in[n]$.
- •
$|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|\leq\sqrt{d\log(12n^{2}/\delta)}$, for any $i,j\in[n]$.

Indeed, this holds by Lemma [57](#Thmtheorem57), Lemma [21](#Thmtheorem21), and the union bound.
We emphasize that the notation $o_{n}(1)$ represents a term that becomes arbitrarily small as $n$ increases, and thus it can be bounded by a small constant if $C$ from Assumption [1](#A1.I2.i1) is large enough.

Next, we show that under a good training set, the model exhibits benign overfitting, already after two iterations. See Remark [18](#Thmtheorem18) for the data setting used throughout the proof.

GD after 1 iteration.
We start by analyzing the first coordinate of $\bm{v}_{1}$ (i.e. $\bm{v}$ after one iteration of GD). By assumption [23](#Thmtheorem23) ( item[6](#A1.I2.i6)), we have that $\bm{p}_{0}=\bm{v}_{0}={\bm{0}}$, which implies that $\ell_{0,i}^{\prime}=-1/2$, for any $i\in[n]$. Hence

$$ $\displaystyle-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{0},\bm{p}_{0})[1]$ $\displaystyle=-\frac{\mathbf{\beta}}{2n}\sum_{i=1}^{n}\ell_{0,i}^{\prime}\cdot y_{i}\bm{x}_{i,1}[1]=\frac{\mathbf{\beta}}{4n}\sum_{i\in\mathcal{C}_{1}}y_{i}\rho+\frac{\mathbf{\beta}}{4n}\sum_{i\in\mathcal{N}_{1}}y_{i}\rho$ $\displaystyle=\frac{\mathbf{\beta}}{4n}(|\mathcal{C}_{1}|-|\mathcal{N}_{1}|)\rho$ $\displaystyle\in\frac{\mathbf{\beta}}{8}(1-2\eta\pm o_{n}(1))\rho$ “good” training set $$

In the same way, we can estimate the second coordinate of $\bm{v}_{t=1}$:

$$ $\displaystyle\bm{v}_{t=1}[2]=\frac{\mathbf{\beta}}{4n}\sum_{i\in\mathcal{C}_{2}}y_{i}\rho+\frac{\mathbf{\beta}}{4n}\sum_{i\in\mathcal{N}_{2}}y_{i}\rho\in-\frac{\mathbf{\beta}}{8}(1-2\eta\pm o_{n}(1))\rho,$ $$

where we remind that $y_{i}=-1$, when $i\in\mathcal{C}_{2}$, hence $\bm{v}_{t=1}[2]$ has the same bounds as $\bm{v}_{t=1}[1]$, just with opposite sign.
We move to analyze the rest of the coordinates of $\bm{v}_{t=1}$:

$$ $\displaystyle\bm{v}_{t}[3:d]=\frac{\mathbf{\beta}}{4n}\sum_{i=1}^{n}y_{i}\bm{\xi}_{i}.$ $$

Overall, we can write $\bm{v}_{t=1}$ as $\lambda_{1}^{t=1}\bm{\mu}_{1}+\lambda_{2}^{t=1}\bm{\mu}_{2}+\sum_{i=1}^{n}y_{i}\theta_{i}^{t=1}\bm{\xi}_{i}$ with

$$ $\displaystyle\lambda_{1}^{t=1}\in\frac{\mathbf{\beta}}{8}(1-2\eta\pm o_{n}(1)),\ \ \lambda_{2}^{t=1}\in-\frac{\mathbf{\beta}}{8}(1-2\eta\pm o_{n}(1)),\ \ \theta_{i}^{t=1}=\frac{\mathbf{\beta}}{4n}.$ (A.5) $$

Moreover, since $\bm{\gamma}_{i}^{t=0}={\bm{0}}$ for every $i\in[n]$, we have that $\bm{p}_{1}={\bm{0}}$ (see Eq. [A.2](#A1.E2)).

Preparation for next iteration. To estimate $(\bm{v}_{t=2},\bm{p}_{t=2})$, we first need to estimate the loss for clean/noisy samples and the score difference, i.e. $\gamma_{i,1}^{1}-\gamma_{i,2}^{1},i\in\mathcal{C}$ and $\gamma_{j,2}^{1}-\gamma_{j,1}^{1},j\in\mathcal{N}$.

We remind that $\left\|\bm{\mu}_{j}\right\|^{2}=\rho^{2}=c_{\rho}^{2}d/n$ (Assumption [23](#Thmtheorem23) (item [3](#A1.I2.i3))).
For $j\in\mathcal{C}_{k}$, where $k\in\{1,2\}$ we have that

$$ $\displaystyle y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ $\displaystyle=\frac{1}{2}y_{j}\bm{v}_{t=1}^{\top}(\bm{x}_{j,1}+\bm{x}_{j,2})$ $\displaystyle\text{since \ }\bm{p}_{1}={\bm{0}}$ $\displaystyle=\frac{1}{2}|\lambda_{k}^{t=1}|\left\|\bm{\mu}_{k}\right\|^{2}+\frac{1}{2}\theta_{j}^{t=1}\left\|\bm{\xi}_{j}\right\|^{2}+\frac{1}{2}\sum_{i\in[n]:i\neq j}y_{i}y_{j}\theta_{j}^{t=1}\bm{\xi}_{i}^{\top}\bm{\xi}_{j}$ $\displaystyle y_{j}\lambda_{k}^{t=1}>0$ (A.6) $$

Since the training set is “good” then by Eq. [A.5](#A1.E5), we can bound $y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ as follows:

$$ $\displaystyle y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ $\displaystyle\leq\frac{\mathbf{\beta}}{16}(1-2\eta+o_{n}(1))\cdot c_{\rho}^{2}\cdot\frac{d}{n}+\frac{\mathbf{\beta}}{8n}d(1+o_{n}(1))+\frac{\mathbf{\beta}}{8n}n\sqrt{d\log(12n^{2}/\delta)}$ $\displaystyle\leq\left(\frac{c_{\rho}^{2}(1-2\eta)+2+o_{n}(1)}{16}\right)\cdot\frac{\beta d}{n}\ \ \ \ \ \ \ \ \ \ \ \ \ \text{Assumption \ref{assumption: gd} (item \ref{assumption: gd-high-dimmension})}$ $\displaystyle=c_{\mathbf{\beta}}\cdot\left(\frac{(1-2\eta)+2/c_{\rho}^{2}+o_{n}(1)}{16}\right)\ \ \ \ \ \ \ \ \ \ \ \ \ \text{Assumption \ref{assumption: gd} (item \ref{assumption: gd-step-size})}$ $\displaystyle\leq\frac{1.1c_{\mathbf{\beta}}}{16},$ (A.7) $$

where the last inequality holds since $c_{\rho}\geq 5$, which implies that $2/c_{\rho}^{2}+o_{n}(1)\leq 0.1$. Similarly, we have that

$$ $\displaystyle y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ $\displaystyle\geq\frac{\mathbf{\beta}}{16}(1-2\eta-o_{n}(1))\cdot c_{\rho}^{2}\cdot\frac{d}{n}+\frac{\mathbf{\beta}}{8n}d(1-o_{n}(1))-\frac{\mathbf{\beta}}{8n}n\sqrt{d\log(12n^{2}/\delta)}$ $\displaystyle\geq\left(\frac{c_{\rho}^{2}(1-2\eta)+2-o_{n}(1)}{16}\right)\cdot\frac{\beta d}{n}$ $\displaystyle=c_{\mathbf{\beta}}\cdot\left(\frac{(1-2\eta)+2/c_{\rho}^{2}-o_{n}(1)}{16}\right)$ $\displaystyle\geq\frac{0.9c_{\mathbf{\beta}}}{16}$ (A.8) $$

For $j\in\mathcal{N}_{k}$ we have:

$$ $\displaystyle y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ $\displaystyle=\frac{1}{2}y_{j}\bm{v}_{t=1}^{\top}(\bm{x}_{j,1}+\bm{x}_{j,2})$ $\displaystyle\text{since \ }\bm{p}_{1}={\bm{0}}$ $\displaystyle=-\frac{1}{2}|\lambda_{k}^{t=1}|\left\|\bm{\mu}_{k}\right\|^{2}+\frac{1}{2}\theta_{j}^{t=1}\left\|\bm{\xi}_{j}\right\|^{2}+\frac{\mathbf{\beta}}{8n}\sum_{i\in[n]:i\neq j}y_{i}y_{j}\bm{\xi}_{i}^{\top}\bm{\xi}_{j}$ $\displaystyle y_{j}\lambda_{k}^{t=1}<0$ $$

Since the training set is good then by Eq. [A.5](#A1.E5), we can bound $y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ as follows:

$$ $\displaystyle y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ $\displaystyle\leq-\frac{\mathbf{\beta}}{16}(1-2\eta-o_{n}(1))\cdot c_{\rho}^{2}\cdot\frac{d}{n}+\frac{\mathbf{\beta}}{8n}d(1+o_{n}(1))+\frac{\mathbf{\beta}}{8n}n\sqrt{d\log(12n^{2}/\delta)}$ $\displaystyle\leq\left(\frac{-c_{\rho}^{2}(1-2\eta)+2+o_{n}(1)}{16}\right)\cdot\frac{\beta d}{n}$ $\displaystyle=c_{\mathbf{\beta}}\cdot\left(\frac{-(1-2\eta)+2/c_{\rho}^{2}+o_{n}(1)}{16}\right)$ $\displaystyle\leq\frac{-0.9c_{\mathbf{\beta}}}{16},$ (A.9) $$

where the last inequality holds for small enough $\eta$ and since $c_{\rho}\geq 5$, which implies that $2/c_{\rho}^{2}+2\eta+o_{n}(1)\leq 0.1$. Similarly, we have that

$$ $\displaystyle y_{j}f(\bm{X}_{j};\bm{v}_{t=1},\bm{p}_{t=1})$ $\displaystyle\geq-\frac{\mathbf{\beta}}{16}(1-2\eta+o_{n}(1))\cdot c_{\rho}^{2}\cdot\frac{d}{n}+\frac{\mathbf{\beta}}{8n}d(1-o_{n}(1))-\frac{\mathbf{\beta}}{8n}n\sqrt{d\log(12n^{2}/\delta)}$ $\displaystyle\geq\left(\frac{-c_{\rho}^{2}(1-2\eta)+2-o_{n}(1)}{16}\right)\cdot\frac{\beta d}{n}$ $\displaystyle\geq c_{\mathbf{\beta}}\left(\frac{-(1-2\eta)+2/c_{\rho}^{2}-o_{n}(1)}{16}\right)$ $\displaystyle\geq\frac{-1.1c_{\mathbf{\beta}}}{16}~{}.$ (A.10) $$

We remind that $-\ell_{1,j}^{\prime}=1/(1+\exp(y_{i}f(\bm{X}_{i};\bm{v}_{t=1},\bm{p}_{t=1})))$ and that $\beta=c_{\mathbf{\beta}}\cdot n/(dc_{\rho}^{2})$ for some constant $c_{\mathbf{\beta}}\geq 16c_{\rho}$. Combine with Eqs. [A.7](#A1.E7) and [A.8](#A1.E8), we have that

$$ $\displaystyle i\in\mathcal{C},\ \ -\ell_{t=1,i}^{\prime}\geq 1/(1+\exp(1.1c_{\mathbf{\beta}}/16)):=m_{\mathcal{C}}^{t=1}>0$ (A.11) $\displaystyle i\in\mathcal{C},\ \ -\ell_{t=1,i}^{\prime}\leq 1/(1+\exp(0.9c_{\mathbf{\beta}}/16)):=M_{\mathcal{C}}^{t=1}\leq 1/(4c_{\rho}^{2}),$ (A.12) $$

where the last inequality holds since $c_{\mathbf{\beta}}\geq 16c_{\rho}$ and since $1+\exp(0.9c_{\rho})\geq 4c_{\rho}^{2}$ for any $c_{\rho}\geq 6$.

Moreover, by Eqs. [A.9](#A1.E9) and [A.10](#A1.E10), we have that

$$ $\displaystyle j\in\mathcal{N},\ \ -\ell_{t=1,j}^{\prime}\geq 1/(1+\exp(-0.9c_{\mathbf{\beta}}/16)):=m_{\mathcal{N}}^{t=1}\geq 0.99$ (A.13) $\displaystyle j\in\mathcal{N},\ \ -\ell_{t=1,j}^{\prime}\leq 1/(1+\exp(-1.1c_{\mathbf{\beta}}/16)):=M_{\mathcal{N}}^{t=1}\leq 1$ (A.14) $$

The notations $M_{\mathcal{C}}^{t}$ and $m_{\mathcal{C}}^{t}$ ($M_{\mathcal{N}}^{t}$ and $m_{\mathcal{N}}^{t}$) denote the upper and lower bounds, respectively, on the derivative of the loss for clean (noisy) samples at time $t$, and we use them throughout the proof. We remind that $\gamma_{i,j}^{t}=y_{i}\bm{v}_{t}^{\top}\bm{x}_{i,j}$. Then
by Eq. [A.5](#A1.E5), for $i\in\mathcal{C}_{k}$ we have that

$$ $\displaystyle\gamma_{i,1}^{t=1}\in\frac{\beta}{8}(1-2\eta\pm o_{n}(1))\rho^{2}=\frac{c_{\mathbf{\beta}}}{8}(1-2\eta\pm o_{n}(1))$ $\displaystyle\gamma_{i,2}^{t=1}\in\frac{\beta d}{4n}(1\pm o_{n}(1))=\frac{c_{\mathbf{\beta}}}{4}(1/c_{\rho}^{2}\pm o_{n}(1))$ $\displaystyle\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=2}\in\frac{c_{\mathbf{\beta}}}{8}(1-2/c_{\rho}^{2}-2\eta\pm o_{n}(1))~{}.$ (A.15) $$

where in the calculation of $\gamma_{i,2}^{t=1}$ we use $\sum_{i\in[n]:i\neq j}y_{i}y_{j}\theta_{j}^{t=1}\bm{\xi}_{i}^{\top}\bm{\xi}_{j}=o_{n}(1)\cdot d$, since the training set is good.
For $i\in\mathcal{N}_{k}$, we have that

$$ $\displaystyle\gamma_{i,1}^{t=1}\in-\frac{\beta}{8}(1-2\eta\pm o_{n}(1))\rho^{2}=-\frac{c_{\mathbf{\beta}}}{8}(1-2\eta\pm o_{n}(1))$ $\displaystyle\gamma_{i,2}^{t=1}\in\frac{\beta d}{4n}(1\pm o_{n}(1))=\frac{c_{\mathbf{\beta}}}{4}(1/c_{\rho}^{2}\pm o_{n}(1))$ $\displaystyle\gamma_{i,2}^{t=1}-\gamma_{i,1}^{t=2}\in\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta\pm o_{n}(1))~{}.$ (A.16) $$

GD after 2 iterations.
Analysis of $\bm{v}_{t=2}$.
Observe that

$$ $\displaystyle-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})=-\frac{\mathbf{\beta}}{n}\sum_{i=1}^{n}\ell_{1,i}^{\prime}\cdot y_{i}\bm{X}_{i}^{\top}\mathbb{S}(X_{i}\bm{p}_{1})=-\frac{\mathbf{\beta}}{2n}\sum_{i=1}^{n}\ell_{1,i}^{\prime}\cdot y_{i}(\bm{x}_{i,1}+\bm{x}_{i,2}).$ $$

We start by analyzing the first coordinate of $\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})$.

$$ $\displaystyle-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})[1]$ $\displaystyle=\frac{\mathbf{\beta}}{2n}\sum_{i\in\mathcal{C}_{1}}-\ell_{1,i}^{\prime}\cdot y_{i}\bm{x}_{i,1}[1]+\frac{\mathbf{\beta}}{2n}\sum_{i\in\mathcal{N}_{1}}-\ell_{1,i}^{\prime}\cdot y_{i}\bm{x}_{i,1}[1]$ $\displaystyle=\frac{\mathbf{\beta}}{2n}\sum_{i\in\mathcal{C}_{1}}-\ell_{1,i}^{\prime}\cdot\rho-\frac{\mathbf{\beta}}{2n}\sum_{i\in\mathcal{N}_{1}}-\ell_{1,i}^{\prime}\cdot\rho$ $\displaystyle=\frac{\mathbf{\beta}}{2n}\left(\sum_{i\in\mathcal{C}_{1}}-\ell_{1,i}^{\prime}-\sum_{j\in\mathcal{N}_{1}}-\ell_{1,j}^{\prime}\right)\cdot\rho~{}.$ (A.17) $$

Observe that

$$ $\displaystyle\sum_{i\in\mathcal{C}_{k}}-\ell_{1,i}^{\prime}-\sum_{j\in\mathcal{N}_{k}}-\ell_{1,j}^{\prime}$ $\displaystyle\geq\frac{n}{2}(1-\eta-o_{n}(1))\cdot m_{\mathcal{C}}-\frac{n}{2}(\eta+o_{n}(1))\cdot M_{\mathcal{N}}$ good training set $\displaystyle>0$ $\displaystyle\text{Eqs. \ref{eq: gd_loss_clean_t=1_lower_bound} and \ref{eq: gd_loss_noise_t=1_upper_bound}},$ $$

where the last inequality holds for small enough $\eta\leq 1/C$, where $C:=C(c_{\rho},c_{\mathbf{\beta}})$ (see Assumption [4](#A1.I2.i4)).
Substituting it into Eq. [A.17](#A1.E17), we obtain that

$$ $\displaystyle-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})[1]>0.$ $$

On the other hand, by Eq. [A.17](#A1.E17), we can upper bound the first coordinate of the gradient of $\bm{v}$ by

$$ $\displaystyle-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})[1]$ $\displaystyle\leq\frac{\mathbf{\beta}}{2n}\left(\sum_{i\in\mathcal{C}_{1}}-\ell_{1,i}^{\prime}\right)\cdot\rho$ $\displaystyle\leq\frac{\mathbf{\beta}}{17}\cdot\rho$ $\displaystyle-\ell_{1,i}^{\prime}<1/16,\text{Eq. \ref{eq: gd_loss_clean_t=1_upper_bound}}.$ $$

Similarly, we can estimate the second coordinate of $\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})$:

$$ $\displaystyle 0\geq-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})[2]\geq-\frac{\mathbf{\beta}}{17}\cdot\rho.$ $$

Write $\bm{v}_{t=2}=\lambda_{1}^{t=2}\bm{\mu}_{1}+\lambda_{2}^{t=2}\bm{\mu}_{2}+\sum_{i=1}^{n}y_{i}\theta_{i}^{t=2}\bm{\xi}_{i}$. Together with Eq. [A.5](#A1.E5), we get that

$$ $\displaystyle\lambda_{1}^{t=2}=\lambda_{1}^{t=1}-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})[1]/\rho\leq\frac{\beta}{8}(1+o_{n}(1))+\frac{\beta}{17}\leq\frac{3\beta}{16}$ (A.18) $\displaystyle\lambda_{1}^{t=2}\geq\lambda_{1}^{t=1}\geq\frac{\beta}{8}(1-2\eta-o_{n}(1))$ (A.19) $\displaystyle\lambda_{2}^{t=2}=\lambda_{2}^{t=1}-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})[2]\geq-\frac{\beta}{8}(1+o_{n}(1))-\frac{\beta}{17}\geq-\frac{3\beta}{16}$ (A.20) $\displaystyle\lambda_{2}^{t=2}\leq\lambda_{2}^{t=1}\leq-\frac{\beta}{8}(1-2\eta-o_{n}(1))~{}.$ (A.21) $$

Next, we analyze the rest of the coordinates of $\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})$.

$$ $\displaystyle-\mathbf{\beta}\nabla_{\bm{v}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})[3:d]$ $\displaystyle=\frac{\mathbf{\beta}}{2n}\sum_{i\in\mathcal{C}}-\ell_{1,i}^{\prime}\cdot y_{i}\bm{\xi}_{i}+\frac{\mathbf{\beta}}{2n}\sum_{j\in\mathcal{N}}-\ell_{1,j}^{\prime}\cdot y_{j}\bm{\xi}_{j},$ $$

and use it to analyze the coefficients of the noise (second) tokens in $\bm{v}_{t=2}$, i.e., $\theta_{i}^{t=2}$.
Indeed, for $i\in\mathcal{C}$ we have that

$$ $\displaystyle\theta_{i}^{t=2}$ $\displaystyle=\theta_{i}^{t=1}-\frac{\beta}{2n}\ell_{1,i}^{\prime}=\frac{\beta}{2n}(-\ell_{1,i}^{\prime}+0.5)$ Eq. A.5 $\displaystyle\in\left[\frac{\beta}{2n}(m_{\mathcal{C}}+0.5),\frac{\beta}{2n}(M_{\mathcal{C}}+0.5)\right].$ (A.22) $$

For $j\in\mathcal{N}$ we have that

$$ $\displaystyle\theta_{j}^{t=2}$ $\displaystyle=\theta_{j}^{t=1}-\frac{\beta}{2n}\ell_{1,j}^{\prime}=\frac{\beta}{2n}(-\ell_{1,j}^{\prime}+0.5)$ Eq. A.5 $\displaystyle\in\left[\frac{\beta}{2n}(m_{\mathcal{N}}+0.5),\frac{\beta}{2n}(M_{\mathcal{N}}+0.5)\right].$ (A.23) $$

Now we move to analyze $\bm{p}_{t=2}$ and show that $\bm{p}_{t=2}$ focuses on optimal tokens for training samples.

$\bm{p}_{t=2}$ focuses on optimal tokens.
Observe that $\bm{p}_{2}=-\mathbf{\beta}\nabla_{\bm{p}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})$. Therefore, for $j\in\mathcal{C}_{k}$

$$ $\displaystyle\bm{p}_{2}^{\top}(\bm{x}_{j,1}-\bm{x}_{j,2})$ $\displaystyle=-(\bm{x}_{j,1}-\bm{x}_{j,2})^{\top}\mathbf{\beta}\nabla_{\bm{p}}\mathcal{L}(\bm{v}_{t},\bm{p}_{t})=(\bm{x}_{j,1}-\bm{x}_{j,2})^{\top}\frac{\mathbf{\beta}}{n}\sum_{i=1}^{n}-\ell_{1,i}^{\prime}\cdot\bm{X}_{i}^{\top}\mathbb{S}^{\prime}(X_{i}\bm{p}_{t})\bm{\gamma}_{i}^{t=1}$ $\displaystyle=\frac{\mathbf{\beta}}{n}\sum_{i=1}^{n}-\ell_{1,i}^{\prime}\cdot\bm{x}_{j,1}^{\top}\bm{X}_{i}^{\top}\mathbb{S}^{\prime}(X_{i}\bm{p}_{t})\bm{\gamma}_{i}^{t=1}-\frac{\mathbf{\beta}}{n}\sum_{i=1}^{n}-\ell_{1,i}^{\prime}\cdot\bm{x}_{j,2}^{\top}\bm{X}_{i}^{\top}\mathbb{S}^{\prime}(X_{i}\bm{p}_{t})\bm{\gamma}_{i}^{t=1}$ $\displaystyle=\frac{\mathbf{\beta}}{n}\sum_{i\in[n]}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{j,1}^{\top}\bm{x}_{i,1}+\bm{x}_{j,2}^{\top}\bm{x}_{i,2})$ Lemma 22 $\displaystyle=\frac{\mathbf{\beta}}{n}(-\ell_{1,j}^{\prime})(\gamma_{j,1}^{t=1}-\gamma_{j,2}^{t=1})(1-\alpha_{j,1})\alpha_{j,1}(\left\|\bm{x}_{j,1}\right\|^{2}+\left\|\bm{x}_{j,2}\right\|^{2})$ $\displaystyle+\frac{\mathbf{\beta}}{n}\sum_{i\in\mathcal{C}_{k}:i\neq j}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{j,1}^{\top}\bm{x}_{i,1})$ $\displaystyle-\frac{\mathbf{\beta}}{n}\sum_{i\in\mathcal{N}_{k}:i\neq j}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,2}^{t=1}-\gamma_{i,1}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{j,1}^{\top}\bm{x}_{i,1})$ $\displaystyle+\frac{\mathbf{\beta}}{n}\sum_{i\in[n]:i\neq j}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{j,2}^{\top}\bm{x}_{i,2})~{}.$ $$

Observe that $\alpha_{i,1}^{t=1}=\alpha_{i,2}^{t=1}=1/2$. In Eqs. [A.15](#A1.E15) and [A.16](#A1.E16) we calculate the score differences (e.g. $\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})$. Overall, we can lower bound the above equation by:

$$ $\displaystyle\geq\frac{\mathbf{\beta}}{4n}\left(m_{\mathcal{C}}\cdot\frac{c_{\mathbf{\beta}}}{8}(1-2/c_{\rho}^{2}-2\eta-o_{n}(1))\cdot d(1-o_{n}(1))\right)$ $\displaystyle+\frac{\mathbf{\beta}}{4n}\left((1-\eta-o_{n}(1))\cdot\frac{n}{2}\cdot m_{\mathcal{C}}\frac{c_{\mathbf{\beta}}}{8}(1-2/c_{\rho}^{2}-2\eta-o_{n}(1))\frac{d}{n}c_{\rho}^{2}\right)$ $\displaystyle-\frac{\mathbf{\beta}}{4n}\left((\eta+o_{n}(1))\cdot\frac{n}{2}\cdot M_{\mathcal{N}}\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta+o_{n}(1))\frac{d}{n}c_{\rho}^{2}\right)$ $\displaystyle-\frac{\mathbf{\beta}}{4n}\left(n\cdot M_{\mathcal{N}}\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta+o_{n}(1))\sqrt{d\log(12n^{2}/\delta)}\right).$ $$

The first term dominates the last term since $d\gg n\sqrt{d\log(12n^{2}/\delta)}$ (see Assumption [23](#Thmtheorem23) (item [2](#A1.I2.i2))).
The second term dominates the third term for small enough $\eta$ (see Assumption [4](#A1.I2.i4)). Overall, we obtain that

$$ $\displaystyle\bm{p}_{2}^{\top}(\bm{x}_{j,1}-\bm{x}_{j,2})>0,$ (A.24) $$

which means that for any $i\in\mathcal{C}$ we have:

$$ $\displaystyle\alpha_{i,1}^{t=2}=\frac{1}{1+\exp(-\bm{p}_{2}^{\top}(\bm{x}_{j,1}-\bm{x}_{j,2}))}>\frac{1}{2}~{}.$ (A.25) $$

For $j\in\mathcal{N}_{k}$,

$$ $\displaystyle\bm{p}_{2}^{\top}(\bm{x}_{j,2}-\bm{x}_{j,1})$ $\displaystyle=\frac{\mathbf{\beta}}{n}(-\ell_{1,j}^{\prime})(\gamma_{j,2}^{t=1}-\gamma_{j,1}^{t=1})(1-\alpha_{j,1})\alpha_{j,1}(\left\|\bm{x}_{j,1}\right\|^{2}+\left\|\bm{x}_{j,2}\right\|^{2})$ $\displaystyle-\frac{\mathbf{\beta}}{n}\sum_{i\in\mathcal{C}_{k}:i\neq j}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{j,1}^{\top}\bm{x}_{i,1})$ $\displaystyle+\frac{\mathbf{\beta}}{n}\sum_{i\in\mathcal{N}_{k}:i\neq j}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,2}^{t=1}-\gamma_{i,1}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{j,1}^{\top}\bm{x}_{i,1})$ $\displaystyle-\frac{\mathbf{\beta}}{n}\sum_{i\in[n]:i\neq j}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{j,2}^{\top}\bm{x}_{i,2})$ Lemma 22 $$

Observe that $\alpha_{i,1}^{t=1}=\alpha_{i,2}^{t=1}=1/2$. In Eq. [A.15](#A1.E15) and Eq. [A.16](#A1.E16) we calculate the score differences (e.g. $\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})$. Overall, we can lower bound the above equation by:

$$ $\displaystyle\geq\frac{\mathbf{\beta}}{4n}\left(m_{\mathcal{N}}\cdot\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta-o_{n}(1))\cdot d(1-o_{n}(1))\right)$ $\displaystyle-\frac{\mathbf{\beta}}{4n}\left((1-\eta+o_{n}(1))\cdot\frac{n}{2}\cdot M_{\mathcal{C}}\frac{c_{\mathbf{\beta}}}{8}(1-2/c_{\rho}^{2}-2\eta+o_{n}(1))\frac{d}{n}c_{\rho}^{2}\right)$ $\displaystyle+\frac{\mathbf{\beta}}{4n}\left(|\mathcal{N}_{k}|\cdot m_{\mathcal{N}}\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta-o_{n}(1))\frac{d}{n}c_{\rho}^{2}\right)$ $\displaystyle-\frac{\mathbf{\beta}}{4n}\left(n\cdot M_{\mathcal{N}}\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta+o_{n}(1))\sqrt{d\log(12n^{2}/\delta)}\right).$ $$

Observe that the third term is non-negative. Moreover, we argue that the first term is at least twice the sum of the second and last terms. Indeed, enough to show that

$$ $\displaystyle\left(m_{\mathcal{N}}\cdot(1+2/c_{\rho}^{2}-2\eta-o_{n}(1))\cdot d(1-o_{n}(1))\right)\geq$ $\displaystyle 2\left((1+o_{n}(1))\cdot\frac{1}{2}\cdot M_{\mathcal{C}}\cdot dc_{\rho}^{2}\right)+2\left(n\cdot M_{\mathcal{N}}(1+2/c_{\rho}^{2}+o_{n}(1))\sqrt{d\log(12n^{2}/\delta)}\right)~{},$ $$

which indeed holds since $n\sqrt{d\log(12n^{2}/\delta)}=d\cdot o_{n}(1)$, and $M_{\mathcal{C}}\cdot c_{\rho}^{2}\leq 0.25$ while $m_{\mathcal{N}}\geq 0.99$ (see Eqs. [A.13](#A1.E13) and [A.12](#A1.E12)).
Overall, for any $i\in\mathcal{N}$ we have that:

$$ $\displaystyle\bm{p}_{2}^{\top}(\bm{x}_{j,2}-\bm{x}_{j,1})$ $\displaystyle\geq\frac{\mathbf{\beta}}{8n}\left(m_{\mathcal{N}}\cdot\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta-o_{n}(1))\cdot d(1-o_{n}(1))\right)$ $\displaystyle=\frac{c_{\mathbf{\beta}}}{8c_{\rho}^{2}}\left(m_{\mathcal{N}}\cdot\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta-o_{n}(1))\cdot(1-o_{n}(1))\right)$ $\displaystyle\geq 2\log(c_{\rho}),$ $$

where that last inequality holds since $c_{\mathbf{\beta}}\geq 16c_{\rho}\log(c_{\rho})$, which implies that $0.9c_{\mathbf{\beta}}^{2}/64c_{\rho}^{2}\geq 2\log(c_{\rho})=\log(c_{\rho}^{2})$.
We conclude that,

$$ $\displaystyle\alpha_{i,2}^{t=2}$ $\displaystyle=\frac{1}{1+\exp(-\bm{p}_{2}^{\top}(\bm{x}_{j,2}-\bm{x}_{j,1}))}\geq\frac{1}{1+\exp(-\log(c_{\rho}^{2}))}=\frac{1}{1+1/c_{\rho}^{2}}$ $\displaystyle=\frac{c_{\rho}^{2}}{c_{\rho}^{2}+1}\geq\frac{c_{\rho}^{2}-1}{c_{\rho}^{2}}=1-1/c_{\rho}^{2}.$ (A.26) $$

We conclude that for any $j\in\mathcal{N}$ we have that

$$ $\displaystyle\alpha_{j,2}^{t=2}\geq 1-1/c_{\rho}^{2},\ \ \alpha_{j,1}^{t=2}\leq 1/c_{\rho}^{2}.$ (A.27) $$

Together with Eq. [A.25](#A1.E25), this proves the third part of the Thm.

The classifier $\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2}))$ classifies correctly clean training samples.
Let $(\bm{X}_{j}=(\bm{x}_{j,1},\bm{x}_{j,2}),y_{j})$ for $j\in\mathcal{C}$. We remind that $\bm{x}_{j,1}=\bm{\mu}_{k}$ for $k\in\{1,2\}$ and $\bm{x}_{2,j}=\bm{\xi}_{j}$. we have that,

$$ $\displaystyle f(\bm{X}_{j};\bm{v}_{t=2},\bm{p}_{t=2})=\alpha_{j,1}^{t=2}\bm{v}_{2}^{\top}\bm{x}_{j,1}+\alpha_{j,2}^{t=2}\bm{v}_{2}^{\top}\bm{x}_{j,2},$ $$

and it suffices to prove that

$$ $\displaystyle y_{j}(f(\bm{X}_{j};\bm{v}_{2},\bm{p}_{2}))>0.$ $$

Indeed,

$$ $\displaystyle y_{j}$ $\displaystyle f(\bm{X}_{j};\bm{v},\bm{p})=\alpha_{j,1}^{t=2}y_{j}\bm{v}_{2}^{\top}\bm{x}_{j,1}+\alpha_{j,2}^{t=2}y_{j}\bm{v}_{2}^{\top}\bm{x}_{j,2}$ $\displaystyle=\alpha_{j,1}^{t=2}|\lambda_{k}|\left\|\bm{\mu}_{k}\right\|^{2}+\alpha_{j,2}^{t=2}\theta_{j}^{2}\left\|\bm{\xi}_{j}\right\|^{2}+\alpha_{j,2}^{t=2}y_{j}\sum_{i\in[n]:i\neq j}y_{i}\theta_{i}^{t=2}\bm{\xi}_{i}^{\top}\bm{\xi}_{j}$ $\displaystyle y_{j}\lambda_{k}>0$ $\displaystyle\geq\alpha_{j,1}^{t=2}|\lambda_{k}|\left\|\bm{\mu}_{k}\right\|^{2}-\alpha_{j,2}^{t=2}n\max_{i}|\theta_{i}|\sqrt{d\log(12n^{2}/\delta)}$ $\displaystyle\geq\alpha_{j,1}^{t=2}\left(\frac{\beta}{9}\right)\frac{d}{n}c_{\rho}^{2}-\alpha_{j,2}^{t=2}n\frac{\beta}{2n}(M_{\mathcal{N}}+1)\sqrt{d\log(12n^{2}/\delta)}$ Eqs. A.23 , A.19 and A.21 $\displaystyle\geq\frac{1}{2}\left(\frac{\beta}{9}\right)\frac{d}{n}c_{\rho}^{2}-\frac{1}{2}n\frac{\beta}{2n}(M_{\mathcal{N}}+1)\sqrt{d\log(12n^{2}/\delta)}$ Eq. A.25 $\displaystyle>0,$ $\displaystyle d\gg n\sqrt{d\log(12n^{2}/\delta)}$ $$

as required.
The classifier $\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2}))$ classifies correctly noisy training samples.
Let $(\bm{X}_{j}=(\bm{x}_{j,1},\bm{x}_{j,2}),y_{j})$ for $j\in\mathcal{N}$. We remind that $\bm{x}_{j,1}=\bm{\mu}_{k}$ for $k\in\{1,2\}$ and $\bm{x}_{2,j}=\bm{\xi}_{j}$. we have that,

$$ $\displaystyle f(\bm{X}_{j};\bm{v}_{t=2},\bm{p}_{t=2})=\alpha_{j,1}^{t=2}\bm{v}_{2}^{\top}\bm{x}_{j,1}+\alpha_{j,2}^{t=2}\bm{v}_{2}^{\top}\bm{x}_{j,2},$ $$

and it suffices to prove that

$$ $\displaystyle y_{j}(f(\bm{X}_{j};\bm{v}_{2},\bm{p}_{2}))>0.$ $$

Indeed,

$$ $\displaystyle y_{j}$ $\displaystyle f(\bm{X}_{j};\bm{v},\bm{p})=\alpha_{j,1}^{t=2}y_{j}\bm{v}_{2}^{\top}\bm{x}_{j,1}+\alpha_{j,2}^{t=2}y_{j}\bm{v}_{2}^{\top}\bm{x}_{j,2}$ $\displaystyle=-\alpha_{j,1}^{t=2}|\lambda_{k}|\left\|\bm{\mu}_{k}\right\|^{2}+\alpha_{j,2}^{t=2}\theta_{j}^{2}\left\|\bm{\xi}_{j}\right\|^{2}+\alpha_{j,2}^{t=2}y_{j}\sum_{i\in[n]:i\neq j}y_{i}\theta_{i}^{t=2}\bm{\xi}_{i}^{\top}\bm{\xi}_{j}$ $\displaystyle y_{j}\lambda_{k}<0$ $\displaystyle\geq-\alpha_{j,1}^{t=2}\left(\frac{3\beta}{16}\right)\frac{d}{n}c_{\rho}^{2}+\alpha_{j,1}^{t=2}\frac{\beta}{2n}m_{\mathcal{N}}d(1-o_{n}(1))-\alpha_{j,2}^{t=2}\frac{\beta}{n}d\cdot o_{n}(1)$ Eqs. A.23 , A.18 and A.20 $\displaystyle\geq-\frac{1}{c_{\rho}^{2}}\left(\frac{3\beta}{16}\right)\frac{d}{n}c_{\rho}^{2}+\left(1-\frac{1}{c_{\rho}^{2}}\right)\frac{\beta}{2n}0.99d(1-o_{n}(1))-\frac{\beta}{n}d\cdot o_{n}(1)$ Eqs. A.27 and A.13 $\displaystyle>0,$ $$

as required.
The classifier $\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2}))$ classifies correctly clean test samples.

Let $(\bm{X}=(\bm{x}_{1},\bm{x}_{2}),y)$ be a fresh clean sample i.e. $(\bm{X},y)\sim\mathcal{D}_{\text{clean}}$. Observe that $\bm{x}_{1}=\bm{\mu}_{k}$ for some $k\in\{1,2\}$ and $y=1$ iff $k=1$. By Remark [58](#Thmtheorem58), with probability at least $1-6n\exp(-d/4C_{1}n^{2})$ for some constant $C_{1}=C_{1}(c_{\rho},c_{\mathbf{\beta}})$ that will be chosen later, we have that $(\bm{X}=(\bm{x}_{1},\bm{x}_{2}),y)$ is a good test sample w.r.t. $C_{1}$ (Def. [20](#Thmtheorem20)).
We work under the event that $(\bm{X}=(\bm{x}_{1},\bm{x}_{2}),y)$ is a good test sample and show that $y=\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2})$.
Recall that $\bm{p}_{2}=-\mathbf{\beta}\nabla_{\bm{p}}\mathcal{L}(\bm{v}_{1},\bm{p}_{1})$ and therefore:

$$ $\displaystyle\bm{p}_{2}^{\top}(\bm{x}_{1}-\bm{x}_{2})$ $\displaystyle=-(\bm{x}_{1}-\bm{x}_{2})^{\top}\mathbf{\beta}\nabla_{\bm{p}}\mathcal{L}(\bm{v}_{t=1},\bm{p}_{t=1})=(\bm{x}_{1}-\bm{x}_{2})^{\top}\frac{\mathbf{\beta}}{n}\sum_{i=1}^{n}-\ell_{1,i}^{\prime}\cdot\bm{X}_{i}^{\top}\mathbb{S}^{\prime}(X_{i}\bm{p}_{t})\bm{\gamma}_{i}^{t=1}$ $\displaystyle=\frac{\mathbf{\beta}}{n}\sum_{i=1}^{n}-\ell_{1,i}^{\prime}\cdot\bm{x}_{1}^{\top}\bm{X}_{i}^{\top}\mathbb{S}^{\prime}(X_{i}\bm{p}_{t})\bm{\gamma}_{i}^{t=1}-\frac{\mathbf{\beta}}{n}\sum_{i=1}^{n}-\ell_{1,i}^{\prime}\cdot\bm{x}_{2}^{\top}\bm{X}_{i}^{\top}\mathbb{S}^{\prime}(X_{i}\bm{p}_{t})\bm{\gamma}_{i}^{t=1}$ $\displaystyle=\frac{\mathbf{\beta}}{n}\sum_{i\in[n]}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{1}^{\top}\bm{x}_{i,1}+\bm{x}_{2}^{\top}\bm{x}_{i,2})$ Lemma 22 $\displaystyle=\frac{\mathbf{\beta}}{n}\sum_{i\in\mathcal{C}_{k}}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{1}^{\top}\bm{x}_{i,1})$ $\displaystyle-\frac{\mathbf{\beta}}{n}\sum_{i\in\mathcal{N}_{k}}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,2}^{t=1}-\gamma_{i,1}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{1}^{\top}\bm{x}_{i,1})$ $\displaystyle+\frac{\mathbf{\beta}}{n}\sum_{i\in[n]:i\neq j}-\ell_{1,i}^{\prime}\cdot(\gamma_{i,1}^{t=1}-\gamma_{i,2}^{t=1})(1-\alpha_{i,1}^{t=1})\alpha_{i,1}^{t=1}(\bm{x}_{2}^{\top}\bm{x}_{i,2})$ $$

Observe that $\alpha_{i,1}^{t=1}=\alpha_{i,2}^{t=1}=1/2$. In Eq. [A.15](#A1.E15) and Eq. [A.16](#A1.E16) we calculate the score $\bm{\gamma}_{i}^{t=1}$. Overall, we can lower bound the above equation by:

$$ $\displaystyle+\frac{\mathbf{\beta}}{4n}\left((1-\eta-o_{n}(1))\cdot\frac{n}{2}\cdot m_{\mathcal{C}}\frac{c_{\mathbf{\beta}}}{8}(1-2/c_{\rho}^{2}-2\eta-o_{n}(1))\frac{d}{n}c_{\rho}^{2}\right)$ $\displaystyle-\frac{\mathbf{\beta}}{4n}\left((\eta+o_{n}(1))\cdot\frac{n}{2}\cdot M_{\mathcal{N}}\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta+o_{n}(1))\frac{d}{n}c_{\rho}^{2}\right)$ $\displaystyle-\frac{\mathbf{\beta}}{4n}\left(n\cdot M_{\mathcal{N}}\frac{c_{\mathbf{\beta}}}{8}(1+2/c_{\rho}^{2}-2\eta+o_{n}(1))\frac{d}{C_{1}n}\right).$ $$

Once again, the first term dominates the last two terms when $C_{1}$ is large enough and when $\eta$ is small enough (Assumption [4](#A1.I2.i4)).
This means that the softmax probability of the first token is:

$$ $\displaystyle\frac{1}{1+\exp(-\bm{p}_{2}^{\top}(\bm{x}_{1}-\bm{x}_{2}))}>\frac{1}{2}.$ (A.28) $$

Let $\bm{x}_{1}=\bm{\mu}_{k}$ for $k\in\{1,2\}$ and $\bm{x}_{2}=\bm{\xi}$. Then,

$$ $\displaystyle f(\bm{X};\bm{v},\bm{p})=\alpha_{1}\bm{v}_{2}^{\top}\bm{x}_{1}+\alpha_{2}\bm{v}_{2}^{\top}\bm{x}_{2},$ $$

where $\alpha_{1},\alpha_{2}$ are the softmax probabilities of $\bm{p}_{2}$ for $\bm{X}$. It suffices to prove that

$$ $\displaystyle y(f(\bm{X};\bm{v}_{2},\bm{p}_{2}))>0.$ $$

Since the test sample is "good", we have that $\forall i:\bm{\xi}_{i}^{\top}\bm{\xi}\leq\frac{d}{C_{1}n}$, which implies that

$$ $\displaystyle yf(\bm{X};\bm{v}_{2},\bm{p}_{2})$ $\displaystyle=\alpha_{1}y\bm{v}_{2}^{\top}\bm{x}_{1}+\alpha_{2}y\bm{v}_{2}^{\top}\bm{x}_{2}$ $\displaystyle=\alpha_{1}|\lambda_{k}|\left\|\bm{\mu}_{k}\right\|^{2}+\alpha_{2}y\sum_{i=1}^{n}y_{i}\theta_{i}\bm{\xi}_{i}^{\top}\bm{\xi}$ $\displaystyle y\lambda_{k}>0$ $\displaystyle\geq\alpha_{1}|\lambda_{k}|\left\|\bm{\mu}_{k}\right\|^{2}-\alpha_{2}n\max_{i}|\theta_{i}|\frac{d}{C_{1}n}$ $\displaystyle\geq\alpha_{1}\left(\frac{\beta}{9}\right)\frac{d}{n}c_{\rho}^{2}-\alpha_{2}n\frac{\beta}{2n}(M_{\mathcal{N}}+1)\frac{d}{C_{1}n}$ Eqs. A.23 , A.19 and A.21 $\displaystyle\geq\frac{1}{2}\left(\frac{\beta}{9}\right)\frac{d}{n}c_{\rho}^{2}-\frac{1}{2}n\frac{\beta}{2n}(M_{\mathcal{N}}+1)\frac{d}{C_{1}n}$ Eq. A.28 $\displaystyle>0,$ $$

where the last inequality holds for large enough $C_{1}$. Overall,

$$ $\displaystyle\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}(y\neq\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2})))$ $\displaystyle\leq\eta+\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}_{\text{clean}}}(y\neq\mathrm{sign}(f(\bm{X};\bm{v}_{t=2},\bm{p}_{t=2})))$ $\displaystyle\leq\eta+6n^{2}\exp(-d/4C_{1}n^{2}).$ $$

By Assumption [23](#Thmtheorem23) (item [2](#A1.I2.i2)), we can also upper bound the above term by $\eta+\exp(-d/C_{1}n^{2}),$ for a slightly larger $C_{1}$. This proves the last part of the theorem.
∎

###### Proof of Thm. 4 .

###### Assumption 23 (Assumptions for GD with SNR = Θ ​ ( 1 / n ) Θ 1 𝑛 \Theta(1/\sqrt{n}) ) .

### A.2 Proofs for Sec. 4

#### A.2.1 Additional Notation

We first introduce some additional notations. Denote

$$ $n_{1}=|\mathcal{C}|,\quad n_{2}=|\mathcal{N}|;\quad n_{1i}=|\mathcal{C}_{i}|,\quad n_{2i}=|\mathcal{N}_{i}|\text{ for }i=1,2.$ $$

Denote the output of the softmax layer $\mathbb{S}(\bm{X}_{i}\bm{p})$ by

$$ $\bm{s}_{i}=(1-\beta_{i},\beta_{i})^{\top}.$ $$

Denote the output of the attention layer $\bm{X}_{i}^{\top}\bm{s}_{i}$ by $\bm{r}_{i}=(1-\beta_{i})\bm{\mu}_{i}+\beta_{i}\bm{\xi}_{i}$, where $0\leq\beta_{i}\leq 1$ is the attention on the noise token of each sample. Then $f(\bm{X}_{i};\bm{p},\bm{v})=\langle\bm{v},\bm{r}_{i}\rangle$ can be treated as a linear classifier on $(y_{i},\bm{r}_{i})_{i\in[n]}$. Additionally, from the property of log function, item [1](#S4.I1.i1) in Assumption [5](#Thmtheorem5) can be understood as $d\geq Cn^{2}\log(\text{poly}(n)/\delta)$ and the same is for item [5](#S4.I1.i5).

#### A.2.2 Proof of Thm. 6

##### Proof Sketch

There are two main parts in our proof. In the first part, we prove that only by selecting signal tokens for clean samples and noise tokens for non-clean samples can we reach the maximum margin when doing SVM on $(y_{i},\bm{r}_{i})_{i\in[n]}$.

###### Definition 24 (Optimal Token) .

We define the “optimal token" for sample $(\bm{X}_{i},y_{i})$ as

$$ $\displaystyle\bm{r}_{i}^{\star}$ $\displaystyle=\bm{\mu}_{i},\ i\in\mathcal{C}$ $\displaystyle\bm{r}_{i}^{\star}$ $\displaystyle=\bm{\xi}_{i},\ i\in\mathcal{N}$ (A.29) $$

Next we define the respective max-margin solution for $\bm{p}$ and $\bm{v}$. We will show that when jointly optimizing parameters $\bm{p}$ and $\bm{v}$ for ([4.1](#S4.E1)), they will converge to their respective max-margin solutions as $R,r\rightarrow\infty$, which are $\bm{p}_{mm}$ and $\bm{v}_{mm}$ defined as follows.

###### Definition 25 .

(p-SVM)

$$ $\bm{p}_{mm}=\mathop{\mathrm{argmin}}\limits_{p}\|\bm{p}\|$ $$

subjected to

$$ $\displaystyle\bm{p}^{\top}(\bm{\mu}_{i}-\bm{\xi}_{i})$ $\displaystyle\geq 1,i\in\mathcal{C}$ $\displaystyle\bm{p}^{\top}(\bm{\xi}_{i}-\bm{\mu}_{i})$ $\displaystyle\geq 1,i\in\mathcal{N}$ (A.30) $$

for all $i\in[n]$. $\Xi=1/\|\bm{p}_{mm}\|$ is the margin induced by $\bm{p}_{mm}$.

Then for a given $\bm{p}$, we define $\bm{v}(\bm{p})$ as the standard max-margin classifier on $(y_{i},\bm{r}_{i})_{i\in[n]}$ and $\bm{v}_{mm}$ as the standard max-margin classifier on $(y_{i},\bm{r}_{i}^{\star})_{i\in[n]}$ which can be understood as the limit scenario when $\bm{p}=\bm{p}_{mm}$ and $R\rightarrow+\infty$ .

###### Definition 26 .

(v-SVM)

$$ $\bm{v}(\bm{p})=\mathop{\mathrm{argmin}}_{\bm{v}\in\mathbb{R}^{d}}\|\bm{v}\|\text{ s.t. }y_{i}\cdot\bm{v}^{\top}\bm{r}_{i}\geq 1,\quad\text{ for all }i\in[n].$ (A.31) $$

$\Gamma(\bm{p})=1/\|\bm{v}(\bm{p})\|$
is the label margin induced by $\bm{v}$ and $\bm{p}$. When $\bm{r}_{i}=\bm{r}_{i}^{\star},i\in[n]$,

$$ $\bm{v}_{mm}=\mathop{\mathrm{argmin}}_{\bm{v}\in\mathbb{R}^{d}}\|\bm{v}\|\text{ s.t. }y_{i}\cdot\bm{v}^{\top}\bm{r}_{i}^{\star}\geq 1,\quad\text{ for all }i\in[n].$ (A.32) $$

$\Gamma=1/\|\bm{v}_{mm}\|$
is the label margin induced by $\bm{v}_{mm}$.

After proving the converfnece direction of $\bm{p}_{R}$ and $\bm{v}_{r}$, we can utilize their properties similar to $\bm{p}_{mm}$ and $\bm{v}_{mm}$ to proceed the training and test error analysis. Therefore proving that the model exhibits benign-overfitting.

It is worth noting that in the first part, we show the optimality of the token selection in ([A.29](#A1.E29)) is strict in the sense that mixing other tokens in $\bm{r}_{i}$ will shrink the label margin. We formalize this into the following proposition:

See [15](#Thmtheorem15)

We will give detailed proof in the following.

###### Definition 24 (Optimal Token) .

###### Definition 25 .

###### Definition 26 .

##### Optimal Token Condition

Since $\bm{v}_{mm}$ satisfies the KKT conditions of the max-margin problem ([A.31](#A1.E31)), by the stationarity condition, we can represent $\bm{v}_{mm}$ as

$$ $\displaystyle\bm{v}_{mm}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum\limits_{i\in[n]}y_{i}\theta_{i}\bm{\xi}_{i}.$ (A.33) $$

Note that the conditions in ([A.31](#A1.E31)) can be written as:

###### Condition 1 (Optimal tokens) .

$$ $\begin{cases}&\bm{v}^{\top}\bm{\mu}_{1}\geq 1\\ &-\bm{v}^{\top}\bm{\mu}_{2}\geq 1\\ &y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in\mathcal{N}\end{cases}$ $$

Plugging ([A.33](#A1.E33)) in the condition [1](#Thmcondition1), we can rewrite these conditions as:

$$ $\begin{cases}&\lambda_{1}\cdot\|\bm{\mu}_{1}\|^{2}\geq 1\\ &-\lambda_{2}\cdot\|\bm{\mu}_{2}\|^{2}\geq 1\\ &\theta_{i}\cdot\|\bm{\xi}_{i}\|^{2}+y_{i}y_{i^{\prime}}\sum\limits_{i^{\prime}\neq i}\theta_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in\mathcal{N}\end{cases}$ $$

Then we introduce a lemma to estimate the coefficients $\theta_{i}$ of $\bm{v}_{mm}$ under this condition:

###### Lemma 27 (balanced noise factor for KKT points) .

Suppose that Assumption [5](#Thmtheorem5) holds, under Condition [1](#Thmcondition1),
we have that for $\bm{v}_{mm}$,

$$ $\theta_{i}=0,\quad i\in\mathcal{C};$ (A.34) $$

$$ $\theta_{i}\in\Big{[}\frac{(1-\kappa)d-4n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)})},\frac{1}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\Big{]},\quad i\in\mathcal{N}.$ (A.35) $$

###### Proof of Lemma 27 .

Note that Condition [1](#Thmcondition1) does not have any constraint for samples with $i\in\mathcal{C}$. Thus we have $\theta_{i}=0$ for any $i\in\mathcal{C}$ in the representation ([A.33](#A1.E33)). For $\theta_{i}$ with $i\in\mathcal{N}$,
we first prove the upper bound by contradiction.
Denote $j=\mathop{\mathrm{argmax}}\limits_{i\in\mathcal{N}}\theta_{i}$. Then we have

$$ $\displaystyle y_{j}\bm{v}^{\top}\bm{\xi}_{j}$ $\displaystyle=\sum\limits_{i\in\mathcal{N}}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle=\theta_{j}\|\bm{\xi}_{j}\|_{2}^{2}+\sum\limits_{i\neq j,i\in\mathcal{N}}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\geq\theta_{j}\cdot(1-\kappa)d-n_{2}\theta_{j}\cdot 2\sqrt{d\log(6n^{2}/\delta)},$ $$

where the inequality is from Lemma [57](#Thmtheorem57) and the definition of $j$. Consider the contrary case when $\theta_{j}>\frac{1}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\top}\bm{\xi}_{j}$ $\displaystyle>\frac{1}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\cdot\big{(}(1-\kappa)d-n_{2}\cdot 2\sqrt{d\log(6n^{2}/\delta)}\big{)}=1.$ $$

By the complementary slackness, if $y_{j}\bm{v}^{\top}\bm{\xi}_{j}>1$, then we must have $\theta_{j}=0$, and thus we reach a contradiction.

Then we prove for the lower bound. For $\forall j\in\mathcal{N}$ we have

$$ $\displaystyle 1$ $\displaystyle\leq\theta_{j}\|\bm{\xi}_{j}\|_{2}^{2}+\sum\limits_{i\neq j,i\in\mathcal{N}}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\leq\theta_{j}\cdot(1+\kappa)d+n_{2}\max\limits_{i\in\mathcal{N}}\theta_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq\theta_{j}\cdot(1+\kappa)d+\frac{n_{2}}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\cdot 2\sqrt{d\log(6n^{2}/\delta)}.$ $$

The second inequality is due to Lemma [57](#Thmtheorem57) and the last inequality is from the upper bound we just get. Therefore, we have

$$ $\displaystyle\theta_{j}\geq\frac{(1-\kappa)d-4n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)})}.$ $$

This completes the proof.
∎

Then we introduce a lemma to estimate $\|\bm{v}_{mm}\|$:

###### Lemma 28 (Norm of $\bm{v}_{mm}$ ) .

Suppose that Assumption [5](#Thmtheorem5) holds, for the solution $\bm{v}_{mm}$ of ([A.31](#A1.E31)) under the token selection ([A.29](#A1.E29)), we have

$$ $\displaystyle\frac{2}{\rho^{2}}+\frac{\eta n}{2d}\leq\|\bm{v}_{mm}\|^{2}\leq\frac{2}{\rho^{2}}+\frac{5\eta n}{d}.$ $$

This implies

$$ $\|\bm{v}_{mm}\|=\Theta\bigg{(}\sqrt{\frac{1}{\rho^{2}}+\frac{\eta n}{d}}\bigg{)}.$ $$

###### Proof of Lemma 28 .

As $\bm{v}_{mm}$ is the max-margin solution and satisfies KKT condition, it can be represented as

$$ $\displaystyle\bm{v}_{mm}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum\limits_{i\in\mathcal{C}}y_{i}\theta_{i}\bm{\xi}_{i}+\sum\limits_{i\in\mathcal{N}}y_{i}\theta_{i}\bm{\xi}_{i}.$ (A.36) $$

As $\bm{v}_{mm}$ satisfies Condition [1](#Thmcondition1), we have $\lambda_{1}\geq 1/\rho^{2}$ and $\lambda_{2}\leq-1/\rho^{2}$. So we could lower bound $\|\bm{v}_{mm}\|$ as

$$ $\displaystyle\|\bm{v}_{mm}\|^{2}$ $\displaystyle\geq\lambda_{1}^{2}\|\bm{\mu}_{1}\|^{2}+\lambda_{2}^{2}\|\bm{\mu}_{2}\|^{2}+\sum\limits_{i\in\mathcal{N}}\theta_{i}^{2}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}}y_{i}y_{j}\theta_{i}\theta_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\geq\frac{2}{\rho^{2}}+\frac{n_{2}(1-\kappa)}{d}+O\bigg{(}\frac{\eta^{2}n^{2}}{d^{3/2}}\bigg{)}\geq\frac{2}{\rho^{2}}+\frac{\eta n}{2d}.$ $$

The second inequality is from Lemma [27](#Thmtheorem27) that $\theta_{i}=\Theta(1/d)$ for $i\in\mathcal{N}$ and the last inequality is from Assumption [5](#Thmtheorem5).

Then to upper bound $\|\bm{v}_{mm}\|$, consider the following possible solution $\widetilde{\bm{v}}$

$$ $\displaystyle\widetilde{\bm{v}}=\rho^{-2}\bm{\mu}_{1}-\rho^{-2}\bm{\mu}_{2}+\sum\limits_{i\in\mathcal{N}}2y_{i}\bm{\xi}_{i}/d.$ $$

For $i\in\mathcal{C}$, we have

$$ $\displaystyle y_{i}\widetilde{\bm{v}}^{\top}\bm{r}_{i}$ $\displaystyle=y_{i}\widetilde{\bm{v}}^{\top}\bm{\mu}_{i}\geq 1.$ $$

And for $i\in\mathcal{N}$, we have

$$ $\displaystyle y_{i}\widetilde{\bm{v}}^{\top}\bm{r}_{i}$ $\displaystyle=y_{i}\widetilde{\bm{v}}^{\top}\bm{\xi}_{i}=2\|\bm{\xi}_{i}\|^{2}/d+\sum\limits_{j\in\mathcal{N},j\neq i}2y_{i}y_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle/d$ $\displaystyle\geq 2(1-\kappa)-2n_{2}\sqrt{\log(6n^{2}/\delta)/d}\geq 1.$ $$

The first inequality is from Lemma [57](#Thmtheorem57) and the second inequality is from Assumption [5](#Thmtheorem5). Therefore, $\widetilde{\bm{v}}$ is a possible solution of SVM problem [26](#Thmtheorem26) when $\bm{p}$ converges to $\bm{p}_{mm}$. So we have

$$ $\displaystyle\|\bm{v}_{mm}\|^{2}$ $\displaystyle\leq\|\widetilde{\bm{v}}\|^{2}=2/\rho^{2}+\sum\limits_{i\in\mathcal{N}}4\|\bm{\xi}_{i}\|^{2}/d^{2}+\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}}4y_{i}y_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle/d^{2}\leq\frac{2}{\rho^{2}}+\frac{5\eta n}{d}.$ $$

The last inequality is from Lemma [57](#Thmtheorem57), Lemma [59](#Thmtheorem59) and Assumption [5](#Thmtheorem5). Combine the results above, we have $\|\bm{v}_{mm}\|^{2}=\Theta(\frac{1}{\rho^{2}}+\frac{\eta n}{d})$.
∎

Based on the lemmas above, we introduce our main proposition in this section:

See [15](#Thmtheorem15)

###### Proof of Proposition 15 .

The main idea is to show the optimality of the token selection rule in the sense that mixing any other tokens will shrink the label margin. For a given $\bm{p}$, we say a sample $\bm{x}_{i}$ is a “mixed sample” if $\bm{r}_{i}\neq\bm{r}_{i}^{\star}$. We say $\bm{r}_{i}$ is a mixture of optimal token and non-optimal token in this case. Note that for any $\bm{p}$ with finite norm, $\bm{r}_{i}\neq\bm{r}_{i}^{\star}$. This notation is introduced for the clearness of the proof.

We use contradiction to prove Proposition [15](#Thmtheorem15) by showing that any token selection different from ([A.29](#A1.E29)) can only result in a strictly smaller label margin than that for the max-margin problem ([A.31](#A1.E31)). Since $\bm{v}$ satisfies the KKT conditions of the max-margin problem, we can write $\bm{v}$ as

$$ $\displaystyle\bm{v}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum\limits_{i\in\mathcal{C}}y_{i}\theta_{i}\bm{\xi}_{i}+\sum\limits_{i\in\mathcal{N}}y_{i}\theta_{i}\bm{\xi}_{i}.$ (A.37) $$

For a given $\bm{p}$, denote $\bm{v}^{\prime}$ as the max-margin solution in ([A.31](#A1.E31)), and $\Gamma^{\prime}=1/\|\bm{v}^{\prime}\|$ as the new label margin. According to Lemma [28](#Thmtheorem28), we have

$$ $\|\bm{v}_{mm}\|^{2}=\Theta\bigg{(}\frac{1}{\rho^{2}}+\frac{\eta n}{d}\bigg{)}=\Omega(1/\rho^{2}).$ $$

Then we have

$$ $\Gamma-\frac{C}{\|\bm{v}_{mm}\|^{3}n\rho^{2}}\cdot\max\limits_{i\in[n]}(1-s_{i\alpha_{i}})\geq\Gamma-\frac{C}{\|\bm{v}_{mm}\|^{3}n\rho^{2}}\geq\frac{\Gamma}{2}$ $$

for sufficiently large $d$. Here the last inequality uses $\|\bm{v}_{mm}\|^{2}=\Omega(1/\rho^{2})$. Thus we only need consider the case when the new label margin $\Gamma^{\prime}\geq\Gamma/2$, or equivalently,

$$ $\|\bm{v}^{\prime}\|\leq 2\|\bm{v}_{mm}\|.$ (A.38) $$

Assume that there are $k$ samples ($0<k\leq n$) that violdate the token selection rule ([A.29](#A1.E29)) and among them, $p$ samples are from clean set $\mathcal{C}$ and $k-p$ samples are from label-flipped set $\mathcal{N}$. Denote the indices of the $k$ samples as $I_{v}$. Then we consider the following three scenarios:

- 1.
$p\neq 0,k-p=0$. (All mixed samples come from $\mathcal{C}$)
- 2.
$p=0,k-p\neq 0$. (All mixed samples come from $\mathcal{N}$)
- 3.
$p\neq 0,k-p\neq 0$. (Mixed samples are from both sets)

We will separately discuss each scenario and show that Proposition [15](#Thmtheorem15) holds in all cases.
Case 1: $p\neq 0,k-p=0$

Under this scenario, we have:

$$ $I_{v}\cap\mathcal{C}=I_{v};\quad I_{v}\cap\mathcal{N}=\varnothing.$ $$

We proceed to analyze this scenario by dividing it into three distinct subcases.

- •
$p<n_{1}$, $I_{v}\cap\mathcal{C}_{1}\neq\varnothing$, $I_{v}\cap\mathcal{C}_{2}\neq\varnothing$
- •
$p<n_{1}$, $I_{v}\cap\mathcal{C}_{i}\neq\varnothing$, $I_{v}\cap\mathcal{C}_{i^{\prime}}=\varnothing$, ($i,i^{\prime}\in[2],i\neq i^{\prime}$)
- •
$p=n_{1}$

Case 1.1 $p<n_{1}$, $I_{v}\cap\mathcal{C}_{1}\neq\varnothing$, $I_{v}\cap\mathcal{C}_{2}\neq\varnothing$

In this case, both clusters exist clean samples that are not mixed. Denote the index of mixed samples $I_{v}$ as $\{k_{1},k_{2},...,k_{p}\}$. For every mixed sample $k_{i}$, we have $\bm{r}_{k_{i}}=\beta_{k_{i}}\bm{\mu}_{k_{i}}+(1-\beta_{k_{i}})\bm{\xi}_{k_{i}}$. Then
the conditions under Case 1.1 become

###### Condition 2 ( $p$ clean samples violating optimal token selection) .

$$ $\begin{cases}&\bm{v}^{\top}\bm{\mu}_{1}\geq 1\\ &-\bm{v}^{\top}\bm{\mu}_{2}\geq 1\\ &y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in\mathcal{N}\\ &y_{i}\bm{v}^{\top}\bm{r}_{i}\geq 1,i\in I_{v}\end{cases}$ $$

From the condition above, we could see that in this case, mixing one more clean sample is equal to adding one more constraint. Therefore, mixing $p$ samples will not result in a better solution than only mixing one sample, i.e. larger max-margin in our setting. So we can reduce this case to mixing only one clean sample with index $k^{\star}=\mathop{\mathrm{argmin}}\limits_{i\in I_{v}}\beta_{i}$. Denote $r_{k^{\star}}=\beta\bm{\mu}_{k^{\star}}+(1-\beta)\bm{\xi}_{k^{\star}}$ for some $\beta\in[0,1)$. Without loss of generality, we assume $\bm{\mu}_{k^{\star}}=\bm{\mu}_{1}$, $y_{k^{\star}}=+1$. Then the conditions become:

###### Condition 3 (one clean sample violating optimal token selection) .

$$ $\begin{cases}&\bm{v}^{\top}\bm{\mu}_{1}\geq 1\\ &-\bm{v}^{\top}\bm{\mu}_{2}\geq 1\\ &y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in\mathcal{N}\\ &y_{k^{\star}}\bm{v}^{\top}\bm{r}_{k^{\star}}\geq 1\end{cases}$ $$

Denote $\bm{v}^{\prime}$ as the optimal solution under this condition. $\bm{v}^{\prime}$ can also be written in the form of ([A.37](#A1.E37)) with coefficients denoted as $\lambda^{\prime}_{1},\lambda^{\prime}_{2}$ and $\theta^{\prime}_{i},i\in[n]$. Plugging this representation into the condition [3](#Thmcondition3), we have:

$$ $\begin{cases}&\lambda_{1}^{\prime}\cdot\|\bm{\mu}_{1}\|^{2}\geq 1\\ &-\lambda_{2}^{\prime}\cdot\|\bm{\mu}_{2}\|^{2}\geq 1\\ &\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in\mathcal{N}\\ &\beta\lambda_{1}^{\prime}\cdot\|\bm{\mu}_{1}\|^{2}+(1-\beta)(\theta^{\prime}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|^{2}+\sum\limits_{i\neq{k^{\star}}}y_{k^{\star}}y_{i}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle)\geq 1\end{cases}$ $$

First, we introduce another lemma similar to Lemma [27](#Thmtheorem27) to characterize the scale of $\theta_{i}^{\prime},i\in[n]$ in this case.

###### Lemma 29 .

Suppose that Assumption [5](#Thmtheorem5) holds, under Condition [3](#Thmcondition3), we have

$$ $\theta_{i}^{\prime}=0,\quad i\in\mathcal{C}\backslash\{k^{\star}\};$ $$

$$ $\displaystyle\theta_{i}\in\Big{[}\frac{(1-\kappa)d-4n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)})},\frac{1}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\Big{]},\quad i\in\mathcal{N}.$ $$

###### Proof of Lemma 29 .

Same as Condition [1](#Thmcondition1), Condition [3](#Thmcondition3) does not have any constraint for samples with $i\in\mathcal{C}\backslash\{k^{\star}\}$. Thus we have $\theta^{\prime}_{i}=0$ for any $i\in\mathcal{C}\backslash\{k^{\star}\}$.

Meanwhile, Condition [3](#Thmcondition3) introduces an additional constraint compared to Condition [1](#Thmcondition1). Consequently, the feasible region for $\{\theta_{i}^{\prime}\}_{i\in\mathcal{N}}$ under Condition [3](#Thmcondition3) is a subset of the feasible region for $\{\theta_{i}\}_{i\in\mathcal{N}}$ under Condition [1](#Thmcondition1). Therefore, the bounds established in Lemma [27](#Thmtheorem27) remain applicable to $\{\theta_{i}^{\prime}\}_{i\in\mathcal{N}}$.
∎

From this lemma, We can see that $\theta_{i}^{\prime}=\Theta(1/d)$ for $i\in\mathcal{N}$.
To proceed, we introduce a crucial lemma:

###### Lemma 30 .

Suppose that Assumption [5](#Thmtheorem5) holds, denote $\bm{v}$ and $\bm{v}^{\prime}$ as the optimal solutions under condition [1](#Thmcondition1) and condition [3](#Thmcondition3) respectively. We have

$$ $\displaystyle\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\frac{C_{1}(1-\beta\lambda^{\prime}_{1}\rho^{2})^{2}}{(1-\beta)^{2}(1+\kappa)d}+\widetilde{O}\Big{(}\frac{\eta n}{d^{3/2}}\Big{)}.$ $$

where $0<C_{1}\leq 1$ is a constant.

###### Proof of Lemma 30 .

We consider two cases under this scenario:

- •
$\theta^{\prime}_{k}=0$ in $\bm{v}^{\prime}$
In this case, from Lemma [29](#Thmtheorem29) we have $\beta\lambda^{\prime}_{1}\geq(1+o(1))/\rho^{2}$ and all other conditions are the same as the optimal selection. In order to get $\min\|\bm{v}\|$, we have $\lambda^{\prime}_{1}=(1+o(1))/\beta\rho^{2}$. Consider another solution $\bm{v}_{0}$ which has parameters $\lambda_{01}=1/\rho^{2}$, $\lambda_{02}=\lambda^{\prime}_{2}$, $\theta_{0i}=\theta^{\prime}_{i}(i\in[n])$. As $\bm{v}_{0}$ satisfies all the inequities under Condition [1](#Thmcondition1), we have $\Gamma_{0}\leq\Gamma$ So we have
$\displaystyle\Gamma^{2}-\Gamma^{\prime 2}$
$\displaystyle\geq\Gamma^{2}_{0}-\Gamma^{\prime 2}=\frac{1}{\|\bm{v}_{0}\|^{2}}-\frac{1}{\|\bm{v}^{\prime}\|^{2}}=\frac{(\lambda_{01}^{2}-\lambda^{\prime 2}_{1})\cdot\|\bm{\mu}_{1}\|^{2}}{\|\bm{v}_{0}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}$
$\displaystyle=\frac{(1+o(1))/\beta^{2}-1}{\|\bm{v}_{0}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}=\frac{(1+\beta)(1-\beta)+o(1)}{\beta^{2}\|\bm{v}_{0}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}\geq\frac{1-\beta}{\|\bm{v}_{0}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}.$
Therefore,
$\displaystyle\Gamma-\Gamma^{\prime}$
$\displaystyle\geq\frac{1-\beta}{(\Gamma_{0}+\Gamma^{\prime})\|\bm{v}_{0}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}\geq\frac{1-\beta}{2\Gamma_{0}\|\bm{v}_{0}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}.$
Set $c=\frac{1}{2\Gamma_{0}\|\bm{v}_{0}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}=\frac{1}{2\|\bm{v}_{0}\|\|\bm{v}^{\prime}\|^{2}}$. we have $\Gamma^{\prime}\leq\Gamma-c(1-\beta)$. Moreover, we could upper bound $c$ as
$\displaystyle c$
$\displaystyle=\frac{1}{2\|\bm{v}_{0}\|\|\bm{v}^{\prime}\|^{2}}\leq\frac{1}{2r^{3}_{mm}}.$
The last inequality is from $\|\bm{v}^{\prime}\|\geq\|\bm{v}_{0}\|\geq r_{mm}$.
- •
$\theta^{\prime}_{k}\neq 0$ in $\bm{v}^{\prime}$
From KKT condition, we have
$\displaystyle\theta^{\prime}_{k^{*}}\cdot\big{[}\beta\lambda_{1}^{\prime}\cdot\|\bm{\mu}_{1}\|^{2}+(1-\beta)(\theta^{\prime}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|^{2}+\sum\limits_{i\neq{k^{\star}}}y_{k^{\star}}y_{i}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle)-1\big{]}=0.$
As $\theta^{\prime}_{k^{\star}}>0$, we have
$\displaystyle\beta\lambda_{1}^{\prime}\cdot\|\bm{\mu}_{1}\|^{2}+(1-\beta)(\theta^{\prime}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|^{2}+\sum\limits_{i\in\mathcal{N}}y_{k^{\star}}y_{i}\theta^{\prime}_{i}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle)=1.$
So we can estimate $\theta^{\prime}_{k^{*}}$ as
$\displaystyle\theta^{\prime}_{k^{*}}\|\bm{\xi}_{k^{*}}\|^{2}$
$\displaystyle=\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{1-\beta}-\sum\limits_{i\in\mathcal{N}}y_{k^{\star}}y_{i}\theta^{\prime}_{i}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle\leq\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{1-\beta}+2n_{2}\max\limits_{i\in\mathcal{N}}\theta^{\prime}_{i}\sqrt{d\log(6n^{2}/\delta)}$
$\displaystyle=\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{1-\beta}+\frac{2n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}.$
(A.39)
The first inequality is from Lemma [57](#Thmtheorem57) and the last equality is from Lemma [29](#Thmtheorem29). We can also lower bound it as
$\displaystyle\theta^{\prime}_{k^{*}}\|\bm{\xi}_{k^{*}}\|^{2}$
$\displaystyle=\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{1-\beta}-\sum\limits_{i\in\mathcal{N}}y_{k^{\star}}y_{i}\theta^{\prime}_{i}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle\geq\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{1-\beta}-2n_{2}\max\limits_{i\in\mathcal{N}}\theta^{\prime}_{i}\sqrt{d\log(6n^{2}/\delta)}$
$\displaystyle=\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{1-\beta}-\frac{2n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}.$
(A.40)
The first inequality is from Lemma [57](#Thmtheorem57) and the last equality is from Lemma [29](#Thmtheorem29). Therefore, we have $\theta^{\prime}_{k^{*}}=\Theta(\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{(1-\beta)d})\pm O(\frac{\eta n}{d^{3/2}})$.
Then from the third inequality in Condition [3](#Thmcondition3), we have
$\displaystyle\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\in\mathcal{N},i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle$
$\displaystyle\geq 1-y_{i}y_{k^{*}}\theta^{\prime}_{k^{*}}\langle\bm{\xi}_{i},\bm{\xi}_{k^{*}}\rangle$
$\displaystyle\geq 1-\bigg{[}\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{(1-\beta)(1+\kappa)d}+O\Big{(}\frac{\eta n}{d^{3/2}}\Big{)}\bigg{]}\cdot|\langle\bm{\xi}_{i},\bm{\xi}_{k^{*}}\rangle|$
$\displaystyle\geq 1-\frac{2(1-\beta\lambda^{\prime}_{1}\rho^{2})\sqrt{\log(6n^{2}/\delta)}}{(1-\beta)(1+\kappa)\sqrt{d}}-\widetilde{O}\Big{(}\frac{\eta n}{d}\Big{)}$
$\displaystyle\geq 1-\frac{2\sqrt{\log(6n^{2}/\delta)}}{\sqrt{d}}-\widetilde{O}\Big{(}\frac{\eta n}{d}\Big{)}$
$\displaystyle=1-\frac{3\sqrt{\log(6n^{2}/\delta)}}{\sqrt{d}}.$
(A.41)
The second inequality is from ([A.39](#A1.E39)); The third inequality is from Lemma [57](#Thmtheorem57) and the last inequality is from the first inequality in Condition [3](#Thmcondition3) that $\lambda^{\prime}_{1}\rho^{2}\geq 1$.
Consider $\widetilde{\bm{v}}=\widetilde{\lambda}_{1}\bm{\mu}_{1}+\widetilde{\lambda}_{2}\bm{\mu}_{2}+\sum\limits_{i\in[n]}y_{i}\widetilde{\theta}_{i}\bm{\xi}_{i}$, which has $\widetilde{\lambda}_{1}=\lambda^{\prime}_{1}$, $\widetilde{\lambda}_{2}=\lambda^{\prime}_{2}$, $\widetilde{\theta}_{i}=\theta^{\prime}_{i}/(1-\frac{3\sqrt{\log(6n^{2}/\delta)}}{\sqrt{d}})$ for $i\in\mathcal{N}$ and $\widetilde{\theta}^{\prime}_{i}=0$ for $i\in\mathcal{C}$. We can verify that $\widetilde{\bm{v}}$ satisfies all conditions for $\bm{v}_{mm}$. For $\forall i\in\mathcal{N}$, we have
$\displaystyle\widetilde{\theta}_{i}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\in\mathcal{N},i^{\prime}\neq i}y_{i}y_{i^{\prime}}\widetilde{\theta}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle$
$\displaystyle=$
$\displaystyle\Big{[}\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\in\mathcal{N},i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\Big{]}/\bigg{(}1-\frac{3\sqrt{\log(6n^{2}/\delta)}}{\sqrt{d}}\bigg{)}\geq 1.$
The last inequality is from ([A.41](#A1.E41)). Meanwhile, we have $\widetilde{\lambda}_{1}\|\bm{\mu}_{1}\|^{2}=\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1$, $-\widetilde{\lambda}_{2}\|\bm{\mu}_{2}\|^{2}=-\lambda^{\prime}_{2}\|\bm{\mu}_{2}\|^{2}\geq 1$. So $\widetilde{\bm{v}}$ is a possible solution for Condition [3](#Thmcondition3), which implies $\|\bm{v}_{mm}\|\leq\|\widetilde{\bm{v}}\|$.
Next we estimate the difference between $\|\bm{v}^{\prime}\|^{2}$ and $\|\widetilde{\bm{v}}\|^{2}$. We write the expansion of $\|\widetilde{\bm{v}}\|^{2}$ and $\|\bm{v}^{\prime}\|^{2}$:
$\displaystyle\|\widetilde{\bm{v}}\|^{2}$
$\displaystyle=\widetilde{\lambda}_{1}^{2}\|\bm{\mu}_{1}\|^{2}+\widetilde{\lambda}_{2}^{2}\|\bm{\mu}_{2}\|^{2}+\sum\limits_{i\in\mathcal{N}}\widetilde{\theta}_{i}^{2}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i,j\in\mathcal{N};i\neq j}y_{i}y_{j}\widetilde{\theta}_{i}\widetilde{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle,$
$\displaystyle\|\bm{v}^{\prime}\|^{2}$
$\displaystyle=\lambda^{\prime 2}_{1}\|\bm{\mu}_{1}\|^{2}+\lambda^{\prime 2}_{2}\|\bm{\mu}_{2}\|^{2}+\sum\limits_{i\in\mathcal{N}\cup\{{k^{\star}}\}}\theta^{\prime 2}_{i}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i,j\in\mathcal{N}\cup\{k^{\star}\};i\neq j}y_{i}y_{j}\theta^{\prime}_{i}\theta^{\prime}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle.$
From the construction of $\widetilde{\bm{v}}$, we have $\lambda^{\prime}_{1}=\lambda_{1}$, $\lambda^{\prime}_{2}=\lambda_{2}$. So we have
$\displaystyle\|\bm{v}^{\prime}\|^{2}-\|\widetilde{\bm{v}}\|^{2}\geq$
$\displaystyle\theta^{\prime 2}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|^{2}+\underbrace{\sum\limits_{i\in\mathcal{N}}(\theta^{\prime 2}_{i}-\widetilde{\theta}_{i}^{2})\|\bm{\xi}_{i}\|^{2}}_{I_{1}}+\underbrace{\sum\limits_{i\in\mathcal{N}\cup\{{k^{\star}}\}}\sum\limits_{j\in\mathcal{N}\cup\{{k^{\star}}\}\backslash\{i\}}y_{i}y_{j}\theta^{\prime}_{i}\theta^{\prime}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{2}}$
$\displaystyle-\underbrace{\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}\backslash\{i\}}y_{i}y_{j}\widetilde{\theta}_{i}\widetilde{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{3}}.$
From ([A.40](#A1.E40)), we have
$\displaystyle\theta^{\prime}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|\geq\frac{1-\beta\lambda^{\prime}_{1}\rho^{2}}{(1-\beta)\sqrt{(1+\kappa)d}}-\widetilde{O}\bigg{(}\frac{\eta n}{d}\bigg{)}.$
We then bound the last three terms respectively. First we have
$\displaystyle|I_{1}|$
$\displaystyle=\sum\limits_{i\in\mathcal{N}}(\widetilde{\theta}^{2}_{i}-\theta^{\prime 2}_{i})\|\bm{\xi}_{i}\|^{2}\leq\bigg{(}\frac{1}{(1-\widetilde{O}(1/\sqrt{d}))^{2}}-1\bigg{)}\cdot\sum\limits_{i\in\mathcal{N}}\theta^{\prime 2}_{i}\|\bm{\xi}_{i}\|^{2}$
$\displaystyle\leq\frac{\widetilde{O}(1/\sqrt{d})}{(1-\widetilde{O}(1/\sqrt{d}))^{2}}\cdot\frac{n_{2}(1+\kappa)d}{(\big{(}1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}\big{)}^{2}}$
$\displaystyle=\widetilde{O}\Big{(}\frac{\eta n}{d^{3/2}}\Big{)}.$
The first inequality is from the definition of $\widetilde{\theta}_{i}$; The second inequality is from Lemma [27](#Thmtheorem27) and Lemma [57](#Thmtheorem57).
Then we bound $|I_{2}-I_{3}|$ as:
$\displaystyle|I_{2}-I_{3}|$
$\displaystyle=\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}\backslash\{i\}}(\widetilde{\theta}_{i}\widetilde{\theta}_{j}-\theta^{\prime}_{i}\theta^{\prime}_{j})\cdot|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|+\theta^{\prime}_{k}\sum\limits_{i\in\mathcal{N}}\theta^{\prime}_{i}|\langle\bm{\xi}_{k^{*}},\bm{\xi}_{i}\rangle|$
$\displaystyle\leq\bigg{(}\frac{1}{(1-\widetilde{O}(1/\sqrt{d}))^{2}}-1\bigg{)}\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}\backslash\{i\}}\theta^{\prime}_{i}\theta^{\prime}_{j}\cdot|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|+n_{2}\theta^{\prime}_{k^{*}}\cdot\max\limits_{i\in\mathcal{N}}\theta^{\prime}_{i}\cdot|\langle\bm{\xi}_{k^{*}},\bm{\xi}_{i}\rangle|$
$\displaystyle\leq\frac{\widetilde{O}(1/\sqrt{d})}{(1-\widetilde{O}(1/\sqrt{d}))^{2}}\cdot\frac{(n_{2})^{2}2\sqrt{d\log(6n^{2}/\delta)}}{(\big{(}1-\kappa)d-2\eta n\sqrt{d\log(6n^{2}/\delta)}\big{)}^{2}}+\theta^{\prime}_{k^{*}}\cdot\Theta\bigg{(}\frac{\eta n}{\sqrt{d}}\bigg{)}$
$\displaystyle=\widetilde{O}\bigg{(}\frac{\eta^{2}n^{2}}{d^{2}}\bigg{)}+\Theta\bigg{(}\frac{\eta n}{d^{3/2}}\bigg{)}$
$\displaystyle=\widetilde{O}\Big{(}\frac{\eta n}{d^{3/2}}\Big{)}.$
The first inequality is from the definition of $\widetilde{\theta}_{i}$; The second inequality is from Lemma [27](#Thmtheorem27) and Lemma [57](#Thmtheorem57).
Combining the above results, we finally have
$\displaystyle\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\frac{C_{1}(1-\beta\lambda^{\prime}_{1}\rho^{2})^{2}}{(1-\beta)^{2}(1+\kappa)d}+\widetilde{O}\Big{(}\frac{\eta n}{d^{3/2}}\Big{)}.$

∎

Now we can prove the main proposition in this case.

###### Proof of Proposition 15 in Case 1.1.

From Lemma [30](#Thmtheorem30) we have

$$ $\displaystyle\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}$ $\displaystyle\geq\frac{C_{1}(1-\beta\lambda^{\prime}_{1}\rho^{2})^{2}}{(1-\beta)^{2}(1+\kappa)d}+o\bigg{(}\frac{1}{d}\bigg{)}\geq\frac{C_{1}(1-\beta\lambda^{\prime}_{1}\rho^{2})^{2}}{(1+\kappa)d}(1-\beta)=T(1-\beta).$ $$

In the last equation we substitute $T=\frac{C_{1}(1-\beta\lambda^{\prime}_{1}\rho^{2})^{2}}{(1+\kappa)d}\geq 0$. Then we have

$$ $\displaystyle\Gamma^{2}-\Gamma^{\prime 2}$ $\displaystyle=\frac{1}{\|\bm{v}_{mm}\|^{2}}-\frac{1}{\|\bm{v}^{\prime}\|^{2}}=\frac{\|\bm{v}^{\prime}\|^{2}-\|\bm{v}_{mm}\|^{2}}{\|\bm{v}_{mm}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}\geq\frac{T(1-\beta)}{\|\bm{v}_{mm}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}.$ $$

Therefore,

$$ $\displaystyle\Gamma-\Gamma^{\prime}$ $\displaystyle\geq\frac{T(1-\beta)}{(\Gamma+\Gamma^{\prime})\|\bm{v}_{mm}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}\geq\frac{T(1-\beta)}{2\Gamma\|\bm{v}_{mm}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}=\frac{T(1-\beta)}{2\|\bm{v}_{mm}\|\|\bm{v}^{\prime}\|^{2}}\geq\frac{T(1-\beta)}{2\|\bm{v}^{\prime}\|^{3}}.$ $$

The last inequality is from $\|\bm{v}^{\prime}\|\geq\|\bm{v}_{mm}\|$. This implies

$$ $\displaystyle\Gamma^{\prime}\leq\Gamma-\frac{T(1-\beta)}{2\|\bm{v}^{\prime}\|^{3}}\leq\Gamma-\frac{C_{1}}{\|\bm{v}_{mm}\|^{3}n\rho^{2}}(1-\beta).$ $$

The last inequality is from our assumption that $\|\bm{v}^{\prime}\|\leq 2\|\bm{v}_{mm}\|$ and $\rho^{2}=\Omega(d/n)$.
∎

Next we consider the other case.
Case 1.2 $p=n_{1}$

Next we consider the case when all clean samples are mixed. In this case, all samples in clean set are mixed, so the first two inequalities in Condition [3](#Thmcondition3) do not hold, which means that $\lambda^{\prime}_{1}$ may be smaller than $\lambda_{1}$. But we could still prove that Lemma [30](#Thmtheorem30) holds. We first write down the condition in this case:

###### Condition 4 (All clean samples violate optimal token selection rule) .

$$ $\begin{cases}&y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in\mathcal{N}\\ &y_{i}\bm{v}^{\top}\bm{r}_{i}\geq 1,i\in\mathcal{C}\end{cases}$ $$

Plugging the representation ([A.37](#A1.E37)) into the condition, we have:

$$ $\begin{cases}&\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in\mathcal{N}\\ &\beta_{i}\lambda_{i}^{\prime}\cdot\|\bm{\mu}_{i}\|^{2}+(1-\beta_{i})(\theta^{\prime}_{i}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{j\neq i}y_{i}y_{j}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle)\geq 1,i\in\mathcal{C}\end{cases}$ $$

###### Proof of Lemma 30 .

First we assume that $\max\{\lambda^{\prime}_{1}\cdot\|\bm{\mu}_{1}\|^{2},-\lambda^{\prime}_{2}\cdot\|\bm{\mu}_{2}\|^{2}\}=q$ in optimal $\bm{v}^{\prime}$. If $q\geq 1$, this is the same as Case 1.3. So we assume that $q\leq 1$. Denote $k^{\star}=\mathop{\mathrm{argmin}}\limits_{i\in\mathcal{C}}\frac{1-\beta_{i}q}{1-\beta_{i}}$ and $\beta=\beta_{k^{\star}}$, consider the following condition

###### Condition 5 (Relaxed version of Condition 4 ) .

$$ $\begin{cases}&\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in\mathcal{N}\\ &\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq\frac{1-\beta q}{1-\beta},i\in\mathcal{C}\end{cases}$ $$

Compared with Condition [4](#Thmcondition4), the second inequality is relaxed for $i\in\mathcal{C}$. Therefore, denote the max-margin solution as $\widehat{\bm{v}}$ under Condition [5](#Thmcondition5), we must have $\|\widehat{\bm{v}}\|\leq\|\bm{v}^{\prime}\|$. Then we will prove that Lemma [30](#Thmtheorem30) still holds between $\|\bm{v}_{mm}\|$ and $\|\widehat{\bm{v}}\|$, which indicates $\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\|\widehat{\bm{v}}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\frac{C_{1}(1-\beta\lambda^{\prime}_{1}\rho^{2})^{2}}{(1-\beta)^{2}(1+\kappa)d}+o\big{(}\frac{1}{d}\big{)}$. Denote the parameters in $\widehat{\bm{v}}$ are $\widehat{\lambda_{1}},\widehat{\lambda_{2}}$ and $\widehat{\theta_{i}}$, we first introduce the following lemma to estimate $\widehat{\theta_{i}}$. Here we denote $\alpha=\frac{1-\beta q}{1-\beta}$ for convenience.

###### Lemma 31 .

Suppose that Assumption [5](#Thmtheorem5) holds, under Condition [5](#Thmcondition5), we have

$$ $\displaystyle\widehat{\theta_{i}}$ $\displaystyle\in\bigg{[}\frac{\alpha}{(1+\kappa)d}\bigg{(}1-\frac{2n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)},\frac{\alpha}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{]},i\in\mathcal{C},$ $\displaystyle\widehat{\theta_{i}}$ $\displaystyle\in\bigg{[}\frac{1}{(1+\kappa)d}\bigg{(}1-\frac{2\alpha n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)},\frac{\alpha}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{]},i\in\mathcal{N}.$ $$

###### Proof of Lemma 31 .

Denote $j=\mathop{\mathrm{argmax}}\limits_{i\in[n]}\widehat{\theta_{i}}$, we have

$$ $\displaystyle\widehat{\theta_{i}}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{j\neq i}y_{i}y_{j}\widehat{\theta_{i}}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\geq\widehat{\theta_{j}}\|\bm{\xi}_{j}\|^{2}-n\widehat{\theta_{j}}\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\geq\widehat{\theta_{j}}((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}).$ $$

The two inequalities are from Lemma [57](#Thmtheorem57) and our definition of j. Consider the contrary case when $\widehat{\theta_{j}}>\frac{\alpha}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}$, we have

$$ $\displaystyle y_{j}\widehat{\bm{v}}^{\top}\bm{\xi}_{j}$ $\displaystyle>\alpha.$ $$

By the complementary slackness condition, if $y_{j}\widehat{\bm{v}}^{\top}\bm{\xi}_{j}>\alpha\geq 1$, then we must have $\widehat{\theta_{j}}=0$, and thus we reach a contradiction.

Then we lower bound $\widehat{\theta_{i}}$, for $i\in\mathcal{C}$ we have

$$ $\displaystyle\alpha$ $\displaystyle\leq\widehat{\theta_{i}}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{j\neq i}y_{i}y_{j}\widehat{\theta_{i}}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle\leq\widehat{\theta_{i}}(1+\kappa)d+2n\max\limits_{i\in[n]}\widehat{\theta_{i}}\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq\widehat{\theta_{i}}(1+\kappa)d+\frac{2\alpha n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}.$ $$

The second inequality is from Lemma [57](#Thmtheorem57) and the last inequality is from the upper bound of $\widehat{\theta_{i}}$ we just derived. Therefore, we have

$$ $\displaystyle\widehat{\theta_{i}}\geq\frac{\alpha}{(1+\kappa)d}\bigg{(}1-\frac{2n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}.$ $$

Similarly, for $i\in\mathcal{N}$, we have

$$ $\displaystyle\widehat{\theta_{i}}\geq\frac{1}{(1+\kappa)d}\bigg{(}1-\frac{2\alpha n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}.$ $$

∎

Note that we only consider the case when $\|\widehat{\bm{v}}\|\leq\|\bm{v}^{\prime}\|\leq 2\|\bm{v}_{mm}\|$. And from Lemma [31](#Thmtheorem31) we have $\widehat{\theta_{i}}=\Theta(\alpha/d)$ for $i\in\mathcal{C}$. So we must have $\alpha=O(\log n)$ is some constant. Otherwise, for $i\in\mathcal{C}$ we have

$$ $\widehat{\theta}_{i}\|\bm{\xi}_{i}\|^{2}\geq\alpha-\sum_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\widehat{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle=\Omega(\alpha).$ $$

It further yields that

$$ $\displaystyle\|\widehat{\bm{v}}\|^{2}=\Omega(\frac{1}{\rho^{2}})+\Omega(\frac{\eta n}{d})+\sum\limits_{i\in\mathcal{C}}\widehat{\theta}_{i}^{2}\|\bm{\xi}_{i}\|^{2}=\Omega(\frac{1}{\rho^{2}}+\frac{\eta n}{d}+\frac{n\alpha^{2}}{d})=\Omega(\frac{n\log^{2}n}{d}),$ (A.42) $$

which contradicts with $\|\bm{v}^{\prime\prime}\|=\Theta(\sqrt{1/\rho^{2}+\eta n/d})$.

Then the difference between $\|\bm{v}_{mm}\|_{2}^{2}$ and $\|\widehat{\bm{v}}\|_{2}^{2}$ becomes

$$ $\displaystyle\|\widehat{\bm{v}}\|^{2}-\|\bm{v}_{mm}\|^{2}\geq$ $\displaystyle\sum\limits_{i\in\mathcal{C}}\widehat{\theta}^{2}_{i}\|\bm{\xi}_{i}\|^{2}-2/\rho^{2}+\underbrace{\sum\limits_{i\in\mathcal{N}}(\widehat{\theta}^{2}_{i}-\theta_{i}^{2})\|\bm{\xi}_{i}\|^{2}}_{I_{1}}+\underbrace{\sum\limits_{i\in[n]}\sum\limits_{j\in[n]\backslash\{i\}}y_{i}y_{j}\widehat{\theta}_{i}\widehat{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{2}}$ $\displaystyle-\underbrace{\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}\backslash\{i\}}y_{i}y_{j}\theta_{i}\theta_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{3}}.$ $$

We will bound every term sequentially. For $i\in\mathcal{C}$, we have

$$ $\displaystyle\widehat{\theta}_{i}\|\bm{\xi}_{i}\|^{2}$ $\displaystyle\geq\alpha-\sum\limits_{i^{\prime}\in[n],i^{\prime}\neq i}y_{i}\widehat{\theta}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq\alpha-n\max\limits_{i\in[n]}\widehat{\theta}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=\alpha-\frac{2\alpha n\sqrt{\log(6n^{2}/\delta)}}{(1-\kappa)\sqrt{d}-2n\sqrt{\log(6n^{2}/\delta)}}=\alpha-\widetilde{O}\bigg{(}\frac{n}{\sqrt{d}}\bigg{)}.$ $$

The second inequality is from Lemma [57](#Thmtheorem57); The first equality is from Lemma [29](#Thmtheorem29) and the last equality is from Assumption [5](#Thmtheorem5). This implies

$$ $\displaystyle\sum\limits_{i\in\mathcal{C}}\widehat{\theta}^{2}_{i}\|\bm{\xi}_{i}\|^{2}-2/\rho^{2}$ $\displaystyle\geq\frac{n_{1}\alpha^{2}}{(1+\kappa)d}-\frac{2}{\rho^{2}}-\widetilde{O}\bigg{(}\frac{n}{d^{3/2}}\bigg{)}\geq\frac{C_{2}n_{1}\alpha^{2}}{(1+\kappa)d}-\widetilde{O}\bigg{(}\frac{n}{d^{3/2}}\bigg{)}.$ $$

The second inequality is due to the SNR condition $\rho/\sqrt{d}=\Omega(1/\sqrt{n})$ so there exists a constant $C_{2}$ that $\frac{2}{\rho^{2}}\leq\frac{(1-C_{2})n_{1}\alpha^{2}}{(1+\kappa)d}$.

Then for $|I_{1}|$ we have

$$ $\displaystyle|I_{1}|$ $\displaystyle\leq(\max\limits_{i\in\mathcal{N}}\theta^{2}_{i}-\min\limits_{i\in\mathcal{N}}\widehat{\theta}^{2}_{i})\sum\limits_{i\in\mathcal{N}}\|\bm{\xi}_{i}\|^{2}$ $\displaystyle\leq\bigg{(}\bigg{(}\frac{1}{(1-\kappa)d-2\eta n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}^{2}-\bigg{(}\frac{1}{(1+\kappa)d}\bigg{(}1-\frac{2n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}\bigg{)}^{2}\bigg{)}\cdot n_{2}(1+\kappa)d$ $\displaystyle\leq\bigg{(}\frac{\sqrt{(1+\kappa)d}}{(1-\kappa)d-2\eta n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}^{2}\bigg{(}1-\bigg{(}\frac{(1-\kappa)d-4\eta n\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d}\bigg{)}^{2}\bigg{)}\cdot n_{2}$ $\displaystyle=\Theta\bigg{(}\frac{1}{d}\bigg{)}\cdot\Theta\bigg{(}\frac{\eta n\sqrt{\log(6n^{2}/\delta)}}{\sqrt{d}}\bigg{)}\cdot n_{2}$ $\displaystyle=\widetilde{O}\bigg{(}\frac{\eta^{2}n^{2}}{d^{3/2}}\bigg{)}.$ $$

The second inequality is from Lemma [27](#Thmtheorem27) and Lemma [31](#Thmtheorem31); The third inequality is from the fact that $\eta<1$.

As for the last two terms, we bound them respectively, for
$I_{2}$ we have

$$ $\displaystyle|I_{2}|$ $\displaystyle\leq\sum\limits_{i\in[n]}\sum\limits_{j\in[n]\backslash\{i\}}|y_{i}y_{j}\widehat{\theta}_{i}\widehat{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|\leq n^{2}\max\limits_{i\in[n]}\widehat{\theta}^{2}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq n^{2}\frac{\alpha^{2}}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})^{2}}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=\widetilde{O}\bigg{(}\frac{n^{2}}{d^{3/2}}\bigg{)}.$ $$

The first inequality is from triangle inequality; The second inequality is from Lemma [57](#Thmtheorem57); The third inequality is from Lemma [29](#Thmtheorem29). Last for $I_{3}$, we have

$$ $\displaystyle|I_{3}|$ $\displaystyle\leq\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}\backslash\{i\}}|y_{i}y_{j}\theta_{i}\theta_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|\leq(n_{2})^{2}\max\limits_{i\in\mathcal{N}}\theta_{i}^{2}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq(n_{2})^{2}\frac{1}{((1-\kappa)d-2\eta n\sqrt{d\log(6n^{2}/\delta)})^{2}}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=\widetilde{O}\bigg{(}\frac{\eta^{2}n^{2}}{d^{3/2}}\bigg{)}.$ $$

The first inequality is from triangle inequality; The second inequality is from Lemma [57](#Thmtheorem57); The third inequality is from Lemma [27](#Thmtheorem27). Combining the results above, we have

$$ $\displaystyle\|\bm{v}^{\prime}\|^{2}-\|\bm{v}_{mm}\|^{2}\geq\frac{C_{2}n_{1}(1-\beta q)^{2}}{(1-\beta)^{2}(1+\kappa)d}+\widetilde{O}\bigg{(}\frac{n^{2}}{d^{3/2}}\bigg{)}\geq\frac{C_{1}(1-\beta q)^{2}}{(1-\beta)^{2}(1+\kappa)d}.$ $$

Therefore, we could then use the same method as above to prove that Proposition [15](#Thmtheorem15) also holds in this case.

Case 1.3 $p<n_{1}$, $I_{v}\cap\mathcal{C}_{i}\neq\varnothing$, $I_{v}\cap\mathcal{C}_{i^{\prime}}=\varnothing$

For the case when only one of the clusters in clean sets are all mixed, we can follow similar method in Case 1.2 to prove that Lemma [30](#Thmtheorem30) still holds. Without losing generality, assume all clean samples with label $y_{i}=+1$ violate optimal token selection while only part of clean samples with label $y_{i}=-1$ violate. we have

###### Condition 6 (One cluster and a clean sample in the opposite cluster violating optimal token selection) .

$$ $\begin{cases}&-\bm{v}^{\top}\bm{\mu}_{2}\geq 1\\ &y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in\mathcal{N}\\ &y_{i}\bm{v}^{\top}\bm{r}_{i}\geq 1,i\in\mathcal{C}_{+1}\\ &y_{i}\bm{v}^{\top}\bm{r}_{i}\geq 1,i\in\mathcal{C}_{-1}\cap I_{v}\end{cases}$ $$

Similar to previous analysis, mixing multiple samples with label $-1$ will not result in a better solution than only mixing one sample with label $-1$. Thus we can reduce this case to mixing only one clean sample and denote this mixed sample as $k_{-1}$. Therefore, we have

$$ $\begin{cases}&-\lambda_{2}^{\prime}\cdot\|\bm{\mu}_{2}\|^{2}\geq 1\\ &\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in\mathcal{N}\\ &y_{k_{-1}}\beta\lambda_{2}^{\prime}\cdot\|\bm{\mu}_{2}\|^{2}+(1-\beta)(\theta^{\prime}_{k_{-1}}\cdot\|\bm{\xi}_{k_{-1}}\|^{2}+\sum\limits_{i\neq{k_{-1}}}y_{k_{-1}}y_{i}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k_{-1}}\rangle)\geq 1\\ &\beta\lambda_{1}^{\prime}\cdot\|\bm{\mu}_{1}\|^{2}+(1-\beta)(\theta^{\prime}_{k_{i}}\cdot\|\bm{\xi}_{k_{i}}\|^{2}+\sum\limits_{i\neq{k_{i}}}y_{k_{i}}y_{i}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k_{i}}\rangle)\geq 1,i\in\mathcal{C}_{+1}\end{cases}$ $$

Denote $q=\lambda^{\prime}_{1}\cdot\|\bm{\mu}_{1}\|^{2}$ and $q\leq 1$. Denote $k^{\star}=\mathop{\mathrm{argmin}}\limits_{i\in\mathcal{C}_{+1}}\frac{1-\beta_{i}q}{1-\beta_{i}}$ and $\beta=\beta_{k^{\star}}$, we can further reduce the condition to

###### Condition 7 (Relaxed version of Condition 6 ) .

$$ $\begin{cases}&\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in\mathcal{N}\\ &\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq\frac{1-\beta q}{1-\beta},i\in\mathcal{C}_{+1}\end{cases}$ $$

Condition [7](#Thmcondition7) relax the constraints in Condition [6](#Thmcondition6). Meanwhile, it differs from Condition [4](#Thmcondition4) only in that the last inequality holds for clean samples with label $+1$. Therefore, we can follow the proof above to show that Lemma [30](#Thmtheorem30) still holds in this case.

∎

Then we consider the second scenario.
Case 2: $p=0,k-p\neq 0$

Similar to the previous part, there are two cases we need to consider under this scenario:

- 1.
$k-p<n_{2}$.
- 2.
$k-p=n_{2}$.

We will go over every case sequentially.
Case 2.1 $k-p<n_{2}$

In this case, part of noisy samples are mixed. Denote the mixed samples as $k_{1},k_{2},...,k_{k-p}$. And for every mixed sample $k_{i}$, we have $\bm{r}_{i}=\beta_{i}\bm{\xi}_{k_{i}}+(1-\beta_{i})\bm{\mu}_{k_{i}}$. Then the conditions under Case 2.1 become:

###### Condition 8 ( $k-p$ noisy samples violating optimal token selection rule) .

$$ $\begin{cases}&\bm{v}^{\top}\bm{\mu}_{1}\geq 1\\ &-\bm{v}^{\top}\bm{\mu}_{2}\geq 1\\ &y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in\mathcal{N},i\notin[k-p]\\ &y_{k_{i}}\bm{v}^{\top}\bm{r}_{k_{i}}\geq 1,i\in[k-p]\end{cases}$ $$

We could also write the last inequality as

$$ $y_{k_{i}}\beta_{i}\bm{v}^{\top}\bm{\xi}_{k_{i}}+y_{k_{i}}(1-\beta_{i})\bm{v}^{\top}\bm{\mu}_{k_{i}}\geq 1,i\in[k-p].$ $$

Therefore,

$$ $y_{k_{i}}\bm{v}^{\top}\bm{\xi}_{k_{i}}\geq(1-y_{k_{i}}(1-\beta_{i})\bm{v}^{\top}\bm{\mu}_{k_{i}})/\beta_{i},i\in[k-p].$ $$

For noisy samples, we have $y_{i}=-1$ when $\bm{\mu}_{i}=\bm{\mu}_{1}$ and $y_{i}=1$ when $\bm{\mu}_{i}=\bm{\mu}_{2}$, so $y_{k_{i}}\bm{v}^{\top}\bm{\mu}_{k_{i}}\leq 0$ and thus $(1-y_{k_{i}}(1-\beta_{i})\bm{v}^{\top}\bm{\mu}_{k_{i}})/\beta_{i}\geq 1$. Compared to the constraint in Condition [1](#Thmcondition1) that $y_{k_{i}}\bm{v}^{\top}\bm{\mu}_{k_{i}}\geq 1,i\in\mathcal{N}$, the new condition is strengthened. So mixing 1 more noisy samples is equal to strengthening 1 constraint in the original setting. Therefore, mixing $k-p$ samples will not result in a better solution than only mixing 1 noisy sample. Similarly, we can simplify this case to mixing only 1 noisy sample and denote this sample as $k_{*}$. We have $\bm{r}_{k^{*}}=\beta\bm{\xi}_{k^{*}}+(1-\beta)\bm{\mu}_{k^{*}}$ and assume that $\bm{\xi}_{k^{*}}=\bm{\mu}_{1}$.

Denote $\bm{v}^{\prime\prime}$ is the optimal solution under this condition, and the parameters in $\bm{v}^{\prime\prime}$ are $\lambda^{\prime\prime}_{1},\lambda^{\prime\prime}_{2}$ and $\theta^{\prime\prime}_{i}$. Then the conditions become:

###### Condition 9 (1 noisy sample violating optimal token selection rule) .

$$ $\begin{cases}&\bm{v}^{\top}\bm{\mu}_{1}\geq 1\\ &-\bm{v}^{\top}\bm{\mu}_{2}\geq 1\\ &y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in\mathcal{N},i\neq k^{\star}\\ &y_{k^{\star}}\bm{v}^{\top}\bm{r}_{k^{\star}}\geq 1\end{cases}$ $$

Plugging the representation ([A.37](#A1.E37)) into the condition, we have:

$$ $\begin{cases}&\lambda_{1}^{\prime\prime}\cdot\|\bm{\mu}_{1}\|^{2}\geq 1\\ &-\lambda_{2}^{\prime\prime}\cdot\|\bm{\mu}_{2}\|^{2}\geq 1\\ &\theta_{i}^{\prime\prime}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in\mathcal{N},i\neq k^{\star}\\ &-(1-\beta)\lambda_{1}^{\prime\prime}\cdot\|\bm{\mu}_{1}\|^{2}+\beta(\theta^{\prime\prime}_{k^{\star}}\cdot\|\bm{\xi}_{k^{\star}}\|^{2}+\sum\limits_{i\neq{k^{\star}}}y_{k^{\star}}y_{i}\theta^{\prime\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle)\geq 1\end{cases}$ $$

We first introduce the following lemma which estimates the parameters of the noises. We define

$$ $\alpha=\frac{1+(1-\beta)\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta}$ $$

for the convenience of the following proof.

###### Lemma 32 .

Suppose that Assumption [5](#Thmtheorem5) holds, under Condition [9](#Thmcondition9), we have

$$ $\displaystyle\theta^{\prime\prime}_{k^{\star}}$ $\displaystyle\leq\frac{\alpha}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}$ $\displaystyle\theta^{\prime\prime}_{k^{\star}}$ $\displaystyle\geq\frac{\alpha}{(1+\kappa)d}\bigg{(}1-\frac{2n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}$ $\displaystyle\max\limits_{i\in\mathcal{N},i\neq k^{\star}}\theta^{\prime\prime}_{i}$ $\displaystyle\leq\frac{(1-\kappa)d+2(\alpha-n_{2})\sqrt{d\log(6n^{2}/\delta)}}{((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)})^{2}}$ $\displaystyle\min\limits_{i\in\mathcal{N},i\neq k^{\star}}\theta^{\prime\prime}_{i}$ $\displaystyle\geq\frac{1}{(1+\kappa)d}\cdot\bigg{(}1-\frac{2\alpha n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}.$ $$

###### Proof of Lemma 31 .

From the last inequality in Condition [9](#Thmcondition9) we have

$$ $\displaystyle\theta^{\prime\prime}_{k_{*}}\|\bm{\xi}_{k_{*}}\|^{2}+\sum\limits_{i\in\mathcal{N},i\neq{k_{*}}}y_{i}y_{k_{*}}\theta^{\prime\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k_{*}}\rangle\geq\alpha>1.$ $$

The last inequality is because $\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1$ and $0<\beta<1$. Denote $j=\mathop{\mathrm{argmax}}\limits_{i\in[n]}\theta^{\prime\prime}_{i}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\prime\prime\top}\bm{\xi}_{j}$ $\displaystyle=\theta^{\prime\prime}_{j}\|\bm{\xi}_{j}\|^{2}+\sum\limits_{i\in\mathcal{N},i\neq j}y_{i}y_{j}\theta^{\prime\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\geq\theta^{\prime\prime}_{j}(1-\kappa)d-n_{2}\max\limits_{i\in[n]}\theta^{\prime\prime}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=\theta^{\prime\prime}_{j}((1-\kappa)d-n_{2}\cdot 2\sqrt{d\log(6n^{2}/\delta)})$ $$

The first inequality is due to Lemma [57](#Thmtheorem57) and the last equation is from our definition of j. Consider the contrary case when $\theta^{\prime\prime}_{j}>\frac{\alpha}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\prime\prime\top}\bm{\xi}_{j}$ $\displaystyle>\alpha.$ $$

By the complementary slackness condition, if $y_{j}\bm{v}^{\prime\prime\top}\bm{\xi}_{j}>\frac{1+\lambda^{\prime\prime}_{1}(1-\beta)\|\bm{\mu}_{1}\|^{2}}{\beta}$ then we must have $\theta^{\prime\prime}_{j}=0$, and thus we reach a contradiction. Therefore, we have $\theta^{\prime\prime}_{k^{\star}}\leq\theta^{\prime\prime}_{j}\leq\frac{\alpha}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}$. Then denote $j^{\prime}=\mathop{\mathrm{argmax}}\limits_{i\in[n],i\neq k^{\star}}\theta^{\prime\prime}_{i}$, we have

$$ $\displaystyle y_{j^{\prime}}\bm{v}^{\prime\prime\top}\bm{\xi}_{j^{\prime}}$ $\displaystyle=\theta^{\prime\prime}_{j^{\prime}}\|\bm{\xi}_{j^{\prime}}\|^{2}+\sum\limits_{i\in\mathcal{N},i\neq j^{\prime}}y_{i}y_{j^{\prime}}\theta^{\prime\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j^{\prime}}\rangle$ $\displaystyle\geq\theta^{\prime\prime}_{j^{\prime}}(1-\kappa)d-n_{2}\max\limits_{i\in[n],i\neq j^{\prime}}\theta^{\prime\prime}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}-\theta^{\prime\prime}_{k^{\star}}\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\geq\theta^{\prime\prime}_{j}((1-\kappa)d-n_{2}\cdot 2\sqrt{d\log(6n^{2}/\delta)})-\frac{2\alpha\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}.$ $$

The first inequality is from Lemma [57](#Thmtheorem57) and the second inequality is from the upper bound of $\theta^{\prime\prime}_{k^{\star}}$ we just get. Consider the case when $\theta^{\prime\prime}_{j^{\prime}}>\frac{(1-\kappa)d+2(\alpha-n_{2})\sqrt{d\log(6n^{2}/\delta)}}{((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)})^{2}}$, we have

$$ $\displaystyle y_{j^{\prime}}\bm{v}^{\prime\prime\top}\bm{\xi}_{j^{\prime}}>1.$ $$

By the complementary slackness condition, if $y_{j^{\prime}}\bm{v}^{\prime\prime\top}\bm{\xi}_{j^{\prime}}>1$ then we must have $\theta^{\prime\prime}_{j^{\prime}}=0$, and thus we reach a contradiction.

Then we estimate the lower bound of $\theta^{\prime\prime}_{j}$ when $j\neq k_{*}$. We have

$$ $\displaystyle 1$ $\displaystyle\leq y_{j}\bm{v}^{\prime\prime\top}\bm{\xi}_{j}=\theta^{\prime\prime}_{j}\|\bm{\xi}_{j}\|^{2}+\sum\limits_{i\in[n],i\neq j}y_{i}y_{j}\theta^{\prime\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle\leq\theta^{\prime\prime}_{j}(1+\kappa)d+n_{2}\max\limits_{i\in[n]}\theta^{\prime\prime}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq\theta^{\prime\prime}_{j}(1+\kappa)d+\frac{1+\lambda^{\prime\prime}_{1}(1-\beta)\|\bm{\mu}_{1}\|^{2}}{\beta((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\cdot 2n_{2}\sqrt{d\log(6n^{2}/\delta)},$ $$

where the last inequality is from the upper bound we just get. Therefore, we have

$$ $\displaystyle\theta^{\prime\prime}_{j}\geq\frac{1}{(1+\kappa)d}\cdot\bigg{(}1-\frac{2n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\cdot\frac{1+\lambda^{\prime\prime}_{1}(1-\beta)\|\bm{\mu}_{1}\|^{2}}{\beta}\bigg{)}$ $$

for all $j\in\mathcal{N}$ and $j\neq{k_{*}}$.
Lastly we lower bound $\theta^{\prime\prime}_{k_{*}}$. We have

$$ $\displaystyle\frac{1+(1-\beta)\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta}$ $\displaystyle\leq y_{k_{*}}\bm{v}^{\prime\prime\top}\bm{\xi}_{k_{*}}=\theta^{\prime\prime}_{k_{*}}(1+\kappa)d+n_{2}\max\limits_{i\in[n]}\theta^{\prime\prime}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}.$ $$

Similarly, we have

$$ $\displaystyle\theta^{\prime\prime}_{k_{*}}\geq\frac{1}{(1+\kappa)d}\cdot\frac{1+(1-\beta)\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta}\bigg{(}1-\frac{2n_{2}\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}.$ $$

∎

After getting the bound of parameters, we could derive the norm difference as above

###### Lemma 33 .

Suppose that Assumption [5](#Thmtheorem5) holds, denote $\bm{v}$ and $\bm{v}^{\prime\prime}$ as the optimal solutions under condition [1](#Thmcondition1) and condition [9](#Thmcondition9) respectively. We have

$$ $\displaystyle\|\bm{v}^{\prime\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\frac{C_{3}(1-\beta)}{d},$ $$

where $C_{3}=\Theta(1)$.

###### Proof of Lemma 33 .

From the third inequality in Condition [9](#Thmcondition9), for $i\in\mathcal{N},i\neq k^{\star}$ we have

$$ $\displaystyle\theta_{i}^{\prime\prime}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\neq i,k^{\star}}y_{i}y_{i^{\prime}}\theta^{\prime\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1-y_{i}y_{k^{\star}}\theta^{\prime\prime}_{k^{\star}}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle.$ $$

Then we add $y_{i}y_{k^{\star}}w\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle$ on both sides, where we set $w=\theta^{\prime\prime}_{k^{\star}}-\frac{\alpha-1}{(1+\kappa)d-2\sqrt{d\log(6n^{2}/\delta)}}\leq\theta^{\prime\prime}_{k^{\star}}$. Then we have

$$ $\displaystyle\theta_{i}^{\prime\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i,k^{\star}}y_{i}y_{i^{\prime}}\theta^{\prime\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle+y_{i}y_{k^{\star}}w\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle$ $\displaystyle\geq 1-y_{i}y_{k^{\star}}(\theta^{\prime\prime}_{k^{\star}}-w)\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle$ $\displaystyle\geq 1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=\frac{(1+\kappa)d-2\alpha\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d-2\sqrt{d\log(6n^{2}/\delta)}}.$ (A.43) $$

The second inequality is from Lemma [57](#Thmtheorem57). Now consider a new $\overline{\bm{v}}=\overline{\lambda}_{1}\bm{\mu}_{1}+\overline{\lambda}_{2}\bm{\mu}_{2}+\sum\limits_{i\in[n]}y_{i}\overline{\theta}_{i}\bm{\xi}_{i}$ with

$$ $\overline{\lambda}_{1}=\lambda^{\prime\prime}_{1};\quad\overline{\lambda}_{2}=\lambda^{\prime\prime}_{2};$ $$

$$ $\overline{\theta}_{i}=\theta^{\prime\prime}_{i}/(1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)})\text{ for }i\in[n],i\neq k^{\star}$ $$

and

$$ $\overline{\theta}_{k^{\star}}=\frac{w}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}.$ $$

We can prove that $\overline{\bm{v}}$ satisfies all constraints for $\bm{v}_{mm}$.

From the first two inequalities in Condition [9](#Thmcondition9), we have $\overline{\lambda}_{1}\|\bm{\mu}_{1}\|^{2}=\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1$, $-\overline{\lambda}_{2}\|\bm{\mu}_{2}\|^{2}=-\lambda^{\prime\prime}_{2}\|\bm{\mu}_{2}\|^{2}\geq 1$. Then by dividing $1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}$ on both sides of ([A.43](#A1.E43)), for $\forall i\in\mathcal{N},i\neq k^{\star}$ we have

$$ $\displaystyle\overline{\theta}_{i}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\overline{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1.$ $$

Lastly we prove that $\overline{\theta}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|^{2}+\sum\limits_{i\neq k^{\star}}y_{i}y_{k^{\star}}\overline{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle\geq 1$. From the last inequality in Condition [9](#Thmcondition9) we have

$$ $\displaystyle\theta^{\prime\prime}_{k^{\star}}\cdot\|\bm{\xi}_{k^{\star}}\|^{2}+\sum\limits_{i\neq{k^{\star}}}y_{k^{\star}}y_{i}\theta^{\prime\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle$ $\displaystyle\geq\alpha.$ $$

Dividing $1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}$ on both sides, we get

$$ $\displaystyle\frac{\theta^{\prime\prime}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|^{2}}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}+\sum\limits_{i\neq k^{\star}}y_{i}y_{k^{\star}}\overline{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle\geq\frac{\alpha}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}.$ $$

Therefore we have

$$ $\displaystyle\overline{\theta}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|^{2}+\sum\limits_{i\neq k^{\star}}y_{i}y_{k^{\star}}\overline{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle$ $\displaystyle\geq\frac{\alpha-(\theta^{\prime\prime}_{k^{\star}}-w)\|\bm{\xi}_{k^{\star}}\|^{2}}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}\geq\frac{\alpha-(\theta^{\prime\prime}_{k^{\star}}-w)(1+\kappa)d}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}=1.$ $$

The second inequality is from Lemma [57](#Thmtheorem57) and the last equality is by our definition $\theta^{\prime\prime}_{k^{\star}}-w=\frac{\alpha-1}{(1+\kappa)d-2\sqrt{d\log(6n^{2}/\delta)}}$. Thus, $\overline{\bm{v}}$ is a possible solution under Condition [1](#Thmcondition1) and $\|\overline{\bm{v}}\|\geq\|\bm{v}_{mm}\|$.

Next we estimate the difference between $\|\bm{v}^{\prime\prime}\|^{2}$ and $\|\overline{\bm{v}}\|^{2}$. The expansion of $\|\bm{v}^{\prime\prime}\|^{2}$ and $\|\overline{\bm{v}}\|^{2}$ are:

$$ $\displaystyle\|\bm{v}^{\prime\prime}\|^{2}$ $\displaystyle=\lambda^{\prime\prime 2}_{1}\|\bm{\mu}_{1}\|^{2}+\lambda^{\prime\prime 2}_{2}\|\bm{\mu}_{2}\|^{2}+\sum\limits_{i\in\mathcal{N}}\theta^{\prime\prime 2}_{i}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}}y_{i}y_{j}\theta^{\prime\prime}_{i}\theta^{\prime\prime}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle,$ $\displaystyle\|\overline{\bm{v}}\|^{2}$ $\displaystyle=\overline{\lambda}_{1}^{2}\|\bm{\mu}_{1}\|^{2}+\overline{\lambda}_{2}^{2}\|\bm{\mu}_{2}\|^{2}+\sum\limits_{i\in\mathcal{N}}\overline{\theta}_{i}^{2}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}}y_{i}y_{j}\overline{\theta}_{i}\overline{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle.$ $$

According to the condition ([A.38](#A1.E38)), we have $\|\bm{v}^{\prime\prime}\|\leq 2\|\bm{v}_{mm}\|=\Theta(\sqrt{1/\rho^{2}+\eta n/d})$, which implies that $\alpha=O(\sqrt{n}\log n)$. Otherwise, we have

$$ $\theta_{k^{\star}}^{\prime\prime}\|\bm{\xi}_{k^{\star}}\|^{2}\geq\alpha-\sum_{i\neq k^{\star}}y_{k^{\star}}y_{i}\theta_{i}^{\prime\prime}\langle\bm{\xi}_{i},\bm{\xi}_{k^{\star}}\rangle=\Omega(\alpha).$ $$

It further yields that

$$ $\|\bm{v}^{\prime\prime}\|^{2}=\Omega(\frac{1}{\rho^{2}})+\Omega(\frac{\eta n}{d})+\theta_{k^{\star}}^{\prime\prime 2}\|\bm{\xi}_{k^{\star}}\|^{2}=\Omega(\frac{1}{\rho^{2}}+\frac{\eta n}{d}+\frac{\alpha^{2}}{d})=\Omega(\frac{n\log^{2}n}{d}),$ $$

which contradicts with $\|\bm{v}^{\prime\prime}\|=\Theta(\sqrt{1/\rho^{2}+\eta n/d})$.
We decompose the difference between $\|\bm{v}^{\prime\prime}\|^{2}$ and $\|\overline{\bm{v}}\|^{2}$ into four terms:

$$ $\displaystyle\|\bm{v}^{\prime\prime}\|^{2}-\|\overline{\bm{v}}\|^{2}=$ $\displaystyle\underbrace{(\theta^{\prime\prime 2}_{k^{\star}}-\overline{\theta}^{2}_{k^{\star}})\|\bm{\xi}_{k^{\star}}\|^{2}}_{I_{1}}+\underbrace{\sum\limits_{i\in\mathcal{N},i\neq{k^{\star}}}(\theta^{\prime\prime 2}_{i}-\overline{\theta}^{2}_{i})\|\bm{\xi}_{i}\|^{2}}_{I_{2}}-\underbrace{\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}}y_{i}y_{j}\overline{\theta}_{i}\overline{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{3}}$ $\displaystyle+\underbrace{\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}}y_{i}y_{j}\theta^{\prime\prime}_{i}\theta^{\prime\prime}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{4}}.$ $$

We now estimate $I_{1}$ to $I_{4}$ sequentially. For the first term,

$$ $\displaystyle I_{1}$ $\displaystyle\geq(\theta^{\prime\prime 2}_{k^{\star}}-\overline{\theta}^{2}_{k^{\star}})(1-\kappa)d=(\theta^{\prime\prime}_{k^{\star}}-\overline{\theta}_{k^{\star}})(\theta^{\prime\prime}_{k^{\star}}+\overline{\theta}_{k^{\star}})(1-\kappa)d$ $\displaystyle=\frac{(\alpha-1)(1-2\theta^{\prime\prime}_{k^{\star}}\sqrt{d\log(6n^{2}/\delta)})}{(1+\kappa)d-2\sqrt{d\log(6n^{2}/\delta)}}\cdot\Omega\bigg{(}\frac{1}{d}\bigg{)}\cdot(1-\kappa)d$ $\displaystyle=\Omega\bigg{(}\frac{\alpha-1}{d}\bigg{)},$ $$

where the first inequality is from Lemma [57](#Thmtheorem57); the second equality is from Lemma [31](#Thmtheorem31); and the last equality uses the fact that $\alpha=O(\sqrt{n}\log n)$. Then we can further upper bound $\max\limits_{i\in\mathcal{N},i\neq k^{\star}}\theta^{\prime\prime}_{i}$ as

$$ $\displaystyle\max\limits_{i\in\mathcal{N},i\neq k^{\star}}\theta^{\prime\prime}_{i}$ $\displaystyle\leq\frac{(1-\kappa)d+2(\alpha-n_{2})\sqrt{d\log(6n^{2}/\delta)}}{((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)})^{2}}=O(\frac{1}{d}).$ (A.44) $$

For the second term $I_{2}$, we have

$$ $\displaystyle|I_{2}|$ $\displaystyle\leq\sum\limits_{i\in\mathcal{N},i\neq{k^{\star}}}(\overline{\theta}^{2}_{i}-\theta^{\prime\prime 2}_{i})(1+\kappa)d$ $\displaystyle\leq\bigg{(}\frac{1}{(1-(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)})^{2}}-1\bigg{)}\max\limits_{i\in\mathcal{N},i\neq k^{\star}}\theta^{\prime\prime 2}_{i}\cdot\eta n(1+\kappa)d$ $\displaystyle=\frac{(\alpha-1)\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d-\sqrt{d\log(6n^{2}/\delta)}}\cdot O(\frac{\eta n}{d})=\widetilde{O}\bigg{(}\frac{(\alpha-1)\eta n}{d^{3/2}}\bigg{)}.$ $$

The second inequality is from Lemma [31](#Thmtheorem31). The first equality is from ([A.44](#A1.E44)) and the last equality is from Assumption [5](#Thmtheorem5).

Then we bound $|-I_{3}+I_{4}|$ as:

$$ $\displaystyle|-I_{3}+I_{4}|\leq$ $\displaystyle\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}\backslash\{i\}}|\overline{\theta}_{i}\overline{\theta}_{j}-\theta^{\prime\prime}_{i}\theta^{\prime\prime}_{j}|\cdot|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|$ $\displaystyle\leq$ $\displaystyle\sum\limits_{i\in\mathcal{N}\backslash\{k^{\star}\}}\sum\limits_{j\in\mathcal{N}\backslash\{k^{\star},i\}}|\overline{\theta}_{i}\overline{\theta}_{j}-\theta^{\prime\prime}_{i}\theta^{\prime\prime}_{j}|\cdot|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|+2\sum\limits_{t\in\mathcal{N}\backslash\{k^{\star}\}}|\overline{\theta}_{k^{\star}}\overline{\theta}_{t}-\theta^{\prime\prime}_{k^{\star}}\theta^{\prime\prime}_{t}|\cdot|\langle\bm{\xi}_{k^{\star}},\bm{\xi}_{t}\rangle|$ $\displaystyle\leq$ $\displaystyle(\eta n)^{2}\bigg{(}\frac{1}{(1-(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)})^{2}}-1\bigg{)}\max\limits_{i\in\mathcal{N},i\neq k^{\star}}\theta^{\prime\prime 2}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle+\eta n\bigg{(}\theta^{\prime\prime}_{k^{\star}}-\frac{\overline{\theta}_{k^{\star}}}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}\max\limits_{i\in\mathcal{N},i\neq k^{\star}}\theta^{\prime\prime}_{i}4\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq$ $\displaystyle\frac{(\alpha-1)\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d-\sqrt{d\log(6n^{2}/\delta)}}\cdot O(\frac{(\eta n)^{2}(1+\kappa)}{d^{3/2}})+\frac{\alpha-1}{d}\cdot O(\eta n\frac{c_{1}}{d})\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=$ $\displaystyle O\bigg{(}\frac{(\alpha-1)\eta^{2}n^{2}}{d^{2}}+\frac{(\alpha-1)\eta n}{d^{3/2}}\bigg{)}.$ $$

The third inequality is from Lemma [27](#Thmtheorem27) and Lemma [31](#Thmtheorem31); The fourth inequality is from the fact that

$$ $\displaystyle\theta^{\prime\prime}_{k^{\star}}-\frac{\overline{\theta}_{k^{\star}}}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}$ $\displaystyle=\frac{\theta^{\prime\prime}_{k^{\star}}-\overline{\theta}_{k^{\star}}-2\theta^{\prime\prime}_{k^{\star}}(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}$ $\displaystyle=\frac{\Omega(\frac{\alpha-1}{d})-O(\frac{\alpha(\alpha-1)}{d^{3/2}})}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}>0$ $$

So we have $\theta^{\prime\prime}_{k^{\star}}-\frac{\overline{\theta}_{k^{\star}}}{1-2(\theta^{\prime\prime}_{k^{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}\leq\theta^{\prime\prime}_{k^{\star}}-\overline{\theta}_{k^{\star}}$; The last equality is from Assumption [5](#Thmtheorem5).

Combining the above results, we have

$$ $\displaystyle\|\bm{v}^{\prime\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\Theta\bigg{(}\frac{\alpha-1}{d}\bigg{)}+O\bigg{(}\frac{(\alpha-1)\eta n}{d^{3/2}}\bigg{)}\geq\frac{C_{3}(1-\beta)}{d}.$ $$

Here $C_{3}=\Theta(1)$ is a constant.
∎

Now we can prove the main proposition in this case.

###### Proof of Proposition 15 under Case 2.1.

From Lemma [33](#Thmtheorem33) we have

$$ $\displaystyle\|\bm{v}^{\prime\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}$ $\displaystyle\geq\frac{C_{3}(1-\beta)}{d}=T^{\prime}(1-\beta).$ $$

Here we substitute $T^{\prime}=\frac{C_{3}}{d}\geq 0$. Then we have

$$ $\displaystyle\Gamma^{2}-\Gamma^{\prime\prime 2}$ $\displaystyle=\frac{1}{\|\bm{v}_{mm}\|^{2}}-\frac{1}{\|\bm{v}^{\prime\prime}\|^{2}}=\frac{\|\bm{v}^{\prime\prime}\|^{2}-\|\bm{v}_{mm}\|^{2}}{\|\bm{v}^{\prime\prime}\|^{2}\cdot\|\bm{v}_{mm}\|^{2}}\geq\frac{T^{\prime}(1-\beta)}{\|\bm{v}^{\prime\prime}\|^{2}\cdot\|\bm{v}_{mm}\|^{2}}.$ $$

Therefore,

$$ $\displaystyle\Gamma-\Gamma^{\prime\prime}$ $\displaystyle\geq\frac{T^{\prime}(1-\beta)}{(\Gamma+\Gamma^{\prime\prime})\|\bm{v}_{mm}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}\geq\frac{T^{\prime}(1-\beta)}{2\Gamma\|\bm{v}_{mm}\|^{2}\cdot\|\bm{v}^{\prime\prime}\|^{2}}=\frac{T^{\prime}(1-\beta)}{2\|\bm{v}_{mm}\|\|\bm{v}^{\prime\prime}\|^{2}}\geq\frac{T^{\prime}(1-\beta)}{2\|\bm{v}^{\prime\prime}\|^{3}}.$ $$

The last inequality is from $\|\bm{v}^{\prime\prime}\|\geq\|\bm{v}_{mm}\|$. This implies

$$ $\displaystyle\Gamma^{\prime\prime}\leq\Gamma-\frac{T^{\prime}(1-\beta)}{2\|\bm{v}^{\prime\prime}\|^{3}}\leq\Gamma-\frac{C_{1}}{\|\bm{v}_{mm}\|^{3}n\rho^{2}}(1-\beta).$ $$

The last inequality is from our assumption that $\|\bm{v}^{\prime\prime}\|\leq 2\|\bm{v}_{mm}\|$ and $\rho^{2}=\Omega(d/n)$.
∎

Then we consider the other case.
Case 2.2 $k-p=n_{2}$

In this case, all noisy samples are mixed. From previous analysis, this is equivalent to strengthening all conditions $y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1$ while other conditions remain the same. As mixing $k-p$ samples will not result in a better solution than only mixing 1 noisy sample, the proof is the same as Case 2.1 and we omit it for convenience.
Finally, we consider the last scenario.
Case 3: $p\neq 0,k-p\neq 0$

This scenario is more complex as both clean and noisy sets are mixed. There are four cases to consider

- 1.
$p<n_{1},k-p<n_{2}.$ (Both clean and noisy sets are partially mixed)
- 2.
$p<n_{1},k-p=n_{2}$ (Clean set is partially mixed, noisy set is all mixed)
- 3.
$p=n_{1},k-p<n_{2}$ (Clean set is all mixed, noisy set is partially mixed)
- 4.
$p=n_{1},k-p=n_{2}$ (Both clean and noisy sets are all mixed)

We will go over every case to prove Proposition [15](#Thmtheorem15) holds.
Case 3.1 $p<n_{1},k-p<n_{2}$

This case is simple because from the analysis above, mixing 1 more clean sample is equivalent to adding 1 more constraint and mixing 1 more noisy sample is equivalent to strengthening 1 original constraint. So mixing both sets will not result in a better solution than only mixing 1 clean sample. Therefore, the proof is the same as Case 1.1 and we omit is for convenience.
Case 3.2 $p<n_{1},k-p=n_{2}$

In this case, all noisy samples and part of clean samples are mixed. We can consider this case as an extension of Case 2.2 by mixing some clean samples. From previous analysis, mixing 1 more clean sample is equivalent to adding 1 more constraint. So this case will not result in a better solution than Case 2.2. The following proof is the same as Case 2.2 and we omit it for convenience.
Case 3.3 $p=n_{1},k-p<n_{2}$

In this case, all clean samples and part of noisy samples are mixed. We can consider this case as an extension of Case 1.2 by mixing some noisy samples. From previous analysis, mixing 1 more noisy sample is equivalent to strengthening 1 original constraint. So this case will not result in a better solution than Case 1.2. The following proof is the same as Case 1.2 and we omit it for convenience.
Case 3.4 $p=n_{1},k-p=n_{2}$

This case is more complex. We cannot simply consider it as an extension of Case 2.2 because the analysis of Case 2.2 is based on the condition that there exist clean samples that follow optimal token selection rule. Denote $\bm{r}_{i}=\beta_{i}\bm{\mu}_{i}+(1-\beta_{i})\bm{\xi}_{i}$ for $i\in\mathcal{C}$ and $\bm{r}_{i}=(1-\beta_{i})\bm{\mu}_{i}+\beta_{i}\bm{\xi}_{i}$ for $i\in\mathcal{N}$. The condition in this case becomes

###### Condition 10 (All samples are mixed) .

$$ $y_{i}\bm{v}^{\prime\prime\top}\bm{r}_{i}\geq 1.$ $$

This indicates

$$ $\begin{cases}\beta_{i}y_{i}\lambda^{\prime\prime}_{i}\|\bm{\mu}_{i}\|^{2}+(1-\beta_{i})(\theta^{\prime\prime}_{i}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{j\neq i}y_{i}y_{j}\theta_{j}^{\prime\prime}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle)\geq 1,i\in\mathcal{C},\\ (1-\beta_{i})y_{i}\lambda^{\prime\prime}_{i}\|\bm{\mu}_{i}\|^{2}+\beta_{i}(\theta^{\prime\prime}_{i}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{j\neq i}y_{i}y_{j}\theta_{j}^{\prime\prime}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle)\geq 1,i\in\mathcal{N}.\end{cases}$ $$

Assume that $\min\{\lambda^{\prime\prime}_{1}\cdot\|\bm{\mu}_{1}\|^{2},-\lambda^{\prime\prime}_{2}\cdot\|\bm{\mu}_{2}\|^{2}\}=q$ in optimal $\bm{v}^{\prime\prime}$. If $q\geq 1$, we can directly follow the proof in Case 2.2. Otherwise, denote $\alpha=\frac{1-\beta_{i}q}{1-\beta_{i}}$. We have $\alpha>1$ due to $q<1$ and $0\leq\beta_{i}<1$.
Without losing generality, we assume $\lambda^{\prime\prime}_{1}\cdot\|\bm{\mu}_{1}\|^{2}=q<1$. Then consider the following relaxed condition

###### Condition 11 (Relaxed version of constraints in Condition 10 ) .

$$ $\theta^{\prime\prime}_{i}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{j\neq i}y_{i}y_{j}\theta_{j}^{\prime\prime}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle\geq\alpha,i\in\mathcal{C}_{1}.$ $$

Denote the optimal solution under Condition [11](#Thmcondition11) as $\breve{\bm{v}}$ and the corresponding coefficients in $\breve{\bm{v}}$ as $\breve{\lambda}_{1},\breve{\lambda}_{2}$ and $\breve{\theta}_{i}$, i.e.

$$ $\breve{\bm{v}}=\breve{\lambda}_{1}\bm{\mu}_{1}+\breve{\lambda}_{2}\bm{\mu}_{2}+\sum_{i\in[n]}\breve{\theta}_{i}\bm{\xi}_{i}.$ $$

Since the constraints in Condition [11](#Thmcondition11) is a subset of the constraints in Condition [10](#Thmcondition10),
we have $\|\breve{\bm{v}}\|\leq\|\bm{v}^{\prime\prime}\|$. Meanwhile, we have the following lemma to estimate $\breve{\theta_{i}}$:

###### Lemma 34 .

Suppose that Assumption [5](#Thmtheorem5) holds, under Condition [11](#Thmcondition11), we have

$$ $\displaystyle\breve{\theta_{i}}=0,i\in[n]\backslash\mathcal{C}_{1};$ $$

$$ $\displaystyle\breve{\theta_{i}}\in\bigg{[}\frac{\alpha}{(1+\kappa)d}\bigg{(}1-\frac{n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)},\frac{\alpha}{((1-\kappa)d-2n_{11}\sqrt{d\log(6n^{2}/\delta)}}\bigg{]},i\in\mathcal{C}_{1}.$ $$

###### Proof of Lemma 34 .

Note that Condition [11](#Thmcondition11) does not have any constraint for samples with $i\in[n]\backslash\mathcal{C}_{1}$. Thus we have $\breve{\theta}_{i}=0$ for any $i\in[n]\backslash\mathcal{C}_{1}$ in the representation ([A.33](#A1.E33)). Denote $j=\mathop{\mathrm{argmax}}\limits_{i\in\mathcal{C}_{1}}\breve{\theta_{i}}$, then we have

$$ $\displaystyle\breve{\theta_{j}}\cdot\|\bm{\xi}_{j}\|^{2}+\sum\limits_{k\neq j}y_{k}y_{j}\breve{\theta}_{k}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\geq\breve{\theta}_{j}\|\bm{\xi}_{j}\|^{2}-2\breve{\theta}_{j}n_{11}\sqrt{d\log(6n^{2}/\delta)}\geq\breve{\theta}_{j}((1-\kappa)d-2n_{11}\sqrt{d\log(6n^{2}/\delta)}).$ $$

The two inequalities are from Lemma [57](#Thmtheorem57) and our definition of $j$. Consider the contrary case when $\breve{\theta_{j}}>\frac{\alpha}{((1-\kappa)d-2n_{11}\sqrt{d\log(6n^{2}/\delta)}}$, we have

$$ $\displaystyle y_{j}\breve{\bm{v}}^{\top}\bm{\xi}_{j}$ $\displaystyle>\alpha.$ $$

By the complementary slackness condition, if $y_{j}\breve{\bm{v}}^{\top}\bm{\xi}_{j}>\alpha$, then we must have $\breve{\theta_{j}}=0$, and thus we reach a contradiction.
Then we lower bound $\breve{\theta_{i}}$. For $\forall i\in\mathcal{C}_{1}$ we have

$$ $\displaystyle\alpha$ $\displaystyle\leq\breve{\theta_{i}}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{j\neq i}y_{i}y_{j}\breve{\theta_{i}}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle\leq\breve{\theta_{i}}(1+\kappa)d+2n_{11}\max\limits_{i\in[n]}\breve{\theta_{i}}\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq\breve{\theta_{i}}(1+\kappa)d+\frac{2\alpha n_{11}\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{11}\sqrt{d\log(6n^{2}/\delta)}}.$ $$

The second inequality is from Lemma [57](#Thmtheorem57) and the last inequality is from the upper bound of $\breve{\theta_{i}}$ we just derived. Therefore, we have

$$ $\displaystyle\breve{\theta_{i}}\geq\frac{\alpha}{(1+\kappa)d}\bigg{(}1-\frac{2n_{11}\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{11}\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}.$ $$

∎

From this Lemma we have $\breve{\theta_{i}}=\Theta(\alpha/d)$ for $i\in\mathcal{C}_{1}$. Similar as ([A.42](#A1.E42)), under our assumption $\|\breve{\bm{v}}\|\leq 2\|\bm{v}_{mm}\|$, we have $\alpha=O(\log(n))$.
Next we estimate the difference between $\|\breve{\bm{v}}\|^{2}$ and $\|\bm{v}_{mm}\|^{2}$. We can prove that Lemma [33](#Thmtheorem33) still holds in this case.

###### Proof of Lemma 33 .

Under this case, the difference between $\|\breve{\bm{v}}\|_{2}^{2}$ and $\|\bm{v}_{mm}\|_{2}^{2}$ becomes

$$ $\displaystyle\|\breve{\bm{v}}\|^{2}-\|\bm{v}_{mm}\|^{2}\geq$ $\displaystyle\underbrace{\sum_{i\in[n]}(\breve{\theta}^{2}_{i}-\theta^{2}_{i})\|\bm{\xi}_{i}\|^{2}-(\lambda^{2}_{1}-\breve{\lambda}^{2}_{1})\|\bm{\mu}_{1}\|^{2}-(\lambda^{2}_{2}-\breve{\lambda}^{2}_{2})\|\bm{\mu}_{2}\|^{2}}_{I_{1}}$ $\displaystyle-\underbrace{\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{N}\backslash\{i\}}y_{i}y_{j}\theta_{i}\theta_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{2}}+\underbrace{\sum\limits_{i\in\mathcal{C}_{1}}\sum\limits_{j\in\mathcal{C}_{1}\backslash\{i\}}y_{i}y_{j}\breve{\theta}_{i}\breve{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{3}}$ $$

We then bound $I_{1}\sim I_{3}$ respectively. For $I_{1}$ we have

$$ $\displaystyle|I_{1}|$ $\displaystyle\geq\sum_{i\in\mathcal{C}_{1}}\breve{\theta}^{2}_{i}\|\bm{\xi}_{i}\|^{2}-\sum_{i\in\mathcal{N}}\theta^{2}_{i}\|\bm{\xi}_{i}\|^{2}-2/\rho^{2}\geq n_{11}\min\limits_{i\in[n]}\breve{\theta_{i}}^{2}(1-\kappa)d-n_{2}\max\limits_{i\in\mathcal{N}}\theta_{i}^{2}(1+\kappa)d-2/\rho^{2}$ $\displaystyle\geq\frac{\alpha^{2}n_{11}(1-\kappa)}{(1+\kappa)^{2}d}\bigg{(}1-\frac{2\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n_{11}\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}-\frac{n_{2}(1+\kappa)d}{((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)})^{2}}-\frac{2}{\rho^{2}}$ $\displaystyle=\Omega\bigg{(}\frac{n}{d}\bigg{)}.$ $$

The second inequality is from Lemma [57](#Thmtheorem57); The third inequality is from Lemma [27](#Thmtheorem27) and [34](#Thmtheorem34); The last equality is due to the SNR condition $\rho/\sqrt{d}=\Omega(1/\sqrt{n})$ so that $\frac{1}{\rho^{2}}\leq\frac{n}{4d}$. For $I_{2}$, we have

$$ $\displaystyle|I_{2}|$ $\displaystyle\leq\sum_{i\in\mathcal{N}}\max\limits_{i\in\mathcal{N}}\theta^{2}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}\leq\frac{2n_{2}\sqrt{d\log(6n^{2}/\delta)}}{((1-\kappa)d-2n_{2}\sqrt{d\log(6n^{2}/\delta)})^{2}}=\widetilde{O}\bigg{(}\frac{n}{d^{3/2}}\bigg{)}.$ $$

The first inequality is from Lemma [57](#Thmtheorem57); The second inequality is from Lemma [27](#Thmtheorem27). Similarly, for $|I_{3}|$ we have

$$ $\displaystyle|I_{3}|$ $\displaystyle\leq\sum_{i\in\mathcal{C}_{1}}\max\limits_{i\in\mathcal{C}_{1}}\breve{\theta}^{2}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}\leq\frac{2n_{11}\alpha^{2}\sqrt{d\log(6n^{2}/\delta)}}{((1-\kappa)d-2n_{11}\sqrt{d\log(6n^{2}/\delta)})^{2}}=\widetilde{O}\bigg{(}\frac{n}{d^{3/2}}\bigg{)}.$ $$

The second inequality is from Lemma [34](#Thmtheorem34).
Combining the above results, we have

$$ $\displaystyle\|\bm{v}^{\prime\prime}\|_{2}^{2}-\|\bm{v}\|_{2}^{2}\geq\Theta\bigg{(}\frac{n_{11}}{d}\bigg{)}-\widetilde{O}\bigg{(}\frac{n}{d^{3/2}}\bigg{)}\geq\frac{C_{3}n(1-\beta)}{d}.$ $$

The remaining proof is the same as Case 2.1 and we omit it for convenience.
∎

Therefore, we complete the proof for all possible scenarios.
∎

###### Condition 1 (Optimal tokens) .

###### Lemma 27 (balanced noise factor for KKT points) .

###### Proof of Lemma 27 .

###### Lemma 28 (Norm of 𝒗 m ​ m subscript 𝒗 𝑚 𝑚 \bm{v}_{mm} ) .

###### Proof of Lemma 28 .

###### Proof of Proposition 15 .

###### Condition 2 ( p 𝑝 p clean samples violating optimal token selection) .

###### Condition 3 (one clean sample violating optimal token selection) .

###### Lemma 29 .

###### Proof of Lemma 29 .

###### Lemma 30 .

###### Proof of Lemma 30 .

###### Proof of Proposition 15 in Case 1.1.

###### Condition 4 (All clean samples violate optimal token selection rule) .

###### Proof of Lemma 30 .

###### Condition 5 (Relaxed version of Condition 4 ) .

###### Lemma 31 .

###### Proof of Lemma 31 .

###### Condition 6 (One cluster and a clean sample in the opposite cluster violating optimal token selection) .

###### Condition 7 (Relaxed version of Condition 6 ) .

###### Condition 8 ( k − p 𝑘 𝑝 k-p noisy samples violating optimal token selection rule) .

###### Condition 9 (1 noisy sample violating optimal token selection rule) .

###### Lemma 32 .

###### Proof of Lemma 31 .

###### Lemma 33 .

###### Proof of Lemma 33 .

###### Proof of Proposition 15 under Case 2.1.

###### Condition 10 (All samples are mixed) .

###### Condition 11 (Relaxed version of constraints in Condition 10 ) .

###### Lemma 34 .

###### Proof of Lemma 34 .

###### Proof of Lemma 33 .

##### Training and Test Error Analysis

From Proposition [15](#Thmtheorem15) we can analyze the properties of both parameters to estimate the training and test error.

In this section, we first get the convergence direction of parameters $\bm{p}$ and $\bm{v}$. The main difference between our setting with [] is that they only consider the infinite case and their results hold only when $R,r\rightarrow\infty$. We extend their results to the finite case.
Specifically, given fixed upper bound $R$ and $r$ for $\|\bm{p}\|$ and $\|\bm{v}\|$ respectively, we denote the solution of the constrained optimization ([4.1](#S4.E1)) as $(\bm{v}_{r},\bm{p}_{R})$ in this section for brevity.

Our main theorem in this section estimates the corresponding deviation of $\bm{p}_{R}/R$ and $\bm{v}_{r}/r$ from their convergence direction $\bm{p}_{mm}/\|\bm{p}_{mm}\|$ and $\bm{v}_{mm}/\|\bm{v}_{mm}\|$. For a given $\bm{p}$, it is elementary that the margin induced by $\bm{p}$ is $\min_{i,t_{i}\neq\alpha_{i}}(\bm{x}_{i\alpha_{i}}-\bm{x}_{it_{i}})^{\top}\bm{p}/\|\bm{p}\|$, thus when $\|\bm{p}\|=1$, the margin becomes $\min_{i,t_{i}\neq\alpha_{i}}(\bm{x}_{i\alpha_{i}}-\bm{x}_{it_{i}})^{\top}\bm{p}$. And for a given $\bm{v}$, the label margin induced by $\bm{v}$ is $\min_{i}y_{i}\bm{v}^{\top}\bm{r}_{i}/\|\bm{v}\|$. Recall that the label margin induced by $\bm{v}_{mm}$ is $\Gamma$ and the margin of p-SVM induced by $\bm{p}_{mm}$ is $\Xi$.

First we introduce a lemma to estimate the norm of $\|\bm{p}_{mm}\|$. This will benefit our proof of the main theorem.

###### Lemma 35 (Norm of $\bm{p}_{mm}$ ) .

Suppose that Assumption [5](#Thmtheorem5) holds, recall that the solution of (p-SVM) is $\bm{p}_{mm}$. With probability at least $1-\delta$ on the training dataset we have

$$ $\displaystyle\frac{1}{\rho^{2}}+\frac{\eta n}{d}\leq\|\bm{p}_{mm}\|^{2}\leq\frac{8}{\rho^{2}}+\frac{17\eta n}{d}.$ $$

This implies

$$ $\|\bm{p}_{mm}\|=\Theta\bigg{(}\sqrt{\frac{1}{\rho^{2}}+\frac{\eta n}{d}}\bigg{)}.$ $$

###### Proof of Lemma 35 .

First we prove the upper bound. Consider the following possible solution $\widetilde{\bm{p}}$:

$$ $\widetilde{\bm{p}}=\frac{2(\bm{\mu}_{1}+\bm{\mu}_{2})}{\rho^{2}}+\sum\limits_{i\in\mathcal{N}}4\frac{\bm{\xi}_{i}}{d}.$ (A.45) $$

We then proved that $\widetilde{\bm{p}}$ satisfies ([A.30](#A1.E30)). For $k\in\mathcal{C}$ we have

$$ $\displaystyle\widetilde{\bm{p}}^{\top}(\bm{\mu}_{k}-\bm{\xi}_{k})$ $\displaystyle=2-\sum\limits_{i\in\mathcal{N}}4\frac{\langle\bm{\xi}_{i},\bm{\xi}_{k}\rangle}{d}\geq 2-\frac{4n_{2}\sqrt{d\log(6n^{2}/\delta)}}{d}\geq 1.$ $$

The first inequality is from the definition of $d$ in Lemma [57](#Thmtheorem57) and the second inequality is from Assumption [5](#Thmtheorem5).
And for $k\in\mathcal{N}$, we have

$$ $\displaystyle\widetilde{\bm{p}}^{\top}(\bm{\xi}_{k}-\bm{\mu}_{k})$ $\displaystyle=-2+\sum\limits_{i\in\mathcal{N}}4\frac{\langle\bm{\xi}_{i},\bm{\xi}_{k}\rangle}{d}\geq-2+4(1-\kappa)+\sum\limits_{i\in\mathcal{N},i\neq k}4\frac{\langle\bm{\xi}_{i},\bm{\xi}_{k}\rangle}{d}$ $\displaystyle\geq-2+4(1-\kappa)+\frac{4n_{2}\sqrt{d\log(6n^{2}/\delta)}}{d}\geq 1.$ $$

The first and second inequalities are from Lemma [57](#Thmtheorem57); The last inequality is from Assumption [5](#Thmtheorem5).

Therefore, the max-margin solution $\bm{p}_{mm}$ must have no greater norm than $\widetilde{\bm{p}}$. So we can upper bound $\bm{p}_{mm}$ as

$$ $\displaystyle\|\bm{p}_{mm}\|^{2}$ $\displaystyle\leq\|\widetilde{\bm{p}}\|^{2}=\frac{8}{\rho^{2}}+\frac{16}{d^{2}}\Big{(}\sum\limits_{i\in\mathcal{N}}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i,j\in\mathcal{N},i\neq j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle\Big{)}$ $\displaystyle\leq\frac{8}{\rho^{2}}+\frac{16}{d^{2}}\big{(}(1+\kappa)n_{2}d+2n_{2}^{2}\sqrt{d\log(6n^{2}/\delta)}\big{)}\leq\frac{8}{\rho^{2}}+\frac{17\eta n}{d}.$ $$

The second inequality is from Lemma [57](#Thmtheorem57); The last inequality is from the definition of $d$ in Assumption [5](#Thmtheorem5).

Then we prove for the lower bound. As $\bm{p}_{mm}$ is the max-margin solution and satisfies KKT condition, it can be expressed as the sum of signal and noise tokens. Then we decompose $\bm{p}_{mm}=\bm{p}^{mm}_{\bm{\mu}}+\bm{p}^{mm}_{\bm{\xi}}$ where $\bm{p}^{mm}_{\bm{\mu}}=f^{mm}_{1}\bm{\mu}_{1}+f^{mm}_{2}\bm{\mu}_{2}$ and $\bm{p}^{mm}_{\bm{\xi}}=\sum_{i\in[n]}g^{mm}_{i}\bm{\xi}_{i}$. Note that $\bm{\mu}_{j}\perp\bm{\xi}_{i}$ for all $j\in\{\pm 1\},i\in[n]$. From Lemma [39](#Thmtheorem39), we have $f^{mm}_{j}\geq 0.9/\rho^{2}$, so we can lower bound $\|\bm{p}^{mm}_{\bm{\mu}}\|_{2}^{2}$ as

$$ $\displaystyle\|\bm{p}^{mm}_{\bm{\mu}}\|_{2}^{2}=f^{mm2}_{1}\|\bm{\mu}_{1}\|^{2}+f^{mm2}_{2}\|\bm{\mu}_{2}\|^{2}\geq\frac{2\cdot 0.9^{2}}{\rho^{2}}\geq\frac{1}{\rho^{2}}.$ $$

As for $\|\bm{p}^{mm}_{\bm{\xi}}\|_{2}$, from p-SVM condition, for every noisy sample we have

$$ $\displaystyle\bm{p}_{mm}^{\top}(\bm{\xi}_{i}-\bm{\mu}_{i})\geq 1,$ $$

which indicates

$$ $\displaystyle\bm{p}^{mm\top}_{\bm{\xi}}\bm{\xi}_{i}=\bm{p}_{mm}^{\top}\bm{\xi}_{i}\geq 1+\bm{p}_{mm}^{\top}\bm{\mu}_{i}\geq 1.9.$ $$

The last inequality is from Lemma [39](#Thmtheorem39). Sum up the inequality for all noisy sample, we have

$$ $\displaystyle\sum\limits_{i\in\mathcal{N}}\bm{p}^{mm\top}_{\bm{\xi}}\bm{\xi}_{i}\geq 1.9n_{2}.$ $$

Thus,

$$ $\displaystyle\|\bm{p}^{mm}_{\bm{\xi}}\|$ $\displaystyle\geq\frac{1.9n_{2}}{\|\sum\limits_{i\in\mathcal{N}}\bm{\xi}_{i}\|}=\frac{1.9n_{2}}{\sqrt{\sum\limits_{i\in\mathcal{N}}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i,j\in\mathcal{N}}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}}\geq\frac{1.9n_{2}}{\sqrt{2\cdot n_{2}\cdot(1+\kappa)d}}\geq\sqrt{\frac{\eta n}{d}}.$ $$

The second inequality is from Lemma [57](#Thmtheorem57) and the last inequality is from Assumption [5](#Thmtheorem5).
Therefore,

$$ $\displaystyle\|\bm{p}_{mm}\|^{2}=|\bm{p}^{mm}_{\bm{\mu}}\|_{2}^{2}+\|\bm{p}^{mm}_{\bm{\xi}}\|_{2}^{2}\geq\frac{1}{\rho^{2}}+\frac{\eta n}{d}.$ $$

Combining the results above, we have

$$ $\displaystyle\|\bm{p}_{mm}\|^{2}=\Theta\bigg{(}\frac{1}{\rho^{2}}+\frac{\eta n}{d}\bigg{)}.$ $$

∎

###### Definition 36 .

Let $f:\mathbb{R}^{2}\rightarrow\mathbb{R}^{d}$. We say that

$$ $\lim_{x,y\rightarrow\infty}f(x,y)=L$ $$

iff
$\forall\epsilon>0\ \exists M$ such that $\forall x,y>M$ we have that $\left\|f(x,y)-L\right\|<\epsilon$.

###### Remark 37 .

Let $g:\mathbb{R}\rightarrow\mathbb{R}$ be a function with $\lim_{x\rightarrow\infty}g(x)=\infty$. Assume that $\lim_{x,y\rightarrow\infty}f(x,y)=L$, then $\lim_{x\rightarrow\infty}f(x,g(x))=L$ and $\lim_{x\rightarrow\infty}f(g(x),x)=L$

Now we introduce our key theorem:

See [16](#Thmtheorem16)

###### Proof of Theorem 16 .

From Proposition [15](#Thmtheorem15), we have that for any $\|\bm{p}\|$, the label margin $1/\|\bm{v}(\bm{p})\|$ is at most

$$ $\Gamma-\frac{C\max_{i\in[n]}(1-s_{i\alpha_{i}})}{\|\bm{v}_{mm}\|^{3}n\rho^{2}},$ $$

where $\alpha_{i}=1$ for $i\in\mathcal{C}$ and $\alpha_{i}=2$ for $i\in\mathcal{N}$. Recall that $\bm{s}_{i}=\mathbb{S}(\bm{X}_{i}{\bm{p}})$ is the softmax probability vector.
We define $q_{i}^{\bm{p}}=1-s_{i\alpha_{i}}$ to measure the amount of non-optimality (attention on non-optimal token).

We first consider the convergence of $\bm{p}_{R}$ and use contradiction to prove the first statement. Denote $\bm{p}^{mm}_{R}=R\bm{p}_{mm}/\|\bm{p}_{mm}\|$ which has the same norm as $\bm{p}_{R}$ and the direction of $\bm{p}_{mm}$. Suppose the margin induced by $\bm{p}_{R}/R$ is at most $(1-\zeta)\Xi$, i.e. $\min_{i,t_{i}\neq\alpha_{i}}(\bm{x}_{i\alpha_{i}}-\bm{x}_{it_{i}})^{\top}\bm{p}_{R}\leq(1-\zeta)R\Xi,\forall i\in[n]$. Note that here each sequence only has two tokens, thus $t_{i},\alpha_{i}\in[2],$ and $t_{i}=3-\alpha_{i}$.

According to Lemma [35](#Thmtheorem35), we have

$$ $\Xi=\|\bm{p}_{mm}\|_{2}^{-1}=\Theta((\eta n/d+1/\rho^{2})^{-1/2}).$ $$

Following the definition of $q_{i}^{\bm{p}}$ above, we set $\widehat{q}_{max}=\sup_{i\in[n]}q_{i}^{\bm{p}_{R}}$ and $q^{*}_{max}=\sup_{i\in[n]}q_{i}^{\bm{p}^{mm}_{R}}$ to be the worst non-optimality in $\bm{p}_{R}$ and $\bm{p}^{mm}_{R}$. Then we have

$$ $\displaystyle q_{i}^{\bm{p}^{mm}_{R}}$ $\displaystyle=\frac{\exp(\bm{x}_{it_{i}}^{\top}\bm{p}^{mm}_{R})}{\sum_{t\in[2]}\exp(\bm{x}_{it}^{\top}\bm{p}^{mm}_{R})}\leq\frac{\exp(\bm{x}_{it_{i}}^{\top}\bm{p}^{mm}_{R})}{\exp(\bm{x}_{i\alpha_{i}}^{\top}\bm{p}^{mm}_{R})}\leq\exp(-R\Xi).$ $$

The last inequality is from the definition of $\bm{p}_{mm}$ that $\bm{p}_{mm}^{\top}(\bm{x}_{i\alpha_{i}}-\bm{x}_{it})\geq 1$, so $\bm{p}^{mm\top}_{R}(\bm{x}_{i\alpha_{i}}-\bm{x}_{it})\geq R/\|\bm{p}_{mm}\|=R\Xi$.
Thus, $q^{*}_{max}=\sup_{i\in[n]}q_{i}^{\bm{p}_{mm}}\leq\exp(-R\Xi)$.
Then denote the output of attention layer $\bm{r}_{i}=\bm{X}_{i}^{\top}\mathbb{S}(\bm{X}_{i}\bm{p}^{mm}_{R})$. Define $\epsilon_{i}=\|\bm{r}_{i}-\bm{x}_{i\alpha_{i}}\|$, we have $y_{i}\cdot\bm{r}_{i}^{\top}\bm{v}_{mm}\geq y_{i}\cdot\bm{x}_{i\alpha_{i}}^{\top}\bm{v}_{mm}-\|\bm{r}_{i}-\bm{x}_{i\alpha_{i}}\|\cdot\|\bm{v}_{mm}\|\geq 1-\epsilon_{i}/\Gamma$. So if we set $\epsilon_{max}=\sup_{i\in[n]}\epsilon_{i}$, $\bm{v}_{mm}$ achieves a label margin of at least $\Gamma-\epsilon_{max}$ on $(y_{i},\bm{r}_{i})_{i\in[n]}$. To better estimate $\epsilon_{max}$, we define $M=\sup_{i\in[n]}\|\bm{\mu}_{i}-\bm{\xi}_{i}\|\leq\sqrt{\rho^{2}+(1+\kappa)d}$, then we have

$$ $\displaystyle\epsilon_{max}=M\cdot q^{*}_{max}\leq M\exp(-R\Xi).$ (A.46) $$

This implies the max-margin achieved by $(\bm{p}_{R}^{mm},\bm{v}_{r}^{mm})$ is at least

$$ $\displaystyle y_{i}f(\bm{p}_{R}^{mm},\bm{v}_{r}^{mm};\bm{x}_{i})=y_{i}\bm{v}^{mm\top}_{r}\bm{r}_{i}\geq r\Gamma-r\epsilon_{max}\geq r\Gamma-rM\exp(-R\Xi).$ (A.47) $$

The first inequality is from $y_{i}\cdot\bm{r}_{i}^{\top}\bm{v}^{mm}_{r}\geq r(\Gamma-\epsilon_{i})$ and the last inequality is from ([A.46](#A1.E46)).

Then we consider the case when $\min_{i,t_{i}\neq\alpha_{i}}(x_{i\alpha_{i}}-x_{it_{i}})^{\top}\bm{p}_{R}\leq(1-\zeta)R\Xi$ the minimal margin constraint is $\zeta$-violated by $\bm{p}_{R}$. Without losing generality we assume that $1=\mathop{\mathrm{argmin}}\limits_{i\in[n]}[(\bm{x}_{i\alpha_{i}}-\bm{x}_{it})^{\top}\bm{p}_{R}]_{t\neq\alpha_{i}}$. Then we have

$$ $\displaystyle\widehat{q}_{max}$ $\displaystyle\geq\frac{\exp(\bm{x}_{1t_{1}}^{\top}\bm{p}_{R})}{\sum_{t\in[2]}\exp(\bm{x}_{1t}^{\top}\bm{p}_{R})}\geq\frac{1}{2}\frac{\exp(\bm{x}_{1t_{1}}^{\top}\bm{p}_{R})}{\exp(\bm{x}_{1\alpha_{1}}^{\top}\bm{p}_{R})}\geq\frac{1}{2\exp((1-\zeta)R\Xi)}.$ $$

From Proposition [15](#Thmtheorem15), optimizing v-SVM on $(y_{i},\widehat{\bm{r}}_{i})_{i\in[n]}$ can achieve the max-margin at most

$$ $\displaystyle\min\limits_{i\in[n]}y_{i}f(\bm{p}_{R},\bm{v}_{r};\bm{x}_{i})\leq\Gamma-\frac{C}{2\|\bm{v}_{mm}\|^{3}n\rho^{2}}\cdot e^{-(1-\zeta)R\Xi}.$ (A.48) $$

And from the definition $\zeta=\frac{1}{R\Xi}\log(2M\|\bm{v}_{mm}\|^{3}n\rho^{2}/C)$, we have

$$ $\displaystyle\frac{C}{2\|\bm{v}_{mm}\|^{3}n\rho^{2}}\exp(-(1-\zeta)R\Xi)>M\exp(-R\Xi)$ $$

for sufficiently large $R$, which implies

$$ $\min\limits_{i\in[n]}y_{i}\cdot f(\bm{p}_{R},\bm{v}_{r};\bm{x}_{i})<\min\limits_{i\in[n]}y_{i}\cdot f(\bm{p}^{mm}_{R},\bm{v}^{mm}_{r};\bm{x}_{i}).$ $$

This contradicts with the problem definition ([4.1](#S4.E1)) to maximize the margin.

Then we prove for the second statement. When the margin induced by $\bm{p}_{R}/R$ in p-SVM is less than $(1-\zeta)\Xi$, we can use the proof above to derive a contradiction, so $(\bm{x}_{i\alpha_{1}}-\bm{x}_{it})^{\top}\bm{p}_{R}\geq(1-\zeta)R\Xi$ must hold. Then set $\widehat{\bm{r}}_{i}=\bm{X}_{i}^{\top}\mathcal{S}(\bm{X}_{i}\bm{p}_{R})$, we have that

$$ $\displaystyle\min_{i\in[n]}y_{i}\bm{v}_{r}^{\top}\widehat{\bm{r}}_{i}$ $\displaystyle\leq\min_{i\in[n]}y_{i}\bm{v}_{r}^{\top}\bm{x}_{i\alpha_{i}}+\sup_{i\in[n]}|\bm{v}_{r}^{\top}(\widehat{\bm{r}}_{i}-\bm{x}_{i\alpha_{i}})|$ $\displaystyle\leq(1-\gamma)\Gamma r+M\exp(-(1-\zeta)R\Xi)r$ $\displaystyle\leq(1-\gamma/2)\Gamma r.$ $$

The second inequality is from previous analysis that $(\bm{x}_{i\alpha_{i}}-\bm{x}_{it})^{\top}\bm{p}_{R}\geq(1-\zeta)R\Xi$, so $|\widehat{\bm{r}}_{i}-\bm{x}_{i1}|\leq M\exp(-(1-\zeta)R\Xi)$; The last inequality is from our definition $\gamma=\frac{2M}{\Gamma\exp((1-\zeta)R\Xi)}$.

Therefore, combining with ([A.47](#A1.E47)), we have

$$ $\displaystyle\gamma\Gamma r/2>rM\exp(-R\Xi),$ $$

which implies

$$ $\displaystyle\min\limits_{i\in[n]}y_{i}\cdot f(\bm{p}_{R},\bm{v}_{r};\bm{x}_{i})<\min\limits_{i\in[n]}y_{i}\cdot f(\bm{p}^{mm}_{R},\bm{v}^{mm}_{r};\bm{x}_{i}).$ $$

Again this contradicts with the problem definition ([4.1](#S4.E1)).
∎

Then we have the following lemma to bound the derivation $\zeta$ and $\gamma$:

###### Lemma 38 .

Suppose that Assumption [5](#Thmtheorem5) holds, consider the same setting in Theorem [16](#Thmtheorem16), we have $\zeta<0.2$ and $\gamma<1$.

###### Proof of Lemma 38 .

From the definition of $\zeta$ in Theorem [16](#Thmtheorem16), we have

$$ $\displaystyle\zeta$ $\displaystyle=\frac{\log(2M\|\bm{v}_{mm}\|^{3}n\rho^{2}/C)}{R\Xi}=C_{1}\frac{1}{R\sqrt{\eta n/d+1/\rho^{2}}}\log(M\|\bm{v}_{mm}\|^{3}n\rho^{2})$ $\displaystyle\leq C_{2}\frac{1}{R\sqrt{\eta n/d+1/\rho^{2}}}\log\bigg{(}\frac{n^{2}(\rho^{2}+d)(\rho^{2}\eta n+d)^{3}}{\rho^{2}d^{3}}\bigg{)}=\frac{C_{3}}{R\sqrt{\eta n/d+1/\rho^{2}}}\log(\rho n)<0.2.$ $$

Here $C_{1},C_{2},C_{3}=\Theta(1)$. The first inequality is from the upper bound of $\|\bm{v}_{mm}\|$ in Lemma [28](#Thmtheorem28) and the last inequality is from the definition of $R$ in Assumption [5](#Thmtheorem5).
And for $\gamma$, we have

$$ $\displaystyle\gamma$ $\displaystyle=\frac{2M}{\Gamma\exp((1-\zeta)R\Xi)}=C^{\prime}_{1}\frac{M\|\bm{v}_{mm}\|}{\exp(R/\|\bm{v}_{mm}\|)}\leq C^{\prime}_{2}\frac{\sqrt{(\rho^{2}+d)(\eta n/d+1/\rho^{2})}}{\exp(R/\sqrt{\eta n/d+1/\rho^{2}})}<1.$ $$

Here $C^{\prime}_{1},C^{\prime}_{2}=\Theta(1)$. The first inequality is from the lower and upper bound of $\|\bm{v}_{mm}\|$ in Lemma [28](#Thmtheorem28) and the last inequality is from the definition of $R$ in Assumption [5](#Thmtheorem5).
∎

Then we can estimate $\langle\bm{p}_{R},\bm{\mu}\rangle$ with the following lemma:

###### Lemma 39 .

Suppose that Assumption [5](#Thmtheorem5) holds, with probability at least $1-\delta$ on the training dataset, $\bm{p}_{R}$ should satisfy

$$ $\displaystyle 0.5(1-\zeta)R\Xi\leq\langle\bm{p}_{R},\bm{\mu}_{j}\rangle\leq R\rho$ $$

for $j\in\{1,2\}$.

###### Proof of Lemma 39 .

The upper bound is given by

$$ $\langle\bm{p}_{R},\bm{\mu}_{j}\rangle\leq\|\bm{p}_{R}\|\|\bm{\mu}_{j}\|=R\rho.$ $$

Then we use contradiction to prove for the lower bound. From Theorem [16](#Thmtheorem16), $\bm{p}_{R}$ satisfies

$$ $\displaystyle\bm{p}_{R}^{\top}(\bm{\mu}_{i}-\bm{\xi}_{i})$ $\displaystyle\geq(1-\zeta)R\Xi,i\in\mathcal{C}$ $\displaystyle\bm{p}_{R}^{\top}(\bm{\xi}_{i}-\bm{\mu}_{i})$ $\displaystyle\geq(1-\zeta)R\Xi,i\in\mathcal{N}$ (A.49) $$

If $\langle\bm{p}_{R},\bm{\mu}_{j}\rangle\leq 0.5(1-\zeta)R\Xi$, then for every clean sample from cluster j we must have $\langle\bm{p}_{R},\bm{\xi}_{i}\rangle\leq-0.5(1-\zeta)R\Xi$ and thus

$$ $\displaystyle\langle\bm{p}_{R},\sum\limits_{i\in\mathcal{C}_{j}}\bm{\xi}_{i}\rangle=\sum\limits_{i\in\mathcal{C}_{j}}\langle\bm{p}_{R},\bm{\xi}_{i}\rangle\leq-0.5(1-\zeta)R\Xi n_{1j}.$ $$

So we could estimate $\|\bm{p}_{R}\|$ as follows

$$ $\displaystyle\|\bm{p}_{R}\|$ $\displaystyle\geq 0.5(1-\zeta)R\Xi\cdot n_{1j}\frac{1}{\|\sum\limits_{i\in\mathcal{C}_{j}}\bm{\xi}_{i}\|}=0.5(1-\zeta)R\Xi\cdot n_{1j}\frac{1}{\sqrt{\sum\limits_{i\in\mathcal{C}_{j}}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i,j\in\mathcal{C}_{j}}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}}$ $\displaystyle\geq 0.5(1-\zeta)R\Xi\cdot n_{1j}\frac{1}{\sqrt{2\cdot n_{1j}\cdot(1+\kappa)d}}\geq 0.4R\Xi\cdot\frac{\sqrt{n_{1j}}}{\sqrt{2(1+\kappa)d}}.$ $$

The first inequality is from the property of innerproduct; The second inequality is from Lemma [57](#Thmtheorem57) and the definition of d in Assumption [5](#Thmtheorem5); The last inequality is from Lemma [38](#Thmtheorem38).
Meanwhile, from Lemma [35](#Thmtheorem35) we have $\|\bm{p}_{mm}\|\leq\sqrt{8/\rho^{2}+17\eta n/d}$. Recall that $\Xi=\|\bm{p}_{mm}\|^{-1}$. Therefore, we further have

$$ $\displaystyle\|\bm{p}_{R}\|$ $\displaystyle\geq 0.4R\Xi\cdot\frac{\sqrt{n_{1j}}}{\sqrt{2(1+\kappa)d}}\geq\sqrt{\frac{0.4^{2}n_{1j}}{(8/\rho^{2}+17\eta n/d)\cdot 2(1+\kappa)d}}\cdot R$ $\displaystyle\geq\sqrt{\frac{0.04(n-\eta n-O(\sqrt{n}))}{(8/\rho^{2}+17\eta n/d)\cdot(1+\kappa)d}}\cdot R>R.$ $$

The second inequality is from Lemma [35](#Thmtheorem35); The third inequality is from Lemma [59](#Thmtheorem59) and the last inequality is from Assumption [5](#Thmtheorem5) about SNR and $\eta$. This leads to a contradiction.

∎

Now we can estimate the output of attention layer for some test sample $(\bm{X},y)$.

###### Lemma 40 .

Suppose that Assumption [5](#Thmtheorem5) holds, with probability at least $1-\delta$ on the training dataset, for a given a test sample $\bm{X},y$, where $\bm{X}=(\bm{\mu}^{\star},\bm{\xi}^{\star})$, $\bm{\mu}^{\star}$ can be $\bm{\mu}_{1}$ or $\bm{\mu}_{2}$, we have with probability at least $1-\exp\big{(}-\frac{1}{2}(\frac{1}{2}(1-\zeta)\Xi-K/R)^{2}\big{)}$ that

$$ $\displaystyle\langle\bm{p}_{R},\bm{\mu}^{\star}\rangle-\langle\bm{p}_{R},\bm{\xi}^{\star}\rangle\geq K,$ $$

where $K\leq\frac{1}{2}(1-\zeta)R\Xi$ and $\zeta,\Xi$ are defined in Theorem [16](#Thmtheorem16).

###### Proof of Lemma 40 .

Note that $\bm{p}^{\top}\bm{\xi}^{\star}$ follows Gaussian distribution $\mathcal{N}(0,R^{2})$, we have

$$ $\displaystyle\mathbb{P}(\langle\bm{p}_{R},\bm{\mu}^{\star}\rangle-\langle\bm{p}_{R},\bm{\xi}^{\star}\rangle<K)$ $\displaystyle=\mathbb{P}(\langle\bm{p}_{R},\bm{\xi}^{\star}\rangle>\langle\bm{p}_{R},\bm{\mu}^{\star}\rangle-K)\leq\mathbb{P}(\bm{p}_{R}^{\top}\bm{\xi}^{\star}>\frac{1}{2}(1-\zeta)R\Xi-K)$ $\displaystyle\leq\exp\Big{(}-\frac{1}{2}(\frac{1}{2}(1-\zeta)\Xi-K/R)^{2}\Big{)}.$ $$

The first inequality is from Lemma [39](#Thmtheorem39) and the second inequality comes from the property of Gaussian tail probability.
∎

We also have the following lemma to estimate $\bm{v}_{r}$. We first prove that $\bm{v}_{r}$ can be expressed as the sum of signal and noise tokens.

###### Lemma 41 .

The solution of constrained optimization problem ([4.1](#S4.E1)) $\bm{v}_{r}$ can be expressed in the form that

$$ $\bm{v}_{r}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum_{i=1}^{n}\theta_{i}\bm{\xi}_{i}.$ $$

###### Proof of Lemma 41 .

Similar to Theorem [16](#Thmtheorem16), define $\widehat{\bm{r}}_{i}=\bm{X}_{i}^{\top}\mathcal{S}(\bm{X}_{i}\bm{p}_{R})$ as the output of attention layer, we have

$$ $\displaystyle\bm{v}_{r}=\mathop{\mathrm{argmax}}\limits_{\|\bm{v}\|\leq r}\min\limits_{i\in[n]}y_{i}\bm{v}^{\top}\bm{r}_{i}.$ (A.50) $$

Then denote $s=\min\limits_{i\in[n]}y_{i}\bm{v}^{\top}\bm{r}_{i}$ and $s_{r}=\min\limits_{i\in[n]}y_{i}\bm{v}^{\top}_{r}\bm{r}_{i}$. Then ([A.50](#A1.E50)) can be written as

$$ $\displaystyle(\bm{v}_{r},s_{r})=\mathop{\mathrm{argmax}}\limits_{\bm{v},s}s,\text{ s.t. }$ $\displaystyle y_{i}\bm{v}^{\top}\bm{r}_{i}\geq s,\quad 1\leq i\leq n$ $\displaystyle\|\bm{v}\|\leq r.$ $$

The corresponding Lagrangian function is

$$ $\displaystyle L(s,\psi)=-s+\sum_{i=1}^{n}\psi_{i}y_{i}(s-y_{i}\bm{v}^{\top}\bm{r}_{i})+\psi_{0}(\|\bm{v}\|^{2}-r^{2}).$ $$

Take derivative of this function on $(s,\bm{v})$, we have

$$ $\displaystyle-\sum_{i=1}^{n}\psi_{i}y_{i}\bm{r}_{i}+2\psi_{0}\bm{v}=0.$ $$

Therefore from the last equation we can get

$$ $\displaystyle\bm{v}=\frac{1}{2\psi_{0}}\sum_{i=1}^{n}\psi_{i}y_{i}\bm{r}_{i}.$ $$

As $\bm{r}_{i}=\beta_{i}\bm{\mu}_{i}+(1-\beta_{i})\bm{\xi}_{i}$ for every $i\in[n]$, $\bm{v}$ can be expressed as the combination of signal and noise token of every sample:

$$ $\bm{v}_{r}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum_{i=1}^{n}\theta_{i}\bm{\xi}_{i}.$ $$

∎

Based on this representation, we can then bound the parameters in $\bm{v}_{r}$:

###### Lemma 42 .

Suppose that Assumption [5](#Thmtheorem5) holds, denote $\bm{v}_{r}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum\limits_{i\in[n]}\theta_{i}\bm{\xi}_{i}$. Then with probability at least $1-\delta$ on the training dataset, we have

$$ $\displaystyle\lambda_{1}$ $\displaystyle\geq(1-\gamma)\Gamma r/\rho^{2},$ $\displaystyle\lambda_{2}$ $\displaystyle\leq-(1-\gamma)\Gamma r/\rho^{2},$ $\displaystyle|\theta_{i}|$ $\displaystyle\leq 2\sqrt{1/\rho^{2}+5\eta n/d}\cdot\Gamma r/\sqrt{d}.$ $$

###### Proof of Lemma 42 .

The first two statements are obvious because from Theorem [16](#Thmtheorem16) we have

$$ $\displaystyle y_{i}\bm{v}^{\top}_{r}\bm{\mu}_{i}\geq(1-\gamma)\Gamma r,$ $$

for $\forall i\in\mathcal{C}$. This implies $|\lambda_{j}|\geq(1-\gamma)\Gamma r/\rho^{2}$ for $j\in\{1,2\}$. Meanwhile, we decompose $\bm{v}_{r}=\bm{v}_{\bm{\mu}}+\bm{v}_{\bm{\xi}}$ where $\bm{v}_{\bm{\mu}}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}$ and $\bm{v}_{\bm{\xi}}=\sum\limits_{i\in[n]}\theta_{i}\bm{\xi}_{i}$. And we can upper bound $\|\bm{v}_{\bm{\xi}}\|$ as

$$ $\displaystyle\|\bm{v}_{\bm{\xi}}\|^{2}$ $\displaystyle=\|\bm{v}_{r}\|^{2}-\|\bm{v}_{\bm{\mu}}\|^{2}\leq r^{2}-\lambda_{1}^{2}\rho^{2}-\lambda_{2}^{2}\rho^{2}\leq r^{2}(1-2(1-\gamma)^{2}\Gamma^{2}/\rho^{2}).$ $$

The first inequality is from $\|\bm{v}\|\leq r$ and the second inequality is from the first two statements we just proved. Therefore, denote $j=\mathop{\mathrm{argmax}}\limits_{i\in[n]}\theta_{i}$, we have

$$ $\displaystyle\theta_{j}^{2}\|\bm{\xi}_{j}\|^{2}\leq\|\bm{v}_{\bm{\xi}}\|^{2}\leq r^{2}(1-2(1-\gamma)^{2}\Gamma^{2}/\rho^{2}).$ $$

Then we can upper bound $|\theta_{j}|$ as

$$ $\displaystyle\theta_{j}^{2}$ $\displaystyle\leq r^{2}(1-2(1-\gamma)^{2}\Gamma^{2}/\rho^{2})/\|\bm{\xi}_{j}\|^{2}\leq r^{2}(1-2(1-\gamma)^{2}\Gamma^{2}/\rho^{2})/(1-\kappa)d$ $\displaystyle=r^{2}\bigg{(}1-\frac{2(1-\gamma)^{2}}{\|\bm{v}_{mm}\|^{2}\rho^{2}}\bigg{)}/(1-\kappa)d\leq r^{2}\bigg{(}1-\frac{1}{(2/\rho^{2}+5\eta n/d)\rho^{2}}\bigg{)}/(1-\kappa)d$ $\displaystyle=\frac{1+5\eta n\rho^{2}/d}{2+5\eta n\rho^{2}/d}\cdot\frac{r^{2}}{(1-\kappa)d}\leq\bigg{(}\frac{1}{\rho^{2}}+\frac{5\eta n}{d}\bigg{)}\cdot\frac{\Gamma^{2}r^{2}}{2d}.$ $$

The second inequality is from Lemma [57](#Thmtheorem57); The third inequality is from Lemma [28](#Thmtheorem28) that $\|\bm{v}_{mm}\|\leq\sqrt{2/\rho^{2}+5\eta n/d}$ and our definition of $\gamma=\frac{2\sqrt{\rho^{2}+(1+\kappa)d}}{\Gamma\exp((1-\zeta)R\Xi)}$; The last inequality is from $\Gamma=\|\bm{v}_{mm}\|^{-1}\geq(2/\rho^{2}+5\eta n/d)^{-1}$. Thus, we can bound $|\theta_{j}|$ as

$$ $\displaystyle|\theta_{j}|\leq 2\sqrt{1/\rho^{2}+5\eta n/d}\cdot\Gamma r/\sqrt{d}.$ $$

∎

Therefore, we can prove the main theorem.

###### Proof of Theorem 6 .

First we show that the model can perfectly classify all training samples. From Theorem [16](#Thmtheorem16), we have

$$ $\displaystyle y_{i}\bm{v}^{\top}_{r}\bm{r}_{i}\geq(1-\gamma)\Gamma r>0$ $$

for $\forall i\in[n]$. The last inequality is from Lemma [38](#Thmtheorem38). Thus $y_{i}=\mathrm{sign}(f(\bm{X}_{i};\bm{p}_{R},\bm{v}_{r}))$ for all $i\in[n]$.
Then we bound the test error. Given a test sample $\bm{X},y$, where $\bm{X}=(\bm{\mu}^{\star},\bm{\xi}^{\star})$, $\bm{\mu}^{\star}$ can be $\bm{\mu}_{1}$ or $\bm{\mu}_{2}$. From Remark[58](#Thmtheorem58), with probability at least $1-6n\exp(-d/4C_{1}n^{2})$,

$$ $|\langle\bm{\xi}^{\star},\bm{\xi}_{i}\rangle|\leq\frac{d}{C_{1}n}.$ (A.51) $$

According to Lemma [40](#Thmtheorem40), with probability at least $1-\exp\big{(}-\frac{1}{2}(\frac{1}{2}(1-\zeta)\Xi-K/R)^{2}\big{)}$, we have

$$ $\displaystyle y\cdot f(\bm{p}_{R},\bm{v}_{r};\bm{X})$ $\displaystyle\geq\frac{\langle y\bm{v}_{r},e^{K}\bm{\mu}^{\star}+\bm{\xi}^{\star}\rangle}{e^{K}+1}\geq\frac{e^{K}(1-\gamma)\Gamma r\|\bm{\mu}^{\star}\|^{2}}{\rho^{2}(e^{K}+1)}-\frac{1}{e^{K}+1}\sum_{i\in[n]}|\theta_{i}|\cdot|\langle\bm{\xi}_{i},\bm{\xi}^{\star}\rangle|.$ (A.52) $$

Let $K=\log(\sqrt{d}\sqrt{1/\rho^{2}+\eta n/d})+C<\frac{1}{2}(1-\zeta)R\Xi$. By uniform bound, we have that with probability at least $1-6n\exp(-d/4C_{1}n^{2})-\exp\big{(}-\frac{1}{2}(\frac{1}{2}(1-\zeta)\Xi-K/R)^{2}\big{)}$,

$$ $\displaystyle y\cdot f(\bm{p}_{R},\bm{v}_{r};\bm{X})$ $\displaystyle\geq\frac{e^{K}(1-\gamma)\Gamma r-n\cdot d/(C_{1}n)\cdot 2\sqrt{1/\rho^{2}+\eta n/d}\cdot\Gamma r/\sqrt{d}}{1+e^{K}}$ $\displaystyle\geq\frac{0.8e^{K}\Gamma r-\sqrt{d}/C_{1}\cdot 2\sqrt{1/\rho^{2}+\eta n/d}\cdot\Gamma r}{1+e^{K}}$ $\displaystyle>0,$ $$

where the first inequality uses ([A.51](#A1.E51)), ([A.52](#A1.E52)) and Lemma [42](#Thmtheorem42); The second inequality is from Lemma [38](#Thmtheorem38) and the last inequality is from Assumption [5](#Thmtheorem5) and our selection of $K$. Therefore,

$$ $\displaystyle\mathbb{P}(y\neq f(\bm{p}_{R},\bm{v}_{r};\bm{X}))\leq\exp\Big{(}-\frac{1}{2}(\frac{1}{2}(1-\zeta)\Xi-\frac{K}{R})^{2}\Big{)}+\delta,$ $$

where $\zeta=\frac{\log(2M\|\bm{v}_{mm}\|^{3}n\rho^{2})}{R\Xi}=\Theta\Big{(}\frac{\sqrt{\eta n/d+1/\rho^{2}}}{R}\log(\rho n)\Big{)}$, $K=\log(\sqrt{d}\sqrt{1/\rho^{2}+\eta n/d})+C=\Theta(\log(\sqrt{d/\rho^{2}+\eta n})$ and $\Xi=\|\bm{p}_{mm}\|_{2}^{-1}=\Theta((\eta n/d+1/\rho^{2})^{-1/2})$. Plugging in the order of $\Xi$ and $K$, we have

$$ $\displaystyle\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}$ $\displaystyle(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{R},\bm{v}_{r})))$ $\displaystyle=\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{R},\bm{v}_{r})),y=-\widetilde{y})$ $\displaystyle\quad+\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{R},\bm{v}_{r})),y=\widetilde{y})$ $\displaystyle=\eta+\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{R},\bm{v}_{r})),y=\widetilde{y})$ $\displaystyle\leq\eta+\exp(-d/C_{1}n^{2})+\exp\Big{(}-\Theta\big{(}\frac{(1-\zeta)}{\sqrt{\eta n/d+1/\rho^{2}}}-\frac{\log(nd\sqrt{1/\rho^{2}+\eta n/d})}{R}\big{)}^{2}\Big{)}$ $\displaystyle=\eta+\exp(-\Omega(\frac{d}{n^{2}}))+\exp\Big{(}-\Omega\big{(}\frac{(1-\zeta)}{\sqrt{\eta n/d+1/\rho^{2}}}-\frac{\log(d)}{R}\big{)}^{2}\Big{)},$ $$

where $\zeta=\Theta\Big{(}\frac{\sqrt{\eta n/d+1/\rho^{2}}}{R}\log(\rho n)\Big{)}$. This completes the proof.
∎

###### Lemma 35 (Norm of 𝒑 m ​ m subscript 𝒑 𝑚 𝑚 \bm{p}_{mm} ) .

###### Proof of Lemma 35 .

###### Definition 36 .

###### Remark 37 .

###### Proof of Theorem 16 .

###### Lemma 38 .

###### Proof of Lemma 38 .

###### Lemma 39 .

###### Proof of Lemma 39 .

###### Lemma 40 .

###### Proof of Lemma 40 .

###### Lemma 41 .

###### Proof of Lemma 41 .

###### Lemma 42 .

###### Proof of Lemma 42 .

###### Proof of Theorem 6 .

#### A.2.3 Proof of Thm. 8

###### Lemma 43 .

Consider the next joint-constrained max margin solution:

$$ $\displaystyle\left(\bm{v}_{t},\bm{p}_{t}\right)=\mathop{\mathrm{argmax}}_{\left\|\bm{v}\right\|^{2}+\left\|\bm{p}\right\|^{2}\leq t}\min_{i}y_{i}f(\bm{X}_{i};\bm{p},\bm{v}).$ (A.53) $$

Let $r_{t}:=\left\|\bm{v}_{t}\right\|$ and $R_{t}:=\left\|\bm{v}_{t}\right\|$, then $\left(\bm{v}_{t},\bm{p}_{t}\right)=\left(\bm{v}_{(r_{t},R_{t})},\bm{p}_{(r_{t},R_{t})}\right)$, where $\left(\bm{v}_{(r_{t},R_{t})},\bm{p}_{(r_{t},R_{t})}\right)$ is a solution to Problem [4.1](#S4.E1). Moreover, under Assumption [5](#Thmtheorem5) (items 1-3), with probability at least $1-\delta$ over the random data generation, we have that
$r_{t}\rightarrow\infty,R_{t}\rightarrow\infty$ as $t\rightarrow\infty$.

###### Proof.

By Proposition [15](#Thmtheorem15), with probability at least $1-\delta$, for all $\bm{p}\in\mathbb{R}^{d}$, the token selection under $\bm{p}$ results in a label margin of at most $\Gamma-c\cdot\max\limits_{i\in[n]}(1-s_{i\alpha_{i}}^{\bm{p}})$ in [26](#Thmtheorem26) (with $\bm{r}_{i}=\bm{X}_{i}^{\top}\bm{S}(\bm{X}_{i}\bm{p})$), where $\alpha_{i}=\mathbb{I}(i\in\mathcal{C})+2\mathbb{I}(i\in\mathcal{N})$, $\bm{s}_{i}^{\bm{p}}=\mathbb{S}(\bm{X}_{i}\bm{p})$ is the softmax probabilities, and $c:=C/|\bm{v}_{mm}\|^{3}n\rho^{2}$ is some constant (which may depends on $n$ and $d$, but not in $t$).

Observe that as the norm of $\bm{v}$ increases, the margin increases; thus, it’s easy to verify that $\left\|\bm{v}_{t}\right\|\rightarrow\infty$ as $t\rightarrow\infty$. We argue that also $\left\|\bm{p}_{t}\right\|\rightarrow\infty$ as
$t\rightarrow\infty$. To see that, assume by contradiction that $\left\|\bm{p}_{t}\right\|\leq R_{0}$ for some arbitrary large $t$ that will be determined later. Set $\Gamma=1/\left\|\bm{v}_{mm}\right\|,\left\|\bm{v}_{t}\right\|=r_{t}$, $\tilde{\bm{v}}_{mm}=(r_{t}-1)\Gamma\bm{v}_{mm}$. Hence $t=r_{t}^{2}+R_{0}^{2}$ and $\left\|\tilde{\bm{v}}_{mm}\right\|^{2}=(r-1)^{2}$. The idea is that by decreasing $\left\|\bm{v}_{t}\right\|$ by $1$, we can choose $\bm{p}$ with $\left\|\bm{p}\right\|^{2}+(r_{t}-1)^{2}=t=r_{t}^{2}+R_{0}^{2}$, i.e., $\left\|\bm{p}\right\|^{2}=2r_{t}-1+R_{0}^{2}$, which can be arbitrary large for large enough $t$.
Set $\Pi:=1/\left\|\bm{p}_{mm}\right\|$ and $\tilde{\bm{p}}_{mm}:=\sqrt{2r_{t}-1+R_{0}^{2}}\Pi\bm{p}_{mm}$.
The proof strategy is obtaining a contradiction by proving that $(\tilde{\bm{v}}_{mm},\tilde{\bm{p}}_{mm})$ is a strictly better solution compared to $(\bm{v}_{t},\bm{p}_{t})$.
Define $q_{i}^{\bm{p}}=1-\bm{s}_{i\alpha_{i}}^{\bm{p}}$ to be the amount of non-optimality sopftmax probability where $\bm{s}_{i}^{\bm{p}}=\mathbb{S}(\bm{X}_{i}\bm{p})$ is the softmax probabilities and $\alpha_{i}=1$ iff $i\in\mathcal{C}$ and $2$ otherwise.
Then we have that

$$ $\displaystyle\max_{i}q_{i}^{\bm{p}_{t}}\geq\kappa$ $$

where $\kappa>0$ is a constant that depends just on $R_{0}$ and data parameters (e.g. $n,d,\rho,\delta$). On the other hand, for every $\epsilon>0$, we have that

$$ $\displaystyle q^{*}=\max_{i}q_{i}^{\tilde{\bm{p}}_{mm}}\leq\epsilon,$ $$

for large enough $r_{t}$ i.e. large enough $t$.
Therefore, By Proposition [15](#Thmtheorem15) (see the first paragraph in the proof), we can upper bound the margin induced by $\bm{v}_{t}$ on $(Y_{i},\bm{r}_{i})$ for $\bm{r}_{i}=\bm{X}_{i}^{\top}\mathbb{S}(\bm{X}_{i}\bm{p}_{t})$ by

$$ $\displaystyle\min_{i\in[n]}y_{i}\bm{v}_{t}^{\top}\bm{r}_{i}\leq r_{t}(\Gamma-c\kappa),$ $$

for some constant $c>0$.
On the other hand, the margin induced by $\tilde{\bm{v}}_{mm}$ on
$(Y_{i},\bm{r}_{i})$ for $\bm{r}_{i}=\bm{x}_{i\alpha_{i}}$ is $(r_{t}-1)\Gamma$. This means that we margin induced by $\tilde{\bm{v}}_{mm}$ on
$(y_{i},\bm{r}_{i})$ for $\bm{r}_{i}=\bm{X}_{i}^{\top}\mathbb{S}(\bm{X}_{i}\tilde{\bm{p}}_{mm})$ is at least

$$ $\displaystyle\min_{i}y_{i}r_{i}^{\top}\tilde{\bm{v}}_{mm}$ $\displaystyle\geq\min_{i}y_{i}x_{i\alpha_{i}}^{\top}\tilde{\bm{v}}_{mm}-q^{*}\left\|\bm{x}_{i}^{(1)}-\bm{x}_{i}^{(2)}\right\|\left\|\tilde{\bm{v}}_{mm}\right\|$ $\displaystyle\geq(\bm{r}_{t}-1)(\Gamma-M\epsilon),$ $$

where $M=\sup_{i\in n}\left\|\bm{x}_{i}^{(1)}-\bm{x}_{i}^{(2)}\right\|$. Observe that this lower bound is bigger than the previous upper bound when

$$ $\displaystyle(r_{t}-1)(\Gamma-M\epsilon)>r_{t}(\Gamma-c\kappa)$ $\displaystyle M\epsilon<-(\Gamma-M\epsilon)/r_{t}+c\kappa.$ $$

Choose large enough $t$ such that $(\Gamma-M\epsilon)/r_{t}<c\kappa/2$ and $M\epsilon<c\kappa/2$, gives us the desired contradiction. Recall that $R_{t}:=\left\|p_{t}\right\|$ and $r_{t}:=\left\|v_{t}\right\|$. Since $r_{t}^{2}+R_{t}^{2}\leq t$, we have that $(\bm{v}_{t},\bm{p}_{t}$ is a solution to Problem [4.1](#S4.E1) with $r=r_{t},R=R_{t}$, and $(\bm{v}_{(r_{t},R_{t})},\bm{p}_{(r_{t},R_{t})})$ is a solution to Problem [A.53](#A1.E53).

∎

###### Proof of Thm. 8 .

By Thm. [6](#Thmtheorem6), with probability at least $1-\delta$, the training set is feasible, i.e. exists $(\bm{v},\bm{p})$ such that $\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{v},\bm{p})>0$. Therefore, for any $\gamma>0$, with probability at least $1-\delta$, we have that $\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{v}_{\gamma},\bm{p}_{\gamma})\geq\gamma$, which proves the first part of the Thm.
Next, we show that the classifier $\mathrm{sign}(f(\bm{X};\bm{p}_{\gamma},\bm{v}_{\gamma}))$ generalizes well, for large enough $\gamma$. Recall the next joint-constrained max margin solution:

$$ $\displaystyle\left(v_{t},p_{t}\right)=\mathop{\mathrm{argmax}}_{\left\|\bm{v}\right\|^{2}+\left\|\bm{p}\right\|^{2}\leq t}\min_{i}y_{i}f(\bm{X}_{i};\bm{p},\bm{v}),$ (A.54) $$

which was introduced in Lemma [43](#Thmtheorem43). Fix $\gamma>0$, and let $\left(\bm{v}_{\gamma},\bm{p}_{\gamma}\right)$ be the solution of Problem [4.2](#S4.E2). Define $t(\gamma):=\left\|\bm{v}_{\gamma}\right\|^{2}+\left\|\bm{p}_{\gamma}\right\|^{2}$. We argue that $\left(\bm{v}_{\gamma},\bm{p}_{\gamma}\right)$ is a solution to Problem [A.54](#A1.E54) for $t=t(\gamma)$. Indeed, let

$$ $\displaystyle m:=\max_{\left\|\bm{v}\right\|^{2}+\left\|\bm{p}\right\|^{2}\leq t(\gamma)}\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{p},\bm{v})$ $$

be the maximum margin for Problem [A.54](#A1.E54) with $t=t(\gamma)$. Assume by contradiction that

$$ $\displaystyle\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{p}_{\gamma},\bm{v}_{\gamma})<m,$ $$

which implies that

$$ $\displaystyle\gamma\leq\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{p}_{\gamma},\bm{v}_{\gamma})<m.$ $$

Let $(\bm{v}^{*},\bm{p}^{*})$ be a solution to Problem [A.54](#A1.E54) with $t=t(\gamma)$ i.e. $\left\|\bm{v}^{*}\right\|^{2}+\left\|\bm{p}^{*}\right\|^{2}=t(\gamma)$ and $\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{p}^{*},\bm{v}^{*})=m>\gamma$. Write $\bm{v}^{\prime}:=(\gamma/m)\cdot\bm{v}^{*}$. We remind that $f(\bm{X};\bm{p},\bm{v})=\bm{v}^{\top}\bm{X}^{\top}\mathbb{S}(\bm{X}\bm{p})$ and overall we get that

- •
$\left\|\bm{v}^{\prime}\right\|^{2}+\left\|\bm{p}^{*}\right\|^{2}=(\gamma/m)^{2}\left\|\bm{v}^{*}\right\|^{2}+\left\|\bm{p}^{*}\right\|^{2}<\left\|\bm{v}^{*}\right\|^{2}+\left\|\bm{p}^{*}\right\|^{2}=t(\gamma)$
- •
$\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{p}^{*},\bm{v}^{\prime})=\frac{\gamma}{m}\min_{i\in[n]}y_{i}f(\bm{X}_{i};\bm{p}^{*},\bm{v}^{*})=\frac{\gamma}{m}\cdot m=\gamma$,

which contradicts the optimality of $\left(\bm{v}_{\gamma},\bm{p}_{\gamma}\right)$ to Problem [4.2](#S4.E2). We conclude that $\left(\bm{v}_{\gamma},\bm{p}_{\gamma}\right)$ is a solution to Problem [A.54](#A1.E54) for $t=t(\gamma)$, i.e. $\left(\bm{v}_{\gamma},\bm{p}_{\gamma}\right)=\left(\bm{v}_{t(\gamma)},\bm{p}_{t(\gamma)}\right)$, where $\left(\bm{v}_{t(\gamma)},\bm{p}_{t(\gamma)}\right)$ is a solution for Problem [A.54](#A1.E54) with $t=t(\gamma)$. Let $r_{t(\gamma)}:=\left\|\bm{v}_{t(\gamma)}\right\|$ and $R_{t(\gamma)}:=\left\|\bm{p}_{t(\gamma)}\right\|$. By Lemma [43](#Thmtheorem43) we have

$$ $\displaystyle\left(\bm{v}_{\gamma},\bm{p}_{\gamma}\right)=\left(\bm{v}_{t(\gamma)},\bm{p}_{t(\gamma)}\right)=\left(\bm{v}_{(r_{t(\gamma)},R_{t(\gamma)})},\bm{p}_{(r_{t(\gamma)},R_{t(\gamma)})}\right),$ (A.55) $$

and that $r_{t(\gamma)}\rightarrow\infty,R_{t(\gamma)}\rightarrow\infty$ as $t(\gamma)\rightarrow\infty$. Clearly $t(\gamma)\rightarrow\infty$ as $\gamma\rightarrow\infty$.
By Thm. [6](#Thmtheorem6), The classifier $\mathrm{sign}(f(\bm{X};\bm{p}_{R},\bm{v}_{r}))$ generalizes well on test data:

$$ $\displaystyle\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{(r,R)},\bm{v}_{(r,R)})))$ $\displaystyle=\eta+\exp(-\Omega(d/n^{2}))+\exp\Big{(}-\Theta\big{(}\frac{(1-\zeta)}{\sqrt{\frac{\eta n}{d}+\frac{1}{\rho^{2}}}}-\frac{\log(d)}{R}\big{)}^{2}\Big{)}$ $$

In particular, there exists $r_{0},R_{0}$ such that for any $r\geq r_{0},R\geq R_{0}$, the above probability can be upper bound by $\eta+\exp(-\Omega(d/n^{2}))+\exp(-\Theta((1/\rho^{2}+\eta n/d)^{-1}))$ (see Remark [7](#Thmtheorem7)). Choose large enough $\gamma_{0}$ such that for any $\gamma\geq\gamma_{0}$ we have that $r_{t(\gamma)}\geq r_{0}$ and $R_{t(\gamma)}\geq R_{0}$. Then we conclude

$$ $\displaystyle\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}\left(y\neq\mathrm{sign}(f(\bm{X};\bm{p}_{\gamma},\bm{v}_{\gamma}))\right)$ $\displaystyle=\mathbb{P}_{(\bm{X},y)\sim\mathcal{D}}\left(y\neq\mathrm{sign}\left(f(\bm{X};\bm{p}_{(r_{t(\gamma)},R_{t(\gamma)})},\bm{v}_{(r_{t(\gamma)},R_{t(\gamma)})})\right)\right)$ $\displaystyle\leq\eta+\exp(-\Omega(d/n^{2}))+\exp(-\Theta((1/\rho^{2}+\eta n/d)^{-1})),$ $$

where the first equality is from Eq. [A.55](#A1.E55), as required.
∎

###### Lemma 43 .

###### Proof.

###### Proof of Thm. 8 .

#### A.2.4 Proof of Thm. 10

##### Proof Sketch

First we prove that in this case, only by selecting the noise token for every sample can we achieve the largest margin in the downstream task,

$$ $\bm{r}^{*}_{i}=\bm{\xi}_{i},\forall i\in[n]$ (A.56) $$

Similarly, we define the respective max-margin solution for $\bm{p}$ and $\bm{v}$ in this case.

###### Definition 44 (p-SVM, negative case) .

$\bm{p}$ should satisfy

$$ $\bm{p}_{mm}(\alpha)=\mathop{\mathrm{argmin}}\limits_{p}\|\bm{p}\|$ $$

subjected to

$$ $\displaystyle\bm{p}^{\top}(\bm{\xi}_{i}-\bm{\mu}_{i})\geq 1,$ (A.57) $$

for all $1\leq i\leq n$. $\Xi=1/\|\bm{p}_{mm}\|$ is the margin induced by $\bm{p}_{mm}$.

###### Definition 45 (v-SVM, negative case) .

$$ $\bm{v}(\bm{p})=\mathop{\mathrm{argmin}}_{\bm{v}\in\mathbb{R}^{d}}\|\bm{v}\|\text{ s.t. }y_{i}\cdot\bm{v}^{\top}\bm{r}_{i}\geq 1,\quad\text{ for all }i\in[n].$ (A.58) $$

$\Gamma(\bm{p})=1/\|\bm{v}(\bm{p})\|$
is the label margin induced by $\bm{v}$ and $\bm{p}$. When $\bm{r}_{i}=\bm{\xi}_{i},i\in[n]$,

$$ $\bm{v}_{mm}=\mathop{\mathrm{argmin}}_{\bm{v}\in\mathbb{R}^{d}}\|\bm{v}\|\text{ s.t. }y_{i}\cdot\bm{v}^{\top}\bm{\xi}_{i}\geq 1,\quad\text{ for all }i\in[n].$ (A.59) $$

$\Gamma=1/\|\bm{v}_{mm}\|$
is the label margin induced by $\bm{v}_{mm}$.

To prove this token selection is optimal, we need to explain that the optimality of the token choice is strict in the sense that mixing other tokens will shrink the label margin. We formalize this into the following proposition:

###### Proposition 46 (Optimal Token Condition) .

Suppose that Assumption [9](#Thmtheorem9) holds, with probability at least $1-\delta$ on the training dataset, for all $\bm{p}$, the token selection under $\bm{p}$ results in a label margin of at most $\Gamma-c\cdot\max\limits_{i\in[n]}(1-s_{i2})$.

Then we derive the convergence direction of $\bm{p}$ and $\bm{v}$ by Theorem [16](#Thmtheorem16). Note that as $\|\bm{p}\|\rightarrow\infty$, the attention is more focused on the noise token for every training sample. Therefore, the output of signal token is upper bounded by a small value.

Consider a test sample $(\bm{X},y),\bm{X}=(\bm{\mu}^{\prime},\bm{\xi}^{\prime})$. As $\|\bm{p}\|$ increasing, the noise token $\bm{\xi}^{\prime}$ will will dominate the overall output if $\bm{p}^{\top}_{R}\bm{\xi}^{\prime}\geq 0$, which indicates the output of attention layer will close to the noise token, $\bm{r}^{\prime}\rightarrow\bm{\xi}^{\prime}$. Meanwhile, we can prove that $\bm{p}_{R}$ and $\bm{v}_{r}$ are near orthogonal, so $\bm{p}^{\top}_{R}\bm{\xi}^{\prime}$ and $\bm{v}_{r}^{\top}\bm{\xi}^{\prime}$ are nearly independent variables subjected to Gaussian distribution. Therefore, the probability that $y_{i}\bm{v}^{\top}_{r}\bm{\xi}^{\prime}<0$ is at least constant order.

###### Definition 44 (p-SVM, negative case) .

###### Definition 45 (v-SVM, negative case) .

###### Proposition 46 (Optimal Token Condition) .

##### Optimal Token Condition

First we find the optimal token selection in this case.
See [46](#Thmtheorem46)

###### Proof of Proposition 46 .

Similar as above, we consider the following three situations:

- 1.
$p\neq 0,k-p=0$. (All wrong token selections come from clean set)
- 2.
$p=0,k-p\neq 0$. (All wrong token selections come from noisy set)
- 3.
$p\neq 0,k-p\neq 0$. (Wrong token selections are from both sets)

We will discuss each situation specifically and prove that Proposition [15](#Thmtheorem15) holds in every possible case.
Situation 1: $p\neq 0,k-p=0$

First, let’s see the condition under the optimal choice of tokens:

###### Condition 12 (Original Condition) .

$$ $y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in[n]$ $$

Similarly, $\bm{v}_{mm}$ also satisfies the KKT conditions of the max-margin problem ([A.31](#A1.E31)) in this case, so we could write $\bm{v}$ as

$$ $\displaystyle\bm{v}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum\limits_{i\in[n]}y_{i}\theta_{i}\bm{\xi}_{i}.$ (A.60) $$

Plugging ([A.60](#A1.E60)) in the condition [12](#Thmcondition12), we can rewrite these conditions as:

$$ $\theta_{i}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in[n].$ $$

Then we introduce a lemma to estimate the parameters of optimal solution under this condition:

###### Lemma 47 (Balanceing noise factor for KKT point) .

Suppose that Assumption [9](#Thmtheorem9) holds, under Condition [12](#Thmcondition12), we have

$$ $\displaystyle\max\limits_{i\in[n]}\theta_{i}$ $\displaystyle\leq\frac{1}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}},$ $\displaystyle\min\limits_{i\in[n]}\theta_{i}$ $\displaystyle\geq\frac{(1-\kappa)d-4n\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})}.$ $$

###### Proof of Lemma 47 .

First we prove the upper bound.
Denote $j=\mathop{\mathrm{argmax}}\limits_{i\in[n]}\theta_{i}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\top}\bm{\xi}_{j}$ $\displaystyle=\sum\limits_{i\in[n]}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle=\theta_{j}\|\bm{\xi}_{j}\|_{2}^{2}+\sum\limits_{i\neq j,i\in[n]}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\geq\theta_{j}\cdot(1-\kappa)d-n\theta_{j}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $$

The last inequality is because Lemma [57](#Thmtheorem57) and the definition of j. Consider the contrary case when $\theta_{j}>\frac{1}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\top}\bm{\xi}_{j}$ $\displaystyle>\frac{1}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\cdot((1-\kappa)d-n\cdot 2\sqrt{d\log(6n^{2}/\delta)})=1.$ $$

By the KKT conditions, if $y_{j}\bm{v}^{\top}\bm{\xi}_{j}>1$ then we must have $\theta_{j}=0$, and thus we reach a contradiction.

Then we prove the lower bound. For $\forall j\in[n]$ we have

$$ $\displaystyle 1$ $\displaystyle\leq\theta_{j}\|\bm{\xi}_{j}\|_{2}^{2}+\sum\limits_{i\neq j,i\in[n]}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle\leq\theta_{j}\cdot(1+\kappa)d+n\max\limits_{i\in[n]}\theta_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq\theta_{j}\cdot(1+\kappa)d+\frac{n}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\cdot 2\sqrt{d\log(6n^{2}/\delta)}.$ $$

The second inequality is due to Lemma [57](#Thmtheorem57) and the last inequality is from the upper bound we just get. Therefore, we have

$$ $\displaystyle\theta_{j}\geq\frac{(1-\kappa)d-4n\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})}$ $$

This completes the proof.

∎

As for the signal parameters $\lambda_{1}$ and $\lambda_{2}$, to achieve the minimal norm for $\bm{v}$, it is obvious that $\lambda_{1}=\lambda_{2}=0$. Then we can estimate $\|\bm{v}_{mm}\|$ in this case:

###### Lemma 48 (Norm of $\bm{v}_{mm}$ ) .

Suppose that Assumption [9](#Thmtheorem9) holds, with probability at least $1-\delta$ on the training dataset, for the solution $\bm{v}_{mm}$ of ([A.31](#A1.E31)) under the token selection ([A.56](#A1.E56)), we have

$$ $\displaystyle\frac{n}{2d}\leq\|\bm{v}_{mm}\|^{2}\leq\frac{5n}{d}.$ $$

This implies

$$ $\|\bm{v}_{mm}\|=\Theta\bigg{(}\sqrt{\frac{n}{d}}\bigg{)}.$ $$

###### Proof of Lemma 48 .

As $\bm{v}_{mm}$ is the max-margin solution and satisfies KKT condition, it can be represented as

$$ $\displaystyle\bm{v}_{mm}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum\limits_{i\in\mathcal{C}}y_{i}\theta_{i}\bm{\xi}_{i}+\sum\limits_{i\in[n]}y_{i}\theta_{i}\bm{\xi}_{i}.$ (A.61) $$

As there is no constraint on $\lambda_{1},\lambda_{2}$, both of them can take 0 to achieve max-margin. So we could lower bound $\|\bm{v}_{mm}\|$ as

$$ $\displaystyle\|\bm{v}_{mm}\|^{2}$ $\displaystyle\geq\sum\limits_{i\in[n]}\theta_{i}^{2}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\theta_{i}\theta_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle\geq O\bigg{(}\frac{n^{2}}{d^{3/2}}\bigg{)}\geq\frac{n}{2d}.$ $$

The second inequality is from Lemma [47](#Thmtheorem47) that $\theta_{i}=\Theta(1/d)$ for $i\in[n]$ and the last inequality is from Assumption [9](#Thmtheorem9).

Then to upper bound $\|\bm{v}_{mm}\|$, consider the following possible solution $\widetilde{\bm{v}}$

$$ $\displaystyle\widetilde{\bm{v}}=\sum\limits_{i\in[n]}2y_{i}\bm{\xi}_{i}/d.$ $$

For $i\in[n]$, we have

$$ $\displaystyle y_{i}\widetilde{\bm{v}}^{\top}\bm{r}_{i}$ $\displaystyle=y_{i}\widetilde{\bm{v}}^{\top}\bm{\xi}_{i}=2\|\bm{\xi}_{i}\|^{2}/d+\sum\limits_{j\in[n],j\neq i}2y_{i}y_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle/d$ $\displaystyle\geq 2(1-\kappa)-2n\sqrt{\log(6n^{2}/\delta)/d}\geq 1.$ $$

The first inequality is from Lemma [57](#Thmtheorem57) and the second inequality is from Assumption [9](#Thmtheorem9). Therefore, $\widetilde{\bm{v}}$ is a possible solution of SVM problem [26](#Thmtheorem26) when $\bm{p}$ converges to $\bm{p}_{mm}$. So we have

$$ $\displaystyle\|\bm{v}_{mm}\|^{2}$ $\displaystyle\leq\|\widetilde{\bm{v}}\|^{2}=\sum\limits_{i\in[n]}4\|\bm{\xi}_{i}\|^{2}/d^{2}+\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}4y_{i}y_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle/d^{2}\leq\frac{5n}{d}.$ $$

The last inequality is from Lemma [57](#Thmtheorem57), Lemma [59](#Thmtheorem59) and Assumption [9](#Thmtheorem9). Combine the results above, we have $\|\bm{v}_{mm}\|^{2}=\Theta(\frac{n}{d})$.

∎

Denote the mixed samples as $k_{1},k_{2},...,k_{p}$. And for every mixed sample $k_{i}$, we have $\bm{r}_{k_{i}}=(1-\beta_{i})\bm{\mu}_{k_{i}}+\beta_{i}\bm{\xi}_{k_{i}}$. Without losing generality, we assume that $y_{k_{i}}=+1$ for all $i\in[p]$. Then
the conditions under Situation 1 become

###### Condition 13 ( $p$ clean samples violating optimal token selection) .

$$ $\begin{cases}&y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in[n]\backslash[p]\\ &\bm{v}^{\top}\bm{r}_{k_{i}}\geq 1,i\in[p]\end{cases}$ $$

Denote the max-margin solution under this condition as $\bm{v}^{\prime}$ with parameters $\lambda^{\prime}_{1},\lambda^{\prime}_{2},\theta^{\prime}_{i}$. Plugging this representation into the condition [13](#Thmcondition13), we have:

$$ $\begin{cases}&\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in[n]\backslash[p]\\ &(1-\beta_{i})\lambda_{1}^{\prime}\cdot\|\bm{\mu}_{1}\|^{2}+\beta_{i}(\theta^{\prime}_{k_{i}}\cdot\|\bm{\xi}_{k_{i}}\|^{2}+\sum\limits_{i^{\prime}\neq k_{i}}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{k_{i}},\bm{\xi}_{i^{\prime}}\rangle)\geq 1,i\in[p]\end{cases}$ $$

We consider two cases: $\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}<1$ and $\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1$. First when $\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}<1$, the condition for mixed clean sample becomes:

$$ $\displaystyle\theta^{\prime}_{k_{i}}\cdot\|\bm{\xi}_{k_{i}}\|^{2}+\sum\limits_{i^{\prime}\neq k_{i}}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{k_{i}},\bm{\xi}_{i^{\prime}}\rangle\geq\frac{1-(1-\beta_{i})\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta_{i}}>1,$ $$

which indicates that the condition for $\theta^{\prime}_{k_{i}}$ is strengthened. So mixing 1 more clean sample is equal
to strengthening 1 constraint in the original setting. Therefore, mixing p samples will not result
in a better solution than only mixing 1 clean sample. Then we can simplify this case to mixing only 1 clean sample and denote this sample as $k_{*}$, $r_{k_{*}}=(1-\beta)\bm{\mu}_{1}+\beta\bm{\xi}_{k_{*}}$. Now the condition becomes:

###### Condition 14 (1 clean sample violating optimal token selection) .

$$ $\begin{cases}&\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in[n]\backslash\{k_{*}\}\\ &(1-\beta)\lambda_{1}^{\prime}\cdot\|\bm{\mu}_{1}\|^{2}+\beta(\theta^{\prime}_{k_{*}}\cdot\|\bm{\xi}_{k_{i}}\|^{2}+\sum\limits_{i^{\prime}\neq k_{*}}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{k_{*}},\bm{\xi}_{i^{\prime}}\rangle)\geq 1\end{cases}$ $$

Similarly, we introduce the following lemma which estimates the parameters in $\bm{v}^{\prime}$. We define

$$ $\alpha=\frac{1-(1-\beta)\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta}$ $$

for the convenience of the following proof.

###### Lemma 49 .

Suppose that Assumption [9](#Thmtheorem9) holds, under condition [14](#Thmcondition14), with probability at least $1-\delta$ on the training dataset, we have

$$ $\displaystyle\theta^{\prime}_{k_{*}}$ $\displaystyle\leq\frac{\alpha}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}},$ $\displaystyle\theta^{\prime}_{k_{*}}$ $\displaystyle\geq\frac{\alpha}{(1+\kappa)d}\bigg{(}1-\frac{2n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)},$ $\displaystyle\max\limits_{i\in[n]\backslash\{k_{*}\}}\theta^{\prime}_{i}$ $\displaystyle\leq\frac{(1-\kappa)d+2(\alpha-n)\sqrt{d\log(6n^{2}/\delta)}}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})^{2}},$ $\displaystyle\min\limits_{i\in[n]\backslash\{k_{*}\}}\theta^{\prime}_{i}$ $\displaystyle\geq\frac{1}{(1+\kappa)d}\cdot\bigg{(}1-\frac{2n\alpha\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}.$ $$

###### Proof of Lemma 49 .

Denote $j=\mathop{\mathrm{argmax}}\limits_{i\in[n]}\theta^{\prime}_{i}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\prime\top}\bm{\xi}_{j}$ $\displaystyle=\theta^{\prime}_{j}\|\bm{\xi}_{j}\|^{2}+\sum\limits_{i\in[n],i\neq j}y_{i}y_{j}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\geq\theta^{\prime}_{j}(1-\kappa)d-n\max\limits_{i\in[n]}\theta^{\prime}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=\theta^{\prime}_{j}((1-\kappa)d-n\cdot 2\sqrt{d\log(6n^{2}/\delta)}).$ $$

The first inequality is due to Lemma [57](#Thmtheorem57) and the last equation is from our definition of j. Consider the contrary case when $\theta^{\prime}_{j}>\frac{\alpha}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\prime\top}\bm{\xi}_{j}$ $\displaystyle>\alpha.$ $$

By the KKT conditions, if $y_{j}\bm{v}^{\prime\top}\bm{\xi}_{j}>\frac{1+\lambda^{\prime}_{1}(1-\beta)\|\bm{\mu}_{1}\|^{2}}{\beta}$ then we must have $\theta^{\prime}_{j}=0$, and thus we reach a contradiction. Therefore, $\theta^{\prime}_{k_{\star}}\leq\theta^{\prime}_{j}\leq\frac{\alpha}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}$. Then denote $j^{\prime}=\mathop{\mathrm{argmax}}\limits_{i\in[n],i\neq k_{\star}}\theta^{\prime\prime}_{i}$, we have

$$ $\displaystyle y_{j^{\prime}}\bm{v}^{\prime\top}\bm{\xi}_{j^{\prime}}$ $\displaystyle=\theta^{\prime}_{j^{\prime}}\|\bm{\xi}_{j^{\prime}}\|^{2}+\sum\limits_{i\in[n],i\neq j^{\prime}}y_{i}y_{j^{\prime}}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j^{\prime}}\rangle$ $\displaystyle\geq\theta^{\prime}_{j^{\prime}}(1-\kappa)d-n\max\limits_{i\in[n],i\neq j^{\prime}}\theta^{\prime}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}-\theta^{\prime}_{k_{\star}}\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\geq\theta^{\prime}_{j}((1-\kappa)d-n\cdot 2\sqrt{d\log(6n^{2}/\delta)})-\frac{2\alpha\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}.$ $$

The first inequality is from Lemma [57](#Thmtheorem57) and the second inequality is from the upper bound of $\theta^{\prime}_{k_{\star}}$ we just get. Consider the case when $\theta^{\prime}_{j^{\prime}}>\frac{(1-\kappa)d+2(\alpha-n)\sqrt{d\log(6n^{2}/\delta)}}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})^{2}}$, we have

$$ $\displaystyle y_{j^{\prime}}\bm{v}^{\prime\top}\bm{\xi}_{j^{\prime}}>1.$ $$

By the complementary slackness condition, if $y_{j^{\prime}}\bm{v}^{\prime\prime\top}\bm{\xi}_{j^{\prime}}>1$ then we must have $\theta^{\prime}_{j^{\prime}}=0$, and thus we reach a contradiction.

Next we estimate the lower bound of $\theta^{\prime}_{j}$ when $j\neq k_{*}$. We have

$$ $\displaystyle 1$ $\displaystyle\leq y_{j}\bm{v}^{\prime\top}\bm{\xi}_{j}$ $\displaystyle=\theta^{\prime}_{j}\|\bm{\xi}_{j}\|^{2}+\sum\limits_{i\in[n],i\neq j}y_{i}y_{j}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\leq\theta^{\prime}_{j}(1+\kappa)d+n\max\limits_{i\in[n]}\theta^{\prime}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq\theta^{\prime}_{j}(1+\kappa)d+\frac{\alpha}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\cdot 2n\sqrt{d\log(6n^{2}/\delta)}$ $$

The last inequality is from the upper bound of $\theta^{\prime}_{k_{*}}$ we just get. Therefore, we have

$$ $\displaystyle\theta^{\prime}_{j}\geq\frac{1}{(1+\kappa)d}\cdot\bigg{(}1-\frac{2n\alpha\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}$ $$

for all $j\in[n]$ and $j\neq k_{*}$.

Last we lower bound $\theta^{\prime}_{k_{*}}$. We have

$$ $\displaystyle\alpha$ $\displaystyle\leq y_{k}\bm{v}^{\prime\prime\top}\bm{\xi}_{k_{*}}$ $\displaystyle=\theta^{\prime}_{k_{*}}(1+\kappa)d+n\max\limits_{i\in[n]}\theta^{\prime}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $$

Similarly, we have

$$ $\displaystyle\theta^{\prime}_{k_{*}}\geq\frac{\alpha}{(1+\kappa)d}\bigg{(}1-\frac{2n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}.$ $$

∎

Therefore, we could estimate the difference between $\|\bm{v}^{\prime}\|^{2}$ and $\|\bm{v}_{mm}\|^{2}$.

###### Lemma 50 .

Suppose that Assumption [9](#Thmtheorem9) holds, with probability at least $1-\delta$ on the training dataset, denote $\bm{v}$ and $\bm{v}^{\prime}$ as the optimal solutions under condition [12](#Thmcondition12) and condition [14](#Thmcondition14) respectively. We have

$$ $\displaystyle\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\frac{C_{1}(1-\beta)}{d}.$ $$

where $C_{1}=\Theta(1)$ is a constant.

###### Proof of Lemma 50 .

From the first inequality in Condition [14](#Thmcondition14), for $i[n],i\neq k_{\star}$ we have

$$ $\displaystyle\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\neq i,k_{\star}}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1-y_{i}y_{k_{\star}}\theta^{\prime}_{k_{\star}}\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle.$ $$

Then we add $y_{i}y_{k_{\star}}w\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle$ on both sides, where we set $w=\theta^{\prime}_{k_{\star}}-\frac{\alpha-1}{(1+\kappa)d-2\sqrt{d\log(6n^{2}/\delta)}}\leq\theta^{\prime}_{k^{\star}}$. Then we have

$$ $\displaystyle\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i,k^{\star}}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle+y_{i}y_{k_{\star}}w\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle$ $\displaystyle\geq 1-y_{i}y_{k_{\star}}(\theta^{\prime}_{k^{\star}}-w)\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle$ $\displaystyle\geq 1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=\frac{(1+\kappa)d-2\alpha\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d-2\sqrt{d\log(6n^{2}/\delta)}}.$ (A.62) $$

The second inequality is from Lemma [57](#Thmtheorem57). Now consider a new $\underline{\bm{v}}=\underline{\lambda}_{1}\bm{\mu}_{1}+\underline{\lambda}_{2}\bm{\mu}_{2}+\sum\limits_{i\in[n]}y_{i}\underline{\theta}_{i}\bm{\xi}_{i}$ with

$$ $\underline{\lambda}_{1}=\lambda^{\prime}_{1};\quad\underline{\lambda}_{2}=\lambda^{\prime}_{2};$ $$

$$ $\underline{\theta}_{i}=\theta^{\prime}_{i}/(1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)})\text{ for }i\in[n],i\neq k_{\star}$ $$

and

$$ $\underline{\theta}_{k_{\star}}=\frac{w}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}.$ $$

We can prove that $\underline{\bm{v}}$ satisfies all constraints for $\bm{v}_{mm}$.

By dividing $1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}$ on both sides of ([A.62](#A1.E62)), for $\forall i\in[n],i\neq k_{\star}$ we have

$$ $\displaystyle\underline{\theta}_{i}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\underline{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1.$ $$

Then we prove that $\underline{\theta}_{k^{\star}}\|\bm{\xi}_{k^{\star}}\|^{2}+\sum\limits_{i\neq k^{\star}}y_{i}y_{k_{\star}}\underline{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle\geq 1$. From the last inequality in Condition [14](#Thmcondition14) we have

$$ $\displaystyle\theta^{\prime}_{k_{\star}}\cdot\|\bm{\xi}_{k_{\star}}\|^{2}+\sum\limits_{i\neq{k_{\star}}}y_{k_{\star}}y_{i}\theta^{\prime}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle$ $\displaystyle\geq\alpha.$ $$

Dividing $1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}$ on both sides, we get

$$ $\displaystyle\frac{\theta^{\prime}_{k_{\star}}\|\bm{\xi}_{k_{\star}}\|^{2}}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}+\sum\limits_{i\neq k_{\star}}y_{i}y_{k_{\star}}\underline{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle\geq\frac{\alpha}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}.$ $$

Therefore we have

$$ $\displaystyle\underline{\theta}_{k_{\star}}\|\bm{\xi}_{k_{\star}}\|^{2}+\sum\limits_{i\neq k_{\star}}y_{i}y_{k_{\star}}\underline{\theta}_{i}\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle$ $\displaystyle\geq\frac{\alpha-(\theta^{\prime}_{k_{\star}}-w)\|\bm{\xi}_{k_{\star}}\|^{2}}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}\geq\frac{\alpha-(\theta^{\prime}_{k_{\star}}-w)(1+\kappa)d}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}=1.$ $$

The second inequality is from Lemma [57](#Thmtheorem57) and the last equality is by our definition $\theta^{\prime}_{k_{\star}}-w=\frac{\alpha-1}{(1+\kappa)d-2\sqrt{d\log(6n^{2}/\delta)}}$. Thus, $\underline{\bm{v}}$ is a possible solution under Condition [1](#Thmcondition1) and $\|\underline{\bm{v}}\|\geq\|\bm{v}_{mm}\|$.

Next we estimate the difference between $\|\bm{v}^{\prime}\|^{2}$ and $\|\underline{\bm{v}}\|^{2}$. The expansion of $\|\bm{v}^{\prime}\|^{2}$ and $\|\underline{\bm{v}}\|^{2}$ are:

$$ $\displaystyle\|\bm{v}^{\prime}\|^{2}$ $\displaystyle=\lambda^{\prime 2}_{1}\|\bm{\mu}_{1}\|^{2}+\lambda^{\prime 2}_{2}\|\bm{\mu}_{2}\|^{2}+\sum\limits_{i\in[n]}\theta^{\prime 2}_{i}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\theta^{\prime}_{i}\theta^{\prime}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle,$ $\displaystyle\|\underline{\bm{v}}\|^{2}$ $\displaystyle=\underline{\lambda}_{1}^{2}\|\bm{\mu}_{1}\|^{2}+\underline{\lambda}_{2}^{2}\|\bm{\mu}_{2}\|^{2}+\sum\limits_{i\in[n]}\underline{\theta}_{i}^{2}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\underline{\theta}_{i}\underline{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle.$ $$

Similar to the condition ([A.38](#A1.E38)), we have $\|\bm{v}^{\prime}\|\leq 2\|\bm{v}_{mm}\|=\Theta(\sqrt{n/d})$, which implies that $\alpha=O(\sqrt{n}\log n)$. Otherwise, we have

$$ $\theta_{k_{\star}}^{\prime}\|\bm{\xi}_{k_{\star}}\|^{2}\geq\alpha-\sum_{i\neq k_{\star}}y_{k_{\star}}y_{i}\theta_{i}^{\prime}\langle\bm{\xi}_{i},\bm{\xi}_{k_{\star}}\rangle=\Omega(\alpha).$ $$

It further yields that

$$ $\|\bm{v}^{\prime}\|^{2}=\Omega(\frac{n}{d})+\theta_{k_{\star}}^{\prime 2}\|\bm{\xi}_{k_{\star}}\|^{2}=\Omega(\frac{n}{d}+\frac{\alpha^{2}}{d})=\Omega(\frac{n\log^{2}n}{d}),$ $$

which contradicts with $\|\bm{v}^{\prime}\|=\Theta(\sqrt{n/d})$.

We decompose the difference between $\|\bm{v}^{\prime}\|^{2}$ and $\|\underline{\bm{v}}\|^{2}$ into four terms:

$$ $\displaystyle\|\bm{v}^{\prime}\|^{2}-\|\underline{\bm{v}}\|^{2}=$ $\displaystyle\underbrace{(\theta^{\prime 2}_{k_{\star}}-\underline{\theta}^{2}_{k_{\star}})\|\bm{\xi}_{k_{\star}}\|^{2}}_{I_{1}}+\underbrace{\sum\limits_{i\in[n],i\neq{k_{\star}}}(\theta^{\prime 2}_{i}-\underline{\theta}^{2}_{i})\|\bm{\xi}_{i}\|^{2}}_{I_{2}}-\underbrace{\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\underline{\theta}_{i}\underline{\theta}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{3}}$ $\displaystyle+\underbrace{\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\theta^{\prime}_{i}\theta^{\prime}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{4}}.$ $$

We now estimate $I_{1}$ to $I_{4}$ sequentially. For the first term,

$$ $\displaystyle I_{1}$ $\displaystyle\geq(\theta^{\prime 2}_{k_{\star}}-\underline{\theta}^{2}_{k_{\star}})(1-\kappa)d=(\theta^{\prime}_{k_{\star}}-\underline{\theta}_{k_{\star}})(\theta^{\prime}_{k_{\star}}+\underline{\theta}_{k_{\star}})(1-\kappa)d$ $\displaystyle=\frac{(\alpha-1)(1-2\theta^{\prime}_{k_{\star}}\sqrt{d\log(6n^{2}/\delta)})}{(1+\kappa)d-2\sqrt{d\log(6n^{2}/\delta)}}\cdot\Omega\bigg{(}\frac{1}{d}\bigg{)}\cdot(1-\kappa)d$ $\displaystyle=\Omega\bigg{(}\frac{\alpha-1}{d}\bigg{)},$ $$

where the first inequality is from Lemma [57](#Thmtheorem57); the second equality is from Lemma [49](#Thmtheorem49); and the last equality uses the fact that $\alpha=O(\sqrt{n}\log n)$. Then we can further upper bound $\max\limits_{i\in[n],i\neq k_{\star}}\theta^{\prime}_{i}$ as

$$ $\displaystyle\max\limits_{i\in[n],i\neq k_{\star}}\theta^{\prime}_{i}$ $\displaystyle\leq\frac{(1-\kappa)d+2(\alpha-n)\sqrt{d\log(6n^{2}/\delta)}}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})^{2}}=O(\frac{1}{d}).$ (A.63) $$

For the second term $I_{2}$, we have

$$ $\displaystyle|I_{2}|$ $\displaystyle\leq\sum\limits_{i\in[n],i\neq{k_{\star}}}(\underline{\theta}^{2}_{i}-\theta^{\prime 2}_{i})(1+\kappa)d$ $\displaystyle\leq\bigg{(}\frac{1}{(1-(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)})^{2}}-1\bigg{)}\max\limits_{i\in[n],i\neq k_{\star}}\theta^{\prime 2}_{i}\cdot n(1+\kappa)d$ $\displaystyle=\frac{(\alpha-1)\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d-\sqrt{d\log(6n^{2}/\delta)}}\cdot O(\frac{n}{d})=\widetilde{O}\bigg{(}\frac{(\alpha-1)n}{d^{3/2}}\bigg{)}.$ $$

The second inequality is from Lemma [49](#Thmtheorem49). The first equality is from ([A.63](#A1.E63)) and the last equality is from Assumption [9](#Thmtheorem9).

Then we bound $|-I_{3}+I_{4}|$ as:

$$ $\displaystyle|-I_{3}+I_{4}|\leq$ $\displaystyle\sum\limits_{i\in[n]}\sum\limits_{j\in[n]\backslash\{i\}}|\underline{\theta}_{i}\underline{\theta}_{j}-\theta^{\prime}_{i}\theta^{\prime}_{j}|\cdot|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|$ $\displaystyle\leq$ $\displaystyle\sum\limits_{i\in[n]\backslash\{k_{\star}\}}\sum\limits_{j\in[n]\backslash\{k_{\star},i\}}|\underline{\theta}_{i}\underline{\theta}_{j}-\theta^{\prime}_{i}\theta^{\prime}_{j}|\cdot|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|+2\sum\limits_{t\in[n]\backslash\{k_{\star}\}}|\underline{\theta}_{k_{\star}}\underline{\theta}_{t}-\theta^{\prime}_{k_{\star}}\theta^{\prime}_{t}|\cdot|\langle\bm{\xi}_{k_{\star}},\bm{\xi}_{t}\rangle|$ $\displaystyle\leq$ $\displaystyle n^{2}\bigg{(}\frac{1}{(1-(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)})^{2}}-1\bigg{)}\max\limits_{i\in[n],i\neq k_{\star}}\theta^{\prime 2}_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle+n\bigg{(}\theta^{\prime}_{k_{\star}}-\frac{\underline{\theta}_{k_{\star}}}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}\max\limits_{i\in[n],i\neq k_{\star}}\theta^{\prime}_{i}4\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq$ $\displaystyle\frac{(\alpha-1)\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d-\sqrt{d\log(6n^{2}/\delta)}}\cdot O(\frac{n^{2}(1+\kappa)}{d^{3/2}})+\frac{\alpha-1}{d}\cdot O(\frac{n}{d})\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=$ $\displaystyle O\bigg{(}\frac{(\alpha-1)n^{2}}{d^{2}}+\frac{(\alpha-1)n}{d^{3/2}}\bigg{)}.$ $$

The third inequality is from Lemma [47](#Thmtheorem47) and Lemma [49](#Thmtheorem49); The fourth inequality is from the fact that

$$ $\displaystyle\theta^{\prime}_{k_{\star}}-\frac{\underline{\theta}_{k_{\star}}}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}$ $\displaystyle=\frac{\theta^{\prime}_{k_{\star}}-\underline{\theta}_{k_{\star}}-2\theta^{\prime}_{k_{\star}}(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}$ $\displaystyle=\frac{\Omega(\frac{\alpha-1}{d})-O(\frac{\alpha(\alpha-1)}{d^{3/2}})}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}>0$ $$

So we have $\theta^{\prime}_{k_{\star}}-\frac{\underline{\theta}_{k_{\star}}}{1-2(\theta^{\prime}_{k_{\star}}-w)\sqrt{d\log(6n^{2}/\delta)}}\leq\theta^{\prime}_{k_{\star}}-\underline{\theta}_{k_{\star}}$; The last equality is from Assumption [5](#Thmtheorem5).

Combining the above results, we have

$$ $\displaystyle\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\Theta\bigg{(}\frac{\alpha-1}{d}\bigg{)}+O\bigg{(}\frac{(\alpha-1)\eta n}{d^{3/2}}\bigg{)}\geq\frac{C_{1}(1-\beta)}{d}.$ $$

Here $C_{1}=\Theta(1)$ is a constant.
∎

Then we consider the case when $\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1$. In this case, the condition for mixed clean sample becomes:

$$ $\displaystyle\theta^{\prime}_{k_{i}}\cdot\|\bm{\xi}_{k_{i}}\|^{2}+\sum\limits_{i^{\prime}\neq k_{i}}y_{ki}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{k_{i}},\bm{\xi}_{i^{\prime}}\rangle\geq\frac{1-(1-\beta_{i})\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta_{i}},$ $$

and $\frac{1-(1-\beta_{i})\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta_{i}}\leq 1$, which indicates that the condition for $\theta^{\prime}_{k_{i}}$ is relaxed. So mixing 1 more clean sample is equal to relaxing 1 constraint in the original setting. Therefore, mixing all clean samples will achieve the best result. From the data generalization model, there are $(1-\eta)n/2+o(n)$ clean samples with label $+1$ and denote $S_{+1}$ as their set. Now the condition becomes:

###### Condition 15 (All clean samples violating optimal token selection) .

$$ $\begin{cases}&\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle)\geq 1,i\in[n]\backslash\ S_{+1}\\ &(1-\beta)\lambda_{1}^{\prime}\cdot\|\bm{\mu}_{1}\|^{2}+\beta(\theta^{\prime}_{i}\cdot\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle)\geq 1,i\in S_{+1}\end{cases}$ $$

We have another lemma to estimate the scale of parameters in the max-margin solution in this case. Here $\alpha=\frac{1-(1-\widetilde{\beta})\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\widetilde{\beta}}$ and $\widetilde{\beta}=\min\limits_{i\in[n]}\{\beta_{i}\}$.

###### Lemma 51 .

Suppose that Assumption [9](#Thmtheorem9) holds, under Condition [15](#Thmcondition15), we have

$$ $\displaystyle\max\limits_{i\in[n]}\theta^{\prime}_{i}$ $\displaystyle\leq\frac{1}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}},$ $\displaystyle\min\limits_{i\in[n]}\theta^{\prime}_{i}$ $\displaystyle\geq\frac{(1-\kappa)d\alpha-2n\sqrt{d\log(6n^{2}/\delta)}(\alpha+1)}{(1+\kappa)d((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})}.$ $$

###### Proof of Lemma 51 .

First we prove the upper bound.
Denote $j=\mathop{\mathrm{argmax}}\limits_{i\in[n]}\theta_{i}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\top}\bm{\xi}_{j}$ $\displaystyle=\sum\limits_{i\in[n]}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle=\theta_{j}\|\bm{\xi}_{j}\|_{2}^{2}+\sum\limits_{i\neq j,i\in[n]}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\geq\theta_{j}\cdot(1-\kappa)d-n\theta_{j}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $$

The last inequality is because Lemma [57](#Thmtheorem57) and the definition of j. Consider the contrary case when $\theta_{j}>\frac{1}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\top}\bm{\xi}_{j}$ $\displaystyle>\frac{1}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\cdot((1-\kappa)d-n\cdot 2\sqrt{d\log(6n^{2}/\delta)})=1.$ $$

By the KKT conditions, if $y_{j}\bm{v}^{\top}\bm{\xi}_{j}>1$ then we must have $\theta_{j}=0$, and thus we reach a contradiction.

Then we prove the lower bound. For $\forall j\in S_{+1}$ we have

$$ $\displaystyle\alpha$ $\displaystyle\leq\theta_{j}\|\bm{\xi}_{j}\|_{2}^{2}+\sum\limits_{i\neq j,i\in[n]}y_{i}y_{j}\theta_{i}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ $\displaystyle\leq\theta_{j}\cdot(1+\kappa)d+n\max\limits_{i\in[n]}\theta_{i}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq\theta_{j}\cdot(1+\kappa)d+\frac{n}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\cdot 2\sqrt{d\log(6n^{2}/\delta)}.$ $$

The second inequality is due to Lemma [57](#Thmtheorem57) and the last inequality is from the upper bound we just get. Therefore, we have

$$ $\displaystyle\theta_{j}\geq\frac{(1-\kappa)d\alpha-2n\sqrt{d\log(6n^{2}/\delta)}(\alpha+1)}{(1+\kappa)d((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})}.$ $$

This completes the proof

∎

Then we can estimate the difference between $\|\bm{v}^{\prime}\|^{2}$ and $\|\bm{v}_{mm}\|^{2}$ with the following lemma:

###### Lemma 52 .

Suppose that Assumption [9](#Thmtheorem9) holds, denote $\bm{v}$ and $\bm{v}^{\prime}$ as the optimal solutions under condition [12](#Thmcondition12) and condition [15](#Thmcondition15) respectively. We have

$$ $\displaystyle\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\frac{C_{2}(1-\beta)}{\rho^{2}}.$ $$

where $C_{2}=\Theta(1)$ is a constant.

###### Proof of Lemma 52 .

Recall the expansion of $\|\bm{v}_{mm}\|^{2}$ and $\|\bm{v}^{\prime}\|^{2}$:

$$ $\displaystyle\|\bm{v}_{mm}\|^{2}$ $\displaystyle=\sum\limits_{i\in[n]}\theta_{i}^{2}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\theta_{i}\theta_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle,$ $\displaystyle\|\bm{v}^{\prime}\|^{2}$ $\displaystyle=\lambda^{\prime 2}_{1}\|\bm{\mu}_{1}\|^{2}+\sum\limits_{i\in[n]}\theta^{\prime 2}_{i}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\theta^{\prime}_{i}\theta^{\prime}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle.$ $$

Then we have

$$ $\displaystyle\|\bm{v}^{\prime}\|^{2}-\|\bm{v}_{mm}\|^{2}=$ $\displaystyle\underbrace{\lambda^{\prime 2}_{1}\|\bm{\mu}_{1}\|^{2}}_{I_{1}}+\underbrace{\sum\limits_{i\in[n]}(\theta^{\prime 2}_{i}-\theta^{2}_{i})\|\bm{\xi}_{i}\|^{2}}_{I_{2}}-\underbrace{\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\theta_{i}\theta_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{3}}$ $\displaystyle+\underbrace{\sum\limits_{i\in[n]}\sum\limits_{j\in[n]}y_{i}y_{j}\theta^{\prime\prime}_{i}\theta^{\prime\prime}_{j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle}_{I_{4}}.$ $$

We now estimate $I_{1}$ to $I_{4}$ sequentially. Here we use the same notation $\alpha=\frac{1-(1-\widetilde{\beta})\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\widetilde{\beta}}$ and $\widetilde{\beta}=\min\limits_{i\in[n]}\{\beta_{i}\}$ as in Lemma [51](#Thmtheorem51). First from our assumption $\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1$ we have

$$ $\displaystyle I_{1}=\lambda^{\prime 2}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1/\rho^{2}.$ $$

Then for $I_{2}$, we have

$$ $\displaystyle|I_{2}|$ $\displaystyle\leq n(\max\limits_{i\in[n]}\theta^{2}_{i}-\min\limits_{i\in[n]}\theta^{\prime 2}_{i})\cdot(1+\kappa)d$ $\displaystyle\leq\Bigg{(}\frac{1}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})^{2}}-\frac{1}{(1+\kappa)^{2}d^{2}}\cdot\bigg{(}\alpha-\frac{2n\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}^{2}\Bigg{)}\cdot(1+\kappa)dn$ $\displaystyle=d(1+\kappa)n\cdot\frac{1-\frac{1}{(1+\kappa)^{2}d^{2}}((1-\kappa)d\alpha-2(\alpha+1)n\sqrt{d\log(6n^{2}/\delta)})^{2}}{((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})^{2}}$ $\displaystyle=O\bigg{(}\frac{n}{d}\bigg{)}.$ $$

The second inequality is from Lemma [47](#Thmtheorem47) and Lemma [51](#Thmtheorem51).

Then we bound $|-I_{3}+I_{4}|$ as:

$$ $\displaystyle|-I_{3}+I_{4}|\leq\sum\limits_{i\in[n]}\sum\limits_{j\in[n]\backslash\{i\}}(\theta^{\prime}_{i}\theta^{\prime}_{j}-\theta_{i}\theta_{j})\cdot|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|$ $\displaystyle\leq(n)^{2}(\max\limits_{i\in[n]}\theta^{\prime 2}_{i}-\min\limits_{i\in[n]}\theta^{2}_{i})\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle\leq(n)^{2}\bigg{[}\bigg{(}\frac{1}{(1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}^{2}-\bigg{(}\frac{(1-\kappa)d-4n\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d((1-\kappa)d-2n\sqrt{d\log(6n^{2}/\delta)})}\bigg{)}^{2}\bigg{]}\cdot 2\sqrt{d\log(6n^{2}/\delta)}$ $\displaystyle=\widetilde{O}\bigg{(}\frac{\kappa n^{2}}{d^{3/2}}\bigg{)}=O\bigg{(}\frac{n^{2}}{d^{2}}\bigg{)}.$ $$

The third inequality is from Lemma [47](#Thmtheorem47) and [51](#Thmtheorem51); The last two equalities are from Assumption [9](#Thmtheorem9).
Combining the above results, we have

$$ $\displaystyle\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\frac{C}{\rho^{2}}+O\bigg{(}\frac{n}{d}\bigg{)}\geq\frac{C_{2}(1-\beta)}{\rho^{2}}.$ $$

Here $C_{2}=\Theta(1)$ is a constant.
∎

Therefore, combining Lemma [50](#Thmtheorem50) and [52](#Thmtheorem52), we have the following statement for the difference between $\|\bm{v}^{\prime}\|$ and $\|\bm{v}_{mm}\|$:

$$ $\|\bm{v}^{\prime}\|_{2}^{2}-\|\bm{v}_{mm}\|_{2}^{2}\geq\frac{C_{3}(1-\beta)}{d}.$ (A.64) $$

Here $C_{3}=\Theta(1)$ is a constant. The inequality is from the SNR condition that $\rho=o(\sqrt{d/n})$.

Now we can prove the main proposition in this scenario.

###### Proof of Proposition 46 in case 1.

From ([A.64](#A1.E64)) we have

$$ $\displaystyle\|\bm{v}^{\prime\prime}\|_{2}^{2}-\|\bm{v}\|_{2}^{2}$ $\displaystyle\geq\frac{C_{3}(1-\beta)}{d}=S(1-\beta)$ $$

Here we substitute $S=\frac{C_{3}}{d}\geq 0$ Then we have

$$ $\displaystyle\Gamma^{2}-\Gamma^{\prime 2}$ $\displaystyle=\frac{1}{\|\bm{v}\|^{2}}-\frac{1}{\|\bm{v}^{\prime}\|^{2}}=\frac{\|\bm{v}^{\prime}\|^{2}-\|\bm{v}\|^{2}}{\|\bm{v}^{\prime}\|^{2}\cdot\|\bm{v}\|^{2}}\geq\frac{S(1-\beta)}{\|\bm{v}^{\prime}\|^{2}\cdot\|\bm{v}\|^{2}}.$ $$

Therefore,

$$ $\displaystyle\Gamma-\Gamma^{\prime}$ $\displaystyle\geq\frac{S(1-\beta)}{(\Gamma+\Gamma^{\prime})\|\bm{v}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}\geq\frac{S(1-\beta)}{2\Gamma\|\bm{v}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}.$ $$

Set $c=\frac{S}{2\Gamma\|\bm{v}\|^{2}\cdot\|\bm{v}^{\prime}\|^{2}}=\frac{S}{2\|\bm{v}\|\|\bm{v}^{\prime}\|^{2}}$, we have $\Gamma^{\prime}\leq\Gamma-c(1-\beta)$. And we can
upper bound $c$ as

$$ $\displaystyle c$ $\displaystyle=\frac{S}{2\|\bm{v}\|\|\bm{v}^{\prime}\|^{2}}\leq\frac{S}{r^{3}_{mm}}\leq\frac{C_{3}}{r^{3}_{mm}d}.$ $$

The first inequality is from $\|\bm{v}^{\prime}\|\geq\|\bm{v}\|$ and the second equality is from $S=\frac{C_{2}}{d}$.

∎

Situation 2: $p=0,k-p\neq 0$

Then we consider the case when all wrong token selections come from noisy set. Same as above,
denote the mixed samples as $k_{1},k_{2},...,k_{k-p}$. And for every mixed sample $k_{i}$, we have $\bm{r}_{k_{i}}=(1-\beta_{i})\bm{\mu}_{k_{i}}+\beta_{i}\bm{\xi}_{k_{i}}$. Without losing generality, we assume that $y_{k_{i}}=+1$ for all $i\in[k-p]$, so the corresponding signal token is $\bm{\mu}_{2}$. Then the conditions under Situation 2 become

###### Condition 16 (Change k-p noisy samples) .

$$ $\begin{cases}&y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in[n]\backslash[k-p]\\ &\bm{v}^{\top}\bm{r}_{k_{i}}\geq 1,i\in[k-p]\end{cases}$ $$

Denote the max-margin solution under this condition as $\bm{v}^{\prime}$ with parameters $\lambda^{\prime}_{1},\lambda^{\prime}_{2},\theta^{\prime}_{i}$, we can interpret the condition for parameters:

$$ $\begin{cases}&\theta_{i}^{\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle\geq 1,i\in[n]\backslash[k-p]\\ &(1-\beta_{i})\lambda_{2}^{\prime}\cdot\|\bm{\mu}_{2}\|^{2}+\beta_{i}(\theta^{\prime}_{k_{i}}\cdot\|\bm{\xi}_{k_{i}}\|^{2}+\sum\limits_{i^{\prime}\neq k_{i}}y_{k_{i}}y_{i^{\prime}}\theta^{\prime}_{i^{\prime}}\langle\bm{\xi}_{k_{i}},\bm{\xi}_{i^{\prime}}\rangle)\geq 1,i\in[k-p]\end{cases}$ $$

Compare with Codition [13](#Thmcondition13), the only difference is that we substitute $\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}$ with $\lambda^{\prime}_{2}\|\bm{\mu}_{2}\|^{2}$. From the symmetry, we can see that the two conditions are actually the same. Thereofre, we can follow the proof of Situation 1 to prove for Proposition [46](#Thmtheorem46) under this situation.

Situation 3: $p\neq 0,k-p\neq 0$

Last we consider the case when wrong tokens come from both clean and noisy sets. Denote the mixed clean samples as $k_{1},k_{2},...,k_{p}$ and the mixed noisy samples as $q_{1},q_{2},...,q_{k-p}$.Without losing generality, we assume that $y_{k_{i}}=+1$ for $i\in[p]$ and $y_{q_{i}}=-1$ for $i\in[k-p]$, which indicates that their signal tokens are all $\bm{\mu}_{1}$. Then the conditions under Situation 2 become

###### Condition 17 (p clean samples and k-p noisy samples violating optimal token selection) .

$$ $\begin{cases}&y_{i}\bm{v}^{\top}\bm{\xi}_{i}\geq 1,i\in[n]\backslash[k]\\ &\bm{v}^{\top}\bm{r}_{k_{i}}\geq 1,i\in[p]\\ &-\bm{v}^{\top}\bm{r}_{q_{i}}\geq 1,i\in[k-p]\end{cases}$ $$

Denote the max-margin solution under this condition as $\bm{v}^{\prime\prime}$ with parameters $\lambda^{\prime\prime}_{1},\lambda^{\prime\prime}_{2},\theta^{\prime\prime}_{i}$, we can interpret the condition for parameters:

$$ $\begin{cases}&\theta_{i}^{\prime\prime}\cdot\|\bm{\xi}_{i^{\prime}}\|^{2}+\sum\limits_{i^{\prime}\neq i}y_{i}y_{i^{\prime}}\theta^{\prime\prime}_{i^{\prime}}\langle\bm{\xi}_{i},\bm{\xi}_{i^{\prime}}\rangle)\geq 1,i\in[n]\backslash[k]\\ &(1-\beta_{i})\lambda^{\prime\prime}_{1}\cdot\|\bm{\mu}_{1}\|^{2}+\beta_{i}(\theta^{\prime\prime}_{k_{i}}\cdot\|\bm{\xi}_{k_{i}}\|^{2}+\sum\limits_{i^{\prime}\neq k_{i}}y_{k_{i}}y_{i^{\prime}}\theta^{\prime\prime}_{i^{\prime}}\langle\bm{\xi}_{k_{i}},\bm{\xi}_{i^{\prime}}\rangle)\geq 1,i\in[p]\\ &-(1-\beta_{i})\lambda^{\prime\prime}_{1}\cdot\|\bm{\mu}_{1}\|^{2}-\beta_{i}(\theta^{\prime\prime}_{q_{i}}\cdot\|\bm{\xi}_{q_{i}}\|^{2}+\sum\limits_{i^{\prime}\neq q_{i}}y_{q_{i}}y_{i^{\prime}}\theta^{\prime\prime}_{i^{\prime}}\langle\bm{\xi}_{q_{i}},\bm{\xi}_{i^{\prime}}\rangle)\geq 1,i\in[k-p]\end{cases}$ $$

We consider three cases: $\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1$, $1>\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq-1$ and $\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}<-1$.

- •
$\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq 1$
First when $\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\||^{2}\geq 1$, we have $\frac{1-(1-\beta_{i})\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta_{i}}\leq 1$, which indicates that the condition for mixed clean samples’ parameter $\theta^{\prime}_{k_{i}}$ is relaxed. Meanwhile, for the mixed noisy samples we have
$\displaystyle-\theta^{\prime\prime}_{q_{i}}\cdot\|\bm{\xi}_{q_{i}}\|^{2}+\sum\limits_{i^{\prime}\neq q_{i}}y_{q_{i}}y_{i^{\prime}}\theta^{\prime\prime}_{i^{\prime}}\langle\bm{\xi}_{q_{i}},\bm{\xi}_{i^{\prime}}\rangle\geq\frac{1+(1-\beta_{i})\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}}{\beta_{i}}\geq 1,$
which indicates that the condition is strengthened. Therefore, this case is an extension of the second case of Situation 1 with strengthening some constraints. These constraints will not result in a better solution than Situation 1. The following proof is the same as Situation 1 and we omit it for convenience.
- •
$1>\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}\geq-1$
In this case, the constraints for both mixed clean and noisy samples are strengthened. So this can be taken as an extension of the first case in Situation 1 with strengthening some constraints. The following proof is the same as Situation 1 and we omit it for convenience.
- •
$\lambda^{\prime\prime}_{1}\|\bm{\mu}_{1}\|^{2}<-1$
In this case, the constraints are strengthened for mixed clean samples while relaxed for the mixed noisy samples. So we consider it as the extension of Situation 2 when $\lambda^{\prime}_{1}\|\bm{\mu}_{1}\|^{2}<-1$ with strengthening some constraints. The following proof is the same as Situation 2 and we omit it for convenience.

Therefore, we complete the proof for all possible situations.
∎

###### Proof of Proposition 46 .

###### Condition 12 (Original Condition) .

###### Lemma 47 (Balanceing noise factor for KKT point) .

###### Proof of Lemma 47 .

###### Lemma 48 (Norm of 𝒗 m ​ m subscript 𝒗 𝑚 𝑚 \bm{v}_{mm} ) .

###### Proof of Lemma 48 .

###### Condition 13 ( p 𝑝 p clean samples violating optimal token selection) .

###### Condition 14 (1 clean sample violating optimal token selection) .

###### Lemma 49 .

###### Proof of Lemma 49 .

###### Lemma 50 .

###### Proof of Lemma 50 .

###### Condition 15 (All clean samples violating optimal token selection) .

###### Lemma 51 .

###### Proof of Lemma 51 .

###### Lemma 52 .

###### Proof of Lemma 52 .

###### Proof of Proposition 46 in case 1.

###### Condition 16 (Change k-p noisy samples) .

###### Condition 17 (p clean samples and k-p noisy samples violating optimal token selection) .

##### Training and Test error analysis

From Proposition [46](#Thmtheorem46) we can derive the convergence direction of $\bm{p}$ and $\bm{v}$, i.e. $\bm{p}_{mm}$ and $\bm{v}_{mm}$.
Note that Theorem [16](#Thmtheorem16) does not depend on the selection of optimal tokens, so it still holds in this case when optimal tokens are noise tokens for all samples. We restate it here for convenience:

###### Theorem 53 .

Suppose that Assumption [9](#Thmtheorem9) holds, with probability at least $1-\delta$ on the training dataset, we have

- •
the margin induced by $\bm{p}_{R}/R$ in p-SVM is at least $(1-\zeta)\Xi$, where
$\zeta=\frac{\log(4\sqrt{(1+\kappa)d}\|\bm{v}_{mm}\|^{3}d\rho^{2})}{R\Xi}.$
- •
the label margin induced by $\bm{v}_{r}/r$ in v-SVM is at least $(1-\gamma)\Gamma$, where $\gamma=\frac{2\sqrt{(1+\kappa)d}}{\Gamma\exp((1-\zeta)R\Xi)}$.

Then we could estimate the test error in this case. From Theorem [53](#Thmtheorem53) we have

$$ $\displaystyle\bm{p}_{R}^{\top}(\bm{\xi}_{i}-\bm{\mu}_{i})\geq(1-\zeta)R\Xi,\forall i\in[n]$ (A.65) $$

$$ $\displaystyle y_{i}\bm{v}_{r}^{\top}\bm{\xi}_{i}\geq(1-\gamma)\Gamma r,\forall i\in[n].$ (A.66) $$

Here $\zeta,\gamma,\Xi,\Gamma$ are the same as the definition in Theorem [53](#Thmtheorem53). Similarly, we have the following lemma for $\zeta,\gamma$.

###### Lemma 54 .

Suppose that Assumption [9](#Thmtheorem9) holds, with probability at least $1-\delta$ on the training dataset, consider the same setting in Theorem [16](#Thmtheorem16), we have $\zeta<0.2$ and $\gamma<1$.

###### Proof of Lemma 54 .

First we upper bound $\|\bm{p}_{mm}\|$. Consider the following possible solution $\widetilde{\bm{p}}$:

$$ $\widetilde{\bm{p}}=\sum\limits_{i\in[n]}2\frac{\bm{\xi}_{i}}{d}.$ (A.67) $$

We then proved that $\widetilde{\bm{p}}$ satisfies ([A.57](#A1.E57)). For $\forall k\in[n]$, we have

$$ $\displaystyle\widetilde{\bm{p}}^{\top}(\bm{\xi}_{k}-\bm{\mu}_{k})$ $\displaystyle=\sum\limits_{i\in[n]}2\frac{\langle\bm{\xi}_{i},\bm{\xi}_{k}\rangle}{d}\geq 2(1-\kappa)+\sum\limits_{i\in[n],i\neq k}2\frac{\langle\bm{\xi}_{i},\bm{\xi}_{k}\rangle}{d}$ $\displaystyle\geq 2(1-\kappa)+\frac{2n\sqrt{d\log(6n^{2}/\delta)}}{d}\geq 1.$ $$

The first and second inequalities are from Lemma [57](#Thmtheorem57); The last inequality is from Assumption [9](#Thmtheorem9).

Therefore, the max-margin solution $\bm{p}_{mm}$ must have no greater norm than $\widetilde{\bm{p}}$. So we can upper bound $\bm{p}_{mm}$ as

$$ $\displaystyle\|\bm{p}_{mm}\|^{2}$ $\displaystyle\leq\|\widetilde{\bm{p}}\|^{2}=\frac{4}{d^{2}}\Big{(}\sum\limits_{i\in[n]}\|\bm{\xi}_{i}\|^{2}+\sum\limits_{i,j\in[n],i\neq j}\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle\Big{)}$ $\displaystyle\leq\frac{4}{d^{2}}\big{(}(1+\kappa)nd+2n^{2}\sqrt{d\log(6n^{2}/\delta)}\big{)}\leq\frac{5n}{d}.$ $$

The second inequality is from Lemma [57](#Thmtheorem57); The last inequality is from the definition of $d$ in Assumption [9](#Thmtheorem9).

Then from the definition of $\zeta$ in Theorem [16](#Thmtheorem16), we have

$$ $\displaystyle\zeta$ $\displaystyle=\frac{\log(4\sqrt{(1+\kappa)d}\|\bm{v}_{mm}\|^{3}d\rho^{2})}{R\Xi}\leq C_{1}\frac{\sqrt{n/d}}{R}\log(4\sqrt{(1+\kappa)d}\|\bm{v}_{mm}\|^{3}d\rho^{2})$ $\displaystyle\leq C_{2}\frac{\sqrt{n/d}}{R}\log\bigg{(}\frac{n^{3}}{d}\bigg{)}<0.2.$ $$

Here $C_{1},C_{2}=\Theta(1)$. The first inequality is from $\Xi^{-1}=\|\bm{p}_{mm}\|\leq\sqrt{5n/d}$; The second inequality is from the upper bound of $\|\bm{v}_{mm}\|$ in Lemma [48](#Thmtheorem48) and the last inequality is from the definition of $R$ in Assumption [9](#Thmtheorem9).
And for $\gamma$, we have

$$ $\displaystyle\gamma$ $\displaystyle=\frac{2M}{\Gamma\exp((1-\zeta)R\Xi)}=C^{\prime}_{1}\frac{M\|\bm{v}_{mm}\|}{\exp(R/\|\bm{v}_{mm}\|)}\leq C^{\prime}_{2}\frac{\sqrt{d\cdot(n/d)}}{\exp(R/\sqrt{n/d})}<1.$ $$

Here $C^{\prime}_{1},C^{\prime}_{2}=\Theta(1)$. The first inequality is from the lower and upper bound of $\|\bm{v}_{mm}\|$ in Lemma [28](#Thmtheorem28) and the last inequality is from the definition of $R$ in Assumption [5](#Thmtheorem5).
∎

Then we have the following lemma to estimate the innerproduct of $\bm{p}_{R}$ and signal token:

###### Lemma 55 .

Suppose that Assumption [9](#Thmtheorem9) holds, with probability at least $1-\delta$ on the training dataset, we have

$$ $|\langle\bm{p}_{R},\bm{\mu}_{j}\rangle|\leq 0.9(1-\zeta)R\xi$ $$

for $j\in\{1,2\}$.

###### Proof of Lemma 55 .

First we use contradiction to prove for the lower bound. Assume that $|\langle\bm{p}_{R},\bm{\mu}_{j}\rangle|>0.9(1-\zeta)R\Xi$. We can estimate $\|\bm{p}_{R}\|$ as

$$ $\displaystyle\|\bm{p}_{R}\|^{2}$ $\displaystyle>(0.9(1-\zeta)R\Xi)^{2}/\rho^{2}>(0.5\Xi^{2}/\rho^{2})\cdot R^{2}\geq(0.1d/n\rho^{2})\cdot R^{2}>R^{2}.$ $$

The second inequality is from Lemma [54](#Thmtheorem54) ; The third inequality is from $\Xi^{2}=\|\bm{p}_{mm}\|^{-2}\geq d/(5n)$; The last inequality is from our SNR condition $\rho=o(\sqrt{d/n})$. This leads to a contradiction.

∎

From Lemma [41](#Thmtheorem41), we can denote $\bm{v}_{r}$ as

$$ $\displaystyle\bm{v}_{r}=\lambda_{1}\bm{\mu}_{1}+\lambda_{2}\bm{\mu}_{2}+\sum_{i\in[n]}y_{i}\theta_{i}\bm{\xi}_{i}.$ $$

Denote $\bm{v}_{\bm{\xi}}=\sum_{i\in[n]}y_{i}\theta_{i}\bm{\xi}_{i}$ as the noise part of $\bm{v}_{r}$. Then we prove that $\bm{p}_{R}$, $\bm{v}_{\bm{\xi}}$ are near orthogonal

###### Lemma 56 .

Suppose that Assumption [9](#Thmtheorem9) holds, with probability at least $1-\delta$ on the training dataset, we have

$$ $\displaystyle|\langle\bm{p}_{R},\bm{v}_{\bm{\xi}}\rangle|\leq c$ $$

for some constant $c\in(0,1)$.

###### Proof of Lemma 56 .

First plugging in the parameters in $\bm{v}_{\bm{\xi}}$ we have

$$ $\displaystyle\langle\bm{p}_{R},\bm{v}_{\bm{\xi}}\rangle$ $\displaystyle=\sum_{i\in[n]}y_{i}\theta_{i}\bm{p}^{\top}_{R}\bm{\xi}_{i}$ $\displaystyle=\sum_{y_{i}=+1}\theta_{i}\bm{p}^{\top}_{R}\bm{\xi}_{i}-\sum_{y_{i}=-1}\theta_{i}\bm{p}^{\top}_{R}\bm{\xi}_{i}$ $\displaystyle\leq(n_{11}+n_{21})(\max_{i}\theta_{i})(R\Xi+O(R\rho))-(n_{12}+n_{22})(\min_{i}\theta_{i})((1-\zeta)R\Xi-O(R\rho))$ $\displaystyle\leq\underbrace{(n/2)(\max_{i}\theta_{i}-\min_{i}\theta_{i})R\Xi}_{I_{1}}+\underbrace{O(\sqrt{n})(\max_{i}\theta_{i})R\Xi}_{I_{2}}+\underbrace{n(\max_{i}\theta_{i})(\zeta R\Xi+O(R\rho))}_{I_{3}}.$ $$

The first inequality is from Theorem [53](#Thmtheorem53) that $(1-\zeta)R\Xi\leq\bm{p}^{\top}_{R}(\bm{\xi}_{i}-\bm{\mu}_{i})\leq R\Xi$ and $\bm{p}^{\top}_{R}\bm{\mu}_{i}=O(R\rho)$ and the second inequality is from Lemma [59](#Thmtheorem59). Then we bound $I_{1}\sim I_{3}$ respectively. For $I_{1}$, we need to first bound $\theta_{i}$. From Theorem [53](#Thmtheorem53) we have

$$ $\displaystyle(1-\gamma)\Gamma r\leq y_{i}\bm{v}^{\top}_{r}\bm{\xi}_{i}\leq\Gamma r,\forall i\in[n].$ $$

Denote $j=\mathop{\mathrm{argmax}}_{i}\theta_{i}$, we have

$$ $\displaystyle y_{j}\bm{v}^{\top}_{r}\bm{\xi}_{j}$ $\displaystyle\geq\theta_{j}\|\bm{\xi}_{j}\|^{2}+n\theta_{j}\sqrt{d\log(6n^{2}/\delta)}\geq\theta_{j}((1-\kappa)d+n\sqrt{d\log(6n^{2}/\delta)}).$ $$

Therefore, we can upper bound $\theta_{j}$ as

$$ $\displaystyle\theta_{j}\leq\frac{y_{j}\bm{v}^{\top}_{r}\bm{\xi}_{i}}{(1-\kappa)d+n\sqrt{d\log(6n^{2}/\delta)}}\leq\frac{\Gamma r}{(1-\kappa)d+n\sqrt{d\log(6n^{2}/\delta)}}.$ (A.68) $$

Then we can lower bound $\theta_{i}$ as

$$ $\displaystyle y_{i}\bm{v}^{\top}_{r}\bm{\xi}_{i}\leq\theta_{i}\|\bm{\xi}_{i}\|^{2}+n\theta_{j}\sqrt{d\log(6n^{2}/\delta)}\leq(1+\kappa)d\theta_{i}+\frac{\Gamma rn\sqrt{d\log(6n^{2}/\delta)}}{(1-\kappa)d+n\sqrt{d\log(6n^{2}/\delta)}}.$ $$

Therefore,

$$ $\displaystyle\theta_{i}\geq\frac{(1-\gamma)(1-\kappa)\Gamma rd-\gamma\Gamma rn\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d(1-\kappa)d+n\sqrt{d\log(6n^{2}/\delta)}}.$ $$

So we can estimate $I_{1}$ as

$$ $\displaystyle I_{1}$ $\displaystyle\leq(nR\Xi/2)\cdot\bigg{(}\frac{\Gamma r}{(1-\kappa)d+n\sqrt{d\log(6n^{2}/\delta)}}-\frac{(1-\gamma)(1-\kappa)\Gamma rd-\gamma\Gamma rn\sqrt{d\log(6n^{2}/\delta)}}{(1+\kappa)d(1-\kappa)d+n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}$ $\displaystyle\leq R\sqrt{nd}/2\cdot\Gamma r\cdot\bigg{(}\frac{1-\frac{(1-\gamma)(1-\kappa)}{1+\kappa}+\frac{\gamma n\log(6n^{2}/\delta)}{(1+\kappa)d}}{(1-\kappa)d+n\sqrt{d\log(6n^{2}/\delta)}}\bigg{)}$ $\displaystyle\leq Rr(\kappa+\gamma).$ $$

The second inequality is from $\Xi=\|\bm{p}_{mm}\|=\Theta(\sqrt{d/n})$ and the last inequality is from $\Gamma=\|\bm{v}_{mm}\|^{-1}=\Theta(\sqrt{d/n})$.

Then we bound $I_{2}$. From ([A.68](#A1.E68)) we have $\max_{i}\theta_{i}=\Theta(\Gamma r/d)$. Therefore,

$$ $\displaystyle I_{2}$ $\displaystyle\leq O(\sqrt{n})\Theta(\Gamma r/d)R\Xi\leq Rr\cdot O(1/\sqrt{n}).$ $$

The last inequality is from $\Gamma,\Xi=\Theta(\sqrt{d/n})$.

Last we bound $I_{3}$ as

$$ $\displaystyle I_{3}$ $\displaystyle=n\Theta(\Gamma r/d)(\zeta R\Xi+O(R\rho))$ $\displaystyle\leq\Theta(r\sqrt{n/d})(\log(4\sqrt{(1+\kappa)d}\|\bm{v}_{mm}\|^{3}d\rho^{2})+O(R\rho))$ $\displaystyle\leq Rr\cdot O(\rho\sqrt{n/d}).$ $$

The first inequality is from $\Gamma,\Xi=\Theta(\sqrt{d/n})$ and the last inequality is from Assumption [9](#Thmtheorem9).

Combining the results above, we have

$$ $\langle\bm{p}_{R},\bm{v}_{\bm{\xi}}\rangle\leq I_{1}+I_{2}+I_{3}\leq Rr\cdot O(\sqrt{1/n}+\rho\sqrt{n/d})\leq c$ $$

for sufficiently large $d$ and $n$. Here the last inequality comes from Assumption [9](#Thmtheorem9).
∎

With the lemmas above, we could prove for the main theorem

###### Proof of Theorem 10 .

First we show that the model can perfectly classify all training samples. From Theorem [16](#Thmtheorem16), we have

$$ $\displaystyle y_{i}\bm{v}^{\top}_{r}\bm{r}_{i}=y_{i}\beta_{i}\bm{v}^{\top}_{r}\bm{\xi}_{i}+y_{i}(1-\beta_{i})\bm{v}^{\top}_{r}\bm{\mu}_{i}\geq\beta_{i}(1-\gamma)\Gamma r-0.9(1-\beta_{i})(1-\gamma)\Gamma r>0,$ $$

for $\forall i\in[n]$. The last inequality is from Lemma [54](#Thmtheorem54). Thus $y_{i}=\mathrm{sign}(f(\bm{X}_{i};\bm{p}_{R},\bm{v}_{r}))$ for all $i\in[n]$.

Then we bound the test error. This is equivalent to estimate $y\cdot f(\bm{p}_{R},\bm{v}_{r};\bm{X})$ and we could write it as

$$ $\displaystyle y\cdot f(\bm{p}_{R},\bm{v}_{r};\bm{X})$ $\displaystyle=y\cdot\frac{\exp(\langle\bm{p}_{R},\bm{\mu}^{\prime}\rangle)\bm{v}_{r}^{\top}\bm{\mu}^{\prime}+\exp(\langle\bm{p}_{R},\bm{\xi}^{\prime}\rangle)\bm{v}_{r}^{\top}\bm{\xi}^{\prime}}{\exp(\langle\bm{p}_{R},\bm{\mu}^{\prime}\rangle)+\exp(\langle\bm{p}_{R},\bm{\xi}^{\prime}\rangle)}.$ $$

We first upper bound the term $y\cdot\exp(\langle\bm{p}_{R},\bm{\mu}^{\prime}\rangle)\bm{v}^{\top}_{r}\bm{\mu}^{\prime}$. From Theorem [53](#Thmtheorem53), the non-optimality of $i$-th sample is

$$ $\displaystyle 1-\beta_{i}=\frac{\exp(\langle\bm{p}_{R},\bm{\mu}_{i}\rangle)}{\exp(\langle\bm{p}_{R},\bm{\mu}_{i}\rangle)+\exp(\langle\bm{p}_{R},\bm{\xi}_{i}\rangle)}\leq\frac{1}{1+\exp((1-\zeta)\Xi R)}\text{ for all }i\in[n].$ $$

The last inequality is from the first statement in Theorem [53](#Thmtheorem53). Consider the sample that contains the same signal token as $\bm{\mu}^{\prime}$, we have

$$ $\displaystyle(1-\beta_{i})\bm{v}^{\top}_{r}\bm{\mu}_{i}=\frac{\exp(\langle\bm{p}_{R},\bm{\mu}_{i}\rangle)\bm{v}^{\top}_{r}\bm{\mu}_{i}}{\exp(\langle\bm{p}_{R},\bm{\mu}_{i}\rangle)+\exp(\langle\bm{p}_{R},\bm{\xi}_{i}\rangle)}.$ $$

Therefore,

$$ $\displaystyle y\cdot\exp(\langle\bm{p}_{R},\bm{\mu}^{\prime}\rangle)\bm{v}^{\top}_{r}\bm{\mu}^{\prime}$ $\displaystyle\leq\exp(\langle\bm{p}_{R},\bm{\mu}_{i}\rangle)|\bm{v}^{\top}_{r}\bm{\mu}_{i}|\leq\frac{\exp(\langle\bm{p}_{R},\bm{\mu}_{i}\rangle)+\exp(\langle\bm{p}_{R},\bm{\xi}_{i}\rangle)}{1+\exp((1-\zeta)\Xi R)}\cdot|\bm{v}^{\top}_{r}\bm{\mu}_{i}|$ $\displaystyle\leq\frac{2\exp(\langle\bm{p}_{R},\bm{\xi}_{i}\rangle)}{\exp((1-\zeta)\Xi R)}\cdot|\bm{v}^{\top}_{r}\bm{\mu}_{i}|\leq\frac{2\exp(\Xi R)}{\exp((1-\zeta)\Xi R)}\cdot|\bm{v}^{\top}_{r}\bm{\mu}_{i}|$ $\displaystyle\leq 2\exp(\zeta\Xi R)\cdot\rho r=(4\sqrt{(1+\kappa)d}\|\bm{v}_{mm}\|^{3}d\rho^{2})\cdot\rho r\leq Cn^{3/2}\rho^{3}r$ (A.69) $$

for some constant $C>0$.
Here the third inequality is from $\bm{p}^{\top}_{R}(\bm{\xi}_{i}-\bm{\mu}_{i})\geq 0$; The fourth inequality is from the fact that $\langle\bm{p}_{R},\bm{\xi}_{i}\rangle\leq\Xi R$ and the last inequality is from $\|\bm{v}_{r}\|\leq r,\|\bm{\mu}_{i}\|\leq\rho$.
Then we can bound the test error as

$$ $\displaystyle\mathbb{P}(y\cdot f(\bm{p}_{R},\bm{v}_{r};\bm{X})\leq 0)$ $\displaystyle=\mathbb{P}(y\cdot\exp(\langle\bm{p}_{R},\bm{\mu}^{\prime}\rangle)\bm{v}_{r}^{\top}\bm{\mu}^{\prime}+y\cdot\exp(\langle\bm{p}_{R},\bm{\xi}^{\prime}\rangle)\bm{v}_{r}^{\top}\bm{\xi}^{\prime}\leq 0)$ $\displaystyle\geq\mathbb{P}(y\cdot\exp(\langle\bm{p}_{R},\bm{\xi}^{\prime}\rangle)\bm{v}_{r}^{\top}\bm{\xi}^{\prime}\leq-Cn^{3/2}\rho^{3}r)$ $\displaystyle\geq\frac{1}{4}\mathbb{P}\bigg{(}y\bm{v}_{\bm{\xi}}^{\top}\bm{\xi}^{\prime}\leq-e^{-R/C}\cdot Cn^{3/2}\rho^{3}r\mid\langle\bm{p}_{R}/R,\bm{\xi}^{\prime}\rangle\in[1/C,C]\bigg{)}$ $\displaystyle\geq\frac{1}{4}(\frac{1}{2}-\frac{cC+C\exp(-R/C)n^{3/2}\rho^{3}}{\sqrt{2\pi(1-c^{2})}})\geq\frac{1}{16}.$ $$

The first inequality is from ([A.69](#A1.E69)); the second inequality use the fact that there exists a constant $C>0$ such that $\mathbb{P}(N(0,1)\in[1/C,C])\geq 1/4$; the third inequality comes from Lemma [60](#Thmtheorem60) and the last inequality uses Assumption [9](#Thmtheorem9).
∎

.

###### Theorem 53 .

###### Lemma 54 .

###### Proof of Lemma 54 .

###### Lemma 55 .

###### Proof of Lemma 55 .

###### Lemma 56 .

###### Proof of Lemma 56 .

###### Proof of Theorem 10 .

### A.3 Supplement Lemmas

Here we list some technical lemmas for the main proof.

###### Lemma 57 .

(Properties of Training Data)
Suppose that $\delta>0$ and $\kappa=O(\sqrt{\log(6n/\delta)/d})=\widetilde{O}(1/\sqrt{d})$ .Then with probability at least $1-\delta$, we have

$$ $\displaystyle(1-\kappa)d\leq\|\bm{\xi}_{i}\|_{2}^{2}\leq(1+\kappa)d$ $\displaystyle|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|\leq 2\sqrt{d\log(6n^{2}/\delta)}$ $$

for any $i,j\in[n]$.

###### Proof of Lemma 57 .

By Bernstein’s inequality (see Theorem 2.8.1 in []), with probability at least $1-\delta/(3n)$ we have

$$ $\displaystyle|\|\bm{\xi}_{i}\|_{2}^{2}-d|=O(\sqrt{d\log(6n/\delta)}).$ $$

Therefore, there exists $\kappa=O(\sqrt{\log(6n/\delta)/d})$ that

$$ $\displaystyle(1-\kappa)d\leq\|\bm{\xi}_{i}\|_{2}^{2}\leq(1+\kappa)d.$ $$

Moreover, $\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle$ has mean zero. For any $i,j\in[n]$ and $i\neq j$, by Bernstein’s inequality, with probability at least $1-\delta/(3n^{2})$ we have

$$ $\displaystyle|\langle\bm{\xi}_{i},\bm{\xi}_{j}\rangle|\leq 2\sqrt{d\log(6n^{2}/\delta)}.$ $$

Applying a union bound completes the proof.
∎

Set $\delta=6n\exp(-d/4C_{1}n^{2})$ for any constant $C_{1}>0$, we can follow the proof of Lemma [57](#Thmtheorem57) and conclude the next remark:

###### Remark 58 .

(Properties of New Test Sample)
Let $(\bm{X}=(\bm{\mu}_{k},\bm{\xi}),y)\sim\mathcal{D}$. Then with probability at least $1-6n\exp(-d/4C_{1}n^{2})$, we have

$$ $\displaystyle|\langle\bm{\xi},\bm{\xi}_{i}\rangle|\leq\frac{d}{C_{1}n}$ $$

for any $i\in[n]$.

###### Lemma 59 .

With probability at least $1-6\delta$,

$$ $\big{|}|\mathcal{C}|-n(1-\eta)\big{|}\leq\sqrt{n\log(\frac{1}{\delta})};\quad\big{|}|\mathcal{N}|-n\eta\big{|}\leq\sqrt{n\log(\frac{1}{\delta})};$ $$

$$ $\big{|}|\mathcal{C}_{i}|-\frac{n(1-\eta)}{2}\big{|}\leq\sqrt{n\log(\frac{1}{\delta})};\quad\big{|}|\mathcal{N}_{i}|-\frac{n\eta}{2}\big{|}\leq\sqrt{n\log(\frac{1}{\delta})},\quad i=1,2.$ $$

###### Proof.

Note that $|\mathcal{C}|\sim\text{Binom}(n,1-\eta)$. Applying Hoeffding’s inequality, we have

$$ $\mathbb{P}\big{(}\big{|}|\mathcal{C}|-(1-\eta)n\big{|}>t\big{)}\leq 2\exp(-\frac{2t^{2}}{n}).$ $$

Let $t=\sqrt{n\log(1/\delta)}$. We have that with probability at least $1-\delta$,

$$ $\big{|}|\mathcal{C}|-(1-\eta)n\big{|}\leq\sqrt{n\log(\frac{1}{\delta}}).$ $$

Similarly, note that $|\mathcal{N}|\sim\text{Binom}(n,\eta),|\mathcal{C}_{1}|\sim\text{Binom}(n,(1-\eta)/2),|\mathcal{C}_{2}|\sim\text{Binom}(n,(1-\eta)/2),|\mathcal{N}_{1}|\sim\text{Binom}(n,\eta/2)$ and $|\mathcal{N}_{2}|\sim\text{Binom}(n,\eta/2)$, we have that each of the following events holds with probability at least $1-\delta$:

$$ $\big{|}|\mathcal{C}|-n(1-\eta)\big{|}\leq\sqrt{n\log(\frac{1}{\delta})};\quad\big{|}|\mathcal{N}|-n\eta\big{|}\leq\sqrt{n\log(\frac{1}{\delta})};$ $$

$$ $\big{|}|\mathcal{C}_{i}|-n(1-\eta)/2\big{|}\leq\sqrt{n\log(\frac{1}{\delta})},\quad i=1,2;$ $$

$$ $\big{|}|\mathcal{N}_{i}|-n\eta/2\big{|}\leq\sqrt{n\log(\frac{1}{\delta})},\quad i=1,2.$ $$

∎

###### Lemma 60 .

Suppose $X\sim N(0,\bm{I}_{d})$, and $\bm{v},\bm{p}\in\mathbb{R}^{d}$ are two vectors with $\|\bm{v}\|=\|\bm{p}\|=1,\bm{v}^{\top}\bm{p}\leq c$ for some constant $c\in(0,1)$. Given some constant $C>1$, for $z<0$,

$$ $\mathbb{P}(\bm{v}^{\top}X<z|\bm{p}^{\top}X\in[1/C,C])\geq\frac{1}{2}-\frac{1}{\sqrt{2\pi}}\frac{cC-z}{\sqrt{1-c^{2}}}.$ $$

###### Proof of Lemma 60 .

Denote $x_{v}=v^{\top}X\sim N(0,1),x_{p}=\bm{p}^{\top}X\sim N(0,1)$. Then we have $x_{v},x_{p}\sim\mathcal{N}(0,1)$. Denote the covariance between $x_{v},x_{p}$ by $c_{0}$, then we have

$$ $c_{0}=\text{Cov}(x_{v},x_{p})=\bm{v}^{\top}\text{Cov}(X)\bm{p}=\bm{v}^{\top}\bm{p}\leq c.$ $$

Note that

$$ $x_{v}\stackrel{{\scriptstyle d}}{{=}}c_{0}x_{p}+\sqrt{1-c_{0}^{2}}r,$ $$

where $r\sim N(0,1)$ is independent of $x_{p}$. It follows that

$$ $\mathbb{P}(x_{v}<z|x_{p}\in[\frac{1}{C},C])=\mathbb{P}(r<\frac{z-c_{0}x_{p}}{\sqrt{1-c_{0}^{2}}}|x_{p}\in[\frac{1}{C},C])\geq\mathbb{P}(r<\frac{z-cC}{\sqrt{1-c^{2}}})\geq\frac{1}{2}-\frac{1}{\sqrt{2\pi}}\frac{cC-z}{\sqrt{1-c^{2}}}.$ $$

∎

###### Lemma 57 .

###### Proof of Lemma 57 .

###### Remark 58 .

###### Lemma 59 .

###### Proof.

###### Lemma 60 .

###### Proof of Lemma 60 .

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: It seems that this paper primarily combines ideas from two prior works, "Benign Overfitting in Two-layer Convolutional Neural Networks" (Cao et al., NeurIPS 2024) and "Max-Margin Token Selection in Attention Mechanism" (Ataee Tarzanagh et al., NeurIPS 2024). By relying heavily on these works, it lacks novelty and inherits their weaknesses without addressing them. 
Additionally, it seems that the weaknesses in these individual studies interact in a way that undermines each other's strengths.

Q1. Can the analysis extend to multiple tokens? Also, is it possible to relax the strong signal-noise separability assumptions that have been identified as limitations in existing studies?
In Cao et al.'s CNN work, only two patches are considered, which has been criticized in OpenReview for oversimplifying the signal and noise separation. Similarly, this paper assumes only two tokens and retains assumptions of orthogonality and separability between signal and noise, as in Cao et al.'s paper. However, in an attention model, especially one focused on token selection, analyzing only two tokens is even more problematic. For applications like Vision Transformers and sequence modeling, the ability to select among multiple tokens is fundamental to the attention mechanism, and overlooking this critical functionality limits the paper’s scope. While Ataee Tarzanagh et al. analyze max-margin solutions for multi-token scenarios, this paper fails to leverage those benefits by retaining the two-token simplification.

Q2. What is the essential difference between this work and benign overfitting in a linear model with input by linearly interpolating signal and noize vectors?
Due to the highly simplified setup, the analysis in this paper appears comparable to a straightforward two-vector linear interpolation. With two tokens—one signal and one noise—the role of attention is reduced to a linear interpolation between the two vectors. Since the signal is fixed and noise orthogonal, analyzing the behavior of the attention weights boils down to tracking a single-dimensional parameter representing this linear combination like lambda*singal+(1-lambda)*noize. As a result, the model resembles a simple two-class linear classification task using signal-noise interpolation. Thus, is this analysis merely a study of benign overfitting in a linear model? Given the extensive research on benign overfitting in linear models, it is unclear what essential differences this work offers from simpler linear models.

Q3. Is it feasible to analyze benign overfitting conditions under fully converged solutions?
Although Cao et al.'s CNN paper contributes by analyzing gradient descent dynamics, this paper only examines two-step gradient descent without exploring convergence behaviors in depth. While it does provide max-margin solution analysis, it lacks a general guarantee of convergence to this solution using gradient descent. This limitation stems from a similar issue in Ataee Tarzanagh et al.'s study, where convergence guarantees hold only for initializations close to the solution—a known weakness that seems inherited
soundness: 2
strengths: The paper provides the theoretical analysis of benign overfitting in attention mechanisms, expanding understanding beyond linear and CNN models. It also establishes conditions for benign overfitting.
summary: This paper studies benign overfitting in a single-head attention model, proving that under certain signal-to-noise ratio (SNR) conditions, the model achieves benign overfitting after two gradient descent steps or through a max-margin solution.
weaknesses: This paper relies on overly simplified settings and assumptions for its theoretical analysis. Additionally, its approach heavily depends on prior research, offering limited novelty and inheriting weaknesses from existing studies.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Would the results generalize to a more realistic model, such as self-attention and with more than two tokens?
- Can the authors discuss the challenges for analyzing further steps for GD instead of just two steps?
- Can the authors include some discussion on how the SNR requirement for benign overfitting for self-attention compares with other architectures, particularly CNNs, since similar data models have been considered for benign overfitting in CNNs?
soundness: 3
strengths: This work takes a step towards extending the understanding of benign overfitting from other neural network architectures to attention models.

Overall, the paper is well-written and easy to follow.
summary: This work examines the phenomenon of benign overfitting in a single-head attention model on a binary classification task with label noise. First, they show that two steps of GD, under signal-to-noise ratio (SNR) of $\Theta(1/\sqrt{n})$, exhibit benign overfitting. Next, they examine the max-margin/min-norm interpolator and show that the SNR requirement in that case is only $\Omega(1/\sqrt{n})$. Finally, they show that this is necessary (upto some constant factors), because for lower SNR, the max-margin interpolator exhibits harmful overfitting.
weaknesses: - The main weakness is that the results are under very strong assumptions.
    - First, the authors consider prompt attention, not self-attention, which is the more prevalent attention architecture. The authors also assume that the number of tokens is two, which is far from practice.
    - Second, the authors only analyze two steps of gradient descent (GD), and not the full trajectory. While the experiments show that with large step size, the model converges in two steps of GD, the theoretical results, which are the main contribution, seem rather weak since only two steps of GD are analyzed.
    - Third, the authors assume that the parameters are initialized at zero, instead of random initialization, which is used in practice.
    
    Given the strong assumptions, it is not clear what insights they offer about training attention models.
    
- Other minor suggestions:
    - There are some missing references in the related work section on optimization in transformers, such as [1-4].
    - In line 89, overfitting is misspelled.

References:

[1] Puneesh Deora, Rouzbeh Ghaderi, Hossein Taheri, Christos Thrampoulidis. On the Optimization and Generalization of Multi-head Attention, TMLR 2024.

[2] Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer, NeurIPS 2023.

[3] Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, and Simon Du. Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention, ICLR 2024.

[4] Heejune Sheen, Siyu Chen, Tianhao Wang, and Harrison H. Zhou. Implicit regularization of gradient flow on one-layer softmax attention, 2024.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can you define the (unusual) notations SNR $\geq \Omega(1/\sqrt{n})$ or SNR $\leq O(1/\sqrt{n})$ explicitly? Is it something like "SNR $\geq f(n)$ and $f(n)=\Omega(1/\sqrt{n})$"?
- Why does $I-\mu_1\mu_1^\top/\rho-\mu_2\mu_2^\top/\rho$ in the covariance make that the noise token is orthogonal to the signal one?
- Can you draw (final or $t=2$ test-accuracy) vs ($n/d$ or $n^2/d$) plot to get a better intuitive explanation?
- Can we somehow neglect the cross term $\sum_{i,j} y_iy_j\theta_j^{t=1}\xi_i^\top \xi_j$ without the assumption item 2, at least experimentally?
- What if we use a normalized noise $\xi \sim N(0, \Sigma/d)$ so that SNR $=\rho$ where $\Sigma=I_d-\mu_1\mu_1^\top/\rho-\mu_2\mu_2^\top/\rho$?
soundness: 3
strengths: I couldn't check all the technical details, but surely this paper has its strengths as follows:

- This paper provides a clear characterization of the role of the SNR (i.e., that of $d$ and $n$) for benign overfitting.
summary: This paper analyzes benign overfitting in single-head attention depending on the signal-to-noise ratio (SNR) of the data distribution.
GD with the logistic loss shows benign overfitting when SNR is $\Theta(1/\sqrt{n})$. For minimum-norm interpolators, the requirement for the SNR becomes weaker ($\Omega(1/\sqrt{n})$) and this is tight.
weaknesses: - Some theory appear to lack empirical evidence. Can we confirm Theorem 6,8,10 (in Section 4) with some experiments on the max-margin learning rule? Experiments in Section 6 seem like they are only for Theorem 4. If not, which experiments (figures) correspond to which theorem?
- It would be better to have experiments (or at least a discussion) on a relatively complex model (e.g., multi-head attention, multi-layer Transformer, more than two tokens). For example, a complex model may have similar behavior with the simple one used in Section 6. We may get some intuition from the experiments that a simple structure is enough to exhibit benign overfitting or some complex structures are crucial.
- The single-attention $S(XWq)$ or $S(Xp)$ considered in the paper is linear in $X$ before taking the softmax which is not usual attention as the authors consider the query vector $q$ independent of the input $X$. Is there any justification that we can use such simplification? This is a big limitation of this paper.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - For **Assumption 3**, why is an upper bound on the Signal-to-Noise Ratio (SNR) necessary? Additionally, how would the results change if a different step size is used during training, such as a larger or smaller step size?

- For **Assumption 3**, what would be the effect of not using zero initialization? For instance, have you considered using Gaussian initialization, and if so, how would this impact the behavior of benign overfitting?

- In the experiments, have you considered using a heatmap for test accuracy to illustrate the separation condition based on SNR values? Such a visualization could help clarify the relationship between SNR and model generalization in different settings.
soundness: 3
strengths: - The authors provide a rigorous theoretical analysis, supported by empirical validation. The proofs are well-structured, with assumptions and conditions clearly stated, lending robustness to the theoretical results.  

- The paper is clearly written and structured, allowing readers to follow the logical flow of arguments and results.
summary: The paper investigates benign overfitting within single-head softmax attention models, foundational components of Transformers, by examining when these models can fit noisy training data while still generalizing effectively to test data. The authors provide theoretical conditions under which gradient descent with logistic loss leads to benign overfitting within two iterations, requiring a specific signal-to-noise ratio (SNR) scaled as $\Theta(1/\sqrt{n})$. Additionally, they show that minimum-norm (or maximum-margin) interpolation allows for benign overfitting with a relaxed SNR requirement of $\Omega(1/\sqrt{n})$. Empirical results validate these findings, demonstrating that a sufficiently large SNR and input dimension are essential for benign overfitting, while lower SNR leads to harmful overfitting. This study offers new insights into the benign overfitting behavior of attention mechanisms, highlighting theoretical and empirical conditions for achieving this phenomenon in single-head attention models.
weaknesses: - Compared to [1], the data model in this paper is similar; however, the attention network considered here is relatively simple. For example, [1] trains both the query and key matrices, whereas this work only trains a single vector within the softmax function. This simplification reduces the complexity of the analysis significantly but may also limit the applicability of the findings to more complex attention networks.

- For benign overfitting with gradient descent, the requirement for both upper and lower bounds on the Signal-to-Noise Ratio (SNR) restricts the scope of the data model, potentially limiting the range of scenarios where the results apply.

- The data model only considers two tokens, which could constrain the generality of the findings. An analysis including more tokens or varying token arrangements would strengthen the applicability of the conclusions.

- Minor: Big-O notation (line 100-101)



[1] Jiang, Jiarui, Wei Huang, Miao Zhang, Taiji Suzuki, and Liqiang Nie. "Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization." arXiv preprint arXiv:2409.19345 (2024).

Meta-review: This paper theoretically investigates benign overfitting for a single head attention model.     

Although benign overfitting of transformers is an important research topic, this paper has several drawbacks: 
(i) The biggest concern of this paper is its novelty. First, the analysis heavily depends on exiting two papers. Unfortunately, this paper does not resolve their weakness. Moreover, the assumptions is quite strong (as described below), and thus is not strong as some other existing work.   
(ii) This paper considers a prompt-attention not self-attention and the token length is limited to 2, which restrict its applicability significantly. Hence, the theoretical implication to practice is quite limited.   

Unfortunately, these issues cannot be resolved by merely re-editing the paper, but It requires substantial revision. Hence, I think this paper cannot be accepted.

Decision: Reject