Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization
Authors: Jiarui Jiang
Wei Huang
Miao Zhang
Taiji Suzuki
Liqiang Nie
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision Transformers
Training Dynamics
Benign Overfitting
Self-Attention
Feature Learning
Primary area: learning_theory

Abstract: Transformers have demonstrated great power in the recent development of large foundational models. In particular, the Vision Transformer (ViT) has brought revolutionary changes to the field of vision, achieving significant accomplishments on the experimental side. However, their theoretical capabilities, particularly in terms of generalization when trained to overfit training data, are still not fully understood. To address this gap, this work delves deeply into the \textit{benign overfitting} perspective of transformers in vision. To this end, we study the optimization of a Transformer composed of a self-attention layer with softmax followed by a fully connected layer under gradient descent on a certain data distribution model. By developing techniques that address the challenges posed by softmax and the interdependent nature of multiple weights in transformer optimization, we successfully characterized the training dynamics and achieved generalization in post-training. Our results establish a sharp condition that can distinguish between the small test error phase and the large test error regime, based on the signal-to-noise ratio in the data model. The theoretical results are further verified by experimental simulation. To the best of our knowledge, this is the first work to characterize benign overfitting for Transformers.

Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization
Jiarui Jiang",, Wei Huang" Miao Zhang Taiji Suzuki Liqiang Nie!
Harbin Institute of Technology, Shenzhen
RRIKEN AIP
University of Tokyo
j jiarui com, wei huang vr@riken. jp, zhangmiao@hit. edu. taiji@mist u-tokyo...... jc. nieliqiang@hit. edu. cn
Abstract
Transformers have demonstrated great power in the recent development of large foundational models. In particular, the Vision Transformer (ViT) has brought revolutionary changes to the field of vision, achieving significant accomplishments on the experimental side. However, their theoretical capabilities, particularly terms of generalization when trained to overfit training data, are still not fully understood. To address this gap. this work delves deeply into the benign overfitring perspective of transformers in vision. To this end, we study the optimization of a Transformer composed
self-attention layer with softmax followed by a fully connected layer under gradient descent on certain data distribution model. By developing techniques that address the challenges posed by softmax and the inter- dependent nature of multiple weights in transformer optimization, characterized the training dynamics and achieved generalization in post-training. Our results establish a sharp condition that can distinguish between the small test error phase and the large test error regime, based on the signal-to-noie to-soise ratio in the data model. The theoretical results are further verified by experimental simulation. To the best of our knowledge, this is the first work to characterize benign overfitting for Transformers.
successfully
Introduction
Transformers (Vaswani et al. 2017) have revolutionized numerous fields in artificial intelligence,
ranging from natural language processing (Devlin et al. 20188 OpenAl 2023). computer vision
Dosovitskiy et al. (2020); He al. (2022), graph Yun et al. (2019) Hu et al (2020) reinforcement
learning Chen et al. (2021a)
J Janner et al. (02T). particular, Vision Transformer (ViT) Dosovitskiy
et al. (2020) has been developed to ad vancinn the computer vision tasks (Liu al. 2021 Khan
et al. 2022 Guo et al. 2022] Tolstikhin et al. 2021 Lin et al., (014)) compared to Convolutional neural networks. Since then, with the high performance, ViT emerged as hot area of research and application. Numerous technologies and methods began to surface, including enhancing ViT's data
and resource efficiency (Touvron al.) 2021 Hassani et al. 2021 Chen et al. 2021b Touvron et al.
2022) reducing heavy computation cost Chen et al. (2022): ZZh et al. 20024
Wang et al. (2023).
Recent success brought by VIT has been inspiring increasing number of works understand the ViT through empirical and theoretical studies. The empirical investigation works study the robustness
(Bhojanapalli al. 2021
Naseer et al., 2021 Paul and Chen 2022 Bai et al. 2021); and the role in
'Equal contribution CCorresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

self-supervision (Caron et al. 2021: Chen et al. [021c; Oquab et al., 2023) and others. On the other
hand side, the theoretical works have been conducted to understand the VTT from the perspective of
expressivity (Vuckovic et al. [0200] Edelman et al. (022), optimization (Zhang et al. 2020 Tian
al.
2023a b), generalization (Li et al. 20233)) and in-context learning (Huang et al. 2023c Garg
et al.
2022
Bai et al. 2024 Ahn et al. 2024).
Despite the insightful understanding provided by the above investigations, remains unclear how ViTs generalize to unseen data when they are trained to overfit the training data. In particular, the conditions under which ViTs can exhibit benign overfitting (Bartlett et al. 2020), where the test error remains small despite overfitting to the training data, are not well understood. Moreover, prior work on the optimization and generalization of ViTs often focuses on simplified settings, such as linear transformers or unrealistic loss functions, due to technical limitations. In this work, we aim to fill this gap through feature learning theory (Allen-Zhu and L.j 2020; Cao et al., (022). Based on a data generative model with M tokens composed of signal tokens and noise tokens and two-layer ViT with softmax attention, we characterize the training dynamics from random initialization and the generalization ability of the VIT after convergence. We establish sharp separation in condition to distinct the benign overfitting and harmful overfitting regime for ViTs. Our contributions are summarized as follows:
We successfully characterize the optimization of the ViT through three different phases in dynamics that exhibit rich and unique behaviors related to self-attention. Building on the convergence results, we further characterize the generalization bounds on unseen datasets.
Technically, we develop novel methods to handle non-linear attention and the inter-dependent nature of multiple weights in transformer optimization when training from scratch.
We establish a sharp separation condition in the signal-to-noise ratio in data to distinguish the benign overfitting and harmful overfitting regimes of ViTs. This separation is then verified by experimental simulation.
2 Related Work
Benign Overfiting in Deep Learning The most prominent behavior of deep learning is its breaking of bias-variance trade-off in statistical learning theory, i.e., good generalization on unseen data even with overfitting training data. This line of research starts from Bartlett et al. (2020) who studied benign overfitting in linear regression, learning data generated by a linear model with additive Gaussian noises. It is shown that at large input dimensions (which leads to over- weer-praaetteizzaioon the excessive risk of the interpolation can be asymptotically optimal. Hastie et al. (002);; Wu and Xu (2020) studied linear regression when both the dimension and the number of samples scale together with fixed ratio. Timor et al. (2023) proposed a unified data model for linear predictors and studied conditions under which benign overfitting occurs in different problems. Besides the studies on linear models (Wang et al. 2021 Zou et al. 2021 Zhou and Ge, (202)). several recent works tried to study benign overfitting in neural networks (Frei al. 2022] Kornowski et al.) 2024 Chatterji et al., 2022 Xu et al.) 2023). In particular, Li et al. (2021) investigated the benign overfitting in two-layer neural networks with the first layer parameters fixed at random initialization. Furthermore, benign overfitting
was characterized in convolution neural networks (Cao ct al. 2022) Kou et al.) 2023). OOD (Chen
af., 2024), federated learning (Huang et al.
2023b), graph neural network (Huang et al. 2023a)
by feature learning theory (A (Allen-Zhu and Li
2020 Cao et al. 2022). Unlike existing research on
benign overfitting, this work focuses on ViT'.
Optimization of Transformers Towards understanding the optimization of Transformers, Zhang et al. (2020) provided the analysis for adaptive gradient methods. Jelassi et al. (2022) proposed a spatially structured dataset and a simplified ViT model and showed that ViT implicitly learns the spatial structure of the dataset while generalizing. Besides, et al. (2023b) provided fine-graines mechanistic understanding of how transformers learn "semantic structure" for BERT-like framework (Devlin et al., 2018). Furthermore, Tian et al. (2023a characterizes the SGD training dynamics of a 1-layer Transformer and multi-layer Transformer respectively. They focused on the unique phenomena and the role of attention in the training dynamics. [Huang et al.] (2023c) studied the learning dynamics of one-layer transformer with softmax attention trained via gradient descent in
order to in-context learn linear function classes. In addition, Li et al. (0024) provided a theoretical

---

analysis of the training dynamics of Transformers with nonlinear self-attention within In- In-Context Learning. In particular, Li ei al. (023a) is most relevant us, as they also study the training dynamies of ViTs with similar data model. However, they considered hinge loss and a specific initialization to simplify analysis and did not characterize the harmful overfitting regime. In summary, while the above work studies the optimization dynamics of attention-ban -aaeed models, they do not characterize benign overfitting as we investigate.
3 Problem Setup
In this section, we outline the data generation model, the Vision Transformer model, and the gradient descent training algorithm.
Notations.
For two sequences {2nn) and [yn we denote vi, (1)) if there exist some absolute
constant C >> 0 and N > 0 such that [xnl < Clyn. for all 2 N. denote I, = (yy)) ) if Yn = (xr,), and denote Zn = (ynn) if Zin = (1n) and Zi, = ). We also denote In, = o(yn) lim / k = O. Finally, we use (-), 92-)) and (-)) to hide logarithmic factors in these notations respectively.
Definition 3.1 (Data Generation Model). Let p4: 11- Rd be fixed vectors representing the signals contained in data points, where l| ||2 = 11--112 = llall2 and (1++, 14- = 0. Then each data point (X,y) with X = (I1, #2...., xM)] E Mxd and y E {-1,1} is generated from the following distribution D:
1. The label is generated as a Rademacher random variable.
2. If then æı is given as μe, if then æ1 is given as p.., which represents signals.
3. A noise vector E2 is generated from the Gaussian distribution N o p+ ||a||/2 p- - (|
4. Noise vectors E3. ,3....,E.,. EM is generated i.dd from the Gaussian distribution CittrributionN((,ee (00, n+ II |||||2 p nr_ ||mlll²))
5. 22 2....., M are given by ......, ... EM, which represent noises.
Our data model is envisioned by considering M tokens of the data points: I1, .2..... EM. which can be seen as patches derived from image data. D embodics the signal that inherently linked to the data's class label, while 22, F3 -- MM represents the noise, which is not associated with the label. For simplicity, we assume that the noise patches is independently drawn from Gaussian distribution
and (O -I -I- I 11 1=ll"=-1 --- ((mmu(1-))) ensuring the noise vectors remains orthogonal to the signal vectors μ4 and μa_. Also, if o is sufficiently larger than op. the input tokens will become sparse, which is consistent with the empirical observation that the attention and activation values in Transformer-bast models are usually sparse (Child al., 2019 Robinson et al. 2023). We denote SNR || Vd) to represent the signal-to-noise ratio.
(0 oz. (I- I |1 2 - I- -
Two-layer Transformer. We consider a two layer Transformer network with self-attention layer and fixed linear layer, which is defined as:
M f(x.0) = M Σ
p( Wo WoW JWWwwo
Here, sp.)) RM - R. denote the softmax function, Wq. Wx E Rdxda Wy E Rdxdd denote the query matrix, key matrix, and value matrix, respectively, and wo E Rd denote the weight for the linear layer. We use 0 to denote the collection of all the model weights. This model is simplified version of the ViT model from Dosovitskiy et al. (2020), making our analysis focus on the self-attention mechanism which is the most critical component of the ViTs.
Given a training data set ((X,,ya) W generated from the distribution D defined Defini- tion 3.1 where the subscript represents the n-th sample, we train the two-layer Transformer by
3

---

minimizing the empirical eross-entropy loss function:
Ls(0) Σ (Xn,
where ((=) = log(1 + exp(-2)) and f(x, 0) is the two-layer Transformer. We further defined the population loss (test loss) (D) ==
Training algorithm. We consider Gaussian initialization for the network weights Wo, WK and Wv. where each entry of We and Wx is sampled from a Gaussian distribution (0, o ) and Wr
sampled from (0, of) initialization. We use gradient descent to optimize training loss Ls and the update of Wo, WK and Wy can be written as follows:
W = W (w
where W()) can be designated as Wo: Wx Wv-
Recalling the VIT defined in Eq. ( we can intuitively analyze its training dynamics and general- ization: if more attention is paid to signals, ie., oj Wo k #= 2 xj We W & for i E [M]/(1}, then the vector (a Wo x x ) has a higher similarity with signals μt rather than with noises E. In turn, Wy together with wo have more chance to learn signals μt and utilize them to make prediction. At the same time, if vector Wywo learns signals μ== more than noises & ie., μu: Wy wo 2 WW then during the gradient descent training, the gradient send to 2 W2 W K #t will be larger than that send to x; WoW W &.. As a result, more and more at- tention is paid on signals and the similarity between signals t.t and Wywo becomes increasingly high. Therefore, the Transformer can perform well on new data points. On the contrary, if much attention is paid some noises in the training dataset and the model utilizes them for making prediction and optimizing the training loss, then the model can fit the training dataset well but might not perform well on the new test data.
4 Main Results
This section presents our main theoretical results which characterize the convergence and generaliza- tion of the ViT model under different sample size N and signal-to-noise ratio SNR = [1/ll2/(ap The results are based on the following conditions.
Condition 4.1. Given sufficientll small failure probability 8 > 0 and target training loss (2) Dimension d = ((c=2N2dd) (3) Training sample size N = (ppolyoo(dd))((4) The number of input tokens M = -(1). (5) The f--norm of linear layer weights ||wlll2 = @(1). (6) The learning rate nj S [mmin[]lml) (o) -1--d df, 3). (7) The standard deviation of Gaussian initialization ov satisfies: ov S @11wolll min
>> O, suppose that:(1) Dimension dn. = 5i max(SNR* SNR ")NN2=2)) ".
d, ). (8) The variance of Gaussian initialization of satisfiees: min || (o,dd) d (log(6N2 M /5) S o S min ( (d,,d)")- d M2
(9) Turget training loss e S 0(1/polylog(d)) (10) The relationship between op and on satisfies ā, = Crop and Cp = M.
learning setting, and similar conditions have been made in the theoretical analysis of CNN models (Cao al. 2022 Kou al. (202)). Condition (3) ensures that there are enough samples in each class with high probability. Conditions (4)-(5) are intended simplify the calculation, and can be easily generalized M 2()1., ||ll = o(1) or
Conditions (1) and (2) ensure an over-parametiie
|woll2 = w1) setting. Conditions (6)-(8) ensure that the Transformer can be effectively trained well. Condition (9) ensures that the Transformer is sufficiently overfitting the training data. Condition (10) ensures the sparsity of the input token features.
We use f((-) and Ō(-) to omit logarithmic terms in the notation.

---

Theorem 4.1 (Benign Overfiting). Under Condition 4.1 iN -SNR2 = =11), then with probability
at least d there exist T = (nn 11,2112 wo 2 such that:
1. The training loss converges to Ls(((T)) c.
2. The test loss is nearly zero: LD(O(T)) - o(1)
Theorem characterizes the case of benign overfitting. shows that as long as N- SNR* S2(1) the Transformer can generalize well, even though it overfit the training data. To complement the above result and highlight the sharpness of this condition, we present the following theorem for the regime of harmful overfitring.
Theorem 4.2 (Harmful Overliting). Under Condition 4.1 if SNR 9((1, then with probability at least d= there exist T (NN)) 38 -o d 1/ 11 such that:
1. The training loss converges to Is(((T)) S.
2. The test loss is high: (D())()) @(1).
Theorem 4.2 shows that -NR S2(1), the trained Transformer has high test loss as result of overfitting the noises in the training data. Theorem 4.1] Theorem 4.2 reveals sharp phase transition between benign overfitting and harmful overfitting
Comparison with other results. Firstly, compared with CNNs with ReLU network result (Cao ct al. 2022), when genal-to-noiss ratio is small (SNR S 1), our results show that ViTs require less number of samples to gengeralize well, which reflects the advantage of Transformers. Secondly, previous theoretical benign overfitting analysis of CNNs with ReLU activations rely heavily on signal strength ||사||| and signal-to-nois ratio SNR (Kou et al. 2023 Meng al., 2023). i.c.,
SNR2 (1/NN) and require |||||| to be large enough, while our analysis does not need to impose restrictions on these conditions. Thirdly, previous works only show when Transformer can generalize well (Li et al. 2023a Deora et al.) 2023), while we demonstrate harmful overfitting as complementary, which reflect that our benign overfitting condition is tighter and more precise.
5
Proof Sketch
This section discusses our main challenges studying ViT's training dynamic, and presents our technical solutions to overcoming them. The complete proofs are given in the appendix.
5.1 Vectorized Q & K and scalarized V
Our first main challenge is to deal with the three matrices Wo. WK and Wv. In contrast to CNN models, whose convolutional kernels can be treated vectors and thus allow for more straightforward analytical approach (Cao et al., 2022 Kou et al.) 2023). the QKV matrices within the Transformer model are inherently complex analyze. Moreover, their mutual interactions further complicate the analysis. To circumvent this complexity. Oymak et al. (2023) and Tian ct al. (2023a) merge the key-query weights(e.g. W := Wq W
k Huang al. (2023c); (Zang al. (2024); Jelassi
et al. 2022) employ a specific initialization (e.g.
W ()) W I). Although their approach simpiiies the analysis, it is not conducive to showing the dynamics of QKV interactions, i.c., how Wq. WK. and Wv affect each other.
In order to simplify the analysis of QKV dynamics without losing rigor, we propose key techniques called Vectorized & K and scalarized V. The basic idea comes from the property that the product of token feature vectors with the QKV matrices results in vectors, e.g., l. Wo, E Wo. etc., which k X can be regarded as the [nI Wo: u W x) Therefore, the dynamics
are more amenable to analysis. Further, each entry of matrix X Wo W
inner product of two vectors, e.g., Ju_ Wo W W 44 of attention can be studied by analyzing the dynamics of the vectorized Q & defined as follows:
5

---

Definition 5.1 (Vectorized Q& & K). Let w and W be the QK matrices of the ViT at the 1-th iteration of gradient descent. Then we define the vectorized Q and vectorized K as follows
a 1 w (1)
q 10 ===I_W w
(t)
En, W
In
Q
k μ W (t)
h n W
h En, W (t)
for e [M]][]), E [N].
With Definition 5.1 denoting S+ == {n E [M] y/. = = S- {u E [M] 1/2 = -1], we further analyze the dynamics of the vectorized Q K. By carefully computing the dynamic of q+ we have
(++1) a+
(1)
Σ RES+
e MalB 1 W x (diog(())))) Pn.1 9 x
(1)T
X, W
9+
NM
R (2)
where p == (, n. Q woW can be viewed in the following forms
TT x a shorthand notation, and w W X and x,, (
wo WOT X = T wo $T,2 (
wo.... ST W W wo),
X, X, K
(k+
(0)
(0)T
n22
n.M
Thus 9+ ++ can be decomposed into linear combination of 444 and k (t) (++1)
(1+1)
knj. Therefore, Eq.
+ Enes Σ an kn where (2)
can be further expanded as q+ q= =
(8) (t)
(0)
Σ (()
(e) w ( ( pn,1 (n.1). In-
to+ n, (.t.M) N.
wES
tuitively, if a = is larger enough than o a,t,r then (q (0) will grow faster than (g (
(t)
which means token 44+ will pay more attention to 44+ rather than En.i- The dynamics of _ an.. k (0 and
(. nn are similar to ( and we can study the dynamics of QK matries by analysing
(t)
(e)
the linear combination coefficients such as a a a t.
As x T (((O) wo are scalars and their dynamics contain factor | we define the scalarized V as follows.
V Definition 5.2 (Scalarized V). Let w( be the V matrix of the ViT at the t-th iteration of gradient
descent. Then there exist coesficients 7v Vv. Pv,n,i such that
(2) (2) (t)
44 W (1)
wo u W. (0)
(1) |wo l22
wo + Vv.+
" w (t)
wo W (0)
w
wo + VV.
(1)
wwoll 122
S., W wo = ET, w wo + PY
[wo
for EM](L),, [N].
With the Vectorized Q & K and scalarized V, one can simplify the study of the Transformer learning process to a meticulous calculation of the coefficients such as a, 7, p throughout the training period. So how calculate the dynamics of these coefficients is key point in our analysis. In the next subsection we describe how to handle softmax function and further give bounds for these coefficients.
5.2 Dealing with the softmax function
Our second challenge is to deal with the softmax function, which is the critical component and introduces non-linear transformation in our Transformer model.
As We and Wx are within the sofimax function and Wy is outside the sofimux function, we divide the dynamic of training process into two key aspects: (1) How Wq and Wx affect Wv: (2) How Wy affects We and WK. Next, we present our approach addressing these two critical issues.

---

1. How WQ and WK affect Wv: Without loss of generality, we take data point (X,: Moo) with
V/ = 1 as an example and provide the dynamics for (t) Lemma 5.1 (Dynamies of y and p).
Vv+ PVn in Wv as follows:
and (e)
?7 2 VV- - V NM
exp
a n.
w W0WW" p4))
Wii K
((+1)
(0
Σ
Q
RES, (.[ exp(xT,, w
W T
p+) + Σ
exp(x w w knkk) Tht
K
k=2
27
M0 exp Bn W W (0) Sn,i)
PVv Prraol
(t+1)
NM
M p+) + Σ
sEIM exp(x w WhaT Q K
exp(x W w Ea,e)
(t)
k=2
fori E [M]{[], E St, where (D == (y (X (())) is a shorthand notation.
In benign overfitting regime where N SNN? = 92(1,, the factor
Σ
alll NM
is larger than
2nC3o2d NM
nES+
Therefore, long as n, ( 14 are not less than 27, W K (1) faster than PV In other words, if qi and (9) prefer to align with k rather than k then Wy would prefer to learn signals rather than memorize noises.
(T(
Sn,i 7V
(0
will grow
(1)
2. How WV affeets We and WK: Recalling that (a (a.... T X (diag((p)))) Eh w
Σ
(t) (!))) where the most complicated part
WES,
(t) the matrix (diag((!")) (1) - 4 pn,1 (1.)). We observe that the matrix diag( two important properties:
Pn,1 sp.,1 has
(+)T
1. The diagonal elements are positive, while the elements on the off-diagonal are negative.
2. The sum of each row and column of this matrix is O.
Based on properties and larger enough than ET, W wo. we have ++++ 2 0 and a,, i I 0, implying that q prefers to align k rather than (0)
deduce the following conclusion: as long μ W (t) wo is
(t)
In summary, if more attention is paid to signals, Wv learns the signals faster than memorizes the noises; conversely, if Wv learns the signals significantly more than it memorizes the noises, more attention will be paid to signals rather than noises. other words, Wo. WK and Wy promote each other.
5.3 Three-Stage Decoupling
Our third main challenge is to deal with the complicated relation among the coefficients, e-g., 7, Ps Inspired by the two-stage analysis utilized by Cao et al. (2022) to decouple the coefficients in CNN models, we analyze the ViT training process in three stage. In the following, we explain the key steps for proving Theorem 4.1 The proof for Theorem 4.2 is similar and we detail it in the appendix.
Stage 1:
The analysis in 5.2 shows that WΩ. WK and Wy promote each other. But this process of mutual reinforcement requires some conditions, e.g. p.j ".T. (f((() wo is sufficiently larger than <T, w uo which does not necessarily hold for Gaussian initialization, complicating the proof. formally Stage was introduced to solve this problem, and Lemma f.rrmaall describes this stage:
Lemma 5.2 (V's Beginning of Learning Signals). Under the same conditions as Theorem 4.1 there exist Ti
10M
such that the first element of vector X,, W wo dominate
nd (NlM 60MPC303d) [ its other elements, that is, p.l w wo 3M IKT, ( wo for all n St, i [M]]1} and u ( wo 5 -3M IET, w wol all S_, i [M]][1]
7

---

we can guarantee the monotonicity of attention on signals for t 2 Ti, which allows us concentrate on analyzing the growth rate of attention the following stages.
With Lemma 5.2
Note that the output of softmax function has an upper bound 1, thus the out- put of ViT can be bounded by max |M 1 W wol. E w woll By Lemma 5.1 it can be proved that there exists T2 = θ 17 11lll2 wwoll 112 (og(6N222/8))) such that: max(Iu] [ wol, I W/O wol} - o11) and further get 1/2 o(1) S -(x0 S 1/2 + o(1) Therefore, one can simplify the dynamics of the coefficients (e.g.. 7, P. a) by plugging the tight bounds of - The following lemma provides some bounds for the dynamics of Wq. WK andWy in Stage 2.
Stage 2:
Lemma 5.3 (Dynamics of QKV in Stage 2). Under Condition 4.1
if SNR² = 21(1, then |lll wo 2 and
with probability at least 1 - d there exist constant C, Ti = θ
17 dd
T- = = (1- ||1 wwol - such that:
1 W.
(B(tt - -T), (9) wo 2 C wo13(t
#4 ( wo 2 3M 3......) -KT, W (T) wol
11_ W wo S -1
Ti).
p W wo S 3M |ST,, W wol
(ol A (of ( log ( (xp((al ((g Th) RTT))
N (1ogGGN'MM (
(t Tt) - Ti 1))
ig k()) 19 ( 2 log exp ((g..) (
(t- Ti)(t- Ti
for E [M]][1], n E [N]. E [T. Tel.
Lemma 5.3 presents the training dynamics of Wo. WK and Wv under benign overfitting regime in two aspects: direction and speed, that is,
direction: Wywo prefer to learn the signals rather than memorize the noises; more and more attention is paid to signals, while the attention on noises is decreasing.
speed:
The inner products of μ+ and Wy grow at linear rate; the inner products of label-related vectorized Q & K grow at a logarithmic rate.
Stage 3: As the training process going on, the loss begins to converge, and the loss derivatives
- no longer remain near 1/2. Therefore, the increasing rate of the inner products of vectorize Q & K and μ__ Wy begins to diminish. Based on the analysis in Stage 2, the attention is sufficiently sparse at the beginning of Stage 3. In other words, the attention on the signals nearly 1, while the attention the noises is nearly O. According to this sparsity, I__ w wo will grow much faster than <T. w
(t) wo by Lemma 5.1 and Lemma 5.4 provides the lower bound of I I wo.
Lemma 5.4. Under the same conditions as Theorem
4. there exists T3
such that:
1. W wo 2 log [exp(1]] w (Ta) wo) + C 113 - Tz)
llog fort E (T2, Ts], where C is constant, T2 is the last iteration of stage 2.
p ( wo S - log (exp(-p_w(T) (Tb) wo) + @Cl Ta)
Lemma 5.4 provides logarithmic lower bounds for I.' W V wo. Plugging = T3 into these incquality, we have | w (1) wol 2 log (0(1/e)) then as long as I w wol is sufficiently small, it can be proved that yn (X..,
Convergence:
n. ((t) 2 log (1/e) for all n E [N]. thus [(y,J(X,,, (())) = log(1 + exp(- log(1/))) S 6 and L (0(1)) S 6. The convergence of ViT is accordingly obtained.
8

---

Consider a new data point (X, y) generated from the distribution defined in Definition 3.1 Without loss of generality, we suppose that the signal token is 4 and the label i.e. X $2,..2..... EM) ) 1. It is clear that ?. We W K u. has a mean zero, which implies that the attention on the signals in the test data may not necessarily be high as that the training data. However, the sparsity of attention during training process creates conditions for Wy to utilize the signals to make predictions, facilitating the generalization the VIT. We provide the lower bound for the output of ViT the unseen data with high probability as follows:
Generalization:
Lemma 5.5. Under the same conditions as Theorem 4.1 there exists T3 =
θ ( || l|wll2 logooll log (2 with probability at least 1 - 6/N2M,
log(C/e)
(f(XX)(T3))
1,
C²
where and C' are constants. 5.5 shows that yf(X, ((())))) large with high probability, and through careful calculation, we can provide a small bound for test loss. More details are in Appendix D.4
Experimental Verification
We present simulation results synthetic data and MNIST dataset to verify our theoretical results.
Synthetic data experiments setting: We follow Definition 3.1 to generate the training set and test set, Specifically, we set token size M = 16 and feature dimension d 1024. Without loss of generality, we set 14+ = |||||22 [1, O,. ,
and μ |ell [0, 1, 0, OJT. We generate noise vector & from the Gaussian distribution (0, T) where op is fixed to 4. Similarly. we generate the other noise vectors &; for i [M] (1, 2} 0.2.
,0 .
from the Gaussian distribution (N o7 where op is fixed to
We consider two-layer Transformer defined in Section The dimensions of matrix Wq. W and Wy are set as d₁, d, 512. The ViT parameters are initialized using Py PyTorch's default initialization method, and then they are divided by 16 to ensure that the weights are initialized small enough. We train the ViT with full-batch gradient descent and learning rate n 0.1, target training loss 0.01. We consider different traning sample size N ranging from 2 to 20, and different signal-to noise-ratio SNR ranging from 0.16 15.6. We evaluate test. loss with 100 test data points after training loss converges to c. All experiments are performed on an NVIDIA A100 GPU.
SNIP
1000
N SXS2 1000
20
20-
0U6
18.
0.5
08
Sample
12-
12
0.5
10
02
-03
4-
2
0.66
50
100
15.0
0.16
10.0
15.0
Signal-to-Noise Ratio SNR
Signal-to-Noise Ratio SNR
(a) Test Loss Heatmap
(b) Cutoff Test Loss Heatmap
Figure 1: (a) is heatmap of test loss synthetic data across various signal-to to-noise ratios (SNR) and sample sizes (N). High test losses are indicated with yellow, while low test losses are indicated with purple. (b) is heatmap that applies a cutoff value 0.2. It categorizes values below 0.2 as 0 (purple),
and above 0.2 (yellow). The expression for the red curves in (a) and (b) is N SNR2 1000.
Synthetic data experiments results: Figure la shows that as N and SNR increase, the test loss tends to decrease. As Figure lb shows, the theoretically derived red curve (N SNR? 1000)

---

almost follows the experimental dividing line between the yellow area (test loss > 0.2) and the purple area (test loss < 0.2). These experimental results further validate our theoretical results, that is, the sharp condition separation between benign and harmful overfitting, where N SNR2 (2(1) is precise condition for benign overfitting and N SNR -2 +2(1 is a precise condition for harmful overfitting. More experimental results on the dynamics of QKV can be found in Appendix
S
00
(a) SNR=0.2
(b) SNR=2.0
(c) Test Loss Heatmap
Figure 2: MNIST experiments
MNIST experiments setting: We add Gaussian random noises to the outer regions of the images with width of 4. The original images and the noises multiplied by diferent factors to generate diferent dataset with specife SNR. For example, Figure 2 (a), we mutiply the original image by 1/6 and the noises by 5/6, thus SNR=0.2: in Figure 2 (b). mutiply the original image by 2/3 and the noises by 1/3, thus SNR=2.0. We consider a VIT model that consists two attention layers, each equipped with four self-attention heads, followed by MLP with ReLU activation. The training sample size ranges from 3000 to 8400 and the SNR ranges from 0.2 to 2.0.
MNIST experiments results: Figure 2 (c) shows the heatmap of test loss, which shows transition between benign and harmful overfitting regimes. The larger the sample size N and signal-to hignal-to-noiee noise ratio SNR, the better the generalization performance.
7 Conclusion
This paper studies the training dynamics, convergence, and generalization for two-layer Transformer in vision. By analyzing the Vectorized Q & K and scalarized V in three-stage decomposition and carefully handling the sofimax function, give the precise increasing rate of the aftention and output of the Transformer in the training process, and further analyze its generalization performance. Our theoretical results reveal sharp condition separation between benign and harmful overfitting. One limitation of our methodology is our proof technique relies heavily on the sparsity of feature strength, e.g., assume the standard deviation op sufficiently larger than Tp. ensuring that more attention is paid to & rather than other tokens. One future direction is to generalize our analysis to study other abilities of Transformers, such as in-context learning, prompt-tuning and time series forecasting.
Acknowledgement
We thank the reviewers for their constructive comments on both theoretical and experimental as- pects. Miao Zhang partially sponsored by the National Natural Science Foundation of China under Grant 62306084 and U23B2051, and Shenzhen College Stability Support Plan under Grant EXWD200311281124400
References
Ahn, K., Cheng, X., Daneshmand, H., and Sra, (2024). Transformers learn to implement pre- conditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36.
Allen-Zhu, Li, (2020). Towards understanding ensemble, knowledge distillation and self-distillation in deep learning, arXiv preprint arXiv:2012.09876.
10

---

Bai, Y., Chen, F., Wang, H., Xiong. C., and Mei. S. (2024). Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36.
Bai, Y., Mei, J., Yuille, A. L., and Xie, C. (2021). Are transformers more robust than enns? Advances in neural information processing systems, 34:26831-26844
Bartlett, P. L.., Long. P. M., Lugosi, G., and Tsigler, A. (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, (77488)30003-30077
Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., and Veit, A. (2021). Under- standing robustness of transformers for image classification. In Proceedings of the IEEE/CVF international conference computer vision, pages 10231-10241.
Cao, Y., Chen, Z., Belkin, M., and Gu, Q. (2022). Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35:25237-252500
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging properties in suler-sue vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660.
Chatterji, N. S., Long, P. M., and Bartlett, P. L. (2022). The interplay between implicit bias and benign overfitting in two-layer linear networks. Journal of machine learning research, 23(263))1-48
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. (2021a). Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15000
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., and Wang, Z. (2021b). Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974-19988.
Chen, W., Huang, W., Du, X., Song, X., Wang. Z., and Zhou, D. (2022). Auto-scaling vision transformers without training. arXiv preprint arxiv:2202.11927 11921.
Chen, X., Xie, S., and He, K. (2021c). An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640-9649.
Chen, Y., Huang, W., Zhou, K. Bian, Y., Han, B., and Cheng. J. (2024). Understanding and improving feature learning for out-of-distribttion generalization. Advances in Neural Information Processing Systems, 36.
Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv preprint /
Deora, P., Ghaderi, R., Taheri, H., and Thrampoulidis, C. (2023). On the optimization and generaliza- tion of multi-head attention. arXiv preprint arXiv:2310.12680.
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. (2018). Bert:
Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1e.iivv/110..0 1810004800.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1199
Edelman, B. L., Goel, S., Kakade, S., and Zhang, C. (2022). Inductive biases and variable creation in self-attention mechanisms. In International Conference Machine Learning, pages 5793-583 PMLR.
Frei, S., Chatterji, N. S., and Bartlett, P. (2022). Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. Conference on Learning Theory, pages 2668-2700 PMLR.

---

Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. (2022). What can transformers learn in in-context? case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598.
Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng. M.-M., and Hu, S.-M. (2022). Attention mechanisms in computer vision: A survey. Computational visual media, 833:33113368
Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, and Shi, H. (2021). Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:210.005744
Hastie, T., Montanari, A., Rosset, S., and Tibshirani, R. J. (2022). Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009.
Hu, Z., Dong, Y., Wang. K., and Sun, Y. (2020). Heterogeneous graph transformer. In Proceedings of the web conference 2020, pages 2704-2710.
Huang, W., Cao, Y., Wang, H., Cao, X., and Suzuki, T. (2023a). Graph neural networks provably ben- efit from structural information: A feature learning perspective. arXiv preprint arXiv:2306. 13926.
Huang, W., Shi, Y., Cai, Z., and Suzuki, T. (2023b). Understanding convergence and generalization in federated learning through feature learning theory. In The Twelfth International Conference on Learning Representations.
Huang, Y., Cheng, Y., and Liang, Y. (2023c). In-context convergence of transformers. arXiv preprint arXiv:2310.05249,
Janner, M., Li, Q., and Levine, S. (2021). Offline reinforcement learning big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286
Jelassi, S., Sander, M., and Li, Y. (2022). Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems, 35:37822-37836.
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., and Shah, M. (2022). Transformers in vision: A survey, ACM computing surveys (CSUR), 54(10s):1-41.
Kornowski, G., Yehudai, G., and Shamir, O. (2024). From tempered benign overfitting in relu neural networks. Advances in Neural Information Processing Systems, 36.
Kou, Y., Chen, Z., Chen, Y., and Gu, Q. (2023). Benign overfitting in two-layer relu convolutional neural networks. International Conference on Machine Learning, pages 17615-17659, PMLR.
Li, H., Wang, M., Liu, S., and Chen, P.-Y. (2023a). A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. arXiv preprint arXiv:2302.06015.
Li, H., Wang, M., Lu, S.. Cui, X., and Chen, P.-Y. (2024). Training nonlinear transformers for efficient in-context learning: A theoretical learning and generalization analysis. arXiv preprint arXiv:2402. 156007
Li, Y., Li, Y., and Risteski, A. (2023b). How do transformers learn topic structure: Towards a mechanistic understanding. In International Conference Machine Learning, pages 19689- 19729. PMLR.
Li, 7.,, Zhou, Z.-H., and Gretton, A. (2021). Towards an understanding of benign overfitting in neural networks. arXiv preprint arXiv:2106.03211
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. (2014). Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part 13, pages 740-755. Springer.
12

---

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang. Z., Lin, S., and Guo. B. (2021). Swin trans- former: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022.
Meng, X., Zou, D., and Cao, Y. (2023). Benign overfitting in two-layer relu convolutional neural networks for xor data. arXiv preprint arXiv:2310.01975.
Nascer, M. M., Ranasinghe, K., Khan, H., Hayat, M., Shahbaz Khan, F., and Yang, M.-H. (2021). Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296-23308
OpenAl (2023). Gpt-4 technical report.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023). Dinov2: Learning robust visual features without supervision. arXiv preprint arX1n:2300..7719
Oymak, S., Rawat, A. S., Soltanolkotabi, M., and Thrampoulidis, C. (2023). On the role of attention in prompt-tuning, International Conference on Machine Learning, pages 26724-26768. PMLR.
Paul, S. and Chen, P.-Y. (2022). Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pages 2071-2081
Robinson, B. S., Drenkow, N., Conwell, C., and Bonner, M. (2023). A sparse null code emerges in deep neural networks. UniReps: the First Workshop on Unifying Representations in Neural Models.
Tian, Y., Wang, Y., Chen, B., and Du, S. S. (2023a). Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. Advances Neural Information Processing Systems, 36:71911-71977.
Tian, Y., Wang, Y., Zhang, Z., Chen, B., and Du, S. (2023b). Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention. arXiv preprint arXiv:2310.0053.
Timor, N., Vardi, G., and Shamir, O. (2023). Implicit regularization towards rank minimization in relu networks. In International Conference on Algorithmic Learning Theory, pages 1429-1459, PMLR.
Tolstikhin, O., Houlsby, N., Kolesnikov, A., Beyer, L.,, Zhai, X., Unterthiner, Yung, Steiner, A., Keysers, D., Uszkoreit, J., et al. (2021). Mip-mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261-24272.
Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., et al. (2022). Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence. 45(4):531445322
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021). Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Vuckovic, J., Baratin, A., and Combes, R. T. (2020). mathematical theory attention. arXiv preprint ar wxXx:2007.00877
Wang, K., Muthukumar, V., and Thrampoulidis, C. (2021). Benign overfitting in multiclass clas- sification: All roads lead to interpolation. Advances in Neural Information Processing Systems, 34:24164-24179,
Wang, W. Ma, S., Xu, H., Usuyama, N., Ding, J., Poon, H., and Wei, F. (2023). When an image worth 1,024 x 1,024 words: case study computational pathology. arXiv preprint arXiv:2312.03558.
13

---

Wu, D. and Xu, J. (2020). On the optimal weighted Eg regularization in overparameterized linear regression. Advances in Neural Information Processing Systems, 33:10112-10123,
Xu, Z. Wang, Y., Frei, S., Vardi, G., and Hu, W. (2023). Benign overfitting and grokking in relu networks for xor cluster data. arXiv preprini arXiv:2310.02541.
Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. J. (2019). Graph transformer networks. Advances in neural information processing systems, 32.
Zhang. J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. (2020). Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 13:15333153333
Zhang, R.. Frei, S., and Bartlett, P. L. (2024), Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1-55.
Zhou, M. and Ge, R. (2023). Implicit regularization leads to benign overfitting for sparse linear regression. In International Conference on Machine Learning, pages 42543 42573. PMLR.
Zhu, L., Liao, B., Zhang, Q., Wang. X., Liu, W., and Wang, X. (2024). Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.0 1..941...
Zou, D., Wu, J., Braverman, V., Gu, Q., and Kakade, S. (2021). Benign overfitting of constant-stepsize sgd for linear regression. In Conference on Learning Theory, pages 4633-4635, PMLR.
14

---

Appendix
Contents
More Experimental Results on Training Dynamics
16
Basic Calculation
16
I Notations
16
B,2 Gradient Calculation
18
B.3 Update Rules
21
Concentration Inequalities
27
Benign Overfitting
29
D.I Stage
29
D.2 Stage
33
D.3 Stage III
44
D.4 Population Loss
59
Harmful Overfitting
66
E.1 Stage
66
E.2 Stage
Stage
75
E.4
Population Loss
79
Complete Calculation Process For Benign Overfitting
80
F1
Calculactions for and
80
F2
Update Rules for Inner Products
88
Proof of Lemmal
97
Lower Bounds of a and
101
F.5
Lower Bounds of q. k
112
F.6
Upper Bounds of q.
118
F7
Bounds for the Sum of a and
126
F8
Proof of Claim 3
130
F.9
Upper Bounds of k.
137
Bounds for the Sum a and
142
Proof of Claim
144
F 12 Explanations of Lower Order Terms
152
Takeaways for Practitioners
154
Broader Impacts
155
15

---

A More Experimental Results on Training Dynamics
In this section, follow similar setting as the experiments in Section and add more experiments. All the experiments in this paper can be performed within hours.
Experiments setting: We focus on tracking the dynamics of QKV under benign and harmful overfitting regime, and the key parameters are as follows:
M
d 2048
d 1024
* de 256
dp 0.05
0p
Specifically, we set 16, |al/2 25 benign overfitting setting, and set 10, ||ll|22 15 as harmful overfitting setting.
Figure 3: Training Dynamies Under Benign Overfitting Regime
Figure 4: Training Dynamics Under Harmful Overfitting Regime
Experiments results: Figure 3 shows the training dynamics under benign overfitting setting, and Figure 4 shows the training dynamics under harmful overfitting setting. In Figure 3 az Wo WT k grow faster than a; Wq W ε2, and more and more attention paid signal p4+- Mean- while, μ. Wo grows faster than Wo In Figure x W WoW grow faster than a, W W K J.4., and more and more attention paid noise ε2. Meanwhile, & Ww grows faster than μ4+ W.w In other words, the ViT prefers to learn the signals rather than memorize the noises under benign overfitting setting, while prefers to memorize the noises rather than learn the signals under harmful overfitting setting.
W
B Basic Calculation
In this section, we introduce some notations for this paper, and give expressions for the exact gradient of loss function Ls(#) with respect to WQ. WK and Wy.
B.1 Notations
16

---

Table 1: Key notations in this paper
Symbols
Definitions
Øn.i
the i-th token in the n-th training sample if E [M]M[]) Dn,i = En...
the i-th row of attention for n-th sample, ie., sp..) = spwe,, W o x
S+,S_ S-
the training samples with +1 labels and labels,
i.c. ,S == {n E [M] = 11. S_ = {m E [N] U/ = =
. --- Am..
vectorized Q, defined as q pe W _0hh n w a.
ET,, W
vectorized defined as k p W (t) 1 = W A E.. W
(()) (() (0)
scalarized V, defined as v := pel W wo. V = n= w wo. vul = E.T, W wo
0 4. A.
linear combinations coefficients for the dynamics of at and q
i.c. q q= 1 Σnes
((+1)
(t)
nES Ex a 1i
linear combinations coefficients for the dynamics a
i.c., gm.s g (.)
((+1)
a ( a,,, (1))) + Σn Em ana
(+)
(1) g(0) 2. m.th
linear combinations coefficients for the dynamics of k44) and k0 i... k (t+1) 5 1
q (t) Σ
Ene
M 9(t) A A
(00 809 n..
linear combinations coefficients for the dynamics (1)
i.e., (t=1)
(1) p(()
g(tt) po,
qtd) (") Σ
w==1 Σ
git)
explig (("')"
softmax ((1))
a general references to
for S4.
exp((a a 100+ E explla (()))
exp((( ((l)
and
for E
explin (1) x (04 E exp(ia (1)))
k
soflmar((gh)) (1) (2))
a general references to
explio
for E St,i [M])(1)
explla & x 1 exp(ia "..
and
for n E S [M]0
exp(la (1) (
+
exp((o 100
softmar(ig] (t) (0)))
a general references to
explia
for E S [M](1),
explly a (01) E
explly (
and
for E S-,1E (MM1)
expllg (nol( E
exp(ig (0))
km2
softmarlig (())))))))))))))))) k())))
a general references to
explla
for E St, E (M]11),
explla (
(0)))
and
for E S_, E (MM))
exp( (q) (0)
+ A
exp(to (000 (99
(1)
t.j. = (4 Ae(0) ( ()))
(4)
A A = (1) k(0) (g (1)
T + (1()
n.t..g
To n.
v (1)
+ (g (1) ((1))) (9 (6) (1)))
( (9 (1)))
(9= (1) (0
n.s. = (1) (1))) (g
(1 = (g k.1)
(g))) k(t)
17

---

B.2 Gradient Calculation
Gradient of Softmax. Before computing the gradient of QKV, solve for the gradient of Softmax function. Suppose there an input vector a (d1, a2, ..., am), then the softmax function can be defined as follows:
softmar(a.) = exp(a:)
exp(a,)
Then we calculate the gradient of softma((as) with respect to a, and ay for i,j E [m], 2 + j.
m Σ ==1
exp(a;)
exp(a;) - exp(a:) exp(a:)
① sof sof softmar(as) 04
m
exp(a,))2
exp(a;)
exp(a;)
(3)
Σ exp(ay)
exp(aj)
softmar(a:) (1 so fttmar (as)
i softmor(a.)
exp(a))xx((o,
aj
Σ exp(a;)))
(4)
1=1
softmar(a.) sof softmar(a,
and (4) show that the gradient of Softmax function can be represented as Jacobi matrix where the elements on the diagonal are so softmaz(a.) (1- softmar((a) and the elements on dff-diagonal are softmaz(a:) so ffmaaraaa) With these properties, the gradient our attention vector an,i (i E [M], (N) can be expressed as follows
Opn.
diagg(pn..) I Pn.
(5)
(x T Wo W x.))
Lemma B.1. the gradients of loss function (s()) with respect to Wo, WK and Wy are given by
Vw Ls(0)
1
Σ (,(0)(4 W. x (diag((vn.)) ,p,,1
NM
nES+
w WT X (diag()pn.)) (T.T WK
2
Σ (, w WT x, (ddagg((nn))) pp.1))
NM
BES_
w T W X.T (diag()Pn..) - (T.T(P)))) X WK.
18

---

Vww Ls(
Σ (X))( (diag((em.)) - -p., Wywon
NM
nES+
M
(ding((Pn..) ( X,, ( We
Σ (X - Wywop"
NM
RES
M
- ( x,, W Wo, -
X n
1
Vw, Ls(0)
NM Σ ynl (w)
Σ Wl W x Xx,)
Considering that in the vector x "., Wa W I (3). only used to multiply Wouss then we have
Proof of Lemma B.1
K XT, the j-th element of x T n. i.c.
ax WoW (ann))is (X, Wx)
OW Qbaa]
And then get
Vaana (x( Wo W (,,,,,)) )x, Wvwo)
0(p Xx, Wvo))
= (Kn.))i) [xx, Wo W 1 XT)
(X,, Wx)
Considering all j E [d], we get
Vwo (p(x)T WoW K X X, Wywo)
O(p,. X, Wvwo)
(6)
=xn,i Ax W
X WK
19

---

1 N
Va (00)
Un (x:
M
yn (0)
Wo X Vwo Wa x, Wywo)
NM
1
M
0(pn, X, Wvwo)
3n
Iniil
X, Wx
NM
0(x; Wq W X
==1
1=1
1
den.,
Σ (0
Σ
Fnii [X,,, w aa Wo Wx X,
Xn
NM
M
1
Un e. (0)
Σ
nn w Wv X (diag(4nn.)) (n, Wx (7)
NM
1
Σ NEST
(,00)(m W. X (diag((nn.1) (n,149,,))
NM
w T W. x (diag(enn.) (T. WK
e, (,(0)(ma_ wo W. X (diag((Pm,1) 4r,, Pm,1)
NM
nES_
M
Wo T Wv x (diag((Pn.s) (T)) WK
where the third equality is by (6) the fourth equality is by chain rule, the fifth equality is by (5) For the last equality, we expand the equality by materializing all the an (e.g., Bn,1 p4+ for n E S4). Using the similar method, we obtain the gradient of WK.
1 N Vw Ls()) = NV
Σ
Unl (Jn,((,,,))
0)
N
M
Σ
Un 2n (0)
Σ
Vwx (p(x,, We W X Xx, Wvwo)
NM
R=1
t=1
1
M
0lpp X, Wy wo)
3 (0)
X
x WQ
NM
(a. Wq W.
t=1
1
Σ yn (1)
M
Oyn.,
X,, Wvwo) IT,, WQ
NM
Oxx WoW XT)
(8)
1
Σ (X,
(diag((e))))) pn. 1) X, Wwwon
NM
RES+
M
(diag( (h. X, Wv w. We
2
(X. (diag((en))) - (T. Wwwop!
NM
n
(diag(een.)) - (T.. X, W.woLT We
20

---

The gradient of Wv can be obtained using the chain rule as follows
1
Ls()
Unl () 8)
N
(9)
N
M
On 2 (0) 1wo
Σ ll (X
NM n=1
B.3 Update Rules
Definition B.1 (Scalarized V). Let W (1) be the matrix of the ViT at the 1-th iteration of gradient
descent. Then there exist coefficients (2) (1) (8)
Vv V, pyn such that
w 1 wo = 1 w wo + Vv 1wo 112
W wo = p w wo + W. 1w l2..
(1)
W (8) V wo ET, w (0) wo Pv, /wo 11
(2)
for i E [M]M1),, n E [M]. We further denote v( n w v wo. ( == L. wo and wo- We refer to it as scalarized V.
((O n,1 = -TT, W. woo
Definition B.2 (Vectorized Q & K). Let wl and w be the QK matrices of the ViT at the 1-th iteration of gradient descent. Then define the vectorized Q and vectorized K as follows
_ = w. (1)
= w. (1)
(t)
5.T, w/l) (1)
Q
( u. ( K
L (W() K
[ = E.T., w
fori E [M]M[], n E [N].
Definition B.3 (Gradient Decompasition). There exist coefficients 0 + n
(1)
n niit
(t)
(1) - 5 g(1)
(0)
((0))
P s(4)_ ,(00
Pn--r p.itt - niim' s such that
g(4) g(1) g(0)
A q+
(t) (1+1)
(1) (1)
a
NES+
1=2
M
Aq
(1)
q
(++1)
q
(t)
(0
RES
1=2
Ag
(1) (t+1) (1) (t)
Un.i gn.. a
a
k
n,
(t)
( ani a al
n'=12=2
M Σ
4k = h
(1+1)
(1)
BES 1=2
M
0 (1) (1++)) - 10 g
RES. 1=2
N + Σ
M
( n.
((t+1) (1) 3 g+ + +
(1)
g(t))
f E (M) and n, n' E (N).
21

---

Remark. With the scalarized V in B.I we can portray the learning process of Wv on signals and (t) noises by analyzing the dynamics of + V. and PV... (or V(O,, v and V.l Also, as scalars, the effect of Wv on the learning process of We and WK is more conveniently to describe. With the Vectorized Q & K in B.2 we can decompose
(2) (1)
Wq X into the inner product of vectors (a generalized reference to a+, 9- and qu,ã) and & (a generalized reference to k4, k- and k,.s), e.g.. (q+, = μ We Wx 14. Then the dynamics of x: WoWKXX can be characterized by analyzing the dynamics of the inner products of q and k. and then further get the dynamies of attentions. The gradient decomposition in B.3 comes from 7 and (8). Note that (7) ends in X, Wx (t)
and ( ends in x ",,," Wq. This suggests that the dynamics of q and In.s can be decomposed into
(t)
linear combinations of k_ and k/ and similarly, the gradient of k and k. can be decomposed into linear combinations of 9 and In.." We calculate (a generalized reference a+,- 04 a_,-, etc.) and 3 (a generalized reference to B+.+. Ba,t __,-, etc.) in detail in Appendix F
(t)
Lemma B.2 (Update Rule for V). The coefficients (0) V VV. PV defined in Definition B.1 satisfy the following iterative equations:
((+1)
(t)
VV++
-V,+
11lll12 Σ
exp(1g ( k(f)) M
NM
nESt
exp((g (9) b.9) + Σ exp((1 19 (20))) k=2
M Σ
explignn
(1) k(11))
M
2=2 exp( (1)
exp((1m)) 10) + Σ
exp( expl((nn)) (0)
(dnjj k(()))
2
11+1) = =v---
W.-
7/1M113 Σ
exp((g_) ()))))))))
NM
M
nES_
exp((g (1 +
Σ exp(1g ( k(2)))
k=
M
exp( (1)
k(11))
M
2= ) 11)) + Σ exp((gn)) (6)
(1)
k((h))
[+1) (f)
Py PV
M
exp((1+) (t) k200pp1)
Σ n'
((t) n'
(iEnss 2'=2
NM
M
exp((g_) (1) x(11)) Σ exp( (1 (22
(1)
M
expl(gnn.) (1)
(En En(,))
J=2
(1) exp((120) ((1)))
+
exp( (a kx
(1)
M
exp( (1)
1
Σ
(() n'
(En Ex(,e)
NM
M
n'ES_
exp((g_) 1 k(0)))
Σ exp((1 (1) x()
2=
M +
exp((g (1) (t)
jz x(x)))
(n (n(.e)
2=
exp
(1)
1000)) + Σ exp( (t)
(an
k=2
for E [M][[],, [N].
Proof of Lemma
B.2 Basc on 9 we have
22

---

N
M
x Vw
yn (0) (
Σ
p Wo )x wwo
wo
NM
Σ
On 2 (0)
X W
NM
l=1
M
exp(xr, WoW Wo 14)
Σ
2
Σ
(1x, m+)
NM
Wo Wo W (4) - Σ exp(x W W kn.))
RES
Wo Em.s)
(r,, Sm.i)
1
M
explxx T W W m+) + E
exp(x We (mn))
1=2
exp(a, n. Wq [K
Σ
e, (0)
(1x
NM
M
nES_
exp(x T WWWWWWH K --)+ + Σ exp(2 T n Wq WK
j=2
exp(xr Wo En.i)
((x, En.i)
1) 10
M exp(xx Wq Wk-)) Σ exp(x x T. We WT km.))
exp(μ I W W WKKM 1+)
Σ (,(0)(((z,,4))
NM
M Σ
nESt
exp(u WeTWWWWHHL
W
R=2
exp(EX., WoWkHH)
(r, (1))
M
j=2
eXp(ETT, Wo WoW + Σ
exp((ET, We WoW W.M))
k=2
exp(μ WOWKE
(1r, En.i)
M ( + Σ WTT
R=2
exp(En.) WilnnnWWWT. WOWTE
(x, En.i)
1w
M
jm2
exp(ETT, Wo WoW exp(En Wo (..))
Σ ((00)(((,,
NM
nES
1-) + Σ explu_Wo W...))
exp(ET, WoW
xx )
M
2=2
eXp(ET,, WoRTT,WWWH w + Σ exp(ET,, Wo WTEN)
k=2
exp(μ We WQW..
(xx, En.i)
M
2=2
W + Σ
exp(x Wo WT..))
exp(En., Wo(En.WWWW' En.i)
(r. En.i)
M
1=2
exp(ET,, WoW Σ
exp(ET, Wa WT km.k)
k=
(10)
23

---

where the second equality we expand X, into vectors and make inner products with z, the third equality we materializing all the In (e.g.. In,1 = =+ for S4). Note the orthogonality between and in,i. we can remove many of the terms in this equation. Take æ μ4 as example, we have
μ FwwLL(())
1 NM n
exp(u_ We W 1b+)
Σ (
exp(u WWW Σ
explu We W En,k)
k=2
(11)
M
explEn, WOWk
1
f=2 e WWW + E exp(En,, WoW
Then we have
Fe W (1+1)
w0 uu w wo = - 1/ (w),,,,s((((t))
explix+ ((()))
Σ (TT
NM
exp( (1+ (1) (441))) + Σ exp((14 (())
nES+
(12)
e (111))
11wol1
M
j=2 exp exp((
(t)
x) + Σ exp((1n)) ()
km
Dividing by wwoll we get
(++1)
= -v,+
Vv+
211اا Σ p(t)
exp((g ( (:())
NM
exp( (9) k (0)) + Σ exp((( (t) 100 M
nES+
(13)
M
exp((an): (1) ((1)))
exp((am)) (1) (2)) + Σ exp((ia (t) ((t) (t)
2=
(0) (0) This proves the update rule for V The proof for VV. and PV.. is similar to it.
(0)
Lemma B.3 (Update Rule for QK). The dynamics of x Wq WK Wx be characterized as follows
( 1 (1+1); - (9
((+1)
(1)
M
(0)
(o + nut. ( (0)
(
(9)
a
nES
1=2
M
+
+
((0)
NES+ i=2
M
(of (0
+
nES 2=2
M
(g(t) (T)
g(t)
(+++9+
+
An.i
nES,
2
24

---

(q ( k 1((+11)/ (q k00)
M
(t)
12
nES im2
+
_
119
(1)
112
(0)
n.
nES-
M
10)
nES_ 1=2
M Σ
(
nES_ 1=2
(gtt)) (t+1) ((+11) - (g b0))
M a (of 12
= |14 121 11 + a - (K )0)
a==1i=2
M
+ 4 144 Ln.td +
n'Es+ 1=2
N
M
+ n.
(0)
+ Σ
n,Y,
M
(0()) (1)T
Σ
3(0)
(T)
++
+ Ia'
n'ES+ t=2
(111) ((+1)) (9 (0)))
(gn
M Σ>>>>>
(1) ||
10 112 + a
i0)
i==l==2
M
+ +0000 (a
R'SS_ 1=2
N M
+
(0)
a n,i,-
(t)
1000 + ΣΣ
n.i. A n1.)
1=
M
(000) -
(1)
g(t) (1)
+
n'
REES_ 1=2
25

---

(q (+1) b - 4 (
(8)
(1))
mg (0 x
a.
n'ES+ i=2
N
11 12 1 (9t 1
(0)
(1)
a 144 22.
(0)
1==1
M
+ (4. + Σ
n'ES+ i=2
N
M
(30) (6)
g(tt)
alt)
(0)
(... a+
+ n.j.
( , 11(+11) - _1 (
M
a (1) (1) + Σ
- ( (0)
n
n'ES- 1=2
N
M
+ -00 an.. 11 (1)
n.J.+ (gf
Jij a 19 (.
M
+
n'ES_ 1=2
N
M
(800 (0)
+ g((()
(0T)
nnj. 9+
Pn.j.
nam 1
n'=1
1=
(gn n
(1+1) ((+11)) (1)
(gn. kn.))
N
M
= niit (b44) , k11))) + 0 02,i
(k (0)
1=2
N Σ
(00 Pnj. (g+ (n.i) + p20)
(0)
(9 +Paj.-19
(0)
g(0)
Andinn a 12 9n.))
(9)
n
N + Σ
M
(000 ( + +a a.,,
(4) (1
(1)
1000
R!
1=2
N
M
(B g+
(
g(t)
T0) +
3(t)
(t)
+
Pnj,-
jn i In'
jori,j E [
Proof of Lemma
B.3 Based on Definition B.3
we have
M
Ag (1) == 94 q
(1+1)
Σ
(t)
(14)
n n.
RES, i=2
M
Ak : k (1+1) - -k(() = ((1)
+
8(1)
R.t. In.i
(15)
RES+
2=2
26

---

Then we get
(9+ (1+1) k4
(((-1)) 1) - ( kk (
=MI(W ( (w
(t) + AAWK K
14 - (9 12
- -Aq (1) k11) + (q 10 AA + (Aq. AART)
M
-aff |1 1 + Σ
nES_ ==2
M
(16)
+ 8 11 11 + Σ
3(0)
nES+ i=2
M
(0)
nES+
1=2
M
T
())
n. 9n,
nES
2=2
where A (W
Q = W
(1+1)
W
AW = w ((++1) W
K the second equality is by Ag
Q
K
a aww (1) Q and AKT
the third equality is by plugging and (15). This proves the update rule for (q))) k()), the proof for (q.... k k()), (4 A02)
(t) pI
K
are exactly the same. (g
Remark. In (6), note that the magnitude of (A: (f)) (a+ (1) is much smaller than that of ((0) | [|g, || l3. and that the magnitude of +.. =. will not be greater than that of (t) g(t) (we will prove it later). Also, since the learning rate 1) sufficiently small, the last term of 166 will be very small. Then (66 can be expressed in the following form
(04
(1-1) 1 ((+1))
(1+ (8) (1)
10
113 + ((()
11 112 + (lower order term]
The dynamics of (g.) (0)), ( x11)) ( ( can be expressed in a similar way. Later we
will rigorously explain the so-call (lower order term
C Concentration Inequalities
In this section, we will give some concentration inequalities that show some important properties of the data and the ViT parameters random initialization.
Lemma C.1 (Lemma B.1 in Cao et al. 2022 ) Suppose that 8 >> O and n 2 8 blog(4/5) Then with probability at least
8.
3N
4 $ Hn [ :n = Kn E [N] Un --1
4
Lemma C.2 (Initialization of V). Suppose that ō > 0. Then with probubility at least 1 - 6,
[ < d t
| d 1
for i E [M]][1], E [N].
Proof of Lemma C.2 is clear that I W (0)
wo a random variable with mean zero and variance
of w /2 Therefore, by Gaussian tail bound and the condition that ov S O11wolllz*
min(llell (dp d # with probability at least 1 5/NM,
II W (0) wol < σv co||wollel|ll|e 1ogg2l// S d
27

---

Moreover, each element of vector w W/ af || [1. Therefore, by Bernstein's 's inequality and the condition of øv, with probability at least 1 8/NM
(0) is a random variable with mean zero and variance
ST. W wol S
S d 7
Applying a union bound completes the proof.
Lemma C.3 (Initialization of QK). Suppose that ò > 0. Then with probability at least 1-6 8,
|| |
11 5 3 11
$ 119
(0)
RRoR
3 2 of
12.2212 <
2
2
o2
30 3
o
In. 11 <
(0)
p
2
2
3110| Bo
< || 10) 1 -
2
2
5 o3oRddn
1, = p 2 0 ddd,
<
2
2
of oz
30 oo
| ( 12
2
2
1a (0) _ s 2
Vdh 18)
(9 (0) (0)) < 2 21
ll 1 dt V dn 11g6)
K (0) k _0) < 100
da 122()
(g (0) ( s 2 Boh dh log(6.=22/8))
(g (0) k < 2 2 V da log6.N-MM20)..
(g k_ \| S 211 V
(0)
Vg (r)) |1 < 25.oRd- da log(NN2190)
(110) (0)
23)1 < 20 v du log(6N2
KA_h ( (0) < 2 12 1 dd V dn log
(g
(0) ( < 2 22 12 1 v du (ogoo(NN2MM/0)
(0) (0) د S 25 Rd Vd, log(6N22218))
Kg
fori,j E [M]M(). 8' E (M](1, and n.n' n' E [N].
Proof of Lemma
is clear that each element of vector q: is random variable with mean zero and variance || 20 By Bernstein's 's inequality, with probability least 1 5/3N2M², we have
C.3
|| (0) 1 |1 0(IIm1 120 V du log(6N2M2/8))
Therefore, as long as dn = 2(log(6N2 /6)), we have
||
1
m11 311
< ول 0 1/
2
2
Similarly, we have
|| 130add
311pd 14 122 dn
<
_
1|q
00 <
3 2 dd
< 19
<
2
28

---

o ozo7dd
30 R
<
(0)
12 -
2
2
1 || 10) 12 -
0
of
30, 2 0f dd
< 2,,111 <
2
2
. ddn
30 dd p
(0)
13
2
2
Moreover, (a_ (0) , has mean zero. By Bernstein's inequality, with probability least 1 5/3N2M2, M2, we have
(0) - s 21 dd, (og(6.
We can prove the rest of this lemma in similar way.
Lemma C.4 (Lemma B.2 in Cao et al. (2022); ) Suppose that > 0 and d = ((1log(4NMM0)) Then with probability at least
5 < 16,2112 < 30
02 < |En,il22 S 30 i R)) s 25 Vdlog(4N2M270)
i [M]M]),,, n, E [N], * or n + n'.
D Benign Overfitting
In this section, we consider the benign overfitting regime under the condition that N SNR 2(1). We analyze the dynamics of V+, Vn,i. the inner product of q Qu and kts kn, during gradient descent training, and further give the upper bound for population loss. The proofs in this section are based on the results in Section which hold with high probability.
D.1 Stage
In Stage I, vy(", v-(t) begin to pull apart until [V[ is sufficiently larger than [V[2] At the same time, the inner products of q and maintain their magnitude.
n,
Lemma D.1 (Gradient of Loss). As long as max((V (1) -1 (, o(1), we have (y (())) remains 1/2 + o(1).
Proof of Lemma D. Note that f(=) = log(1 + exp(-2)) and -P' exp(-2)/(1 + exp(-2)) without loss of generality, we assume yn 1, we have
- (X 0(1)))
1 Σ p Q 1 1 exp((
M
( X x,) X, wo) wo)
Note that
I S
M Σ
pl w/ w )x,,, w wo max 1,1r1(9
Then have
1
-X's(X", ((1)) >
2
2+ o(1) o(1),
1+ exp(0 + o(1))
exp + (())
o
(X, ((t)) < exxe +
5 +
1+1+o(1)
29

---

Lemma D.2 (Bound of Attention). As long ( Kg 11)) K 1
o{1), we have
M (1) - softmar((g (0) k()) v + +
1
M
M (11 - softmar (g ( v
1
1
+ +o(),
M
M o(1) f softmar (9 {1)))) _
1
1
+ +1))
M
M ((1 v so softmar (g (2)) S M + (1).
1
Proof of Lemma we have
D.2 It clear that exp(o(1)) = = (1).. Therefore, as long as (g A = =0(1,
M o(1) 1+ (M 1) + (M - 1)o(1) 1+ (M 1) exp(o(1))
exp( (1)) + (M - 1) exp(o(1)) S ssftmaax (1 (0) (111)) exp(o(1)) + (M - 1) (xx))xxp(-ol cxp(-o(1))) exp(o(1))
exp(-o(1))
exp(o(1))
1 + o(1)
1
M +
exp(o(1)) + (M 1) 1 1+ 01) + (M -1)
Similarly, we have
M o(1) s sos softmar (g (1) k(")) - +o(l),
1
1
M
M o(1) < so so smflmar (a= x2)))) <
1
M + o(1),
1
M o(1) S coftmar((g.)
1
m (1)))) S
M + +1()
Lemma D.3 (Upper bound of V). Let To = 0
Then under the same conditions as
nd |1 11
Theorem 4.1 we have
IV
= 0 1=
for E [0, Tol.
Proof of Lemma
D.3 By Lemma B.2 have
(++1)
exp((g(")
(1) ((()))))))
Σ ((())
$
NM
M
nes,
exp((g (8) (2)) + Σ exp((4t) (())))
(1)
k
exp((g ((1)))))
+ Σ
M
J=2
exp((g.) k (1)) + Σ exp( (t)
( (1))))
km
711m4112
3NV
(1+ (M - 1))
NM
4
3 31l|ua
4
where the second inequality is by Lemma C.1 and - (()
1. Similarly, we have
3ml | 1
(t+1)
(0)
hv=
30

---

By Definition
B.1
we have
| -
[1 +
Mi+ - - 11woll
(a+1)
5 1 +
(s+1) 1++
7o. /
==
301
$ dr
|00
12 0
4
nd
(d
where the first inequality is by triangle inequality, the second inequality by Lemma C.2 have [
Similarly,
O(d, *).
By Lemma
B.2 we have
expl(g (9 (1)
n
Σ
(())
(En. En(
NM
M
exp((xt) (1) x441 + Σ expl(q)) (1) ())
n'E8+
2=2
(2 .1 k1,)))
E En En(
))
exp( .z k (00)) +
exp exp((a (q (0,,)))
k=
M
exp( k mx,e))
7 NM
(En, En(,e)
n'ES_
1=2
exp((q (1) k(0 + expl 10 (((aa))
(17)
M
(xp J: k((())))
(En,is En(
1=2
expl(g (1)
k (1)) + Σ
exp( (1) (qn' k,,k)))
k=2
3n0 d
M +
" NM
MN 25 v Vdlog(4N2222/5)
2NM
2n
where the second inequality is by Lemma C.4 and -(10 I 1, the third inequality is by d δ
(0)
N2 2 4N Vlog(A (22) the last inequality is by N SNR2 = 2(1). Then by
Definition B.I have
[
(1) n.t
(p - -lwolll
< |
1 - -
||
d
0n w 0(
nd
|| woll
(d
where the first inequality is by triangle inequality, the second inequality is by Lemma completes the proof.
which
31

---

Lemma D.4 (Inner Products Hold Magnitude). Ler To = 0
Then under the same
nd | l
conditions as Theorem
we have
Kg1 k1 M. K9n. M. Ka 1
(8)
=0( max (11l ozd] of dd log (6NM
Kg /
(0) 111 16 20 L= > Kin 2.)i
=0 max |1 of of (dlbo(66N23/310))
(1) 2
= 0 max (11 12, d. of v da (og
|1 || 112 = 011111220d))
1 1 10 0 ,
joj E [M]M[]] n' E [N] and t E Tol.
The proof for Lemma D.4 is in Section F.3
Note that o I minillulle?, (o)d) d [log(6N²M//8)) thus O max 1l of of vd, 1og()N2MM//6)) = o(1)
Lemma D.5 (V's Beginning of Learning Signals). Under the same conditions Theorem 4.1 there 10M(33/++)))
exist Ti
such that the first element vector X,, W. wo dominate
nd (0M0B33-)MM33333)) lwoll
its other elements, that is, v 2 3M 3M-IV for all n E S.. [1]
[V for all n E S4, i [[M][L and v < -3M - [V
Proof of Lemma D.5
Let C' be a constant larger than 10M + 1), then as long as N SNR Z 10M N and further get
N 1 odd
(3M we have N N| 3| 60M2C3ofd
C
10M 1)N
T
nd (2 - 60 63MCC303d)l l2
nd || w
ndd || woll
which satisfies the time condition in Lemma D.3 and Lemma D.4 we have
Then by Lemma D.I and Lemma
(()))
+ oo)),
2
M o(1) s so (g ((())))) -
1
+ o(1)
M
M o(1) < so softmar( (g (1) (0) <
1
1
+ o(1),
M
M o(1) < softmar (4 (0) <
1
1
(1))
M
1
o(1) s softmar((q
(1 <
+
M
32

---

for i,j E (T), E [N] and E [0, Ti]. Plugging them in the update rule for (0)
VV
showed in
Lemma B.2 and we have
(1+11)
(t)
V.+
7
1 Σ
exp((4+) (6) k()))
NM
M
neSt
exp(1a+ (8) 1)1) + E exp((gt kkppk)))))
h=2
expl k)4))
(18)
M
exp( 1) + Σ
expl (un (
k=2
N
(5 =
M
1
=
NM 4
M
10M
Then by Definition B. and taking a summation, have
V (
2 - Ti
10M
13M + I)N
n 12
= =d
wooll
10M
(19)
(3M + 1)N 1
d%
d [N
60M C3o3d)
Similarly, we have
(TT) < dr
(3M I)N 1
(20)
(N
60M
Similarly, by PVv., PV.n,
(1+1)
(8)
2n in
we have
2no d N
(T))
I ||1001
+i
1
=dp+ 7
10M(3M + 1)N
2no
+
Iwoll
nd (4 60M2C333d)
woll
N
(21)
20MC + 1)07d
dn +
di [N 2OM
According to 19
20) and 21 easy to verify that v (T) 3M A.M.IV--. (T(Ta) 2 0 and ((T)) +3M
[v(TT]
< 0, which completes the proof.
D.2
Stage IT
In stage II, (q+, k4), (qr,t k4) grows while (q+ knng), (qn.is kng) decreases, resulting in attention focusing more and more on the signals and less on the noises. By the results of stage I, we have the following conditions at the beginning of stage II
(T)) 2 3M [/ /b
(TT) < -3M [1()))
(T))))
[v(T)) [((ii O(d, 1)
33

---

(o ( - 1 1.1.( (a - 1 (1) | Ka 19
= =0 maxflle pم l/ ofd. oo v dp log(6.> -
T1) ( (Ti)
وم 11 ozd] of vhh logg (oooeaaa
q ( | 1/ (.?
47
= o max
(A k ااااتتية
n..
(Th)
= o max
ofd] of dh logg
(Ti) 11: | 1/ 0(11m1303d))
1
| 11 1 1 o(oRoRddn)
(Ti)
for E [M]M[]}, n, n' E [N].
Some of the proofs at this stage are based on the above conditions. Notations. To better characterize the gap between different inner products, we define the following
notations:
denote 2.t (a k ( (q
(1)
S+.
denote (*)
n SES. S+. denote A n. = (g k (0)) (q (
- = (q
(1) (1) (
denote A p, (g k (11)
(g
n S_.
Lemma D.6 (Upper bound of V). Let To = 0
naa wo 1og(6N
Then under the same
conditions as Theorem 4.1 we have
|V
( | o(1)
fort E [0, To].
The proof of Lemma i. similar to that of Lemma D.3 except that the time To is changed.
Let T2 = @ 11 wo 1 (o(6N then by Lemma D.6 and Lemma D.I we have 1 o(1) s
29 S ] o(1) for [M] E [Ti, T2], which can simplify the calculations of a and / defined in Definition B.3 by replacing e((0)) by their bounds. Next prove the following four propositions B(t), C(t). (t),, Et) by induction on t for E [Ti,Ty]:
. :B())=
v 2 7/C3 oo [201 - Ti)
v(0) C 1M wo 112(1 Ti)
v 2 3M -11
((0) 5 3M |r
| s 0(0 b) + maallll woll@wlll Ti)
v. 00 4) n 20 Ti)
i [M]11), [N].
34

---

C(t))
ll o || - ((lulBoad))
1d 2212 6 (of0)ddn)
1g 9 _
- o(1); d(),
( 4 1
for i,j E nM E [M] i # or on
. DD()):
(q (++1) 1 (0 (
(gm k= (+11))
( (++1) k s (ql
(g knj (lt)) S
(++1)
n CC 1 wo B2d
n.d.j 2 log exp(A (T))
(t+1)
n.i.))
(TL)
+
(1- -T+)))
N (1og((6N2233/8)
CCso3dl 1 ww B2
(t+1) 2 log (exp(A(Ti)))))
(11 -1- (1-T))((-+++))
((1og(62222//00)
for [M]
=()):
KgL (0) k M. ( 10 1 1.),
(1) 2,11 S log(d))
K 2 =1(1)
for i E [M]VLJ,,,, n, [N] n + Tr.
By the results of Stage I, we know that B(Ti). ) C(Ti). E(Ti) are true. To prove that B()), C(I,, D()) and (L) are true in stage 2, we will prove the following claims holds for E [2]:
Claim 1. D(Ti), D(L 1) = 1
Claim 2.
1) = D(I)
Claim 3. B(T))....,,BU D(L -(
E(1) = C(1+1)
Claim 4.
= E(L+1)
D.2.1 Proof of Claim
By the results of Stage I, we have
(g (T) A
(T)) )1 (9 n.j M (g k y. Kg [ o(1)
(T)) (T)) (T) (T))
Assume that D(Ti). .................. D(t 1) (t (T)Te) are true, then ( k())), (g.0) ( are monotonically non-decreasing and (a()) E [ 1), so we have
(s) b.)) ( (l)) are monotonically non-inereasing for
(a ( k() (.) ((m 2 - (1))
(c) (1g) (d. (33 < o(1),
35

---

for s E [T..t]. Further we have the lower bounds for the attention on signal μ== as follows for
E [Ti,)] t]:
softmar (19 (6) ,
exp(-o(1))
(M exp(((1))
1+ (M - 1) exp(o(1))
(22)
1
1+ (M- - 1) + (M 1)o(1)
M -()),
where the second equality is by exp(o(1)) = 1 + o(1) Similarly, we have so (g. k(()) 2 M - -1). Plugging them in the update rule for vi+ showed in Lemma B.2 and we have
(s+1)
Vv+
7V+
1
exp((4+) (8)
((()))
Σ 0
NM
exp((9 (0) 1)))) + Σ exp((a ,((")))
M
nES
k=
exp((an.) (())))
(23)
j-2 exp( ( an.j.
((01)))) + Σ exp (nn. (1(1))))
k=2
2 N
1 o(1) M 1
o(1)
NM
4
M
10M
for (]. Then by Definition B.I and taking summation, we have
v((+1))
(Ti) + Ti 1) 001 |
10M
(24)
2 C | / wo 11:(1 - + 1),
where the last inequality is by (!T)) and M (1). Similarly, we have
((++)) < C 1 1 wo |/ Ti + 1).
By PP Pivmal
(1+1) (2)
2na
in and taking summation, we have
2n N
2n pd
(v(++)]] < |V(TTi) + (t Ti +
| < n2, | (1+T
|w0
N
(25)
Combining 24 and 25
we have
((+1)) -MM ((+1)
2
2 v (T)) + (t- - Ti + 1)
woll2 3M (|v(TT)) (t-
[wo
12)
10M
n.
N
(26)
N M 60M20803d)
((T)) 3M | (t T + 1)
10NM
0,
where the last inequality is by (T)) 2 3M (T]) and requires N SNR > 60M The proof
for (e+1) < -3M (t+1) is the same.
36

---

Next, we prove the upper bound for V+ and Va Based on the upper bound of attention(<1) and En S have
M Σ
(a+1)
(8)
Σ
W++
S VV
NM
nES,
5=2
3ry A
2
(27)
(8) $ V
+
S vv. + +n/C's
1A//
Then get that
v+ (++1))
S (T)))
+ +7,,tt (++1) (Ti)
-v,+ wo 2
S ((T)) + Calle 2 1oo
(28)
- o(d + C wo - wo
where the first inequality is by the monotonicity of rV and the definition of V4, the last inequality by the result stage I where v((T) 1). Similarly, we have
( O(d, +) + 1 wo lāat Ti + 1)
(29)
which completes the proof for the upper bound of V+. Expanding (55) yields
2noRoodd
S |
| 12:(1 Ti + 1)
N
(30)
S <0(d
n C
l Ti + 1)
where the last inequality is by the result of phase I where ||( Od,, 4) and the condition that
N SNRR2 > ((1).
D.2.2 Proof of Clain
By the results of F5
we have the dynamic of (q, k) as follows
(q (1+1) k(-
(q ()) (()))
n Ce 1 [ wo 12 Ti)
(31)
N
exp(A
(g (0+1) ا(ا+1)
و (b) (())
C μ | wo
123 3 dn (8 Ti)
(32)
N
(x (4)
m.
(q (o+1) x
( (8) 2(0)
1/ (ooffdl1111 w wo
fo Ti)
1
(33)
N
exp(A
(q (0+1) . - (ه (3) k1n))
70007dd11115 wo 20 Ti)
1
(34)
N
exp(A (+)
37

---

(CooBdl wo N
1
(35)
exp(A (s)
an.. k!ا)
@ w ds Ti)
da
1
(36)
exp(A (8)
n.d
(0-1) knn k(tttly (a... ( Goof | woll Ti)
(37)
N
exp A n.tjt
for 8 [Ti, f]. The seven equations above show that (
(q.. are monotonically in-
creasing and (q1 (8) k. (gn,i (()) are monotonically decreasing. Next, we provide the logarithmic
(a+1) increasing lower bounds of A #, and "
By 3 and 33 we have
(++1) m,tt R = (19 k_ (
(s+1)
(9 (b_ ((94) (a+11 A - -
1 Col μ
wo 1 dn - -T1)
N
exp(A!1) ()t
1/ Gooffdl
fo TT)
N
exp(A
(38)
1 1ad
-i))
exp( (1
|C Tı)
(1oog66N22M3//00
exp(A
where the last inequality is by of 2 min | - (0 -1 I1d (log(6N2M2/8) Multiply both
sides simultaneously by exp(A (8)
(.) and get
A 2 7 11. Ti)
exp(A R.tJ
(++1)
(39)
N (ogg(6N2M2/0))
Taking summation from Ti to and get
(+) (A
( Aotis - -1.t1)
" Ba Ti)
(40)
N N (log(6N22M/0))
2 p wo Ed
(t) 1).
N (1og()N2M2100)
By the property that A (0)
t monotonically increasing, we have
exp(x)dr
exp(A (+) n.t,) (A (+)
(41)
[ (t Ti)(t - +1+1)..
N (1og(6N222/88)
38

---

By exp(x)de -
expp=)dd=exp(Ap-tt exp(Anit?) exp(A we get
1 2 log (xpp(A + wolld -t (t-T)((t-++1))
(42)
((oo((NN2M/1/01)
Similarly, we have
(T1) pC wo 13d
(+1) 2 log (xpp(A
(t- T,
- -T++))
(43)
(1og(GN22M²/0))
By 35 and 37 we have
(++1)
((qn1) .k (((tt))
( k( (1q (xt1))
1 (.))))
Goof ( dn - -ii)
1
exp(A (x)
17 Coofh p wo Bo Ti)
1
+
N
exp(A n..
17 Cr max 11 12 33l Ti)
exp(A n,i++
wo Tı)
1
(10g(6N22M3/6))
exp(A
(44)
Then using the similar method as for Ant... we get
A.i.o log
(++1)
(Ti) (x
(t- Ti)(t Ti T1+1)).
(45)
N ((og(BNN2MM/)))
which complete the proof. The proof for Claim 3 is in Section F8
D.2.3
Proof of Claim
By the results of F.6 we have
1C exp(1a ,1(())))
(g
(s+1) 14 x(ot1)) - (())
(46)
for [2i,4] t). Further we have
@ColllllBoffdn exp((e(.) ,((")))
exp((g (s+1) k a((t1l)) s cxp
(( (((())))))
nhall a 100
exp (aa n((i)))
(47)
exp exp((g!") ((1)))
$ Cu exp (gg k(1))
S Öminn (ofd) d, 2) A (Ti) = o(1) and the monotonicity < exp(o((1) < Cn. Multiplying both 1201))
For the last inequality, by " min(l1eell? 2 (o (log(6N2 M² /8)) (q
of (g( ( for E [Ti, t). we have exp
#Colllul fa exp((g)))
39

---

sides by (aaf ( (a. k( simultaneously gives
expl(a+ ( b((+l))) ( (aaf 1e - ( k (at x( (laf _ - (af 104)))
S nn exp
(48)
s 111
where the last inequality is by plugging 46) Taking summation we have
(0)) ((11)) exp(x)dr
exp(1e (u+1)
kt
((ttl))) (la blltt)) (x4))())))
xc)))
(49)
1/ialllll1 ==
s T2 n (112ll11220800
Mo
where the first inequality is due to (q k (x))) is monotone increasing, the
last inequality is by T2 = 1(1~1|| 2 wwwlls 1og(6N2 M² (2/8)-1) and of S
10 (1-1) a ,(th))
min milll (ozd)-1)ed) (log(6N2M* M² M2/8)) 3.
By
( (ft)) exp(x)dr
exp((g+ (e+1) ((+1))) exp((14 (T1) k. (T))) we have
d log log(6N2
(q (1+1) b((t1) s log (af (T)) (T)))
s log (ok).
(50)
By the results F.6 we also have
(g (((+))) ((ot1)) - (q k()))) 1Coollu1100dd
(51)
exp((g (8) (3))))
knj (6t1)) (qLh)) (a) ((()) 2 201002d1141100
(q)
(st1)
exp((ae") (0) (.())))
(52)
N
(g/0t1) (00t"),,(((+1)) (((+1)) (g..)) (())))
(53)
N exp( (esp((g/1)) (((",,(()))))
(go (())) 2 nf of (.
(g (s+1)
(5)
exp((a (1) k())))
(54)
N
Then using the similar method as for (q (1+1) (((+1)),, we get
( ((+1)) < log (ah)
(gt ((+1) ((t2)) - log (dh)
40

---

In.i ((+1)) k s log (12)
(n, ( log (dh)
(1+1)
Next we provide the upper bound for (94 (1+1) have
(1+1)
By the results of
F.7 we
19
Σ
|3(0)
| p
oN
1++++
+.+b
71
T
s
(55)
i [M]{[),nEsS
an.t.b
Σ
la (0)
- = (N--bd6
(56)
s=
for E [M]{[],,nes_
Σ
[0(+)
1
Σ
[g(+) 1 = -O(SNR. v-
(57)
s=
s=7;
f [M][1], n E S+.
la anitt
Σ
la (a.11-1
(a)
Σ
aa
(a)
Σ
[3(6)
(58)
Tm Tj
Ti
for E [M]\[1] E St.
(8)
[8(0)
n 0a (o
1og(6N2 1og(6N2(2210))
(59)
41

---

for i.j E a, E [M * n'. Plugging these into the update rule of ( (t) ) (g & and assume that propositions C(Ti)...., C(() hold, we have
14 (+1)) k (1+1))1 $ k( (Ti) k
(q (5-1) x (0t1)) - (/ k (
+
71
1(44 (21) x
M
nES+
Σ nES_ 1=2
M Σ
(5)
RES+ i=2
(BL)) (T)
RES 1=
(TT) (T))))
(60)
+ Σ M
+ Σ |44.-11
(k: 7,10
n,
nES_ i=2 s=
M
+ Σ
(36
1
Σ
Σ
[8(()
(g -
(3)
s
nES x=
+ flower order term)
=1(t (7h) k (
ON ( +) .01)+ + N. M 0 N ( .0(1)
0 N ( )) .0(1) NN M. AMMMMOOSSM O(SNR. N t o(11)
-1(44 (1)) k (()))) + o (vbdd )) + o (SNNR. Ntd
o(1),
where the first inequality is by triangle inequality, the second inequality is by 213). the last equality
by (( (Th). (T)) (T)) = o(1) and d, = ② maxx(SRR) SSNR-4)N2
N2N22 -2). Similarly we have
42

---

(g_ ) = o(1)
9n.i 1411) ا S Kg k x +
(21)
(gnstt) kns bott (q (222))
(Ti) (
N
M
anitt (K+ (()) + (8)
(e)
(k ( +
an,i
n'
1=2
+ m.j.t (9 (3)
[nn.s) + 2
In.s)
P a (e nn.1)
(8)
1=1
N
M
44) + +an
(6)
(0) +
nnin' , k
+ a
n'=1 1=2
((5) (T)
+ - -LoTT
(0)T
J. I 9 ())
T.s+ 94
Mnnd (())>>1
(T)
[1< ( +
Ja R,,
[1( الال)م
(c)
(t)
(6) [133))1
m
R.j
2
(61)
n ,Il
(8) (())))
n¹ Mn (1/ n/ x5)
s= 73
| M...
1 11(g (0) -g
n...
s=
g (3)
+
[
R.j.a.
(3)
1=2
[g(0]
Rij.y 2 (9) 9.33
n' #n)
flower order term]
K (Tt) (TT)))
+ ((rtt)
o(1) + 1 10/ O(03oRddh))
+ M. 0 (1h H) o(1) + . o t 1ogg(oo 16)) o(1)
(N(Nd -01
= -aatt A > + of (11))) )
+ + (a bd of Nd bd log(6N2 12)
= (1))
where the first inequality is by triangle inequality, the second inequality is by (196). the second (o)d)= dr ((og(6N222//8)) - the last equality is by (T)) <((i>>) = o(1), ((--2h) ) and dn = ō max SNR--JNN -)N22
equality is by of <
(g
43

---

D.3
Stage III
In Stage III, the outputs of ViT grow up and the loss derivatives are no longer at o(1). We will carefully compute the growth rate of V+ and Vi,i while keeping monitoring the monotonicity of
(q. k). By substituting 1 = T2 = o
| into propositions B()), C(), D()), E() in Stage
II. we have the following conditions at the beginning of stage III
( 11 (T)), ((Ta) o(1),
((Ta) 2 3M. - ||(
(-TT) 2 -M |V[
19 1 | (3) 1/ = e (11113
||
(Tb)) 12 1 11 = e (ddd)
(g (Ta) q (Ta) |1 Kg (2)) (Ta))L M. Ka TT) (T2)
9 - o1))
K b (T)a))
( (Ta) ((3) o(1,
i, E [M]M n, n' E [N], i # or n tn'
at N(oog(6N2 M22 (03))/))))))))))))
(72) 2 log (cxp(A (Ti) + +
oRddR
nnid 2 log (exp(A
(1b)
(3i) + 0
11 (og(((/N00))))))///)))) ))
(gtt k (T))) Ka (Tb) knn 1/1 Kg
(T))
(T2) (T)
(T2) (B))) s log(d))
( (Tz) ( |. (0.0)) |
(2b)
for j E [M] n. TE [N) + n.
Let T3 = 0 कpا 1
Next we prove the following four propositions F(t). 9(),, H()).
by induction on t for E Ta]:
. F(():
Vi-h 2 3M |V(1)
((t) s 3M [|00]
||00| = o(1),
11 -Ta) S v/1² 2o (0 )) log (exp(V((a)) + 1/111|1111111 //
-2log (o(²)) s 0 < log (exp(-v(Ta)) + n 11 1 13)
for i E [MMM),, [N].
. :()):
ll ||a1111111111 @(1lu130ddn)
ll 1420 ( 03
kg S.P.29)11 " -
K A R (6 11 ( (11)
for E (M)()),,, E [N], # n
44

---

H((t)
(9= (+1)
ky (((t1) 2 (9]
(0) k
(nd k 1++1)) 2 (m.
(++1)
(q (+1)) kn ((+l)) s (a
ann
(++1) k = (q.))
e E MM
I(t): 1i 10 &. Anjil Kg & A log(E (d))
kg k >1
x0)1 = o())
for f, E [M)(1), E [M]. n + nT
By the results of Stage II, know that F(Ti), G(T2), I(T²) are true. To prove that F((), 9(1), H(t and [(t) are true in stage 3, we will prove the following claims holds for I. E [72, Ts]:
Claim 5. H(T2),....
- - F(L+
Claim 6.
((, - H(t))
Claim 7. F(..)....,, F(t), (L.T)) H(t 1),T(T))......... I(t) = g(t 1)
Claim 8.
((7)). -((1-))
- 1) = I(1+1)
D.3.1 Proof of Claim
The proofs for V(O 2 3M [V and ((0) --M | are the same as for D.2.1 Based
H(T2),.... ..., H(t) where (q1)
(a) k(()) and (g..) k ( monotonically non-decreasing and
max (1) (0)) k max
(9) (())) monotonically non non-increasing for [T2, t 11, we have
A 1 2 A (Tz) 2 log (exp(A
(T)) (T))
n. + O
(62)
N (log(662M3/8))
oz
1 A... A A.i.t. 2 log ( (
(63)
| log(6N2//2/80)3)
i,r E (M]{(1), E [N], E t). We further get
exp((q (1)))
exp(1e (8) ((1))) +
Σ
M
exp((g (8)
y=2
exp((gl") (5) ((1))))
C Cexpl(at") )
Cexp(A
(64)
Cexp(A (ATT)) + @
N (og(6N+
0
N (IoR(6N7M3/0))
N W1og(66N22M) 2?? 22/)) 18))
0
dt
45

---

For the first inequality, by 66 and the monotonicity of ((00,,())) (if AC(")) is increas- ing and (at k is decreasing), there exist a constant C such that C'ep(( (x) )(( z exp(111 (5)
The second inequality by plugging 62). Similarly, we k('))) + E exp((q (a) ()
M
p=2
have
exp(( k..)))
exp((an..t 1) +
Σ
(o) expl (an. ((
-
(65)
Cexp(A m
=0 NI Nl (og(
ofdd
Plugging 64
and (65) into the update rule of Pv.. in Lemma B.2 and get
(x+1) PV.n
N 21/ 1og(6N2M3//8) (L(3 + +0
7
|
115m (o
$
NM
dz
ozddž
N ((00(66N2M//00)
M2 ))
1
Σ
1 (w (o
+ +0
NM
dz
0z
#'iiz!
3nd3d
(0 (0 ( (11o((0N22 + +0
N (log(622)
2NM
d
ozdd
M 20, 283VVoog(NN222M// (o (V(00g(6N22MM110) )
N (1og(())) /8))
+ +0
NM
a
ozdd
2n
(0 No Nogd( + 0 N (og(G((22 /61)))
NM
dh (
db
-0 no log
(og((N22()))))
d3
(66)
where the second inequality is by Lemma C.4 and (00) I 1. For the last inequality, since d
( N2d we have N M 263 d < Lozd. By Definition B.1 and taking
summation we have
(1(+))
1 (T2) n,t
(a+1) PVin.
S
+
s=
T2
< n,5
+ T3
(1+1)
Pv..al ||wo 12
nofa( (
+ nllell (0s(6NNM2//0)
50(() +
||wol|2
176 1e wo |23
di
d
(1og(6N2N///8) edj
o(1) 0
=o1)+ +o(1)
= o(1)
(67)
46

---

where the first equality is by N SNR? > 2(1), the second equality is by dn á max SNR SNR= Then we have a constant upper bound for the sum of Vngi as
follows:
| = (M- - 1) o(1) - C1s
(MM)))
for E |M] [T2, f].
Expanding (64) and (65)
we have
exp( ( (
0 N (1og(6N2M2//0) = o(1) (68)
M
exp((g (1))) +
Σ
exp((1 k..)))))
di
j==2
where the equality is by dn = ( max/SNRR, SNR 11N22-2) N 1 log(
expl (0) (11)))
o(1),
(69)
explig (1)) Σ
M
explign. k)
oRdd
where the equality is by dn ② max SNR -)N22-2) Then we have
softmar (9= (+) kiz (1))))
(M- 1) 1) o(1) 2 o(1);
(70)
softman (1n k4 ス 1 - -M o(1) 2 o(1)
(2)
(71)
for i E [M]\(1),n n E [N]. E [T2, t). Next we provide the bounds for -(( Note that {(2) = log( [1 (-xx((=)2) and -f" = exp( - (1 21 (1+ without loss of generality, assume yn 1,
we have
1
((((Xx,
(72)
M
1 exp( 1
Σ f(
w
w (T)T (X,) W wo)
M
M
where (x
Σ
w
W ( (X₂) T)X, W
wo can be bounded follows:
M
M
p
w
w (Xn) T X w wo
M
e
1=1
1 ((50stmarc
(g (8) {(()))
so softmar (gg.)) (8)
((1 -1. W wo
M
M
Σ (softmaz((g) (6) (0) +
so softmaz (gg.)) (6)
( (()))))) S.. w wo)
(73)
JEMM]
M M (1-0(1)) v(0) + M o(1)
1
v(o)
JE(M)V()
((0)
2 2
for s E [T2 t). The second equality is by plugging
688
699
70
and
71 the inequality is by
((()) 3M (v(6) for s [T2, Similarly, have
M
Σ
p(x n W
(OT
(X,)
)x w wo
M
-=1
< max (v(*) p(s))
(74)
aE(M]11
(s)
47

---

Plugging
73
and
74
into 72
we have
+ exp(
M
1
Σ
M
p
n.
w (xx))) x, ( wo)
w
Q
W
1
(75)
+ )
C16 exp(V
where the first inequality is by plugging 74.. For the last inequality, note that V (72) monotonically increasing, there exist constant Cls such that 1+exp(V11)) have the upper bound
0 and V is
Gie We also
exp(V("))
+ exp( M Σ
p( w wh (TxT)( (X,) w(() un)
(76)
1 + exp(V /2)
1
exp(V /2)
Plugging 770)
and 75
into the update rule of (8)
VV. and in Lemma B.2 and get
(a+1) V+
Tv.
expl(g (8) (1)))))
Σ e((0)
NM
exp((14 (9) (1)))) + Σ expl(el (1) (((h))
M
nE8_
M
exp((gn.). (5)
k(22)))
M
J=2 exp((gn.) (8) {())) + Σ exp((gn b(
(77)
k=2
Σ ( - o(1)))
NM
nES_
C16
(1 - o(1)) (1-o(1))
N
e
1irll
(xp(v(oo)
where the second inequality is by (75). Then by definition we get
(( -10-))___66 v(6)
- 11 (117||111@@ww)
(8)
exp(V\(")
Multiply both sides simultaneously by exp( ((())) and get
exp(V(")) - v((l) 111rllal8]@0013
48

---

Taking summation from T2 to and get
(V
-v(")
(78)
[C1rllm4@@| 2012
72
*Ciillll wol T2 + 1)
By the property that v( is monotonically increasing, we have
v(+)
y exp(x)de
(v(ott)
(79)
2 C wo 1oll T2 + 1)
victa) By fyon exp(x)dr= exp(V((+1)) exp(Vi get
v((t2) 2 log (ep((v(2)) + n 11 T/2 +1)
(80)
Similarly, we have
v(++)) log eexp(- (Ta)) + C 11r11ul|@@ - T2 1))
(81)
Next we provide upper bounds for v((+1) and ((+1) we have
By the update rule of (0) vv. and in Lemma
B.2
(a+1)
-
VV.
exp((g))) (R(()))
Σ (())
NM
expl (x) x(21)) Σ exp(((()) ((2211) M
NES+
(5)
M
exp((gn.) k4 (1))))
1=2 exp (xp((g x42)) Σ eepp(1nn)) (5)
((())))))
((s) a
M
NM
nESt
3N
N
exp(V(()/2)
37/111112
where the second inequality is by (76). Then by definition B.1
we get
v(o+1) v(0) (0(2)) - No. 11
(e)
31/ 1
wo 1
(82)
4exp V /2)
Further we have
exp(V(at!) /2) s exp(v!!) /2 31l1m1121100l1
Sexp(Vi() /2)
= exp((\(0) /2) exp( 8exp(V (9) /2)
3nl 14 /3 wo 12
(83)
5 ( /2).
49

---

For the last inequality, by 1 S Ómmin () .d, 3) of (+() we have exp( 3n | l22 I exp(o(1)) - Cis. Multiplying both sides by (viot1) /2 8exp(V+ /2)
= (1) and the monotonicity
(1)
v v+ /2) simultancously gives
exp(V/e+1) (2 - C1s exp(V /2) < (2
(84)
3nCB1allu@@11000 woll
8
where the last inequality is by plugging 82
Taking summation we have
1 exp(x)dr
exp(V /2) (v (v(0-1)22 -2v1 -/-1 /2)
37 wo
(85)
8
<= T2
31/C || wo
θ
n7 1
12 By 12 exp(x)dr exp(v /2) exa((
/2) have
v((+1) < 2lo (exp(V((") /2) o(2) log (0(5))
Similarly, we have
v((+1) > -2og (0(
D.3.2 Proof of Claim
By H(T2), ... H(t 1), we have softmar(( (1) k((1))) coftmar((gg. (0) x0(1)) o(1) and (g
softmar( ( (1) (1 ), sof poftmar(('g.. (1) (()))) o(1), which have been proved in D.3.1 By the
results of [44, have the signs of and B as follows:
4 0 _0 - ( 80
(0)
(0)
300)
300 2 0,
ai n.
11)
(6)
(
an a n n Pnj.m, 0.
50

---

Then combined with (T) and we have the dynamics of (q. k) as follows:
11-1)
(
(8)
14 k
(1)
n
+
11
10)
nES 2
(86)
+
(0)
(t)
(1)
n.t n,))
NES 2=2
(OT
3(0) t, In
()
+
RES+
(0)
112 + (
1 1/22 + &loweer order term]
0
Similarly, we have
(q (+1+) 1(++1)/ -
_a (0) , 2 0.
(87)
In 1 ((+1)} - (0 b19))
(+1)
N
M
=a n [ 112 + a n,i
(1)
(4
i==1
M
+ +/00 (4+ In.il
(t)
((t) 1 (1 )
n'ES+ 1=2
N
M
(88)
+ a
(0)
(1)
n, 1000 +
naia W1)
n==1 1=2
M
(0()) (T)
Σ
Σ
((0)
(T)
W'SS 1=2
112 + ( 1l 10 112 + flower order term)
2 0.
Similarly, we have
(gn
(1+1) ((+1)) - (4.00 (00)) 2 O.
(89)
51

---

(q ( ( k
M
=f (n)) + Σ
Σ
a
+
n'eS+ t=2
N
M
800 1 11 112 + (
(1)
-
(9) (0)
.9-1
+
3
(0)
n==1
M
(90)
+ (aff 6 +
n' k
E' t=2
N
M
10 (0)T
g(0)
(1)T
9 paam'1992
(0T)
Pnjjt a+
+ PPn.j.
n nig + 9 1 + flower order term]
0
Similarly, we have
(a- (++1) 111t1)) - (a
(2) (2) S O.
(91)
(nni knj
(1+1) (0+1))
(In.i (
M
= a niit (14 b!1)) + a n,i,
(0)
n a mid (
(1
M
+ (00)) +
(9+ -n.i) + -
(6)
(q
((t))
Pij.n a 19 9n.))
n 2
(one,
(6) 10 (0)
ani, k
(0)
(1)
(92)
n!
N
M
(g(t)
(+)T
g(t)
(1)T +
Σ
3(0)
(t)T
P+ + n.j.
In'
n'=1
Onamm n- 112 + 8(1)
(1)13
flower order term}
0,
which completes the proof. The proof for Claim iss in Section F.11
D.3.3 Proof of Claim g
By the results of F9 we have
(at x: ((+1)) - "q k(0) < @C1ollul@oodd log (0())
(++1)
(93)
exp((g !1) k(
Further we have
((+1) ; ((+1)))) s exp ((0 k00) + n (1ollll1 f dn log (0(!)
exp((g((
exp((g4)) (1) (0))))
11 log (0(
=exp (ig rc(1)) exp
(94)
exp((g!) k()))
S C11 exp (a b())))
52

---

For the last inequality, by n S (mlnn (ozd)=1) ((0 dr of s min(llullz (o d à (log(6N2 (og(6NN22/00)) / 3 (a) (Ti) k o(1) and the monotonicity of (of k44) for s e
(7,,4, we have exp 10o Re lox (o(t) s exp(o(1)) Cn1. Multiplying both sides by
exp((1 ,
(l4+t), x k - (4t x())
simultaneously gives
exp((at ((a k4 {(t"ll)) (af (ot++)) *(ttl) - (at (1) k(.)))
s Cn Cn (la k()) (la!
l(ttl) (a (0) k((1)
(95)
<nCilllul22 log (0(
where the last inequality is by plugging (93). Taking summation we have
exp(x)dr
19 (
exp((g(+t1) (lttll)) (gl(tt), 1.0t2)) (9 xx()))
/CC2lll41 log (0(
(96)
<
==
S T3 (121llu11205 log (0
=0 d log (O(
(a) 4 is monotone increasing, the last equality is by where the first inequality is due to (af")
Ta 0 ---
f ) ") |wol 12 @(1) and o S min(llull (ozd)) -1 dh exp(x)dx exp((9+ k (+11) ((+1)) - -exp((1) (Tb) ((aall))
(log(6N²M1/88) - By S
S/ (22) x
we have
( (TTT)))) o( di log (O(t)
(g((+)), ((+11)) s log (exp((9 (Tb),
s log(e--dd b (97)
e(log(6 (6.2 2 /2)
where the last inequality is by (T2) k+ ((2)) < log (dā ). By the results of F.9 we also have
(++1)) xs (1+11) (a) (0 < 71olll fo dn log (0(
(98)
exp( (0))))
(q (++1) (((t - (T) (2) > n02u0 1 120 f dn log (O())
exp((g (1) (.))))
(99)
N
C1o0 dd 4 10 log (O(E))
(((11) (((-1)) (9.0)) k(0))
(100)
N Wexp(
11) ". (1)))
01000A oo log (0(!)
(241) k/ (((41)) ( (2))))))22
exp((a (.1)))
(101)
53

---

Then using the similar method as for (q (2)) b we ger
(a ( ((ttt)) S log(E -1 -Idf).
(9= (++1) 1 2 log(t -dd))) _kn
(n. ki 16(t1)) s log(c= dt))
(t+1)
( knj (ttt)) 2 log(( 1og(c-1d)))
Next we provide the upper bound for (4 (t+1) k (((+2)), (a (e+1) ( By the results of F.10 we have
( (log(6N2 N2
SNR2
log (O(:))
Σ
(
- = 0
(102)
edi
s 73
T2
for e [M]][]),ne S=.
la
-
Σ 1
Σ
1800
18
(
(
n,J,
s=
T=
s=12
s= T2
==
(103)
N (log(6N22M* M 18)) log (O(2))
ed
for E [M]][]], S+.
la n. abb
la 1
Σ
la n,d
o niin
Σ
[g(0))
==
T3
s= T2
T2
T=
(104)
0 [log(6N2MM2 (1)) 3
log (o(t))
edp
for E [M]((1).nES..
la(0) n = (10g(((N (/2//0)))0))))))))) (10)) 'og (o(t))
(105)
ed d
54

---

for for E [M] n'n E [N]. # n'. Plugging these and proposition ((t) into the update rule of
[ (9 xx) <b>1l (6) k and get
19 (11), ا S 1ka= (T))
k x2)l
Σ
1a4 (((tt), (lot2l/ - (a ( ااا"
s=12 12
$ 1( (Ta) {
M
T2
nES; 2=2
M
+ (
nES
1=2
M
(0)
n,,, n,
NES+ t=2
M
(+)
((0)
In.i
RES 1=2
1(9 (22) (T))))
x
la aa (1: (1)
(0
(0)
nES T=
(0)
Σ RES_ 1=2
Σ
13 n,-
(g (2))
10)
=T
+ flower order term}
(T2)
(T)
N ((oog(6N2 M (2))) log (O(H)) -011) + N N--- M-O( (10g(6N22 12))) log (O(t)) o(1)
+ 0
edi
edī (Nng 12 3 log (o(t)) o(1)
N N(oog(6N2M' M (2/0)//80)0 log (O(+))
N- M = =I<4 (T22 k (T)) + of N (1og(6NN9MM/00 (8)) log (O(t))
(
edi
edi
N. SNR2 (log(6N2 M 15)) log (O(2))
+ 0
cd
cd
=(1),
(106)
where the first inequality is by triangle inequality, the second inequality is by 213). the last equality
by (g
(Tb) (T2) (Taa = o(1) and d, = ② maxxxxNxN "SNRR",SRR""""" SNR-4)NN N2 -2). Similarly we have
55

---

(o ) = o(1)
(9 (111) ااالام - [g (Tb) (
(x+1)
(an.. |
Un.i
==12
(Ta)
N
M
n,,,+ (14 ) + +
(8)
(k
((0) R.j
an (h)
==
W==l1=2
M
Po.+ An.e) + PT.). (q
((4) (5)
(4)
(4) (8)
Tuij.t An.. In.i)
1=2
N
M
(000
(0)
[( + t0p.i.
(())
(3)
+ miit
aniin n',1)
m==1 1=2
N
M
a(()) (s)T
Tx)
T
T.J.+ 9+
+ +.,,,,
n
(Ta) (Ta))) }
(4)
(0)
+
T=
M
+
ny
1
(1)
n,1
==
8=
T3
1o
(k
1 (
f nAl Mlzjvv'' V'v'' x
T=
|,(0))
18 Rij 11Ka d.
M
1a(0)
Anj.nall 2 +
(0
m.mn (g (9 222
1=2 ==2
Σ
Σ
[3(0))
Rigine 11 (.)))
n #TA #n) o=
(lower order term)
= =((ght)) (Ta) x(
( (log(6N2 18) log (o(t)) o(1) + 0 (log(6N2 log (0(2))
@(o3o7ddn)
edidd
+ +M- 1M 0 (log(6.22M)) M² 10) log (0(+)) o(1) + N M -0
(log(6N2 M²
/8)) 22
log (O(A)) d(1)
cdf
cat
/8) 2200)//80))) N NN 2
log (O(t)) (o(t))) :(11)
0
td
)1 + N (1gg(6N=M//8)) log (O())
(22) = =I(g R.j
(Ta)
edp
(Iog(6N2 log (o(t)))
of N /1108(6N2M////6) 1 log (O(t))
+0
tdà
ed d
=0(),
56
(107)

---

where the first inequality is by triangle inequality, the second inequality is by (16). the second equality is by oh < 11 -
(o d MP) - the last equality is by
(25) ( = o(1), d = 52 N22d) and dh = . max SNR N (gna))
Lemma D.7 (Convergence of Training Loss). There exist 'T'
by > such that
(s()())) Se
Proof of Lemma D.7
Recall (68)
69
70
and
68
we have
0( N (10og(6222/110)
TT) softmar( ((g
(108)
softmant (a... ((())) ( N (1x(01(0 0
NNNR (Iog(6N²M² 22 /2))
3
0Rdd;
(109)
(V(lo((6N2M (M222M23))))
softmar((d) A (0) = o(
(110)
d
N. SNR? NSSRR
(og(66222//0))
softmor((d (g k(()) = -0("
(111)
dz
Substituting = T
C nell wol
into propositions F(t) and get
v(1) > log
(exp(V n 117lr/ll 10 Tb))
z log eexp(V(T2)
C20
(112)
C2o
> log
|r00 O(1).
(113)
57

---

For 12 S+, we bound f(X,. n: 0(t)) as follows
M
f(xn,
Σ
p(a Wo WoW ), Wrwo
M
exp( (((1))
1
((
M
M
exp((1)
x411) + Σ Eep(((e (1) (2
1=2
exp( k
Σ
((T)
M
exp( (g (1) k (0)) + 2 expl k())))
1=2
M
expl ( (1)
(
Σ
exp((g (t) (01)) + Σ exp((a (t) (.))))
1=2
5=2
exp((gm.111
(1) k.))) k(1)))
v1l
(9. (1) + Σ expl
M
(8)
knj
(114)
N (10g(6NN3MP111)) M (1-0(
1
SNR" (log(6N2MM288)) 1)) log C20
+ +M- -[11, 1).
1
di
M (M 11). (0( N (1og(6N2M33/8))
+ +M - 1) -) N SNR² (log(6N>MM2/8)) M -0(1) (M-1)
di
N WSNRR (log(6N2 log Can
(220)
= log
d3
C20 logg(Co))
log
(
log
where the first inequality is by plugging 1088) (009 110 and 1111). For the second inequality, Eza
by dn = क max(SNRR SNR SNR we have O N-SNR² (log(6N2 /2/1))
log
o(1) log(1 + o(1)) log(C20) long C20 is sufficiently large. Then we have
e = log (1 + eexp(=f(Xx,, .(t)))
< exp(-f(X,,, ((1)))
(115)
log (t))
< exp
Se.
Similarly, we have (9 < for n S_. Therefore, we have Ls(O(T)) = Σ
(0) Se.
n=1
58

---

D.4 Population Loss
Consider a new data point (X, y) generated from the distribution defined in Definition 3.1 With- out loss of generality, we suppose that the signal token is μ++ and the label is 1, i.c. X (14. E2. ....................... ..... E.r) y 1. We will first give the bounds of attention, then the bounds of the output of ViT, and finally the bound of the expectation of loss.
Lemma D.8 (Bound of Attention). Under the same conditions as Theorem with probability at
least 1 - 8/2N2 M, we huve that
(T') exp(u
W
Q
W
K
(T&) p4)
1
2M
exp(u w (T)
W p4) cxp(
(
W (25)
w
(25)
Ei)
Proof of Lemma D.8
By the result of D.2 and D.3 monotonically non-decreasing for E [3], we have
W W TT)
p+ o(1)
and p WlO w
μ w (15)
W T
14 -0().
By proposition (tt) in D.3 have || W TT) 1 (11ul( In order to give bounds for (Ts)
&. provide bound for the F-norm of W w According to the properties of F-norm, we have
W (
[ (Ts)
w || S W (Ti) w |F + || (T>)
W (T))
|| W
(Ta) K w (Ti)
(T))
(T))
(22) |F (116)
Next, we will provide the bounds of I|W (Ti) w ||., I (T2) W
(0)
(T))
" and Ilk (Ts)
K || respectively.
Bounding | (0) Illt Recall that
Vw (s)
1
Σ (x))), -P., Wywop!
NM
nESt
X (diag((pn..) ) Wn We
Σ (X) (diag(Pmm.) [[))(x](diig)p - " Wywon
NM
nES
M
+ 1
X _ (diag - (T..))..))) WywoE., Wo.
so we have
[ (++1)
W []V (1) (1) fl.)) wo aFllllllu (' ||
Σ (1
(0) ||=](diagg((1)) 4n,1
NM
RES+
M
| || [[(((ia((() - pn.. (1)
( w woll w(("
1220 [I][( p.T
(
- X W wo w Iw("
NM
nES
+ M
X |F (ddiag(s
th (Q) (9) P W woll=lleT (1
(117)
We bound 1200)) |xallr, ([diag(yp!!) 100T IIETT (1 [[ respectively.
Mille |X W wolls. Il [ I| and
59

---

1 1
By Lemma C.4
| IF S
1 + (M 1) 30 = O max op vvv))
Note that each element of vector (Pn, has an upper bound 1, and the number of tokens has constant upper bound (M (())), then we have ([diag((P)))) Pn, (n O())
(4) (4)
By Lemma D.3 we have |V [V
1 1.1%
O(dh +), we have
.
\ (1/0)² Σ
(v(l))² VMM M (dr O(dh*)
X, Wywwle
1=2
* By Lemma D.4 we have l| W =
- @(Iull2ond)) 1e,, W ((ppondtdt)
Plugging them into 117) and get
[ (t+1)
w
||
7
NM- O( [1ax vay) O(d, 0max Vdyon
NM
O( max{||||13
12 ozd} -o,di)
(118)
Taking a summation we have
T
1
W
TT)
x
w((++) K
W IF
K
K
==0
<0(-
O( max 11 -nd)))
nd ||
1ur 12
(119)
max
| odd
-hh
|
< 0 (onn)
where the last inequality is by N SNR? = 9(1).
Bounding | (W(T)) (Ta) w K
|F= By E7 we know that each element of matrix (diag(<!..)
(t)
PR,1 ( have upper bound O
(t)
and each element of matrix (1 ((-T))(--T--)
1+
N (12)))))
(T) (diag(() ( - -nns (nn)) have upper bound O
Considering
14 1103d1m111m00132 (I-T.)
that the number of tokens has a constant upper bound (M (1)), we have
([diag(<)!.) pn.1 11 = 0
(1)T
(120)
1
(t (t)((-
N MM9)6)
( (t) - 1p.01 (( 0
1+ PRaddllal@@@wolll
- -1; (121)
(l (
N (ng(6N7212/88)
By B(t) in D.2 we have 1r'"),1VV'' - Odd, #) C7 wol((( - Ti) for 1. E [Ti, 72). We
further have
X, ((t)) (t) wolle 0
+ 1 3 - -T:)).
60

---

Then we have
[I(dia((e)))) pn,1 (1 X| Wi woll
=0
1
(dpx = Ti)
v2 v 133d (t Ti)(t - -
1+
(1gg66N225388)
= =O(N+d) log(6N M² 1og(6N22 /8).
(122)
The reason for the last equation is similar to 256 We also have
[[diag(()))))
(t)T 4 P.llle I, wollr
=0
1
(dhh m Ii))
1 #0330ll113l000003 (t 77
N( (10E(6N7M///8)
= =0 ||alla N'd, log(6.2 M2/8)) 1)))
(123)
Plugging them into (17) and get
W
(1+1))
(++1)
w
(0
ll
7
N (1axx(lllll, vddd)) (o(Nád,, 1og((N2MM O 011141220
K
K
NM
+ (M- - 1) 0 ||e||2 0011ll N2d, log(6N2
0 opvd
1 1og(6N2 1221))
(124)
Taking summation we have
T2-1
(T)))
(2)
|| <
| (++))
K
K
==
00
<
o (1max(114ll22 N |all (g((6222r/6))
14 wo 1B
max{11aa| op V N No 1og(6N2M M 18)
|/1//2
N <
SNR Don 2 M 18))
(125)
Bounding | (Ta)
w (22)
w || Recall that 64). 65
exp( (g (t)
N 2 /5))
M
exp((g (1) ( + Σ exp((g (1)
2-2
exp( (m (1))))
N NI (ogg(
expl(g
(9) 1 (0)) + Σ
M
expl(g!)) (1)
(1)))))
ozdd
1=2
i,j E [M]](1), E St, E [T2, T3]. We have
N (1og(6.N2MM)/)
(ddiag(p)))
(O) 2 0
(126)
di.
61

---

N M 1og(6N2 (o0)2M228)))
[I(diag (1)) - nnn,i 4
(t)
(1) =0
(127)
oz
By F(t in D.3
we have
|V 2og (0(2))
|V = 0(1)
further have
X,, W. wolle ollg (11g)(
Plugging them into 117 and get
10-0 (mam (max ( (11ull2ond;
(++1)
W | S
7
NM
di
N ( (1g((((2M1) /01)
+ (M 1) 0
Olo, ((onnd))))) o (o
ozddR
Nllulll (Iolllooo06N"
3 log (o
= =0 mallma 0,d]
apd'
(128)
Taking summation we have
T--1
w (Ta)
(Ta)
W
(Ta)
W (++)) - Ir
T2
17 max || oz N Mula (log((/22
log (O(#))
s 0
7/6
| woll 12
ond dz
N (1og(6NN2M /8)
log )0
o
mmxx{llulB o
e|| 20
(129)
Combining
116
119
125)
and
129 and get
W
w
|l - w
T1)
w
x |l + W (Tb)
w (1)) ||
I
w (Ta)
K
K
= (O))
0 (1
(1R 1og M² /6)
0 maax(111122 M(og((NN2M2/00) log (0(
<|I1|120,pd
=O(ondi))
(130)
where the last equality is by d, á max[SNR* SNR SSNSSSRRNNN 2 (Ts) Bounding p._ [.T((Ts)) (Tx)
E. We decompose W(Ts)
(Ts) into W
(0)
( (Ts)
wl0) then
have
T (T')
w W
(T))
&: W. (Ts) w T & &.+ (
(Ts)
W TTaT (T)T
(0))
14
Now bound μ W (T) W (O) Bounding 14 W
&, and μ ((a))
(Ts) w((TaT - T))
E. respectively.
(TT)
W
(O)T
E: Note that || (T)) 1 = (1Iu[139ddd) and each element
of W T is sampled from Gaussian distribution (0 of), have that each element of
62

---

is random variable with mean zero and standard deviation smaller than C21 C21l|u1|20ffd df Therefore, for any i E [M][[ by Bernstein's inequality, with probability at least 1 8/3N2M².
14 W. (25)
W (O)T
11 (T) W
K &,
/ =[1),
where the last equality is by oh I min (o $
( (1og(6N2M//00))
Bounding μ w (15)
W (T5)
w
(0) )E: By 130 we have
(
(Ta)
( }|} - | la
(TST W
( |r
Q
K
(131)
< <callulllooddh.
Therefore μ. W.
(Tb)
WK
( - WK (0)T
(Ta]T
&: is a Gaussian distribution with mean zero and standard By Gaussian tail bound, for any E [M]][1], with
K
deviation smaller than O22 p 12
probability least 1 5/6N2 M
w ){ s Cza 222 v 22 (12N3M3/5) o(1), 14 W.(T (Ta) W (Ts)
W
where the last equality is by d = (( (N and of S minill (o,d)) dr
(0g(6N222/000) Applying union bound, with probability at least 1 8/2N2 M,
(T)T &: = =.. w (T) wx ())
(T)T w E: & + u. (.TW(T
K
k
= o(1) - o(1)
o(1).
Therefore, we have
W TT)T K
exp(#
p+)
exp(-o(1))
(Ta) W T exp(p) [o([[/(TT)))
M+)+ + Σ explu
11
W (Tb)
wTTTT Ei)
exp(-o(1)) + (M - 1) exp(o(1))
K
Q
1+ (M 1) exp(o(1))
1
1+ (M 1) + (M 1)o(1)
2 M
o(1)
2M
which complete the proof.
Lemma D.9 (Bound of V). Under the same conditions as Theorem I with probability at least 1 - 8/2N2M, we have that
IET
W (Ty) V wo $
Proof of Lemma D.9 By 9
we have
W (1+1)
wo w wo NM
(tt) Un n
( WO Σ W( w X )x,]" wo
(132)
Σ Un (((t)
Σ X
W
((( X wol...
NM
63

---

Then we have
N
M
(++1) wo - W/
wo wol2
"
Σ
Un
(())
x fl/p((nm, W T x, 111l wol2
NM
n=
- NM
NM O max (1ulll2 (l vD 0(1) 1/wol
= O
(133)
where the second inequality is by (7 < 1, |x,||F = O [(mxx(lull2:0p and (t)T
w
K (x,) xx)22 = 0(). Taking summation and get
|/ (Tb) V
woll2 v W (0)
woll2
w (1+1)
wo w wollz
| wollz
0 0 max([l -p (1ooll
n7 1 wo 12
= = + 0 max (1aaa11222 (
<||ma|1
= =o (ax(1]mll2
op
<||
(134)
Then E. W (T5)
wo is a Gaussian random variable with mean zero and standard deviation smaller than O [ 2lll a By Gaussian tail bound, for any i E [M]][1}, with probability at least 1 - 5/2N2M2, elaall
@oolll [8. (T( wol I 0 fol1wool22ppVd max (ax Vp .0)) v 2 (4N2M2/0)
c/|p4112
1
where the last inequality by oy O(1wolll min [1all (apli",(oppa (a N SNR 2(1)
and d 5 e2 N2d)) and dn d max{(SR** ,NN "ARR-) Applying union bound completes the proof.
Lemma D.10 (Population Loss). Under the same conditions as Theorem
we huve
(D())(s)) = o(1)
Proof of Lemma D.10 Let event A be the event that Lemma D.8 holds, and event B to be the event that Lemma D.9 holds. By a union bound, P(A B) 2 1 - 8/N M. We decompose Lp (0(Ta) into the following form:
(D((33)) (AA(A A BX
())
(135)
Then we bound (A A (XX, ((Ta))) and ([A A ()) (XX 0(Ta)))) respectively. When event A and event B hold, have
Bounding E[[AA
M
v(((X0((T)) = M
(
W
(T))
W
T
XW wo
K
1=1
(Ts)
wo
- -(1
max(/e, W (Ta) (T)) wolk
2M
2M
(136)
C20
2M log
log
Cw
- 1,
C21
64

---

where the first inequality is by Lemma D.8 last inequality by M (1). Therefore.
the second inequality is by (112 and Lemma D.9 the
(A B)AWS(( < (()s(X
< log(1 + exp( (XX,
(oxp(-uf(X,, ((T))))
log
Czo
(137)
I <exp/ (1
C2
log
C20
I exp
C21
Bounding
((T 0(TS))) <V B)²) v (X
s ()((AA)))) Blexpp 2y (X, ((T3)))
(138)
87
EEexp(-2f(A ((T))))
N
where the first inequality is by Cauchy-Schwartz inequality, the second inequality is by log(1 exp(-2)) exp(-2).
Next we provide bound for [feexp(-2(Xx,,) ((x))) Note that
1 M
v(((x ((Ts))) = Σ
(Ts) x X wo w
(Ts)
M
and μ. W wo 2 0, we further have
M Σ Is,
(f((X0((Ts)) 2
(T))
wol
1=
Therefore,
E[(ep(- ((Ts)) < E2exp(2 Σ
& W (T) wol)]
2=2
E
exp(2
| W (7)) wol)
M
E
exp W (7b) wo) + expp(-2E, W (75)
wo))l
(139)
1=2
M
E exp(22: W (T5) wo) + expl-26 2/ (T')
wo))
2M
II Elexp W (T') wo)l
where the third inequality is by |x| < max(æ, - The fourth inequality by the independence of wo.
E, W. (T))
65

---

We denote [ (1b) woll. then (T5) W. wo is a Gaussian distribution with mean zero and variance smaller than C200 p. Therefore,
(2 W wo)) S exp(22
max
| ozd] )
exp (2Cro7, of
<
max ozdl of 2
(140)
= exp
<
= exp (o(1)
= -1+
where the first equality by 134 the third equality is by ov v O11
min(llall (opvd)-1) d )) N SNR? 2 2(1) and d = 52 (2 Then we have
M
Elexp(-2f(X (0 - MM
II
Elexp(2 W (Ts) wo)l
f=2
(141)
s 2M (1 + o(1))
= 0().
We further get
52
C21
(L((AAAB))(F(XX00000)) <
(fexp(-2f((x001)))
<
(142)
N
N
Plugging
137
and
(122)
into
135
we have
log
C20
O21 N
Lp (0(T))) s 3exp
+
o(1).
C21
E Harmful Overfitting
In this section, we consider harmful overfitting case under the condition that N SNR 2(1). The proofs in this section are based on the results in Section C which hold with high probability.
E.1 Stage
n,t begin to pull apart until [v is sufficiently larger than [V[[] At the same time, the inner products of and maintain their magnitude.
In Stage I, ((()) (())
Lemma E.1 (Upper bound of V). Let To = O
Then under the same conditions as Sdl o3d wo ll
N
Theorem 4.2
we have
|V [V(0)),
| 2)
fort E [o]
66

---

Proof of Lemma E.7
By Lemma B.2
we have
(1+1)
mit
1111112
exp((4 (3) ,b("))))))))
Σ
NM
M1
nES
exp((4+) (8) 11 + Σ expl(g+)
(0) (
2=
exp( k4
+
Σ
M
j=2
exp((?n.)) k () + Σ exp (1) (upl(anns k,01a)))
k=
n
3NV
(1+ (M 1))
NM
4
3 21
4
where the second inequality is by Lemma
C.1 and
((0) 1. Similarly, we have
3r p 13
hv=
(++1)
-vp-1
4
By Definition B.I
we have
|r
=
V100 +
(v, [v++ )1 1//200110 11
(s+1)
0 0
To
$ 1 (0)
+
his -v,+1
-1 l1
3
311111113
N
I d
||lol12
00
4
bll ofd]|woll
N 1
dx²
+ +0
d o3d
= =(d, )
where the first inequality is by triangle inequality and t < To, the second inequality is by Lemma C.2 the last equality is by ~-1, SNR -2 22)).. Similarly, we have ||_0|| O(d,, )
67

---

By Lemma B.2 we have
expl ( 1: k,,,1))
Σ
Σ
( Ent,e)
NM
M
n'ES+
1
exp((et) (1) 1 ))+ +
Σ
exp (x= ,
k=2
M
explign g: k0,))))
En Sn(ie)
M
j=2
exp((gn'.) 1) +
E
exp( An'
(1)
k=2
M
exp(1q (1) k
7
En En(,e)
NM
M
E'ES
exp((g (0) )( + Σ exp( ( k 1
(143)
M
exp( jo k
Enn En(re)
M
j=2
exp( (1) b!))) + Σ expl An' (1))))))
(1)
3m02d
1
M +
MN 25, v Vdlog(4N2M22/))
2
NM
2n0dd
N
N
where the second inequality is by Lemma C.4 and -P1 < 1, the third inequality is by d =
(())
(222d the last inequality is by N SNR? = 2(1). Then by Definition B.1 we have
|
1V100 (0)
() wwo
$ <11100 n2
Σ
(ett)
(1) V.nsl 1|00
20
00 N
I dh + N
|wo 12
o wo 12
=(d }
where the first inequality is by triangle inequality, the second inequality is by Lemma which completes the proof.
Lemma E.2 (Inner Products Hold Magnitude). Let To O(
N
Then under the same
nd wodl ll
conditions as Theorem
4.2 we have
(9 k
M1
of max || 12, on of ddh log(6.22
[a - 1011 1 (1)) . (
= =0 max (11 112 o.d] of da logg(6N222155)
(119 k 11) (
1 1
(0) 2001
68

---

0 max ozd] of dh (o..og())
11 11. [1 0(1
14. || 2 (d
for i,j E [M]][[}, n.n' [N] and Tol.
The proof for Lemma E.2 is similar to D.4
Lemma E.3 (V's Beginning of Learning Signals). There exist T₁
4M(3M+1)N
such that
nd (C3 (23-24M)33 wo
the second element of vector X,, W. (T)) wo dominate its other elements, which means ((Ti) max ( + V for all n E S4, i [M]]], and v(TT
n.2 2 3M
n.2 I -3M max ( (T2)
[T)
n,i
n2
for all n E S_, E (M),
Proof of Lemma E.3 According to Condition 4.1 where CP 25M, we have Then we have
4M(3M + 1)N
4(3M + 1)N
N
T
ndd (25M- 24MM woll
ndj ofdd
nd oz wo
which satisfies the time condition in Lemma
E. and Lemma
Then by Lemma D.I and Lemma
D.2 we have
(((()
o1),
2
M o(1) S so so foftaa (gf (1) k(
1
+ o[1)
M
M o(1) s softmar((1 (g (0) (1)) <
1
1
+ o(1),
M
M o(1) < soʃ softmax((( (4 (1)) <
+ +0(1,
M
1
M o(1) softmaaa (g (1
+ +(1
M
for i,j e [M]] n e [N] and = [0, Ti]. Plugging them in the update rule for VV+ showed in
Lemma B.2 and we have
(1+1)
(1)
-v+
exp((at) (t) x(")))
Σ (((
= NM
(0) exp((1 (2) (01)) + Σ exp((1) (())))
nESt
M
exp( (g (1) (((2))
2= exp((gn))
(0)
k(1))) Σ exp( (1 (1) (1))
(144)
3.V
1
<
+ o())
M M
+ oo())
NM
4
2M
2nd
NM
69

---

where the last inequality is by N 1----SN SNN -2 = =hen by Definition B.1 and taking summation, we have
2n
[V
1 1 (0)
+ Ti
/ 1
NM
4M + 1)N
2n d NM
dp
w
ydd (CB 24M)
(145)
8(3M +
d
di (0z 24M)
Similarly, we have
8(3M +1)
v (T))
S dn
(146)
(03 24M)
For S4, by Lemma B.2
we have
(1-1) (1)
PV
PV,n2
M
cxpl(9 (4) p(,e))
7
Σ
(1)
2n
NM
#'ES+
P=2
exp( (1) (2)) + Σ
exp((at)) (t) (
k=2
M
exp((a (1)
n 2, En(,t)
M
j=2
expl Lapl(ann k 11 + Σ
exp( (q jn (w,)))
k=2
exp((g (1) p(
11
Σ
(t)
Enire)
NM
M
n'ES-
P=2
exp((g 1 (n + Σ expl(a
(0) (0
(147)
k=
exp((g :f
(1) ((x)))
En, En(re)
M
2=
exp(1gn k 1)) + Σ
exp
( jn w9,N))
k=2
noād
M 1
o(1)
1 NM
.MN- 25
v M
2NM
M
nopd
4NM
Choodd
4NM
where the second inequality by Lemma C.4 and
(0 < 1, the third inequality is by
NN2-Nd. Then by Definition B.1
have
((T) (0)
pV,m2)ll Σ (Pvm2 (6)
4M 1)N
2 - -dpt
[oo
(148)
4NM
ndl (C) 24M 34M) wwol|
C2(3M +1)
2 d
di (C2 24M)
for n E S4. Similarly, for E S_, we have
C23M+1 +1)
(T,) n2
< d
(149)
d (C 24M)
70

---

By Lemma B.2 we have
(++1) Pvn
M
exp((1 (2) 18
<
Σ
(00
(S En(er)
NM
M
n'ES+
1
exp((et) (t)
Σ co
(0) N
k=
M
exp( (e
Sm ))
j=2
exp((gn k (0)
(1)
Σ
eapplan
(t)
k=2
M
exp( (1) k
Σ
pit))
(150)
NM
exp((g (0) (00))) E expl(q 10
E'
M
explig (t)
k(,))))
En,i En(,e)
j=2
exp((an..)) (11))) + Σ
exp( (n (11
h=2
poo d
M(
+ +(1)
" NM
MN 25 /dlog(4N2M2//0
2NM
2n
s NM
for E (1,), where the second inequality by Lemma C.4 and
((() S 1, the last inequality
by d -2 N2,) ). Then by Definition B.1 have
|
Vini M.
(a+1) (8)
(e+1) (2) 11
(151)
4M( + 1)N
2no
S d +
1|
bd (C3 24M)o woll
NM
8(3M 1
d/
(C3 - 24M)
for i E [M,21,,), where the first inequality is by triangle inequality, the second inequality is by
Lemma C.2
and 151 is easy to verify that ((TT) 2 3M | for all E St.iE [M))(1,2 and v(T))
According to 145
146
148).
max (T1)
max)(V((T
(T))
n,2 S -3M max (
n,
for all n E S.,iE E [M]M1,,2 which completes the proof.
E.2 Stage II
In stage II. (q+, k.)), (gn, k4) grows while (q+, kng) (An, kn decreases, resulting in attention focusing more and more on the signal and less on the noise. By the results of stage I, we have the following condition at the beginning of stage II
(TTi) n.2
2 3M max ( (Ti) [T)))|I)(()))
for all n E S4, i E (1)..
-3M max m (21) (Ti)
|
71

---

for n E S_, [M]
/ | = 0(d
1
T1) k
(
=0 maxi|l 1 112 of oo dh log(6.222118)
Kot mi) 4F
( Kgn. (Ti)
= 0 max (11 ozd oh dh log(6N22M2))))
(K4 (T1) k (TT)) |1 n,
(T1) (Th)
nn (T)) (7))
o max |م 112 o.dd3 of V log (6.V2
| 11 || (T 1 011
1 12: ( 112 @ ddlp)
for i,j E [M]][ n, n' [N].
Some of the proofs at this stage are based on the above conditions.
Notations. To better characterize the gap between different inner products, we define the following notations:
denote
w/ (a+ kn,) 19+ .4) n
(1)
denote y (4)
n (g+
(1) n,2) (q !
ne St, [M]ML,2),
denote v (q
(1) (9) - (a
neS_.
denote y
n (q (2) (q ngh
(1)
ne S-.j M1.
denote v (4)
+, (q
(1) (0) (9
n S [M]
denote v (4)
n, (an..:
(1) ((2)) (an
neS_, E [M]
denote (t)
n.i.j (g (1)
(g
i [M] E [M]][,,2].
n.j
Lemma E.4 (Upper bound of V). Let To O
nazd| wo 11 (=
Then under the same
conditions as Theorem we have
| o11)
for E [0, To].
The proof of Lemma E.4 is similar to that of Lemma except that the time To is changed.
Let T2 0 Mofdloo log 6N (NN ). then by Lemma and Lemma D.I we have 1 o(1) <
-((() S + o(1) for [N] E [Ti, T2], which can simplify the calculations of a and B defined in Definition B.3 by replacing e(()) by their bounds. Next we prove the following four propositions I (1), K(I), (()), M(I by induction on for / [Ti:
Jt)):
nCgond wo wol Ti)
v(t) n22 >
N
"C3ofdllwoll (t Ti)
vit) n2
<
72

---

V12 2 3M max m ( |
for all n St,iE E [MM (1.
n,2 S -3M max{ 1 ( }
for all n E ES. MM, 2}.
I s o(d,))) + BC wo lBat Ti)
| S o( (CCofdluurolB Ti)
N
for n [N].
. :(((t):
11 12. 11 = e (lu1Boodd))
11.. 10:1 e (o
kg __ ( (0)
o(1)
(11)
f, E [M]][]], n, n' E [M] + or n #
C(t):
(9) ((+1) k 1( (4 ; (12)) 2 2 (++1) k4 ((41) - 10 k())
(0)
(gf+t) k(( - (9 (0)) 2 2(
(+11)
2nj ((t1)) - ( 2))
(go kn2 ((tt))) ) (gn
(++1)
(8) (1) 2 2((
((+1)
bx ((41)) (g
(1) (1))
(q (0+1) kn (bt11)) (9 (2) 2 2(( (((
1 (011))
(q (1))
1 11110 wo [g
T ((+1) z log cxp( iT (Ti) (T)))
(t (t) Ti + 1))
No N3(oo((AN2M22/1
#C 13
q (((+1) n, z log (epp( (TT)
(t- Ti)(t Ti 11+1)
(( 18))
(t) - -TI+1) n2 wo [2
v m 2 log (exp(v (T))
N2 (1oo((NNN2 18))
Cro ww [2 N2 log (0
v(l+z) m,i.j
> log (exp(V
(1- - -T +1)
fori e [M] E [M]]f1,2]. n St.
. M(t):
| /v n.
(0 2 S log(d})
(( (9) 1 10))),, (0) (
Kg (1) 2.,2111 s 2 log(d
(8) || o(1)
for i,j [M](1),,5' E (M) n, [N], n + n.
By the results of Stage I, we know that (Ti). K(Ti), M(Ti) are true. To prove that K(), C(t) and M(t) are true in stage 2, we give the following claims holds for E TT,Ta] Claim 9. L(Ti), ... L(t 1) = 1) Claim 10. (()
C(t - 1) = L(t)
73

---

Claim 11. TCC
K(t]- 1)
Claim 12. "CCT,,,,,,"C,,K.........."",,,,,
M(++)
The proofs of Claim 9 12 are similar to the proofs of Claim 5 8 Next, we show some important
procedures for proving Claim TO
Proof of Claim 10:
Similar to D.2.2 we have the dynamic of V as follows
v (
(+)
, wo Tı)
1
(152)
Nº [2((og(6N''M20/0)
exp(u!())
17 C wo (s Ti)
(+)
2
(153)
2,
N² N2((og M2
exp( m.
"C wo Ed (s-
((+2)
((())
(154)
nii n.Id
N²2(log (22g(222)/)))
exp(V(l...))
"C wo [d Ti)
vl++1) n.i.j
V((e)
(155)
n.j.
(1(10((6N2222/))
exp n.i.)
for n E [M] e [M]{1 [M]M[,,2], E [Ti t) Next, we provide the logarithmic increasing lower bounds of V. Recall that
"CallulBo wo 1s -s
(156)
N2 N3(1og(6N22
exp(v(())
Multiply both sides simultaneously by (xp( and get
- #C 1 wo 122d Ti)
exp(v(()) (((+tt) v1(1)
(157)
22)
Taking summation from Ti to and get
exp(v(()) (/(+t)) -((+1)_v((+)) v((1)
CCllulBodd [2 (s Ti)
(158)
N2 (7oog(6N3M3//00
7 CC [C 1ld (t Ti) Ti +).
N2 (((oo(6N22MP/)))
By the property that ((() is monotonically increasing, we have
M exp(x)dr N
exp v (?) ( (1((41)) vo(1)
(159)
(t ( 1).
N2(1og(6N2MM2/66)
((+ By f exp(x)dr= exp(x)de=exp(VV - (T)) get
exp
"2 wolldd
(((-1) > log ((xp(
(1- (i)( -
(160)
N2 (Poog(6N222/5))
74

---

Similarly, we have
3 woll (t- (161)
( 2 log (exp(v(Ti)) +
No (110((6N2233/)))
v v "n 2 log [exp(v C wo B2
<=(oo(662222/00)) (1-T))(-Ti+ (162)
Croof wo wo
(((+1) n, 2 log (exp(
N2 (10o((02221/3/8) (t- -t ( 1)).
(163)
u (1+1) n,i,j 2 log (exp(v ming + C 10 wool
wo (11g0((NN7MM)))))
-
(164)
f n E [N] [MM(( (MM,))) (
E.3
Stage III
In Stage III, the outputs of VIT grow up and the loss derivatives longer o(1). We will carefully compute the growth rate of Vi and Vn,; while keeping monitoring the monotonicity of
(q, k). By substituting T2 = θ
woll into propositions [(t,, KC()), C(t), ((t) M(t) in Stage
II. we have the following conditions at the beginning of stage IIII
(Ta) (
(22)
[(Taaa = o(1)
for all n E [N], i E [M][[
(T) n2 2 3M max max (Ti)] |
n,
for all n E S E [M]M1,2).
((
s -3M max [V (T)) | (
n.2
for all n E S_,iE ( (1),
11
(T))
2 |
(T))
= @ 11
|1q.
(Tb)
(Tb) 11 @ 7d
(( (T2) (T2)
(T)) 1.
Inc (T)) o()),
(K(T)) k (bb)
((22) o(1),
i,r E [M]M{1),,,,' n, e [N], # or n n'.
V (2) 2 log (exp(w(T)) (T))
ofd 2dd(og(6NN3MMP/0))
Cr112||11 uo 11:
y (Tg) n.t. 2 log (exp() m +
ogd( (og M /5))
C7 17llw0 [2a
y (Ta) 2 log exp(viTT) (Ti)
(log(6N2M22/05
75

---

CrlluooBBdy
n,i,j 2 log
(exp(V m.i.g
(TT))
(1og(0N2/3//6)) 3))
1 ب m N.ng log(d))
Kg (T))
k_ (TT)) y. Kg (To) (7:)) (Ta) (3))) (7:) | = 2 log(d))
1( (25) (1b) (2:) ا || o(1)
n, RJj
for i,j E [' (M.)) n, E [ n + n.
Let T3 = O neoo2dl wo 1 by induction on t for t E [s):
Next we prove the following four propositions (()), 0(), P(). 2()
(())):
V. 2 3M max max
for n S+, i E [M] (1.
< -3M max([V((t)) ( }
n.2
for n E ES, (MM(()112)))
IV f |.V = o(1),
for E [M],,,2), [N].
log (exp(V/(?>)) "Ciro wo wol( Tz)) s v/l2 < 2log (0
N
for all n E
*Cnooodll 1B:t T2)
-2 (0 < V/0 - logg (
N
for all S_.
. O(t):
l |I o @(Il)12oddh)
l || 1 θ (z
Kg q (. 2 20
= o(1)
K4 k (3) ).
(1)
for [M]M[1], n, n' E [N], jr nn.
. P()):
(of ((+1) ((=1)) (9t (0) (1) 2 2(
k ((4)) - (1 (()
(q ((+1) (((-1)) - (q (0)) 2 2((1
(011) - (0)
(++1) ( (q (1) 2 2((
(e+1) k(+1)
( ()))
(gnd
(gn.) kn2
(++1) ((t1))) (g (8)
m2 > 2(
(1t1)) (q.h) (1))
v v((+ 0 (
((+1)
n.
w " 2 "
v (1+1) 2 (1))
n.i.j
for (MMO), e (M]\112)), E St.
76

---

=(())
I T 10 n. 10 2. (0 n.gh - [og(c=111)
Kg 2 1
(/03)) S 2 2log( -1d
Kg A 11 / (11)
for i,j [M]M1]) E [M]] n. [M] n +
By the results of Stage II, we know that (T))., (T2)) 2(T)) are true. To prove that (Vt), O(t), P(t) and (t)) are true in Stage III, we will prove the following claims holds for t [T2, Ta]:
Claim 13. P(T2)..... ..., P( (i(t-1) 1) -
= N(t 1)
Claim 14. (tt), P(T2), (())), P(t - 1) = >>>p(()) -(())) P(t)
Claim 15. (T N((), (())) P.. ((t-)))).,,(((())))))))))) 1(T))), 2(t=
- = (t+ 1)
Claim 16. , (T), P(t Q(++1)
Lemma E.5 (Convergence of Training Loss). There exist T
CiaNN voodd/wool3
such that
(s( 2 0.1
Proof of Lemma
E.5
Similar to D.7
we have
softmaz (gg (, = =0 (3d(log((6 ( /
(165)
| ww 12
softmart (9f (.())))))) = o oz log(6N M 10))
(166)
| m0 [2
costmax((gg)) (2) = o( (log(6 M² 12))
(167)
11w0 1124
(log(ON2M2) M2218) 3
softmar((g (1) ((1))))) 0(
(168)
|
softmar(1g (1) (2(1)) = 1 o dd (1og(6/22M22//0)
(169)
||H1131 w 112d
softmaz((g.) ((()))) -00 (log(6N2MM2/4) 3
(170)
||wo||3d3
for E [M]][], E [M]V(12),, [N].
For E S4, substituting t = T
C meofdlloll b
into propositions ((t) and get
nC1r0 woll (t- T2)
v log
exp((v((a))
+
N
> log
exp((V(Ta))
n2
+ O2o
(171)
2 log Cz0 (
|v |, | o(1)
(172)
77

---

we bound f(Xn. ((t) as follows
((x,. = M
E
n.
W (1) W
( x x, w wo
1=1
expl 10
1
M
M
exp 12)) + Σ exp( (q (1) (1))))
j=2
exp( (q (
) v17
+
exp( x( + Σ exp exp((g (1) ((02))
exp((q (1)
exp((g (1) (())) + Σ
expl1a (())))
5=2
cxp( exp((qn.1. k
) -112
M
exp( (pp((ann))) (00))) + Σ
exp( (9 (t) (1
y=2
M
exp((g+", (k())))))
exp((g- (1) k ((11)) Σ exp((14 (1) ((1)))
1=2
(173)
exp((g..)) (1)))
M
exp( (0)) + Σ expl(g (1))))
2 M (1-0 ozd( log 6.N (6. (6N )
|1 ww 11
(1- (log(6.V2 M /8)) 3
+
-1) +M-1))
))
log C20
/wolldd
1. M (M
(M-1) -1) (0( ozd
|| 11112120ol1dd
+ +MM 11) - (log(6.22 M2 18)) )) 0(1)
||wol
Oz0
0 of [ M
M 16))
= log
| 1ml3l/woll
wo lo
Ozo
2 log
(oo))
> log
)
For the second inequality, by dp = ā max(SNNR SNR "1N22-2). we have
2 (0000(6N22//80)) o(1) log(1 + o(1)) S log (2o)) long Czo is sufficiently large.
wo Ba
78

---

Then we have
d = =og (1 + exp( (xxxp-s((x,)) -X(,,, (()))
I exp(-f(XXn (())))
(174)
log (t))
$ exp
cc
Similarly, we have e v é for n E S_. Therefore, we have (s()(()) = -N
Σ
1 S
n=
E.4 Population Loss
Lemma E.6 (Population Loss). Under the same conditions as Theorem 4.2
let T3 =
we have
nEO ddl wo 11
LD (0(T)))) 2 0.1
Proof of Lemma E.6 Similar to
133
W (t+1)
w wo -
17
Un ((t))
Xn | (((. W
( (xx)")ll Il
wo
NM
$ NM
NM O( (1al2, -(1) 1
0 n- op Vd. |
(175)
Taking a summation, we have
Ta-1
(T)) wolla <
W wollz +
(1+1) 11+1)
wo W wollz
= | woll2 + 0
(n.1oppa 1ovad-11loo / 1
(176)
moo3d wo
= = woov + 0 ca,vd
Then E,- W wo a Gaussian random variable with mean zero and standard deviation smaller than O [ov Towllla woll2a, Vd .2a.. By Gaussian tail bound, for any i [M]][1), with probability at least 1 1/2M.
E W (Ts) wol s <0 Vd+
N ) 22 (4M) $ 1/2,
where the last inequality is by oy Ō(1wolll min (opV -1 dr and d δ 2 da). Applying union bound, with probability at least 1 1/2, E W (Ta) wol $ 1/2.
Recall that v (Ta)) o(1), with probability at least 1 1/2, we have
M
v ((Tx)) = M
Σ
w
(Ts)
(T) X J wo
(177)
1=1
> log(1 + 1122)
Thus. (D((())) > log(1 + 1-12) 0.5 > 0.1.
79

---

Complete Calculation Process For Benign Overfitting
In this section, we show more calculation process under benign overfitting regime. The calculactions for harmful overfitting is similar.
F1 Calculactions for a and B
In this subsection, give the calculactions for and 3 defined in Definition B.3 Restatement of Lemma B.1. Wy are given by
the gradients of loss function Ls(0) with respect Wq, Wx and
Ww (s(
1
Σ
(100) W. x (diag(4nn.1) 1P,,19m..1
NM
nES+
S w T o W X (diag(4nn.)) (n.)), WK
(178)
e (0) WT x (diag((Pn.)) 0 pm,1)) Σ
NM
BES_
M
wo W. x (diag(?nn.) (n;)), WK,
Σ (X,, (diag(<Pn.1) - (m,1) wow
NM
NES+
M
x (diaglynns) - :T..P..)) Wy Wo
(179)
Σ (diag(l) pT,, Pm, Xn Wywon
NM
nES_
M
(diagl(nn.) - (,, W,
We will give expressions for a and B defined in Definition B.3 based on these two equations above.
By
178) and the orthogonal relation between f and &, we have
4g n
TAW
"
Σ
pl(e)
w X, (diag((en.)) (n,, W!()
NM
nES+
M
(+)
nES_
(180)
T where w. (OO
(t)
and X, ( can be viewed in the following forms
n
w W X (V(')
V(O)
v-00
X, W (1)
(R: (T)
(T) .....
K
80

---

Then we can express a 4++ and t as follows
7
Σ
NM
exp(14+ ; (.4)))
(v
exp(1a+ (8) 1) + Σ exp(114 (()))))
M
j=2
exp((9+) (1) x()1)
}3)
exp((g4) (. b(b1)) + Σ exp( (1) ((())))
M
(181)
M
exp( (t) k k((1)))
(v())
exp((gf" (1) x(1)) Σ exp(((4)) (t) (ili))
-=2
exp((gt) (t) (k("))))
M
exp((1+0 10 1( + >
cxp((g (9) ,(
==2
(1) 1 p(()
ant,
NM
exp(11+ (1))))
M
exp((gt)) (2)) + Σ exp((1t)) (0) (()))
3=2
exp((gt) (1) ((ll)))
exp((g4) (2) ((11)) + Σ exp((g (2)
M
j=2
exp(1a+ (1)
(()i))
exp((g() x ((0))) + Σ Exp((1 (1) ((1)))
(182)
3=2
exp(19 (1) k(11))
)²)
M
exp((gt) (1) ((1)) + Σ exp (q k())))
1=2
exp (1) ((!!))
Σ (v n,k
(g exp((g_) (1 b(11) + Σ exp((g (1) ((0)))
exp((1() (1) ((tl)))))
M
exp((q_", + Σ exp((a ,,
11)
Using the similar method as for (o (1)
we get the other and B as follows
81

---

(0) 7 Σ
( 1 11
NM
nES
exp(1 (1)
(v.0)
exp(1a (4) k 1) + Σ expl(aa (3())))
M
j=2
exp((9 k0))))
exp((e_) k (1)))
M
exp(( ((())))
(183)
1=2
M
exp((g (1) ((
(v(t)
n,j
1=2
exp((9_))
(4) (00)) Σ exp((q" (1) ()))))))
cxp((9 (t) (())))))
M
exp((g_) 11 Σ
exp( (q _.,))))
==2
(0) (((() |000 12
-n
NM
exp(1g (1) (10)))
M
_xp((g_)) (8) (b + E exp((a_)
(1) (())))))
1=2
exp((1) (1) k((1))))
M
exp((g"),,1 Σ exp( (t) (21)))
1=2
exp(1q (2))))
M
exp((g_) (0) k (1))) + Σ
(1))))
(184)
1=2
exp(1g (1)
j²)
ex (0) k. (0)))) expl exp((1")) (1) x(())))))
exp( (0 ((0)))
(v())
M
exp((g (1) (1)) + Σ exp( (q (t) (())))
j=2
exp((g (t) ((1))))
M
(9) (xp(((_0AA00)))) + Σ
exp( (q ((11)))
コ=2
82

---

(1)
77
Σ
pit))
Σ
NM
nES
i=2
expl 1:44))
M
expl (g 1) +
Σ
exp( (ap((agn)) An (0))))
-=2
exp( (1)
nn
)²)
exp((g k (11)))
exp(
(1)
(185)
exp((g k:p)))
(v-9)
M
exp(ign.: (1) 124)) - Σ exp( (t)
(g (2))))
j=2
exp((gn..) (1) k1.k)))
exp((g..)) (0) k (( + Σ (1)))
M
(1)
5=2
(1) 17
Σ
((())
Em.))
NM
nES_
(1
M
exp(
!1 1) +
Σ
expl expl(ann.o (())))
1=2
exp((g (0)
100))
)²)
exp((gg) 1 ((10))) expl (t)
(g ((1))))
(186)
exp( (ap((1 nn k 1(())))
(1)
(v(0.
M
exp((gu)) (1)) + Σ
exp (opp(a!!))
( (())))
1=2
exp( (t)
expl(g..) (1) 1 (00))) + Σ
M
expl
(g
(q
((0)))
2=2
83

---

7
((())
v Em,a)
NM
exp( (1)
ke k((1))
(n.
-1
M
exp( (t) k
(42))
Σ
exp(ia ((
(1)
3=2
exp((g (21))
M
explig.. (1)
(11)) + Σ
exp( exp((gn.0: (
1
exp((gn. (nnk ((1))
m.
exp((a (1))) Σ
exp (1) ex
ko (!9)))
(187)
1=2
exp((1 (())))
M
exp((1nn (1)
((1)))
Σ expl expl(1 ( k!(1)))
-=2
exp( (2)
(v
exp(1g (01))
Σ exp(
(ap((gg.)) (0
(g ((0)))
f=2
exp((g...:: ((0)))
M
exp(ign0) k ((1))) + Σ exp((g (:11))))
==2
for n E S4.
84

---

M
T ((t))
Σ
NM
k=2
(0) 1
v(0)
M
exp((gm..) k 1) + Σ
explignk k/1.)))
5=2
exp( (1) k.111))
exp((gnn (1) R b1
Σ exp( (t) ((n))))
M
(g
==2
exp(( (t)
(111))
exp(( k (1) +
Σ
M
(1) (0))))
(188)
expl (1) ((11))
j²)
exp((g (1) k (1))))))
expl eepp((g (1)
(gnkk ((1)))
exp((gnnk (t)
((1)))
E (V/1-
M
(1) exp( (1))) + Σ (x ((.)))
j=2
exp( (1) k(())))
exp(ig!... (1)
b()))) + Σ (xpp(191.00 (1())))))
2=2
for n E S_,
n111113 Σ
300
NM
n
nES
exp((g (2) (())))
(V
M
exp(1a (9) k (0)) + Σ _e (())))))))))))
(1)
f=2
cxp((g (1) k())))
)²)
exp((g_.) (1) k ((1))) Σ exp ((g (() ((1)))))))
(189)
M
exp((g") (1()))))))
(v(()
1=2
e
(1) (0))) - Σ exp( (q (t) ((()))) (()
j=2
exp((q (t) {()))))
M
exp((g_) 10 A(0))) (000( + Σ
expl (q (1) (.00))
2=2
85

---

300 m,t,
(())
NM
expl 1opl(gn.110 k:1))
(v20(
M
exp( (1)
1) + Σ eexp((gn. (2))))
1=2
exp((g (8)
k
)²)
expl 11) b(1))=
M
expl (g 1 b!0))))
(190)
exp( (g, k44
(1)
(v(9)
M
exp k b(1))
Σ
exp (ap((gg)) (0) (2))))
j=2
exp((g (1) k(kk))
M
expl(g (1) k
111)) + Σ
exp( (q, (01)))
2=2
300 0 NM
(()))
expl expl(g
(1) (. (20))
(v!0)
M
exp((gn.)
(1) 11) + Σ
eepl( (10)))
1=2
exp((g (0)
(00)))
)²)
exp(1g k. 1(00))) expl (t)
(g [(1))))
(191)
exp( (xp((111) 1 k:! ((())))
(1)
(v(0)
M
exp((gu)) (1) (1)) + Σ
exp (xp((a!!))
(n. (!))))
j=2
exp((g!)) (t)
expl(g..) (1) 1 (0)))) + Σ
M
expl (
(())))
2=2
86

---

71 NM
nES+
M ( Σ
exp((g+) (6) -1
(En En,i)
M
1=2
explixt 1( + Σ eepp((a) to (
==2
exp(19 m0)))))
M
exp((9+) (1) k () +
Σ
exp((19)) (9) (22)))
j=2
expl(0t (1)
(192)
n,k (iEm' En,a)
M
expl(g (2) (,,,(()))) + Σ expl(1 (2))
j=2
M
exp( k(
(Sar,, Ena)
M
exp(114), (1)
Σ exp((14) (0) (2))))
/=2
exp((ge) (1) k(0)))
M
exp((et) x(11))+ Σ exp((a
12 (x))))))
1=2
-h 27
Σ ((())
NM
nES_
M
exp((g (1) (((())))
( ( Σ ((En,,' En,i)
1=2
exp((g (1) x()))) + Σ exp((1g (0) k!!l))))
j=2
exp((a (1) (1
M
exp((g (0) (1)))) + Σ exp ( x(0)))
M
exp( (1)
(193)
(() (1En, En.k)
ex (9) x(0)) + Σ
exp( (1) b(1)) j=2
M
exp((g ((1)))
(En En..))
cxp((g (t) k (1))) + Σ (xp((g") (1))))
M
1=2
j=2
exp((q (2) k(()))
exp((g_0) (1))) + Σ
M
expl (q (9) (0)
j=2
87

---

B (00)
7
Nn, ni
NM
exp( (2) kk (2,)))
- v
(K Sm.k)
explig b(1))) + Σ exp((e (9) (2))))
exp( (pp( 1. ((1)))
exp((gn.: 12))
expl(1 k 11)))))
exp((g nn k
(194)
n,k ((En' Sm,a)
M
exp((g...) (1) ((1)) + Σ eepp(inn. (4) (1))))
M
expl(g (t)
(( En.))
1=2
exp((dn.) k ( + Σ
explignst (1) (2))))
y=2
exp( (pp((g..1111 (8) (1))
M
exp((ann) ((1))) + Σ
exxp(1nn)) (
(!)
y=2
for n E S+
3 m
11 p(1) NM
expl 11) xxp((?m....'))))
(1))))
(- v
(EEn,e, En,k)
exp(1g ((00))) + Σ exp((e (0.. (())))
2=2
exp( (2p((gnn)))) (1)
exp((g
expl ( n (1))))
exp( (q nn.55 (1)
(1) R,k (EE'' En.ik
(195)
M
exp((g (1) (1))) + Σ exp(1a (())))
-=2
M
exp((g (1)
((0)))
(1E En.1)
exp((ge)) k (()))
exp( (
j=2
exp((g (1) ((0)))
M
exp((g (())))))))))) + Σ exp ( (1)
for n E S_.
F.2 Update Rules for Inner Products
In this subsection, we give the update rules for the inner products of and k.
Restatement of Lemma B.3 The dynamics of a Wo W X can be characterized as follows
88

---

(a+ (++1) _ ( - (44 (1))
(1) 111:01101 11 + Σ -
ant (
(0)
n
+ + 1
nES im
M
+ ((o00 (0)
(4)
n,
WES 2=2
M
(34)) ())
300
(
(
aata In.i
nES 2=2
(q (+)) 11411) _q !!
M 12) 12 + Σ Σ
(1) 1
nES_ 1=2
M
+ 0 -
(2)
((t)
n.
NES_ 1=2
M
2( (1)
+
BES
M
(310 T
3 R
I
BES
(111) ((+1)) (9 x()))
(gns
M Σ>
(1)
|k 11 11 + a
(K.
(1)
i0)
10)
n==li=2
M
+ +.4 (
n' 1=2
N ΣΣ
M
(0)
(t)
(0) +
n,4,
1=
M
(g(t) T
g(t)
()
+,+
n' ES 1=2
89

---

(1+1) k ((t1))
1g k )
(1)
N Σ
M
-n,i, 11=h
10 112 + a
(k+
10 +
Σ
2-
M
Σ E'ES
l=2
N
M
(0)
10 (1)
(0) Rdw . 413)
n,,,
+
n/=
M
(5 (1)
3 n'
+
Anl
n'ES_ 1=2
(q (41)) k - (a (
(0) (1) ((1)) + Σ
(
n'ES+ 1=2
N
M
+ 1124 +
(0)
Pn.. (4+
(1) (t)
ann' a (4+ 22..)
1=2
(0
Σ
(0) W1
W'SS+
(09) ( 9+
(()
3(0)
(
Σ
(0)
aa Ani.
n.j.
(q (+1) k 11111)1 - (q (1) k(0))
(1) (k E'ES 1=2
Σ
k00
+ +
119
(4)
n.J.+ (g+
(1)
Σ
M
+
(0)
Σ
n'ES_ 1=2
(80) 9+
(t)T
g(t))
g(t)
(0)
+
n.j.
q
ag 9n'
90

---

(nn ((t))) k (g kn))))
= (bof) ( + p 1,
(1)
(c)
(k (1)
N
+ 3 (at In.it + - (a_ In.it
8(1)
nighn 1nt (n.
(0
N
M
(nns 144 + +O k
+ Σ
n/=1 1=2
( + Pn.. q
(O)
(())
(0)
Σ
(0) In'1
i.j E [M]{[)), E [N].
In addition to these equations above, we give the complete inner product update rule as follows
(nni knj
(t+1) ((+1)) (0) (
= =TT
@ + (W
(t) (W
+ +AW ST, - (. (
= (Ag BB
(1)
+ (q
(Ag (1) Ab
a (l+ kig
(1) (8)
(1)
(k_)) k
(1)
(0)
(k k
N
M
(196)
+ + (at In.el
(q 310
In..
((t)
Faghon a (g gn.))
(9)
N
M
(0 (onh + 4 k +
(2)
nad 1 n'1)
n/=1
1=
(0()) 1 + =0000
(0)
(()
Σ
(C)
q
Jan In'
for n # nT,
(q 1+1) k ((+1)) _
(0) (t)
= (Aq (00) + _ AK (0)
+ (A (1) AK
M
(0) (1)
Σ
(1) 44)
nES
1=2
M
+ 10 (0) (1)
(197)
NES+
1=2
M
10
+
WES_ 2
M
(8
10+++ g
(t)T
(
(T)
nES.
2
91

---

(q. - (111)) - (t)
(+1)
(q q )
=Aq (1) (1)) + (q
(t) Aq #)
(Aq (1)
Aq
( (q
(1) (0) Σ
n.
nES
107
(1)
E
Σ
(198)
nES_
Σ
nES+
M
(8) (
LT)
RES t=
11
(++1 (
12 |1 1
(t)
=2(A (1) (1) + (Ag
(9) A441) (1)
M (1) ( +2 2 Σ Σ
20 (0
10
(9)
nES+
(199)
M
+aaff
(0)
(1)
MES 2=2
(
BES+ 1=2
11 (1+1) 11 1 1| (1)
12
2( (t) + (A (8) Aq
2a (4) (8) 2 Σ Σ
(0)
an-
BES 2m
(200)
M
(1)
(t)
WES
2=2
(1) (
(0T
BES 2=2
1g (1t11)11 11 12
=((Ag
(8) (
(Ag (1) (9)
1 Ago
N
M
20 (1m.
(1))
+ 20 n, (1
+ 2
n Aan (0))
n
(201)
N
(a 10 + az,i k
(0)
n/=1
M
(a k
1000 (a
(()
n.inn k (t)
n'!1
m/ 1=2
92

---

(9: Innt (9+ In.il
(++1) ((+1)) (1)
= =A (moi) + (9 1 Aqnd (
(0)
(0) Ag.
(
N M Σ
(0)
(9) .k + (0)
n.s,- (q
Σ
(202)
M
Σ
+
m.
nES+ 1=2
N
M
a n,,,
[!)
+ ta a
(0)
Σ
(9 (1+1) (1+1); (1) (0
An.i (q 1n.i
(Aq (1) + {q" Ag (Aq (0)
(1)
E' 1=2
N
M
+ .
(
(1) + a
n
nnint a _ (0
(203)
(6)
nES im
(0 (
(0T +a (8)
(1)
Σ
a
1=2
(t+1) (1+1))
(0)
(n,i : In. (q In.j)
= =(Ag M an.)
+ (g !
+ (Ag..... (1)
a (g
(t)
(1)
n,ú,- (g...)
wnind (g (00)
(
N
M
+ anngh (Tu.
(1)
+ anj.
(1)
(g
Onjan (g (2))
(204)
M
+ (arit 1 + +an
(1)
(1)
n'=l
N
M
(000 1:T + +ap
(1)
1(OT
ni k (i)
+
n'!
1=
93

---

(1+1) ((11)) (1) (1)
(nni 98. in P
-Agg [TT)
(t) (t)
(n.
(t)
(Ag. AqTi)
M
(t) (t)
(t)) +
(t))
Onin' a (9..j kg)))
= aniit (mn.j
On..- (m.
n/=1 1=2
+ an
(1) (1) (0)) (0)
(n,
ans-
(g k
angnn' 1 In,i n'.)
(0)
(0) 10
(205)
N
M
+ a a
1: + +a
(0
n.i,n R!!1)
N Σ
M
(0 (110T (1)
A (0) +
Biin' k (0)
w/=1Ea
1=2
for n + n,
(1- 11 -
-11
-(
(0) +. . + (A (0) Ak
M
23 (9 (0) (0) +
nES,
(206)
(of
(()
n 1=2
M
(0 4+
(A)
Σ RES+ 1=2
| 11 - -1
-2(Ak (1) (,A(9)) + (AA A
M
23th (q +2 Σ
g
(1)
BES_
1=2
(207)
M
(300
a(t)
Inoi
nES_ 2=2
M
(300) g
(t)T
(t)
RES_
1=2
94

---

| 1 114
2(A (0) (300)) + (Ak A1
N
M
23 m.d.t (4+ (0) + 23
3 Minn' (o k2
+
2
n.J.
(208)
+ (
9+
(1)
-Pni
(t)
n,i,' Ann'
n/=l Im2
N
M
(B00 ( + +1((1)
(O)
(O)
+ Σ
g(0)
(0)
q
Riin'' 9
(44 (1+) 1: (((t1)) - _ (2)
= (Ak (0) (1) + (K AR. + (AA AA
(0)
301)
(8)
E'
N
+ M. (9
(0) (0)) + P.,,-
(1)
s(t)
a. / 19 (0)
(t)
(209)
M
+ (84) q+
(8)
(1) n' In's)
E'ES.. 1=2
M
(0 9
(O)
+ n,,
+)((t)
(
Σ
gio)
niin' . In'd (0T))
(1t11) (.t - 10) (2)
= -AA
(0) (1) + (0) (R + (A (A)
-.0 (a (1)
Σ
g(t)
(g (()
10
n
D'ES 1=2
N Σ
+ -00 -00 (0 +
p(0) P.,, ( 10
Σ
Ag 11 (()))
(t)
(210)
M
+ +(300 (1)
g
(t) In'
n'
n'ES_ 1=2
M
(B (001
(0)
g(t)
q (iT
git)
(T))
In'.
n==l== 1=2
95

---

( k - (Ki m1n
= =(Ak ") k + (k n... A1 + (AA n.s:
(1)
(1)
M
- ( (8) ( + (00
00 (
800
n.
1=2
N
M
+ p (4+ ( + -...j (9-
1,30
+
3(0))
(211)
+ (p00 (9)
(p + + q (1)
(1))
(t)
m In'd)
n'=1 1=2
M
(3 (n 7t + F.jj. 9
(OT
+.000
(
(1)T)
m. In's1
1 1==1=2
(1+1) ((+1)) _ ( 2
<Ak (1) + (k ( A + (AA
(0)
AKD))
N
M
= - (9 R.j
+ ((0) 1 (q M
(1)
R'=1
+ 000 (9 (1) (0) 300
Mun a A k11))
(212)
+ (B 1 1 q
(1) +1,00
((t)
n. In'.)
N
M
( (20)+ 9+ + T.j.
(0)
+03
(t)T
(t)
(T)
+
m.j.n a
Ix'.1
12 w/=1= 1=2
for n + n
(+1) k ((t1)) -(g.0 k (1)
(0)
(9
= =Ag (100) + (9 AK (0) + (Agth
(1)
(
M
a,t (k)
(0) (1) 10
(0)
(0)
WES
2=2
M
+ 00 (1) (
(213)
nES_
i=2
M
+ 0+++
nES, 2=2
-000 g (f)T
((t)
(i)
+
RES
1=2
96

---

( (+11 k 1 -4 (
(Ak (1) R00)) + (k44 A (AR AA!
m _ kn))
RES+
M
30 (q
19
Σ
(214)
RES
(
An.s
n 1=2
(C)
(
9n.i
nES
1=2
F.3 Proof of Lemma D.4
By Lemma D.3 we have 1. 1.1 O for t E
Let To = O
yd | woll
[0, To] by Lemma
D.3 Plugging this into the expression for and gives
Ja
(1)
71
Σ
ge(t)
NM
nES_
expl(c4 (2) (:(
(v
M
expl1g (1) k x21) + Σ
expl(1f) (1) (())))
3=2
exp((gt) (1)
k(')))
)²)
exp(11t (1)
k(11)
Σ exp( ((1))))
M
f=2
M
exp( (1) k((0))
(v()) n.
exp((g (t) k(1)) + Σ exp(((e) (1) (
exp((g (t) ((t)))
M
exp((g)) (1) K (()) Σ
expl (q (1)
(())))
1=2
11 3NM
o
NM 4
O 1llpal
)
dA
where the inequality by e(t)
< and the property that attention smaller than e.g.
exp((g)") (1))))
< 1 ). We also have
expl(g () (0 E exp( (0 (0);))
1 (((() |
NM
97

---

(-v! exp(1et") (2) ,k4)))
M
expl(1+ (0)
. ,b4 + Σ exp((14)
(1) )
5=2
exp((1 (1) k((1))))
M
exp((g (1) 14 + Σ exp((a (t) (2
5=2
expl (4) (2))))
M
exp((1 1 (11)) + Σ exp( (1) ((09))
1=2
exp(194) (1) k((1)))
)9)
exp((gt) (8) k k(1))) + Σ exp ((9 (1)
M
k(r)))
exp((g (1) (!(!))
(vrl) RgH
M
exp((1 (1) ((21) + Σ exp( (a+ (t)
(11)))
j=2
exp((1+) (t) k((a)))
exp((14) 19 R 1(1)))= + Σ exp((14)) (1)
M
j=2
M M.OM O(d,ht
NM
O n
N
where the inequality is by- 20 < 1 and the property that attention is smaller than 1. We also have
M
Σ p((t)
n Σ (en)
NM
S
exp( (. (())))())))))))
(e)
expl(gn)) ( k ((1)))
Σ
exp((1 (8)
(1 (())
1=2
exp( ( {(1)))
(1)
)2)
M
b24)) + Σ
exp( (epp(aa.)) ( (())
j=2
M
exp( (g (1))
(v(() n,
M
exp((g 10) (1))))
+ Σ
expl ((1)))
y=2
exp((g (0) (10))
x(0))) + Σ exp( exp((g (g (11)))
17
332d
3N
x2_
d dlog (4N2 M - O(d
NM
2
4
98

---

0 nozd
where the inequality is by - ((0) S 1, Lemma and the property that attention is smaller than 1.
Similarly, we have
(0)
11 0
d
(
n
13 m.+.1: 1...111111--11111 = 0
dj N
n. = -0 nood
la n.1 13
- a. w (s(t)
d N
for E To]. Next we use induction to show that the following proposition A(t) holds for t E [0, Tol
A(t):
[g k " Kg 11). Kg 12 Kg 2
o max (a oz of v dn 1=g(6.22
Kgl 4 19)1,(99 d (g 2:2
= o max (1114 ozd] of da (dllog(622M210))
(1:4)) ( (k 2. 11 16 n.. ((,g)
o max (11p4112, ozd] of dn log(6N22218)
11 12 | 10 (1111110,d,0
1| 1 -
(0o30,ddn)
for i,j e [M]] n, [N].
By Lemma C.3 we know that A(0) is true. Now we assume A(O)...., is true, then we need (c( (T+1)
to proof that A(T 1) true. We first proof (g
k ((++)))] = o (max [ of
/ dy Mo2(6N22218)) as an example.
99

---

1 k+
( (0)
M
= a44 (t)
12
aata (k4 AN
(4)
nES+
i=
M n
+ 1
11
(4)
g(1)
n (9+ I2..
(1) (0)
i0 n.
+ a K
2m2
(215)
T
RES
1
S 0
e (11
di
+ NM
7
o( max{]l 112,0 oz on ddn (og(622222218)
NE
+ flower order term)
= (1
Taking a summation, we obtain that
T
19 k4
(T+1)
x - 1(
(0)
+
(q
(1+1)
(of k ((t1))
(2) (
1=0
To
< 1 (0) (0)) ) +
(q
(++1)
k +
(t-1))
- (g+
(t)
1=0
s 0
max [|| d3 σo dn log(2222211)
+0
(nilullloo))))
nd |1
0
(1ax o. o.
ddn logg6222M23
0 011111202
[0max(]1u1] o on ldgoo
(216)
Similarly to (q ((2) ) it is easy to know that the inner product does not change by a magnitude
more than the product of maxfa, B} and max (q. q). (k. k)} in single iteration, which can be expressed as follows
(attt)) - (1) |:0))
= =0( (maxx
glllel12 nozd ))
of
d d
(217)
mmllul2 -(
e (max || 4e d 2
d
max(]|u o3al)
100

---

where the second equality is by the condition that N SNR 22(1). Taking summation, obtain that
T-1 (+t1) 1((1t)) ,100))))
Kg (1 x(++11) - ( 5
1=0
To
mill Bo dh
max 1
2, o oz
(218)
t=0
o (1 [0l
maxi 02)
gdg | 1m11@|
( max [ ozdd oa..).
It is clear that the magnitude of (q (T-),,, (((+1)) (q k:0)) is smaller than max ozd] of /d, log( Thus the magnitude the bound for (q (T++),, k((+1)) the same as that of (q 1(()). The proof for (T+++), a (((+1)) and (k(T+1), k(T+1)) is exactly the same, and we
conclude the proof by induction.
F.4 Lower Bounds of a and [
In this subsection, present some bounds for and 8 which be used in D.2 and D.3 All the calculations this subsection are based on the precise expression for a and / in F. and assume that B(Ti)..... B(s)) D(Ti), ..... D(s hold (s E t)). Then the following propositions hold:
v(() 2 3M |V/(
V(0) S -3M -1V49)
so softmar (1tmar(gg (6) (( ", softmar((d..)) x()))) 2
M o(),
soflmar (g (())))) so softmar((d (()))) S
+ o(1).
M
Now we give the bounds respectively for a n ny- n. a nin' g(0)
(8)
(8)
(t) (s)
(e)
g(s)
Pa,t5 g(o) g(o) g(0)
P pnit+ a(e) Pn..- g(o)
-
101

---

exp((<t") (3) x xx
(1)
17
Σ
NM
M
RES+
explie+" x( + Σ exp ( k!
_=2
expl11 ,.()))))
(v(6)(
exp((g+ 1) + Σ expl(14 (5) (2))))
M
exp( (0)
M
exp((1 k(
Σ exp( (1
-=2
expl (1 kb:)))
Σ
NM
exp((a( x(0))) + Σ exp((14) ( x(()))
ES
(219)
expllg (.((1))))
(v
M
exp((1t (0) b(4)) + Σ exp((al)) (1) (!
j-2
M
exp(<g k!!l)?)
(qt exp((4t) (8) {((")) + Σ exp((1 (8)
M
exp((1))) (0) (((()))))
1 Σ
M(s) | 11
2
2NM
exp((14)) (5) b((11))= + Σ exp((1+ (())))
M
(x)
nES+
y=2
expl(94 (5) k k(())))
(11
exp((g (8) b(4))) + Σ exp((g()) ((())))
(8)
==2
where the first inequality is by v(() > 3M of attention equal to Similarly, we have
||(]] the second inequality is by the fact that the sum
+.+
exp((g..) (8) x((()))
Σ
((s)
2NM
exp((gf") ((0))) + (M - 1) exp(max (( (x) exp(max (( (11)))))
b!))))))
nES+
(220)
exp((g!)) (c) 1( + (M- 1) exp(max ( (q b!
(
exp((g (5) (((1))))))))
17 Σ ((() (1(
2N
exp((q ()) (1)) + (M 1) exp(max (g (x(")))))
nES
(221)
exp(max( (z) b
exp((g()) x (1))) + (M- - 1) exp(maxf((g (g (1)))))
102

---

85-7- 2
expl (0) ,b((")))
7 Σ 2 [
2NM
expl(g_") (c) ا(اي) + (M- 1) exp(max (ي_)) (11))))
nES_
(222)
exp(max{ (3) (
3 exp((g_)) ; k 1) + (M- - 1) exp(maxx { (a (0) 1
explief) (5) (k(1))))
π ((+)
1
n
NM
exp((o b("))) + > exp((al (0) (A(3))))
M
y=2
exp((at") (1) b(,b))
exp((g/ (00((g (()))) + Σ exp((a (5) (b()))
1=2
exp((et" (8)
k((l))
n.i
exp((g_)) (3) ((()))))) + Σ expl(g (.(0))))
M
5=2
exp( (1 (5)
Σ (v kitt
n
expl1g (5) x(41))
Σ exp ((g (e) ,((1)))
M
j=2
exp((gt+) (c) ((tt)i)
NM
exp((1! (1) k(())) + Σ exp((al)) (1l)))
M
y=2
expli24" (3) ; ((1)))
exp((g (1) (( + Σ exp((g (s) (!))))
M
y=
+ +|((b] +
(1~53) ( //4 +o(1))))
lgfti
exp((g()) (5) x((o)))
< n (1(0))
NM
exp((g!)) (((0)) + Σ exp((a ((1)))
2=2 exp((g ((4))) + Σ expl (0) b(())))
exp((at") (1) 11 (((1)))) M
v((0)
+ 2M max [V(1])
3=2
exp((g4) (8)
: ((()))))
|P1/ 17 (M)) NM "
exp((g (") ((اا) + Σ exp (0) ((1)))
M
y=2
explia!") (3) ((())))
(-v())
expl(g ((a))) + Σ exp (1 (5) {(())))
y=2
103

---

exp((1 (0) (k(-))) M
+ (66))
exp((et" (5) k 1² + Σ
expl (5) An1))))
==2
exp( (1) 10(11))
4NM (()) p (3
exp(ief" ((())))
Σ eep(((9) (gp (") (A()))
5=2
expl(a!") ,+()))
M
expllal x(1))) + Σ Eepp((g) (g ( x(()))
j=2
where the third inequality is by v(2)= 2 3M [v and softmar( b()))) M o()) Similarly, we have
a S
expl (+) k.(1))))
1 (1
((())
4NM
exp((g") )(ا" + Σ exp( (q (5) ,.(3)))
M
exp((g(") (5) k (1)))
exp((g_") (0) (b(()))) + Σ exp ( k!!))))
M
1=2
g(t) 71lpe11 p(s) n.t, NM x
exp((g (c) k b(11))
(v
M
exp((g!.) 1111)) + Σ
eepp( (g ml13))
(0)
==2
exp((gn2) In.i b((11)
)²)
(()))) + Σ
M
exp( (ap((g!.) (8)
(g k1(0))))
y=2
M
exp((g..) k(())))
(v(e) n,1
M
exp((d!.) (8) (1)) + Σ exp(( (8) (!!))))
5=2
exp((gni) k((hk))
exp((g (e) {(0))) + Σ explig (8)
M
k(()))
nllul1 p(s)
exp( (6) {(1))))
NM
exp((g x(0)))) + Σ exp( ( (1)))
3-2
exp((g (+)
: b(())))
(1
exp((a (())))) + Σ exp((g ((())) ((()))
(5)
2-2
M
(xp( k(())))
(v(+)
2=
n,
exp((g!.) (5) A((o)) + Σ
M
exp( (pp((ian))) (0)
(nn b.())))
2=2
104

---

nllul (%)))
exp( (6) ((21))
2
NM
1 Σ eepp( k11)))
expl b(11)))
(v
exp((an.) (6) (( + Σ
expl(gn (5) ((1)))
j-2
((0) M
exp((an..: k(1h)))
Σ
2
expl (5) 1 +
Σ exp (N) pip (121))
M
5=2
expl k(111)
n||n| (()) p()) (s)
2
M
expl (21) + Σ explligh
M
/=2
expl(gn.1 (11)))
explian.) (11))))) + Σ explign))
M
(4) ((())))
where the first inequality is by V4 2 3M
||[]] the second inequality is by
M
softmax((a (do.) ((2e)) = (1 softmar( (( k. 1)) Similarly, we have
-
exp( (5) k. k2())))
n 0111u123 (*(s) (s)
(()
2NM
exp((an.) (10) + Σ expl(gn.
(5) k())))
exp(ig b((1)/
M
exp((g (5) )) + Σ exp( (8) (.1)))
1=2
(3) 17
Sw (0()))
NM
(5) exp((g exp (11)))
exp((gn0 (6) k (1)))) + Σ exp((a (8)
1=2
expl(g x k())))
)7)
M
exp(1g p, (()))) + Σ
exp (n (1))))
1=2
M
(g k((1)))
(v(s)
n'
exp(1g ((11))
expl(gg (8)
(n b()))))
exp((gle) 1((0) (1)2))
M
exp((a!") y A: x((0))) +
(3)
E
explial (()))))
3=2
+ flower order term)
105

---

n 11
expl(g (s)
, k(,)))
C/ (6(s))
NM
exp((gn'." (2)))
expl(gn kob1))
expl(g {
(v
M
explidheee 1 ( + Σ
expl(g b(
j-2
M
(v(l) mip
expl (2)
k((kk))
explign (0)
b(11)) + Σ expl(e (0)
+ (lower order term]
nl Ew 12
exp((gn (e)
k()))) (())))
C. (0(s))
NM
expl k (((1))
Σ expl exp(1a k(2)))
exp((g y k(
(v
M
expl )) + Σ exp((1n b10)))
M
exp((1n))) k2
2
k=2
expl(gn) (5)
ke ((111) +
Σ
M
expl (xpp((
-
+ flower order term}
12
expl(g b(())))
(O(s)))
2NM
M
expl(a 2
(())))) + Σ expl (q (3
j-2
exp((gn) 2 (()))))
expl(an)) (5) k
(22)))
+ Σ
M
exp((1 (a)
1=2
+ flower order term]
no
exp(ign")
{(()))
E ()s))
5NM
expl(g!?) x(40)) 1)
(6) expl(ee k/(0
y=2
expl(gn0) , (())))
M
exp((g (0) ((11) + Σ
3=2
where the flower order term] is by the property that |m [12 is much larger than (Emr,r, En,x) in
Lemma C.4 the condition d d= =d= (23 the first inequality is by v((1) 2 3M [V[[] the
zdd
last inequality is by & >
2" and absorb the flower order term). Similarly, we have
nood 5NM
exp((g (8)
22, (2 (((s))
expl(g k (1)) + Σ Eepp(( (5) M
(g (1)))))))))))))))
j=2
106

---

exp((ah",00) (0) , k(o)))
expl(g b(0))
explign b
Pa S (0())
NM
exp((g4) (0) ((())))))
exp((g4)) (0) b(11) + Σ exp(1a4)
(6) (())))
3=2
exp(1g (-) /1))))
exp((g!") ((1))) Σ expl (0) (
j=2
cxpl(9t k1
(r(a)
M
exp(1at") k((2)))) + Σ exxp((at) (0) (
exp((1 (0) ,,/)),)))
expl(gt)) 1()1 Σ
exp( (q (e) eep((19
5=2
+ flower order term}
exp((a5") (5)
nl + 212
(B(s))
NM
exp((14") (1) x()) + Σ exp(ia ((22))
M
5=2
exp((14 k(())))
(- v
M
expligt) (1) x(1))) + Σ expl (5) (2
j=2
exp((gt")
+ [(()) "0.0" (1
exp((g (5) (())) + Σ exp((g (6) (4, (1))))
(lower order term]
exp(1q (5) 1 [1
n 12
(0(s))
NM
expl(at") x(21)) Σ exp((a (1) ,(()))
5=2
exp((al") (3) k()))))
((0)
exp((al)) (()))) + Σ exp((al)) (0)
M
==2
exp((g(") (3)
((())))))
1 (c)
2
M
exp((at)) )(())) + Σ exp((a ((:)))))
2=2
n1l v 11
exp(1g (5) k k1,)))))
C (B()))))
2NM
expl(g!") (2) {(:))) + Σ expe (5) ((1))))
/1
2=2
107

---

exp((e4") (2) ,b())))
exp((g (8) k((1)) + Σ
exp((14
(.) b1))))))
5=2
flower order term}
1 nond
exp((14" (5) k,()))))
5NM fw (0(s))))
exp((14 b(411)) Σ exp((14 (4) (.2)))
exp((1)) (2))))
expl(a k(2) Σ j=2
exp(((( (
where the flower order term} is by the property that I| 112 is much larger than (Ew,a,, Sn,a) in
Lemma C.4 and the condition d = 50(5 N22). the first inequality is by v(() 2 3M [V the
last inequality is by En 112 N a.d and we absorb the flower order term}. Similarly, we have
(/4) Par S
nopa 5NM 5NM
expl(g_ (6) k.)),:)))
fn (0))))))
exp((g_)) (5) ((o"")),,[["))) + Σ
exp( (ppl(a90 (1 (
==2
exp((g_)) (3) (1)
exp((g_") (5) x(0))) + Σ exp((1 (1) (2()))))
exp((g!1)) (s)
k.13))
Onims
(5) = Σ
7 p(s))
Sn En,k)
NM
k=2
exp((2 (e)
b(()))
+ Σ
M
exp( (8)
P=2
exp((g (z) k((())
((0))
M ((1))) + Σ exp((gnn) (0) (((?)))
expl(g (5)
P=2
expl(gnnk: (())))
exp((g! ((11)) Σ expp((a() (())))
exp((g (()))))
- (v(())
expl(g (t)
((11)) Σ expl(g (8) ((g)))
1 E (s))
exp((g 1
NM
exp(ig (+)
+ Σ explig!)
exp( I (0) k(2)))
(-v(
exp((g()) (1)) + Σ exp((g!))
(s) (())))
exp( (1 (1))))
(5)
+ ((8)
M
exp((g ((:))))))
Σ
exp( expp(gn (s)
((())))
P=2
108

---

exp((a (2) ((1))
Σ (Vil)) lifi
expl (0) 11 + E expl(1n (0) (
+ flower order term]
7 E (())
exp((
(nn.i (1)
S
NM
expl (4) b((1)) +
Σ expl(1 k(10)))
M
(5)
1'2
exp(( (6) 10
(-v4a)
expl(gn. k (()))) + Σ exp( (6) (()))
P==2
+ n.j
+ (Irla)
M + o(1)))
+ flower order term]
exp( (5)
(q k
S 7 g())
NM
exp(ignn) (1)
( b(11))
+ Σ
(5) expl(gnai k(9))))
2=2
expl (5) (121))
(-v
2 max (r((a)
expl( ((1))) + exp exp(((g (5)
(g ((()))
+ flower order term}
exp((g (5) (0)
7 (ss)
S
NM
expl (e) (()))) +
Σ
M
(5) ((1
1'22
cxp((g..)
(8) (:)1))
(- (-v[)
(4) k()))) expl(gn.) (5) ((11)) + Σ exp((gn.)
P=2
exp( (5)
(9 (())))
3 (())
+
4
exp( (8) (111) + Σ exp((gg)) (4)
(gn, (1)))))
2=2
+ flower order term}
exp( (5)
(g k(a))) ))
111 p(s) v(o)
4NM
exp((gni) (3) (())))++ + Σ expl (9.0) ((
M
exp((g (11)))
exp expl(an, k4 ((2))) +
Σ expl(g (4) (())))
M
+ flower order Lerm]
nood
exp((q (x)
9NM M eh p(s))
M
exp(ig (3)
(()))) + Σ expl exp((gu!! (1))))
p p=
109

---

explidn (5) b(,1))
exp( 1) Σ exp((inn. (5) (())))
where the flower order term} is by the property that l|Em,ill is much larger than (En', En,s) in Lemma C.4 C.4 and the condition d = 2 2222 ). the third incquality is by v1+) z 3M |V/.?]
and so so fofama(a(gg (*) k(())) 2 _ of (1,, the last inequality is by ||5m.:112 2 = and we absorb the {lower order term). Similarly, we have
n, S
nood
exp((gn.)) (. k.(3)))
(()))
9NM
M
exp( (8) x(/1)/² +
Σ
exp( (5)
(gn (1.
2'
exp( (5) k(1)))
M
k((1)) +
Σ
exp (u ( k1
2'
17
(1()) n
p.j.m
NM
M
exp(iq (5) (ikkk(4)) k())))
(0)
(EEm.) Em.h)
exp((gn.) (o) (44)) + Σ expl( (1) ((2))))))
M
k=2
cxp((an.. (
M
exp((g (9) 1 11 +
Σ
expl(g (0) ((())) ))
1==
exp((g k((h)))
vila) (.mn:::E.hh
exp((an ((11)) + Σ exp(((e (6) ((
M
1=22
exp((gn.) (6)
Σ ((En.):
explig (5) ((1))) +
Σ exp( (pp((gn))) (9
1=
exp((gn.) k(111)
M
exp((gn.) (5) (1)))
+
Σ
exp in k(l))))
2=
expl k..)))
En 12
p((e)
NM
expl (5) 1(411) + Σ
expl eexp((an.) (an ((9
1=2
expl
(q (x) (10)))
expl(g (3) x(:))) + Σ
M
(8) (t) expl(g
(011))
exp((g..)
(2)
+ v(() an
exp((g (c) b(0)) + Σ (x
M
(s)
(())))
1=2
+ flower order term)
110

---

expliann (0) b!o)))
7 En.j 112
M(())
NM
explign h b411) Σ expl((an1) (11))))
exp((gn.)) (4) (b(,))
v
+ max |ilg 1)
expl (()))) + Σ exp(( ( (1)))))
+ (lower order term)
exp( (up((gni: (0) (
7 En. 1 ((s))
NM
explig (0) k4 k(2))) + Σ (a k1)
(e)
2==
exp( k(1)))
- v(())
exp((g (4) ((1))) + Σ
expl(g (8) ((0)))
2=
exp((gn..e ((11))
1 +
2
expl(g (9) x(2)))=
Σ
M
(4) k())))
+ flower order term}
7/ Em. 112 p((e)
(8) b..)))))
2NM
M
exp((g (5)
|(= + Σ exp(ia (c) k(?)))
2=
cxp( (1))
M
expl ((:)
Σ
expl (g ((1))))
P=2
+ (lower order term)
< pood gls)) (())
exp((g (8) (t)
5N
M
explig (1)))) + E
exp( (g k.2))))
1=
exp( Exp((an..s k(()))
M
exp(
Σ
exp((g (1))))
where the flower order term] } is by the property that =| is much larger than (Ewre" En,s) in Lemma and the last inequality is by ||5m N 2
and the condition d = ② =2N=dd) the second inequality is by v(*) 2 3M [V())
add and we absorb the (lower order term). Similarly, we have
2100 P S
nopd
exp((g (e) ((0)))
(())
5NM
exp(1g x (()))
E (exp((g..)) (1) k((
M
1=2
exp((g!.)) (())))
M
(pp((gl.) Σ exp( (((((a k()))))
P=2
Summarizing the above equations, we have the signs of and B as follows:
111

---

(6) ((0) g6.) (3)
g(6)
tits on
2 0,
R,t
(a) (5) g /00) 3/6) ninnn S 0.
an rs ning ni
F.5 Lower Bounds of ( (q (q.k)
In order to give the lower bounds for (g, k). we need to rewrite the bounds of a and /8 in a more concise form. We first expand the equations in under the assumption that B(s) holds for [Ti,4.
(x)
exp((g+" (8) k((1))
Σ
-
(( p
(3
2NM nES+
M
(0) , explixf) k x( + Σ exp((1+)
2
exp((14 (5) (((1)))
exp((et)
(4) ((11)) + Σ exp((1t))
M1
(8) A((g)11)
y=2
exp((g4) (())))
1
Σ
2
(0) (xp(((")A(()))) + Σ exp(ig (8) ,1(0),)))
EES
y'=2
Σ exp((1 (8) k1 p=2
M
M
expligt) (2) k {e))))
+
Σ
expl(at (e) k(()))))
-=
expl (1 , k(()))) M
(223)
77
(( 1111 11(
2NM
exp((g4) (9) x(11)) + Σ expl(1 (5) ,(.))))) ,
2
exp( (
M
exp((el) ((m))) + Σ
expligt"
(5) (/13)11
n
p()) 1 |
M o(1)
1
2NM
expl1gg) (0) ((
expligt) (4)
x(11) + Σ exp((gt))
M
(8) ((())))
2
exp((ef)) (1()))))
11 ((() |1(
M o(1)
2
C Cexp((g) (0) k(2)))
O2O @ IB(( Tı).
1
exp(
(8)
where the (g!h) (())) in the second incquality is a particular choice (we will characterize the dynamic
of (g 1(()) - (q (o))), ) the third inequality is by softmaz((q"" (( (+) x(7))) 2 (1r - -o(1). In
the fourth inequality, by (T), ((TT) = o(1) and the monotonicity of (q k(())) (gg (8) x(7) is
increasing and (g( (1) kll)) is decreasing), there exist a constant such that C eep(pppe (g k x("))) Z
(4)
expl(gl", ((())) + E exp((a)", (()))) In the last inequality, we plugging the lower bounds of
112

---

V and ((0) and then absorb all the constant factors. Similarly, we have
(05llll@@wlllll Tı)
860 2
(224)
N
exp(A (9)
| Tı)
( 2
(225)
N
cxp(A n.-1))
الCا|اااا|| wo (3: -T))
B(0] 2
(226)
N
exp(A m.
(8)
antt
exp((g (8) ; _k(())
4NM 11 (2
11 (((x)
M
exp(ia (()1 + Σ exp( ..b.),))
exp(1q b(1)))
exp(ig(") (1 + Σ exp(((!)) (3) ((f
M
4NM 11
(11 o(1))
1
(227)
M
expl(g (3) k(o1))
exp((g {(0))))
Σ expl(e (8) (()))))
p=2
M o(1)) expl(g!") (+) ((1))))
4
Cexp(ia!") (0) ((11))
17CCs 1 Ti)
N
exp(A
where the inequalities similar to 223).
Similarly, we have
woll(( Tı)
n.-J s
N
exp(A n.-.))
113

---

antt
=11lp11 (()))(()))
expl(gn.!
(6)
2NM
explign (4) k (())))
explignn1 kile)
3'=2
expl(gn.) (11)))
M
exp((gn0 {(1))) + Σ expl(ig (1) (() ((().))
2²=2
n p(( v()
(N (1))
1
2NM
M
M
Σ exp((in..)
(8)
k(.))))
(228)
exp((g (0) k(e)))
Σ explian. (5) ((3)))
M
j'=2
n (()) (0)
2
(())
M ((1)
2NM
expl An (1))))
explig ((())) + Σ expl (8)
M
(qn.i: k(())))
y'=2
exp((g k(1))))
n| (())
(3)
o(1)
2NM
M
C Cexp(1an.) (3))))
(x)
3CColllll@lwol/// Ti)
1
(o)
where the inequalities is similar to (23)) and (g ( is a particular choice (we will characterize
the dynamic of (g k(2)) - (g () Similarly, we have
8(0) n r 1 wo [[s Ti)
1
(229)
N
exp(A ()
17 (1 (T))
(9)
n
2
N
(4)
(230)
1/ 1 - Ti)
0
(0)
(231)
N
exp(A(.i.-.)) .)
8(0) kngt < 11 Ti)
(232)
N
exp(A (8)
8(0) 8 < 133 Ti)
1
(233)
N
exp(A (o)
13 IB(s 7i)
(3) Cnjinn 5
(234)
N
(x (.)
1 c -i))
9(6) Bujmn
S
(235)
exp (1
114

---

With the concise lower bounds for a and |i above and proposition C(s), we will give the lower bounds for the dynamics of (q. k).
(
(0+1) k ((++1)) - ( (1))
=0++
(3)
|
(+)
12 +
(
nES i=2
+0(0 1
+
nES
(o_))
BES+
(236)
M
(pL)) (3)
(T)
Un
nES+
(2)
g
1 10
flower order term)
Co
|
/woll
Ti)
@(1111/03dd))
N
exp(A
flower order term]
[Col1111111 ds
N
exp(A
where the first inequality is by (233) (224). the bound of (0) 12. ( in stage II, the second incquality is by absorbing the {lower order term) and the constant factors. Similarly, we have
(q (xt))) (6+1)) (ي.0) k (a))
C wo 30 dn Ti)
(237)
exp(A n,
(8)
115

---

(9 ( 11 (99 (1(1))
= a++ (14 ((a))) +
(6)
(6) R'11)
n.j
nt n 1=2
N
M
810)
+ g(5)
Pay.- (9+
gi8) asin //4 .22.))
n==ld
R' 2=
((()) T
g(t)
(8)
g(0)
(T) mann mowaan
In'd
(238)
+ n. |1119/ (4) 112 + {lower order term}
C wo 12 -Ti)
(ooRRddn)
exp(A(1)
(8)
Tí)
+ (11111207,1)
N
exp(A (8)
(.t
&lower order term]
17 petpd//lll@@@wwwoloop - Ti)
<
N
exp(A (
m.tu
where the first inequality is by (227). (232), the bound of [|< 12. 1 in stage II, the second inequality is by absorbing the flower order term) and the constant factors. Similarly, we have
(q (o+1) k ((021)) (a k!.)))
7'0o0dd111115 wo 12 Ti)
1
(239)
(
(8)
116

---

(s+1) k
( (
N
M
(8)
[ 1/ + Ox an
nan (kf) (2))
=
t==1
+ + (q 9nn.
n'ES, Im2
+ (ns. kt
(5)
(4)
(c) nad (11
n'=
(0/)) (
T
g(o) (
(240)
E' 1=2
= a [144 1 + (602
(5)
flower order term]
GCozdd
| wo 11 -i))
(11e1120Rch)
N
cxp(A m.j+t
C
wo
(s T2)
(0zoRddn))
N
exp(A (6)
(iitt)
+ flower order term}
17 GCooBddl wo fo - -T1)
N
exp(A mii+t
where the first inequality is by 233 (228). the bound of || 11 in stage II, the second inequality is by absorbing the flower order term) and the constant factors. Similarly, we have
(0-1) ((t22) (q
(q
7CCoodd1uu115
/ wo 12 Ti)
1
(241)
(8)
117

---

( 1 (a ( knn)))
N M
= anitt (16 (3) (()) + (a)
(8)
an, (k 1-) nnj
(6)
nnin a (k k
(6)
+
n'
I=2
+ (9 .qn.) ((0)) -Pn..-
(8)
g(0)
agnnt (9 1a 9
(
an 14
+
(5)
n'=1
N
M
(ph). (T) +8(0) (N)T
(x)T
(242)
Pns at
Pnjn' fn
n'=1 1=2
= aninngl 1l + pn...n.ill
+ Hlower order term}
2n0 2 Wo ds
1
(0
N
exp(A (8)
flower order term]
p p wo ds Ti)
1
N
exp (A (5)
where the first inequality is by 234 (35).. the bound of
||
(
I2. l in stage II, the second
inequality is by absorbing the (lower order term) and the constant factors.
E.6 Upper Bounds of q, k
In order to give the upper bounds of (g. k) in stage II, we need to give the upper bounds of a and B based on the equations in F.l under the assumption that D(Zi)...., D(s 1) hold for s [T..t..
(*(()) 11111
NM
exp( (((1))))
(V!
exp((gt) (8) ((1))) - Σ exp((1f (5) (()(())
exp( (5) b(11))
)2)
exp((g( (c) x((1))) Σ exp((g (q (0) x(())))
2=2
M
cxp((q (x) (1(1))))
(v(0)
exp((gl") ((()))) + E exp((al)) (5) b(()))
3=2
exp((g (8) ((()))
M
exp((g(") (5) (((ي))))) + Σ exp((g!") (((")))))))
2=2
expl(q!" (8) (((()))
"
p | Σ
NM
nES.
exp((g(") (1) x(()))) + Σ expl (2) (.()))
1=2
118

---

M Σ 5=2
exp (19 (0) (b
(v
M
expl(et (0) 11 + Σ exp((at) (0) ((l)))
/=2
M
expl(a (0) k(()))
v(6)
M
expl(g+") 1( + Σ expl(q (5) 1
2
j=2
Σ expl(a (6) k//1))))
Σ
||al23 (v
y=2
NM S
exp((g (1) x(4))) + M
Σ exp(((t") (5) k((3))
St
M
Σ exp( (1= (a) b1!b)))) 1=2
+ max [V(2]
M
exp 11 (8) 1)
Σ exp( b/
+
j=2
7
3N
|| 12 (v(..
C
max |
C
NM
4
exp((94) (8)
b(m)))
exp( (0)
.x4 ,((1)))
n |
exp( (g (8) 1)))))²
where the first inequality is by - (1 < 1 and softmar(() (g (3) (()))) S 1. For the second inequality,
Σ exp((a!" (1 ((b)))
exp(ia) 2 ..67)))
we first consider
exp(la))) (()))))"<())))> then by the monotonicity
exp(id)" a( 1)+
E
M
exp(ta (())))
1-2
(2) (21))) S C for s E [Ti, f). The last of (g) Al()) and (q (T)) k = (1) we have exp((gl
inequality is by v(0), v(()) have
n,s = o(1) for s E [Ti, t] and absorbing the constant factors. Similarly, we
1
(5) W
cxp(19 (x) b1))))
exp( (5) (k(")))
8 < n/C' |
exp((g (5))))
a nnt NM 1111
(5) "1 ((())
exp((g()) (e) k(()))
exp(1g_") (5) k x(11)) + Σ
expl(1 (2) ..b9))1)
2=2
exp((g(-) (5) ((())))))
M
exp((g!") x(اi)) + Σ exp (2) ((
expl (x)
(91 n.
+
exp((gl") ا(اي) + Σ exp( ((3))))
M
y==
119

---

explie _0) (1
)9)
exp((1f" x4)))
Σ eepp(la (
y==2
exp((1! (
n. kWd
M
exp((gf" (0) x(41) +
Σ exp((14)) (6) 1/
exp((g (0) k(ikk))
exp((of)) ())))) Σ exp((a!)) (6) ((())))))
y'=2
explig!" k
2 NM
(()))
M
exp((g(") x(()) +
Σ expl(a!)) ( 1) /==2
explie+" (1) b(1)))
(-v())
M
exp(ig (0) b( + Σ exp((a (0) b/22/11) (.
y=2
expl ((!)))))
- -Iv((
exp((g[") b(o))) + Σ exp( (9 (0) x11))1))
M
f'=2
Σ kØj
exp( (9 ( ((())))
- -max [V(.)]
M
exp((gt") A(:1)) +
Σ exp((14))
(5) (.())))))
j²=2
exp((1!)) (5) (!!))))
2 NM 11
2n
exp((gf) x(m)) Σ expl(at" k.
M
8==2
(1 exp((al) (!
2m
|1
NM
n/C C mll
N exp((at)) (a) ((())))
where the second incquality is by v10) 2 3M |V 1. ((() < 1 and the property that attention < 1.
exp((a) (z) x (())))
explta (2) x (( y
For the third inequality, we consider
exp((t ( ,6' first, then
explla!") ")p((a"",,(()))))+ E explla (0 y
by the monotonicity of (g) {())) and ( (T)) ((T))) we have exp( (q (0) k 2 C for s E t].
The last inequality is by v(() o(1 for E [Ti, f and absorbing the constant factors. Similarly, we have
n 2 n
N exp((g (b()))).
(60)
13 ((a)
2.t. NM
expl (5) ((1)))
(V
exp((g (2) k((2))) + Σ
M
exp((g (e)
ح=
120

---

exp((g {((111)
19
exp((a!.)) (8) ((11)) + Σ exp( (8)
M
exp( 9n.1
(V(()
M
expl (1) { + Σ
exp( (gn (b()))
2=2
exp( (qp k
exp((an.) (6)
((4)))² + exp
(g (5) (((1))
- (())
expl(g k(1)))
NM
expl(gn (21)) + Σ
(2) (111))
1=2
exp( 1o(
(1
M
explig (1) 1) + Σ
exp (8) k(11))
M
exp( k(nk))
(vl
expl(g ((21)) + Σ expl(ln (5)
M
j=2
exp((gn.) (5) (())))
(((s)
NM
M
(2) exp((g + Σ exp((g!!!.. 1/()))
1=2
exp( (2) b((())
exp((g (5)
{())))) + Σ
(x) ((()))
M
Σ expl ( k1(1)1)))
|V/1) ((
k=2
exp((g!.)
(5)
((2))) +
Σ
M
exp( (ep(( ((1)))))
y=2
< ~1
(())
NM
a5
exp((g (()))
(5)
(V
M
exp((gl.)) (6) (()e)))
+ Σ exp(ig (0) (())
M
Σ expl((a
(8) (!()))))
+ |V((]
exp((g!") (5)
x(0))) + Σ exp (opp((a)))
(5) ((()))
2=2
_ p(s) (v(
C
+ ||]
C
NM
exp((g!.)
(8)
exp( (
(b) x ,00)))
nColl 1
N Wxxp(gg..).
{(( ((0)))²''
121

---

where the second inequality is by soflmar( b(())) S 1. For the second inequality, we exp((al
expi( ((3)))
first consider
expl(al.) (in)) then by the monotonicity of expl ( 11+ 2 expllain. (2
M
(n ( and (In, 1 ( - o(1) we have
> explian (.) S C for 8 E [Ti, The last inequality is by v(o), n, = o(1) for 8 E [Ti,4] and absorbing the constant factors. Similarly, we
(T))
have
nalll 8 p I N exp((g ( k!')))
(0) "16mmall -((x)
NM
exp( (1 (x) ((21))
(V
exp((gn.) (3) b(a))) + Σ expl(g (8) (())))
exp(ign1. k((1))
)3)
1)+ exp((g x(()))+ +
Σ expll (5)
M
(q. (!!)))
1=2
exp(
(x. (6) ((1)))
(v(() n,
exp( ((o + Σ
M
exp( ((!)))))
5=2
expl (5) k((l)))
M
expl b(211) +
Σ
expl(g
j=2
+ flower order Lerm)
exp((a!:.) (e) ((()))
g(sa)
NM
11
expl(gn.) (0) (()))) + Σ
exp( (g ((o)))
j=2
exp( (1 s (()))))
(v1'
exp((g (x) ((0))) )) + Σ exp((g!?)) (5) (((i)))
M
exp( (5)
(1)1))
(v(()
2=2
exp((gnn) (()))
Σ
exp (qn ((()))
2=2
+ (lower order term)
7
NM
exp( x(())))
(1
exp((g!.) (5) {((2)) + Σ
M5
expl( (z)
k((l))
j=2
122

---

M
exp( (0)
(v(()
exp(
(
expl(gn, b::)))
+ flower order term}
M
Σ exp( (q 13,1)))
11 S
(v
3=2
NM
expl
(
2 exp((g!..) (2))))
f=2
(11b)))
exp
+ ||(0|
expl k (421))
exp((g (5) 121)))
+ flower order term)
(v(*).
+ ||100]
NM
exp((gn,))
(x) exp((an,..k4.) (a...4.)) k(2)))
+ flower order term)
n/Crofd
N Nexp((g k())))
where most of these processes are similar to the other equations above, and the last inequality we absorb the constant factors and the {lower order term]. Similarly, we have
(x) < 0 n,
N Nexp(ig!.) (1))))
n/Cuopd
1/1) 2
exp((1)) (2) (.1
N
r/Coo
((0)
exp((g (()))))
2
n,,,
N
n/ogo?d
(6)
exp((g.") (())),
N
Pnj.n
exp(
(8)
(.)))
N
Similar to we apply the bounds of a and / above to give the upper bounds for the dynamics (q, k).
123

---

10 ( x - 1 (())
Σ
an (K k2)
S+ 2=2
nES,
i=2
M
(8)
RES
im2
(243)
M
T
((*)
(x)
In.s
BESI 1=2
1| 1
l1a
1 Llower order term}
21/0bl1a11
@11 + flower order term)
exp((at") k(
@Collle 11
expl(gt) (s) 1("))))
similarly, we have
r/iolllll100f
(q (5+1)) 1(0+1) - (g"h"K'9 s
(244)
exp((9 (0) ())
(q (o+1) 1l1t (اوh) (5) kl)
M
= a (b.4)) ((2)))
(5)
nt' S 1=2
M
11
1B +
(g
an (9 ( go)))
M
+ a
+
n' n'ES 1=2
(B(0) T
g(t)
N
M
g(0)
(s) )
(245)
n.j. U+
+
Pn..
2
و|| (0) 11 + flower order term]
moll
expl(g!
(4) (.)) e (opohdenn)
N
naoff
exp((a
(4) (1)) [111
N
+ flower order term]
(C exp((a(") (6) k(3))))
N
similarly, we have
(q (++1) knj
(6011)) (x) (((())) >
exp((a [.))))))
(246)
N
124

---

(9 (0+1) k - (/) {
= (4) = 11c
(8)
(
a
1=2
+ + (0f
(6)
(o.it +
8(0)
n
N N'ES 1=2
N
+ a n.
(4)
k.
(4)
+
Σ
n.in k R'.)
==11=2
M
(
3(0)
(x)
+ 1
(247)
n' 2m2
anit
(8)
((()
-
{lower order term)
Coo
+ (1 (o
N Vexp((dn.. ((01)))
N (app(a k(11))
+ flower order term]
(CloodillulBBBRRdd
N expp(1g (e) x(,)))
similarly, we have
(s+1) (otti,,((tt)) (0tt1) - (cli)aaa))) S n0ooodddlll1 11200 (glett)
(248)
N Vexp((g (g (z) |ااا)))
(g knj
(0+1) (0+1)) (q (e)
N
=a (k4 b!:))) + a p,
N
M
+
9(6) Pnnit (4+ (n.)) + (/55)
(q
In.
g(s)
Fnginn a Aan' Io.i)
n'= 1=2
M
(e) (0)
k(0) + t
(8) ((0)
(6)) n',
N
M
(249)
at (
(T)T
+ n.j. 9
g(s)
(T)T +
Σ
g(s)
An'
(
(s) =0( - [| 112 + (l..m.ill!! (e)12
+ flower order term]
2nCo
N exp (8) ((.))) 0 (0
+ flower order term]
Cr of
exp((g (.))))
125

---

F.7 Bounds for the Sum of and B
The gradients of the inner products of q and ls contain lot of coefficients and 8, and in order conveniently give the upper bounds some lower order inner products, we will give upper bounds
for the summation of and 3 (e.g. Σ
Note that in the Jacobi matrix of the Softmax function, the elements on the diagonal are so somtmar(aa) ( softmar((;)) and the elements on the off-diagonal are so (oftmaa()a) softtoofm(a( (onnazzt)ssofttma((( In Stage
II, the attentions on signals flt increase and the attentions on noises & decrease, then we can consider the following cases
if a; (q+ k-) a; k4), softmaz(a:) has constant upper bound 1, (1 softmar(a:)) decreases softmaz((1, (a:) increases. So the upper bound of so softmax((.) (1 softmar(a;) decreases softmax(a,) increases.
af = (g+,k;) or a; = (q., kz). (1 (of has a constant upper bound So the upper bound of sof (maa((a)) (1- s0ftmar((ss) decreases as sof (oftmar(a)) decreases.
if = (q- kj) or = (qs, kj), softmar((4 has a constant upper bound 1. So the upper bound of so softmar(a)) so softmar(a,) decreases as so softmaz(a,) decreases.
Based on the above cases, we first study the bounds of the following terms
softmar( (g (9) (())))
softmar (gn..) )
(5)
softmar((q))) k())))
softmar (qn,s (())))
Note that 1 - softmar (c) k(()))) = softmar((g"" [.l)))) and 1 softmar( (e) k((2)))
Σ soft softmar (qn,er ((())) we only need to give the upper bounds for so softmor (q (6) [(())) and
(8)
}
soʃ coftmax( ( (+))))
Assume that the propositions B(Ti), .... B(s). D(Ti), D(s 1) hold (s E [T),t), have
|V | 044 CC 11 Ti),
(250)
12 C wo wol3dh ((-TT)(( (T((-----))) (251)
A n 2 log (exp(A(T)))) (Ti)
N (1og(66N2M2/88)
1 wo (s (s- (s)ss)(s Ti (0-11) -1)). (252)
(Ti) A (8) An. 2 log ( (xp((A +
N(1og(6N2113/10)
for E [M]][ nE [N [T.t].
126

---

Then we have
exp(<g (5) k1/3)))
M
expl(ef") 1) + Σ exp(((ef)) (6) (k
j'=2
exp((g (8) k((7,))
Ceep((a= ( k(")))
Cexp(Andd))
(253)
Coxp(An,t)) (T) + PGCI111)) - (s) - Ti 1)
n( (10R(0N=M3/11)
C13 + Cralll Allwol13d (s) Ti
N (1og(6N=M1118)
For the first inequality, by (T), (T) l(()) o(1) and the monotonicity of (a) k())) ((g x()) is increasing and (g(h) (4)
k(()) is decrcasing), there exist constant C such that C Cexp((g)))) (4) (1))))
M
exp((gl) {()) + Σ exp((g )) The second inequality is by plugging 251). For the
f=22
last inequality, by A A C13 < CsC. Similarly, we have
o(1), there exist constant Cia such that C13 I C' C'exp(A (TT) and
explig..)) (5) k(())))
expl(g!i)) x(())) E exp((g (6) (
(())))))
(254)
(6) (5)
Cia + "CCis3ddMlal8 woll (s Ti)(s Ti 1)
n (10g(0)333/10)
Plugging 250 253
and (254) into the expressions of a, 3 we have
127

---

NM Σ
pt( 11
10+
nES+
exp((q (0) (((1)))
(v!")
exp((et)) (0) (( + Σ exp((1 b(11))
M
(5)
y=2
exp((1 (0) 1((1)))
)²)
exp((g1) x2 exp( (q (6) ((1)))
M
expl(at") (1) k((1)))
(v(()
exp((g b()))) + Σ exp((a (0) k!1)))
2=2
5=2
expl(at) (8) (()))
exp((g (4) ((")) + Σ exp((gl)) x(1)))
M
3-2
(255)
7lll1/ 3N
(o(4, b) + Call
Ti)
NM 4
1
Cia + [Csssll[[[lwwo (s (s- Ti)(s- -Ti -1) 1)
N M22 /19)
1l| 1111112 1
=0(
C1 + P'rssllu N (18g(687*M188))
- -T)((s - Ti 1)
12 wo lla(s Ti)
+0
C13 + 1
(s Ti) Ti - 1)
(1oR(0N223/111)
-0(1l1(1B0))) + o
1 1ل111|1111 woll Ti)
C13 + P I ا|
(s- Ti) Ti 11 1)
N (om(6N=M2//0)
128

---

where the third equality is by 2 Ci N (1g(6N>>22/))
(8 (8- (s)s -T - 1) 2 0 for a E [7i, Next,
1 1311 1
give upper bound for
follows:
C1s+ " ا 22
(3-
w woll((s -T)
Arslllu Hll
C13
(s Ti)(s 1)
N (19((6N2M//68))
1 w0
Cu + CCaallal 3d 2 (s Ti) N 1og(6N22
C E ue 1122
(a-Ti) N (19g(6N3M3/80)
po [alll wo 11343 2
(256)
N (10g(6N2M3//6)
N log(6N+A13/6))
x/² 14
woll
2n 11 uro laad
(og(6N3M3//8) N2
N (og(6N2432/66
|| wo 11
θ 1ml la4,
N log(6.N2M3788)
0 1 1og(6N222/8))
where the inequality is by ax b 2/bb for x > O. the third equality is by absorbing the lower
order term [ wlll 2, the last equality is by wollz = (1). Plugging this into 255 and
get (1og(6N=M47/6)
= O 1 #) + (1lllllB (g((622M//0))
(257)
0 millul 1 N (0g(62222/0)
Similarly, we have
la 11144 1,3((0) (111111111 1og(6)2222))))
(258)
1 1 la (22,, (111u11/N log(6N
(259)
1860) 104.b1184 10 0 m1lle18 log(6N22 /01) = 0 0111113 SNR. N &- 2 M2)8))
ond't vtdd
(260)
fori E [M]{(1),nES)
(5) (5) 1 o 0011ll po 120 log(6N2M2/8) = =0(1llu1d, 1og(6N222//)) (261)
for E [M]}(1), E Sz, the last equality is by N SNR2 2 2111)
11 = 1ogd (6N2
0 (1g/:1/811,211100000.1111111 (262)
2id
129

---

for [M] E S2, the last equality is by N SNR? 2 2(1).
Jandngbl (implbl91...... mng.n.ll - o 1og(6N2M*/0) / O (1illl112dh 1og(6N2
(4)
N2d
(263)
i E [M]]{]] n E [N], the last equality is by N SNNR? 2 2(1).
lnn,nn' 13 m.s.nn = o ( (6. M2 15) log(4N2 (2/8)) /0)) = -0(illal|Bd dd " (Iog(6N2 M² 11)))
Nid
(264)
for i,j E MM E [N], + n', the last equality is by N SNR? 2 2(1). Taking
summation we obtain that
0 17 11 wo 3 log(6N2213/6)
(mllllliNN 1og(6N22M118)
(265)
= =N
where the last equality is by ||wol| (l)) Similarly, we have
Σ
+
|86 1.
1 + 1. Σ
1 ON 1
(266)
7
T
for E [M]][1), n E S..
(3)
-. (N-
(a)
(267)
for e [M
Σ
[3(00) / - O(SNR--N' SONR--Ntd,. &x
(268)
==
for E [M].'),neSL S1.
(5)
(8(0) m = ((1xb)
a aninll
(269)
for E [M]]{1], n e St.
lonane sl
Σ
[8() p
=0
(270)
f, E [M]M(1], n, n' E (N,n
With these sums of a and 3 above, we can easily prove Claim 3 and Claim 4
F8 Proof of Claim
In this subsection, we assume that E(Ti), E() hold, and then proof that C(t + 1) is true with the result E7
(t+ 1
112
1 (s+ 1) 112 11 (+)
1
130

---

M
2a (of - k
+ Σ
Σ
(9)
5
==
nES_
2=2
M Σ
+ Σ
+
R.t. n.))
RES,
i=2
M
(TT)
n,
nES,
I 2
Σ s=
a
Σ
Σ
a
(5)
(2)
EES
7
+ flower order term}
=NN O(log(dR)) +N. N. M (N O(log(dh))
= + log(d;))
where the first inequality is by triangle inequality, the second inequality is by 199 Since of 2
(max (ozd, (lxl())) -1 d (log(6N2 and da 5 max 4 SNR 1),2 -2)
2). we
have N V.d : log(d)) 01I121100 so Similarly, we have
11 (++1) 1) || (Ti 1| = 0(1l (107)))))
11a - 11
0 - l log(d))) o(1
11
(1 d log(d))) - a( (3)
|1
"12 - 1 = 0 $ log(d)) = ol
||
|| (.) 12 o Nb th log(d))) o(07o)ddn)
so we have
11
1|
e 011 1 ( ||2 (o)or)ddn)
|| 1
|| (
for E [M]{(1), e [N].
(9
(1-1) .9 _ ((+1)/]]
S 1 (T)
_ ( (1))) +
(q .
(1-1) (0+1)) - (g(".(()) .g(h))
==
< 1(9 (Ti) q rr)y
M
(5) k()))
a
nES
1=2
M
nES_ i=2
M
RES, 2
M
T
(T)
+
RES_
1=2
131

---

144 (2)) |
10 (1) .ll -.12)
(0)
nESt ==
M
1( )
Σ Σ
Σ
0n -
(4)
(0) |
==
nES_ 1=2 ==
(lower order term]
S (a . J
o (Nttd )) o(1)+(+)) N.N O(N (otxt)))) )) log(d))
1ga ( + O dá log(
= =1),
where the first incquality is triangle incquality, the second inequality is by (198), the last equality is by dn- = ② ( SSRR 2)N22=
19 1+ (uiti)))) S ((
(T))
(Ti) 1 +
Σ 5 7j
( (0+1)
clatt1)) - (9 .
-nnt
< ((9 Anni
(Ti) (Ti))]
M Σ
+ Σ (+)
+.+
n'ES_ 1=2
N
n.it (g (0)) + p
(5)
(g
Riin (g (6) (
(4) + M
a
RES+ i=2
M Σ
N
(n.s k4
(T)
+ (e)
(T) +
nii i k (T)T)
an,,
1=2
1(9
(Ti) (Tily)
M
+
nn (1
(s) (5) x(0) +
Σ
la
(0)
(5)
1=2 s=
E'ES An' o I= s
111))++ +
1a 11(g
(5) (
+
(t)
(0)
2=
M
(2) (.
fn 1=2 =
+ flower order term)
< 1(
(Ti)
Inni AoT
M o N2 (r log(dd)) + M. +M. (N log(ak)
N. M- 0 (N- +) o(1) (15+t) log(at)
+ MM O .lg(())) N. M. 0 b log(6N2///1)) log(d?)
132

---

= (T)) -1(),
o(Na ) + + + o OoNd-adj 2d, 1og(6N2M///)logg)))
where the first inequality is triangle inequality, the second inequality is by (202). the last equal-
ity is by da = 2 max(SNR* SNR=SNN2 *) and d 2 222d Similarly, we have (g (+1), an, ( o(1).
Kgni qnn. )
(1+1)
= (q 9nn.j +
(2)) (3))
Σ
(q
(x+1)
Inn.
(ott))) - (g (1))))
7=
(T))
N
(8) (8) (()) + a
(8)
(0)
(10)
Σ
a An.j'
(9n.j"
N
M
(1)) +
(5)
nn of In.e (1))))
(8)
+ Bngt An.
+ong. (An,e:
+
'2' n'=1 1=2
h24 + +0
(c) 1/6))
(0)
(())
Rii
N
M
(ngtth kt (G) + (x)
(+)T
(O)T)
+ap
Kg : In.j
(Ti) (2)))1
M
(o) (0)
(010)\1 +
la a
(e)
(0)
(5)
la (e)
anin | Kann k(tll
1=2 s=
M
nin' 1
(5)
1/
la nad
+
la (x)
n.j. 11: A ا(0م
n' fn 1=2 T=
s=
s=
M
M
mn (0) (111))
n.j.n , .ll b(()))1
+
1=2 ==
W n' 1=
+ flower order term]
< Ka
(T))
(3))))
+ 0 (dpxt) -lo((d)) + M ((T+t) log(aE)
+N. M O d & d log(6N2 M² 16) o(1)
(
(3)) ) 0 (dh² t log(d)) +
+o(Nd (Nd (Na dd Log(6N>M2/0)
o(1)
for j E [M]M[ # j,n E [N]. The first incquality is triangle inequality, the second inequality is
by (204) the last equality is by dn ă max{(NN* ,NN AI.N2 2) and .t c 2 22222 vd,d)
light In
(0+1)y) S (( I..j
(T))
Σ
(a Ini
(0+1) (at1)) - (1 Gil
==
< <19g (o.
(Ti) (Ti)g]
N
(s) (5)
(())))
+ (5)
(3)
(1))
(s) nann (9.. (,0))
an (gn.)
a
+
133

---

N
M
+ miit (dnn 1of2)) + ans.
(6)
(a k (3))
aainn a An k/5))
+
M
( 44 + +0
(2) (8)
(x)
(3)
n/=1 l=2
M
(6)T (0)
annjtth k + +an.
(Ty) (1)i)1l
1g I7..
M
n,i,+ Tm. 1
|| +
la (6)
n,i, 17..
(0) (1) +
1o n.i,, mnin Ill (+) A
11
T= T
M
aa (5)
n.amm 11
(6)
Σ
(5) |1ll [T k//,>>1
n.i.'
==
a'znAn' f
M
(5)
k+
x401++
||
la (5)
(5)
(s) m
(5) k!!))
2=2 T
M
M
(s)
(
E
2=
Wnnnn' fn 12 =
flower order term}
= 1a
(TT) In.s"
+ + -logdd)) + M- M- Ol 2d 1og(6N2MM10)) (0M log(dh)
+ M. O (Th)) (01 + N. M 0a 2 log(6N221/2/8)) /8)) o(1)
= =1(mb)) (T)) (T)) )1 + 0 thh log(at)) + o Nd (od 10g(6)2M/216))
(drx
=0(1)
for i,j E [M E [N + n. The first inequality is triangle inequality, the second
inequality is by 205 the last equality is by dn = Φ0 max{(SR* =NNR-JJ.VZE and d σ (E NN=
((((et1), k 11++1)1)1 S ((((
(T))
&
xTuyy]++ +
( (()+1)) (x6(") |(00g>
$ (K(TT) ke (Til)))
M
/ (a
((0)
(q ())
+
S
1=
M
+ 8(0)
Σ
g(5)
(0))
DES_
M
+ (oL)
NES, 2=2
M
T
RES_ 1=2
(1( (T)) ((
134

---

M
x + Σ
(3(1)
n Kg N: |(10)
+
SES+ 1=2 s=
M
|3(+)
11(0 (3) 1+
Σ
Σ
Ka k42)
(+)
+
n,
==
nES_ 2=2 ==2
+ flower order term)
= 1(kf 1
+ +N(2d +) log((d)))) + N M O . -N -r log(d;)
1444 R ( + 0 (SNR. Nàd log(d)))
= =1))
where the first inequality is triangle inequality, the second inequality by (214). the last equality is by dn = = max(SNR* ,NN ")NN2=
(K4 (1(+)) (( S (( (
(14 (
< [k (2)) w
M
Σ
Σ
n' + Io k(ll))
n'ES+ l=2
N
M
k441 + Pn,, {a_)) .k4
n'=1 i=2
M
(10)
3(0)
(5) In'
2
n'ES+
M
(x) g(0)
T
ma y Inn.
<
M +
Σ
Σ
13
R.+ Ill (2) ((?1)
1=
M
(8(0) n'
(
(a(e) {
(0) x(7)
n'E An'
(8(0
(1. R {اا)
N
M
[3(0)
n. fll. . ا600
in 1=2 T=
flower order term]
= =KK4TT ((T))))
+ =O(toott log(d)) M - O(SNR. -N log(a))
N. M O(NNR. N x-bottt o(1) + (N log(d))
135

---

+ M. ( log(d)) + N. M of dd (0g())222/1))) log(df))
= =Kuoft ( 1 + of (SNR. Nid +) + (Nttd, log(d)))
+ +O(Nd 3 d 1og(6N2MP/0)) log(dh))
where the first incquality is triangle inequality, the second inequality is by (29). the last equal- ity is by dn = ( max(SNR*, SNR ",N,"S"N'' (RR==== and d 22(c=NN2d, Similarly, we have (a ) o(1).
(1 1 ((())))s S (T)) +
Σ s=
(B k (2 11/23))
n
N
M
+ 3(0) (c) (111 + 3 (9 "g_", ( +
m (
Σ
3(4)
nan 2
N
M
+ 2100 (44) [b!i))) + -10)-11 (1-
3(5)
ma a (1 (())
1=2
N
M
+ (2h)) ( + 116)) q -
(0)
(0)
m 1 Inr.)
n 2
N
(o + n..j. d
(J)
-
+3(+)
(+)
(T)T
nignn' 1 In'
< ( (())))))
M
(8(1) P 111)++ +
1n,,, 11kg_ (1) |.(0))
[3(8) main MaannIlKgg (3)
+
Ti
1=2 ==
N M
Rin a
(1)
(t)) +
(3))) + 11
n.j 1 [1( (3) (((
s=
e=
N
M
(3(b) il R. (())
Σ
Σ
Σ
(
(5)
a'zn
+ flower order term)
(e!Th) (Ti)))
+ (N 1) log(d))) + M (chh)) log(d))
+ N 0 ( d log(GN2 2² /0) o(1)
=K (T) + 0 (N 3 log(d)) + of (Nd dd Log(6N2M//1))
= =o(1)
for i,j E [M]M)), i ij, E [N]. The first inequality is triangle inequality, the second inequality is
by 21I). the last equality is by dn = 52 max((SR' ,NN SANR-11.. 'IN22 2) and ii(==2N2d, dn).
(++1) | < (
< +
Σ
21ot - (k x(21))
k
s T
< 16
(T)) ((3)))
136

---

N
M
(a ( +
B (a
g(s)
+
==
(
(6) k1 31 + (a
night An' k
(3)
N
M
+ (0 ( 9- + M,
n'=1
N
M
( + +
(O)T
(+)
(a)T)
In'
n'=1
$ (k (Ti) (
M
18(8)
106 p 11 (5)
18(()) n.
(5)
M
M
(8(0)
n.j.m. 1q x + Σ
(g(0)
(0)
p,in 1
1=2 s=
1 n'Znn'' T 1=2
M 2
1865) (6)
n.j. 11
| +
(33ba) n.j 11
M
M Σ
(3(0) n.mm
11 /
|g(e)
Ragin ,1l (g x())
==
n Au 1=2 1=2 T=
+ flower order term}
= -Ia ((fii>))
+ +N(Nd log((d)) + M. od dd log(6N2 log(d))
+ M O0-00(1h' (ch o(1) N M 0 od 2d log(6N> o(1)
= =I( ( (
+ + log(dt) + of Nd (d id log(6N2
=
for i,j E [M]i{)),,, E [N].n * n. The first inequality is triangle inequality, the second inequality is by (212). the last equality is by dn = ō max(SNR*, SNR "NNR=")N' N222)=)) and = (22222 N22.).
F.9 Upper Bounds of ( q, k)
In order to give the upper bounds for (g. k) stage III, we need to give the upper bounds of a and / based on the equations in F.I. The main difference between this subsection and is that the bounds
of [V+ Vo is log (O(5) in this subsection, while the bounds of [V+|, [V,, is log (O(2) in F.6
resulting in different bounds for and B. Now we take a (*) as an example
Σ R
(1)) |
NM
exp(1g (0) ((اا)
(v(
expl(g(.) ((1))) - Σ exp(( (8)
M
(4+ ((0)))
137

---

exp((1l)) (0) (x()))))))) (221))
)2)
15
exp((g4) ())))) + Σ exp( (g+ )(2)))
M
expl(at") (0) 1((1))
n,
exp((a4) (0) x(1))) + Σ expl(a (0) k!1)))
5=2
exp((af) (-) ((bl))
exp((94 (.) , ,k("))) + Σ exp(1a (5) ((())))
M
exp((1+ (8) , _k(41) M
7 NM
Σ
e( 1al2
exp((gt) x( + 2 expl (5) ,.
nES+
y=2
M
Σ expl (8) (b()))))
(V
2
M
exp((14 (8) 11421)) + Σ exp((el)) (o)
M
exp((14) (0) k11))))
v(6)
M
exp((of") x(4))) + Σ exp(((g
(q (5)
2
j=2
Σ exp((1 (a) b(()))) 1=2
7 Σ
|mll2 (v
NM
M
exp(<g (0) { +
Σ exp((at)) (:) (((1))))
nES+
j=2
M
Σ exp( (pp((q (a) [(!:)))))
+ max [V
2=2
M
exp( (gg (8)
11
Σ exp( (
+
1=2
7
3N
| 12 (v(.)
C
+ max [V(0]]
C exp( ,(,))))))
NM
exp((2f)
(6)
((0l))
(0)
n 14 log (0(2)
exp((g)) (0) ((())))))
where the first inequality is by- - ((()) < 1 and softmar
(q (. ((())) < 1. For the second inequality,
Σ expl1o (4) (())))
Σ exp((a (0) (())))
we first consider
explig x!") then by the monotonicity
exp((g (()))) 1)+ E Exp(ta??) (())
(5) (())) and (a+ of (g
(Ti)
(9) (0)) C for e [T,Ta]] The k (71)) o1) we have Σ exp((g)))
last inequality by v/(0) ||
s 2 log (O(+)) for E [T2, Ta] and absorbing the constant factors. Similar to F.6 we can give the bounds for the other a and as follows:
C (0l1m112 log (O(2))
(0)
exp((g (1)))
n/Collul13 log (0(
8(()
+.+ <
exp((g (5) k(a)))
138

---

8 r/Collulléé log (0(2))
exp((g (a) (x))))
70lll log (0
exp((1 (1)))))
nnt
N
Collllll log (0(2))
(6)
exp((g()) (3) ;.!"))))
N
Collull log (0
g(0)
N eepp( (q k(
Colllell log (O( N eepp( (!. k (11)))
nooofd (O())
(4)
and
N eexp((9..) (5) k ((,1)))
nCoofd (O(t))
(s)
O < N Nexp((a!.) (5) k (())))
nCgoddoog (0(2))
g(0 2
exp((g!.) ((()))
N
Coooddoog (0(2))
exp((g_)) ((!!))
2
N
nooofd (0()))
exp((a!:) ((2)))
ajinj 2
N
nCoood log (O(!)
8(0) pn
exp((g 123)).
N
Similar to we apply the bounds of a and / above to give the upper bounds for the dynamics (q, k).
139

---

a ( 1 - ( )
M = = 14 122 + Σ Σ
R 1=2
M
+ +
Bla)
R
nES t=2
M
(8) n,
+
2= nES,
(271)
M
((0) n.+. In.
nES,
114 12 + $4.1 11
(5) 1 + {lower order term}
2nColl 2l log (O(+)) O [1 (o) + flower order term]
expl(et") (0) 1
11Collull A log (0(H)
exp((at) (4) 11
similarly, we have
(9 (())) < @Colllul3o log (O())
(q (0t))) k ((0t1))
(272)
exp((g_)) (5) 1111))
(g (s+1) ( - (4 (1
a
(x) (1))
n
1/866)
a.ttt 11
(8)
8/6) (a+
an 1(4 (9)
M
Σ
+ (at+
W'ES+ 1=2
(B(0) n.j. 9+
( 3(0)
M g(x)
(2)T
(273)
aggn I
n.j.
| 12 + 3/09
11 (1) 112 + flower order term]
Collil11 log (o
exp(1q (+) ((b))))) e (o
N noooddoog (O
exp((g_)) (1l))) θ (1111B20dd))
N + llower order term}
( log (O(E)) exp((g!") (.("))
N
similarly, we have
log (O(?)
(-t1)) x (o(",(((.) 2
exp((g(") [!!))) (274)
140

---

(an. (1t))) x
( )
N
M
=a - |1 + (+)
a (K (0)
)
(
1=2
+ (g In.i)
An.i 9.l
n
n'ES+
N Σ
(5)
ki
(3)
(+)
(6) R.dnd n1,))
n,,
n,
M
(p T
8(3)
(T)
(275)
E 1=2
=
(0) |
11 ((.]
qn / flower order term]
no (O(.))
(1111277d))
N Wexp((g (())))
Callull log (O(2)
2 (dd
N (exp((g.).. 1(
+ flower order term]
nC1003d 12 log (O()))
N eexp((g))) (5) K ))
similarly, we have
(3) ( 01000dd1m1113 log (O())
(276)
Nexp((g..) (3) (())))
(0+1) (0+11) -G (e))
(gnji
a (k4 (11))) +
(5)
N
M
+
g(5)
Pniit (4+
In.i + g/6)
nij.n (
In In.i)
w==1
M
+ (0 + a
(5) (2)
(6)
(6)
(1) R.k n',1
R'=1 1=2
N
M
(277)
((())
T
g(0)
(T)T
((s)
(x)
( q + n.j,
=o(() mning n.j
(s)
g(0) P
(lower order term}
2n (O(t))
exp((g ((1))) - @
N + flower order term]
Croo of log (O()))
exp((g k!!)))
N
141

---

E.10 Bounds for the Sum of a and B
Assume that the propositions
F()), H(T)),............ ..,,, H(ss - 1) hold (s [Ti, (), we have
|V s log (0(
(278)
| 0(1),
(279)
at.
A 2 A 2 log (spp(A/..)) + +
(280)
N (1og(6NN2 M² MM/52// 18))
oz
( R. 2
n. 2 log (x (Tt)
(Tp)
| (oog(6N2 ))
(281)
for E [M] (1), n E [N]. t.. Similar to 64 and 65
we have
exp((g]) (0) k1))))
N M02g(6222/000))))
M
exp((g!"), () + Σ exp((e
(5) k..3)))
d
(282)
j'm2
exp((g k(1)))
0 N Ml 11111B(1000) (log(6NN
exp( (8)
(1))) +
Σ
M
(xxp((gn.) (5) exp( (g ((g)))
07
(283)
2'm2
Plugging 282 (83) into the expressions of a, B and letting O(
(log log (0(t)) be the upper bound
for V (a) [ls)
we have
Σ
((()) 1111
NM
n
RES+
exp((g (5) (21))
(v(*)
exp (x) x121) Σ
eep(((al) (x) ((2)))
5=2
expl(q x((1)))
j2)
exp( (0) x((1)) + Σ exp( (3) {(())))
j=2
expl(g ((1)))
(v())
(284)
M
exp((14)) (0) ((1)))) Σ exp((al)) (8) ((()))
exp(1g (0) ((ا111))
exp((g()) (4) x(())) + Σ exp((g()) (5) (((l))
all 3N
0 (0(+)) o N (0o0()NN222/000)
NM
4
d
N || (oggall(((((6322M/// log (0(
d
142

---

Similarly, we have
1a 1. 1..+t1113 y (
( (lo((N /0)) log )).
(285)
d
log(6No 1 ( A²2 /6))) log (O(t)))
10 (0) 0
(286)
dz
[og 22 /8) ( log (0
18 b 18 0
0z
(287)
- SNR? (og(6222M log (
- (
for E [M]][]], S=.
la p |a =0 0 (0 (ogg 18)) 3 log (0(+))
(s)
(288)
f [M]][]], E Sz.
10 11 o mofdl
( log log((N log (0(
dz
(289)
n/ Mull 2 / log (0(t))
d
for E [M]1(1),n E Sz. the last equality is by N SNR (1))
jnimm 133 133k)mmm - 1
log
log (O(t))
(290)
dz
for E [M]][1}, E [N].
[log(6N2 2 M2 (1))/)))))))))))) (1gg6]] log (O(E))
(291)
dàd
for i,j E [M]][1], n,n' n' E [N], + n'. Taking summation we obtain that
1 (1ggNN log (O
1
nel 1 | woll
d3
(292)
N 2 M 15)) log (0(2))
ed
where the last equality is by ||woll = 0(1). Similarly, we have
SNR? (NRR"(og(66N2)) M² / /8))*
log (0(t))
(8(()) - = 0
(293)
ed
==72
for E nM).
Σ
gg())
J, Σ
1
((())
| n.-ai
[g($)
| n,
D
T=
==1z
==7²
T3
s= 73
N (log /8)) log (O(
ed,
(294)
143

---

for i E [M] nE E St.
Σ
abb
1 n,
0 Cinnn
18
==
s=12
==12
=
==13 11
s= 12
(295)
-0( (log(6N2 M /8)) log (O(
i, E [M][]] S+.
( m - ( M2 /0)) 'log (0(
(296)
ed d
T2
for i,j E [M]](1], n, n' E [N #n.
With these sums of a and /3 above, we can easily prove Claim 7 and Claim g
F.11
Proof of Claim 7
In this subsection, we assume that I(T2),..., I() hold, and then proof that g(t + 1) true with the result F..0
1
14
112
11
(1
(++1)
1 Σ
11
1 112
-11
T2
M
(8)
(0)
a k..)))
<
20
(4+
+2
=
nES+
2m
RES
2=2
M
n. (TT
nES, 1=2
M
2 Σ
la k
x +2 Σ Σ Σ
la
s=1²
nES. i=2 ==
(lower order term)
2 (1oog66N3M200)) o N
log (O(t)) log(c -td))
edi +N. M-MMO0 o (log(6N' M 18)) $1 log (O(t)) log(e log(E-1d -d)
ed log (O log(e- (ogle-la))) = -0 No N(oog(6N3M /8)
ed
where the first inequality is by triangle inequality, the second inequality is by 199 the third inequality is by / < Ta. Since of 2 ( (max(ofd, [1xl(2)) dr (log(6N22 M M2/8)) -2 and
dn á max((NNN)) SNR- we have N WloRioN22 M2 /1)) log (o log(s +a
144

---

al
dlluIBoūdd) so ||
1++1
|1 (1)
- ollua| = 011ua180 Similarly, we have
N (oog /0)) ' log (o())) log(c -d,)) ( =ollaal
||0 1 11 - 12
0
edi
(3000662M3//000 15)) ((LgGN)) | ( |/ - -14 1 |2| = =0 (1+
log (o(:)) log( ( = o =b((lulBooddd)
edi
(1og(6N22//00)) /8)) 3 log (O( log(e log(( =
1 1 = =0(
112
edi
N /5) log (o(t)) log(c (bp) = olo30
[ n,
12
12 = 0
ed
so we have
| 12.
= 0(1
| " 112 || 1 = @(ofoRdda)
for i E [MM{1),n [N].
(at
(1-1) q _ 1 S (a (Ta) ( (aay + Σ
(q (1-1) _ (0+1)) - -g( .
s=
T2) (Ta))]
M
a k()
(3)
nES, 1=2
M
(e)
nES_
i=2
M
RES
1=2
M
T
+
RES_ 1=2
(22)
(Tay)]
M
Σ
Σ
1a (. (l
(0) x!!i)
nESt i=2 s=1=
M
11
Σ Σ
Σ
(5)
T2
RES_ ==
[lower order term}
kg TT)
(Tay)
N (log(6.N' M² 18))² log (O(t)) o(1)
o
eat
+ +N M. O (log(6N2M3/0)) /0)) log (O(t)) log(c -1b))
cd
145

---

(Ta) (T) + +0 N N(log(622M//8) log (O( log(E (11)) = -(g .
edi
=1), where the first inequality is triangle inequality, the second inequality is by (998), the last equality is by dn = /2 (SNRR ,NR "NNR=")N& N
19 ((+1) (+)))) S 1(9 (T2)
(22) In.i
| + Σ
(q
(++1)
-9n, 9n..1 - ( In.il
5 T=
< 1(9+ (T2) (Ta))]
M
Σ (5)
Σ
R'ES
1=2
(q
( + a
(q
Σ
Σ
Rid ( (1)))
n.it
M
k+ + Σ
n,t, 1
a+.+
NES+ 2=2
N
M
(
(8)
(+)T
k
n.
n,,,
nin i n!!
t=2
(T2)
(22)))
An.i
M
+
+
n 11 (qn.i:
(e)
(6) (()))| +
la (e)
1=2 == 12
n'ES+ An fn 1=2 s=
mai 11 ((00)| +
la |
(1 (6) x605) +
M
la
nand 1lla (5) (.(0))
n fn 1=2 T=
+ flower order term)
<
1(4
(T2)
In.t
(Taly
N N (log(6N23M3 (8)) 3 log (o(t)) (og(e-!dt) + M. MMO(
(log(6N2M2/5)) log (0())) ed 2
(
log(l log(c-1at)
h
+N 0 (log(6N2 10) log (o(t)) o(1)
edi
3 + M- O (0og(6N2M² 10) cd + N M- 0 Mog(6N222/0))) log (O(t)) log(e =dft)
log (0(+)) ) log(E log(e-1dh)) (d²)
ad
N (log(6N2M2/5)) / 3 log (O( log(e -1db)
=1(g4 _q.. + (
(T2) (Ta)
edi
0 (108(6NN233//1) log (O())) log(e (1ob)
ed' d²
146

---

(1))
where the first inequality is triangle inequality, the second inequality is by (202). the last equal- ity is by dn = 57 max(SNR', (SNNSSSRRRR"R""""" ,SNR ()N22 and d 52(c 222 dr Similarly, we have 1( ((+1) "nn.."))) = o(1).
(aatt)) Inn ( S (g T (Ta)
(0+1)
In.."
In.
1i ynj
(Tp) (Tb))]
N
M
(5)
k
(8)
a 9n.j.
+
(qn.j.
+
T2
M
+ n..
(gn. k4
(5) (2))
and-
(8)
(g +
nij . In.in k..))
M
(oan 144 + +0
nnin' knn!!
n'=1 1=2
M
(3) (T)
+.
(+)
A
(+)T
(T)) n'd
(Tb) (T2))
An
M
(1)
la (6)
n.s.m [an (
a n,it Ang.
+
aa n.i.
s= T2
1=2 s= T2
(e)
l 1 ( !
(e)
la
fn 1=
s=1b
M
M Σ
(5) m ((0)
n.j. Allgg (3) (
2=2 s=
w n'ynn 1= s=
flower order term}
(log (log(6N2M///8) log (O(!)) log(e (d)) 'd))
+ +0(
ed
(log(6N2 M² (8) log (0(2)) log(e 'dbt)
+ M. O
edp
+ N. M. 0 (log(6N² M22 log (O(t)) o(1)
ad
(og((6N2M2/6)) log (0 log(e (1()) ed
= = mn
(1b)
(1b)) O
N ((oog(6N2M288)) 18))* log (0(2)
+0
edt d
=((1)
147

---

for i,j E [ i + j. n E [N]. The first inequality is triangle inequality, the second inequality is
by (204), the last equality is by dn- = ñ max ,NN SNNR=)JN2 1)N2 and d = 52 50(e= N22
Kgn
(tt)) qnj
(
S (g yn.
(T2)
+
(..tt.ahtt)) (q (
==1²
(T2) [Ta))]
19 9
N
M
(8) (3) (0) + - (Tr.j
Onin ( ( (
+
n²=1 1=2
N
M
+ n. (q (()) +
a
(3)
(q
+
Tng a (9 (5)
(9 (21)
N (0)
M
(0)
(8)
+ ( + p.
n'=1
1=2
N
M
(s) (T)T
+ (5)
+a
k
k
(1)TT
n'!
(T2) (Ta))]
M
On [a (1)))
Σ
l mniim,llaa (c) k(!h)!
T=
T2
M
Σ
n.i.n,
1=2 T2
n An माा 1=2
11 (3)
k4 +
l1 (8)
m Kl (9) (())))
T.J:
M
M
(5) (3)
a a (1 A
+
2=2==2
alennnn fm t= s= T2
(lower order term)
(T2) :
(Ta)
+ +0( (log(6N2 M2 /8)) 3 log (0()))) log(e-!at) laat)
edi
+M-0 (log(6N2 10)) 4 log (O(t)) log(e log(E-!d?) -d2))
ed dk +M 0 [log(6N²MM 2 log (0(:)) o(1) + NN- MM- - o (log(6N2MM) 10)) log (O(t)) o11 ed's dz
edp
(1g((622M2/00) log (O( log(e (os(6-2d) -2d
=I( P + O
(Ta) (Ta)
edi
N (Iog(GN²MM2/00) log (O())
+o
ed
=(( for i,j E [MMM), n, n E [N]. F n. The first inequality is triangle inequality, the second inequality is by (005). the last equality is by dn = á( max{(SNR SNR NNNN= -2) and =
148

---

((e-2N2da).
1 1 s (bf x +
(k(f 1(5t1)) - (16 |6.0))
< [k+ k (
M
((0) D (an. (())
S 1=2
M
g(+)
+
BES
2
M
+ (5f))
g(*)
In.t
nES1 1=
M
(OL)) T+)
nES_ 1=2
(Te) (Taly
M
+
| I1K9
(
19 k: |(0)>
n.
EES,
i=2 s= T2
M
Σ nES
Σ
q k |(ا))
(5)
+
T2
i=2
+ flower order term}
N (log(6NN2 /0))) log (O(t))) loogle (og(e-1dh)
((4 (T2) (Ta)
edi
SNR? SNR2(Log(6N22) M² 18) log (O(t)) log(( 2d))
+ N.. M 0
cat
N (log(6N2MM//0)) /)) 3 log (0(2)) log((- (d
= -I(49 (3)) k Maall +
edi
3 log (O(t)) log(E -l
+ +0 N-
ed
o(1),
where the first inequality is triangle inequality, the second inequality is by 214), the last equality is by dn = γ max(SNR* SNR-SNRRR
SSNR 2)N2==
((6( Knd
(t)) 1 S (K(TT)) (Tx)
|
Σ
(
s= T3
< (1
(T2) (
M
n' t (gr (1))
+.t
n'ES+ 1=2
M
g(o)
n.it (9
(+) x(00)) + 3/02__ (0) k (o(1))
1=2
149

---

M
(84%) 9 +
g(a)
(5) (
n' 1=2
N
M
(1 ( (T)
+ 1/100 q
(+)
g(0)
(+)T)
ni I In'd
M
+it 11
+
18 [
n, . Ilgg |
2=2 s=
M
t (0) .44")
" n'E An' fn 1=2 ==
1860)
(3(5)
18(8) nam llg (x)
flower order term}
+ ( N (1oog M² 18)) log (O(t)) log(( l
ed SNR? (log(())2) M2 3 log (o(H)) log(e log(e=!dR)
+ M O
edj SNR? (log(6N2 M / log (0(t) o(1)
+N. M.O 0
od
N (1oo((NN2MM)))) 18)) log (O(2)) log(( -1d)) o
edi (log(6N> log (O(+)) log(e= -1())
M. 0
ed ( /8)) log (O(H)) log(c (c (ztptt
+N. M- 0
ed dx
N (1ogg(NN2M///1)) 3 log (O(.) log(c log(c=!d,)
(bb)) = -K4 (T2) k 1
+
ed
N SNR? ANR2((1gg() (206662222/0)) 3 log (O(+))
o
ed
o(1)
where the first inequality is triangle inequality, the second inequality is by 209.. the last equality is
by da = 22 (max(SNT) (.S-SSRN.N Similarly, we have (xx (e+1) k y = o(1).
(111))
(o+1) (((++1))11 < (TT)
(T)) (T + Σ
(1 k(0 - (x (0) (10)))
T2
(T5) ((33))) < (6.7))
150

---

N
M
(11))) +
31 (q
3(5)
+
==
+ 110)
(g ( =((e)
(8)
((0)
an a In' (())
(5)
-
N
M
+ (10% R 9 + +
n 1 In
n==1 1=2 N
M
(0n3 9 + (/1)) +.n.j---
(S)T
(T)
(x)T nij In'
n==10 1=2
< (k (2)) ا
M
1860)
18(0) n.J.
11:0 (1)
18(1) Baind (gn.i |,
(5)
==
M
(360)
((0) n.J++ (1
(c) +
[ |n (| (0) ,b(s))]
+
n'
1=2 T=
T=
M
N M
13
insmillaa (e))
+ Σ
Σ Σ
18 8(0)
Pn |1l (())))
1=2 =2
n' T2
a
+ (lower order term}
= - (
+ ( N (log(6N2M /8)) log (O(:)) log(e ddh))
edi
+ M 0 (oog(6N=M²) /)) log (O(t))) log(E -10/)
cd + N- M. OMMO0( ( (log(6N2 M2 log (o(t)) o(1)
ed' tt
N (1og(0N2M9/6)) log (O())) log(c-1d
= =K(k!
(1b)
((53)) + +
edi
N (log(6.N2 (3 (8))
log (O(t))
+
cdåd
for i,j E (M jnn E [N]. The first inequality is triangle inequality, the second inequality is
by (1T)), the last equality is by d, = á max(SNR' SNRR N)222)2 and d 5t (2
K& (++1) <0t))1 (k
(bb) K
( + Σ /o/tttt) klott))) (1/0)) ((19))
(Ta) (Ta))]
(kb k
N
M
g(s) (x) ( 2l (9
g(s) Paint An k(
n'=1
M
g(0)
(q ((0)) + B (q (+) [(a))
g(o)
Rggmn a (o k!(1))
(5)
151

---

M
+ (b0) (8) +.3%%)
((5)
(5)
+ 9
n.. q
n Ann
NV
M
(+) (8)T
T
(8)
(s)
n.J+ + TT..
n2=1
l=2
(1a (T2)) }
n, mnj
18 n.it 11
|3(0)
11
1g(0) n. (g ا
M
(3(+)
nii.n
An,1
(2)
R.
[g($)
n.i,'' z (1 (5)
9 |
1=2
1=2
M
(6)
+
[a15]
11
(0)
g(s) in (lKg (x) (!!)
R.J.
n.J:
8=12
1=2 1=12
Σ
(g(+]
E
(8(0)
12
AnAn 1=2
+ flower order term}
(T)) (
+ 0 N (log(6N22M11)) log (0(+)) log((= M 0 (1og(6N²M2/66) log (O(+)) log(e -1
c +M 0 [log(6N2 / log (O(E)) o(1) + N. W.W.MM. 0
ed d
[oog(6N2M>/0)) log (O(#)) o(1)
edp
ed d
= - n.
(Tb) ((a))
N ((lo((NN2MM/8) log (O( log(r ( + of N N(1og(6N2M2/00) log (O
+0
ed
ed dz
-()
for i,j E [M]\[1),,, E [N].n * n. The first inequality triangle inequality, the second inequality is by (12 the last equality is by d, δ max(SNR* SNN-S (NNR-1]N22 =)NN2 and ā( N2dd).
F.12 Explanations of Lower Order Terms
In this section, we provide some explanations of lower order terms to demonstrate the rigor of our proof.
To bound the so-call flower order term], we condition that dimensions d, d, are sufficiently large and learning rate is sufficiently small. Next, we show how we utilize these three parameters.
Sufficiently mall learning rate 7 Recall the dynamics of QK
(-11),, (q))) ((),AA k()) (A (())) + (g) A + (A, (,(()) (8) Note that terms (1), 1: (9), (1) (1() contain factor 77, and term (A) (t) contains factor n/2. Therefore, as long as yy is sufficiently small, (!)()) A is sufficiently small than (!) k!0)
and (9) (t)))). Now take the dynamic of (g ((())
(1)
example.
152

---

( (1+1) {((+1)) (q+ k
M
12
(4 (0)
R + 1=2
M
+ 1
(
n.
RES 1=2
M
(2)
+
nES+ x=2
M
(00)
T
(+++
4
a,t In.i
nES+
Under benign overfitting regime, we have || O do) (a ( s Ioo((= and
( (11 Therefore, as long as o( dr (log C (df))) we have
(a (2) k (4) 4
2T ol (1)
113)
Similar method can be used for other items in
a+.+
a
(1)
k + Σ
Σ
(sff (O)
RES x
Σ
M
g(t)
n.t.o n.t.
(1)T) At last, we have
nES,
(0 ((t2) x((11)
144 (())
M Σ
-. 11:44111
Rit 1 9
RES+ i=2
(297)
M
+ 144+1114 (1)
NES+
+ flower order term].
Sufficiently large dimension d, Take 297 as an example, Noting that ||< 11 = (nll
and (1. ((1))) o(L). Therefore, as long as d, sufficiently large, [| |/3 is much larger than
(R:4 (()). Besides, by the property that the sum of each row and column of matrix (ding((?!.))
M
9n,1 ( (00) is O, we have a 10 Σ Σ
n.t. 0. We also prove a (1) 2 0 and a 0
(0)
S
2=2
under benign overfitting regime, thus the magnitude of a (1) is smaller than 4 All all, it can
M
be proved that
Σ
(e)
(1 (1))) This method can be further applied to (k(2) ((99) = o(a (2)
[1)..
nES_
M
800
11
(8)
12
Σ
Σ
nnt
(9
(1)
( At last, simplify (297) follows:
NES+
t=2
(q (++1)
k (1+1)) - (9 (
(298)
a + [|
(t) (t) 112 =0(0 11 (t) 12 flower order term].
153

---

Sufficiently large dimension d: Take (0)
0 as an example:
7
Σ
S Sma))
NM
exp((gn..0 1.1)))
(v
M
exp( (1) 1) + Σ
exp( (q, (n))))
j=2
exp((g (0)
)2)
exp((gn)) k (211))
Σ
expl (1)
(gn. b11))))
(299)
-=
exp( (n, k b44)))
(v())
M
exp(iam.) k())
Σ (g
(1)
exp((g (
M
exp( (t)
()))
Σ exp((a
(11)))
Note that (Eni,e, En,s) can be divided into two types: || 11 and (Ewr,, En,) for fori or
By Lemma
C.4 we have
o.d/2 S En,21 < d/
0 16n,i12 S 30 i En.e) s 2 Vdlog(4N23M/0) i, [M]\1)),,, E (N), * 1' n n n'.
As long as is sufficiently large, / 1/ much larger than (Enry, Eng)) for i + or n + n'. Therefore, 299 can be further simplified as follows:
(0)
NM E 11
"
())
expl N. b())))
M
explig (1)
1²))+ + Σ
exp (gp (1)
-
exp((g (1)))
j²)
1
exp((g.)) 10 A (1)))
+ Σ
exp (n'' , [1
1=2
(300)
M
exp( (gp((g)))) g k(1)))
(v(t)
exp(1a 1 (211
Σ exp( (q (.
(1)
cxp((g 1.n 11)1))
(1)
M
exp xxp k22) + Σ Ee 10
(t)
(000))
3=2
+ flower order term).
G Takeaways for Practitioners
Our theoretical results mainly focuse on the impact of different N and SNR on the generalization performance. So we can provide guidance from the perspective increasing N, SNR and N SNR*. The following are some practical scenarios
154

---

Researchers sometimes employ the technique of data augmentation by intro- ducing controlled noise into their datasets. From the perspective of our paper's results, this method reduce SNR but improve N because we generate "new" data point by adding noises. As reducing SNR may be harmful to generalization performance, we must make sure that we use enough data points to train the model( enough sample size N) )
Data Augmentation:
Semi-Supervised Learning: Semi-supervised learning is useful when you have small amount of labeled data and a large amount of unlabeled data. Labeled data can be seen as data with high SNR, while unlabeled data with low SNR because for some unlabeled samples, we may mistake their labels, making them equivalent to noises. In this scenario, we need to ensure that we have sufficient unlabeled data (enough sample size N) and make full of labeled data (high SNR data points).
Overall, we need to consider both the sample size and the signal-to-noise ratio SNR to train the model.
H Broader Impacts
This work focus on theoretically studying the training dynamics and generalization of Transformer in Vision. The techniques used in this paper may be generalized to study other abilities Transformer or other network models. Besides, the theoretical results in this paper may inspire more attempts at training large foundational models with high-quality data. We do not foresee any form of negative social impact induced by our work.
155

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract reflects the paper's scope, and the introduction reflects the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitation in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
156

---

We provide the assumptions in Condition 4.1 The detailed proofs are in the appendix. We also provide a proof sketch in the main paper.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: provide the experiments setting this paper, and the codes are submitted as supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
157

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The codes are submitted as supplementary material, which include readme
file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligggddeeeeeeeeeeee Cotee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed data generation methods and hyperparameters in the experiments setting.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: plot the 2-sigma error bar for the training dynamics in Appendix A Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
158

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the computer resources in experiments setting, All the experiments in this paper can be performed within hours.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work focuses on theoretical study of ViT, and all the data is synthesized. We see ethical or immediate negative societal consequence of our work, and it will not violate the Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the societal impacts in Appendix H There are no negative social impacts in this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
159

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All the data this paper is synthesized. This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: All the data in this paper is synthesized. This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
160

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
All the data in this paper is synthesized. This paper does not release new assets.
Justification:
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
161

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
162


Reviews:
[{"review_id": "45vrnvSMT3", "reviewer": "Reviewer_GNLt", "summary": "The paper studies the benign overfitting phenomenon for a two-layer Transformer in vision. The paper characterizes the optimization of the ViT through three different phases in training dynamics and finds a sharp separation condition of the signal-to-noise ratio to distinguish the benign and harmful overfitting of the ViT.", "questions": "In Theorem 4.1 and Theorem 4.2, the requirement is related to SNR. Why does $\\tilde{\\sigma}_p$ not occur in the requirement?", "limitations": "The authors have stated the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper gives a sharp transition between benign and harmful overfitting for ViT, which can be verified by a simulation experiment.\n2. The paper proposes a novel method of vectorized $QK$ and scalarized $V$ to simplify the study of Transformer.\n3. The paper successfully deals with the challenges caused by softmax and multiple weights.", "weaknesses": "1. Lack the introduction of the benign overfitting phenomenon in the first section.\n2. It is better to clarify some notations like $\\Omega(\\cdot)$, $\\Theta(\\cdot)$, $\\omega(\\cdot)$...\n3. Data generation is specified, thus this model may be a little bit limited.\n4. A small typo: (Line 149) $\\mu||_2^{-2}$ --> $||\\mu||_2^{-2}$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback! We address your questions and concerns as follows. \n\n---\n\n**Q1**. Lack the introduction of the benign overfitting phenomenon in the first section.\n\n**A1**.  In the first section, we first introduce the Transformers and ViT models, and then the empirical and theoretical studies of ViT, and finally the benign overfitting phenomenon, followed by our contributions. The introduction for benign overfitting is in Lines 35 - 38. Benign overfitting is a phenomenon where \"**the test error remains small despite overfitting to the training data**\". A more detailed introduction is in the first paragraph of the second section. We plan to add more explanations for benign overfitting.\n\n---\n\n**Q2**. It is better to clarify some notations like $\\Omega(\\cdot), \\Theta(\\cdot), \\omega(\\cdot)$\n\n**A2**. Thank you for pointing it out, we will add a paragraph to explain them.\n\n---\n\n**Q3**. Data generation is specified, thus this model may be a little bit limited.\n\n**A3**. The data generation model in this paper follows the common assumptions made in previous studies of benign overfitting ( Cao et al. (2022) [1], Kou et al. (2023) [2]). To demonstrate the practicality of our theoretical results in real-world datasets, we perform some experiments on MNIST dataset. The results are detailed in the uploaded PDF file. The experimental result shows a transition between benign and harmful overfitting regimes. The larger the sample size $N$ and signal-to-noise ratio SNR, the better the generalization performance.\n\n---\n\n**Q4**. In Theorem 4.1 and Theorem 4.2, the requirement is related to SNR. Why does $\\tilde{\\sigma}_p$ not occur in the requirement?\n\n**A4**. In Condition 4.1 (10), we require that $\\tilde{\\sigma}_p = C_p \\sigma_p$ and $C_p = 5 \\sqrt{M}$. We do not make any requirement on $\\Vert \\mu \\Vert $ and $\\sigma_p$ in Condition 4.1 so that they are flexible. For example, if we want $\\mathrm{SNR} = \\Theta(1)$, then we can choose:\n\t  $\\Vert \\mu \\Vert = \\Theta(d^{1/2}) , \\sigma_p = \\Theta(1)$\nor   $\\Vert \\mu \\Vert = \\Theta(1) , \\sigma_p = \\Theta(d^{-1/2})$.\nThis setting enable the input to be more general. In real-world dataset, the normalization methods for data may not be the same. Sometimes images are normalized to 0 - 1, while sometimes  they are 0 - 255.\n\n---\n\n**References**\n\n[1] Cao, Y., Chen,Z., Belkin,M., and Gu,Q. Benign overfitting in two-layer convolutional neural networks. In NeurIPS 2022\n\n[2] Kou, Y., Chen, Z., Chen, Y., and Gu, Q. Benign overfitting in two-layer ReLU convolutional neural networks. In ICML 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5i6hRKlfI3", "reviewer": "Reviewer_tKtT", "summary": "This study investigates the theoretical aspects of Vision Transformers (ViT) with a focus on their generalization capabilities, particularly under conditions of benign overfitting. Through a detailed analysis of the optimization process involving a self-attention layer and a fully connected layer, optimized using gradient descent on a specific data distribution modal, this work addresses the complexities introduced by softmax functions and the interdependencies of multiple weight configurations in transformer models. By developing novel techniques, the researchers delineate the training dynamics that lead to effective generalization in post-training scenarios. A key contribution is the establishment of a sharp condition based on the signal-to-noise ratio within the data, which predicts whether a small or large test error will occur. These theoretical findings are supported by experimental simulations, enhancing our understanding of transformers' performance in vision tasks.", "questions": "See in Weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper rigorously analyzes the training dynamics of a simplified ViT model. Specifically, its technical contributions related to \"Vectorized Q & K and Scalarized V\" and \"Dealing with the Softmax Function\" may have a broader impact on subsequent theoretical research.", "weaknesses": "The current experiments only validate the theoretical results on synthetic datasets. It is recommended that the authors consider adding some experiments on real datasets to test the effects, such as experiments on benign overfitting of the ViT model on MNIST and CIFAR10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback! We address your questions and concerns as follows. \n\n---\n\n**Q**. The current experiments only validate the theoretical results on synthetic datasets. It is recommended that the authors consider adding some experiments on real datasets to test the effects, such as experiments on benign overfitting of the ViT model on MNIST and CIFAR10.\n\n**A**. We perform some experiments on MNIST dataset. Please find the results in the uploaded PDF file. The experimental result shows a transition between benign and harmful overfitting regimes. The larger the sample size $N$ and signal-to-noise ratio SNR, the better the generalization performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GKpIB3rmhq", "reviewer": "Reviewer_UpG9", "summary": "The paper investigates **the benign overfitting phenomenon in Vision Transformers**. The authors adopt a theoretical framework similar to that proposed by Cao et al. (2022), which use **a data model consisting of label-dependent signal and label-independent noise**, but employ **a two-layer Transformer architecture** instead of a two-layer convolutional neural network. They provide conditions under which benign and harmful overfitting occur.", "questions": "* In the data distribution (Definition 3.1), what is **the role of the larger noise $\\xi_2$ in the analysis**? What would happen if the data distribution were the same as that considered in Cao et al. (2022), which consists of a single signal patch and a single noise patch?\n* In the numerical results section, it would be beneficial to compare the results with those for a two-layer convolutional neural network, as considered in Cao et al. (2022), to emphasize the advantages of the transformer architecture.\n\nReference\n\n[1] Cao, Y., Chen,Z., Belkin,M., and Gu,Q. Benign over fitting in two-layer convolutional neural networks. In NeurIPS 2022", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper provides **conditions under which benign and harmful overfitting occur** in a two-layer transformer, with results that are **tight up to a constant**.\n* The authors overcome challenges in analyzing the highly complex training dynamics of transformers by introducing **a novel technique called vectorized Q&K and scalarized V**, successfully addressing the learning dynamics. To the best of my knowledge, this is **the first work to successfully address the learning dynamics of even a simple transformer architecture without unrealistic assumptions** (e.g., merging the key-query weights). If all proofs are correct, this represents a significant technical contribution.", "weaknesses": "While I believe that the vectorized Q&K and scalarized V techniques are significant contributions, **it is difficult to verify the correctness of the proofs** due to readability issues in both the main text and the appendix. I suggest that the authors **improve the clarity and writing of their technical terms and proofs**.\n\nMinors and Typos\n* The notion of $\\mu$ in Definition 3.1 seems unnecessary\n* Line 9: modal→model\n* Line 541: Eexperimantal Rresults → Experimental Results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback! We address your questions and concerns as follows. \n\n---\n\n**Weakness**. Readability issues\n\n**A**. We realize that readability is important for readers to understand and further apply our techniques. In order to enhance readability, we have made the following efforts:\n\n- We present a proof sketch in the main text, which contains three main challenges and our solutions, allowing readers to quickly access our techniques.\n- We provide a notation table in page 16, containing the key shorthand notations in this paper.\n- In the appendix, we separate the high-level proof process from the low-level proof process. For example, in appendix F, we present some bounds for $\\alpha$, $\\beta$ and so on (low-level). In appendix D, we use the bounds in appendix F to complete the proofs for the key steps in the training dynamics (high-level).\n\nWe will make more efforts to further improve the readability as follows:\n\n- extend the proof sketch: We consider providing a more comprehensive proof sketch in the appendix to fully describe the key steps in the proof process.\n\n- simplify the notations: The large number of symbols makes it difficult for readers to recall the meaning of a particular symbol. So we will do efforts to make it easier to remember and understand.\n- improve the clarity and writing: We will add more textual descriptions and remarks during the proof process to help readers better understand the proofs.\n\n---\n\n**Q1**. In the data distribution (Definition 3.1), what is the role of the larger noise $\\xi_2$ in the analysis?\n\n**A1**. By Condition 4.1.(10), we ensure that the norm of $\\xi_2$ is sufficiently larger than that of other noises. Then under harmful overfitting regime, it becomes easier to prove that $\\xi_2$ attract most of the attention and the model learns little signal, thus the test loss will be high. \n\nWithout Condition 4.1.(10), it is much more challenging to prove harmful overfitting results. Next we discuss the difficulty we face when $\\xi_2$ share the same variance with other noises ($\\tilde{\\sigma}_p = \\sigma_p$)\n\nTo prove harmful overfitting, we must prove that the model learn little signals, i.e., signals attract less and less attention and $W_V$ learn little signals.\n\nTo prove that the model learn little signals while the training loss converges, we must prove that the model memorizes the noises, i.e., noises attract most of the attention and $W_V$ memorizes the noises well.\n\nBut it is difficult to characterize the attention attracted by the noises. For example, consider the scenario at initialization where there are noises $\\xi_i$ and $\\xi_j$ satisfy: more attention is paid on $\\xi_i$ than $\\xi_j$ ( attn($\\xi_i$) $>$ attn($\\xi_j$) ), and $W_V$ memorizes $\\xi_j$ better than $\\xi_i$ ( denoted by $\\rho_j > \\rho_i$ ) (under Gaussian initialization, this situation may occur). As we mention in Section 5.2, QK affect V ,and V affect QK. Therefore, the condition $\\rho_j > \\rho_i$ may lead to an increase in attn($\\xi_j$) and a decrease in attn($\\xi_i$) in the next training iteration. Meanwhile, the condition attn($\\xi_i$) $>$ attn($\\xi_j$) may result in $\\rho_i$ growing faster than $\\rho_j$. Therefore, in the next iteration, the following situations may happen:\n\n- attn($\\xi_i$) $>$ attn($\\xi_j$),  $\\rho_i > \\rho_j$\n- attn($\\xi_i$) $>$ attn($\\xi_j$),  $\\rho_i < \\rho_j$\n- attn($\\xi_i$) $<$ attn($\\xi_j$),  $\\rho_i > \\rho_j$\n- attn($\\xi_i$) $<$ attn($\\xi_j$),  $\\rho_i < \\rho_j$\n\nWe cannot know which situation will occur, let alone give them precise bounds. New techniques need to be developed to handle this difficulty.\n\nIn this paper, we let $\\xi_2$ to be stronger than other noises, thus much attention will be paid on $\\xi_2$, and $W_V$ will memorize $\\xi_2$ well under harmful overfitting regime.\n\n---\n\n**Q2**. What would happen if the data distribution were the same as that considered in Cao et al. (2022), which consists of a single signal patch and a single noise patch?\n\n**A2**. It is a special case of our Condition (number of input tokens M=2), and the conclusion will remain unchanged.\n\n---\n\n**Q3**. In the numerical results section, it would be beneficial to compare the results with those for a two-layer convolutional neural network, as considered in Cao et al. (2022), to emphasize the advantages of the transformer architecture.\n\n**A3**. Thank you for your suggestion. We made the comparison in Lines 176 - 178. We will consider adding numerical results to emphasize it.\n\n---\n\nReference\n\n[1] Cao, Y., Chen,Z., Belkin,M., and Gu,Q. Benign overfitting in two-layer convolutional neural networks. In NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TGGVsiJ5vZ", "reviewer": "Reviewer_ysmG", "summary": "This paper provides a sharp theoretical characterization of the transition between benign and harmful overfitting regimes for Vision Transformers trained on linearly separable data. The authors carefully analyze the optimization dynamics and provide generalization bounds that depend on the signal-to-noise ratio of the data. Extensive experiments validate the theory.", "questions": "* If an MLP with ReLU activation was used instead of the simple linearly separable task, would the optimization dynamics converge faster or slower? Would the SNR requirements be stricter or more relaxed?\n* What guidance can the authors provide to practitioners based on these theoretical results?", "limitations": "The authors acknowledge focusing only on the simplified linearly separable setting. More discussion on how the insights could extend to real-world nonlinear problems would be valuable.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Provides a precise characterization of benign vs harmful overfitting regimes for Vision Transformers\n* Writing is clear and easy to follow\n* Novel results on the harmful overfitting regime\n* Extensive experiments align well with and validate the theory", "weaknesses": "* Only considers the linearly separable setting, which is already solvable by existing vision and language models, so the conclusions are not very surprising even if initialization and model details differ\n* Lacks clear takeaways for practitioners - how can these theoretical insights be used to improve real-world vision models?\n* Could the authors provide a more rigorous perspective on transformer bias (e.g. low rank structure) from an optimization perspective under their model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback! We address your questions and concerns as follows. \n\n---\n\n**Q1**. Only considers the linearly separable setting, which is already solvable by existing vision and language models, so the conclusions are not very surprising even if initialization and model details differ.\n\n**A1**. We follows the common assumptions made in previous studies of benign overfitting ( Cao et al. (2022)[1], Kou et al. (2023)[2] ) for the data generation model in this paper. Although this problem seems simple, rigorously proving these conclusions to show benign overfitting phenomenon is very challenging when considering the complexity of ViTs. In addition, rather than only presenting benign overfitting conclusion as most existing works, we also present harmful overfitting conclusion to show that the ViT cannot even learn a linearly separable dataset when the sample size $N$ and signal-to-noise ratio SNR is low.\n\n---\n\n**Q2**. Lacks clear takeaways for practitioners. What guidance can the authors provide to practitioners based on these theoretical results?\n\n**A2**. Our theoretical results mainly focuse on the impact of different $N$ and SNR on the generalization performance. So we can provide guidance from the perspective of increasing $N$, SNR and $N \\cdot \\mathrm{SNR}^2$. The following are some practical scenarios.\n\n- Data Augmentation: Researchers sometimes employ the technique of data augmentation by introducing controlled noise into their datasets. From the perspective of our paper's results, this method reduce SNR but improve N because we generate \"new\" data point by adding noises. As reducing SNR may be harmful to generalization performance, we must make sure that we use enough data points to train the model( enough sample size $N$ ).\n- Semi-Supervised Learning: Semi-supervised learning is useful when you have a small amount of labeled data and a large amount of unlabeled data. Labeled data can be seen as data with high SNR, while unlabeled data with low SNR because for some unlabeled samples, we may mistake their labels, making them equivalent to noises. In this scenario, we need to ensure that we have sufficient unlabeled data (enough sample size $N$) and make full use of labeled data (high SNR data points).\n\nOverall, we need to consider both the sample size $N$ and the signal-to-noise ratio SNR to train the model.\n\n---\n\n**Q3**. Could the authors provide a more rigorous perspective on transformer bias (e.g. low rank structure) from an optimization perspective under their model?\n\n**A3**. We present a brief discussion about the rank of matrix $W_V$ under some conditions as follows:\n\nSuppose that $SNR \\rightarrow \\infty$, then we have $\\Vert \\mu \\Vert / \\Vert \\xi \\Vert \\rightarrow \\infty$. Recall Equation (9) (Line 583), we have:\n\n$\\lim \\ \\nabla_{W_V}L_S(\\theta) = ( a \\mu_+ + b \\mu_- ) w_O^\\top$\n\nwhere a and b are linear combination coefficients for $\\mu_+$ and $\\mu_- $. Note that $\\mu_+$, $\\mu_- $ and $w_O$ are vectors (rank = 1), so we have:\n\n$R(\\lim \\ \\nabla_{W_V}L_S(\\theta)) = 1$\n\nConsidering that the weights of model are usually initialized to be very small, matrix $W_V$ tends to decrease its rank during training because it changes towards its gradient direction $\\nabla_{W_V}L_S(\\theta)$. The analysis of QK is also similar.\n\nAs for a more rigorous proof, we will consider developing new theoretical techniques to prove it.\n\n---\n\n**Q4**. If an MLP with ReLU activation was used instead of the simple linearly separable task, would the optimization dynamics converge faster or slower?\n\n**A4**. Kou et al. (2023)[2] investigate the benign overfitting phenomenon in two-layer ReLU CNN. Note that their data model only contains one patch of signal and one patch of noise, we can consider their CNN model to be a special MLP, especially in cases of very high or low SNR. Their results show that their model can converge to training loss $\\epsilon$ at $t = \\eta^{-1} poly(\\epsilon^{-1}, d, n, m)$, while the ViT model we consider can converge at $t = \\eta^{-1} poly(\\epsilon^{-1}, \\Vert \\mu \\Vert, \\Vert w_O \\Vert)$. Ignoring parameters such as network size, sample size, norm of input and so on, we found that their convergence time is proportional to $\\eta^{-1}$ and $\\epsilon^{-1}$. So we can think that their convergence speed is of the same order.\n\n---\n\n**Q5**. Would the SNR requirements be stricter or more relaxed?\n\n**A5**. According to Condition 4.1. in Kou et al. (2023)[2], they require $\\mathrm{SNR}^2 \\le \\tilde{O}(1/N)$ and require $\\Vert \\mu \\Vert_2$ to be large enough, which is stricter than our condition.\n\n---\n\nReference\n\n[1] Cao, Y., Chen,Z., Belkin,M., and Gu,Q. Benign overfitting in two-layer convolutional neural networks. In NeurIPS 2022\n\n[2] Kou, Y., Chen, Z., Chen, Y., and Gu, Q. Benign overfitting in two-layer ReLU convolutional neural networks. In ICML 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PMNsIINeyJ", "reviewer": "Reviewer_4uNF", "summary": "The paper investigates the benign overfitting phenomenon in Vision Transformers. By examining the training dynamics and generalization of a two-layer Transformer model, it establishes a condition to differentiate between benign and harmful overfitting based on the signal-to-noise ratio in the data model. Theoretical results are supported by experimental simulations.", "questions": "The authors are suggested to explore scenarios that deviate from these assumptions to understand the limitations of the theoretical results. For example, more signal patches and more noise patches are contained in all patches. Tests on real-world tasks are better.", "limitations": "See Questions and Weaknesses.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper provides a deep theoretical understanding of how Vision Transformers can achieve benign overfitting, filling a gap in the literature.\n2. The derivation of the conditions for benign and harmful overfitting is mathematically rigorous and well-founded. Strong conclusions drawn based on appropriate assumptions.\n3. The theoretical findings are supported by experimental simulations, which confirm the sharp condition separating benign and harmful overfitting.", "weaknesses": "1. The sparsity assumption is too strong, as it relies on the signal being contained within one patch and the noise being contained within another, which may not align with real-world data distributions.\n2. Experiments are limited to settings that perfectly align with theoretical assumptions. The paper should also explore scenarios that deviate from these assumptions to understand the limitations of the theoretical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback! We address your questions and concerns as follows. \n\n---\n\n**Q1**. The sparsity assumption is too strong, as it relies on the signal being contained within one patch and the noise being contained within another, which may not align with real-world data distributions.\n\n**A1**. The data generation model in this paper follows the common assumptions made in previous studies of benign overfitting ( Cao et al. (2022) [1], Kou et al. (2023) [2]). This assumption is not so strong for recent studies on benign overfitting, especially when we investigate on more complex ViT models. As Reviewer UpG9 says, \"To the best of my knowledge, this is the first work to successfully address the learning dynamics of even a simple transformer architecture without unrealistic assumptions\". \n\nWe also acknowledge that the data generation model may not align perfectly with real-world data distributions. In our perspective, the signals in the data model simulate the targets within an image, while the noises simulate the background in the image. We perform some experiments on MNIST dataset to verify our theoretical results (see next question), and we will try to make our theoretical results helpful for more practical applications.\n\n---\n\n**Q2**. The authors are suggested to explore scenarios that deviate from these assumptions to understand the limitations of the theoretical results. For example, more signal patches and more noise patches are contained in all patches. Tests on real-world tasks are better.\n\n**A2**. We perform some experiments on MNIST dataset. In these experiments, we used different dataset, different noise, and different network model than in this paper. So the results regarding N and SNR may be different, which shows the limitations of our theoretical results. \n\nOne of our important conclusion is valid, that is, the larger the sample size $N$ and signal-to-noise ratio SNR, the better the generalization performance. Please refer to the uploaded PDF file. \n\n---\n\nReference\n\n[1] Cao, Y., Chen,Z., Belkin,M., and Gu,Q. Benign overfitting in two-layer convolutional neural networks. In NeurIPS 2022\n\n[2] Kou, Y., Chen, Z., Chen, Y., and Gu, Q. Benign overfitting in two-layer ReLU convolutional neural networks. In ICML 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We kindly thank all the reviewers for their time and for providing valuable feedback on our work.

To validate the practicality of our theoretical results in real-world datasets, we perform some experiments on MNIST dataset. The experimental result shows a transition between benign and harmful overfitting regimes. The larger the sample size N and signal-to-noise ratio SNR, the better the generalization performance.

For more details, please see the PDF.

Meta-review: The reviewers unanimously agree that this paper should be accepted and presented at neurips, and the AC sees no reason to overturn this recommendation. However, as noted by the reviewer, there are two caveats:
1) Real experimental results are needed to empirically validate the derivations. The authors added MNIST experiments during the rebuttal, which need to be included in the final version of the paper.
2) The proofs are difficult to read, follow, and verify. It sounds like none of the reviewers, nor the AC, was able to fully follow and verify them. Hence, improved clarity is needed. However, we hope that the publication of this paper attracts further scrutiny by the community and leads to follow-up works.
Decision: Accept (poster)