On the Generalization Ability of Next-Token-Prediction Pretraining
Authors: Zhihao Li
Xue Jiang
Liyuan Liu
Xuelin Zhang
Hong Chen
Feng Zheng
Venue: ICML 2025 poster (2025)
Keywords: Next Token Prediction; Decoder Only Models; Generalization Bounds
Primary area: theory->learning_theory

Abstract: Large language models (LLMs) have demonstrated remarkable potential in handling natural language processing (NLP) tasks and beyond. LLMs usually can be categorized as transformer decoder-only models (DOMs), utilizing Next-Token-Prediction (NTP) as their pre-training methodology. Despite their tremendous empirical successes, the theoretical understanding of how NTP pre-training affects the model's generalization behavior is lacking. To fill this gap, we establish the fine-grained generalization analysis for NTP pre-training based on Rademacher complexity, where the dependence between tokens is also addressed. Technically, a novel decomposition of Rademacher complexity is developed to study DOMs from the representation learner and the token predictor, respectively. Furthermore, the upper bounds of covering number are established for multi-layer and multi-head transformer-decoder models under the Frobenius norm, which theoretically pioneers the incorporation of mask matrix within the self-attention mechanism. Our results reveal that the generalization ability of NTP pre-training is affected quantitively by the number of token sequences $N$, the maximum length of sequence $m$, and the count of parameters in the transformer model $\Theta$. Additionally, experiments on public datasets verify our theoretical findings.

On the Generalization Ability of Next-Tooee-Prediicion Pretraining
Zhihao Li" Xue Jiang? Liyuan Liu ! Xuelin Zhang'
Hong Chen 14 Feng Zheng? 2
Abstract
1. Introduction
Large Language Models (LLMs) have emerged as pow- erful generative models in solving (seq2seq) tasks (Ott al., 2019), which not only have achieved tremendous progress in various NLP tasks (Malach, 2023), but also have realized remarkable performance in other domains (Li et al., 2024). Surprisingly, several exist- ing LLMss, such as GPT3 (Brown et al., 2020), OPT (Zhang et al., 2022), BLOOM (Workshop et al., 2023), Llama (Tou- vron et al., 2023), Deepseek (Liu et al., 2024a) and Qwen (Yang et al., 2025), share two common characteristies: (i) Employing decoder-only architecture based on the masked- self-attention( Vaswani et al., 2017). (ii) Adopting the un- supervised pre-training method of Next- Tew-e--ooo-eiccceee (NTP) (see Figure/ 1 (a)), which is to predict the next to- ken based on all previous context tokens in each step (Qi et al., 2020). The predominant expense training large language model is typically incurred during the pre-training phase (Zhao et al., 2024). Consequently, it is very important to examine the DOMs-based NTP pre-training.
Large language models (LLMs) have demon- strated remarkable potential in handling natu- ral language processing (NLP) tasks and be- yond. LLMs usually can be categorized as transformer decoder-only models (DOMs), uti- lizing Token-Toke-BB Prediction (NTP) as their pre- training methodology. Despite their tremendous empirical successes, the theoretical understand- ing of how NTP pre-training affects the model's s generalization behavior is lacking. To fill this gap, we establish the fineggrained generalization anal- ysis for NTP pre-training based on Rademacher complexity, where the dependence between to- kens is also addressed. Technicalll,, a novel de- composition of Rademacher complexity is de- veloped to study DOMs from the representation learner and the token predictor, respectively. Fur- thermore, the upper bounds of covering number are established for multi-layer and multi-head transforme-ddccode models under the Frobenius norm, which theoretically pioneers the incorpo- ration of mask matrix within the self-attention mechanism. Our results reveal that the gener- alization ability of NTP pre-training is affected quantitively by the number of token sequences N, the maximum length of sequence m, and the count of parameters in the transformer model e. Additionally, experiments on public datasets ver- ify our theoretical findings. Our code is available at http://gittub.com/Liziieee
Recently, there have been increasing efforts to evaluate the DOMs-based NTP pre-training empirically. Shlegeris et al. (2022) found that language models are consistently better than humans at NTP tasks by performing two distinct ex- periments. Malach (2023) demonstrated when trained on Chain-of-Thought data, even a lincar next-token predictor can possess high fitting ability. Bachmann & Nagarajan (2024) designed a minimal planning task and demonstrated that NTP pre-training cannot accurately predict the first position in some tasks. Li et al. (2024) utilized a single self-attention layer to explore the mechanics of NTP. While these works justify the use of NTP pre-training in the corre- sponding regimes, they do not provide rigorous analysis of the training mechanism, especially from the perspective of generalization theory. This motivates natural question:
'College of Informatics, Huazhong Agricultural University 'Dparrmeent of Computer Science and Engineering, Southern University of Science and Technology 'Department of Com- puter Science, Hong Kong Baptist University 'Engineering Re- search Center of Intelligent Technology for Agriculture, Min- istry Education, China. Correspondence to: Hong Chen <cchenh @ mail. hww..cdu.cn
"Can we establish the generalization analysis of NTP pre- training, which probably explains the effects of model pa- rameters?"
This paper answers the above question positively. The DOMs (see Figure 1 (b)) usually consist of two compo- nents (Zhao et al., 2024): multiple layers of transformer- decoder-blocks, shortened as Representation-Leareee (R-L), and a snseific processor, noted by Token-Predicta Predictor (T-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On the Generalization Ability of Next Text-Token-Predictoon Pretraining
E2
3
T-P linear & softrma
Add & Layer-Noms
transformer-deet ceaesfomee--eecdee-bbeeee decoder-block-L.
Feod-Porward
LM (Decoder-Only-Mode))
Add & Layer-Noms
transformer- deapemeer-ddood--bboooe
R-L
Embeddling
Maskeed- Mollied-Multi-HHaee Hadd---n--
to
2
(a)
(b)
Figure 1. (a): How NTP works utilizing dccoder-only model (DOM), for every input token (0 S < m), we can get an output token (/+1 whose label is 1,+1. and the dashed line here represents context used. (b): The architecture of DOM, which consistent with the GPT-3 (Brown et al., 2020). (c): The architecture of wansformer-decodes beccker-blook (Vaswani et al., 2017).
P). Similar to Zhang et al. (2024a); Deng et al. (2024), we abstract the DOMs into loken-based composite function made up of two separate functions for R-L and T-P, respec- tively. We consider the dependence between tokens and utilize şp-mixing to delineate the inter-token dependencies, which is commonly used tool in non-independent scenes (Masuda, 2007; Mohri & Rostamizadeh, 2010; McDonald et al., 2015; Wong et al., 2020). We then propose theoret- ical framework for NTP from the perspective of statistical learning. To bound the excess risk of NTP, we introduce the concept of Rademacher complexity for composite func- tion classes and propose a decomposition law, as stated in Proposition 4.8. We then establish two distinct bounds on the generalization capability of NTP, depending whether Proposition 4.8 is applied.
A novel Rademacher complexity decomposition method: We consider the dependence between to- kens and provide a theoretical framework for NTP pre-training (Section 3). On this basis, we establish the Rademacher complexity upper bounds of excess risk by novel Rademacher complexity decomposition method (Section 4. 1),, which shows that the generaliza- tion performance of NTP pre-training is related to both sequences and tokens.
A refined covering number for multi-layer, multi-head wransormer-decokee models: We establish bounds for the covering number of a function space derived from a multi-layer, multi-heat cunsformer-decodee model based on sesked-self-attentioo (Section 4.2). Unlikely the previous works, theoretical results are the first to consider the mask matrix in the metric induced by the Frobenius norm.
To further assess the impact of model parameters of DOMs on NTP, we provide a refined estimation of the covering number for multi-layer, multi-head transformer-decode blocks. We are the first to consider the mask matrix in self-attention, which is crucial for NTP. We then use the covering number of DOMs get the corresponding Rademacher complexity upper bound by utilizing the the- ory of Bartlett et al. (2017); Lin & Zhang (2019) and es- tablish the generalization bounds for DOMs-based NTP pre-training. Our results primarily encompass three key pa- rameters: the number of token sequences N, the maximum sequence length m, and the count of transformer model parameters O Our generalization bound can be expressed as (( /0/Nm + v1/m), where O ((ve) ((VQ/Mm) signifies the generalization capability across token sequences, and O (VIIm)) denotes the generalization capability among individual tokens. Our bounds remain valid even with modi- fications to the structure of the transformer-decoder block. Our main contributions are summarized as follows:
based on
A generalization bound for DOMs-based NTP pre- training: We use the Rademacher complexity upper bound and covering number to establish the generaliza- tion theory of DOMs-based NTP pre-training (Section 4.3). Theoretical results imply that the generalization bound mainly depends on: the number of token se- quences N, the maximum length of the token sequence m, and the number of model parameters O. Data ex- periments in Section 5 verify our theoretical findings.
2. Related Work
Next-Token-Predictin (NTP). Beyond its prominence in NLP (Moon et al., 2021; De Souza Pereira Moreira et al., 2021), NTP has found applications in diverse domains, in- cluding object recognition (Yue et al., 2024), sensorimotor trajectory prediction (Radosavovic et al., 2024), autonomous driving (Wu al., 2024; Jia et al., 2024), and code-related
2

---

On the Generalization Ability of
Pretraining
Table Generalization bounds for transformer-based models in different pre-training scenarios. (N: the number of token sequenees, m. the maximum sequence length, I': the number of prompts. L: the number of transformer layers, d: the model dimension of transformer, 12Ld: the number of transformer model parameters. C: the constant bigger than 1).
Ref.
Scenario
Technique
Bound
c(V
CoLL (n(Nmd)
Edelman et al. (2022)
seq2seq Pretraining
Rademacher Complexity
Li et al. (2023)
ICL Pretraining
Stability
C'I VNT dd In(1+NTC)
Zhang et al. (2023)
ICL Pretraining
Operator Approximation
VNT LimC)o(M) Ind
Deng et al. (2024)
MAE Pretraining
Rademacher Complexity
Ours
NTP Pretraining
Rademacher Complexity
o(v
@l((I+Nmm))
Nm
tasks (Izadi et al., 2022; Kim et al., 2021; Qi et al., 2024). While vision applications remain less explored, Kilian et al. (2024) demonstrated that NTP excels in prompt adherence and throughput efficiency for image synthesis, though dif- fusion models achieve superior image quality and lower latency. Extensions to standard NTP include multi-token prediction (Qi et al., 2020; Gloeckle et al., 2024) and Diffu- sion Forcing (Chen et al., 2024), a hybrid training paradigm combining NTP with diffusion for sequence generation.
layer muti-heed-attenn models based on the stability of SGD Lei & Ying (2020); Zhang et al. (2024b). Recently, theoretical understandings have been provided for the gen- eralization ability of In-context Learning (ICL.. The ICL. pre-training is investigated theoretically from the aspects of multi-task learning (Li et al., 2023) and Markov pro- cesses (Zhang et al., 2023), respectively. Notably, Lotfi et al. (2023) derived the first non-vacuous generalization bounds for pre-trained LLMs. Later, Lotfi et al. (2024) in- troduced martingale- bound that captures token-level dependencies.
Theoretical insights reveal NTP's foundational capabilities: Malach (2023) proved autoregressive NTP can approximate complex functions using a novel length-complexity mea- sure. Thrampoulidis (2024) identified an implicit bias to- ward structured solutions in gradient-based NTP optimiza- tion at low training loss. Flemings et al. (2024) proposed PMixED, differentially private protocol for L..--based NTP. Madden et al. (2024) established memory capacity bounds for decoder-only transformers in NTP. Li et al. (2024) showed self-attention learns token-retrieval automata via token-priority graphs.
Table I highlights the key differences of our theoretical result by comparing it with the most related progresses in (Edelman et al., 2022; Li et al., 2023; Zhang et al., 2023; Deng et al., 2024).
Notation. Throughout our paper, we denote set {1,... _n as [n]. And for matrix W, ||W]le, := maxi, W.,..
3. Preliminaries
Generalization theory for pre-training and transformer- based models. Generalization characterizations of pre- training have been stated for many learning paradigms, such as curriculum learning (Zhou al., 2022), transfer learn- ing (Tripuraneni et al., 2020; Xu & Tewari, 2021; Lotf et al., 2022), reinforcement learning (RL) (Ye et al., 2023; Lin et al., 2023), etc. Moreover, Zhang et al. (2024a) con- structed the generalization theory for supervised pre-training and fine-tuning to explore the trade-off between intra-class and inter-class diversity in pre-training datasets. Deng al. (2024) developed a generalization bound for the unsuper- vised pre-training of Masked Autoencoder (MAF), their results are mainly based on the covering number theory of the transformer-encoder models, which was established by Edelman et al. (2022).
This section introduces the framework of NTP pre-training and defines the architecture of DOMs.
3.1. Problem Setting
Consider a set of tokens T whose vocabulary size is ny [T]. Given a pre-training dataset D = { S X. where X, denotes the i-th token sequence, X' is an instance domain such as sentences. We assume there exists an known distribution D that (X, J
D and all the se- quences are independent of each other. Each sequence X, is composed of m tokens (ti. (ti.... t S T, where tj E R. denotes the j-th token of i-th sequence, and m denotes the maximum input length of a language model L.M. Note that the token sequences we consider here have all undergone series of preprocessing operations, such as cropping, mask- ing, and patching, so all sequences have the same length. We denote T = (t), t ,t/_1] as the context of tj, where
For the generalization of transformer- models, Deora al. (2023) derived generalization bounds for the single-
3

---

On the Generalization Ability of Next-Neoknn-PPPe Preeeeooken- Pretraining
i tg E T denotes the given begin sign <| im_start |>>
3.2. Decoder-only Models
for all sequences. Note that To is the empty context for tip-
For a given token sequence [t,,- tm] E Ruxx, we denote Z [to, t,,... .tm-i] Rmx*, the in- put matrix, which contains all the context information. We consider the L-layer and H-head decoder-only trans- former model as the R-L, which mainly consists of one Embedddigg-lyyer and L layer mennsformer-dcoder---e (see Figure (b)). We use to denote the model dimension, d d H denotes the attention dimension, and dj = 4d denotes the feed-forward dimension throughout the paper.
For NTP, we abstract the model LM : X x + T as an algorithm that maps the con- text - to a function LM (T ) similar idea can found in Li et (2023). Then, we imput the token t,-1 to the above function, which will get a response:
Sext-Token-Preiceee
t, LM (T).MTTT..tJ--
we hope that i can be as close to 1 as possible. More de- tails about the next-token-pres token-predictoon can be found in Figure Note that the model LM belongs decoder-only model (DOM), which is usually composed of a Representation- Learner (R-L) h E H S (T + I) and (T-P) E 9 C (I + Th. where I denotes hidden repre- sentation space. We can represent the model LM via
Embedding, Token vectors are one-hot vectors, which are in discrete form. We need to convert the discrete vectors into continuous vectors first through the Embedding operation:
z Embeddingg(2 ( = ZW + W p
where Z denotes the embedded token sequence of Z. We E R=. denotes the token- eoken-embeding matrix, and W, p E Rmxdd denotes the position-embeding matrix.
LM (T) - 9 (h
Denote z] (T) t2) as the j-th training sample of i-th sequence, where T3 (T;-i://-11 Then the empirical risk based on NTP with the i-th sequence can be defined as
Trans-omee-deeoder-bbkk.. Let Im denote the Layer- normlization operator, and a denote the non-linear acti- vation function ReLU. Denote TFw the 1-th layer cunsformer-decoderr with parameter set
Êx, h) =
Σ (gg (())(2)))))) _E(m)
W = -(WII W (Wo, W W W WK W | 1=1)
KA
V.
where W Fi E Rdxdd, w F2 E Rddxd, w O E Raxd W Q. Radxdu, W K,,
where (ggoh()), X' X + IR represents the pre-training loss function, usually cross-entropy loss, and
Rdxd, W V. E B and Z' TF we (Z' (1 2 1) denotes the output of 1-th layer block, which can be formulated by
{(g ((ooh(z))) h(z)). = (((((T)-
denotes the loss of sample 2). Pre-training based on NTP is to train each token sequence in the dataset D according to formula (1), further obtaining the optimal R-L and T-P. We can denote the objective function based on the empirical risk minimization (ERM) as
TF TF Wt
1Z' IIgrr (o W + )
(4)
Y
Σ A... on + +
hE(m]
min ED (go = E Lx, h). gEG.NEH
Here A denotes the masked self-attention of the h-th head.
(2)
RE[N]
Masked Self-attention. Denote softmax as the row-wise softmax operator, Q Z W Q K,
Let Lo, (go h) E[ (g o h]] denote the population risk of Lx. (g h), and
Z W K2
Z z V/ we have
(K M +
Lo h) BBcoh)==[((((C (g
Co, hg) VE[N]
Σ
A softmax
V
(5)
Vdl
where M E gm is a mask matrix defined as
be the population risk of ED (g o fi). Then, the excess risk for NTP pre-training task can be represented as
0,
jši
Mij
-00
j>i
(p(g.h) h)) = (
min SEG
CD (goh), (3)
Then, the R-L can be mathematically formulated by
where ğ E & and E H denote the optimal R-L and T-P we learned by solving (2) respectively.
h(Z) = -TFw (...TFw (Embedding(Z)))
(6)

---

On the Generalization Ability of TeeeeTooken-roiiccioo Pretraining
The T-P is composed of linear projection and softmax:
4.1. Rademacher Complexity Upper Bounds
9 (h (Z)) softmax (h (Z) W W E Rddx.. (7)
To mitigate the excess risk defined in Equation (3), we in- troduce a metric for assessing the complexity of a function class, known as Rademacher complexity (Mohri & Ros- tamizadeh, 2008).
4. Main Results
Note that the tokens in one sequence are dependent, which we usually call a non-i..d. process. We introduce the con- cept of g--mixing processes to characterize the dependency relationship between tokens.
Definition 4.6 (Rademacher complexity). Given a sample set = fz1, (2,,.... 2n S Z and function class F - I. the empirical Rademacher complexity of F is defined as
Definition 4.1. Let T (ty) be a stationary process (Hirschfeld, 1935). T is said to be exponentially sp-mixing (Dobrushin, 1956) if there exist some constants yo > 0, P1 > 0 and r > 0 such that the ş-mixxng coefficient
Rg (F) == Eg sup fEF eE[m]
E axf (x)
(8)
where E = (ei,..., En are i.i.d. Rademacher random vari- ables satisfying P(s; = 1) = P(s, = -1) 0.5, i E [m].
(A:) == sup sup AEo
|Pr[A| B] - Pr[A]]
Due to the dependence of tokens within a sequence, and the independence of distinct sequences, it is essential to establish two separate measures of Rademacher complexity. By setting S = D and J = é o G a H. in Equation (8), we can define the empirical Rademacher complexity of the composite function class G H for as
BEO
S pp exp (-ppk") Vk N"
where of denotes the o-algebra generated by the random variables 1,,". ty.
Based on the above definition, we make the following assumption on dataset D.
RD (Lo :- E2
sup LgE
1 N
Σ SE[N]
ExLx, (g (1
Assumption 4.2. Assume that X; = (t tm is gen- crated by a gp-mixing distribution o, for all i, and there exists an unknown distribution U such that U = ) U.
For ease of representation, denote Ej - e (g o h(zz) ((22))22)) 2)). Referring to the definition of Rademacher complexity of multi-task learning in Wang et al. (2022b), we can also consider all token sequences, and define the following multi- sequence Rademacher complexity:
Remark 4.3. The above assumption is widely adopted in the study of non-i.i.d. processes such as Ralaivola et al. (2010); Heinrich & Pawlas (2013); Vankadara et al. (2022): Liu et al. (2025b). In Definition 4.1, lim (>:) - O means that and B will become independent as & increases. When A and B represent two different sentences, the farther the distance between A and is (coming from two different articles), the smaller the correlation between A and B will be. Therefore, Assumption 4.2 is reasonable.
RD o H) = E sup LeggheN Nm SE[N]
EΣ
Remark 4.7. The Rademacher complexity defined here closely resembles the "representation-induee Rademacher complexity" delineated Deng et al. (2024). However, a notable distinction exists: the innermost function in their composite function fixed, whereas in our definition, encompasses the entire function class H.
Assumption 4.4. There exists a constant By E R+ satisfy- ing E(t, t)] < Bg for any f, t E T, and is G.-Lipschitz w.r.t. t.
Assumption 4.4 is commonly used in learning theory (Bartlett & Mendelson, 2002; Shalev-Shwand & Ben-David, 2014; Liu et al., 2022; Deng et al., 2024; Liu et al., 2025a). Definition 4.5 (Discrepancy measure). Given the set of distributions U (dx)) we define its discrepancy as
We primarily focus on the performance of R-L because applies to various downstream scenarios after pre-training, whereas T-P changes downstream tasks evolve. Conse- quently, we propose decomposing the Rademacher complex- ity of g o H into the complexities of the individual function classes G and H. This approach allows for a more precise analysis of the influence exerted by H. defined as follows: Proposition 4.8 (Rademacher complexity decomposition). Let F. Z + R be composite function satisfying J fog o H. where e is a loss function and H. G are function classes. Given a sample set S = {Z1,... .... kn} C for any g g 9 satisfying Gy-Lipschitz w.r.t. h E H and E satisfying Ge-Lipschitz w.r. g o h E G H. we have
disc(U) = sup kE[N]
Σ l6 &2 I
N
KE[N]
where ||0; d& |Ivv = sup
(t) - (x)l denotes total tET variation distance between two distributions.
As shown in (Kuznetsov& & Mohri, 2020; Wang et al., 2022a), the closer the distributions in set U are, the smaller dUsc(U')
will be. In particular, dUsc(U) = 0 when Φ1 = .... NN-
Rg (Eo 1oogo o H) S &GGG (H) + Gess(g i))
5

---

On the Generalization Ability of Next- Teeeeeeennn Pretraining
where i is any given funcciin in H.
4.2. Capacity of Transformer-deccdder Models
Theorem 4.9. Given pre-training dataset D containing N token sequences {X:) ==1 C X, satisfying the distribution conditions in Assumption 4.2. Denote & and h as the optimal
To investigate the effect of the parameters within the transformer-decoder model (i.e., R-L) the generaliza- tion performance of NTP, use the covering number to quantify the Rademacher complexity of R-L. We begin by providing a general definition of the covering number.
R-L and T-P derived by solving Equation (2), respectively. Then, under Assumption for some so > O, ip > 0 and
r > O, there holds
Definition 4.13 (c-cover and covering number). Denote (U, || -1) as a metric space and V C U. For any e > 0, V called an e-cover of U if for any u E U, there exists v E V such that ||u - vll S e. The covering number of (U, || [l) is the cardinality of the smallest c-cover, which is defined by
8ln Ep(g. h1) & GAD (lo + B
N
Am I log 2
Be
+4B, dUsc(U),
N(U, e, I| I)) = min(V] V is the e-cover of U}.
2m
Assumption 4.14. Assume that
with probability at least 6, where || | oo I 1 2 2 (kk) and (kk) S yo exp k-)) Vk E [m]]
Tneen is G--Lipschitz with the n-rno i.e., Vt,, t2 E Rdd ||[ooom (tg) - Inoem (tz)llea I Gz - tyllez-
The proofs of Proposition 4.8 and Theorem 4.9 are provided in Appendix and Appendix B, respectively. Remark 4.10. Item I represents the generalization error of NTP pre-training on the dataset D, reflecting the model's ability generalize unseen token sequences and its over- all generalization capability within the sequence space. Item II denotes the average generalization capability on individ- ual token sequences X,, indicating the model's local gen- eralization ability within the token space. The last item, disc(U) (see Definition 4.5), reflects the influence of the quality of the pre-training dataset. Since yo, (fi, and are all greater than 0, the positive series Σk= (pk)) is conver- gent. Therefore, there exists a constant Cg.. > such that || 11 < C.. For simplicity, we will use the constant Cppr to represent the upper bound of |m 1l in the subsequent amalysis.
VI E [L] and h E [H]] there exists constants Cj such that | (K)) //deell.. SG.
AI E [L] W E WW, there exists constants B, satisfy- ing W | S Bj.
The second assumption in Assumption 4.14 is reasonable due to the presence of the scaling factor ddg in the self- attention mechanism. The first and third assumptions have been previously used in the analysis of the transformer cov- ering number (Edelman et al., 2022; Deng et al., 2024).
Since the learnable parameters of the Transformer model are all fully connected layer parameters, we introduce the following lemma proposed by Lin & Zhang (2019):
Lemma 4.15. Let X E Raxde. be a given input matrix with a bounded Frobenius norm, and W E Red Xdout such that [| I a. Then, we have
The following corollary can be derived by combining Theo- rem 4.9 and Proposition 4.8.
( ||W]FF aa) (0,
Corollary 4.11. Under the same assumptions as Theorem 4.9, if g is Gy-Läpschitz w.r.t. h for any g E G H. there exists constant Cip.r >> 0 such that the following inequality holds with probability at least 1 - 8:
Sdindout In (1+ 2a 2a||X||]
6
Ep S (M, + (G
Lemma 4.15 emphasizes the impact of model parameters on the covering number, aligning with our research objectives. Based this lemma, provide the upper bound of the logarithmic covering number for masked self-attention as follows:
(I)
8l N
Cer log
+ B&
+ B
+ 4B, disc(U..
2m
Lemma 4.16 (Simplification of Lemma C.10). Given an input sequence S = {Z!...., Zun} E Rmxdd denote Z = Zi...., ZN] E R Nmxd as the concatenated data matrix. Consider the masked self-attention head A (-) (ig- nore the layer and head indices) defined in Equation (5), the corresponding function class can be defined as:
Remark 4.12. Item (I) is exclusively associated with the complexity of R-L, while item (II) depends solely the complexity of T-P. These two items operate independently. allowing for separate analysis of the effects of R-L and T- on generalization performance. This independence also simplifies the process of replacing T-P. as it only requires redefining item (IT).
Hz. == (z A(Z) W WK, Wvllr S B).
6

---

On the Generalization Ability of Next-Token-Predictie Pretraining
Then, we have
where O R 12Ld is the number of model parameters and
IN (H&)e,
e,ll Il (| S dd& In (1+ 1
|| 12
SL
II + ( BJH + 1). LE(L]
+
vd&e
1+ BBBV N lmm N UNillF
+ df In
Remark 4.21. As demonstrated in Bartlett et al. (2017), the logarithm of the covering number of H under the infinite- norn is bounded above by that under the Probenius-nome i.e., In (H,, e, !le_.) < In WnH,,,, e, || ||r). However, our approach bounds the Frobenius- norm covering number with a smaller order of (L) compared to the orders CO(L) (where > 1) reported in Edelman et al. (2022); Deng et al. (2024). This indicates that our method offers significant advantage over the previous studies.
where ||
[ [Z,|F.
Then, based on Lemma 16, we can further obtain the upper bound of the logarithmic covering number for a single-layer menfforner-decode--bbbeeee
as follows:
Proposition 4.17. For the cenffommer-dccoder--beeee TFw (ignoring the layer indices) defined Equation (4), the corresponding function class can be defined as
4.3. Generalization Bounds for DOMs
H3S = 12 | S B,W Then we can get the following covering number bound:
Inspired by Bartlett et al. (2017), we use the covering num- ber to deduce the upper bound of Rademacher complexity, Then, the excess risk for NTP pre-training can be bounded by integrating Corollary 4.11 and Theorem 4.20.
hN .ell (I)) S 4d + 3) In (1+
Theorem 4.22. Let [N) E R NNMn, be the input se- quences generated from dataset D. Denote 9 and h as the optimal R-L and T-P learned from Equation (2), respectively. Then, under Assumptions 4.2, 4.4 and 4. 14, there exists constant C.. > 0 such that the following inequality holds with probability at least 1 - 8:
where w =
B'H (Mlle
The following two lemmas mainly explore the Lipschitz continuity of the transformer decoder model.
Lemma 4.18. For a single transfomer- TFw (-) parameterized by W (ignore the layer indices),
let Z, Z E Remxd be any input matrixes. Then, there holds
Ep(f, 50
@dHmm (Vottm Nm
+ Gey
dn Nm
|IFw (Z) TFW (2) (2)lE
(
In 4
Co.r log a
+ Br
+ V
44diso(U))
(B2 +1) (6 (BCBBHH
B² Hmd 11) +1) --2 --
N
2m
where O is the number of model parameters, and TI In (1+PLSL), with PL = -1 B3 and constant s defined in Theorem 4.20.
Lemma 4.19. Let Z' E mRa as the output matrix of the 1-th layer sicoder-block, we have:
z'
(B3 +1) (c (3 |1 JER]
II
Remark 4.23. We focus on three parameters: the num- ber of token sequences N, the maximum length of the token sequence m, and the number of model parame- ters O. Our bound is O( OO/Om + /Cp.r/mi, where ((ve) O/Nm) reflects the generalization ability between token sequences, and O(v/Cp./m reflects the general- ization capacity among tokens within a sequence. Here, C.p, is constant related to the e--mixing coefficient, indi- cating the distribution quality of a single token sequence, while disc(U) reflects the overall distribution quality of all token sequences. Our bound captures the impact of both dataset quality and individual sample quality on general- ization performance. Unlike previous works (see Table I), we show that effective generalization requires both larger N and a larger m, enabling the model to generalize across both sequence space and token space. Additionally, as θ in- creases, the total number of tokens Nm should also increase to achieve better generalization.
Based Proposition 4.17 and Lemmas 4.18 and 4.19 (proved in Appendix C.3), we obtain the following loga- rithmic covering number upper bound for the R-I..
Theorem 4.20. Let D = N be dataset contain- ing N token sequences and let Z = IZ....,,Zy, IR Nmxu. be the input matrix generated from D, and denote []] E mmNd as the embedded matrix. The function class of the R-L defined in Equation (6) can be defined as
H. = (z + h(Z): || - Bi. w' E WWW W' [4]]
Then, under Assumption 14, we have
0H-
LBB (xy1l1))
In (H, e1 -1p)) <
In 1+
L
7

---

On the Generalization Ability of Next- Next-Tooen-rreiiction Pretraining
8
(a)
(b)
(C)
Figure Experiments on MiniMind and DAMO.NL datasets.
Remark 4.24. It should be noted that our bounds remain valid even when modifications are made to the structure of the transformer-decoder block. For instance, replacing post-layer Normalization with pre-layer Normalization or substituting the ReLU activation function with SiLU. This attributable to the fact that such modifications exclusively affect the logarithmic terms PL and SL.
8x NVIDIA A800-8068 8000800B GPUs using DeepSpeed-Zenne (Rajbhandari et al., 2020). For optimization, we utilized the A (Loshchilov & Hutter, 2017) optimizer, combined with a cosine learning rate scheduler that includes 20-step warm-up phase during the initial training stage.
Full specifications for model architecture, dataset prepro- cessing, and training configurations are detailed Table
5. Experiments
5.2. Main Results
To validate the theoretical contribution of this paper, specifi- cally, Theorem 4.22, we performed a set of NTP pre-training experiments in DOMs. These experiments were designed to systematically examine the influence of model parameters and sample size on generalization performance.
Maximum sequence length m. As demonstrated in Fig- ure 2(a), we performed experiments with varying maximum sequence lengths m (64, 128, 256, 512) the training dataset while holding all other parameters constant. No- tably, the test dataset retained fixed maximum sequence length of 512 across all evaluations. While models trained shorter sequences converge more rapidly, they exhibit limited generalization capability when applied to longer sequences. This observation accounts for the monotoni- cally increasing test loss trend observed for miniNTP-1, and 3 as the training sequence length decreases. As shown in Table 2, models trained shorter sequences deliver strong performance quences. Importantly, models trained on longer sequences maintain robust performance even when evaluated on shorter sequences, highlighting their adaptability.
5.1. Setup
Model Architecture. Our architecture largely follows the GPT-2 framework, with two key modifications: (1) the adop- tion of the SiLU activation function and (2) RMSNorm nor- malization (Zhang & Sennrich, 2019). As demonstrated in Remark 4. 4.24, these adjustments do not compromise the validity of our theoretical framework.
test cases with similarly short se-
Datasets. For pretraining, we employ the MiniMind dataset', while our test set consists of 199 samples (with a maximum sequence length of m < 512) carefully selected from the DAMO. NLP dataset Both datasets belong to the category of Chinese generation datasets. Due the con- sistent pretraining corpus, adopted the same tokenizer as MiniMind preserving vocabulary size of n, 6400.
Table 2. Test sample perplexity (PPL) variations across models under variable maximum sequence lengths (m)
Model
m 64 m
128
m
256
512
miniNTP-1
1.10
69.35
1578. 14
316024.25
miniNTP-2
1.24
1.31
224.25
130613.71
Training Protocol. Our training methodology follows the approach outlined MiniMind. To optimize efficiency, we employ FlashAttention (Dao et al., 2022) for accelerated attention computation and conduct distributed training on
miniNTP-3
1.03
1.05
1.08
24343.04
miniNTP-4
1.03
1.16
1.24
1.49
The number of sequences N. For models with identical architectural configurations, we demonstrate enhancement in generalization performance with increasing training se- quence quantity. As illustrated in Figure 2(b), while holding model parameters, training hyperparameters, and maximum sequence length (m = 512) constant, we evaluated perfor-
hnttps://ww.mmmelsspps cn/datasets//
hnttps//www.mddeescoo https//opo...oattttt////aaaaaae
NiP/:csts_test__ee hhttps://github com/ http:///ww.....ggggogoo/miiiiiin
8

---

On the Generalization Ability of Next-Token-Predictie Pretraining
Table Model architcctures, training data specifications, hyperparameter configurations, and PPI. (m 512).
Model
+
L
H
d
N%
Batch Size
Learning Rate
PPL
MiniNTP-1
0.029B
8
8
512
64
100
0.5M
5.0e-4
316024.25
miniNTP-2
0.029B
8
8
512
128
100
0.5M
5.0c-4
130613.71
MiniNTP-3
0.029B
8
512
256
100
0.5M
5.0e-4
24343.04
miniNTP-4
0.029B
8
512
512
100
0.5M
5.0e-4
1.49
miniNTP-5
0.029B
8
512
512
50
0.5M
5.0e-4
3.17
miniNTP-6
0.029B
512
512
75
0.5M
5.0e-4
1.95
miniNTP-7
0.002B
128
512
100
0.5M
1.0e-3
5.76
miniNTP-8
0.09B
768
512
100
0.5M
6.0e-4
1.13
miniNTP-9
0.31B
1024
512
100
0.5M
3.0e-4
1.05
mance across 50%, 75% and 100% subsets of the complete pretraining dataset. The experimental results reveal that diminishing training data volume not only compromises model generalization but also significantly impairs conver- gence characteristics and training stability.
eralization in the NTP pretraining task. Notably, increas- ing produces more substantial performance gains than expanding N, though this comes with increased training complexity. Additionally, while larger model parameters facilitate learning richer token representations and elevate the model's capacity ceiling, this architectural expansion should be matched with a corresponding increase total token quantity mitigate overfitting risks.
Model size O. Our analysis Figure 2(c) evaluates how model parameter size (+) affects generalization perfor- mance under fixed training sequence count (N) and maxi- mum length (m), with configurations adapted from Bider- man et al. (2023). Early training (step< 100) shows smaller models converge faster, but prolonged training demonstrates larger models achieve superior convergence rates and lower final losses. As predicted by Theorem 4.20, this divergence arises from capacity limits: smaller models saturate earlier while larger ones continue learning from additional tokens.
6. Conclusion
This paper presents a generalization error analysis for next- token prediction pre-training, a widely used paradigm in large language models. Our theoretical results enhance the understanding how model parameters influence general- ization ability. We find that generalization depends on the number of token sequences, the maximum sequence length, and the number of parameters in the transformer model. Em- pirical evaluations confirm our theoretical findings through data experiments.
RollPantess
7. Future Work
L9S
In the rapidly advancing field of large language models, theoretical foundations remain underdeveloped. While our study addresses part of this research gap, we identify sev- eral promising avenues for future work. First, although our e--mixing data modeling approach demonstrates theoreti- cal validity, empirical verification practical applications requires further investigation. Beyond mixing processes, de- veloping language-specific data distributions could provide deeper insights into how linguistic properties affect model behavior. Second, this work primarily uses Rademacher complexity for theoretical analysis, other frameworks like stability based (Liu et al., 2024b; Zhang et al., 2024b) or infformation-therree (Lu & Van Roy, 2019; Livni, 2023) methods are viable alternatives. Finally, given the antic- ipated evolution toward unified multimodal architectures. extending this research to incorporate diverse data modali- ties represents crucial direction for future exploration.
Figure Model generalization ability radar chart.
Figure 3 provides comprehensive visualization comparing the nine models across five distinct evaluation dimensions, with radial distance from the center indicating performance quality in each dimension. The analysis reveals that ex- tending both the maximum sequence length (m) and the number of training sequences (N) improves model gen-

---

On the Generalization Ability of Sext-ooken-Prriiction Pretraining
Acknowledgments
Deng, Y., Hong, J., Zhou, J., and Mahdavi, M. On the generalization ability of unsupervised pretraining. In Artificial Intelligence and Statistics, 2024.
This work is supported by the National Natural Sci- ence Foundation of China (NSFC) (Nos. 62376104 and 12426512) and the Open Research Fund of Engineering Research Center of Intelligent Technology for Agriculture, Ministry of Education (No. ERCITA-KFO02).
Deora, P., Ghaderi, R., Taheri, H., and Thrampoulidis, C. On the optimization and generalization of multi-head attention. arXiv preprint arXiv: 2310. 1310.12680, 2023.
Dobrushin, R. L. Central limit theorem for nonstationary markov chains.i. Theory of Probability & Its Applications, 1(1::72-88, 1956.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Edelman, B. L.,, Goel, S., Kakade, S., and Zhang, C. Induc- tive biases and variable creation in self-attention mecha- nisms. In International Conference on Machine Learning, 2022.
References
Flemings, J., Razaviyayn, M., and Annavaram, M. Differ- entially private next-token prediction of large language models. arXiv preprini arXiv: 2403. 15638, 2024.
J.,. Kiros, R., and Hinton, G.E. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
Bachmann, G. and Nagarajan, V. The pitfalls of next-token prediction. arXiv preprint arXiv: 2403.06963, 2024.
Gloeckle, F., Idrissi, B. Y., Rozière, B., Lapez-Paz, D., and Synnaeve, G. Better faster large language mod- els via multi-token prediction. arXiv preprint arXiv: 2024.
Bartlett, P. L.. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(11):463-482, 2002.
Heinrich, L. and Pawlas, Z. Absolute regularity and brillinger-mxiing of stationary point processes. Lithua- nian Mathematical Journal, 53(3):293 2013.
Bartlett, P. L.,, Foster, J., and Telgarsky, J. Spectrally- normalized margin bounds for neural networks. In Ad- vances in Neural Information Processing Systems, 2017.
Hirschfeld, H. O. A connection between correlation and contingency. In Mathematical Proceedings of the Cam- bridge Philosophical Society, volume 31, pp. 520-524, 1935.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2430. PMLR, 2023.
Izadi, M., Gismondi, R., and Gousios, G. Codefill: Multi- token code completion by jointly learning from structure and naming sequences. In International Conference on Software Engineering, pp. 401-412, 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Jia, X., Shi, S., Chen, Z.,, Jiang, L.., Liao, W., He, T., and Yan, J. Amp: Autoregressive motion prediction revisited with next token prediction for autonomous driving. arXiv preprint arXiv:
2403. 13331, 2024.
Chen, B., Monso, D. M., Du, Y., Simchowitz, M., Tedrake, R., and Sitzmann, V. Diffussin forcing: Next-token pre- diction meets full-sequence diffusion. arXiv preprint arXiv: 2407,01392, 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A.. Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv: 2001 000100866, 2020.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory efficcienn exact attention with io-awareness. Advances in neural information processing systems, 35:16344-166551 2022.
Kilian, M., Jampani, V., and Zettlemoyer, L. Computa- tional tradeoffs in image synthesis: Diffusion, masked- token, and next-token prediction. arXiv preprint arXiv: 2405. 2024.
De Souza Percira Moreira, G., Rabhi, S., Lee, J. M., Ak, R., and Oldridge, E. Transformers4rec: Bridging the gap be- tween nlp and sequential/sessionn based recommendation. In ACM Recommender Systems, 2021.
Kim, S., Zhao, J., Tian, Y., and Chandra, S. Code predic- tion by feeding trees to transformers. In International Conference on Softwar Engineering, 2021.
10

---

On the Generalization Ability of Next-Token- Toreeeeee--roiiiio Pretraining
Kuznetsov, V. and Mohri, M. Discrepancy- theory and algorithms for forecasting non-stationary time series. Annals of Mathematics and Artificial Intelligence, 88(4): 367-399, 2020.
Loshchilov, and Hutter, Decoupled weight decay regu- larization, arXiv preprint arXiv:1711.051011 2017.
Lotfi, S., Finzi, M., Kapoor, S., Potapezynski, A., Goldblum, M., and Wilson, A. G. Pac-bayes compression bounds so tight that they explain generalization. In Advances in Neural Information Processing Systems, 2022.
Lei, Y. and Ying. Y. Fine-grained analysis of stability and generalization for stochastic gradient descent. In Interna- tional Conference on Machine Learning, pp. 5809-5819. PMLR, 2020.
Lotfi, S., Finzi, M., Kuang, Y., Rudner, T. G., Goldblum, M., and Wilson, A. G. Non-vacuous generalization bounds for large language models. arXiv preprint arXiv: 2312.17175. 2023.
Levin, D. and Peres, Markov chains and mixing times, volume 107. Americun Mathematical Soc., 2017.
Li, Y., Ildiz, M. E., Papailiopoulos, D., and Oymak, S. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pp. 19565-19594, 2023.
Lotfi, S., Kuang, Y., Finzi, M., Amos, B., Goldblum, M., and Wilson, A. G. Unlocking tokens as data points for generalization bounds on larger language models. Ad- vances Neural Information Processing Systems, 37: 9229-9256, 2024.
Li, Y., Huang, Y., Ildiz, M. E., Rawat, A. S., and Oymak, S. Mechanics of next token prediction with self-attention. In Artificial Intelligence and Statistics, 2024.
Lu, X. and Van Roy, B. Information-theoreti confidence bounds for reinforcement learning. Advances Neural Information Processing Systems, 32, 2019.
Lin, L., Bai, Y., and Mei, Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. arXiv preprint arXiv: 2310.08566, 2023.
Madden, L., Fox, C.,, and Thrampoulidis, C. Upper and lower memory capacity bounds of transformers for next- token prediction. arXiv preprint arXiv: 2405.13718. 2024.
Lin, S. and Zhang, J. Generalization bounds for convolu- tional neural networks. arXiv preprint arXiv: 1910.01487, 2019.
Malach, E. Auto-regressive next-token predictors are uni- versal learners. arXiv preprint arXiv: 2309.06979, 2023.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arX1v:2412.19437, 2024a.
Masuda, H. Ergodicity and exponential 8-mixing bounds for multidimensional diffusions with jumps. Stochastic processes and their applications, 117(1):35-56 2007.
Liu, L., Song, B., Pan, Z., Yang, C., Xiao, C., and Li, W. Gradient learning under tilted empirical risk minimiza- tion. Entropy, 24(7):956, 2022.
McDonald, D. J., Shalizi, C. R., and Schervish, M. Estimat- ing beta-mixing coefficients via histograms. Electronic Journal of Statistics, 9(2), 2015.
Liu, L.,, Chen, H., Xiao, C., and Li, W. The consistency analysis of gradient learning under independent covariate shift. Neurocomputing, 635: 1:5:1290 19883,, 2025a.
Mohri, M. and Rostamizadeh, A. Rademacher complexity bounds for non-iid processes. In Advances in Neural Information Processing Systems, 2008
Liu, L., Chen, Y., Li, W., Wang, Y., Gu, B., Zheng, F., and Chen, Generalization bounds of deep neural networks with r -mixing samples. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15, 2025b. doi:
Mohri, M. and Rostamizadeh, A. Stability bounds for sta- tionary ---mixing and ß-mixing processes. Journal of Machine Learning Research, 11(2), 2010.
10.1 1.11099/NNS..0 2.0.NNNN...222552222555
Moon, J.. Park, G., and Jeong, J. Pop-on: Prediction of process using one-way language model based on nip ap- proach. Applied Sciences, 11(2):864, 2021.
Liu, X., Zhang, H., Gu, B., and Chen, H. General stability analysis for zeroth-order optimization algorithms. In The Twelfth International Conference on Learning Represen- tations. 2024b.
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng. N., Grangier, D., and Auli, M. fairseq: A fast, extensi- ble toolkit for sequence modeling. In North American Chapter of the Association for Computational Linguistics, 2019.
Livni, R. Information theoretic lower bounds for informa- tion theoretic upper bounds. Advances in Neural Infor- mation Processing Systems, 36:37716-37722 2023.

---

On the Generalization Ability of Next-Tokee-Predictton Pretraining
Qi, M., Huang, Y., Yao, Y., Wang, M., Gu, B., and Sun- daresan, N. Is next token prediction sufficient for gpt? exploration code logic comprehension. arXiv preprint arXiv: 2404.08885, 2024.
Wang, R., Walters, R., and Yu, R. Data augmentation vs. equivariant networks: A theory of generalization dy- namics forecasting. arXiv preprint arXiv: 2022a.
Qi. W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., and Zhou, M. Prophetnet: Predicting fu- ture n-gram for sqquence-oo-sqquee pre-training. arXiv preprint arXiv: 2001.04063 2020.
Wang, R., Walters, R., and Yu, R. Meta-learning dynamics forecasting using task inference. In Advances in Neural Information Processing Systems, 2022b.
Wong, K. C., Li, Z., and Tewari, A. Lasso guarantees for B- mixing heavy-tailed time series. The Annals of Statistics, 48(2): 124-1142, 2020.
Radosavovic, I., Zhang, B., Shi, B., Rajasegaran, J., Kamat, S., Darrell, T., Sreenath, K., and Malik, J. Humanoid locomotion as next token prediction. arXiv preprint arXiv: 1400.194699 2024.
Workshop, B., Scao, T. L., Fan, A., Akiki, C., Pavlick. E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F,, Gallé, M., Tow, J., and M., A. Bloom: A 176b-pan cooob-prrmmter open-access multilingual language model. arXiv preprint arXiv: 2211.05100, 2023.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1- IERE, 2020.
Wu, W., Feng, X., Gao, Z., and Kan, Y. Smart: Scalable multi-agent real-time simulation via next-token predic- tion. arXiv preprint arXiv: 2405. 15677, 2024.
Ralaivola, I.., Szafranski, M., and Stempfel, G. Chromatic pac-bayes bounds for non-iid data: Applications to rank- ing and stationary 3-mixing processes. The Journal of Machine Learning Research, 11:1927-1956, 2010.
Xu, Z. and Tewari, A. Representation learning beyond linear prediction functions. In Advances Neural Information Processing Systems, 2021.
Shalev-Shwartz, S. and Ben-David, S. Understanding ma- chine learning: From theory to algorithms. Cambridge university press, 2014.
Yang, A., Li, A., Yang, B.. Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lvv, C.,, el. Qwen3 technical report. arXiv preprint arXiv:2505.09 09388, 2025.
Shlegeris, B., Roger, F., Chan, L., and McLean, E. Language models are better than humans at next-token prediction. arXiv preprint arXiv: 2212. 2022.
Ye, H., Chen, X., Wang. L., and Du, S. S. On the power of pre-training for generalization in rl: provable benefits and hardness. In International Conference on Machine Learning, 2023.
Thrampoulidis, C. Implicit bias of next-token prediction. arXiv preprint arXiv: 2402. 1002.1855). 18551, 2024.
Yue, K., Chen, B.-C., Geiping, J., Li, H., Goldstein, T., and Lim, S.-N. Object recognition as next token prediction. In Conference on Computer Vision and Pattern, 2024.
Touvron, H., Lavril. T., Izacard, G.. Martinet, X., Lachaux, M.-A., Lacroix, T., Roziére, B., Goyal, N., Hambro, E., Azhar, F, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2302.13971, 2023.
Zhang, B. and Sennrich, R. Root mean square layer nor- malization. Advances in Neural Information Processing Systems, 32, 2019.
Zhang, J., Wang. B., Hu, Z., Koh, P. W. W., and Ratner, A. J. On the trade-off of intra-/inter-class diversity for super- vised pre-training. In Advances in Neural Information Processing Systems, 2024a.
Tripuraneni, N., Jordan, M., and Jin, C. On the theory of transfer learning: The importance of task diversity. In Advances in Neural Information Processing Systems, 2020.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv: 2205.01068, 2022.
Vankadara, I. C., Faller, P. M., Hardt, M., Minories, L.., Ghoshdastidar, D., and Janzing, D. Causal forecasting: generalization bounds for autoregressive models. In Un- certainty in Artificial Intelligence, pp. 2002-2012. 2022.
Zhang, X., Chen, H., Gu, B., Gong, T., and Zheng, F. Fine-graines analysis of stability and generalization for stochastic bilevel optimization. In Proceedings of the Thirty-Thine International Joint Conference on Artificial Intelligence, pp. 5508-55 5516, 2024b.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E.., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, 2017.
12

---

On the Generalization Ability of Next-Token-Predictie Pretraining
Zhang, Y., Zhang, F., Yang, Z., and Wang, Z. What and how does in-context learning learn? bayesian model averag- ing, parameterization, and generalization. arXiv preprint arXiv: 2305.19420, 2023.
Zhao, W. X., Zhou, K., Li, Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv: 2803. 2024.
Zhou, D., Zheng, L., Fu, D., Han, J., and He, J. Mentorgnn: Deriving curriculum for pre-training gnns. In Conference on Information and Knowledge Management, 2022.
13

---

On the Generalization Ability of Text-iookenrrraiittoo Pretraining
Appendix
Summary & Technical Route
This paper conducts comprchensive theoretical analysis by focusing the pre-training methodology of Large Language Models (LLMs) known Sext-Token-Prediciie (NTP). It categorizes LLMs as transformer decoder-only models (DOMs) and delves into the empirical successes of NTP despite a lack of theoretical understanding. This work also establishes theoretical framework to analyze the generalization behavior of NTP pre-training. We introduce a novel decomposition of Rademacher complexity to study the representation-lcareer and token-predictor components of DOMs. The paper also addresses the dependence between tokens using se-mixing, a tool commonly used non-independent scenarios, to delineate inter-token dependencies. This approach allows for fne-grained analysis of the generalization ability of NTP pre-training, considering the model's structure and the nature of the training data.
The technical route of the paper involves developing theoretical framework for NTP from statistical learning perspective. This work proposes decomposition law for Rademacher complexity to bound the excess risk of NTP and establish different bounds on the generalization capability. We refine the estimation of the covering number for multi-layer multi-head transformer-decodder models, pioneering the incorporation of the mask matrix within the self-altention mechanism under the Frobenius norm. This paper uses the covering number to derive the corresponding Rademacher complexity upper bound, extending the theory of (Bartlett et al., 2017) and (Lin & Zhang, 2019) to establish fine- tnneggaiiedd generalization bounds for DOMs-based NTP pre-training. The results expressed in terms of key parameters that affect the generalization ability, providing clear and quantifiable understanding of how NTP pre-training behaves in practice.
Outline of the Appendix
The appendix is mainly structured follows,
Section A: Proof of the Proposition 4.8.
Section B: Proof of the Theorem 4.9.
Section C: Capacity of DOMs.
Section C.1: Introduction to the model architecture.
Section C.2: Restatement to some useful lemmas.
Section C.3: Proof the Proposition C.11.
Scction C.4: Proof of the Theorem 4.20.
Section D: Proof of the Theorem 4.22.
14

---

On the Generalization Ability of Next- Prreennnn-rriiiiio Pretraining
A. Proof of the Proposition 4.8 Proof.
A Ee
sup
Σ (z)
fer n ic(n]
(a) 2 GgEa
sup DEG. 71
E (z))
rE(m)
S R.
sup DE 7 iE[n]
8
-0
+ +G
sup Σ
o 9 o (zx)
gEt n
:E|m]
(6) B.
1
((x) - - -
(G
sup heH TI
8
KE[N]
Be
Σ sup NEW n
1
I1 (z)il
+ G.G.Be
Σ c h iE|ml
( + (G
sEin]
(
Rs(H) + G h),
where h is a any given function in H. Here (a) is by Ledoux- -aalorrnd contraction inequality, (6) uses the Lipschitz conditions of g. and (c) uses the property that the Rademacher random variables are i.i.d. with zero mean.
B. Proof of Theorem 4.9
Firstly, we give two necessary assumptions which have been mentioned before. Assumption B.1. Assume that X, = [2]. z E R= is generated by ---mixing distribution di, and there exists an unknown distribution U such that U = {or ~ U. Assumption B.2. Assume there exists a constant Bg E IR+ satisfying [é(t, t) S Be for any i, t E T, and is Ge-Lipschitz w.r.t. É.
Then, introduce some related lemmas which will be used in our proof. Since (X,) NN independent of each other, {Ộx, (g o ) are also independent each other, so the generalization error based on the dataset can be bounded by the following common theorem.
Lemma B.3 Shalev-Shwart. & Ben-David (2014)). Given dataset D = (X D, if loss function é T x T +
[- B,, B,), Bg E RR,, then with probability at least - 5, the following inequality holds for any h E H and E G:
2 N
Colgoh) < Lp(goh) ( + +B
For the -on-independent case, Mohri & Rostamizadeh (2010) gave Rademacher complexity bound under the c--mxing distribution. Therefore, under Assumption B.1, define the generalization error based on single token sequence X: mainly using the following theorem:
Lemma B.4 (Mohri& & Rostamizadeh (2010)). Given token sequence X; -tii,... tim and loss function e: T x T + [-Be, Bel, Be E R if (t)) follow sp-mxing distribution 6ú then for some > O, f1 > 0 and r > 0, with
probability at least - 5, the following inequality holds for any E H and E G:
A 1 log
Co. (g h) - Lx. & BBy
2m
where | loc S1+ 21+2Em (kk), and (p(k) S exp (-pikk) for all & E [m].
15

---

On the Generalization Ability of Next-Token-Prediition Pretraining
Lemma B.S. For the two Rademacher complexity over the dalaset D mentioned before, have the following inequality: RD (6 H) 3RD (1 o H).
Proof. Let e' (E (())) E',E" are independent of each other. For case of representation. we denote lj l o h(zz)) 21)) Then, we have:
((e D be three i.i.d. Rademacher random variable collections, and
AD (lo E.
sup gEG,NE R. N
E 6
E e (g0o(12))
iE[M]
jE[mL
ΣΣ
(4E) =jj +Ejet)
Eg'
sup
NEGHNEH
Nm JElm!
Σ ((4 =ÿ fj +
E
E
sup
BEGHNEH Nm
jE[m)
< Eg"
Eg'
sup SEGHEEH
Σ E (c eg
+ EE
Eg'
sup nEG.NEN
Σ iE[N]
Σ
Nm
Nm
VE(N] JElm)
J
(1I)
For part (I), denote é (1 1m 1 where Esg [c) (E/-EE) -Eÿ) the distribution is:
It's easy to get are i.i.d. random variables, and
1/4, Eij
p (E1g)
1/2, Eij 0
1/4, Eig = -1
Then, by the independence of e' and e", we have:
(I) 2E 2E "
E
sup
1
Σ
Σ
c {5) ej
DEG,NEH
Nm
2
JE|m]
2E
sup ΣΣ
EijlE
gEG.NEH
Nm jE((m
< 2E
sup 9ER Nm
Σ
Σ
Eg
jEIM]
For part (II), we denote ē = {fEss 1. 1 where Eij- = Elee.
"
It's easy to get are i.i.d. Rademacher random variables.
We have:
(TT) E
E.""
sup DEGHNEH
Σ
Nm (E[N] JE(m)
Eg'
Bd"'
sup DEGHNEH
Σ
Σ
Nm TE[N] JE[m]
E
sup
Σ
ExplY
gE9,NEN
Nm TE[N) jE[m]
Σ Σ
Eg
sup 9EG,AEN
Nm (E[N) JE[m)
16

---

On the Generalization Ability of Next-TokennPredditioo Pretraining
Therefore we get: RD (& g H) s (7) (IT) s 3dD o H).
Lemma B.6 (Levin & Peres (2017)). Given two probability measures Ti and T2 over instance space X, the following equality holds:
1 Σ (T((z) - T2(2)-- 2
| -h1-72llm T2
2EA
We then give some necessary symbol descriptions as follows:
9" hh*- = arg min LD h))
(9)
gEg,NEM
A arg max Ls. (g" h*). :E[N]
(10)
In the above symbols, 9" and h" in (9) are the Token-Predicton and Representation cepresentation-Leanee Learner that minimize the expected risk, exactly the best and h that we hope to learn through (2). k represents the subscript that maximizes the expected risk based
on distribution To (k E [N]) when using 9" and h", therefore Ta represents the worst distribution in l {T&) Based on the above lemmas and notations, we begin the proof of Theorem
Proof. We first perform an error decomposition on the excess risk defined in (3):
{ h) LDlğ e h) Lo (g" o h²)
= Col0o o h) Col0 h) + Lp o h) Lp (g' o1))
Col h) Lnl0 o h) + Cn (9 o Lo (g o oh),,
Bounding 1: According Lemma B.3, we can get
I = Lolğ h)) Ln o
log
$ 2Dx (eo + +4BB
N
Bounding II:
11 En (9" hh) Lo (0"="")))
1
Σ (Ex (g) -Cr (g) ())("))1'))
N
iE[N]
1
Σ
(ox (g* -Lo, (g*oh) + Lo. (g. ( - Lo. (g" (g* oh)))
N
KE[N]
<
1
Σ
Exi 6" o Lo, (9* (g
Σ (For (g" oh) (g)
Cos (9" o0"0""10 o1t)).
N
RE[M]
sE[N]
Bounding III: According to Lemma B.4, we can get
III = Σ
Lx.
or Lo: (9" o
N
SE[N]
Ammm 1l2 log
<
B
2m
17

---

On the Generalization Ability of Text-eooknnnroiiitioo Pretraining
Bounding IV:
IV
1
Σ
Cos (g" oo)) Lo. (g" ohr))
N
REIN)
Σ
(e (g" ohh La. (g" ))))) +
Σ
(b (g" oh) LTx (9" oh))
N
N
[E[N]
iE[N]
Σ RE[N]
Σ VEX
(T(z)) (,(2)1 E(g (2)
Σ
Σ AEE
[0=(2) Tx(z)] / (0 o (((2222))
N
N
iE[N]
2Br
(di(z) T((zi))
IE[N]
4B,
Σ
| - Th TV
N
NEIN]
S 4B, disc(U),
where (i) by Lemma B.6. Combining the above processes and by Lemma B.5, Theorem is obtained.
C. Capacity of DOMs
C.1. Model architecture
In this section, we describe the architecture and function class of the decoder-only transformer model in detail. Given a pre-training dataset D { S Rm*X, containing token sequences, where m represents the maximum word vector length and n, represents the vocabulary size. We can get N input matrixes { S TR.., We first introduce two normalization operations that will be used. For given matrix e Raxm, we denote softmax (-) as the row-wise softmax operator, which be defined
exp(A
softmax (A) =
(11)
E(mmm) (xp(As.)))
Let IT NTnem ()) denote the Layer-norm operator (Ba et al., 2016), which can be defined as:
1 E jjem As, Aij - a where = EeE(m) (Aij 11)2
morm (A)
(12)
8
We consider L-layer and H head decoder-only transformer model as our Representation-Leane h(-), which mainly consists one Embeedding-layr and layer transformerr-decode-bboe. We use denote the model dimension, dş = d/ H denotes the attention dimension, and dy 4d denotes the feed-forward dimension throughout the paper. Given token sequence Z E mmxx. as input matrix, the Ith layer's output is:
Z' Embedding (Z), 1= TF (Z
(13)
I LE [L]
here Embedding (Z) ZW, + Wp, where We xd denotes the coken-embeding matrix, W, E Ruxd denotes the position-emheding matrix. Note that matrices W. and W, are learnable, but also directly the pre-trained W. and calculate W, using sine and cosine functions, the specific calculation method can be found in (V Vaswaniet al., 2017). To simplify our analysis, we choose the latter.
TFw (-) denotes the 1-th layer transformer- block with
W/.. W (W) " (w! (Wi
2 " : Wn J (wh)) 1 E Rdxd, Rdy xd Raxd dddde ddxx Raxd
(Wk,
W
(14)
18

---

On the Generalization Ability of Mexterookenirriiction Pretraining
parameters, which be defined
TF Wi (Z'-1)- mmmm (FFN (Y')):
(15)
Y nnom (MHA (Z (Z)
where FFN (+) denotes the Feed-Forward Neural Network with Residual Connections:
FFN (Y') = (Y'W W Y
(16)
where a (-) denotes the activation function, and we use ReLU throughout the paper. MHA (-) denotes the Masked- -util-
Hcad-Attention with Residual Connections:
MHA (
E
A (Z'-1) Wo, +Z'-1
(17)
he(H)
here A (.) denotes the Self-Attention head:
Qh (Kr) + M Vdh
A (z softmax
V/
(18)
where Q/ = = T.. Qn
K) = Z W Kv V = -
W Vi
denote Q, K, matrix respectively, and
0
00
-00
-00
0 0
-00
-00
M
E RR mmXm
(19)
is a given mask matrix. Therefore the Representation-Leann can be defined as h(X) = Z And the hypothesis class of (XX) is defined as:
X HTFW (TFw .... 1... ...: (Embedding (X)))
H
| w l. Wo W/ p II
W k. |I Wt - Bi, BI [L] E [mn]
(20)
The Ooken-Predictor has many options, there we use simple linear projection layer and softmax:
9 (h (X)) softmax s
(21)
where WP E RRdxx.
Assumption C.1. Assume that
[mom is G--Lipschitz with the f2-norm, i.e., Vt,, t2 E Rd,
Hoom (t1) _m (te) S G₁ Gx tzlle
E [L] and h E [H], there exists constants C such that
(kh)) St.
WE [L], w E W there exists constants B; satisfying
W F S B
19

---

On the Generalization Ability of Next-Token-Predictin Pretraining
C.2. Useful Lemmas
Here, the covering number bound Rademacher complexity, first introduce the definition of the covering number and provide some useful lemmas and propositions.
Definition C.2 (e-cover and covering numbe). Denote (U, || [l) normed space and V C.. For any > 0, V is called an e-cover of U if for any E U, there exists E V such that I|u - wlll < The covering number of the normed space [I, II) is the cardinality of the smallest e-cover, which is defined by NU, Il -1) == min[IV]: V is an e-cover of U U}. Lemma C.3 (Lemma 9 Lin & Zhang (2019)). Let W == {w: : w E R. || I a), then for any > 0, we have
In NW, e, |l (12) - dln (1+ 2a
)
Lemma C.4 (Lemma of Lin & Zhang (2019)). Let E guxdin be given input matrix with bounded F-norm, and W e Rdin dout satisfying [W|F a I a, then
In | S aat (1x) Z dindona In (1 2a 22l|X(1E)
Proof. Let W be the e-cover of (W: ||]| - a} such that |W- W S then
XW XW P - X X |W W|l 1 |||
This means that any e-cover of (W:
| F < a is also an F-covrr for { ||]F S a). we have
( | S sat,c, c. - S In (w W S sa),
InW
| Il -11-111)
We denote W E Rdindout the one dimensional vector which is obtained by reshaping W. Then by Lemma C.3, we have
In (IW: s at, |X||=" 11
= InN (IW:IWI: S sa)
1l |1.
|X||="
S dint In (1+ 2a xx|
c
Lemma C.5 (Extension of (Bartlett et al., 2017)). Let F be real-valued function class taking values in [0, c). and assume that 0 E F. Then the empirical Rademacher complexity of F can be bounded as:
4a
12
In movn n
VIN (Fisse,) l12)de
RsFF)
inf
a>0
Vn
where S represents the dataset containing n samples.
Lemma C.6. Assume that f() is a continouous function on [a, b] satisfying f(x) O, and g(x) is continouous concave(downward) function the range of f (x)). Then we have:
b-a [ o(f()))dr 5 (1-aS. ((x))a
Proof. We divide the interval [a, b] into n eaqual parts, let x; = a + (b - ai = 0, 1,2, ,n), then A, = 2, - 24-1 b-a = 1,2, ... n). Since g(x) is concave function, we can use Jensen's 's inequality to get:
Σ (((()))) = x -1 ((((,))) و
Σ E))(0
b Σ f(,))
a
Combining the integrability of continuous functions and the definition of integral, let - oo in the above formula, we can get the result.
20

---

On the Generalization Ability of SeteToken-Prrdiction Pretraining
Lemma C.7. Let II norm ()) be the layer normlization operator defined in (12), for any matrix X Rmxd have
nonm (X) 1| Vmd.
Proof.
X
[ norm
(x) || =
Σ
+ 3 E f X
xe(mi jE(d]
x b
ke(m) jE(d] a Σ Eee X
md
Lemma C.8. Let softmax ()) be the row-wise softmax function defined as (11), for any matrix X E mR obeying
ex S C, we have:
||softmax (X)lle S e and ||softmax (X M) S e Vnn,
where M is the given mask matrix defined in (19).
Proof. Denote (211g) mxm and softmax (X) (Vis) mxm' we have:
6
e
20
Vij Eye(m)) ehtyy e
S
me C m
Then get:
22C
|sof[max (X)IlF
Σ (E(m) jE(m)
S
ΣΣ
Mij
ec
m
KE [E(m)
JE(m)
here uses [me(m) Vig 1, Vi [m]. When adding the mask matrix M, have:
311
00
Vi1
0
0
F21
T22
00
1/11
1/2
0
X M
=
softmax (X+M)
Fm1
Fm2
Fm3
Fmm
Fm
V'me
V'a
Fmo
where
.J S k .J > k
Eyeny Byy
Similarly we can get:
2C
|sftmaa (X + M)
Σ
Σ
ec
Σ
k
kE|mJ JE(M
kEim]
Here we use the fact: (1 + + + Inm) + 7, where 7 R 0.577218 called Buler constant.
Lemma C.9. The softmax G. Lipschitz in the norm, and G, have:
4/3 which means for any x.y IR
4/3
|softmax(x)) - softmax(y) < 9 x - ylle.-
21

---

On the Generalization Ability of
Pretraining
Proof. Let e, denote the jth element of softmax (x), the Jacobian satisfies:
[](x)]]= [ddiag(sffmmxx()) softmax(() softmax(x) Fl
(c: (I 5| =e,))
EE mm) JE[m]
Σ ox
1- Ei +
1
Σe, Ej
km(ml
jfi
4e (1 (1-e;) ei)²
EE m
4es (e 26 + es)
2E(m)
16
27 RE|m]
4v3
where the last inequality the fact: x² 2x x 5 E [0,
Denote y according to the definition of derivative, we have:
softmax(x + ōz) softmax(x) (x)2.
lim 6-0
5
Integrating along 8 = 0 to under |x(xx) S 4v3 9
||z|le, can obtain the result.
C.3. Proof of Proposition 4.17
Lemma C.10 (Covering number of masked self-attention head). Given an input sequence S = (Z....., Zn} E Ruxd denote [N = [Z,,..., ZN] E Nmxd as the concatenated data matrix. Consider the Self-Attention head A (.) (ignore the layer and head indices) defined in Equation (18), the corresponding function class can be defined as:
A (Z,)
Z.Wa ZWW +M
A
A (Z:) = sofumax
Z.W.
H
vdg
A (ZN)
Wolle, WKlle, [ SB
then we can get the following covering number bound:
2c BVN
Z 4NllF
8G, Z || [
IN (HS f1 IIF) s dl In 1+
+ 2dd In 11
Vdtee
Proof. Step 1: Denote Cv év-oover of set HY = (V = -Z/z) ZinDWv Wv
Wrrle & B}, then by Lemma 3.6 we have:
2B 1
In (H |l. (|8) < In In [1
cV
Step Let Co to be a cover of set H (Q= ZWq : || S B), and Cg to be cover of set
Hk [K- ZW k || || w |r < B). We can use Co and Cx construct a following:
S
0
Z.Wo (Z. T +M)
Cs =
S =
S, softmax
We Co, WK EK
dd
0
SN
22

---

On the Generalization Ability of Texteiooknnrrriiction Pretraining
We consider the following set:
S
Z.Wo (Z,WK) K) + +M)
H =
s
S - softmax
||Wolle;/WKlle | SB
vd
N
then for Ve and any S E Hg. there exists Cs such that:
Si
S1
0
s S
S
0
SN
Z.Wo (Z, W k)
M
Z Z.Wo
Z, (Z.WK)
+ +M
Σ
softmax
softmax
vdx
ddl
MM[M
G2
Z.Wo (Z.Wk)= Z.Wo Z.WKk )
Σ
dd
EE(N
Gs
(I(2.wo ZWW)) (Z.Wk) (1. + 12.+Z z.wo (z.Wk) -(z,Wx) ) 2
Σ
V dd
iE|M]
G.B
(EQ2 + te)
17
VE(N)
G.
(cQ + (k) |Z
(
Evoking Lemma C.9 we have (i), and by setting cQ cx
Jic
get (ii).
Therefore Cs is cover of MG..
2G, Ml f
then by Lemma C.4 we have:
4G, 4G, [MmllE
IN (HS, e, 11 11x) - In [Cs] < /n/l + In CCx s 2ddz In 1+
vdke
where | = maxie[N] |Z.||F
Step 3: For every given V. E Cy, we can construct the set H o V = SSV S E nz) and Cs o v = (
we denote C (Cs v,es, Illl- (IF) as es-cover of Cs V. Then for any SV E H v, we can find SV E
C (Cs v.es. llr) such that:
SV sv s s|s
IIVI= - S S
I| e사 i get that C (Cs vess,11-112 is a covovr of H V which can be denoted as
We can Choose Es
(HS. o v, Veea,ll -117) Then we have:
sup In l (H$ V e. -[m] veCr
S sup In le (Cs 71 s 2dds In (1 (1+
4G, B3 Z' 1
VdLEA
Vecy
Then we construct a set CA:
CΛ = U C
HS V. .. -11).
VeCr
23

---

On the Generalization Ability of Next- Poreeeeeennroiiiiioo Pretraining
which is easy get:
In Ca - S In In|Cul + sup In | (NS) o v, |1+)
VeCv
2B (M]]lE) + 2dd In (1 4G, 1 |2 |N 11?
I d In
1+
cv
Vd
Step 4: Next we will proof that CA covers HS. For any A E HS. we can find Å E CA such that:
A
A
SV sv
=
F
P
I SV - Sv + S|S - -VV-SVI
S | v v +
|S.11.Ev EA
(EIN]
S V N me év + EA,
where the last inequality Lemma C.8. Then by setting cv 2e CA 1. get:
In In (HS, 11 (1F) < In Caal
2e BV
Inm NII + 2dd In (1+ 8G,B3 || ZiN
< dd In
1+
vdue
Proposition C.11 (Covering number of transformer-dccoder block). Given an input sequence S (Zi.... Zn E Ruxd denote [ i...... Zv] Nmxx the concatenated data matrix. Consider the connsormer-decoder--be TF (-yignore the layer indices) defined in Equation (15), the corresponding function class can be defined as:
Z ++lomom (YWWF)) WF2 Y), Y -Inorm
ΣAN An (Z) Wo. +2)
hEH
HgP
ZW Qa (ZW K. )
M
An (Z)- = softmax
ZW vv
Vdz
WFille W || Wo.
W (Q Il
[[ K. Il, Wv,ll; S B, vh E {H]
then we can get the following covering number bound:
GBB +1) (CBBHyNNnme
| Ir
In (HSP, e, (Ix) S (
+
1+
Proof. Step
We first consider the function class of muti-head-attenttee MHA (+), which can be defined as:
MA = Σ
A Wo, + Z A E HAA, Wo, s V. [m]
HS
RER
where
A (Z))
Z.Won (Z,WK) K.) + +M
Ap =
An (Z.) softmax
Z.
H.A.
Vile
A (ZN)
1 Q. Ie, | lle Wv, S
24

---

On the Generalization Ability of Next-Tooee-Prediction Pretraining
For any h E [H], denote CA. as a a of H select element A, E CAs construct the following set:
A Wo, (A. : || S B].
let C (An o Wo <d. 11-11)>) ca-covers A, Wo,, we have:
C (An Wo.. ed. 1H1p) - sup INN (Ån W Edi ll-l AnA
2B A 1|
$ sup AnECAn
df In 1+
Ed
2e B² v N In (m1llF)
$ d In
14
cd
Now we can construct the eg-cover of set
An Wo. An E Hg ./Woolllle S B as:
U c (An o Wo,, cd. l11l))))))
Chead,
ALECAN
Combined with Lemma C.10, is easy to get the following covering number bound:
In (15 ca. ll IIx) SH In Cheadsl
SH In CA
In l (An Wo..ed;
ed. (lon)))lllle))
sup ARECA
2e BV Z ZINIlE
+ 2dd H in (1+ 8G 8G,B$112 1z" |1 +1 |NN
Hd
1+
EA
VdNEA
26
B2
V N m ANyllE
+ H1
1+
Step 2: Now, we consider the Feed-Forward Neural Network FFN (.) defined in (16), which consists of two fully-connected layers. The hypothesis class of fully-connected layer can be defined as:
H = Y WF (N = nomm (MA),MA E H WFill - B
Denote Car as a eover of HSs MA we select one element MA E Cm to construct the following set:
MA o WFI = [nurm (M^)WFi: Wrille $ B).
let C (MA Wrr.er1:llll eF covers MA WFI. we have:
Inle (MA+ o (EL v sup In (MA o WFL,FFI, MAECH
2B Immm (Ma) ||
< sup M.ECr
ddy in
1+
EF1
2BG [H] (H| F (M)l))
$ sup ALECAn
ddy In
1
CFI
2BG (CBPHVNNm M
< ddy In
1
EF1
25

---

On the Generalization Ability of Next-Token-Predictin Pretraining
Now can construct the eo1-cover of H
CFi
U
c (MM WeL)eFi,ll-Ie)
MAECM
We have the following covering number bound:
In le (MA o WFI, (ri,1-l))l)
In Cr, - <ln Carl+ +
sup MaECи
eeeBBBBB BVN lnml (Mille
8G, IZ'
<d Hn 1+
2ddd In |+
VdrcA
2BG (cC Nmm + 1) [emlll
Ba B²
2e
Inm (N1lEE
+ H
1+
ddz + +dd In 1+
EFI
Step 3: Next, we consider the hypothesis class of fully-con cully-connected layer 2:
H55 for (F(N) WF2 YN FIN E H5 YN Imam (MA),M, MA H |eaall S B}.
For every element FINj E CF, and MA E CM, we construct the following set:
F [ij WF o MA =
= =fo (F(M)) WF2 + Imorm (MA): WF - sB].
let C (FIM WFi (2.lll)) EF2-covers Flv WF o MA. we have:
ln (PMM We MA,6r2,1l11a)
In (FIM) WF MM,
sup PINJECAI MAECU
In (fIM= WF1,cr2,11-1e
= sup FInjECa I sup FINjECA
2B F
Wwlllo
ddj ln
1+
6F2
2B²
Nnomn (M^)llr
s sup MACCM
dd, In
1+
6F2
2B2G, (H]) + |ZIN|||>)
< sup
ddş In
1+
AnECas
6F2
2B (c Inm + 1) I
< ddy In
1+
6F2
Now we can construct the cover of H.F as:
CF.=
U
c( Wr M2.l22.lll
FIMECA, MAECM
26

---

On the Generalization Ability of Next--ooee--reedcctin Pretraining
We have the following covering number bound:
In Cpp In [Cr,
+ In [Cul+ +
sup
l c (FiN) WFI o MA. (F21111)))))))
FIN MAECA
2c By Nlnml (Mlll
+ 4dda In (1+
8G, Z
1| 12
H2 1m 1+
VdleA
2BG CBBHHVNlm + 1) (N
2c B² V N In m (NIlEE + dd, In
+2d in
11
1+
ed
Efl
G (CBBHVNMm +1 |
+ ddd In
1
CF2
Step 4: To get the covering number of HTF. we construct the following set:
C = [ (F2) : F2 E Crah.
which is an e-cover of HDS that can be verified. For any ZIN] E HTF, we can find a NN E Cr such that:
| - 2 = 1mom (o (Y WFI) WF2 + ((N)) - Inoor (x(XIN) WF2 + MIN))
<Gr 0 (Y [N] WFI) WF a (YIN) WFI) WF2/
Gxx 0 (Y WFI) WF2 a (X(N) WFI)
WF2 + Gx Y|M] 1 YYN
<Gx FFF + B (NN) WFI WFI
+ +||IIM- -Y(N) lp)
F
<Gx FFF + B Y(N) WFI WINWWFI
+ +B X
+ WF Y WF/ F + +||X|m] - (
F
<Gx (F2 + BEFI + (B² 1) | - YIM|1
For YIM- - {(m]
we have:
|NIN] - {IM|
= 1l
Σ An An Wo. +ZIN)
-loom
Σ KEH
Wo. + +Z
An Wo.
VEE
< G Σ
Won An An.oll
KEll
< G Σ
A.. A.Wo + A. -/A.Wo..A,.o.l1/
hER
< G. + BHHA).
In summary, we have:
|INI 2/m1/1 S Gr (eF2 + BEFI + G+ + (Ha BHHa))
= =G + GGBEFI + (B + 1)Hfa + GZ + B)HAA.
We can conclude that Cr e covers H By setting
E BG + B)H
EF2
4G , EFI
4G, B Ed 4G2 1)H
27

---

On the Generalization Ability of Next-Token-Predictie Pretraining
From the definition of Cy, have:
In [rl - In CF2l
1+ (B. B3) Inm Z mmll
<4d H in
32c 32c G + B Z' Z 4M1lE
+ Add,
1+
vdRe
B e (CBHNNN + 1) 1Z
+ 2dd In 1+
BB +1) ( (CBBBVNNTm+) (mpllr) 1
S(Ad H 4dd, H + 2dd,) In
Combining de = d/H. d = 4d, we have:
GBB +1) (CHHyNNmm +1) [Z IF
In InN(H) (HT", :6ll (1=) S In In S (H+ 1+
Lemma C.12. Let ('N) E Nmxd be the lth layer's concatenated output matrix of our transformer model defined in
Equation (13), we have:
III CB +1) (c B3 +1) Izinlle.
jEl
Furthermore, let Z E mxd as the output matrix, we have:
| || < II CB +1) ( |
SE(4
Proof.
Inorm (o(Y(N)) Whn) w {IM))
(1)
Gx o (YiNy Wha) w - XIMIIE=
S
GG(B +1)
Σ Al (z
Wo, + Z [N]
hE(H]
< GB +1)
B Σ
+ +1
hE(H]
< CB +1) (e (BBBHVVNIng +1) |zint|la I|
Il (B) +1) ( B, 1) 1l JEL
Here (1) uses Assumption C. and (ii) uses Lemma C.8. Similarly, we can get the second result by setting
28

---

On the Generalization Ability of Next-Tokee-Preedicton Pretraining
Lemma C.13. For a single tranformerr-eecale--bbccc TFw (-) parameterized by W (ignore the layer indices), let [Nj TR. Nax as the concatenated input matrix, have:
TFw (ZIN) - -TF (2IN)) S GB +1)
Furthermore, let Z E Raxd as the input matrix, we have:
TFw (Z) - TFF (3)11= s ( (e
1)) Hmd 1) -- 2
Proof.
|FW (Z(N)- - TFw (2N) |1 s <G> (Y WFI) WF2 + YINI (Viv WFl WF2 YINIE
S CB +1)
Σ NEIH
An (Z(N)) Wo. + Z Σ An (Zmm) Wos ZiNi
he[2s]
$ G² +1)
B Σ hE(m)
Aa (Z(N) A (2mN) + Z
| - 22ml11)
For any h E [H], we have:
|| An (Z(M)- An (2(M)
11- - S Wv. - -S,ZIN Wv.
11 1
s (|5,Z S
SAAZIN-SS,ZN l1p) + | 2,
F.
S B /
Sn. - 5/
+ c C.VNNn [ - (
For Sp. S/ p. have:
Sy.
St
Shi Sni 12
Z, (Z, kK) M
Z.W Qh
(2.WK)) kn
+ M
Σ
softmax
softmax
Vdk
Vilk
KE(N
G.
Σ
Z. on (Z.Wkn)T Wor (2.Wkn) kh
7/2
Vdx
VE(N
G.
Σ
(I(Z, -2 Wax) (2. (2.Wkn) |
2, Won Qn
(T.W -(2, kb) "[])" 2
--
Vdx
VE(N)
G2
Z. -
Vil
KE[N]
2G.B² Vmd
Σ
Zi - 2/
Vdu
F
N
2G 2G,B² mdd
ZINi 2IN
29

---

On the Generalization Ability of Next-Tooen-Predictin Pretraining
Combining the above results, we can get:
@BBHVNNm + 2G,B*Hv HVNmd +1) |IN1 -2mm
TFw (Z(N)) TFw (2|M) GB +1)
vdk
5 GB ( ( 2 2
C.4. Proof of Theorem 4.20
Proof. Step 1: We firstly define the class of lth layer's output as:
H' TWF (T
1.. TFw: (Embedding (X)))
Wi
W F
W 2. 1| W F. /wall S BB. E {
P1
P2
For / = 1, we consider the set:
TF, (iN) (FW (z'N)= w }
where Z M]
Embedding (X N:) (X))) represents the embedded token sequences, and W defined in (14). We denote
C = C TF, (wi)) se, 1.ll-11F)) as the -o-co of TF, (xNT) Then for < <<+++ S L, let Cj be a cover of H', we
select element
[NN]
E Ci to construct the -cverr of following set:
TF1+1 (Zix)) = {TFwiI (ZIN): w e wtt1).
here we denote с (TF (z))) .441,11-111 (11)) as the cover. Then by Lemma C.12 and Proposition C.11, we have:
B2 + 1) (cenn Bi H V V + 1) 11 lFr
In le (TF (Z(m) ,e411:11117) 4H +3)
1+
411
17415441 514
< 4d (H + 33
1+
4+D
In
where 8141 H +1)
(BBB+1)
(1)BBBB 1).
Step Now, we can cunstruct the cover of H as:
U e (TEwi (Z(M) E4411 1441,11-111 11.11l))))
C411
2 ix
Then we can get:
U e ((TFW (Z(m) (442:11-112) 1C - I[N; Ng. j=1
1C4+1|
EC
We have:
4+1
In C++1l S Σ In,,
1=1
1+1
B3s, 11 10
< 4d +3)
Σ In
1
j=1
30

---

On the Generalization Ability of Next-Tokee--redditioo Pretraining
Step 3: Next go verify that Cz is cover of H4. By Lemma C.13, for any Zívy E 74 can find 2/x that:
C2 such
ZKN
2
TF TFW We
(
TE (Z(N1)
< wF w. (Z/N) Ww. (2(x1) 11, + TF (Z(r))
-TF vipe (2 111
- 6 (B)
+1)
(e. HiHNNe Nmdd + +1)
|Z [/Ti]
+ +
Σ II (B). +1) (c B3,WVNd++++
j'=t+1
We set CL i. and for 1 & 1 < L - 1, we choose (IIT C +1) (,, + +1)) c.
We denote 8141-1 = 1 G (B3 + 1) (c B3, H HVdd + 11). then we can get the covering number of H6 as following:
In (H 61 (1F) & In Cr
Bf 7 IF
s 4d +3)
Σ
In 1
E1
1=1
B7 (7
L 1F
S 44 +3) Σ
1
Bß 1. 1 | (Nn]lr
= 4d + 3)
Σ
1
1=1
Furthermore, let X E pmxa, as the input matrix, and Z -Embedding (X), by scaling N to 1, we have:
1+ BB
In 61 s (H 3)
Σ
1=1
where <t, Ney C2 + +1) (c Bi Hmd
D. Proof of Theorem 4.22
Proof. Step 1: We first bound the Rademacher complexity (D).). For Z; [t. ... , -1 E Rmxmy note that we do not input each token one by one order, but input the entire sequence at once, and get representation matrix h(z.) E Raxd Due the existence of mask M, each token can only use the information before the current node. We can naturally abstract the above process into using the jth token t² to perform queries in sequence. Therefore Ht (H) be defined as:
(H) E.
ΣΣ Ei I| (C-D)lr sup hEH Nm SE(N]
By Lemma C.7 we know: for any i E j E [m], we have /| (15-1)l S Vd. Therefore, according to Lemma C.5, we have:
4a
12
Nmd
RD (H) < inf
V InN (H. 1.l lla
a20
VNm
Nm
31

---

On the Generalization Ability of Next-Token-Prediction Pretraining
We denote X (H + 33 and PL E'= Bf. then Theorem 4.20, can get:
Nmd
VNmd
BR Is 1 (N]) )
1F
IN
S VAL
Σ In
1+
de
e
Pr.Sr. LZ (olllo ))
Nmd
VAL
In 1+
(6)
//mmd
1 PLSL
IZ
VLL Nmd - a)
de
VNmd Ja
= VVL(VVNNI) Nmd -a) a) In PLSL Z 1 In /Nmm
VNmd
lI In N/Mmd
< VLLNmd In 1 (N)
VNmd
Here (a) uses Jensen's 's inequality, (b) uses Lemma C.6. By setting a VNmd we have:
In 1 PLSL 1| [N) || In Nmd
4a
12
RD < inf as0
+
VLLNmd
VNm
Nm
VNmd - a
12v ALd 4
PLSL IZ le In In(Nmd)
In 14
Nmvd
VNm
Nmd 1/VNnd Nmd
IA 2V3
VALd
/In (1 + PL.LL)
VNm
12 (H + 3) In (1 + PL8L)
Nm
where (1) uses the fact that IxlllE is of /Nmd.
Step 2: Next, we consider the Rademacher complexity (D ht):
RG o h)) E
Σ Σ
Eij |sotmax (t(+) w
sup ||WPIDSS
Nm
iE[N] jE(m]
We consider the set
G o h /sottaax (tf WP):
|| - R
where 2/in e Cz. By Lemma C.4, C.7 and C.9 easy to have:
In (gohh,l (1x) s dn In (
2RG, 12 (2)]lo
2RG, v VNmd
1+
< dn, In 1+
e
Then, by Lemma C.5, we can get:
dn, In(1 2RG.) Nm
Rp o h) s
32

---

On the Generalization Ability of Nexe-iioeenrreaictton Pretraining
Step 3: For any [m]] let Ta IR be the input token sequence. Then for any h h E H and g E G, have:
| (1 (Tx)) ( (Tw)) = 111 (T&) W (Txs)) woll.
(Ts)) - h (Tw)lP
Therefore Gg = R, then by Corollary 4.11, we have:
8ln
Co.r log 2
h) < RG,,G,RD(H)) + (G o h) +B
+ BBV
+ 4BB disc(U)
N
2m
(II)
12Ld H + 3) In (1 PI.
dn, In(1 + 2RG,)
<
+ 6GLV
Nm
Nm
8ln
Cr. log
+ BeV
+ B
+ 4B,
N
2m
In PLSSL)
dn Nm
8 N
C.rr log à 2m
SGERVOdH
+ Gey
B
Be
+ 4B, disc(U).
Nm
Note that, the number of model parameters is L(10d 2/1d²). Since d is usually much larger than H, we follow the approach of Kaplan et al. (2020) which use θ 12Ld approximation to represent the number of model parameters. O
33


Reviews:
[{"review_id": "Otswe6DQ0X", "reviewer": "Reviewer_cRs1", "summary": "This paper investigates the generalization properties of Next-Token Prediction (NTP) pre-training. The derived generalization bounds highlight the influence of key model parameters, such as the number of token sequences and the maximum sequence length. Specifically, the contributions include: establishing a Rademacher complexity bound for the excess risk, deriving covering number bounds for the function space of transformer-decoder models, and providing a generalization bound for decoder-only models (DOMs) in the context of NTP pre-training.\n\n## update after rebuttal\nThe authors have addressed my questions, and I will maintain my current score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and support of our work. Below, we provide a detailed comparative analysis of our work with the two most relevant previous studies, [1] and [2]. We sincerely hope this will address your concerns. All of the following content has been added to the discussion section of our paper:\n\nOur approach shares the overall methodology and thought process with [1] and [2], where we first decompose the excess risk error to derive an upper bound on the Rademacher complexity, and then obtain a more refined generalization error bound by analyzing the model capacity, i.e., the covering number. However, there are several key differences:\n\n1. **Task Context:** We focus on NTP pre-training, using $\\varphi$-mixing to model token sequences to capture dependencies between tokens within a sequence, whereas [1] and [2] only consider independence between sequences.\n\n2. **Rademacher Complexity:** Since DOM can be viewed as composed of two parts, its function space is considered as a composite of two function spaces. Therefore, we introduced the Rademacher complexity of composite function spaces and derived the corresponding decomposition theorem, Proposition 1. Compared to the direct analysis of a single function space in [1] and [2], our method facilitates the analysis of pre-training generalization when changing DOM task heads and makes it easier to transfer to different downstream tasks.\n\n3. **Covering Number Analysis:** Like [1] and [2], we analyzed the covering number of Transformers, which is why we share Assumption 4.14. Similar assumptions are common in covering number analyses, such as in [3] and [4]. However, [1] and [2] analyze the covering number of encoder-only models under the spectral norm, while we analyze decoder-only models under the F-norm. Our challenge includes considering the masked self-attention mechanism, with the main impact of the mask matrix detailed in Lemma C.8.\n\n4. **Final Results:** [1] and [2] show polynomial dependence on data norms and parameter bounds, and logarithmic dependence on the number of model parameters, which is the opposite of our findings. Our results nearly eliminate dependence on data norms and show polynomial dependence on model parameters, aligning more closely with the Scaling Laws for large models [5], which suggest linear growth of data size with model parameter count. This is mainly because we cleverly applied the convexity and concavity of functions, as stated in Lemma C.6, to Lemma C.5, with detailed applications in Appendix D.\n\nAdditionally, Assumption 4.2 is crucial for considering dependencies between tokens within a sequence. Based on Assumption 4.2, we can use Lemma B.4 to analyze the model's generalization ability on a single token sequence.\n\n**Reference:**\n\n[1]Edelman, B.(2022) Inductive biases and variable creation in self-attention mechanisms.\n\n[2]Deng, Y.(2024) On the generalization ability of unsupervised pertaining.\n\n[3]Bartlett, P.(2017)  Spectrally\u0002normalized margin bounds for neural networks.\n\n[4]Lin, S. and Zhang, J. Generalization bounds for convolutional neural networks.\n\n[5]Open AI.(2020) Scaling Laws for Neural Language Models.", "claims_and_evidence": "The claims in this paper—particularly the generalization bounds—are well-supported and convincing.", "methods_and_evaluation_criteria": "This is a theoretical work, and while the simulation settings are relatively simple, they help reinforce the theoretical findings.", "theoretical_claims": "The theorems and proofs provided in this paper are convincing.", "experimental_designs_or_analyses": "This is a theoretical work, the experiments are valid to verify the theoretical results.", "relation_to_broader_scientific_literature": "This work provides a thorough discussion of related literature and compares its results with prior studies in Section 2. The authors also strengthen their assumptions by referencing established works. However, my main concern is the lack of discussion on how their proofs relate to those in previous research. For example, Assumption 4.14 was also used in Edelman et al. (2022) and Deng et al. (2024). While the authors compare their bounds with those in these two papers, they offer limited insight into how their proof techniques differ from or build upon those earlier works.", "essential_references_not_discussed": "The related works discussion is sufficient.", "other_strengths_and_weaknesses": "As mentioned earlier, my main concern is the lack of discussion on how the proofs in this paper relate to prior work. For instance, Assumption 4.14 also appears in Edelman et al. (2022) and Deng et al. (2024). However, it is unclear how the proofs in this paper build upon or differ from those studies, and what specific challenges the authors faced. As a result, the novelty of the proof remains unclear. Additionally, Assumption 4.2 appears to play an important role, but it is unclear how this assumption contributes to the development of the proofs.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See comments from Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4GmcP6ycwA", "reviewer": "Reviewer_axCZ", "summary": "This work derives new bounds on the generalization power of multi-layer multi-head transformer models pertained through Next-Token-Prediction mechanism. The first theorem in the paper shows that the generalization error is bounded by the Rademacher complexity of the class of $\\mathcal{G}(\\mathcal{H})$ (where $\\mathcal{G}$ is the  token predictor (decoder) and $\\mathcal{H}$ is representation learner) and other additive terms that decay with the number of training examples and the sequence length. The next main result bounds the above Rademacher complexity through covering number and shows that it is bounded by a terms that is increasing with the total number of parameters of the model and dimension, and decays with the number of training examples and the sequence length. The combination of these two gives a generalization bound that is mildly deteriorating with $'L'$ number of layers (only linear compared to previous works which was either exponential or quadratic) and decreases with the sequence length as $1/\\sqrt{m}$. The method used in this work also leverages $\\Phi$-mixing to account for the inter-token dependencies and allows for the masking operation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted to receive your recognition and support for our work, and we appreciate your careful attention to details we might have overlooked. We have made corrections to the relevant sections of the paper based on your suggestions. Additionally, we plan to release the code upon acceptance of the paper. Below are our responses to your questions, which we hope will address any concerns:\n\n**A1:** Our work primarily focuses on the proof details, and we initially lacked a thorough technical comparison with previous works. We have now enhanced these sections. Compared to earlier works such as [1,2], our main technical challenges and innovations include: \n\n(1) Considering both the independence between sequences and the dependency between tokens within sequences. Unlike previous works that only considered independent and identically distributed token sequences, we introduced the $\\varphi$-mixing tool for analysis. This increased the complexity of error decomposition but also enhanced the interpretability in the NTP pre-training scenario, specifically the bound's decrease with sequence length. \n\n(2) We introduced the Rademacher complexity of composite function spaces and proposed its decomposition theorem (Proposition 1), enabling separate analysis of different function spaces. This greatly simplifies the generalization analysis when changing DOMs task heads or transferring pre-trained DOMs to different downstream tasks. \n\n(3) We performed a detailed analysis of the covering number for masked-self-attention-based decoder-only models. Our approach maintains a high structural consistency with the original Transformer [3] and analyzes under the F-norm, which was not discussed in previous works. The main challenge in considering the mask matrix is defining the upper bound of the attention matrix's norm, as seen in Lemma C.8. This results in an additional $\\sqrt{\\ln(m)}$ factor compared to analyses that do not consider the mask matrix's norm upper bound.\n\n\n**A2:** Thank you for your question. We compare our work in two aspects, denoting the input data matrix as $\\mathbf{X}_{[N]} \\in \\mathbb{R}^{Nmd}$: (1) Regarding the dependence on $\\epsilon$ and data norm, their bound is $\\sqrt{\\frac{||\\mathbf{X}\\_{[N]}||^2}{N}}$, while ours is $\\sqrt{\\frac{\\ln(||\\mathbf{X}\\_{[N]}||\\_{F}/\\sqrt{Nmd})}{Nm}}$. Given that the order of $||\\mathbf{X}\\_{[N]}||$ is $\\sqrt{Nmd}$, their bound has a greater dependence on the data norm, whereas our bound almost eliminates this dependence. (2) Regarding the dependence on model dimension $d$, since the model parameter count $\\Theta=\\mathcal{O}(d^2)$, their bound has a logarithmic dependence on $\\Theta$ due to its logarithmic dependence on $d$. Our bound has a polynomial dependence on $\\Theta$. According to the Scaling Laws for language models [4], the dataset size should grow sub-linearly with the model parameter count, making our bound more consistent with the Scaling Laws.\n\n**A3:** In our work, `disc(U)` primarily measures the overall quality of pre-training data. We assume all sequences are generated by a $\\varphi$-mixing process, defined as an infinitely long sequence, implying that most sequences may originate from the same mixing process. On one hand, two different sequences might be extracted from the same corpus or even the same article. On the other hand, due to the nature of human language, even seemingly unrelated sentences like \"Large language models benefit humanity\" and \"The weather is nice today\" can appear in the same context when read together. This indicates that any high-quality sentence conforming to human language rules can be interconnected through language. Conversely, sequences like gibberish, grammatical errors, unclear expressions, or incorrect knowledge struggle to establish connections with high-quality human language. This underscores the importance of data cleaning quality [5]. Fewer low-quality sequences in the pre-training data result in a smaller `disc(U)`, thereby enhancing the model's generalization performance.\n\n**A4：** Regarding Lemma 4.16, we acknowledge that the notation $||W_Q,W_K,W_V||\\_F \\le B$ was indeed a space-saving simplification in the main text. The complete formulation, specifying $||W_Q||\\_F ≤ B$, $||W_K||\\_F ≤ B$, and $||W_V||\\_F ≤ B$ individually, is provided in Lemma C.10. We have now added explicit clarification of this point in our paper. Additionally, we confirm that the issues raised regarding line 186 and line 427 have been addressed. We thank the reviewer again for bringing these important details to our attention!\n\n**Reference:**\n\n[1]Edelman, B.(2022) Inductive biases and variable creation in self-attention mechanisms.\n\n[2]Deng, Y.(2024) On the generalization ability of unsupervised pertaining.\n\n[3]Vaswani,A.(2017) Attention is all you need.\n\n[4]Open AI.(2020) Scaling Laws for Neural Language Models.\n\n[5]Penedo, M.(2023) The Refined Web Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only.", "claims_and_evidence": "Yes. Theoretical results are rigorous, proofs are provided in the appendix and real-world experiments align with the theoretical claims.", "methods_and_evaluation_criteria": "The results are sound.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "they sound valid. however, code is not provided by the authors.", "relation_to_broader_scientific_literature": "The work presents a theoretical upper bound for the transformer DOM. This is closely related to Zhang et al. 2023 and Deng et al. 2024 which also utilizes Rademacher complexity. Compared to closely related work of (Deng et al. 2024) which considered $\\mathcal{H}$ as fixed (cf. summary section), here the Rademacher complexity is computed over both $\\mathcal{H}$ and $\\mathcal{G}$. Also, the dependency on the number of layers in the resulting bound is improved.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\nThe considered model is quite sophisticated and close to what can be used in practice. The method is rigorous with few assumptions on the model and also considers inter-token dependencies via $\\Phi$-mixing. The authors use an interesting combination of known methods to bound the Rademacher complexity of both the representation learner and the token selection part of the transformer through bounding the covering number of networks with bounded Frobenius norm. \n\nWeaknesses:\n\nThe paper doesn't discuss in detail about the new/challenging steps in the proof compared to previous works. the authors mention they're the first to consider mask matrix but it is not discussed how it affects the approach or the final results. Considering NTP paradigm and obtaining bounds decreasing in terms of sequence length are interesting contributions of this work but the paper lacks a high-level discussion on the challenges. \n\nIt seems Lemma 4.15 which bounds the covering number of Frobenius-norm bounded weights (by $a$) and is only logarithmic in $\\epsilon,a$ and data norm is essential for obtaining improved results in terms of number of layers. This is comparable to Lemma 9 in (Deng et al.2024) that obtains a bound on the covering number of $\\ell_{q,s}$-norm bounded weights and has a polynomial dependence on $a$ and data norm but with better dependence on $d$. Can the authors comment on the worse dependence in $d$ in their result compared to Deng et al.2014? \n\n\nThe effect of $disc(U)$ is not discussed in detail in the paper and it's not really clear how it quantitatively affects the bound, except that it's zero if the distributions in $U$ are equal.", "other_comments_or_suggestions": "typo(?): how is $||W_Q,W_K,W_V||_F$ defined in lemma 4.16? \n\ntypo in line 186: samples\n\nthe title \"experiments\" is line 427 seems redundant.\n\nIn the related work section, it seems appropriate to include comparisons with previous studies and clarify how your results differ from them.", "questions_for_authors": "please see sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JZZahFCiAu", "reviewer": "Reviewer_fzti", "summary": "This paper presents new Theoretical results to study the generalization of decoder-only transformer based LLM models. The paper revolves around using a Rademacher complexity argument to bound the generalization risk of a multi layer transformer model with fixed position encoding used for tokens. This work focuses on NTP pretraining of LLMs while taking into account the dependence between tokens. The paper then provides results for the covering number of their decoder model and finally bound the pretrained model's generalization using a Racdemacher complexity upper bound derived using the covering number. Large portion of the paper's work is regarding the derivation of the covering number, taking a step by step approach from each component of attention and ffn and building up to the full transformer decoder model. \n\nThe result provide a upper bound for the model with an experimental result to confirm the validity. As expected, increasing number of tokens and context size help close the test vs train generalization gap. Additionally the experimental results regarding fixing datapoints and increasing number of parameters suggest that the model tends to overfit on the limited data, suggesting that any scaling on parameters should be followed by a scaling in number of datapoints.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, thank you very much for your recognition and support of our work. We have incorporated your valuable suggestions by adding a comparative discussion of related past and recent work in the appendix and have made every effort to supplement our experiments. Additionally, we are more than willing to address any questions you have about this work:\n\n**A1:** The expression $\\ln(1+x/e)^{0.5}$ can be seen as $g=f^{0.5}$ where $f=\\ln(1+x/e)$. The function $g$ is \"continuously concave (downward).\"\n\n**A2:** This uses a property of the softmax function: $\\sum\\_{j\\ne i} e\\_j=1-e\\_i$, so the expression inside the parentheses becomes $2(1-e\\_i)$.\n\n**A3:** Please note the definitions of $\\mathcal{C}\\_{Q}$ and $\\mathcal{C}\\_{K}$ in line 1093, where the input data is any $\\mathbf{Z}$. Here, $||\\mathbf{Z}^*||\\_F$ corresponds to $||\\mathbf{X}||\\_F$ in Lemma C.4, as we take the maximum value of the norm for the upper bound. Meanwhile, $||\\mathbf{Z}\\_{[N]}||\\_F$ results from substituting the value of $\\epsilon\\_{Q}$.\n\n**A4:** This is an excellent question! Since we are discussing the pre-training phase, as described in our paper and the paper[6], this phase is not autoregressive. Autoregression is the mechanism used during model reasoning. In the pre-training phase, $m$ queries (tokens) are input simultaneously, and $m$ output tokens are produced simultaneously. Due to the masking mechanism, each query can only use preceding information, mimicking the autoregressive mechanism without seeing future information.\n\n**A5:** In line 1618, the right side of the inequality is a lower bound. Theoretically, $\\alpha$ should be chosen to minimize this expression, i.e., where the derivative is zero. However, since deriving the expression with respect to $\\alpha$ is complex, we set $\\alpha=\\frac{1}{\\sqrt{Nmd}}$. Although this does not reach the lower bound, it simplifies derivation and analysis.\n\n**A6:** The training data used is too limited, so dropout is added to prevent overfitting, similar to the experimental setup in [1].\n\n**A7:** This does not involve any parameter optimization issues; it is solely to analyze the impact of parameter changes on model performance based on experimental results. We have clarified this in the paper.\n\n**A8:** Thank you for your question. To explore the optimal selection of parameters $N$ and $m$ under a fixed total number of tokens, as you suggested, we should vary the sequence length while keeping the total number of tokens constant. Our experiments aimed to verify theoretical results, but indeed, the total number of tokens increased. We are working on this exploratory experiment and aim to present the results in the final version.\n\n**A9:** Many empirical studies indicate that increasing model parameters excessively, while keeping the number of pre-training tokens constant, can harm model performance and generalization ability. Both Chinchilla's Law [2] and Scaling Laws [3] suggest that token numbers should increase alongside model parameters. This may be because larger models with fixed training tokens are more prone to over-parameterization and overfitting. As research in Nature paper [5] shows, for simple tasks with fewer tokens, large models may \"memorize\" noise or spurious patterns in the training data rather than learning underlying general rules, leading to poorer performance.\n\n**A10:** Your understanding is correct. For example, in the sentence \"I went to the restaurant today, the food was delicious, and after eating, I went to the park, which was crowded,\" let event A = \"I went to the restaurant today,\" B = \"the food was delicious,\" and C = \"the park was crowded.\" According to the definition of $\\varphi$-mixing, the time step $k\\_1$ between A and B is much smaller than the time step $k\\_2$ between A and C, so we have $\\varphi(k\\_1)>\\varphi(k\\_2)$, indicating that the dependency between A and B is stronger than between A and C. The dependency decreases as the time step $k$ increases.\n\n**A11:** For the response regarding `disc(U)`, please refer to response **A3** in our reply to **Reviewer axCZ.**\n\n**Reference:**\n\n[1]Edelman, B.(2022) Inductive biases and variable creation in self-attention mechanisms.\n\n[2]DeepMind.(2022) Training Compute-Optimal Large Language Models.\n\n[3]Open AI.(2020) Scaling Laws for Neural Language Models.\n\n[4]Penedo, M.(2023) The Refined Web Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only.\n\n[5]Zhou, L.(2024) Larger and more instructable language models become less reliable.\n\n[6]Bachmann, G.(2024) The pitfalls of next-token prediction.", "claims_and_evidence": "For the most part the claims in the paper are theoretical, followed by solid proofs and foundations. The main result that can be supported with experiments are the ones w.r.t Thm 4.22 on the generalization. In this regard, while the results are promising for the real data experiments, I would like to see further ablation studies or perhaps additional experiments with synthetic data. Given that the upper bounds are not tight, it would be beneficial to include additional toy experimental results. However, I don't believe this is necessary given the highly theoretical aspects of the paper and the limited assumptions made on model/data.", "methods_and_evaluation_criteria": "More of a theoretical paper so evaluation criteria for comparison to other methods and models is not as critical.", "theoretical_claims": "I have gone through the proofs of the paper for the most part. From my understanding I haven't found significant issues, but I would like some clarifications on certain parts of the proofs. Namely: \n\nQ1: In lemma C6, when the authors discuss \"continuous concave (downward)\", what condition on the function are they applying ? This is later applied to the function $ln(1+x/e)^{0.5}$ for the proof of Thm 4.22. \n\nQ2: For the proof of C9, on line 1055, shouldn't it be an inequality ? I don't think it has major significance but just asking to make sure I didn't miss anything.\n\nQ3: For the proof of C10, on line 1133 I missed how $||Z^*||_F$ into the bound. Could you please elaborate ? \n\nQ4: For the proof of Thm 4.22, can the authors elaborate on the comment made on line 1582 regarding the masking of queries. I believe this is an important part of the proof and I don't exactly understand the argument. I don't understand how the current formulation incorporates the auto regressive nature ? \n\nQ5: Proof of Thm 4.22, could the authors please elaborate on why the inequality is valid for line 1621 when they set$\\alpha=\\frac{1}{\\sqrt{Nmd}}$ ? Specifically for the second term. \n\n\nTo be clear, whenever authors use results or lemmas from previous work, I took it for the most part at face value.", "experimental_designs_or_analyses": "As mentioned previously, the experimental results provided on the real data setup confirms the theoretical findings. However, I do believe providing more results with synthetic data could help improve my confidence. \n\nregarding the experimental design, I do have a number of minor questions: \n\nQ6: Why the inclusion of dorpout ? \n\nQ7: What do the authors mean when they suggest grid search for $N$ and $m$ ? I just want to confirm that means changing the parameters not any optimization concern. \n\nWith regards to the results analysis, \n\nQ8: For the conclusion from the first figure to the left, does having larger $m$ imply seeing more tokens ? Its a bit difficult for me to justify the conclusion drawn, given that based on the text, longer context length implies more total tokens seen given that the number of iterations for training are kept the same for all experiments (the same 2000 epochs) \n\nQ9: Given some famous results such as double descent, I do wonder what the implications are where increasing model size hurts generalization performance. Does this mean the model is under parametrized w.r.t to the dataset ?", "relation_to_broader_scientific_literature": "The authors do provide comparison to previous generalization bound for LLM models, having mentioned that their method is the first to consider the NTP pretraining regime.", "essential_references_not_discussed": "None to the best of my knowledge.", "other_strengths_and_weaknesses": "I would like sections in the appendix to further discuss the comparison of their to other bounds for LLMs, such as the ones suggested in table 1. I do understand that the previous work studies different training setups however I do believe such comparisons of bounds could help find potential aspects that impact different training setups.", "other_comments_or_suggestions": "I have given this work a 4 as I can't see any major issues w.r.t the claims made theoretically and given the novelty of the work (w.r.t analyzing NTP pretraining). I would like to see more experimental results and some further discussion regarding the observations.", "questions_for_authors": "Q10: I would like to have a better understanding of what of what $k$ represents for the Def 4.1. To clarify, given a single sequence, $k$ determines how far off the two sample $A,B$ sequence are from one another. Is that correct ? and this is used to help ease the non-iid'ness of sequenced tokens. \n\nQ11: Could the authors expand upon the $disc(U)$ and how it reflects on the quality of data. From my understanding, this term carries the bulk of non-iid'ness of tokens and their relation to each other within one sequence.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "bWIcwBOU21", "reviewer": "Reviewer_astS", "summary": "The authors give a generalization error bound for decoder-only transformer language models trained with next-token prediction objective. The bound is a function of the number of training sequences $N$, number tokens $m$ per such sequence, number of model parameters, etc. Within a sequence, they assume that tokens are dependent but follow certain mixing conditions (Definition 4.1). They further justify the error bounds with simulations on a small dataset.\n\n**Recommendation**\n\nThe bound has a $O(1/\\sqrt{m})$ term. If the sequence length is fixed, then even as the number of sequences $N \\rightarrow \\infty$, the bound does not tend to 0. This appears to make the bound weaker than bounds in the literature which rely only on the number of sequences (scaling like $O(1/\\sqrt{N})$. Therefore, I recommend a reject unless the authors have some strong arguments why their bound is the tightest so far.\n\n**Questions / Comments**\n\n1. It seems the generalization error bound is at least as large as $O(1/\\sqrt{m})$, even as the number of training sequences $N\\rightarrow \\infty$ no matter how large the model is. This seems to make the bound weak in practical scenarios. Specifically, the bound never goes to 0 if the sequence length is fixed. Can the mixing assumption explain this irreducible bound? Is there a worst-case mixing scenario where the generalization error is $O(1/\\sqrt{m})$ however large $N$ is?\n\n\n2. The generalization error bound (in Theorem 4.22) increases as the model size increases. But in practice, larger models have smaller generalization error. How do we understand this apparent contradiction? I see Figure 2 (rightmost) plot supporting the error bound. However, I am not convinced that this is how generalization error scales in general. Is there a certain regime, such as low-$Nm$ where this happens?\n\n3. How do we interpret the discrepancy term $\\mathrm{disc}(U)$? One could set $\\bar \\phi = \\frac{1}{N} \\sum_{j=1}^N \\phi_j$ and think that we draw sequences from the same $\\bar \\phi$ distribution. Then $\\mathrm{disc}(U)$ would be $0$. What other term in the bound increases with this reparametrized $\\phi$? The mixing term $\\Delta_m$?\n\n4. Minor notation issue: Is the first argument of $\\ell$ token probability vector, as suggested by equation (7)? Or is it a token, as suggested by the text right after eq (1)? In Assumption 3.4, the Lipschitz smoothness assumption on $\\ell$ needs further clarification based on this answer. \n\n5. Can the authors justify the mixing assumption in Definition 4.1 with some simulations? \n\n6. In the experiments section, the fact that the bounds are in the same order of magnitude as the generalization error is surprising. To compute the generalization bound from Theorem 4.22, what values were used for the various quantities in the bound? Can the authors explain why they train for a relatively high 2000 epochs?", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and questions. We have carefully considered them and have added supplementary explanations in the relevant sections of the paper. Additionally, we have incorporated the suggested references into our paper. Below are our responses:\n\n**A1:** This is indeed a significant question, and we fully understand your concerns. We must acknowledge that the sequence length $m$ is constrained by available training resources. However, from a theoretical standpoint, we assume that the sequence length $m$ can approach infinity. Firstly, from the intrinsic nature of human language, language sequences can indeed extend indefinitely, much like time series.\nFurthermore, based on the two papers [1, 2] we are aware of that also consider the generalization bounds of language models at the token level, their bounds include $\\mathcal{O}(\\sqrt{1/m})$, although they do not specifically target the NTP pre-training task. For instance, in Theorem 4.3 of paper [1], there is a term $\\mathcal{O}(\\sqrt{\\frac{1}{T_p}})$, where $T_p$ represents sequence length, similar to $m$. Paper [2] also considers dependencies between tokens, and thus in their Theorem 1, the generalization bound is $\\mathcal{O}(\\sqrt{1/m})$. We hope this explanation alleviates your concerns. Additionally, we will enhance the discussion on sequence length in our paper and look forward to reaching a consensus with you.\n\n**A2:** It is true that larger models often perform better in practice. However, according to the scaling laws for large models [3], larger models also require more tokens for pre-training.  In the third experiment depicted in Figure 2 of our paper, we observed that when the total number of tokens is fixed, larger models tend to generalize worse. This is primarily because the limited number of tokens leads to overfitting in larger models. This observation aligns with our theoretical results.\n\n**A3:** For our detailed explanation regarding disc(U), please refer to ​Response **A3** in our reply to **Reviewer axCZ**. Furthermore, $C_{\\varphi,r}$ represents the upper bound of $\\Delta_{m}$ ( Remark 4.10) and also varies with $\\phi$, measuring the diversity of the token sentences. We aim for a smaller $C_{\\varphi,r}$ to enable the model to learn more diverse knowledge. We will further explore `disc(U)` and $\\Delta_{m}$ in the discussion section of the paper.\n\n**A4:** In Section 3.1 of the paper, we clarify that all tokens $\\mathbf{t}^i_j \\in \\mathbb{R}^{n_v}$ are vectors, meaning that the first parameter of the loss function $\\ell$ is the probability vector computed by Equation (7). \n\n**A5:** We apologize for not being able to experimentally demonstrate in a short time frame that human language sequence is a mixing process. However, we provide two theoretical supports: (1) In numerous existing NLP studies, language sequences are often modeled as Markov processes [5,6]. Under certain conditions, Markov chains are indeed mixing processes [7]. (2) Under certain conditions, autoregressive processes are equivalent to mixing processes [8].\n\n**A6:** Regarding the calculation of the error bound, we followed the approach outlined in [9], our computed generalization bound is $\\sqrt{\\frac{\\Theta \\tau_{1}}{Nm}}$. (1) We set the $G_{\\pi}$ to 1 based on Lemma A.9 in [10]; (2) $B_{l}$ and $C_{l}$ were calculated by extracting the model parameter matrix and the attention matrix during training. \nWe set the epoch to 2000 to ensure the model converges as much as possible [11], as smaller datasets require more epochs for generalization [12]. The primary purpose of this experiment was to demonstrate that even with a limited number of training tokens, our generalization bound remains valid and does not collapse as discussed in [9]. Additionally, we are actively working to supplement our study with more experiments.\n\n**Reference：**\n\n[1]Gong.(2025) Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization.\n\n[2]Lotfi.(2024) Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models.\n\n[3]Open AI.(2020) Scaling Laws for Neural Language Models.\n\n[4]Penedo, M.(2023) The Refined Web Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only.\n\n[5]Zhang, Y.(2023) What and how does in-context learning learn? Bayesian model averaging, parameterization, and generalization.\n\n[6]Li, Y.(2023)Transformers as algorithms: Generalization and stability in in-context learning. \n\n[7]Meyn.(2012) Markov chains and stochastic stability.\n\n[8]Krishna.(1986) Mixing properties of harris chains and autoregressive processes.\n\n[9]Galanti.(2023) Norm-based Generalization Bounds for Sparse Neural Networks.\n\n[10]Edelman.(2022) Inductive Biases and Variable Creation in Self-Attention Mechanisms.\n\n[11]Madden.(2024) Upper and lower memory capacity bounds of transformers for next-token prediction.\n\n[12] Power.(2022) Grokking: Generalization beyond overfitting on small algorithmic datasets.", "claims_and_evidence": "Please see above.", "methods_and_evaluation_criteria": "Please see above.", "theoretical_claims": "Please see above.", "experimental_designs_or_analyses": "Please see above.", "relation_to_broader_scientific_literature": "See above.", "essential_references_not_discussed": "It would good to add this reference which also gives error bounds for language models:\n\nSanae Lotfi, Yilun Kuang, Marc Anton Finzi, Brandon Amos, Micah Goldblum, Andrew Gordon Wilson,\n\nUnlocking Tokens as Data Points for Generalization Bounds on Larger Language Models, 2024, NeurIPS 2024", "other_strengths_and_weaknesses": "Please see above.", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors establish a Rademacher complexity bound for the excess risk of NTP training and derive covering number bounds for the function space of transformer-decoder models, thus yielding a generalization bound for decoder models. Within a sequence, they assume that tokens are dependent but follow certain mixing conditions.

Three reviewers remain positive about the paper after raising several technical questions which have been successfully addressed by the authors. The fourth reviewer raised concerns regarding suboptimality of the bound, particularly with respect to previous works, in that the bound scales as 1/\sqrt{m} where m is the size of the sequence and thus won't go down as the number of sequences N increases. The authors have in my opinion successfully clarified that results in referenced previous works are of similar nature and have also clarified that compared to previous works they here tackle the challenge of sequence dependencies (albeit using standard assumptions on mixing conditions).

Overall, I suggest that the authors take care in revising their manuscript according to the feedback. Please include all additional explanations given during rebuttal to make the results more interpretable and clarify the technical contributions compared to previous works. Edge cases such as those asked by Rev. astS on what happens when all \phi_i are equal are useful and should be included in the revision. Additional discussions regarding the experiments as included in the rebuttal should also be added.

That said, study of pretraining generalization properties is an understudied topic and this paper makes a meaningful step in this direction. Given this and the overall positive feedback, I recommend acceptance.
Decision: Accept (poster)