Impact of Layer Norm on Memorization and Generalization in Transformers
Authors: Rishi Singhal
Jung-Eun Kim
Venue: NeurIPS 2025 poster ()
Keywords: Memorization
Generalization
Transformer
Layer Norm
Primary area: deep_learning

Abstract: Layer Normalization (LayerNorm) is one of the fundamental components in transformers that stabilizes training and improves optimization. In recent times, Pre-LayerNorm transformers have become the preferred choice over Post-LayerNorm transformers due to their stable gradient flow. However, the impact of LayerNorm on learning and memorization across these architectures remains unclear. In this work, we investigate how LayerNorm influences memorization and learning for Pre- and Post-LayerNorm transformers. We identify that LayerNorm serves as a key factor for stable learning in Pre-LayerNorm transformers, while in Post-LayerNorm transformers, it impacts memorization. Our analysis reveals that eliminating LayerNorm parameters in Pre-LayerNorm models exacerbates memorization and destabilizes learning, while in Post-LayerNorm models, it effectively mitigates memorization by restoring genuine labels. We further precisely identify that early layers LayerNorm are the most critical over middle/later layers and their influence varies across Pre and Post LayerNorm models. We have validated it through 13 models across 6 Vision and Language datasets. These insights shed new light on the role of LayerNorm in shaping memorization and learning in transformers.

Impact of Layer Norm on Memorization and Generalization in Transformers
Rishi Singhal
Jung-Eun Bun Kim Department of Computer Science North Carolina State University
Department of Computer Science
North Carolina State University
Abstract
Layer Normalization (LayerNorm) is one of the fundamental components in trans- formers that stabilizes training and improves optimization. In recent times, Pre- LayerNorm transformers have become the preferred choice over Post-LayerNorm transformers due to their stable gradient flow. However, the impact of LayerNorm learning and memorization across these architectures remains unclear. In this work, we investigate how LayerNorm influences memorization and learning for Pre- and Post-LayerNorm transformers. We identify that LayerNorm serves key fac- tor for stable learning in Pre-LayerNorm transformers, while in Post-LayerNorm transformers, it impacts memorization. Our analysis reveals that eliminating LayerNorm parameters in Pre-LayerNorm models exacerbates memorization and destabilizes learning, while in Post-LayerNorm models, it effectively mitigates memorization by restoring genuine labels. We further precisely identify that early layers LayerNorm are the most critical over middle/later layers and their influence varies across Pre and Post LayerNorm models. We have validated it through 13 models across 6 Vision and Language datasets. These insights shed new light on the role of LayerNorm in shaping memorization and learning in transformery
Introduction
Layer Normalization [Lei et al. 2016] greatly contributes stabilizing training and optimizing performance in deep learning models, especially in transformers. It works by normalizing the activations each layer, ensuring a more consistent gradient flow during training. In recent years, transformers are primarily designed with two options by LayerNorm (LN) placements: Pre-LN and Post-LN. Post-LN transformer, introduced by Vaswani al. [2017). applies normalization after the addition of the layer's output with the residual connection's 's output and has been showing competent performance in language modeling and machine translation. However, due to the issue unstable gradient flow [Liu et al.) 2020] in Post-LN models, Pre-LN transformers [Xiong et al. [2200] were introduced, where normalization is applied before self-attention and forwwrd layers. This configuration stabilized training and achieved faster convergence by improving the gradient flow, making the preferred choice modern architectures such as GPT, Llama, and Vision Transformers.
Even though transformers have demonstrated remarkable capabilities in learning rich representations from data, they exhibit a strong tendency memorize some samples due their complex nature, which is commonly known as Label Memorization [Feldman) 2020 Feldman and Zhang 2020). where a model memorizes labels training without learning the relevant patterns that generalize to unseen data, leading to overfitting. Recent studies have explored whether memorization can be localized to specific layers [Maini et al., 2023 Baldock et al., 2021 or components such as attention heads and the
network (FFN) [Haviy et al. 2023 Geva et al. 2023 Yu et al.) 2023 Apart from the attention heads and the feed-forward network (FFN), LayerNorm stands as
* "Correspondigg author
"Code at:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Becovery Score 1%11
Removal
Memorization
Random Presirtions 1%5..
1.00
80
Score 14g
60
60
40
20
20
GPTN80
Owenz
G912
Vn-88
Delt
Vi-5
GPTN00
Queett
OPT1Z
1961
VITS
Model Names
Model Names
Learning (test) accuracy for Pre-LN mod-
(b) Memorization, Recovery Random prediction scores for Pre-LN models
cls
Recovery Score 1%6
Before LN Removal
After LN Removal
Nemorizatio Score (%)
Random Predictions 1%1
100
E80
Accurass
Scare 155
80
20
9530
Longlomer
ResE3tth
Desiiitt
Bletts
DERT
Longformke
RoBeRia
Beeta
Model Names
(c) Learning (test) accuracy for Post-LN mod- els
(d) Memorization, Recovery & Random prediction for Post-LN models
Figure 1: Impact of LN layer on memorization and learning of Pre- and Post-LN models. (a) shows a clear impact of LN in Pre-LN models, whereas (c) shows no impact of the removal of LN parameters in Post-LN models for learning. (b) exhibits that, without LN layers, the Pre-LN models struggle with high memorization and random predictions (red-color-famil) bars), while (d) exhibits that in Post-LN models, removing LN parameters recovers a significant portion of correct predictions (green bars).
pivotal component in the transformer architecture that further shapes its optimization dynamics and performance. Outlier neurons in LN layers have been shown impair transformer performance and hinder quantization Kovaleva et al.
2021 Puccetti et al. 2022 Bondarenko et al.) 2023 He et al.
2024 [Xu et 2019] hints that LN may contribute to increased overfitting Pre-L. models.
In our paper, we identify that in Pre-LN transformers, LN is critical to learning and removal of its learnable parameters exacerbates overfitting and disrupts learning. On the contrary, in Post-LN transformers. LN plays a significant role in memorization, where by eliminating LN parameters, memorization is suppressed by recovering the genuine labels without affecting the model's learning capability. We rigorously validate our claims through various models BERT, Longformer, RoBERTa, DeBERTa, ELECTRÁ. DiST GPTNeo, GPT2, Qwen2, RoBERTa- PIBERTa--PeLLyyeNomne ViT-Base, ViT-Small, and DeiT, across both Vision and Language tasks. In summary, the core findings of our paper regarding the impact of LN on memorization and learning in transformers are follows:
Learning Stability, Memorization Suppression & Label Recovery: We identify that LN is crucial for learning in Pre-LN models, unlike Post-LN models. For Post-LN models, LN learnable parameters removal suppresses memorization and recovers genuine labels, whereas in Pre-LN models, LN removal exacerbates overfitting, with persistent memorization.
Early LNs are Critical: We uncover that removal of LNs parameters in early layers most impactful in mitigating memorization for Post-LN models, and destabilizing the learning in Pre-LN architectures.
Gradients Explain LN's Impact: We explain the divergent impacts of LN in Pre- and Post-LN models by comparing learning and memorization gradients, which reveal why LN parameter removal causes learning disruption and memorization suppression in Pre- and Post-LN models, respectively.

---

2 Related Works
Memorization & Learning: Transformers excel at learning general, simple patterns [Arpit et al.
2017 Shah al., 2020 Zhou and Wu 2023], but also tend to memorize rare, mislabeled, or complex
examples [Stephenson et al. 2021 Baldock et al. 2021 Agarwal et al., 2022 Feldman and Zhang 2220)] Feldman 2020] formally define label memorization, while [Blldock et al. [[021] proposes
prediction depth to capture example difficulty. Other works (Jiang et al. [2020] Ravikumar et al.
2024 Garg al. 2023] associate high curvature and consistency with long-tai -tii led or mislabeles
samples. Beyond identifying memorization, several studies [Haviv al.,
2023 Geva et al. 2023 Dai
ct al. 2021] investigate how self-attention and feedforward layers contribute to factual recall across
transtormer layers. More recent work (Yin et al. 2023 Lad et al. 2024 Men et al. 2024| Li ct al.
2024 Sun et al. 2025] highlights the limited effectiveness of deeper layers on learning in Pre-LN
transtormers. Despite these insights, the distinctive impact of LayerNorm in shaping memorization
and learning across both Pre- and Post-LN architectures remains poorly understood.
Understanding LayerNorm (LN) in Transformers: In addition to self-attention and feedforward networks (FFNss, Layer Normalization (LN) plays a critical role transformer models. Prior work [Brody et al.] 2023 Wu et al.) [024] has demonstrated that LN is essential to the overall expressivity of transformers. Beyond its utility, LN has been found to contain outlier neurons [oovlleaa et al.) 2021 Puccetti et al. 2022], whose removal severely degrades model performance. These outliers have also been shown to hinder the quantization of transformer models [Bondarenko et al. 2023
He al. 2024] Moreover, several studies (Xiong al. 2020 Liu et al. 2020 Takase et al. 2022
Xie et al. 2023 Kim et al.) 2025 have highlighted that Post-LN architectures can cause gradient
instability during training, while Pre-LN configurations may lead to exploding gradients in early
layers prompting the development of techniques (Shleifer et al. [021 Wang et al.. 2022 Kumar
al. 2023 Qi et al., 2023] Jiang al. 2023) to address them. Additionally, Xu et al. 2019
suggested that LN parameters may contribute to overfitting in Pre-LN models.
However, provide a far more nuanced understanding of LN's role: in Pre-LN transformers, LN is essential for learning but not memorization, whereas in Post-LN models, LN is crucial for memorization but not learning. This distinction offers novel contribution understanding the function of LN in transformers for learning and memorization.
3 Prelimnaries
3.1 Understanding LayerNorm in Transformers and Defining Memorization & Learning
LayerNorm Operation. Let x = (I1, ................................ Ta) be the input of size d to the LayerNorm function
LN(x) which first normalizes the input æ as (xx) using mean μ and standard deviation Then it re- re-scales and re-centers (xx) using the learnable parameters (weight) and (bias). The output of
the LayerNorm layer is then given by:
LN(x) w N(s)+b n
Di
x 2)2 N(x)
(I)
i=1
where C denotes the dot product operation.
Pre-LN & Post-LN Transformers. In the Pre-LN Transformer, LayerNorm is applied before each sub-layer- Multi-Head Self Attention (MHSA) and Feed-Forward Network (FFN). On the other hand, in the Post-LN Transformer, LN is applied after the residual connection. We represent the key difference the architectural design of the two configurations as follows:
Pre-LN:
z' -'= MHSA(LN, (x))
Post-LN:
x' LN MHSA(z))
(2)
y =' FFN(LN
y = LN2 (x' FFN(x'))
Understanding Learning and Label Memorization (LM). Deep neural network models, like trans- formers, learn meaningful relationships between features and labels during training and generalize the learned representations to unseen test data the phenomenon
well understood as learn- ing/generalization. At the same time, these models also have the tendency to memorize training data points which are complex in nature, commonly known as label memorization (LM) [Feldman) 2020 Feldman and Zhang 2020), where the model just memorizes the labels during training without
3

---

capturing meaningful patterns that generalize to new data, resulting in overfitting, Label memorization known occur due to multiple factors such as complex, ambiguous features, and noisy labels BBllockk et al., 2021). which makes it difficult for the model to learn any meaningful relationship.
In this work, we specifically focus on introducing noisy labels way to study memorization, where we change the label of a particular class sample to a randomly chosen class label that is different from its original label. To ensure that the noisy label samples are memorized, we train the model until achieves 100% training accuracy. Throughout our experiments, we introduce random label noise in all datasets by modifying 1% of the training set labels, maintaining consistency across evaluations.
3.2 Investigating LayerNorm (LN) Impact on Memorization and Learning
Removing LN parameters. To examine the role of LayerNorm (LN) in memorization and learning within transformers, we analyze the effects of omitting its learnable parameters. during training. This provides insights into how LN influences the balance between learning and memorization for both Pre- and Post-LN models. We precisely analyze the impact of LN on memorization and learning in Sec. Please note that we use LN removal and LN parameters removal interchangeably in our paper. They both refer to removal of learnable parameters of the LN layer, while keeping the normalization operation, (xx), intact.
Effect of LN Removal across Layers. To further understand the impact of LN at different stages of the model, we categorize the layers into early, middle, and later layers (described in detail in Appendix F.4. We then selectively remove LN parameters from one set at a time to analyze how their absence affects learning and memorization. This analysis reveals which set of LNs the most influential towards memorization and learning behaviors in Pre- and Post-LN transformers. The experiments and results are discussed in Sec. [5]
LN Gradients Analysis across Layers. To support our observations on the influence of LN, we compute the gradient of the loss function (L) with respect the input of LN, J. represented as ac This measure quantifies how much the input to the LN layer affects the model's loss, thereby its learning and memorization ability. To understand the sensitivity of each layer's LN towards memorization and learning, we compute the L2-norm of this gradient (i.e., 119= Specifically, to quantify sensitivity towards learning. i.c., how the model generalizes the patterns to the test set, compute 1l9 for every test-set sample and average it across all of them, obtaining learning gradient norm, denoted by |[ For memorization, we compute | llz for each of the noisy labels samples that we injected into the train set and then averaged across all such noisy samples to obtain memorization gradient norm, denoted by [lg 122
V.C 9z 27.
A higher gradient norm indicates that the layer's ss LN significantly influences the model's 's ability to memorize or learn, while a lower gradient suggests minimal impact. The discussion of memorization and learning gradients and their significance is shown in Sec.
3.3 Key Metries: Learning Accuracy Memorization, Recovery & Random Predictions Score
To evaluate the impact of LN on the learning and memorization ability of the transformer models, we focus several key metrics that provide insights into their behavior and effectiveness in the presence of noisy labels during training.
Learning (Test) Accuracy ( (%) refers to the model's performance on the test set, depicting how well generalizes the learnt relationships to unseen data, marking it as a core indicator of the model's learning progress ( ##Comeet predictions test
# Total test set samples x 100). A high learning accuracy signifies that the model has learned meaningful patterns and is able to generalize well to unseen data. On the contrary, a low learning accuracy depicts poor generalizability.
Memorization Score (%) serves as an indicator of the model's tendency to memorize noisy labels that are irrelevant or erroneous, rather than genuinely learning the true underlying relationships #Noisy label samples memorized x 100). A high memorization score indicates that the model has overfit the # Total noisy label samples
noisy labels, effectively "memorizing" them.
Recovery Score (%) is crucial metric that helps in understanding the impact of LayerNorm (LN) on memorization of noisy labels. It measures the model's 's ability to recover the genuine, true labels after the removal of LN parameters # Recoverrd noisy label samples as true labels
x 100). A high recovery # Total noisy samples

---

Table 1: Summary of the impact of LN layer in Pre- and Post-LN Models.
LN Removed
Learning Intact?
Memorization Mitigated?
Recovery Happens?
Pre-LN Model
Learning Disrupted
X Memorization Still Present
Negligibbe Recovery
Post-LN Model
Stable Learning
Memorization Mitigated
Genuine Labels Inferred
score indicates that the model can successfully recover the original, correct labels by suppressing memorization.
Random Prediction Score (%) measures the percentage of noisy label samples whose predictions were changed to random labels after the removal of LN parameters. These predicted random labels are neither genuine nor the noisy label # Random prodictions of noisy label samples
x 100). Although this is not ideal, it provides a complete picture of the impact of LN parameters removal and indicates the extent to which the model can recover the true labels. A high percentage of random predictions suggests that the model struggles to recover the true labels effectively.
# lotal noisy label samples
3.4 Datasets Models Used
We empirically verify all claims and show extensive results against both language and vision modali- ties, including 3 language and vision classification datasets, and Pre-LN and Post-LN transform-
ers architectures, as follows:
Datasets: CIFAR10 (Krizhevsky et al.
2009
NICO+ [Zhang al.
2023 UTK-Face hZhnng TweetTopic (Antypas et al.
al. 2017 Emotions
(Saravia et al. 2018 News 9kkite97 2024 and
2022
Post-LN Models: BERT Devlin et al.
2019 RoBERTa Yinhan et al.
2019], DistilBERT SSanh Longformer
et al.
2019]. DeBERTa
He et al. 2020
ELECTRA
Clark 2020]. and
Beltagy et al.
2020
Pre-LN Models: ViT-B [Alexey] 2020]. ViT-S [Assran et al. 2022]. DeiT
Touvron et al. 2021]
GPT2 (Radford et al., 2019]. GPT-Neo [Black et al. 2022 Qwen2 Yang al. 2024 and RoBERTa-
Prel.ayerNorm [Ott et al. 2019].
It needs to be acknowledged that only language modality is available for the Post-LN architec- ture in practice/literatuee. We provide a thorough discussion of the datasets, models, and training configurations in Appendix F All our experiments run across 3 random seeds.
Impact of LN on Memorization and Learning
In this section, examine the distinct impact of Layer Normalization (LN) memorization and learning Pre-LN and Post-LN transformers. To assess its influence, we train two versions of the model with LN parameters removed and with them intact and compare their performance using learning accuracy, memorization, recovery, and random prediction scores.
4.1 Learning Stability
From Figs. Tal& Ic. we observe that removing LN parameters in Pre-LN transformers significantly disrupts learning, whereas Post-LN transformers remain robust, maintaining their learning accuracy even after LN parameter removal.
This discrepancy becomes even more evident when analyzing the progression of learning in epochs, as depicted for Qwen2 (Pre-LN) in Fig. 2a & ELECTRA (Post-LN) in Fig. 2d For Qwen2, once learning is disrupted by LN parameters removal, it does not recover till the end of training, indicating fundamental instability. However, ELECTRA maintains stable learning throughout training, showing no signs of degradation, further highlighting its resilience to LN parameters removal. Similar results are observed for other Post-LN (BERT, DeBERTa, Longformer, RoBERT) and Pre-LN models (GPT2, GPTNeo, ViT-B, DeiT, VIT-S), as shown in Appendix G.T
4.2 Memorization Suppression & Label Recovery
We now examine the role of LN in memorization and label recovery. From Figs. lb & 1d
we observe
that Post-LN models, IN governs memorization, its parameter removal mitigates memorization and enhances true label recovery, reflected in lower memorization scores and higher recovery scores.
5

---

20
Trat Acuurayy
frannng
Speets
(a) Learning (Test) accuracy over
(b) Memorization, Recovery and Ran- (c) Overfitting gap Pre-
epochs for Pre-LN Model (Qwen2)
dom Predictions over epochs for Pre-
LN Model (Qwen2)
LN Model (Qwen2)
IN present
removed
(d) Learning (Test) accuracy epochs for Post-LN Model (ELEC- dom Predictions over epochs for Post- TRA)
(c) Memorization, Recovery and Ran-
(f) Overfitting gap for Post- LN Model (ELECTRA)
Model (ELECTRA)
Figure 2: LN removal destabilizes learning in Pre-LN models, while mitigates memorization in Post-LN models (News Dataset): LN removal in Pre-LN models critically affects learning (accuracy gap in (a)) while Post-LN models remain robust (negligible gap in (d)); LN removal helps in effective mitigation of memorization and high recovery in Post-LN models (green bars in (e)), while memorizatiin/rauddom predictions still persist Pre-LN models (red-color-family bars in (c)); LN removal in Pre-LN models exacerbates overfitting explained by increasing train-test accuracy gap (c), and for Post-LN models it decreases due memorization mitigation (see (f)).
In contrast, for Pre-LN models, LN parameters removal does not mitigate memorization, indicated by persistently high memorization and random prediction scores.
This effect is even clearer when analyzing memorization over epochs, as shown for ELECTRA (Fig. (2ep and Qwen2 (Fig. 2b). ELECTRA, memorization decreases over time, with label recovery improving training progresses. Conversely, in Qwen2, memorization persists throughout training, and label recovery remains poor, indicating that LN parameter removal does not suppress memorization or aid label recovery in Pre-LN models. Similar patterns observed in other Post-LN (BERT, DeBERTa, Longformer, RoBERT) and Pre-LN models (GPT2, GPTNeo, ViT-B, DeiT, ViT-S), as shown in Appendix G.I These findings offer nuanced perspective on LN's role: it is crucial for learning in Pre-LN models but does not influence memorization, contrary to prior work [Xu et al. 2019]. which suggested that LN in Pre-LN models contribute to overfitting.
In summary, LN is essential for stable learning in Pre-LN models, hence its parameters removal significantly destabilizes learning and widens the train-test accuracy gap PAPre Aeggit) i.e., exacerbating overfiting/memooriztiion as illustrated in Fig. 2c In contrast, Post-LN models, LN parameters removal suppresses memorization and enhances true label recovery, thereby narrowing the train-test accuracy gap (A Post as shown Fig. 2 This distinction is further illustrated Table which provides a comparative overview of LN's role in learning and memorization across Pre-LN and Post-LN model. Similar observations are observed other Pre-LN (GPT2, GPTNeo, ViT-B, ViT-S, DeiT) and Post-LN (BERT, RoBERTa, DeBERTa, Longformer) models reported in Appendix G.I
5 The Pivotal Impact of LN in Early Layers
Building on the observation that LN has distinctive impacts on learning & memorization for Pre- and Post-LN models, respectively, later LN layers on Pre- and Post-LN models. Fig. depicts that early LNs are more significant than middle/later LNs in driving learning and memorization in both Pre- and Post-LN models.
precisely investigate the impact of the early, middle, and
In the Pre-LN model (DeiT, Fig. Baj, the removal of early LNs parameters significantly disrupts the learning process, highlighting their importance in learning for Pre-LN models. In the Post-LN model

---

Exily Remrovdd
Midd Lis femoved
Lacer Berroved
Toxt Acc when Presert
Sunn present
Nocovey
(a) Impact of carly, middle, later LNs on learning (test) accuracy, memorization, recovery and random predicitons scores for Pre-LN models (DeiT, UTK-Face)
Eaily Via Ramovad
Vidd Lite temoved
Later Lha Removvdd
Text Acc when LN Present
... Aowar shan LN present
Aowert
(b) Impact of carly, middle, later LNs on learning (tcst) accuracy, memorization, recovery and random predicitons scores for Post-LN models (DeBERTa, Emotions)
Figure 3: Pivotal impact of early LNs for learning and memorization across Pre- and Post-LN models. (a) clearly shows impact of early LNs removal on destabilizing learning in Pre-LN models, accompanied with higher train-cestaaccuracy gap, A overii early LNs removal help in suppressing memorization and improving recovery in Post-LN models, alongwith lower train-cest-accuracy gap,
early than later layers, whereas (b) shows
Post, early than later layers.
everlit
(DeBERTa, Fig. 3b). the removal of early LNs parameters mitigates memorization and enhances true label recovery most significantly compared to the cases of middle later layers. This contrast highlights the pivotal impact of early LNs in shaping learning and memorization dynamics, positively in Post-LN and negatively in Pre-LN models. Aligned trends are observed for other multiple Pre-
and Post-LN models, as presented in Appendix G.2 Prior studies [Gromov et al. 2024 Li et al.) 2024 Lad et al./ 2024 Men et al.) 2024] highlighted the limited effectiveness of deeper layers for learning in Pre-LN models. Our observations take this a step further by precisely identifying that LN in the early layers is critical factor memorization in Post-LN models, presenting novel and
distinctive observation.
The distinctive effect of early LNs parameters ditoovallddssuiiingggggga learning in Pre-LN models while mitigating memorization in Post-LN models-is further supported by the train-test accuracy gap (A (verrtit) Specifically, in Pre-LN models, removing them leads to
pronounced increase early compared to middle or later LNs, whereas Post-LN models, removing early LNs A overitt
parameters results in a sharper decrease in A poen Post,earl early
This trend is shown in Fig, (bar plots) and
formalized follows:
Pre, carly overfit
middle
Pre, overfit
and
Post. carly overfit
Post, midilee overfit
A Avee
overfit
In summary, observe that early layers LN more significant than later layers LN, where their removal disrupts learning, explained by high A Are, early in Pre-LN models. On the other hand, their removal suppresses memorization while recovering true labels in Post-LN models, illustrated by low
Post, early Similar trends are observed in other Pre-LN (GPTNeo, Qwen2, GPT2, ViT-B, ViT-S) and
overfit
Post-LN (BERT, RoBERTa, ELECTRA, Longformer) models as illustrated in Appendix G.2
Gradients Explain LN's Impact
To better understand the role of layer normalization (LN) in learning and memorization, we compute the gradient norms associated with both processes across different layers (gu). Specifically, we measure the norms for learning (gleam) and memorization (g.ee) gradients separately, allowing to quantify their relative contributions throughout the network.

---

Gradient Norm
bmmb
Gradient Norm
Rotlo
Ratio
Cocs3
Ratio
3
0.03
0:42
0
1.45
182
Layer Index
Layer Index
(a) Pre-LN (GPTNeo)
g Jearn
l1g m
analysis
(b) Post-LN (DeBERTa) 9 leam
ll2
lz analy-
acTOSS layers
sis across layers
Figure 4: Learning Memorization Gradients in Pre- and Post-LN Models: (in Emotions Dataset) Results clearly exhibit high gradient norms of early layers LNs than later layers for both learning and memorization Pre-LN (GPTNeo) and Post-LN (DeBERTa) models. Importantly, the learning gradient norm ([g. ( is consistently stronger than the memorization gradient norm (Igg 222 across all layers. Furthermore, the ratio ||g]_am / |lg ll is significantly higher in Pre-LN models compared to Post-LN models.
Theorem 1 (Learning Gradient Norm, leurni 2 is greater than or equal Memorization Gradient Norm, 9 x
2 across all layers). It is formally represented as follows:
Jeam |12 2 |g
across all layers
(4)
A proof of Theorem is provided in Appendix B
From Theorem 1, we observe that the learning gradient norms are generally greater than the memo- rizing gradient norms across all layers for both Pre- and Post-LN models. This observation is further validated empirically from the trend as seen in Fig.
6.1 Understanding the Distinctive Impact of LN in Pre and Post-LN Architectures
Having identified importance of early layers LNs (Sec. 5), we now focus on explaining why the removal of Layer Normalization (LN) in Pre-LN models hinders learning, while in Post-LN models, mitigates memorization without disrupting learning. To do so, we focus on the ratio of
seaming-to-memoriiaee gradients norms
red-color line plots in Fig. & 4b across layers.
[9
Based on the results, we uncover the following phenomenon:
Jearn
1
Jeam
across all layers
(5)
mem
Pro-LN
POSE-LN
This indicates that in Pre-LN models, LayerNorm primarily facilitates learning, as evidenced by the dominance of ||g][am| over [|9_-==|| ||2. Consequently, the removal of its parameters disrupts learning and exacerbates overfitting. In contrast, in Post-LN models, ||g]_mm|| and || are of comparable magnitudes. As result, removing L.N parameters effectively mitigates memorization by restoring genuine labels without disturbing learning. Consistent trends are observed for other Pre-LN (GPT2, Qwen2, ViT-B, DeiT, ViT-S) and Post-LN (RoBERTa, BERT, Longformer, ELECTRA) models, illustrated in Appendix G.3
6.2 Why are LNs in Early Layers Important for Memorization and Learning?
In this section, we explain why the early layers LN are pivotal in governing memorization and learning across Post and Pre-LN models, through the lens of gradient analysis.

---

Theorem 2 (Gradient norm of loss L w.r.t input of LN is upper bounded).
Post-LN: Let 2; denote the input to L of the 7th Post-LN model layer. Then,
aC 3z
1 1 1 (ar(MHSSA(Z))
11 1a =
1
Var(FFN(22)))
N
N
II
Smax (Friv) [1
II -=++1
(1 Smax (J (HSS))
(6)
Pre-LN: Let Ti denote the input to LN1 of the tth Pre-LN model layer. Then,
N
N II
11 112 =
0L
Smax
(P2)
II
Smax (N (1
LNg
(1+
Smax (N
JIN
f=1
j=i
A proof of Theorem 2 along with the expressions for LN₂ in both Pre- and Post-LN setup, is provided
in Appendix C
Theorem 3 (Upper bound of the gradient norm of Early Layers LN are higher than those of Later layers LN). is formally represented as follows:
UB(Igg, 112)
UB( 12) 2 ... UU([19zxlle) 112); for both Pre- and Post-LN models
(8)
where UB (2) denotes the upper bound of 92z ll2 and x; is the input to the ith layer's 's LN.
proof of Theorem 3 provided in Appendix D
The results depicted in Fig/4 eemprricll confirm the trend established in Theorem 3 Specifically, we observe that both [|g]_m]]] 12 and ||,9==1|| are significantly higher in the earlier layers compared the later ones. This gradient decay trend is consistent across both Pre-LN (GPTNeo, Fig 4a) and Post-LN (DeBERTa, Fig[4b) architectures. Similar trends are observed for other Pre-LN (GPT2, Qwen2, VIT-B, DeiT, ViT-S) and Post-LN (RoBERTa, BERT, Longformer, ELECTRA) models, as shown in Appendix G.3
Thus, the theoretical upper bounds not only provide an analytical explanation for the gradient magnitude behavior but also align closely with the empirical patterns observed across a wide range of transformer variants. This alignment helps explain why the removal of early layers LN parameters leads to disruption of learning in Pre-LN models and mitigation of memorization in Post-LN models, highlighting their predominant role in the entire network because of their higher gradient norm.
In addition to the isolation of learning and memorization early layers, we observe another interesting pattern. For Post-LN models (Fig. 4b). both [|g][am l2 and ||g || decrease gradually over layers LN. However, for Pre-LN models (Fig. 4a the gradient norms are predominantly high in the first layer, with the following layers having almost negligible norms. This observation explains why the removal of early layers LN parameters did not significantly affect learning for Post-LN models. That because, in Post-LNN models, later LNs compensate for the absence of the early ones, recovering learning, while mitigating memorization, due to their comparable gradient norms. However, this does hold for Pre-LN models, where the high gradient norms in the early layers LN are critical, and their absence severely disrupts learning.
Similar observations are observed in other Pre-LN (GPT2, Qwen2, VIT-B, DeiT, VIT-S) and Post-LN (RoBERTa, BERT, Longformer, ELECTRA) models, as shown in Appendix G.3
In summary, gradient analysis highlights why removal of LN parameters significantly affects both learning and memorization: (1) disrupts learning Pre-LN models and mitigates memorization in Post-LN models due to the distinct behavior of their gradient norms ratio; and (2) it reveals the particular significance of early layer LNs, which exhibit stronger gradient norms and thus play a more influential role in both learning and memorization processes.

---

7 Conclusion
In conclusion, our study highlights the pivotal role of Layer Normalization (LN) in governing both memorization and learning across two different Transformer configurations: Pre-LN and Post-LN. We identified that the removal of LN parameters in Pre-LN models significantly destabilizes the learning process, leading to persistent overfitting. In contrast, removing LN parameters from Post-LN architectures effectively mitigates memorization and enables the recovery of genuine labels. More precisely, we find that L.N in the early layers are especially critical-removing them has the strongest impact disrupting learning in Pre-LN models and mitigating memorization in Post-LN models. By analyzing the learning and memorization gradient norms, we further reveal how LN distinctively influences these two mechanisms across Pre- and Post-LN models. We show that this distinctive behavior across a wide range of model architectures, spanning several vision and language datasets. Overall, our findings uncover a crucial connection on how layer normalization impacts learning and memorization in transformer models, with its broader impacts discussed Appendix K
References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints, pages arXiv-1607, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint (XXi::2004.88249 2020.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524-10525 PMLR, 2020.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954 1844959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Pratyush Maini, Michael C Mozer. Hanic Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? In International Conference on Machine Learning, 2023.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10889, 2021.
Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. Understanding transformer memorization recall through idioms. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 248-264, 2023.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. arXiv preprint urXiv:2310.15910 2023.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.06990 2021.
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell' Orletta. Outliers dimensions that disrupt transformers driven by frequency, arXiv preprint arXiv:2205. 11380, 2022.
10

---

Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances Neural Information Processing Systems, 36:75067-75096 2023.
Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Understanding and minimising outlier features in transformer training. Advances in Neural Information Processing Systems, 37:83786-8338 83846, 2024.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances neural information processing systems, 32, 2019.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242.
PMLR, 2017.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
Xiaoling Zhou and Ou Wu. Which samples should be learned first: Easy or hard? IEEE Transactions on Neurul Networks and Learning Systems, 2023.
Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Yeon Chung. On the geometry of generalization and memorization in deep neural networks. arXiv preprint arXiv:2105. 14602, 2021.
Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10368-10378, 2022.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint 2rXIv:2002.032066 2020.
Deepak Ravikumar, Efstathia Soufleri, Abolfazl Hashemi, and Kaushik Roy. Unveiling privacy, memorization, and input curvature links. arXiv preprint arXiv:2402.18772 18726, 2024.
Isha Garg, Deepak Ravikumar, and Kaushik Roy. Memorization through the lens of curvature of loss function around samples. arXiv preprint arXiv:2307.058311 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint (XXi:2111400660 2021.
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yo Hsieh, Yaqing Wang. Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Im to high sparsity. arXiv preprint 15Xiv:2310.05175, 2023.
Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of Ims: Stages inference? arXiv preprint arXiv:2406. 19384, 2024.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.0368
2024.
Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-In: Unleashing the power of deeper layers by combining pre-In and post-In. arXiv preprint arX1v:2412.13795, 2024.
Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. The curse of depth in large language models. arXiv preprint arXiv:2502.05799 2025.
Shaked Brody, Uri Alon, and Eran Yahav. On the expressivity role of layernorm in transformers' attention. arXiv preprint Stti:2305.02882.. 2023.
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie, On the role of attention masks and layernor in transformers. arXiv preprint arXiv:2405.1888) 2024.
11

---

Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. B2t connection: Serving stability and performance in deep transformers. arXiv preprint arXiv:2206.003300, 2022.
Shufang Xie, Huishuai Zhang, Junliang Guo, Xu Tan, Jiang Bian, Hany Hassan Awadalla, Arul Menezes, Tao Qin, and Rui Yan. Residual: Transformer with dual residual connections. arXiv preprint arXiv:2304. 14802, 2023.
Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontack Oh, Beomjun Kim, Tachwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-In: Revisiting layer normalization in the transformer architecture. arXiv preprint arXiv:2502.0273:1 2025.
Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization, arXiv preprint trXiv22//0004555 2021.
Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng. Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. Foundation transformers. arXiv preprint 22X:::2210006423, 2022.
Manoj Kumar, Mostafa Dehghani, and Neil Houlsby. Dual patchnorm. arXhv:2302.01327 2023.
arXiv preprint
Xianbiao Qi, Jianan Wang, Yihao Chen, Yukai Shi, and Lei Zhang. Lipsformer: Introducing lipschitz continuity to vision transformers. arXiv preprint arXiv:2304.09833 2023.
Zixuan Jiang. Jiaqi Gu, Hanqing Zhu, and David Pan. Pre-rmsnorm and pre-crmsnorm transformers: equivalent and efficient pre-In transformers. Advances in Neural Information Processing Systems, 36:45777-44793, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico++: Towards better benchmarking for domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16036-16047, 2023.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/reression by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 5810-5818, 2017.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii, editors, Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10118653/ URL httts::/acclattholoo wwrooollgyoooooooooooooooooooo.. hrtp:/o//////000000
Okite97. News data dataset. httppp//wgguugggeeeccc/tttttttte Accessed: 2024-03-03.
2024.
Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados, Leonardo Neves, Vitor Silva, and Francesco Barbieri. Twitter Topic Classification. In Proceedings of the 29th International Confer- ence on Computational Linguisties, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, Levy Omer, and Lewis Mike. Roberta: A robustly optimized bert pretraining approach (2019). arXiv preprint urXiv:/907) 1692, pages 1-13, 2019.
Victor Sanh, I. Debut, Chaumond, and T Wolf. Distilbert, a distilled version of bert: Smaller, faster, 2019.
cheaper and lighter. arxiv 2019. arXiv preprint
12

---

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhann enhanneed bert with disentangled attention. arXiv preprint arXiv:2006. 03634, 2020.
Clark. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint (tti::200400555500 2020.
Dosovitskiy Alexey. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arviv: 2010.11929, 2020.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. European conference on computer vision, pages 456-473. Springer, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow, 2021. URL: https://doi. org/10.528l/zenodo. 5297715, 2022.
Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15115: 2024.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Waleed Ammar, Annie Louis, and Nasrin Mostafazadeh, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties (Demonstrations)) pages 48-53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/119-4000 URL http:///aaaaaaaolooooroo////////
Andrey Gromov, Kushal Tirumala. Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887 2024.
Roger A. Horn and Charles R. Johnson. Topics in Matrix Analysis. Cambridge University Press, 1991.
Ola Hössjer and Arvid Sjölander. Sharp lower and upper bounds for the covariance of bounded random variables. Statistics & Probability Letters, 182:109323, 2022.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
13

---

Appendix
Table 2: Comparison of Post-Layer Normalization (Post-LN) and Pre-Layer Normalization (Pre- LN) Transformer Setup. MHSA = multi-head self attention, FFN = feed-forward network, LN: LayerNorm, N number of transformer layers, 2; input to tth layer. Y/1 output of 7t layer, 2 & 2x intermediate vectors Post-LN model, which are imputs to LN1 & LN₂ respectively.
Post-LN Transformer Setup
Pre-LN Transformer Setup
2; MH)A(a;: + Ti
MHSA(LN (2:)) Z; y1 F + #4
4 LNI(2)) (zi)
(FN +
M/ (2)
Yout classfiicatinnhhadd..
C CrossEntropyloss(Joot 3tree)
A Problem Setup for Gradients Analysis:
Consider a training dataset Drain consisting of C classes. We make an assumption that all samples in class well represent class Then, we introduce single noisy label by selecting sample (I1, Me) from class where e E C, and modify its label to a different (incorrect) label, 3NL. where NL(E C) A c. The noisy sample now represented (I1, 2NI.). At the same time, we do modify any other training samples from class to ensure that the model has access to correctly labeled examples for effective learning as well. Next, we train transformer model on this modified dataset until it reaches 100% training accuracy. At this stage, the model has fully memorized the noisy- labeled sample (I1, 3NL) while also learning class features from correctly labeled training samples. Now to measure the notion of memorization and learning, we compute memorization gradient norm (Igg (|2) and learning gradient norm leam (|2) as discussed in Sec 3.2 and compare them.
B Theorem 1: Learning Gradient Norm, geaarn| || is greater than or equal to Memorization Gradient Norm, ||g across all layers.
It is formally represented as follows:
leam
2 19. mmme
ll2.
across all layers
(9)
Proof:
Based on the Problem Setup as discussed in Sec. A layers, for both Pre- and Post-LN models follows.
we prove that (gleum || 2 |1g_emm || across all
B.1 For Post-LN model:
Firstly, we elucidate the architecture of Post-LN transformer Tab. Based on the Post-LN architecture, during backpropagation, layer. Since there are two LNs in every layer, we compute and compare the learning and memorization gradients corresponding the inputs of both LNs, i.c., Izg and 92. (refer to Tab. 2).
compute derivatives of loss w.r.t input of LN for every ith
B.1.1 Backpropagating gradient analysis for LN₂ (9=;):
The backpropagating gradient for Iz,, can be expressed as follows:
ac Oc Yyou
Ty
d2
drl
0zj Oz,
Oy: Ozf
92
02 yoo
OyN
II j=++1
dz
drl
Zzj
(10)
where 2++1 = Yi because th layer's output yi the input of 1)th layer, 01+1- To measure memorization and learning, compute these gradients for both (œ1, yNL) and (Iz, Be), denoted as
14

---

92. and germ, respectively. In both gradients, Yous, Vj. 2/. 2/ 2z. 2 are only dependent on the imput samples #1 and æ2, respectively, and not on their labels, where both and I2 genuinely represent class c. Therefore, Ouaa 0y, dz Ouy deg
Bys will not be significantly different for both Jr. and
GyN 02 aue Øzs" Bx," 82;
g. thus we regard the term, it does vary across inputs). The only difference between the two gradients due to oc because Byie..
[[i=+t
8z
02, 32g
as A for both cases (as D5
ByN
the loss C is dependent on the label of both samples, i.e. Me and UNL, as follows:
L=
Σ yk. log(ūr.) 1
(11)
where μık = 1 if k; is the ground truth class, otherwise O, and Wk, is the predicted softmax probability of class ki, As result, mem and leam can be respectively represented as follows:
mm
OCleam
mem
AA
&
Iearn 92.
A
(12)
Oymem
Oyleam
Jearn
Comparing learning and memorization gradients Now the problem boils down to comparing
the L2-norms of agnn CrossEntropyLoos between the predicted softmax probability vector ý = Softmax(Joutl and the
Oygr and aekmm Oytir To do so, we need to first define ac We know that L is the
asua
ground truth y as defined in Eq. (IT). Hence, ac can be written as follows:
Byar
ac Oyou
(13)
Now we can understand what it really means for memorization and learning. During inference of (I1, YNL). Vmem will be vector which consists of very high probability (= 1) for class yNL while assigning extremely low probabilities (== 0) to remaining classes due to overfitting. Therefore, Ymiem UNL will almost be O-vector, i.e., all the elements in the vector would be almost O. This phenomenon is formally represented as follows:
Jmem ~ 00
0],
UNL 00 0..... 1....,100 0],
(14)
Umenm
UNL R 10...., .,...,.0) 0].
where the index corresponding to class UNI. is 1, and all other elements are (close to) O.
Now, take the L2-norm of both sides in Eq. 113 for the memorizing sample. Since Yniem UNL
close to a O-vector, its L2-norm Rs O. Therefore,
OL
0
(15)
mem
On the other hand, for (I1, Me), even though the model has learned generalizable features for class it has also been overfitted on noisy label samples. Hence, Yteam will not assign very high probability Ye but instead will distribute some probability mass across multiple classes. Therefore, yheam Be contains non-trivial, not near-zero values. This behavior formally presented as follows:
Úeem = [Pi, Pg. .c)) 0],
ye 0.....
(16)
Uteam
Yo = P1. Py. 1 pcl.
where Py. is not close 1, and other probabilities pı are non-negligible.
Now, by taking the 1.2-norm on both sides of Eq. (13 for the learning sample. Since yiem ye contains non-trivial, zero values, its L2-norm 0. Therefore,
15

---

a Cleam
(17)
y
learn
Therefore, by comparing Eq.
15
17
we can establish the following relationship:
Cleam
m
(18)
Yyou
learn
dym mem
because overfitting noisy samples, causes the memorizing gradients be smaller than learning gradients. However, note that an ideal case of perfect learning, where 100% memorization and 100% learning co-exist, is achievable in practice memorization inherently hinders generalization. Hence, the equality from the inequality can be disregarded in almost all cases.
Now, substituting the relation found in Eq. 18 to the L2-norma of ب and mm in Eq. can formally conclude that:
learn
92
mem
(19)
This proof explains why (ommm)) is lower than in Fig.
leamn
across all layers, as also empirically observed
B.1.2 Backpropagating gradient analysis for LN1 (9x;);
Similar to 9at= we can express Is, as follows:
DC OC
Oyou
Dyy 0z Ou'y Ozj dz' Or' 0z, 0z,
0y:
azk
Dxk
II j=++1
9=. 02;
(20)
Oyou
Oyn
Oz
Ox!
O2.
Here, Z141 = Yı because 7th layer's output ys is the input for (i + 1)th layer, 2i+1-
We compute Da.m and g:.um for both memorization (I1, UNL. and learning (I2, ye) samples, respec- tively. Based on the discussion in See. B.1.1 on the similarity of 21 and X2, since they both originally belong to the same class we can write g and glaum as follows:
mem
Cleamm
Oymem
A2
&
9
Jearn
Oy
A2
(21)
where, Ag
Ottsa OyN
02!
0x
Bug dz
0=
8m 0z.
which does not vary across
j=it1
82
B2g
0z,
inputs.
To compare mem
and Jearn
we use the argument made in Eq. 14
16
which explains why
2
Using the above results and subsituting the relation in the L2-norms of gl..m and mem in Eq. 21) we can conclude that:
(22)
In conclusion, both Eq. 19 & formally demonstrate that the L2- no-morm of learning gradient, leam is greater than or equal to memorization gradient, mmm across all layers of a Post-LN model.
B.2 For Pre-LN model:
Firstly, we describe the architecture of the Pre-LN transformer in Tab. Based on Pre-LN architecture, during backpropagation, we compute derivatives of loss wrt input of LN for every ith layer. Since there are two L.s in every layer, we compute and compare the learning and memorization gradients corresponding the inputs of both LNs, i.c., Ir,, and 9x:-
16

---

B.2.1 Backpropagating gradient analysis for LN (Im;);
The backpropagating gradient for Jxy can be expressed as follows:
0C
Oc
yyour
N II 5=++1
Jg Oxy
day
Dy: Ork
0x dx
(23)
yont
NyN
dzj
where 2++1 Yi because hth layer's output yi is the input of + 1)th layer, I141- To measure
memorization and learning, we then compute these gradients for both (x1, ( and (22, We). In both gradients, Yout; V/j xrj; 2i are only dependent on the input samples x1 and 42, respectively, and not on their labels, where both 2 and genuinely represent class c. Therefore,
Byg Ba
Oy: 8m
will not
Bay
be significantly different for both Ix Ilum and gr -r thus regard the term,
Bys 02
82
Bax
Dz,
By, as B1 for both cases (as it does not vary across inputs.) The only difference between the two Bey
gradients is due to By shown in Eq. As result, Day mem and Dzz dearnn can be respectively represented follows:
ac because the loss L. is dependent on the label of both samples, i.e., Ye and yNL
mcme
Ix
memm
B1 &
Jearn 9x
B
(24)
Oy mem
dylam
Comparing learning and memorization gradient norms
Now the problem boils down to com-
paring the norms
aeum
and
agtmm
Synen
By/tmm
From Eq. (18) we know the following relation:
a Cleam
mem
(25)
Dy Jearn
mem
Now, substituting the relation found in Eq. 18) the 12-norms of g.. learn and g in Eq. (4). can formally conclude that:
learn
mem
(26)
This proof explains why consistently observed in Fig.
mem
lower than
learn
across all layers, as also empirically
B.2.2 Backpropagating gradient analysis for LN, (gx,):
Similar to 9x., we can express Iz. as follows:
0C DC
Oyout
dy:
Ozy
Oy:
dx!
I=: Ir,
OyN
II
Zn'
0n;
D.l
Or,
(27)
where Fi+1 Ys because th layer's output is the input for (i + 1)th layer, 21+1, and compute g and geam measure memorization and learning respectively. Based the discussion in Sec. B.1.1 on the similarity of 2 and 22 since they both genuinely belong to the same class c, we can write g mem and g
learn follows:
mmem
aceam
mem Ias
B2
&
Jearn
B2
(28)
dymem
Oy
where, B2
Byoo
GyN
Ti=it1
Ty: day
da 32,
Bys Uz
au', DE,"
which does vary across inputs. To compare
mem and g we use Eq. (18). which states that
alamm Dyycam
à m
2

---

Using the above results and substituting the relation in the L2-norms of g and go; in Eq. (28) we can conclude that:
19..
(29)
In conclusion, both Eq. 26
formally demonstrate that the L2-norm of learning gradient, gam is greater than or equal to memorization gradient, Iarm across all layers of Pre-LN model.
29
C Theorem Gradient norm of loss L w.r.t input of LN is upper bounded.
Post-LN: Let 2; denote the input to L of the th Post-LN model layer. Then,
OC
1
11 12
<amax (Pi)-
0z,
II s=
Var(FFN (2²))) (II-VVV((MSSA
N
N
II
(1 mmax (FFEx)
II j=++1
(1 + 8max (JMiHSa)
(30)
Pre-LN: Let z; denote the input to LN1 of the th Pre-LN model layer. Then,
11x; |12
0L
8max (P2)
II (11 Smax (No(e ((r))
II
(a1 Smax MIISA JEN.
(NN(E)))
Ør:
(31)
Proof:
C.1
For Post-LN model:
The Post-LN model setup for 7h layer can be represented as follows:
/ LN (Z: + MHSA(x;))
(32)
3: LN2 (z) + FzN(2!))
where x; + (H)A(r;) and x. + FFN(2:) are the inputs LN, and LN2, respectively. Later, we substitute and use them as
2 + MHSA(x;)
(33)
= (FN(
Since there are two LayerNorm (LN) operations in every layer, prove it seperately for both of them.
C.1.1
Backpropagation analysis for LNg (g=;);
By applying Eq. 333. we obtain z/J = x (.)(2))) 2j xy + MHSA(rr)) Hence, we can write
dz (from Eq. 10h for the 7th layer as follows:
ac
OC
Oyou
Ty dzh dz
dx
0zj dr,
Oy: Ozh
92
II j=CH1
(34)
Oz
Dyou
OyN
day
Ōzy
ac Otsa independent of the transformer's layers as they are computed using the classifica- Here, 8yea Byn tion head's output. Therefore, we can treat them P1 (which does not vary across layers). We also compute the corresponding derivatives of 2n and 2j. Lastly, I++1 1 is same yt because y, is the
18

---

output of the tt layer which becomes input 2;-1 of the (2 + 1)th layer. By applying all of these, we obtain the following equation:
ac
N
Ouj 0z
O + FFN(()))
Oz'z
a(z, + MHSA(I,))
9s;
02
Pi
II -=++1
Ozz
dx,
dzk
(35)
N
dy 02}
)FFN(E)),
ÖMHSA(I,)
Dy:
P1
II j=++1
(I+
(I+
(36)
Oe'
dzj
Ozy
0zk
We also acknowledge that Dy 02j" 0
0x
and Cy are derivatives of output of LN w.r. their input, which DE!
can be simply represented as Jacobian matrices, J J and Jix,, respectively. Likewise, @FUN(a)) and OMHSA((E)) are also derivatives of the output of FFN and MHSA w.r.t their inputs, and 8
be represented as Jacobian matrices, JiiN and J respectively. After substituting these terms,
we obtain the following:
9=
dc
II 1=i+1
(JiN.- (I SHiN JRN. (I+ (WAiSa) Jú
(37)
2z
Now, we take the L.2 both sides of Eq. follows:
10=;112 =
02 1
0C
=
Pi
II j=i+1
N
(JiN.
(I+ JFEX) JEN (I+ (MISN))
(38)
We know that L2-norm of matrix is equivalent to its largest singular value [Horn and Johnson 199T]. Hence, we can further write Eq. (38) as follows:
1
Ac
Smax (PI
(JLN. (1 -=++1
II
PEEN JEN (I (iitsa)
-
(39)
where
Smax
outputs
the
largest
singular
value of (P)
I
JIN.
(I
FFN)
JIN (I+ Jillss)
J. From
properties of singular
values [Horn and Johnson 1991]. know that
Smax (AI ... An. Smax (aix(AA)smnn Smax (A2)
Smax (An)
(40)
max (Ai + A2) S smax (A1) + smax Smax(A2)
where 8max (Aa) maximum singular value of matrix Av. After applying these properties to Eq. 39 we get the following:
0c
1 112 =
Oz 2
< Smax (Pi)
II j=i+1
8max (J (smax (I) Smmx (JeiN)
(41)
Smax (J (Smax (I) Smax (Jritss)
Smax (J
According to Xiong et al. 2020 we can rewrite the Jacobian of LNs as follows:
and Jtb σz
IN
JINz σz
(42)
19

---

where xzy, o. and o are the standard-deviation of 24, agj. and z/2 respectively. Therefore, we obtain the following equation:
ac
119 122
221 2
I mmax (P1)
II
Smax
(8mmax (I) + Smax (FEiN))
(43)
f=1-1
1 Tx!
(8max (I) + Smax (J (MitEss))
max
mmax
Another property of singular values states that all singular values of identity matrix I are I [Horn and Johnson 1991]. i.e., (I) 1. Therefore substituting with that Eq. 43 we obtain the following:
ac
N II j=++1
| ll2
< max (P1)
(1 + Smax (J
(1 + 8max (amaxJJiisss (NMiIs))
0=4 2
(44)
By re-arranging the terms, finally obtain the following equation:
ac
11g=:1l
smax (Pz)
IT TT Ozy
IT Ozs
(45)
N
II j=1+1
((1+
smax((FFF)) (1+ Smax (JMHHS))
Now, we need to investigate how Øsy and Dzy behave. We know that Ø2 v Var(z)) and 0z V (Va(<z))) where Var(-) represents variance. Therefore, we can instead focus on Var(z;) and Var(z)). We know that 2j = Iÿ + MHSA(#;) and 2/j = x'j + FFN((1)). Therefore, their variances can be written as follows:
Var(2,) Var(z, + MHSA(z,)) Var(z)) Var(z' + (x'()
(46)
Now to compute the upper bound of Eq. 445 we need to substitute the lower bound of Øz, and dz as they are in the denominator. The lower bounds az, and Tz2 would basically lower bounds of Var(z;) and Var respectively. From Hössjer and Sjölander 2022], know that for any two matrices A and B,
Var + BB >
(VVa((A) v VVr(B)
(47)
Therefore, the lower bounds of Var(z;) and Var(zz) can be written as follows,
Var(2;) 2 (v Var(ay) Var(MHSA(z;)) 2
(48)
Var(z/) 2 (v Var((z)) Var(FFNN(1))) 2
Since F, and 2 are the outputs of the two LN layers, their variance 1. Therefore, we can rewrite Eq. (48) as follows:
Var(z;) 2 1 1- V--VV((MHSA((;))) 7 Ozy v (ar(MHHA((z)))
(49)
Var(z))
2 (1-VVa(FFEN(''))
0z 2 Var(FFN(<3)))
20

---

Hence, we can we-te Eq. 45 as follows:
1 - (ar(FFN())))) )
((I>
0C
119
Smaa (Pi)
0zg 2
1
v Var(MHSA((T))
N
II ((1
(1x+ Smax
(1 + Smaa (J (JMliss)) (FEts) (
/=(+
(50)
C.1.2 Backpropagation analysis for LN1 (g2,):
Similar to Ouf+ we can express 9s, as follows:
OC
OC
yout
Oy Ozy
0z1 Ou'
0u'y
0zj
Dy.
az
dxk
9s, Oz
II j=(+1
(51)
Yout
OyN
0zj
0xy
Oz
Ox:
0z
0C Gy independent of the transformer's layers as they are computed using the classifica- Here, Bie Buy tion head's s output. Therefore, we can treat them as Pi (which does not vary across layers). We also compute the corresponding derivatives of Zy and Zj. Lastly, Ii+11 is same as yı, because yi is the output of the ,th layer which becomes input -;-1 of the ( 11th layer. By applying all of these, obtain the following equation:
Oc
By:
0 + +FFN(a)))
Ou', 0(z, + MHSA(2;)) Oz,
II i=++1
9s.
0z,
Ozy
On',
Oz,
(52)
0y
0a + FFN({
Ox!
0z
Oal
Ozi
Tys 02y
a OFF((
0x', 0zj
@MHSA(0))
P1
1+
1
Ox'
Ox,
j=++1
(53)
0y:
i FFN(
Dx! Oz,
0z
Ouk
Clearly, we can see that J5,
0y.
Gy: D D5
0x
and 0z Dz,
are derivatives of output of LN w.r.t their input, which
can be simply represented as Jacobian matrices, J JiN, J.N. and Jik,, respectively. Likewise, OFFN(a)) and @MHSA(#;) are also derivatives of the output FFN and MHSA w.r.t their inputs, and 3my
can be represented as Jacobian matrices, JriN and JHHSAA respectively. After substituting these terms,
we obtain the following:
OC
Pi
II s=1+1
(J (I+ FFEN) JRh (I (iiIsN) Thh. (I+ JEAN JEL, (54)
02:
Now, we take the L2-norm on both sides of Eq. (54) follows:
1g ll2
II j=++1
(Ji (I+
Jis (I+ JMinsa)
Jike (I+ JofN)) J
0z
1
(55)
21

---

We know that the L2-norm of a matrix is equivalent to its largest singular value [Horn and Johnson 199I]. Hence, we can further write Eq. (55) as follows:
9 ll2
0L 0zs 1
Smax (P
II -=++1
(J (I JEEN Jis (I (M+ss))
J
(I
JFiN)
JIN
(56)
where
largest I ( (I J JIN (I+ (MHSA)
Smax
outputs
the
singular
value
of
(Pr
JIN
(I+ JFPN) J Now from properties
of singular values defined Eq. (40), we can further rewrite Eq. (56) as follows:
Ac 19
8zg l2
N
S 8max Pi))
8max (JiÑs)
(Amax (I) + 8max (JrEx) Smax (Jit,) ) (8max (I) + 8max (JMiisa))
f=1+1
8max (JiNz)
(on))))(mmmm (Smax (I) + Smax (JHéN) Smax (J
(57)
From Xiong et al. 2020 we can rewrite the Jacobian of LNS as follows:
JIN.
JIN2
and J
(58)
22,
d=
where Zzy oz2, o2t and os are the standard-deviations of 2j, 2//, 24. and 2 respectively. Therefore, obtain the following equation:
ac
11 l1 =
3zs l2
N
< 8max (Pi)
I
(8mmxx (I) + 8max (JFiN) 8max
I
(Smax (1) + Smax (J
8max
0zy
f=++1
I
(8max (I) + Syax (JFiN)
8max
8max
Oz
(59)
Another property of singular values states that all singular values of identity matrix I are 1 (Horn and Johnson 1991]. i.e., (I) = 1. Therefore substituting with that in Eq. (59). we obtain the following:
Oc
11 112
1
< Smax (P1)
Smax
(1 + Smax (JFfx)) Smax
+ Smax (J (Fritsaa)
0z
Tzg
j=it1
(1 + Smax (J
8max
Smax
Tz
(60)
By re-arranging the terms, we finally obtain the following equation:
11
0C
< 8max (Pi)
1
II
N
1 + 8max
(JFEN)
II
N
(1 + Smax (J
0zi 1
O2
j=i
j=
(61)
22

---

Now, based on Eq.
49)
we can we-weite Eq.
61 as follows:
aC
11 lla
Σ Smax (Pt)
02,
2
11
Var(FFN((3)))
Var(MMSA((
(62)
NV.
+ 8max (JFEN)
(1 + 8max (JMIISA)
j=i
j=x+1
C.2
For Pre-LN model:
The Pre-LN model setup for 7th layer can be represented as follows:
a/ = 24 + (
(63)
3/2 2 + FFN(LN2
where z; and d are the inputs L and LNg, respectively.
Since there are two LayerNorm (LN) operations in every layer, we separately prove both of them
C.2.1 Backpropagation analysis for LN₂ (ga:) ):
We can write Izद for the jth layer as follows:
OC
Oc
yont
dy,
Oa's
By: Dx
9x Ox
II
(64)
Oyou
OyN
Ox,
0z;
0C Byue is independent of the transformers layers they computed using the classification Here, 8yie dyx
head's output. Therefore we treat them as P2 (which does not vary across layers). Furthermore, we expand y, z + FFN(LN, (z') )), x', = zj ( using Eq. (63). and compute their corresponding derivatives Eq. (64). Lastly, x++1 is same as Vi, because yi the output of the 7th layer which becomes input 1 of the (i + 1)th layer.
After substituting, get the following equation:
Oc dx
N
((al FFN(LN2(E())) 0(z, + (HH) 0z'
0(1) + 2F Ox
9n
P2
II jmi+1
dx,
(65)
OFFN(LN((12)),
OMHSA(LN) (Iz)) -II+
OFFN (x{))
Pa
II |=/+1
(I+
dx; (I+
Ox'y
(66)
N P2 II (1+ 0FFN(LN,, (2()) OLN2(T)),
(MHSA(LN((E)) (
(LN2(x
Ox²
OLN, (2,)
Oz,
j=++1
(67)
(FFN(LN,((
(LN2(11)
-I+
0LN₂ (x))
da!
Here, in Eq. 667
(L,)
(L Ou
are both derivative of output of LN w.r. their inputs, and hence
duy
can be represented as Jacobian matrices, J and J JNN respectively. Similarly,
(M and
BUN, (mgg)
23

---

( (E))
are derivatives of output MHSA/FFN w.r. their inputs, and can also be represented
as Jacobian matrices, (L,) and FEN
L
MHSA
respectively.
Using these relations, re-write Eq.
66 follows:
0C 9x
Ps
II
JIt (Nale))
J.RR)) (I+
MHSA -INN )) (I+ (Na(t))
JINo
dz
j=it1
(68)
We can further re-arrange the terms in Eq. (68) as follows:
0c 9o! Or
P2 II [1
JEN
(Na
II jmit1
(1+
J (my)
MISSA
JEN.)
(69)
Now, we take the L2-norm at both sides of Eq. 669 and since we know that L2-norm of matrix is equivalent to its maximum singular value. Therefore, get the following equation:
119x:112 =
Oc 0x 12
mmx PP
II [I
III J=i
N (Na(E))
FEN LN2
II j=it1
N
I
LN,((2))
MHSA
(70)
where
Smmx is the maximum singular (x,) MHSA JIN,
value of (P2
I
(I+
(I+ I From the singular values properties, discussed Eq. 440 we can write the upper bound || l12
J(X,))
follows:
19_:112
0C
< 9max (P2)
II
Smax (I) + Smax (J LNa(aa)
II j=++1
Smax (I) + Smax (NNN (xx,).
0x,
j=
(71)
know that singular values an Idenitiy matrix I are I, i.e., (x) 1. Thus,
1
0L dx 1
S Smax (P2)
II (1+ Smax [Na(a))
II
(1+ Smax MJSA J
f=141
(72)
C.2.2 Backpropagation analysis for LN1 (gx;); ):
We can write gz, for the jth layer as follows:
OC 0C
yout
Oy Oxy
Oy:
dxy
Im: Dx.
II
(73)
Oyout
OyN
Ox,
Ory
Ox;
DC Oyon is independent of the transformers layers they are computed using the classification Here, Byon 8yx
head's output. Therefore we can treat them as P2 (which does not vary across layers). Furthermore, we expand y, 2' + FFN(LN, (x') )) 2/ xi their corresponding derivatives Eq. (73). Lastly, 2++1 same as ys, because yt the output of the 7t layer which becomes input 24+1 of the (i + 1)th layer.
(HSA((NN((zz)) using Eq. (6) and compute

---

After substituting, we get the following equation:
0C
N
0(1) + FFN(LN2(EE)) D(z MHSA(LN, (23)))
9x.
0z,
P2
II
dz'z
dx,
/=++1
(74)
0(1 + FFN(LN)(Z)))
az + (
Ox²
Oz,
P2
II j=i+1
1+
OFFN (z
(1+
@MHSA(LN, (xj))
Oal
dx,
(75)
(1+ (FFN(
(
1+
Ox,
Ox:
(1 OFFN(LN, (2()) (LN2(T))
(I+ (MHSA(LN, ØLN
OLNI (xz))
PPa II
(L)2(x))
On'z
OLNI(T,)
Ox;
j=++1
(I+ (z.)
(LN2(11) (I+
@MHSA(LN (c.)) (LN)(
(L))(
dx,
(z))
Ox: (76)
Here, in Eq. 76
(L,) Che
(LN.(a))
are both derivative of output of LN w.r. their inputs, and hence
can be represented Jacobian matrices, J and JiN, respectively. Similarly, @MHSA(LNI (2s)) and (1r)
derivatives of output of MHSA/FFN w.r.t their inputs, and can also be represented (F
(a))
as Jacobian matrices, LL (xz) and JFEN respectively.
LN2(21))
Using these relations, we can re-write Eq. 76 follows
I2: dr.
OC
P2
II
N
(1+
JIN
Jin.)) (1+
M JEN, )
j=++1
(77)
(I
LNa(z()
JEN
(I
(N.( MHSA
Jik,
We can further re-arrange the terms in Eq.
77 as follows:
9x.
P2 0x:
OC
N IIIII111 5=i
I+
FFN
LN2OE)
JUN
II(
II
MHSA
(N,(
(78)
Now, we take the L2-norm at both sides of Eq. 788) and since we know that L2- 12-norm of matrix is equivalent to its maximum singular value. Therefore, get the following equation:
19 112
0L 0x.
Smax (P2
II (1+ J
N
(N
II jmi
N
I+
(N,
-EX.))
(79)
MHSA
jmi
(I+ INa(T))
where
Smax is the maximum singular JNN (x,)
value of (P2
I1
T (I+ From the singular values properties, discussed in Eq. (40) we can write the upper bound of ||
MHSA JEN
as follows:
25

---

1g0:112
0C
I max (22)
II
Smax (I) + Smax ((NNe(())
II
N
8max (1) + Smax MMHSA
0x:
2
(80)
We know that all singular values of an Idenitiy matrix I are 1, i.e. (I) 1. Thus,
19e, 12
0C 0x: 2
8max (22)
N II (1 j=1
+
Smax (JENN
N II
(1
Smax (NN)
JENN
J=i
(81)
D Theorem 3: Upper bound of the gradient norm of Early Layers LN are higher than those of Later Layers LN.
formally represented as follows:
UB( 9z (12) 2 UB( l12) 2 ... UB 112) for both Pre- and Post-LN models (82)
where UB (B(1le, (2,
denotes the upper bound of 11 llz and x; is the input to the 1t layer's LN.
Proof:
D.1
For Post-LN model:
D.1.1
Analysis for LN2 (u=:):
We prove that UB(llg& (2) 2 UB( l12), corresponding to 7th and (4 + 1)th layer.
We can compute UB UB[lg (2) and UB UB (119, (12) using Eq. 50 as follows:
UB(19=;]12) =mma (Pi)
[I
--VVaaai (ar((FFN((3)))
[T
(VH((M)))(zz)))
II
((1+
Smax Sman (JEtx)) (1 Smax (J (Missa))
f=it1
(83)
(II
UB UB([ 112) Smax (Pi)-
I 1 v (ar((FFN((
/ Var(MMHAA(zz)) (I;))
N II j=1+2
Smax Smaxx (1++ (JEfN) (1+ Smax (JitHts)))
(84)
We then substitute these expressions in the inequality UB([Ig:{11)) 2 UB(1192 [12), to prove that carly layers have higher gradient norms in comparison to later layers.
26

---

1
Smax (P1)
N
V Var(FFNN(x)))
j=itt
1
Var(MMMSA
II ((1+ Smax $maae(FFin)) (Frie))
Smax ( (Niissa))
j=1+1
)
1
2 8max (P1)
[1 1 /ar((FFN(>))))
1
Var (MHSA
II /=1+2
(1++ 8max (nfee)) + Smaae Smmee (Miiss))
(85)
This can be further rewritten as follows:
(1 + Smax (JFPN () + Smax (MaHIAA) )
21
(86)
(ar(FFN(()))
(MHSA (z++1))
From Horn and Johnson 1991]. we know that every singular value of matrix A is greater than or equal 0, i.e., (A) 0, Vk. Hence, for every transformer layer,
(1 + Smax (HiHS))) 2 and
(1 Smax (J Z
(87)
Now, to prove Eq. 86 to be true, we need to prove that
<<< V (ar(FFF(())) <1 & 0 0< V Var(MMHSA (xi+1))
(88)
We do not consider the scenario where either
(ar(NFN(
0, because if either/both of them becomes O then the gradi- ent norm would go infinity, which we do not observe in real-world models either.
Var(MHSA((x41))
We prove Eq. 88 for the FFN component follows (MHSA component will also have similar proof):
Firstly, 1 (FF((FF( be rewritten as follows:
(ar(FFN( DD),, if V (ar(FFN(( <
Var(FFN(a:))
(89)
Var(FFN(2 1, if Var (FF(FF(((:)) >
We then apply the inequality described in Eq.
88 and Eq. 89 together as follows:
0 < Var( 1 if (ar(FFN(())) <1
<<
(ar(FFN(2
<1
00 (ar(FFN(())) 111, if (ar(FFNN)))) >
(90)
Further solving Eq. 90). we obtain the following:
0 Var(FFN (x))) <1. if (ar(FF())()) <1
V Var(FFN((
$1
1 (FF(FFN 2()) 22, (arFFN(( > (91)
27

---

Hence, the inequality
(FF
1 holds true when
OFN, (ar(FFN((2)) [0, [
(92)
Likewise, the inequality 0 < V Var(MMHSA S holds true when
MHSA,
Var(MHSAA( -
(93)
Hence, from Eq. 92 & Eq. 133, we prove that
(1+
PEN ()(1+))nm
AIRAA
2 1, further
1 (ar(F
Var (MHSAA(()))))))
proving that UB (11951112 UB (1 ll))
Consequently, we prove that the upper bound of L2-norm of gradients for Early Layers LN2 are higher than the one of Later Layers LN₂ in Post-LN models, formally represented as follows:
UB ( (1a) 2
UB (
UB( ll),
(94)
when 0 < Var(FFNN(:))
- and 0 <
Var(MHSAA( S.
D.1.2 Analysis for LN1 (gz²):
We need prove that UB (z (la) 2 UB(1g l12), corresponding hth and (1 + 1)th layer.
We can compute UB(1192, (2) and UB(l (I2) using Eq (62) as follows:
UB (12) =8max (P1).
[T 1 -
1 (ar(FF)(( 1 (ar(MMSA(
(95)
Smax (Fees)
II (1 Smix (JMitss)
jmit1
UB(1 (12) -max (Pt)
II 1 /ar((FFN( (2)))) 1 Var(SMAAAAT
(96)
N II
(1+ Smax ( (FFIN)
II j=++2
(1 + Smax (Jaiss))
We then substitute these expressions in the inequality UB(119,, (ll2) 2 UB(1l= 122) to prove that early layers have higher gradients in comparison to later layers.
1
Smax. (Pi)
11
Var(FFNN(z)))
VaH(MM
N
N
+ Smax (FFN))
II
(1 +1++ Smax (Jee
j=
s=+1
(97)
Smax (Pi)
[1 Var(FFN(x
Var Var( (MHSA(z;))
II
[ Smax (JEAN)
II
(1 Smax (J
j=it1
j=++2
28

---

This can be further rewritten as follows:
(1 + Smak (JFFN) (1 + Smax (J
MHSA
21
(98)
V Var(FFN
Var (MHSA(z:))
We already know that + 8max (MMSAA)) 2 and (1 + Smax (Jrin)) 2 from Eq. 87
Now, to prove Eq. (98) to be true, we need to prove that
0 Var(FFN(2 af))) < & 0<
1 0<
V Var(MHSA(I.)) $
(99)
We do not consider the scenario where either
or 0, because if either/both of them becomes then the gradient norm would go infinity, which we do not observe in real-world models either.
Var(FFN(()))
1 Var(MHSA(z.))
From Eq. 92 & Eq. 993 we know Eq. 99 holds true under the defined conditions.
(1-8ma (Jmm) (1 9pm (J
Hence, we prove
1, further proving that UB(l l12) z
1- (arFFN(()( 1- (MMSAA(()))
UB(1 Ila). This then inductively proves that the upper bound of 1.2-norm of gradients for Early Layers LN, are greater than the one of Later Layers LN; in Post-LN models, formally represented as follows:
UB ( Ila) 2 UB(([e2211) 2... UB( 19 112)
(100)
when 0 <
Var(FFN(()))) < and 0 <
(ar(M < 1.
E
For Pre-LN model:
E.0.1 Analysis for LN₂ (Ja;);
We need to prove that UB (1: (12) 2 UB( 12), corresponding to tth and (á + 1)th layer.
We can compute UB(Ilg.; (2) and UB U l22) using Eq. (72) follows:
UB(19>:1122 Smax (P2)
II (1 Smax (N)( x(
II (1 Smax (JHHA {Lhn))
(az))
J
(101)
N
UB [B(1) = Smax (P2)
II j=i+1
(1+ Smax (NNa(1a)) JiÑe))
II (1 (1+ Smax (J Juh
j=++2
(102)
We then substitute these expressions in the inequality UB (12) 2 UB( [12), to prove that early layers have higher gradients in comparison to later layers.
S Smax (1))
II j=i
1 Smax (JENSCOE) 1(r)))
II (1 Smna (JuMSA JEX.)) j=it1
(103)
N1
N
Smax (P2)
II
11 Smax (JIN
(No(oo))
((Nz))
II (1 Smax (JANSCE) JEXZ.) j=it2
j=++1
29

---

This can be further re- re-written as follows:
(1 + mmax (JANN ) (1+ Smax (JuHS'a
JIN;
(104)
We know that every singular value of matrix greater than or equal i.c., sx(A) Hence, for every transformer layer,
vk.
(1+ Smax
(JLNa JiN.s) )) 2 and (I+ Smax
(J JIN, )) 21 (105)
This proves that Eq. (04) is true, hence also proving that UB( (2) 2 UB( 112)
Ví.
This then inductively proves that the upper bound of L2-norm of gradients for Early Layers LN2 greater than the of Later Layers LN₂ in Pre-LN models, formally represented follows:
U (Ia) 2 UB(1 (12) 2 ... 2 UB( ll2)
(106)
E.0.2 Analysis for LN1 (gm;);
We need to prove that UB (z. (|2) 2 UB(l [2), corresponding to hth and (i + 1)th layer.
We can compute UB(ll 12) and UB (12) using Eq. (81) as follows:
N II
N
UB = 8max (P2)
1
8max (JEEN
(N2(x())
(V.)
Smax (JUNS)) SEX
(107)
UB (10 Smax (22)
II f=++1
(1+ Smax (LNola))
II (1+ Smax (JNNs J(N,))
(108)
We then substitute these expressions in the inequality UB( 112) Z UB(l 112), to prove that early layers have higher gradients in comparison to later layers.
N
N
Smax (P2)
II
+ 8max
(JAN
II jmm
+
Smix
(JUNSKE) JEN.)
jmm
(109)
N II jmi+1
N
Smax (P2)
(1+ Smax
(LNN(10)
J(K.))
II jm2+1
(1+ Smax (JANSSA J(R.)
This can be further re-written as follows:
11 + Smax
(JIN T(kz)) (1 Smax (JuNs JUn J(nn))
)) 21
(110)
(1 Smax
LNa(o!) JEN
JEi.)) 2 1 and (I+ Smax (JuNslEe) JEN
We know that
2 1 from
Eq. (105..
This proves that Eq. (10) is true, hence also proving that UB(llga, I12) 2 UB(lle Vi.
This then consequently proves that the upper bound of L2-norm of gradients for Early Layers LN, greater than the of Later Layers LN, in Pre-LN models, formally represented as
follows:
UB [|2) 2 UB(19221122 l12) 2 2 UB (B((gg=y) l1
(111)
30

---

F
Training Details
This section outlines the detailed configurations of the datasets and models used in our experiments, including dataset splits, pre-processing steps, model architectures, and training settings.
F1
Datasets
Below discuss the 6 datasets used in paper.
dataset, proposed in Saravia et al. 2018), consists of 16,000 train, 2,000 validation and 2,000 test samples, each sample belonging one of the 6 classes (class 0-5): sadness, joy, love, anger, fear and surprise. To induce the notion of noisy labels, we randomly flip labels of 1% of class 5 train samples any another random class label and let the model train till we reach 100% train accuracy (memorizing the noisy labels).
Emotions
TweetTopic dataset, developed in Antypas et al. 2022]. consists of 2,858 train, 352 validation, and 376 test samples, spanning across 6 classes (class 0-5); arts trtiss_ccuturre business_ laaiiesss__eeetteeee &_entrepreneurs. pop_culture, science__t Lete&&t_eeehhp__e
sports_& fatisss_geaat &_gaming, daily _ife.. To introduce memorization of noisy labels, we flip labels of 1% of class 3 train samples to any another random class label while training the model till it reaches 100% train accuracy.
News dataset, proposed in Okite97 [024), consists of 4,686 train and 828 test samples, spanning across classes (class 0-5): business, sports, politics, health, entertainment and tech. We then split the train set to train & validation using 90: 10 stratified split over the class labels. Also, we introduce noisy labels, by flipping labels of 1% of class 5 train samples to any another random class label, while letting the model overfit to 100% train accuracy.
CIFARIO dataset, first introduced in Krizhevsky et al. (209)] consists of 60,000 samples, with each belonging to one of the 10 classes (class (1-)): airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. our setup we consider subset of 16,000 train, 4,000 validation and 10,000 test samples. We also resize the images to size (224x224x3) for training the transformer model. To introduce noisy labels, we flip labels and train the model till it reaches 100% train accuracy.
of class train samples to any another random class label,
UTK-Face dataset, proposed in Zhang et al. [2017). consists of 23,705 samples and classes (classs 0-4) depicting ethnicity: white, black, asian, indian and others. We split the dataset into train, validation and testing using 65:15:20 stratified split. We also resize the images to size (224x224x3) for training the transformer model. We then introduce noisy labels, by flipping labels of 1% of class 2 train samples to any another random class label, and train the model till it achieves 100% train accuracy.
NICO+ dataset [Zhang et al., 2023] consists of approximately 60,000 images distributed across 60 categories of everyday objects. In this study, we select subset of 15 object categories, including car, flower, penguin, camel, chair, monitor, truck, wheat, sword, seal, lion, fish, dolphin, lifeboat, and tank. The dataset is partitioned into training (80%), validation (10%), and testing (10%) sets using stratified sampling. Images are resized to (224x224x3) for consistency with the model input requirements. We then introduce noisy labels, by flipping labels of 1% of class train samples to any another random class label, and train the model till achieves 100% train accuracy.
F.2 Models
Below we discuss the 13 transformer models (6 Post-LN and 7 Pre-LN) considered as part of our paper. We utilize the Sequence Classification variant of all the models available on Huggingface
31

---

Post-LN Models
Description
BERT Devlin et al. 2019
12-layer bidirectional transformer for masked language modeling and next sentence prediction.
DeBERTa He et al.
2020
12-layer model with disentangled position/content embeddings and decoding-enhaneee attention.
RoBERTa
Yinhan et al.
12-layer robustly optimized BERT variant trained with dynamic masking and more data.
2019
ELECTRA Clark 2020
12-layer model using replaced token detection for sample- efficient pre-training.
DistilIBERT
Sanh al.
6-layer distilled BERT that is smaller and faster variant of BERT,
2019
retaining strong performance.
Longformer
Beltagy al.
12-layer model with sparse attention for efficient sengseeqeence processing.
2020
Pre-LN Models
Description
GPT2- Medium
Radford
24-layer unidirectional transformer trained for causal language modeling.
ct al. 2019
NPP-Nco-125M
Black et al.
12-layer open-source causal language model trained on The Pile dataset
2022
Owen2-0.5B
Yang et al.
24-layer efficient LLM optimized for generative tasks, using RMSNorm [Zhang and Sennrich, 2019]
2024
EEEEERTA-PreLayetee
24-layer variant of RoBEKIa with Pre-LN setting for improved
Out et al.
2019
training stability.
V
Alexey
2020
12-layer Vision Transformer Base model for image classification.
ViT-S
Assran el al.
2022
12-layer smaller VIT variant trained with Masked Siamese Net- works (MSN).
DeiT Touvron et al.
2021
12-layer Data-efficient Image Transformer trained with distilla- tion, without external data.
Table 3: Overview of the 13 transformer models categorized into 6 Post-LN and Pre-LN architec- tures.
F3
Training Settings & Hyperparameters
In our study, we explore various combinations of different models and datasets across our experiments, follows (1) Emotions dataset with BERT (Post-LN), DeBERTa (Post-LNN, DistillBERT (Post-LNN) GPTNeo (Pre-LN), (2) News dataset with ELECTRA (Post-LNN, Longformer (Post-LN), Qwen2 (Pre-LN), (3) Tweets dataset with RoBERTa (Post-LN), GPT2 (Pre-LN), RoBERTa-PPee Poyerayr
(Pre-LN), (4) CIFARIO 10 dataset with ViT-B (Pre-LN), (5) UTK-Face dataset with DeiT (Pre-LN), and (6) NICO+ dataset with VIT-S (Pre-LN), We fully unfreeze all layers of the pre-trained models during training.
Regarding the hyperparameters, consider a learning rate of 2e-5 and batch size of 16 for all models. Then, we train the Post-LN models for 40 epochs and Pre-LN models for 70 epochs. We run 70 epochs for Pre-LN models, because after removal of LN parameters, the learning accuracy is impacted significantly in Pre-LN models from the start. Hence, we examine if the accuracy would increase by letting the model train more. In addition that, we do not any data augmentation in our training procedures. We also provide the code for our experiments in the supplementary file.
F.4 Grouping of Early, Middle, and Later Layers
Early Layers
Middle Layers
Lator Layers
Exrly Layers
Mindle Layers
Later Layers
(a) Pre-IN architecture
(b) Post-IN architecture
Figure 5: Pre-LN vs. Post-LN architectures depicting LN placement and categorization of early. middle and later layers
32

---

To investigate which group of layers' Layer Normalization (LN) contributes most to memorization and learning, we divide the transformer layers into three subsets: Early, Middle, and Later layers as shown in Fig.
Formally, for transformer model with N layers (where N is divisible by 3), define:
Early Layers = {1, 2,
N
Middle Layers = { 3
N
+ 1,..., 2N
(112)
Later Layers
{ 12 +,, +,...,,
This grouping helps separately examine which set of layers most significantly influences learning and memorization in transformers.
G Additional Experiments & Results
This section presents additional experiments (on top of the ones discussed in Sec. Sec. and Sec. 6)and results on supplementary datasets and models, expanding on the analyses Sec. G. G.2 and G.3. These experiments aim to: (1) establish the distinctive impact of Layer Normalization
(LN) on memorization and learning Pre-LN vs. Post-LN models, (2) assess the role of LN in early layers, and (3) investigate how gradient behavior accounts for the observed phenomena.
G.1 Impact of LN on Memorization & Learning in Pre- and Post-LN models
In this section, we present the results corresponding the distinctive impact of LN of memorization and learning for the remaining Pre and Post LN models spanning multiple datasets. These results further corroborate our finding that removal of LN parameters in pre-LN models critically destabilizes learning while exacerbating overfitting, while for Post-LN models, LN parameters removal, suppresses memorization and facilitates true label recovery without impacting learning-
The results are verified against Pre-LN models GPTNeo, GPT2, ViT-B, DeiT, VIT-S, RoBERTa- PreLayemnrrm and Post-LN models BERT, DeBERTa, Longformer, RoBERTa, DistilBERT spanning across multiple language and vision datasets Emotions, News, Tweets, CIFARIO, NICO+, UTK- Face, are provided in Figs.
12
and 16
G.1.1 Pre-LN models Learning Destabilized
Randor
14
Tran Asturacy Liirurzze
Ace beierr cerveoel
Spects
IN removed
(a) Leamiing (Test) accuracy epochs for Pre-LN Model (GPTNeo)
(b) Memorization, Recovery and Ran-
(c) Overfitting gap for Pre- dom Predictions over epochs for Pre- Model (GPTNeo)
Model (GPTNeo)
Figure 6: LN removal destabilizes learning in Pre-LN model GPTNeo, Emotions Dataset: LN removal critically affects learning while memorization still persists GPTNeo. This further exacerbates overfitting, explained by increasing train-test accuracy gap when LN is removed, due the drop in test-accuracy.
33

---

17.5
12.5-
16.0
Tha Adtraay Liirmzz
7.5
3rd Nr parr barkes Surevwol
5.0
25:
0.0
L prwsent
LN
(a) Leamiing (Test) accuracy epochs for Pre-LN Model (GPT2)
(b) Memorization, Recovery and Ran-
(c) Overfitting gap for Pre- LN Model (GPT2)
dom Predictions epochs for Pre-
Model (GPT2)
Figure 7: LN removal destabilizes learning in Pre-LN model GPT2, TweetTopic Dataset: LN removal critically affects learning while memorization still persists in GPT2. For GPT2, the overfitting gap decreased after LN removal, because the model could not even stabilize during training due to the destabilization of learning. Hence both train and test accuracies remain low and comparable. However, the learning accuracy still remains low when LN absent in comparison when LN was present, and struggle with high memorization and random predictions (red-color family bars).
Kucdar Robiie
Sunwy
30
Tean hrrriryyyy Sireeso Accunccy nirorol tra
20
10-
presont
(a) Learning (Test) accuracy over epochs for Pre-LN Model (ViT-B)
(b) Memorization, Recovery and Ran-
(c) Overfitting gap
dom Predictions over epochs for Pre- LN Model (ViT-B)
LN Model (VIT-B)
Figure 8: LN removal destabilizes learning in Pre-LN model VIT-B, CIFARIO Dataset: LN removal critically affects learning while memorization still persists in ViT-B. This further exacerbates overfitting seen by increasing train-test aceuracy gap when LN removed, due to the drop in test- accuracy.
Kodia Bobilesfos
Broms
30-
Soom DN
20
fot Are aite siveesa the and traning cenii
10
Speene
LN presen
removeed
(a) Learning (Test) accuracy over epochs for Pre-LN Model (DeiT)
(b) Memorization, Recovery and Ran- dom Predictions over epochs for Pre- LN Model (DeiT)
(c) Overfitting gap for Pre-
LN Model (DeiT)
Figure 9: LN removal destabilizes learning in Pre-LN model DeiT, UTK-Face Dataset: LN removal critically affects learning while memorization still persists in DeiT. This further exacerbates overfitting seen by increasing train-test accuracy gap when LN is removed, due to the drop in test- accuracy.
34

---

Train Astuaey aftrr Uiremnr
Sports
Uirahoved
(a) Learning (Test) accuracy epochs for Pre-LN Model (ViT-S)
(b) Memorization, Recovery and Ran-
(c) Overfitting gap for Pre- dom Predictions over epochs for Pre- Model (ViT-S)
LN Model (VIT-S)
Figure 10: LN removal destabilizes learning in Pre-LN model VIT-S, NICO+ Dataset: LN removal critically affects learning while memorization still persists in ViT-S. This further exacerbates overfitting seen by increasing train-test accuracy gap when LN removed, due to the drop in test- accuracy.
112.
Doow
aiter venoval
7.5
Taa Accusssy betira renovll
5.0
2.5
0.0
LN presont
LN remevos
(a) Learning (Test) accuracy over epochs for Pre-LN Model (RoBERTa- dom Predictions epochs for Pre- PreLayerNorm)
(b) Memorization, Recovery and Ran-
(c) Overfitting
gap for Pre-LN Model (RoBERTa-
LN Model (RoBERTa-Prele Preayyerom)
PreLayerNorm)
Figure 11: LN removal destabilizes learning in Pre-LN model RoBERTa-PreLayyrNNo
TweetTopic Dataset: LN removal critically affects learning while memorization still persists in RoBERTa-PreLLyerti
For Prraaaaroore the overfitting gap decreased after LN removal, because the model could not even stabilize during training due to the destabilization of learning. Hence both train and test accuracies remain low and comparable. However, the learning accuracy still remains low when LN is absent in comparison to when LN is present, and struggles with high memorization and random predictions (red-color family bars).
35

---

G.1.2 Post-LN models
Suppression of Memorization & True Labels Recovery
Tran Asuraey Limumar
Oporrs
rereoveed
(a) Leariing (Test) accuracy epochs for Post-LN Model (BERT)
(b) Memorization, Recovery and Ran-
(c) Overfitting gap for Post-
dom Predictions over epochs for Post- Model (BERT)
LN Model (BERT)
Figure 12: LN removal suppresses memorization & facilitates true label recovery in Post-LN model - BERT, Emotions Dataset: LN removal in BERT suppresses memorization and facilitates true label recovery, while keeping learning intact. This further reduces overfitting seen by decreasing train-test accuracy gap when LN is removed.
Monaa Siiior RANdAY ProDCci Mare
10
Sweet:
Scom
"hat Acc aver bertoa rensovl
Cporra
UN present
LN removed
(a) Learning (Test) accuracy over epochs for Post-LN Model (DeBERTa)
(b) Memorization, Recovery and Ran-
(c) Overfitting gap for Post-
dom Predictions over epochs for Post- LN Model (DeBERTa)
LN Model (DeBERTa)
Figure 13: LN removal suppresses memorization & facilitates true label recovery in Post-LN model DeBERTa, Emotions Dataset: LN removal in DeBERTa suppresses memorization and facilitates true label recovery, while keeping learning intact. This further reduces overfitting seen by decreasing train-test accuracy gap when LN is removed.
fardur hylolesion
Noves
14
Tein Asuraay aftrr Lirumr
Accuncey
Seices
(a) Learning (Test) accuracy over (b) Memorization, Recovery and Ran- epochs for Post-LN Model (Long- dom Predictions over epochs for Post- former)
(c) Overfitting gap for Post-
LN Model (Longformer)
LN Model (Longformer)
Figure 14: LN removal suppresses memorization & facilitates true label recovery in Post-LN model Longformer, News Dataset: LNN removal in Longformer suppresses memorization and facilitates true label recovery, while keeping learning intact. This further reduces overfitting seen by decreasing train-test accuracy gap when LN is removed.
36

---

Nedia Prod NOIN
16
14-
12
10
Troi Acturaty Liiemnn
trannng
Soceta
(a) Learning (Test) accuracy over epochs for Post-LN Model (RoBERTa)
(b) Memorization, Recovery and Ran- (c) Overfitting gap Post-
dom Predictions over epochs for Post- LN Model (RoBERTa)
LN Model (RoBERTa)
Figure 15: LN removal suppresses memorization & facilitates true label recovery Post-LN model - RoBERTa, TweetTopic Dataset: LN removal in Longformer suppresses memorization and facilitates true label recovery, while minimal drop in learning. This further reduces overfitting seen by decreasing train-test accuracy gap when LN is removed. Note: For RoBERTa
slight drop in learning accuracy which is much lower in comparison to the huge drop which happens in GPT2 setup in Fig. 7 Furthermore, memorization is still suppressed in comparison to GPT2 where memorization still persisted.
Sardee PrOdli
Recwney
lea Accasiey after nenoial
the ondie connort
2
Speets
LN present
LN removved
(a) Learning (Test) accuracy over
(b) Memorization, Recovery and Ran-
(c) Overfitting gap for Post-
epochs for Post-LN Model (Distil- dom Predictions over epochs for Post- BERT)
LN Model (DistilBERT)
LN Model (DistilBERT)
Figure 16: LN removal does not suppress memorization in DistilBERT (Emotions Dataset) but does not affect learning: LN removal in DistilBERT does not suppress memorization (but does not impact learning) as we observe for all other 5 Post-LN models. We think other components in the Transformer architecture might have more profound impact on memorization in DistilBERT. But, since this work is primarily about LN impact, we refrain from dealing with other components, making it an interesting future work.
G.2 Significance of Early Layers LN
In this section, we illustrate the results corresponding to the significance of Early Layers LN in im- pacting learning and suppressing memorization for remaining Pre- and Post-LN models, respectively, across multiple datasets.
The results verified against Pre-LN models GPTNeo, Qwen2, GPT2, RoBERTa-PreL RoBERRaa-PeLayemmom ViT-B, VIT-S, and Post-LN models BERT, ELECTRA, Longformer, RoBERTa, DistilBERT are shown in Figs.
26
and 27
G.2.1 Pre-LN Models: Early Layers LN drives learning
Meneraatann Sark
Secetr Scrm
8
Ceech
Escore
Figure 17: Pivotal impact of carly LNs for learning in Pre-LN model (GPTNeo, Emotions Dataset). We clearly observe the impact of early layers LN removal destabilizing learning in
GPTNeo, accompanied with higher train-test-accuracy gap, A Ptc. early than for later layers, A Ptc. later overlit
and poor memorization suppression.
37

---

Exily Removed
Midd Lis femoved
Lacer Rerroved
Toxt Acc Presert
Suen When prisent
Backs
Figure 18: Pivotal impact early LNs for learning in Pre-LN model (Qwen2, News Dataset). We can clearly observe the impact of early layers LN removal on destabilizing learning in Qwen2, accompanied with higher train-test tes--uuurtcyy gap, A Pre, early than for later layers, A Averrit memorization suppression.
Pre, later and poor
overfit
Eaily UG Removee
Middlc Lis hamoved
Later Aorrowzd
Tor Acc when cscsk
Aownr Mich present
Sora
1 5
Epechs
Opech
Cocchs
Esocts
Figure 19: Pivotal impact early LNs for learning Pre-LN model (GPT2, TweetTopic Dataset), We can clearly observe the impact of early layers LN removal on destabilizing learning in GPT2, accompanied with higher train-test-acuuracy gap, Are
carly ,than for later layers, A later and poor memorization suppression. Note: When all layers LNs are removed, A all when we removed all LNs from GPT2 then the model could not stabilize in training, reaching very low train accuracy, similar train-test accuracy
overfit is low because
in Fig. 7a hence low A Pre. all However, when just removed early LNs, the model is able to converge sufficiently high train accuracy, but the learning is impacted much severely in comparison to later layers.
overfit
Extly (Ns Semroed
Widd LNe. Semoved
Lattrr INS Remound
The Art when Presect
.on Qownn Sthnn present
Arvert
Spede
Figure 20: Pivotal impact of early LNs for learning in Pre-LN model (RoBERTa-PreLayerVee) Tweet'opic Dataset). We observe that for RoBERTa-PreLayeer PreLayerNorm. removing early layers LN impacts learning, however, memorization also seems to get suppressed, when compared with removing all LNs where we observe persistent memorization and destabilized learning (Fig. Tab. Despite this unique trend of RoBERTa- RoBERTa-PreLayemnon PreLayernorm (not following the consistent trend just as in the other Pre-LN models), early layers LN still remain the most significant.
38

---

Exily Remored
Midd Lis femoved
Lacer Rerroved
Toxt Acc when Presert
Sunn prisent
2 9
5
Tpa
Figure Pivotal impact of early LNs for learning Pre-LN model (ViT-B, CIEARIO Dataset). We can clearly observe the impact of early layers LN removal destabilizing learning in ViT-B, accompanied with higher memorization suppression.
gap, A bee, early than for later layers, A Pre, later and poor
Exfly (Ns Remroedd
Wide INe Semoued
Eater INs Remound
Tee Arc when Precent con. Qownn Shen present
Figure 22: Pivotal impact of early LNs for learning in Pre-LN model (CIFARI00, ViT-S Dataset). We can clearly observe the impact of early layers LN removal on destabilizing learning in ViT-S, accompanied with higher memorization suppression.
Pre, later early than for later layers, A Averritt and poor gap, A Averfit
G.2.2 Post-LN Models: Early Layers LN suppresses memorization
Eafly UNS Remroved
Widele LNd. Removed
Later LBS Removed
Test Acc when Present
... Aowns Shan present
200
g
Recounte
0
8
Opecta
Coocha
Coota
Figure 23: Pivotal impact early LNs on memorization in Post-LN model (BERT, Emotions Dataset). We can clearly observe the impact of early layers LN removal on suppressing memorization & achieving true label recovery in BERT, accompanied with higher train-test-accuracy gap, A Pre, early than for later layers, A Pte, later while learning being intact. overfit
A
39

---

Eaily Lis
Lacer
Tex, ALC
Aownt
5 g 8
Duvertt
Figure 24: Pivotal impact of early LNs on memorization in Post-LN model (ELECTRA, News Dataset). We can clearly observe the impact of early layers LN removal on suppressing memorization & achieving true label recovery in ELECTRA, accompanied with higher cain-test-accuracy gap, Pre, early than for later layers, A Pre. later while learning being intact. overfil
A overlit
Aown
ww
Scom
Duport
Figure 25: Pivotal impact of early LNs on memorization in Post-LN model (Longformer, News Dataset). We can clearly observe the impact of early layers LN removal on suppressing memorization & achieving true label recovery in Longformer, accompanied with higher train-esttacccuracy gap, while learning being intact.
A Pte, early than for later layers, A Pre. later overfit
overfit
Eady Semoved
Widd Life Semoved
Later Remoued
Teat Art
Presect
Avenc
245
8 g
Auvertt
3
Epecho
Cpech
Cooch
Eoocta
Figure 26: Pivotal impact of early LNs on memorization in Post-LN model (RoBERTa, Tweet- Topic Dataset). We can clearly observe the impact of early layers LN removal suppressing memorization & achieving true label recovery in RoBERTa, while learning being minimally impacted. Note: In the case of RoBeRTa, A Pv, early is slightly higher than carly because even though removing early LNs, led to a greater memorization suppression than later LNs, but learning also got affected slightly, hence, the overfitting gap increased. However, need to compare this Post-LN result with its Pre-LN counterpart of GPT2 (Fig. 19), where early LNs removal, affected learning more, without suppressing memorization.
2
Eanchs
Coera
So0ths
Eoctss
Figure 27: Pivotal impact of early LNs on memorization in Post-LN model (DistilBERT, Emo- tions Dataset). For the case of DistilBERT, observe that just like when all LNs removal could not suppress memorization, the same happens with early LNs removal. However, we want to draw attention the delay in achieving memorization. i.c. early LNs removal results in the highest delay in memorization compared to Middle/Later LNs removal. This shows that Early LNs are still the most significant.
40

---

G.3 Gradients explain the impact of LN on Memorization & Learning
In this section, we provide additional results understand the distinctive impact of LN on memoriza- tion and learning through the lens of gradients. These results provide a deeper understanding of why (1) LN removal destabilizes learning Pre-LN models, and suppresses memorization in Post-LN models, (2) Early Layers LN are more significant than later layers LN in driving these phenomena.
G.3.1 Language Datasets Results
In the language modality, we experimented with additional Pre-LN (GPTNeo, GPT2, Qwen2) and Post-LN models (BERT, RoBERTa, Longformer, ELECTRA) across Emotions, News, and Tweets
Datasets. The results for the same are depicted Figs. 28
29 and
30
1000
Gradient Norm
Ratio
4000
Ratio
80
3000
10
Layer Index
(a) Pre-LN (Qwen2) ||9|
& @
analysis
across layers
Gradient Norm
Ratin
Gradient Norm
Raho
Ratio
XX4
D
L004
2004
Layer Index
Layer Index
(b) Post-LN (ELECTRA) Jeara analysis across layers
&
ll mema
(c) PosI-LN (Longformer) analysis across layers
||
2 &
1,92
mcm
Figure 28: Learning vs. Memorization Gradients in Pre- and Post-LN Models (News Dataset): Results clearly exhibit high gradient norms of early layers LNs than later layers for both learning and memorization in Pre-LN (Qwen2) and Post-LN (ELECTRA, Longformer) models. Importantly, the learning gradient norms ([1g (2) are consistently higher than the memorization gradient norms (Ilg I22) across all layers. Furthermore. the ratio ||g] 2l// ||g_em| ll is significantly higher in Pre-LN models compared to Post-LN models.
41

---

Gradient Norm
Gradient Norm
L Ratio
8
4122
F:51 1555 Layer Index
Layer Index
(a) Pre-LN (GPT2)
Jeam
|19 mem
analysis
(b) Pre-LN (RoBERTa-PreLLaeeet ||9 |9
||
across layers
ll analysis across layers
411
Gradient Norm
1
Ratio
6000
370
Ratio
250
230
6000
230
200
6000
Layer Index
(c) Post-LN (RoBERTa) Jearn
& 119 mem
analysis
across layers
Figure 29: Learning Memorization Gradients in Pre- and Post-LN Models (Tweet Dataset): Results clearly exhibit high gradient norms of early layers LNs than later layers for both learning and memorization in Pre-LN (GPT2, RoBERTA-PreeayerNON
and Post-LN (RoBERTa)
models. Importantly, the learning gradient norms ([gg/am ( are consistently higher than the mem- orization gradient norms (1ggemml (|2) across all layers. Furthermore, the ratio ||g]/m/ ||2 1em ll is significantly higher in Pre-LN models compared to Post-LN models.
E.0%s
Gradient Norm
Roto
5.05
5.00
0.00
0.09
0.70%
5.00
Layer Index
(a) Pre-LN (GPTNeo) sis across layers
||
1|2
& |1g.
1|2 analy-
Gradient Norm
Bani
Gradient Norm
Bhump Ratn
Rabio
Rabio
80000
0000
Layer Index
Layer
(b) Post-LN (BERT) across layers
||g Jeam|
||,
& 1 mem ll analysis
(c) Post-LN (DistilBERT) ||g Jam analysis across layers
& ||g
mom
l
Figure 30: Learning vs. Memorization Gradients Pre- and Post-LN Models (Emotions Dataset): Results clearly exhibit high gradient norms of early layers LNs than later layers for both learning and memorization in Pre-LN (GPTNeo) and Post-LN (BERT, DistilBERT) models.
Importantly, the learning gradient norms ( ([ (1 consistently higher than the memorization leam
gradient norms (ll ( across all layers. Furthermore, the ratio [g higher in Pre-LN models compared to Post-LN models.
|| [l, is significantly
42

---

G.3.2 Vision Datasets Results
For the vision modality, it needs to be acknowledged that Post-LN architectures not available in practice/literature, and only Pre-LN models are available. Hence, we provide additional experiments for Pre-LN models VIT-B, ViT-S, and DeiT using multiple datasets CIFARIO, CIFARIOO, 100, UTK- Face. The results for the same are presented in Figs. 31 32
and 33
Gradient Norm
Ratio
6:33
120
Ratio
0:43
0.34
90
0:23
10
6.41
0.90
Layer Index
(a) Pre-LN (ViT-B)
9
learn
I men
ll analysis across
layers
Figure 31: Learning vs. Memorization Gradients Pre-LN Models (CIFAR1O Dataset): Results clearly exhibit high gradient norms of early layers LNs than later layers for both learning and memorization in Pre-LN (ViT-B) models. Importantly, the learning gradient norms (I Jeam [[]) are consistently higher than the memorization gradient norms (1lg 2 across all layers.
Gradient Norm
Igranp
0:55
Ratio
0.54
0.22
8
1 [ mom
12
12
0.32
0.00
Layer Index
(a) Pre-LN (ViT-S)
g
Jeam
& 11 mem analysis across
layers
Figure 32: Learning vs. Memorization Gradients in Pre-LN Models (NICO+ Dataset): Results clearly exhibit high gradient norms of early layers LNs than later layers for both learning and memorization in Pre-LN (VIT-S) models. Importantly, the learning gradient norms (1gg are consistently higher than the memorization gradient norms (Ilgmem |[2) across all layers.
Gradient Norm
Ratio
0.75
Ratio 1
0.35
0.54
0.32
0.32
0.51
6.90
Layer Index
(a) Pre-LN (DeiT)
||g Jeam
||,
I| mom 1l analysis across
layers
Figure 33: Learning vs. Memorization Gradients in Pre-LN Models (UTK-Face Dataset): Results clearly exhibit high gradient norms of early layers LNs than later layers for both learning and memorization Pre-LN (DeiT) models. Importantly, the learning gradient norms (Ilg). consistently higher than the memorization gradient norms (Ilgmem Il2) across all layers.
[|lleam [|,) are
43

---

H
Analysis across multiple Noisy Label Ratios and Optimizer
We extend our analysis evaluate whether the clavn-removing LN parameters mitigates memoriza- fion in Post-LN models while impairing generalization in Pre-LN models holds consistently across higher noisy label ratios (2% and 5%) the training dataset, as shown Table
Noise
Model
Setting
Learning (1)
Memorization (1)
Recovery (t)
Random Prediction (L)
Post-LN (BERT)
Before
91.70
100.00
0.00
0.00
After
92.00
20.62
76.25
3.12
2%
Pre-LN (GPT-Nee)
Before
91.35
100.00
0.00
0.00
After
84.85
66.87
16.56
16.56
Post-LN (BERT)
Before
90.35
100.00
0.00
0.00
After
91.25
27.00
66.88
6.12
5%
Pre-LN (GPT-Neo)
Before
90.35
100.00
0.00
0.00
After
82.60
67.50
11.00
21.50
Table 4: Results for higher noisy label ratios (2% and 5%) on the Emotions dataset.
Additionally, assess the robustness of this finding under different optimization setting by experimenting with Muon optimizer other than Adam, which was used in the primary experiments, shown in Table
Model
Setting Learning (1)
Memorization (L)
Recovery (f)
Random Prediction (L)
Post-LN (BERT)
Before
91.95
100.00
0.00
0.00
After
92.00
25.00
62.50
12.50
Pre-LN (GPT-Neo)
Before
91.70
100.00
0.00
0.00
After
85.55
32.50
29.38
38.12
Table 5: Results with the Muon optimizer with Emotions Dataset.
Overall, the results demonstrate that our observations agnostic to optimizer and noise-label ratios.
I Loss and Gradient Norms Analysis across Epochs
In Sec. 4 we showed how on LN parameters removal Post-LN models, memorization mitigated epochs, while in Pre-LN models, the testing accuracy drops as training progresses, without recovering at any time. To further corroborate this claim, we plot the train-test loss functions for Post-LN and Pre-LN models, where the overfitting gap decreases in Post-LN models, while for Pre-LN models it increases, upon LN removal, as shown in Fig. 34
Apart from the loss curves, also provide how the gradient norm evolves across epochs for both Post-LN and Pre-LN models in Fig. 35
44

---

6.4
Test Loss before LN removal
1.2
Test Loss after removal
1.0
LOSS.
0.4
02-
Epochs
Epochs
(a) BERT (Post-LN) before LN removal.
(b) BERT (Post-LN) after removal.
1.6
Train Loss before LN removal
Train
1.4
1.2
1.0
0.6
0.2
6.0
Epochs
(c) GPT-Neo (Pre-LN) before LN removal.
(d) GPT-Neo (Pre-LN) LN removal.
Figure 34: Train and test loss trends across epochs for Post-LN (BERT) and Pre-LN (GPT-Neo) models before and after LayerNorm (LN) removal. In Post-LN models, LN removal reduces the overfitting gap (difference between train and test losses), mitigating memorization. In Pre-LN models, LN removal widens the overfitting gap, indicating impaired generalization.
Layer
Layer
Gradient Norm
0.0010
Layer
Layer
Gradient Norm
Laper
Layer
Layer
Layer
0.0008
Layer
Layer
0.0006
Layer
Layer
Layer
Layer
0.0004
0.0002
0.0000
Epoch
Epoch
(a) Gradient norms for BERT
(b) Gradient norms for GPT-Nco
Figure 35: Gradient norms gradually increase across epochs for both Pre-LN and Post-LN models, with early layers exhibiting the highest gradient norms.
J Validity of Results for Generative Modeling Task
To verify the robustness our findings in a generative modeling context, we conduct Next Token Prediction (NTP) experiment on the Emotions dataset using both BERT (Post-LN) and GPT-Neo (Pre-LN) models.
Each input sample is reformulated as:
original text + "This emotion is [type]"
where the model predicts the token corresponding the emotion label [type]] one of six possible emotion types. To introduce noisy labels, we randomly replace the [type] token for 1% of the training samples with different emotion label. Two different model configurations trained and evaluated: (i) before LN parameters removal, and (ii) after LN parameters removal.
As shown in Table removing LN parameters substantially reduces memorization in Post-LN models (BERT), while impairing learning and generalization in Pre-LN models (GPT-Neo). This confirms that the trends reported in the main paper hold consistently for generative modeling tasks as well.
45

---

Table Results Next Token Prediction task with noisy labels.
Model
Setting
Learning (1)
Memorization (L)
Recovery (T)
Random Prediction (4)
Post-LN (BERT)
Before
92.14
100.00
0,00
0,00
After
91.95
28.12
60.62
11.25
Pre-LN (GPT-Neo)
Before
91.70
100.00
0,00
0,00
After
85.55
51.25
19.38
29.38
K
Broader Impacts and Limitations
Our study reveals that LayerNorm (LN) affects memorization and learning differently across trans- former variants: disabling LN parameters suppresses memorization and aids label recovery in Post-LN models, but destabilizes learning in Pre-LN models. These insights can inform architecture design and robust training in noisy labels settings, where memorization needs be controlled. While we focus on LN due to its central role and controllability. other components- like residual paths, attention, and feedforward layers-also influence memorization and merit further investigation. We hope this work can provide insights to the community to encourage further follow-up studies.
46

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have systematically demonstrated the distinct impact of LN on memoriza- tion and learning across Pre- and Post-LN models using various experiments and results in and
Sec. 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper reflects the limitations of prior work, while emphasizing the novel results of the distinctive impact of LN memorization and learning.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
47

---

Answer: [Yes]
Justification: Yes, complete and correct proofs are provided for the Theorems provided in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, the paper fully discloses all the important information in replicating the experiments done in it.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
48

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Yes, code corresponding to the results is provided in the supplementary file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co cc/
wtt.://wbmmmiiiooooooooooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co. ccippi.cc guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, proper training details (data splits, hyper-parameters, optimizer, etc.) are provided in the Appendix. Furthermore, the code corresponding the results is also provided in the supplementary file.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the experiments are done across 3 random seeds for robustness in results, while providing necessary error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
49

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, necessary compute resources specification are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httele:///euuuiee /t/preeurp...// cwtrrips.ccppuullle cccppbliccBthiccciieeel...
Answer: [Yes]
Justification: Yes, the paper conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we have added seperate section on discussing broader impacts and limitations of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
50

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, all existing datasets are properly cited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
51

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, proper documented code is provided for the experiments done in the paper.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
52

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: httpps:///eurips.... /ctp////eeeeeeecc///22///////// for what should or should be described.
53


Reviews:
[{"review_id": "GxPYQw9LzI", "reviewer": "Reviewer_Fd4h", "summary": "The paper investigates how Layer Normalization (LN) affects memorization versus learning in transformers with Pre-LN and Post-LN designs. The key takeaway is: removal of LN parameters in Pre-LN models significantly destabilizes the learning process, leading to persistent overfitting. In contrast, removing LN parameters from Post-LN architecture designs effectively mitigates memorization and enables the recovery of genuine labels.", "questions": "Please see Strengths and Weaknesses Section for a list of questions.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) **Clear, under-explored question.** LN is ubiquitous in modern transformer design, yet its role in memorization vs. learning has not been clearly established; the paper studies this gap.\n\n2) **Broad empirical study.** Extensive results for both language and vision modalities, including 3 language and 3 vision classification datasets, and 7 Pre-LN and 6 Post-LN transformers architectures.\n\n3) **Simple, reproducible intervention.** Dropping γ, β is easy to implement and isolates LN’s specific contribution.\n\n4) Code is provided.\n\n\nWeaknesses:\n1) **Noise regime.** Results use 1 % random label noise. Do the Pre-/Post-LN findings persist with higher noise rates? A sensitivity plot is required to verify the findings.\n\n2) **Task scope.** All benchmark tasks are classification. Can you report at least one generative task (e.g., language modelling) to demonstrate that the conclusions generalize? A toy/ synthetic task is fine.\n\n3) **Standard deviation of experiments.** The paper mentions three seeds, but error bars are missing.\n\n4) **What optimizer did you use?** Could the optimizer choice—say switching from Adam to a modern variant like Muon—alter the Pre- vs. Post-LN learning/memorization findings? A brief explanation would be helpful.\n\n\nRandom question for my own learning: We know that many Pre-LN transformers produce large activation values—this is true even for generative models like DiTs. Do you think there’s a relationship between LayerNorm-related overfitting and the presence of activation outliers, or are the outliers simply an artifact of the optimizer?\n\nIf the authors satisfactorily address the questions above, I’m prepared to raise my score to 5 (Accept). Overall, this is an interesting paper with several insights.", "was_revised": true, "final_justification": "Thank you authors for the rebuttal. Authors have addressed all my concerns.\n\nIn the revised version, please include results on different noise ratios, next-token prediction task and Muon optimizer.\n\nOverall, this paper is interesting, well executed, and constitutes a good scientific contribution in my opinion. I will increase my score to 5 (Accept).", "rebuttal": "**We thank the reviewer for their comments and feedback. Below, we address their questions to the best of our ability in the given time frame.** \n\n>**Weaknesses:**\n>>**Q1:** As per your suggestion, we provide experiments on higher label noise ratios - 2% and 5% for the Emotions dataset using BERT (Post-LN) and GPTNeo (Pre-LN) models. \n>>|Noise|Model|Setting|Learning&nbsp;(%, ↑ is better)|Memorization&nbsp;(%, ↓ is better)|Recovery&nbsp;(%, ↑ is better)|Random Prediction&nbsp;(%, ↓ is better)|\n>>|-|-|-|-|-|-|-|\n>>|2%|Post-LN (BERT)|Before|91.7|100|0|0|\n>>|||After|92.0|20.62|76.25|3.12|\n>>||Pre-LN (GPTNeo)|Before|91.35|100|0|0|\n>>|||After|84.85|66.87|16.56|16.56|\n>>|5%|Post-LN (BERT)|Before|90.35|100|0|0|\n>>|||After|91.25|27.0|66.88|6.12|\n>>||Pre-LN (GPTNeo)|Before|90.35|100|0|0|\n>>|||After|82.6|67.5|11.0|21.5|\n>>&nbsp;\nFrom the results we can clearly confirm that even for higher proportions of label noise, LN has an impact on memorization in Post-LN model (BERT), where their removal suppresses memorization. While in Pre-LN model (GPTNeo) case, LN removal impairs learning without mitigating memorization.\n\n>>**Q2:** We acknowledge the reviewer’s comment and carry out a Next Token Prediction Task on the Emotions dataset for both BERT (Post-LN) and GPTNeo (Pre-LN). We modify the classification dataset as follows - **original text + “This emotion is [type]”**. We then predict the **“[type]”** of the token which can be one of the 6 emotions. To induce the notion of noisy labels in language-modeling, we change the **“[type]”** token for 1% of the samples to a random different emotion in the train set. Following this setup, we train 2 variations of both Pre- and Post-LN models, (1) without removing LN, and (2) with removing LN and provide the results below.\n>>Model|Setting|Learning&nbsp;(%, ↑ is better)|Memorization&nbsp;(%, ↓ is better)|Recovery&nbsp;(%, ↑ is better)|Random Prediction&nbsp;(%, ↓ is better)|\n>>|-|-|-|-|-|-|\n>>|Post-LN (BERT)|Before|92.14|100|0|0|\n>>||After| 91.95|28.12|60.62|11.25|\n>>|Pre-LN (GPTNeo)|Before|91.7|100|0|0|\n>>||After|85.55|51.25|19.38|29.38|\n>>&nbsp;\nWe observe that even for this language modeling task, LN has an impact on memorization in Post-LN model (BERT) while it impairs learning for Pre-LN model (GPTNeo). As a result, **we verified that further results on generative tasks also corroborate the claims of the paper.**\n\n>>**Q3:** Although we provided the error bars in most of the plots, if, in some charts, they are not that visible, that is because the std-dev is too less. However, we could not provide error bars in Memorization-Recovery-Random Prediction charts due to their nature (bifurcated into 3 regions), however, we will provide them in a tabular form in the Appendix in a revised version.\n\n>>**Q4:** We used Adam as the optimizer. As per the reviewer’s suggestion, we also evaluate our setup with Muon optimizer. For Muon as well, we observe that  LN has an impact on memorization in Post-LN model (BERT) while it impairs learning for Pre-LN model (GPTNeo), as shown below:\n>>|Model|Setting|Learning&nbsp;(%, ↑ is better)|Memorization&nbsp;(%, ↓ is better)|Recovery&nbsp;(%, ↑ is better)|Random Prediction&nbsp;(%, ↓ is better)|\n>>|-|-|-|-|-|-|\n>>|Post-LN (BERT)|Before|91.95|100|0|0|\n>>||After| 92|25|62.5|12.5|\n>>|Pre-LN (GPTNeo)|Before|91.7|100|0|0|\n>>||After|85.55|32.5|29.38|38.12|\n\n>>**“Random question…”:** That is an interesting and thoughtful point to consider. It sounds indeed plausible that while memorizing a label, the model can rely on certain tokens, which could lead to disproportionately large activations, from a general perspective. Having said that, as it is not what we have been thinking about rigorously, we may not be the best to adequately answer the question. We humbly relay the opportunity to explore the point to the reviewer or other adequate researchers. Hope we could see some interesting insights regarding the point in upcoming conferences in the near future. **Thank you again for extending thoughts, building on our work.**\n\n**We hope our response resolves your concerns, and helps you consider raising the score. We would be happy to answer any more questions that you may have. Thank you.**"}, {"review_id": "KTMMf67KNf", "reviewer": "Reviewer_qby4", "summary": "The paper investigates how Layer Normalization (LN) influences label memorization and learning in transformers configured with pre-LN and post-LN placement across 13 models (7 pre-LN, 6 post-LN) and six language and vision classification datasets. This paper shows that the LN parameter is crucial in pre-LN models, unlike post-LN models. Without LN, pre-LN models suffer large drops in test accuracy, indicating that LN is essential for learning stability and mitigating memorization. However, the removal of the LN parameter in post-LN models suppresses memorization and recovers genuine labels. This Paper also argues that early LN is crucial and explains those phenomena with the gradient of each LN.", "questions": "- Q1: The paper says removing the early layer of LN significantly disrupts performance. I suspect this may stem from attention entropy collapse [1]. It would be informative to compare attention entropy in pre-LN models with and without LN and to examine how those entropy changes correlate with model accuracy and label memorization.\n- Q2: Once the model has learned all data patterns, its logit vector norm continues to grow, driving the cross-entropy loss down excessively and leading to memorization of the training set [2]. This paper also shows that applying simple logit vector normalization markedly boosts out-of-distribution performance. I’m interested in how the logit norm behaves with versus without LN and how that correlates with your empirical observations.", "limitations": "The Paper properly addressed the limitation. But I think this paper would be more valuable if it also dealt with a more realistic scenario. One example would be the impact of LN in causal LM (with language modeling task) under non-iid text data.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- S1: The paper is easy to follow, and the argument has not been studied before.\n\n### Weaknesses\n- W1: It would be ideal to run a controlled experiment using a single model and learning objective, since the outcomes can be affected by various hyperparameters.\n- W2: I’m not clear on how gradients alone can measure a model’s learning versus its tendency to memorize labels, and there’s no clear explanation regarding this (line 131-132). Also, since each example yields multiple LN gradient values (about 2N for N layers), I’d expect both the variance and the mean of those gradients across samples to be quite large, so they can’t reliably indicate learning or memorization on their own.\n- W3: Arguing LN in pre-LN helps stabilize training only by showing the accuracy gap before and after removing LN sounds overclaims to me. It would be great to show the loss and gradient norm graph to see how spiky the training was.", "was_revised": true, "final_justification": "The authors added controlled experiments comparing five models under identical data and hyper-parameter settings, consistently reproducing the effect of LN placement, and showed that LN-input gradient norms validly separate learning from memorization. They further extended the study to attention entropy, logit norms, and a generative next-token prediction task, fully addressing W1, W2, and key questions Q1 & Q2. Loss and global gradient-norm plots are still pending, but the empirical evidence now appears solid. I therefore raise my rating.", "rebuttal": "**We thank the reviewer for their thoughtful and constructive feedback. Below, we address their questions in detail, while covering requested additional experiments in the limited time frame with our best effort.**\n\n>**Weaknesses:**\n>>**W1:** In order to address the reviewer’s request  we include additional results so that we can overall compare results for Emotions dataset through three Post-LN models, BERT, DeBERTa, and ELECTRA, and two Pre-LN models, GPT-Neo and Qwen2. \n>>|Model|Setting|Learning&nbsp;(%, ↑ is better)|Memorization&nbsp;(%, ↓ is better)|Recovery&nbsp;(%, ↑ is better)|Random Prediction&nbsp;(%, ↓ is better)|\n>>|-|-|-|-|-|-|\n>>|Post-LN (BERT)|Before|92.30|100|0|0|\n>>||After|91.95|19.16|72.5|8.33|\n>>|Post-LN (DeBERTa)|Before|92.58|100|0|0|\n>>||After| 91.98|21.66|70.41|7.91|\n>>|Post-LN (ELECTRA)|Before|93.20|100|0|0|\n>>||After|92.70|17.5|68.75|13.75|\n>>|Pre-LN (GPTNeo)|Before|90.98|100|0|0|\n>>||After|86.16|73.75|14.79|11.45|\n>>|Pre-LN (Qwen2)|Before|93.25|100|0|0|\n>>||After|89.4|97.50|0.62|1.88|\n>>&nbsp;\nThe results again confirm our claims that for Post-LN models, LN removal mitigates memorization, and for Pre-LN models, LN removal impairs learning.\n\n>>**W2:** Thank you for this important point. We use gradient norms as a proxy to examine how LN removal would impact memorization and learning in Pre- and Post-LN models. The gradient shows how the loss is affected due to the input of the LN for each sample, thereby commenting on learning and memorization through the analysis of clean and noisy samples, respectively. Our results consistently show that in Pre-LN models, learning gradients dominate, while in Post-LN models, memorization and learning gradients are of comparable magnitude. This offers an explanation of the discrepant impact of LN in Post-LN (mitigates memorization) and Pre-LN models (impairs learning) upon its removal. Please kindly note that, while individual gradients may vary, averaging across multiple samples yields stable trends that align with our empirical observations on LN removal.\n\n>>**W3:** We agree that loss analysis could further support our claims. However, we have included test accuracy and memorization dynamics over epochs (Fig 2), which already reflects training instability Pre-LN removal due to a drop in test-accuracy, which the model can not recover back over the course of training. Having said that, we will try to include loss analysis plots in the final version of the paper per your request.\n\n>**Questions:** We like the reviewer’s inquisitiveness in our paper. Since, the reviewer hasn’t provided us with the exact papers that they are referring to, we present the results for Q1 and Q2, to the best of our knowledge in the limited time frame.\n\n>>**Q1:** We compute attention entropy across layers in Pre-LN models - GPTNeo (12 layered), and Qwen2 (24 layered), before and after LN removal. We provide results as mean and standard deviation for all samples across all layers. \n>>| Model|Setting|L1|L2| L3   | L4   | L5   | L6   | L7   | L8   | L9   | L10  | L11  | L12  | L13  | L14  | L15  | L16  | L17  | L18  | L19  | L20  | L21  | L22  | L23  | L24  |\n>>|-|-|-|-|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|\n>>|GPTNeo|Before| 5.03±0.02 | 4.90±0.01 | 5.06±0.02 | 4.88±0.01 | 5.04±0.02 | 4.87±0.02 | 5.05±0.02 | 4.88±0.02 | 5.04±0.02 | 4.85±0.02 | 5.03±0.02 | 4.86±0.02 |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |\n>>|| After   | 5.00±0.03 | 4.83±0.03 | 5.01±0.03 | 4.84±0.02 | 5.00±0.03 | 4.83±0.03 | 5.01±0.03 | 4.84±0.02 | 5.00±0.03 | 4.84±0.02 | 5.00±0.03 | 4.83±0.03 |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |  -   |\n>>|Qwen2|Before|5.99±0.01 |5.99±0.01 |6.00±0.01 |5.96±0.01 |5.97±0.01 |5.95±0.01 |5.96±0.01 |5.97±0.01 |5.98±0.01 |5.96±0.01 |5.95±0.01 |5.95±0.01 |5.97±0.01 |5.96±0.01 |5.96±0.01 |5.95±0.01 |5.96±0.01 |5.95±0.01 |5.95±0.01 |5.96±0.01 |5.95±0.01 |5.95±0.01 |5.93±0.01 |5.97±0.01|\n>>|| After   | 5.92±0.01 | 5.97±0.01 | 6.00±0.01 | 5.99±0.01 | 5.99±0.01 | 5.99±0.01 | 5.99±0.01 | 5.99±0.01 | 5.99±0.01 | 6.01±0.01 | 5.99±0.01 | 6.01±0.01 | 6.00±0.01 | 6.01±0.01 | 6.00±0.01 | 6.00±0.01 | 6.01±0.01 | 6.00±0.01 | 6.01±0.01 | 6.01±0.01 | 6.01±0.01 | 6.02±0.01 | 6.01±0.01 |5.98±0.01|\n>>&nbsp;\nThe results show no significant drop in attention entropy, indicating that performance degradation is **not due to entropy collapse**. While entropy remains stable, learning is still impaired, suggesting there might be other mechanisms at play, making it an interesting direction for future work. \n\n>>**Q2:** We analyze the norm of the logit vectors for both Pre-LN and Post-LN models, for before and after LN removal.\n>>|Dataset|Model|Setting|Logit Norm(Mean±Std)|\n>>|-|-|-|-|\n>>|Emotions|Post-LN (BERT)|Before|9.34±1.22|\n>>|||After|8.80±1.15|\n>>||Pre-LN (GPTNeo)|Before|15.23±2.71|\n>>|||After|8.14±2.31|\n>>|News|Post-LN (ELECTRA)|Before|9.72±0.96|\n>>|||After|8.75±1.61|\n>>||Pre-LN (Qwen2)|Before|11.40±4.37|\n>>|||After|6.51±3.18|\n>>&nbsp;\nIn Post-LN models (BERT, ELECTRA), **logit norms remain stable** after LN removal, consistent with our observation that their learning dynamics are not substantially disrupted. In contrast,**logit norms in Pre-LN models decrease** after LN removal, indicating impaired learning and higher prediction uncertainty. This further aligns with our gradient analysis, which shows learning is more dependent on LN in Pre-LN architectures.\n\n>**Limitations:**\n>We acknowledge the reviewer’s comment and carry out a Next Token Prediction Task on the Emotions dataset for both BERT (Post-LN) and GPTNeo (Pre-LN). We modify the classification dataset as follows - **original text + “This emotion is [type]”**. We then predict the **“[type]”** of the token which can be either of the 6 emotions. To induce the notion of noisy labels in language-modeling, we change the **“[type]”** token for 1% of the samples to a random different emotion in the train set. Following this setup, we train 2 variations of both Pre- and Post-LN models, (1) without removing LN, and (2) with removing LN and provide the results below:\n>Model|Setting|Learning&nbsp;(%, ↑ is better)|Memorization&nbsp;(%, ↓ is better)|Recovery&nbsp;(%, ↑ is better)|Random Prediction&nbsp;(%, ↓ is better)|\n>|-|-|-|-|-|-|\n>|Post-LN (BERT)|Before|92.14|100|0|0|\n>||After| 91.95|28.12|60.62|11.25|\n>|Pre-LN (GPTNeo)|Before|91.7|100|0|0|\n>||After|85.55|51.25|19.38|29.38|\n>&nbsp;\nWe observe that even for this language modeling task, LN has an impact on memorization in Post-LN model (BERT) while it impairs learning for Pre-LN model (GPTNeo). As a result, **we verified that further results on generative tasks also corroborate the claims of the paper.**\n\n**We hope our response resolves your concerns and questions, and helps you consider raising your score. If there is anything that we can further clarify, please let us know. We appreciate your time and effort again.**"}, {"review_id": "x65iPW1Wii", "reviewer": "Reviewer_rmJJ", "summary": "This paper presents a comprehensive analysis of the role of **Layer Normalization (LN)** in transformer architectures, contrasting **Pre-LN** (where LN is applied before attention and feedforward layers) and **Post-LN** (where LN is applied after residual connections) designs. The authors study how LN affects both learning stability and label memorization, particularly under training with noisy labels. \n\nThe core findings are: **(1) Learning vs. Memorization:** LN is essential for learning in Pre-LN transformers; removing LN parameters destabilizes training and worsens overfitting. In contrast, for Post-LN models, LN plays a key role in memorization, and removing LN parameters suppresses memorization and restores correct labels without hurting generalization; **(2) Importance of Early LN Layers:** The influence of LN is concentrated in the early layers—removing early LNs harms learning in Pre-LN models and effectively suppresses memorization in Post-LN models; **(3) Gradient-Based Analysis:** The authors compute learning and memorization gradient norms, showing that LN in Pre-LN models contributes more to learning gradients, while in Post-LN models, learning and memorization gradients are comparable—explaining the divergent behaviors upon LN parameter removal.", "questions": "This is a well-executed analysis paper with a thorough investigation into the role of Layer Normalization in Pre-LN and Post-LN transformer architectures. The paper is already strong in its current form and I lean toward an accept recommendation. That said, I have a few suggestions and questions that I believe could improve the scope and impact of the work even further, should the authors choose to explore them:\n\n1. Scale of Vision Datasets (e.g. run few experiments on Imagenet-1k)\n2. Alternative Memorization Probes (i.e. explain differences and why noisy labels were chosen)\n3. Architectural Conversion Experiments (e.g. transform ViT-S to Post-LN)\n4. Extension to Multimodal Models (e.g. run few experiments with CLIP)\n\nThese are suggestions for potential enhancement, not critical gaps.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is exceptionally **well-written** and **highly accessible**. The flow of ideas is clear and logical, with no ambiguities or gaps in explanation. The authors strike an excellent balance between depth and clarity, relegating technical details and supplementary results to the appendix, keeping the main paper focused and uncluttered.\n\n- The experimental validation is **compelling** and **clearly supports the theoretical claims**. The experiments are consistent, well-designed, and span a variety of architectures and datasets. The figures and plots are not only visually informative but also instrumental in building trust in the findings.\n\n- Although the topic may appear narrowly focused—centered solely on the role of Layer Normalization (LN)—the thoroughness of the analysis and the consistency of the insights make this study **valuable** to the broader community. While it doesn’t propose a new method, the paper’s novelty lies in its analytical depth and empirical findings. It is clearly an analysis paper, and should be *reviewed as such*.\n\n- The metrics introduced to quantify the impact of LN on learning and memorization—*Learning Accuracy*, *Memorization Score*, *Recovery Score*, and *Random Prediction Score*—are thoughtful and enhance interpretability across experiments.\n\n**Minor Weaknesses and Suggestions**\n\n- The vision datasets used—CIFAR-10, UTK-Face, and NICO++—might be seen as relatively *small-scale* or “toy” by today’s standards. Including experiments on more challenging and large-scale datasets such as *ImageNet-1k*, *Places365*, or *iNaturalist* could further strengthen the empirical claims and broaden the paper’s applicability.\n\n- While the use of noisy labels is a standard approach to studying memorization, a brief discussion of **alternative methods** (e.g., studying forgetting dynamics, example forgetting statistics, or influence functions) would help contextualize the choice. A comparison of the advantages and limitations of noise injection would also improve clarity.\n\n- It would be interesting to include an ablation where a *Pre-LN model is manually converted to Post-LN*, and vice versa (e.g., modifying ViT-S accordingly), to directly observe how the same base architecture behaves under the two LN configurations. This would further cement the core message about architectural asymmetries in LN’s impact.\n\n- The paper currently evaluates only unimodal models (vision-only or text-only). It would be interesting to investigate whether the findings regarding LN’s role in learning and memorization transfer to *multimodal architectures* such as vision-language models (e.g., CLIP). Given that these models often use LayerNorm in both visual and textual branches, this could shed light on whether the observed effects generalize beyond the unimodal setting.", "was_revised": true, "final_justification": "I recommend accepting this paper. I will keep my initial score 5: Accept. Authors did a great job in this rebuttal.", "rebuttal": "**We thank the reviewer for recognizing our work’s value, and suggesting several points that can potentially further improve our work. We report back what we were able to collect within the time frame.** \n\n>**Minor Weaknesses/Questions**\n>>**Q1:** We provide additional analysis on the Places365-Mini (We couldn’t do Places365 because of the resource and time constraints). We tested out, ViT-B and DeiT Pre-LN models against the same.\n>>|Model|Setting|Learning&nbsp;(%, ↑ is better)|Memorization&nbsp;(%, ↓ is better)|Recovery&nbsp;(%, ↑ is better)|Random Prediction&nbsp;(%, ↓ is better)|\n>>|-|-|-|-|-|-|\n>>|ViT-B|Before|93.4|100|0|0|\n>>||After|56.2|78.33|13.33|8.34|\n>>|DeiT|Before|91.2|100|0|0|\n>>||After|51.2|95|0|5|\n>>&nbsp;\nFrom our results, we can again verify that for Pre-LN models, LN primarily influences learning, where their removal impairs learning without mitigating memorization.\n\n>>**Q2: Thank you for such productive comments.** We chose random label noise because it offers a simple and reliable way to study memorization. Since these examples are known a priori to be unlearnable, any correct prediction must stem from memorization rather than generalization. In contrast, methods like forgetting dynamics or influence functions require additional computation to first identify memorized examples, adding complexity to the analysis. While such heuristic-based methods can also be used, we utilized random label noise for its conceptual clarity and ease of implementation. Having said that, we agree that such further explorations would help extend this work. \n\n>>**Q3:** Following your suggestion, we attempted this for Pre-LN models (ViT-B and DeiT) by reordering the LN placements to emulate Post-LN behavior. However, even prior to any LN removal, these converted models failed to converge properly, as reflected in poor train-test accuracy at the end of training, as shown below.\n>>|Model|Train Accuracy (%) |Test Accuracy (%) |\n>>|-|-|-|\n>>|ViT-B|10.15|10|\n>>|DeiT|42.77|42.52|\n>>&nbsp;\nThis suggests that such conversions may require additional tuning or stabilization strategies. We consider this as a promising direction for future work to systematically convert Pre-LN models to Post-LN and examine their behavior under LN removal.\n\n>>**Q4:** We appreciate the reviewer’s suggestion. Due to resource and time constraints, we were unable to run multimodal experiments, but we instead provide new results on a language modeling task using a modified version of our classification dataset. Each input is rephrased as: **original text + “This emotion is [type].”**, where **[type]** is one of 6 emotions.\nTo simulate noisy labels, we randomly flip the **[type]** token for 1% of the training samples to a different emotion. We then train both Pre-LN (GPTNeo) and Post-LN (BERT) models under two settings: (1) with LN intact, and (2) with LN removed.\n>>|Model|Setting|Learning&nbsp;(%, ↑ is better)|Memorization&nbsp;(%, ↓ is better)|Recovery&nbsp;(%, ↑ is better)|Random Prediction&nbsp;(%, ↓ is better)|\n>>|-|-|-|-|-|-|\n>>|Post-LN (BERT)|Before|92.14|100|0|0|\n>>||After| 91.95|28.12|60.62|11.25|\n>>|Pre-LN (GPTNeo)|Before|91.7|100|0|0|\n>>||After|85.55|51.25|19.38|29.38|\n>>&nbsp;\nWe find that, **consistent with our original results**, removing LN impairs memorization in Post-LN (BERT) but hurts learning in Pre-LN (GPTNeo), re-verifying our main claims in a generative modeling setup. We hope this demonstrates that our findings generalize beyond classification tasks.\n\n**We sincerely thank you for your valuable comments and suggestions, and hope that our responses answer them. We would be happy to answer any further questions that you may have.**"}, {"review_id": "sQ4F7c5tRg", "reviewer": "Reviewer_Kc5j", "summary": "This paper studies the effect of Layer Normalization on memorization and learning in Transformer architectures, comparing Pre-LN (where LN is applied before main layers) and Post-LN (where LN is applied after) settings. The authors systematically investigate what happens when they remove LN's learnable parameters, measuring how this affects memorization (overfitting noisy labels), genuine learning, and recovery of true labels.", "questions": "Please see the weaknesses section for broader concerns. In addition, I have the following specific questions and suggestions for clarity:\n\n1. Certain models (e.g., DistilBERT) appear to deviate from the general trends observed, but these exceptions are not discussed in depth. Do the authors have any intuition or explanation for why such deviations occur?\n2. While I enjoyed reading the paper, I believe the following points would help improve its clarity:\n\n2.1. In Figure 1, what dataset is used to generate the plot? It would help to include this in the caption or main text.\n\n2.2. In Figure 2, the second column presents results without layer normalization (LN). It would be helpful to also show the same configuration with LN for direct comparison. Without it, it's difficult to assess the actual impact of LN removal, and the effect might be overestimated.\n\n2.3. The paper claims that in the Post-LN setup, removing LN suppresses memorization. However, from the plots, this claim is hard to confirm. While it may help with recovery of true labels, the link to memorization suppression is unclear. Could the authors elaborate on how they diagnose memorization from these plots?\n\n2.4. In Figure 3(a), third column, I expected the effect of LN removal to show a monotonic trend. However, the blue line is non-monotonic. Do the authors have an explanation for this behavior?\n\n2.5. On page 8, lines 265–267, the sentence beginning with \"In contrast, ...\" is unclear to me. Specifically, how is the order of magnitude of gradient norms related to the restoration of genuine labels without interfering with learned features? Some further clarification here would be appreciated.\n\n3. The authors mention the effect of LN removal from early layers, mid layers and final layers, however, there is no definition what layers are assumed to be early, middle or last. Is the number of layers in each group equal?", "limitations": "Please see weaknesses and questions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths:*\n1. The paper is well-written, with a logical flow, and provides a clear explanation of the problem's importance, novelty, and relevant background.\n2. The authors studied the effect of LayerNorm across various models and datasets, offering a relatively large-scale and robust study. Results appear relatively stable across different settings.\n3. The connection established between gradient norms and learning/memorization provides theoretical grounding for the empirical observations. This also introduces a useful metric for practitioners.\n4. The findings could help inform the design of models that are more robust to label noise or less prone to memorization.\n\n*Weaknesses:*\n1. Memorization is tested via random label flipping (1% noise). As far as I can tell, there is no investigation of how the results change when the proportion of noisy labels is increased. While the findings likely hold up to a certain point, it would be interesting to see how they behave as label noise increases.\n2. The study focuses exclusively on removing LN parameters (scaling and bias), while retaining the core mean-variance normalization. The impact of removing the normalization itself is not explored. If relevant prior work exists, it should be cited; otherwise, could the authors clarify why they did not also test the effect of removing the normalization operation?", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the reviewer’s recognition of our work, and appreciate their insightful comments. Below we address the questions to provide more clarity regarding our paper.**\n\n>**Weaknesses:**\n>>**W1:** Thank you for the suggestion. We provide additional experiments on the Emotions dataset for 2% and 5% label noise. From the results we again observe **consistent results** that LN removal mitigates memorization in Post-LN models (BERT), while it impairs learning in Pre-LN models (GPTNeo).\n>>|Noise|Model|Setting|Learning&nbsp;(%, ↑ is better)|Memorization&nbsp;(%, ↓ is better)|Recovery&nbsp;(%, ↑ is better)|Random Prediction&nbsp;(%, ↓ is better)|\n>>|-|-|-|-|-|-|-|\n>>|2%|Post-LN (BERT)|Before|91.7|100|0|0|\n>>|||After|92.0|20.62|76.25|3.12|\n>>||Pre-LN (GPTNeo)|Before|91.35|100|0|0|\n>>|||After|84.85|66.87|16.56|16.56|\n>>|5%|Post-LN (BERT)|Before|90.35|100|0|0|\n>>|||After|91.25|27.0|66.88|6.12|\n>>||Pre-LN (GPTNeo)|Before|90.35|100|0|0|\n>>|||After|82.6|67.5|11.0|21.5|\n>>&nbsp;\n\n>>**W2:** Yes, in this study we primarily focus on removing learning parameters. We do this by following prior work done by [1], who also removed the same to analyze LN’s impact.\n\n>**Questions:**\n>>**Q1:** Regarding DistilBERT, we observe that even though LN removal did not mitigate memorization, early LNs removal at least delayed memorization when removed, because early LNs have higher norms. That being said, we suspect that other components (like FFN, MHSA) in DistilBERT might be driving memorization more than LN. Hence, LN removal didn’t mitigate memorization. But since this paper is solely focused on the impact of LN, we refrain ourselves from distributing our focus on other components.\n\n>>**Q2.1:** In Fig.1 we used multiple datasets-models configuration for diversity in our analysis, as stated in Appendix F.3 - (1) Emotions dataset with BERT (Post-LN), DeBERTa (Post-LN), GPTNeo (Pre-LN), (2) News dataset with ELECTRA (Post-LN), Longformer (Post-LN), Qwen2 (Pre-LN), (3) Tweets dataset with RoBERTa (Post-LN), GPT2 (Pre-LN), (4) CIFAR10  dataset with ViT-B (Pre-LN), (5) UTK-Face dataset with DeiT (Pre-LN), and (6) NICO++ dataset with ViT-S (Pre-LN). \n\n>>**Q2.2:** We take into account the reviewer’s suggestion and will add before LN removal memorization plots over epochs for Fig.2 as a reference. This would essentially show that without LN removal, model achieves 100% memorization at the end of training.\n\n>>**Q2.3:** As explained in our paper, we introduce “noisy labels” to study memorization. If the model overfits on these noisy labels, then it has memorized them. However, in Post-LN models, after removing LN, we observe that the model does not overfit on these labels and reverts back to their genuine class labels, meaning that memorization was mitigated as shown by high recovery scores (green bars) and low memorization scores (red bars), in Fig.1d.\n\n>>**Q2.4:**  We appreciate the reviewer’s observation. Here is our explanation for this effect: In the very early training epochs (epochs 1–3), removal of **early LNs** disrupts the model’s capacity to learn and generalize, resulting in poor test accuracy. Consequently, the model behaves nearly randomly causing high random prediction scores and poor recovery. As training progresses, however, the model gradually learns and generalizes better to the test set. This improved learning temporarily enhances recovery (reduces memorization) for the noisy samples, explaining the increase in the blue curve (early LNs). But with further training, memorization sets in, and recovery again worsens and hence the dip.  On the other hand, when removing **middle/later LNs**, which our gradient analysis shows to have lower learning gradient norms, the model's learning behavior is less disrupted, even in early epochs. This causes the model to make correct predictions from the start, and thus the recovery score doesn't begin at zero, and hence leading to a more monotonic-like trend. Thanks to your comment, we will incorporate this explanation in the revised version.\n\n>>**Q2.5:** First, we would like to clarify that the gradients offer possible explanations for the observed phenomena across Pre- and Post-LN models. From Eq 5, we observe that the ratio of learning-to-memorization gradient norms are much higher for Pre-LN models than Post-LN models. This means that for Pre-LN models, the learning gradient norms are significantly high, hence LN removal disrupts learning. On the other hand, for Post-LN models, both learning and memorization gradient norms are similar. This suggests that LN removal won’t influence learning that much while impacting memorization. \n\n>>**Q3:** Yes, we define early, middle and later layers by grouping consecutive layers. We have provided the explanation for the same in Appendix F.4.\n\n[1] Xu, J., Sun, X., Zhang, Z., Zhao, G. and Lin, J., 2019. Understanding and improving layer normalization. Advances in neural information processing systems, 32.\n\n**We thank the reviewer for their comments and suggestions and hope that our responses provide more clarity regarding our paper.**"}]

Author rebuttal:

Meta-review: This paper discusses the impact of LN layers on the training dynamics of transformers in a supervised classification setup. It systematically analyzes the role of Layer Normalization showing that removing LN parameters: 
1) destabilizes learning and increases memorization in Pre-LN models and 
2) suppresses memorization and aids recovery of true labels in Post-LN models.
The reviewers have praised this work for its simplicity and clarity. The rebuttal successfully addressed most of the reviewers comments.

My only concern with this work is the discrepancy between the possible impact breadth and evaluation thoroughness. This work has broad impact, as transformers are ubiquitous in modern machine learning (vision, language, speech, audio). The paper does a decent job at providing empirical evidence on several domains (vision and language). However, as noted by reviewers Fd4h and rmJJ the evaluations should include proper language modelling and larger-scale datasets. The reviewers provided experiments on a "generative" variant of Emotions dataset - but I think the evidence would be much stronger on proper language modeling, measure using perplexity. Language modeling with transformers has an abundant literature with decent mid-scale benchmarks from the pre-LLM era. 

Overall, despite the point raised above, this is a good paper. I recommend accepting the paper as a poster.
Decision: Accept (poster)