SeeDNorm: Self-Rescaled Dynamic Normalization
Authors: Wenrui Cai
Defa Zhu
Siyuan Qiao
Qingjie Liu
Qiyang Min
Venue: ICLR 2026 Poster (2026)
Keywords: Normalization Layer
Primary area: foundation or frontier models, including LLMs

Abstract: Normalization layer constitutes an essential component in neural networks. In transformers, the predominantly used RMSNorm constrains vectors to a unit hypersphere, followed by dimension-wise rescaling through a learnable scaling coefficient $\gamma$ to maintain the representational capacity of the model. However, RMSNorm discards the input norm information in forward pass and a static scaling factor $\gamma$ may be insufficient to accommodate the wide variability of input data and distributional shifts, thereby limiting further performance improvements, particularly in zero-shot scenarios that large language models routinely encounter. To address this limitation, we propose SeeDNorm, which enhances the representational capability of the model by dynamically adjusting the scaling coefficient based on the current input, thereby preserving the input norm information and enabling data-dependent, self-rescaled dynamic normalization. During backpropagation, SeeDNorm retains the ability of RMSNorm to dynamically adjust gradient according to the input norm. We provide a detailed analysis of the training optimization for SeedNorm and proposed corresponding solutions to address potential instability issues that may arise when applying SeeDNorm. We validate the effectiveness of SeeDNorm across models of varying sizes in large language model pre-training as well as supervised and unsupervised computer vision tasks. By introducing a minimal number of parameters and with negligible impact on model efficiency, SeeDNorm achieves consistently superior performance compared to previously commonly used normalization layers such as RMSNorm and LayerNorm, as well as element-wise activation alternatives to normalization layers like DyT.

Published as a conference paper at ICLR 2026
SEEDNORM: SELF-RESCALED DYNAMIC NORMALIZATION
Wenrui Cai Defa Zhu Siyuan Qiao*, Qingjie Liu Li Qiyang Min "Bytedance Seed China, School of Computer Science and Engineering, Beihang University [wenzui.cai, singjie. 1iu @buaa edu. en (zhudefa, siyuan, qiao, minqiyang (aregllaynnnnyyyaaaaaaaa (ebytedancee........ com
ABSTRACT
Normalization layer constitutes an essential component in neural networks. In transformers, the predominantly used RMSNorm constrains vectors to a unit hyper- sphere, followed by dimension rescaling through learnable scaling coeffi- cient y to maintain the representational capacity of the model. However, RMSNorm discards the input norm information in forward pass and a static scaling factor ? may be insufficient to accommodate the wide variability of input data and distribu- tional shifts, thereby limiting further performance improvements, particularly zero-shot scenarios that large language models routinely encounter. To address this limitation, we propose SeeDNorm, which enhances the representational capability of the model by dynamically adjusting the scaling coefficient based on the current input, thereby preserving the imput norm information and enabling data-dependent, self-rescaled dynamic normalization. During backpropagation, SeeDNorm retains the ability of RMSNorm to dynamically adjust gradient according to the input norm. We provide detailed analysis of the training optimization for SeedNorm and proposed corresponding solutions to address potential instability issues that may arise when applying SeeDNorm. We validate the effectiveness of SeeDNorm across models of varying sizes in large language model pre-training as well as supervised and unsupervised computer vision tasks. By introducing a minimal number of parameters and with negligible impact model efficiency, SeeDNorm achieves consistently superior performance compared to previously commonly used normalization layers such as RMSNorm and LayerNorm, well element-wise activation alternatives to normalization layers like DyT.
INTRODUCTION
Normalization layers have become fundamental building block in modern deep neural networks, playing a key role in stabilizing training and accelerating convergence (Toffe & Szegedy 2015). By enforcing statistical regularity activations, normalization techniques help prevent issues such as exploding or vanishing gradients, thereby enabling deeper and more expressive models. Over the past decade, normalization has proven indispensable, especially in large-scale architectures for both language modeling (Vaswani et al. 2017) and computer vision (He et al. 2016) fields.
However, this stability comes a cost: conventional normalization methods (such as LayerNorm (Bal ct al. 2016) and RMSNorm (Zhang & Sennrich, (2199) tend to discard or diminish information about the input norm, which can restrict the expressive capacity of the network and hinder the preservation of crucial scale-related features. Although modern normalization layers introduce learnable parameters to restore some network expressivity, these parameters are static and input-independent, which presents challenges in scenarios such as zero-shot generalization.
Alternatively, saturation activation functions such as tanh or its dynamic variants (Zhu et al. 2025b) offer the potential to retain norm information by constraining outputs within fixed range. While these functions can preserve the relative scale or norm of the input, they inevitably suffer from the vanishing gradient problem in extreme cases, and these functions are unable to dynamically adjust
equal contribution; project leader; corresponding author.

---

Published as a conference paper at ICLR 2026
Downstream ACC.
Training
Training
7.44
3.63
3.54
k.
2.54
Thardin
2.33
Phaz
3337
0/60
Tokore (Bllond)
Tokens (Billins)
Steps DK]
Steps (X]
Figure Comparisons between SeeDNorm and prior methods across diverse tasks language modeling and computer vision. The first two figures depict training loss curve comparisons and average downstream task aecuracy between the OLMoF-11.3B (Muennighoff et al., 2024) baseline (using RMSNorm (Zhang & Sennrich 2019)) and the SeeDNorr-equiipeed model, training on 500B tokens. The latter two figures show training loss comparisons between Dy b-T-dd (Zhu et al. 20256) baseline models and SeeDNorm- models
image generation and MAE (He et
2022 pre-training. All loss curves are smoothed with a 0.99 EMA.
gradients based on input norm during backpropagation like RMSNorm (Zhang & Sennrich 2019). which leads to inefficient optimization and slow convergence.
This dilemma raises fundamental question: Is it possible to design method that combines the training stability, the optimization efficiency, and the ability to preserve input norm information?
In this work, answer this question affirmatively by introducing Self-Rescaled Dynamic Normal- ization (SeeDNorm). SeeDNorm achieves stable and efficient training while explicitly retaining norm information throughout the network. Extensive experiments demonstrate that SeeDNorm consistently accelerates convergence and improves performance across both language modeling and vision tasks, offering a simple yet effective alternative existing normalization and activation approaches. As illustrated in Figure I integrating SeeDNorm into language and vision models leads faster convergence and consistently improved performance across variety of downstream tasks. In summary, our main contributions are as follows:
We propose SeeDNorm, a dynamic normalization layer that generalizes RMSNorm and adaptively adjusts its scaling coefficient based on the current input, preserving the input norm information and offering improved adaptability to data variability and distributional shifts.
We conduct detailed and comprehensive analysis of the forward pass and gradients in the backpropagation of SeeDNorm, demonstrating the advantages of our method over existing normalization and dynamic activation alternatives, while also proposing techniques to enhance training stability.
Extensive experiments on large language models with both dense and MoE (Du et al. 2022) structure, as well as computer vision tasks, show that SeeDNorm consistently accelerates convergence and improves performance over R.M.Norm, LayerNorm and DyTT baselines, with minimal additional parameters and computational cost.
We believe SeeDNorm offers a simple yet effective path towards more robust and flexible normaliza- tion in large-scale neural networks, especially for scenarios requiring strong generalization across diverse or shifting data distributions.
2 RELATED WORK
Normalization layers. Normalization layers are widely employed in modern deep neural networks, and the formulation of most normalization layers be described
Norm(æ) = Vo² +
2 11
19.
(1)
is extremely small value to prevent division by zero. Given BxNND normalization layers enforce to transform the distribution over specific dimensions of the input to standard normal
All downstream tasks evaluated in zero-shot manner, all tasks shown in Table
2

---

Published as a conference paper at ICLR 2026
distribution, ensuring that data distributions across network layers remain stable during training and accelerating model convergence. Subsequently, learnable parameters y and [3 are used to adjust the distributions of each layer, enabling diversity in the distributions across network layers and alleviating the degradation of the expressive capacity.
Batch normalization (BN) (Toffe & Szegedy 2015) operates by normalizing all elements within each channel across the batch dimension, where the channel-specific mean μl and variance a are calculated as jic J Eo.n Fo,n,e and o² BN Ex. (Zo,, Me)² respectively. BN has gained prominence in computer vision tasks, partieularly in convolutional neural networks (He et al. 2016), due to its alignment with convolutional operations: identical kernels process features within the same channel across all spatial positions and batch samples. As discussed in (Santurkar et al. 2018), BN effectively smooths the loss landscape and enables more stable training of the model. However, BN not suitable for sequence modeling tasks and may leak context information across samples within a batch. As result, BN is seldom adopted in large language models or generative models.
Layer Normalization (LN) (Ba et al., 2016) addresses the limitations of BN in sequence modeling. Instead, LN normalizes the imput across the feature dimension for each individual sample, where μ and 2² in Eq equation denote the mean and variance computed along the last dimension of ar LN henesoorerrrrchiteete is widely adopted in language modeling and Transformer architectures due to its independence from batch size.
Root Mean Square Layer Normalization (RMSNorm) (Zhang & Sennrich, 2019) further simplifies LN by omitting the mean subtraction and normalizing only by the root mean square of the input:
RMSNorm(1) = RMS(x) where RMS(#)
D
Σ
of +
(2)
is element-wise multiplication along channel dimension. RMSNorm has been shown provide stable training and competitive performance, especially in large-scale Transformer models (Touvron) al., 2023). Despite the effectiveness, a fundamental limitation shared by these normalization methods is that stability is obtained by sacrificing information related to the scale of the input, which potentially restrict the network's expressive capacity.
Saturation activation functions. Recent works have also attempted to use saturation activation functions to replace normalization layers. For instance, /huu et al. (2025b) proposes using dynamic tanh (Dy')) to substitute normalization, which can be formulated as:
DyT(2:) = 7 @ tanh(ax:) + ,
(3)
The input-dependent statistics replaced with activation function and learnable scalar parameter The scaling coefficient y and shift coefficient / is also preserved. DyT explicitly preserves the norm of the input vector æ in the forward pass and constrains extreme values via tanh, thereby mapping input vector within a hypersphere with radius of vD to enhance training stability. However, DyT exhibits vanishing- graddient problem in its extreme regions. Since DyDT(( osech? (azc)"(az)) diag(y), when 7 is too small, a is either too small or too large, or & is excessively large, the gradient tends to approach 0. Since sech? 2 (x) ) a higher-order infinitesimal of 1 it still leads to the problem of gradient vanishing when backpropagating to the preceding layers. Furthermore, in Proposition [B.]] we demonstrate that under the assumption of constant input norm, RMSNorm is equivalent to Dy terms of gradient w.r.t ar, which also indicates that DyT lacks the ability of RMSNorm to adaptively adjust gradients based on input norm during backpropagation.
z.
Motivated by the limitations of existing normalization layers and saturating activation functions, present Norm. SeeDNorm incorporates input norm information during the forward pass and mitigates the gradient vanishing issue in backpropagation observed in DyT, while can also adjust gradients based on input like RMSNorm.
3 SELF-RESCALED DYNAMIC NORMALIZATION (SEEDNORM)
The core design of SeeDNorm lies in dynamically adjusting the scaling factor of normalization layer based on the input, while incorporating input norm information. Building upon RMSNorm,
3

---

Published as a conference paper at ICLR 2026
implement dynamic adjustment of the scaling parameter 7. Given the input R NxD, SeeDNorm can be formulated as follows:
111 D Σ 2 + (4)
SceDNorm(z) (o -
where RMS(æ)
RMS(æ))
where 7 E RIID denotes the learnable scaling factor in RMSNorm, 3 RIX represents the self- rescaling parameter. SeeDNorm performs matrix multiplication between the input æ and 3 then activates the result using the nonlinear function obtain intermediate output o (3") E RNx11 To further enhance the dynamic adjustment capability of SeeDNorm, the intermediate output is subsequently multiplied with another learnable parameter a RIxD producing an element-wise rescaling matrix (o(x ) a| E IR- This rescaling matrix is conditioned on æ itself and modulates the static scaling factor 7, thereby incorporating input norm information and endowing SeeDNorm with the ability to dynamically adjust the rescale factor for diverse inputs. In our implementation. the function is instantiated using the tanh activation, which inherently constrains the output range to [-1, 1], ensuring that large outliers in the input æ: do not exert a significant influence on the scaling coefficient.
SeeDNorm can be used to replace all normalization layers in current transformer-based models, including QueryNorm and KeyNorm (Henry et al. 2020) that is commonly employed in the attention modules in LLMs. Algorithm| | presents the Py Torch--st pseudocode for the SeeDNorm imple- mentation. The initialization method of y is consistent with that of RMSNNrm, using I-initialization, while ,3 is initialized with zero. The initialization of a can be adjusted via hyperparameters. In the following sections, we will also discuss parameter initialization methods combined with the analysis of SeeDNorm.
During the training process, for the parameter y, we maintain the same regularization scheme as the baseline model; otherwise, by default, we do not apply weight decay or other regularization techniques. For a and B, however, we apply regularization, which is more beneficial for model training, and alleviating overfitting.
3.1 ANALYSIS OF SEEDNORM
In this section, we present a detailed analysis of the forward and backward propagation of SeeDNorm. For forward propagation, our primary focus is on its scale invariance, specifically whether it can maintain numerical stability when the input scale or norm undergoes significant changes. As for translation invariance. it has already been demonstrated in RMSNorm that this does not impact model performance, so we will not pursue further analysis on this aspect. For backward propagation, we concentrate the gradients of SeeDNorm with respect to each parameter a, 3, 7 and the input æ.
Invariance Analysis. While SeeDNorm does not exhibit the same strict scale invariance as RMSNorm, demonstrates insensitivity to input scaling.
Since c is an extremely small value, for ease of notation, we will omit it by default in the subsequent discussion. For given input R xDD, when is scaled by multiplying factor of k. SerD can be expressed as:
[a(ku 3r) 7
ka
[kkk 3²) a +7] o
(5)
E. (kra)²
D (Ra)²
Therefore, when æ is scaled by factor of k, the only component in SeeDNorm that changes is the self-
rescaling matrix, which transforms from f(x) = (o 3T)) t7] to f f(kx) = [o B a a 7 The derivative of ff with respect to v is: V.. = sech 2 ( BT) 3a)) T.TT) = (1- 1--tanh? BT) () 3).
For very large values of æ, soch²(x) approaches O and Vaf approaches 0, indicating that f f(x undergoes minimal change. Conversely, as approaches O, V Vaf reaches its maximum value Therefore, to preserve insensitivity of SeeDNorm to input scaling. we initialize 3 to 0 to make v is initialized with 0, and we also add weight decay a and B. Additionally, when is close to 0.
B.
4

---

Published as a conference paper at ICLR 2026
f()) is primarily dominated by 7. Consequently, SeeDNorm is not significantly affected by the scale of the input magnitude.
Gradient Analysis. Since SeeDNorm operates each token individually, we assume by default that a E RIXD, For notational convenience, let = o(x ) Oc. The gradients of the output of SeeDNorm with respect to a RRID, B RR 1x R and æ are following:
3 SeeDNo)m
diegí RMS
7 i SeoDNo)
RMS(w) [o(a 3 Lox
i SecDNorm(2
((a RMS(#)
(x (xx"8")
8
SccDNorm(z)
(s - 1IxD @ (x"". mz)
a RMS(#)
8 RMS(())
diag(s 7)
0 æ
D (M)
Detailed derivations of the gradients are provided in Appendix
Gradient of 7. As demonstrated in Equation 5 the term RMS( is ocale-invariant Therefore, the gradient of y is not affected by samples of æ that are abnormally large abnormally small. This property contributes to the stability of the training of 7.
Gradient of a. The gradient of a also includes the scale-invariant term RMS(#)) but it is multiplied by ox )²)) When (-)) () is implemented using tanh, for abnormally large values of æ. the value can be constrained within 1, thereby preventing gradient explosion. an abnormally small input occurs, the gradient of a will similarly become small, but y can still update normally. Additionally, since a directly multiplies in the gradient of J3, and ,3 also directly influences the gradient of a, and B cannot both be initialized to O simultaneously.
Gradient of [.. a'xx -3r) cash When æ is abnormally large, cosh(a)) higher-order infinitesimal of æ, so the gradient of B approaches O. Similarly, when æ is abnormally small, the gradient of /3 also approaches O, thus avoiding the risk of gradient explosion. Since /3 is often encapsulated within or o' in the gradient, which constrains its range, while a is directly multiplied, we initialize zero, ensuring that the gradient of a starts from 0 in the early stages of training, thereby enhancing training stability. At the same time, given that almost all gradients involve a and G we need to control the scale of a and ,3 to prevent them from continuously increasing and causing excessively large gradients. Therefore, we apply weight decay both parameters during the training process.
Gradient of æ. Unlike the gradients of other parameters, is the activation output of the preceding layer, and its gradient is propagated to the previous layer for parameter updates. Therefore, when a excessively large, SceDNorm should output proportional small gradient w.r.t. æ, and vice versa. Assuming æ is scaled to an abuormally large kæ, the first term approaches 0, and in the second term, remains unchanged. Therefore, the gradient of kas is
approaches I, while (km) (kmr) (km)
n²
(kz)
RMS (a))
Therefore, the gradient of SeeDNorm with respect to kæ decreases by the same factor k. Thus, during backpropagation, SeeDNorm can dynamically adjust gradients based on the input norm like RMSNorm. Similarly, when kæ is abnormally small, the second term is significantly larger than the first term and also dominated by RMS(km)) Therefore, SecDNorm exhibits favorable adaptive gradient adjustment properties during backpropagation. More analysis regarding gradients can refer to Appendix C
primarily dominated by RMS(ke)
(RMSS(#)
3.2 MULTI-HEAD SEEDNORM
The (x (3T) and a' 3T)) term affects the gradients of a, B, and æ. In our previous analysis, we focused only extreme values, but in the actual training optimization process, such situations rare. To further ensure training stability under non-extreme conditions, our strategy is to reduce the variance of this term, specifically, to decrease the variance of x (T
Theorem 3.2. In dimesiiman space, the variance of the dot product of two random vectors is proportional to their dimension D.
5

---

Published as a conference paper at ICLR 2026
The proof of Theorem 3.2 can be found in the Appendix C.6
Therefore, we propose multi head form of SeeDNorm, which splits a and /3 into multiple sub-vectors and computes the dot product between these sub-vectors. This operation reduces the dimensionality of each dot product, and the results are then concatenated back to the original dimension, thereby achieving the goal of reducing variance. The process be formally described as:
= [ Bhgg ... Fhal. 1 33 11h1 Bhgy Bianl
MHSeeDNorm = [Dn) Bh , Thn 8h.. + 2 RMS(x)
Under the multi-head form, the gradients of SeeDNorm with respect each parameter and the input are also analyzed in detail in Appendix C And Algorithm 2 the pseudocode. The primary change is that (-) and o'(-) are also transformed into multi-head form, thereby achieving the goal of reducing gradient variance.
EXPERIMENTS
To validate the effectiveness and generality of SeeDNorm, we conduct comprehensive experiments across both language and vision tasks. During the experimental process, we systematically replace all normalization layers or saturation activation functions in the baseline models with our proposed SeeDNorm.
4.1 LARGE LANGUAGE MODELS
Our experiments language modeling primarily focus on the pretraining of large language models. We conduct experiments under both dense and mixture-of-experts (MoE) (Shazeer et al. 2017 Fedus et al. 2022) model architectures. We selected OLMoE (Muennighoff et al. 2024) the baseline for MoE architectures and OLMo2 (OLMo et al. 2024) as the baseline for dense model. To ensure experimental consistency, We utilize the identical training corpus OLMoE-mmxx0022 (Muennighofi ct al.) 2024) as specified the original OLMoE implementation, and employ the OLMo-mix- 7124 (LMo al. 2024) dataset identical with OLMo2.
Both OLMoE and OLMo2 adopt RMSNorm as normalization layer. In addition to attention layers and FFNs, RMSNorm are also applied output normalization, QueryNorm, and KeyNorm. In our experiments, we replace all normalization layers in the models with SeeDNorm, and perform QueryNorm and KeyNorm in each attention head. the experiments of language modeling, the parameter a of SeeDNorm is initialized to 1.
Training Lost WiEt Tobate
Training Loce W.Et Tohans
232
dulla Jick
20
230
Takers (Biliona)
150
(NE
23
Figure 2: The first two subplots respectively show comparison of training loss and average downstream task accuracy between OLMoE-7B baseline and the counterparts with SecDNorm. The last two subplots show a comparison training loss OLMo2-550M and OLMo2-1B IB baseline models and their counterparts with SeeDNorm incorporated. All curves are smoothed using EMA with a coefficient of 0.99.
Downstream Tasks and Evaluation Metrics. Beyond evaluating each model based on the con- vergence behavior the cross-entropy loss during training in Figure 2 we also report the average accuracy curves for the downstream tasks listed in Table 5 Table presents several representa- tive dataset metrics, we report the average perplexities (PPL) and losses on the s4-envalldattin dataset, and simultancously evaluate accuracy metrics ARC- ARC-Challenge (Clark al. [2018). ARC- Easy (Clark et al. 3018), HellaSwag (Zellers et al. 2019), MMLU- Var (Hendrycks et al.), and PIQA (Bisk al. 2020).

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of dense model and MoE models of different sizes the c4 en-validation
dataset and multiple downstream datasets, where all metrics downstream are reported as %.
Training Tokens (B)
c4-on-validation
Downstream Evaluation
Models
Loss 4.
PPI.
ARC-T
ARC-ET HellaSwagt
MMLL--Var
PIQAT
MoE Models
OLMo
500
2.922
18.63
32.3
62.2
55.2
32.4
72.6
OLMoB-1.33-DD
2.968
19.45
30.4
61.9
53.2
30.5
70.6
OLMeE-1.3B- 1.MGE-1.B-SoeDNorm
500
2.900
18.12
34.5
65.4
56.8
33.2
73.1
OLMeE-7B
1000
2.644
14.07
40.8
73.7
71.2
38.8
76.6
1000
2.631
13.88
44.5
76.1
71.8
40.2
79.1
Dense Model
OLMc2-550M
400
3.011
20.30
30.0
62.7
52.3
31.5
71.5
OLMe2-550M-5eDNom
3.008
20.24
31.4
63.4
52.0
71.5
O1.Me2-I8
500
2.884
17.88
35.6
68.7
60.4
33.9
74.5
OLMe2-IB-SeeDNorm
500
2.879
17.79
37.8
70.0
61.0
34.8
74.5
MoE Models. We conduct experiments on OLMoE-1.3B and OLMoE-7B, with activated parameter counts of 260M and IB, respectively. All experiments based on OLMoE-1.33 and OLMoE-78 are trained on 500B tokens and IT tokens using the corresponding corpus datasets. Detailed configura- tions of additional experiments are provided in the Appendix D As illustrated in Figure | and Figure 2] applying SeeDNorm to both the OLMoE-1.33 and OLMoE-7B models significantly accelerates the convergence during training. Furthermore, as the number of training tokens increases, models using SeeDNorm exhibit increasingly larger improvements in training loss compared to their baseline counterparts, Moreover, Table |
and Figure T shows that both the 1.3B and 7B models achieve comprehensive improvements in various validation metrics and accuracy in downstream tasks. In contrast, replacing the normalization layers of the OLMoE- 1.3B model with saturation activation function like DyT (Zhu al. 2025b) leads to slow convergence and degradation in performance.
#DDIGrGel.Si Stase
FDD ICFCel
Training Lass Stops
Training Loss Steps
8
Beisss
C1400
C344
48
60 30
Stace 30
Steps
Figure 3: The first two subplots show FID comparison at different training steps wich CFG=1.5, and the last two subplots show loss curves of DiT-XL/2 in image generation and VIT-L in MAE, All models respectively augmented with our proposed SecDNorm DyT.
Dense Models. Although MoE-based model architectures currently dominate LLMs, we also evaluate the performance SeeDNorm in dense model. Our experiments are primarily conducted based on OLMo2-550M and OLMo2-1BB, training on 400B and 500B tokens, respectively. Detailed configurations of additional experiments are provided in the Appendix D As shown in Figure 2 and Table [ SceDNorm continues to yield benefits in dense models, with the training loss improvement over baselines still widening as the number of tokens increases. But the advantage SeeDNorm in terms of training loss is reduced compared to MoE models. This may be because dense models do not require dynamic activation parameters, leading to more stable training, and each parameter is sufficiently trained, which diminishes the accelerated convergence advantage brought by SeeDNorm. However, in zero-shot evaluation tasks, such as ARC-C and ARC-E, the application of SeeDNorm can still significantly enhance performance. And the dynamic architectures MoE models are better able to amplify the advantages of SecDNorm.
4.2 COMPUTER VISION TASKS
We conducted experiments supervised, unsupervised, and generative visual tasks, using the multi- head version of SeeDNorm across all tasks except image generation. More details are provided in Appendix D
7

---

Published as a conference paper at ICLR 2026
Image Generation. We evaluate the effectiveness of SeedNorm using Diffusion Transformer (DiT) (Peebles & Xie 2023) as the baseline. Experiments are conducted on two model sizes: DiT-B/4 and DiT-XL/2, where 4 and 2 denote the patch sizes. It is noteworthy that SeeDNorm cannot directly replace AdaLN, the normalization layer within DiT. This limitation stems from the mechanism of A that incorporates class-specific information by predicting scaling parameter 7(c) and shifting parameter (c)) conditioned on class label c. Therefore, we retain the shift and scale terms of AdaLN, removed the y inside SeeDNorm, and adopted the following form that includes label conditions:
AdaSeeDNorm(z..d (0 (7) (.l))aa11 a+1)
x RMS (a)
+1+ (((())) where is the condition
We train DiT on the ImageNet-IK IK (Krizhevsky et al. 2012) dataset and evaluate on the ImageNet validation set, which comprises total of 50,000 images. Since DyT (Zhu et al. (025b) has already achieved better results than DiT baseline, we directly compare SeeDNNrr with DyT in Figure and Figure 3 We present comparisons of the loss curves and FID (Heusel et al. (017) across different training steps. During evaluation, the cfg-scale is set to 1.5, consistent with the optimal value used for DiT (Peebles & Xie 2023). Additional contiguration details are provided the Appendix D.4
Supervised Learning. We conduct image clas- sification experiments on two representative ar- chitectures, ViT (Dosovitskiy et al. 2021) and ConvNeXt (Liu et al., 2022). All models are trained on the ImageNet- K (Krizhevsky al., 2012) training set and evaluated on the test set. Additional configuration details are provided in the Appendix D.2 The results in Table demon- strate that applying SeeDNorm achieves better performance compared to both DyT and Layer-
Table Acc@1 ImageNet- I classification; (MAE)
denotes fine- ttn-ituning MAE pretrained models.
Model VIT-B
LayerNorm
DyT
SecDNorm
82.3
82.5
82.7
VIT-L
83.1
83.6
83.6
ConvNeXT-B ConVNeXT-L
83.7
83.7
83.7
84.3
84.4
84.6
ViT-B (MAF)
83.2
83.2
83.5
ViT-I (MAF)
85.5
85.4
85.5
Norm.
Self-Supervised Learning. We select the representative self-supervised mask reconstruction task MAE (He et al. 2022) for experiments, which are conducted on ViT (Dosovitskiy et al., [202)) All models are initially pre-trained on the ImageNet-IK (Krizhevsky al. (012) dataset, and then fine-tuned on ImageNet-IK. Additional configuration details are provided in the Appendix [D3 Figures [ and Figure 3 demonstrate that SeeDNorm significantly accelerates convergence during the pre-trate tre-training stage, while Table 2 shows that it also holds an advantage during fine-tuning.
4.3 ABLATION STUDY
our ablation studies presented in this section, we primarily focus on language modeling, conducting experiments with OLMoE-1.3B and OLMoE-7B the bascline models. In the computer vision domain, we use ViT as the baseline model.
Implementation of Activation Func- tion a in SeeDNorm. In SeeDNorm, the activation function o is set to tanh by default. As demonstrated in Figure conduct experiments with different implementation choices for σ. When employing activation functions with an unbounded output range, such as GeLU (Hendrycks
Training Loss wint Tokens
Training Loss w.nt Tokens
2,85
2.75
CMPAOL pew.seeoooeeeeeeeeooooo peoid
Taannngg LOSE
Training
2,70
2.70
2.65
2.50
2.55
2.55
Gimpell 2016) and Swish (Ramachan, dran et al. 2017), the model fails to converge properly. In contrast, us- ing bounded functions like sigmoid, tanh, and hardtanh allows the model to converge successfully. All bounded functions achieve performance superior the baseline, with tanh yielding the best results.
Tokers (Billionss
100
Tokers (Bilioons)
Figure 4: Two subplots present training loss curves of SeeDNorm with different activation functions, and with various a initialization strategies, respectively. The loss curves are smoothed using EMA with a coefficient of 0.99.
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation studies of SeeDNorm based OLMoE- 1.3B, all experiments training for 500B tokens. We evaluate various models based on validation loss and PPL on the o4 Len-validation dataset, Acc. % on different downstream tasks. "t" denotes initialization, "æ @ 3 indicates that æ and / perform element-wise multiplication in SecDNorm, "w/o" indicates removing the corresponding parameter, and "FCxn" indicates model with Dynamic Inc-Connections at frac-rates.
ca_en-validation
Downstream Evaluation
Models
Loss
PPL.
ARC-C1 f
ARC-EF
HellaSwag"
MMLU-Var Var?
PIQA
OLMoBB1133B
2.922
18.63
32.3
62.2
55.2
32.4
72.6
Dor (a - 1)
2.900
18.12
34.5
56.8
33.2
73.1
OLMoB-1.330 S.eeeee (ax 0.1)
2.912
18.39
63.7
55.6
32.3
72.8
M.Mo0--.3B-SeeDNNNA (ar 10)
3.154
23.42
27.8
53.0
43,0
28.6
68.2
OLMcE-1.33 1.B--ScDooo (scalar a)
2.909
18.33
32.6
62.2
55.9
32.4
72.6
B.MooE-..3--seDDNMe (æ @ (3)
2.909
18.33
36.5
64.9
55.7
32.0
72.7
1.3bB--1.B-SceeNNoo (w/o o))
2.907
18.29
32.1
67.0
56.5
32.9
73.2
L. (w/o (
2.911
18.37
63.7
$5.4
31.7
72.9
1.MoBB-133--eeDDNNo (w/o
2.913
18.41
33.7
65.4
56.0
32.5
72.8
OLMoB-1.3B 111--.33--NNNOAAAI
2.902
18.20
34.1
64.2
56.2
32.6
74.2
LLMMB--.3B-MuulihcudSceee
2.904
18.25
31.5
63.9
55.7
32.9
71.9
OLMoE-1.3B-FC
2,908
18.32
32.1
63.5
55.9
31.6
72.4
2.899
18.11
34.8
64.7
56.9
33.4
73.9
OLMoB-7B
2.644
14.07
40.8
71.2
38.8
Noom
2.631
13.88
44.5
76.1
71.8
40.2
79.1
OL MMxF-
2.630
13.88
44.3
75.6
71.9
39.6
78.2
2.629
13.86
44.9
76.6
72.4
39.9
79.1
Initialization of a. As depicted in Figure experiments are conducted with initialized to 0.1, and 10, respectively. Figure iigureFiidicatoo indicates that excessively large values of a adversely degrade training stability and lead poorer final performance. In contrast, smaller initial maintain training stability, and suitably larger initial a can accelerate model convergence.
Whether o vector or scalar. In SecDNorm, is designed vector, which is used to generate an element-wise self-rescaling matrix of the same shape as the input æ. Table 3 presents our evaluation of the performance impact when replacing with single scalar parameter. With a as a scalar, the self-rescaling matrix of SeeDNorm can only adjust each input vector uniformly, rather than providing element- specific scaling. As shown in Table substituting the D-dimensional vector with scalar leads to degradation in model performance, but it still surpasses the baseline.
The type multiplication between and æ. SeeDNorm uses the dot product between ,3 and by default. We further experimented with element-wise multiplication between 33 and æ. Although this does not affect the shape of the self-rescaling matrix, the results Table 3 indicate that using the dot product method offers better expressive power and performance.
Impact of Different Parameters SeeD- Norm. SeeDNorm incorporates three learn- able parameters: a. B, and 7, all of which D-dimensional vectors. Table 3 presents an ablation study evaluating the impact on model performance when each of these parameters is removed. When removed, similar the case where is scalar, the self-rescaling matrix is no longer element- wise; when /3 is removed, SeeDNorm loses the ability to adjust the shape of the nonlinear function a, and the computation with becomes a matrix mul- tiplication; removing makes SeeDNorm equivalent to directly replacing scaling factor RMSNorm with the self self-rescaling matrix.
Table 4: Ablation Study on whether to apply weight decay to a and /9 and the number of heads in SeeDNorm in ImageNet- IK classification.
Model
Acc@
Aco@5
ViT-B BeeBB-ee (16Head)
82.7
96.1
WT---SeeDNNe (1Head)
Fail to converge
BT--B-SeeDNom (8Head)
82.5
96.0
ViT-B-SecDNorm (32Head)
82.5
96.0
VTT-B-SceDoome (w/o Weight Decay)
82.4
95.9
ViT-I--Se:DNom (8Head)
83.4
96.3
ViT-L-SeeDNorm (32Head)
83.6
96.5
Impact of Multihead SerDNorm. In vision tasks, we employ multi-head SeeDNorm to reduce gradient variance and enhance training stability. In Table we experiment with varying the number of heads for the image classification task. When the number of head is 1. the model fails to converge. And the results indicate that increasing the number of heads contributes improve performance, but

---

Published as a conference paper at ICLR 2026
an excessively high number lead to reduced gradient diversity, thereby degrading performance. On larger models with greater hidden dimensions, SeeDNorm can similarly utilize a higher number of heads.
Whether to apply weight decay to a and G. Our previous analysis suggests that regularizing a and G enhances gradient stability. Thus, we test omitting weight decay for and /3 in supervised image classification task. Results in Table44iidicaal that removing weight decay of a and B will lead to inferior performance. And the result validates our theoretical analysis.
When applying SeeDNorm in QKNorm in the task of language modeling, perform normalization on each attention head, computing it in the same dimension as multi-head attention. In Table 3 we experiment with retaining the original structure of OLMoE, performing normalization across the entire hidden dimension. The results indicate that normalization on each attention head yields slightly better performance, though the difference is not significant.
Applying SeeDNorm in Each Attention Head.
Multihead SeeDNorm in OLMoE. In OLMoE, we do not use multi-head SeeDNorm. On one hand, training on large corpus less prone to overfitting and does not exhibit the high gradient variance seen in vision tasks with multiple training epochs. On the other hand, MoE models require appropriate gradient variance to dynamically train more experts. Table 3 we also conducted experiments using 16-head SeeDNorm configuration. The application of multihcad SeeDNorm does not improve the performance.
Combine with Advanced Structure. In Table 3 we also presents performance evaluations of SeeDNorm on more advanced model architectures. For these experiments, we selected OLMoE variants improved with Cran-Connection (Zhu et al., (025a). We conduct experiments with both 1.3B and 7B parameters, where the frac-rate is set to for the 1.3B model and 4 for the 7B model, respectively. SeeDNorm further enhance performance, with the effect being more pronounced in downstream tasks.
models
5 CONCLUSION
In this paper, we propose a novel normalization method SeeDNorm. SecDNorm dynamically adjusts the scaling factor based the input as condition, thereby incorporating input norm information during the forward pass, which is overlooked in previous normalization layers like RMSNorm, while also enhancing the model's adaptability to diverse inputs. During backpropagation, SeeDNorm retains the capability to dynamically adjust gradients based on input magnitude. Experimental results demonstrate that SeeDNorm achieves faster convergence and superior performance compared to previously commonly used normalization layers saturated activation functions across various tasks language modeling and computer vision. We hope that this work will draw more attention try to improve current normalization layers.
Limitations and Future Work. To demonstrate the intrinsic effectiveness of SeeDNorm and its robustness to hyperparameters, we use the same configuration for all normalization layers across all models, except that multi-head variants use different numbers of heads for different model sizes. However, in modern large language models, the optimal initialization strategy and other applicable techniques often vary across different normalization layers, which remains to further explored. Furthermore, this paper verifies that improving normalization layers can bring substantial performance gains, but we introduce only a very small number of additional parameters. Therefore, we believe that investigating how to effectively scale up the parameter capacity of normalization layers to further improve the performance is a highly promising research direction.
REPRODUCIBILITY STATEMENT
In Section 3 we present a detailed, equation-level specification of the proposed method, and we provide Torch implementation pseudocode in Algorithm [ and Algorithm well as Triton implementation pseudocode in Algorithm 3 to facilitate faithful reproduction. The Appendix D further details the configuration of each experiment and all optimization hyperparameters, thereby ensuring result reproducibility. We also include comprehensive loss curves on various validation sets and accuracy curves downstream tasks for side-by-side comparison.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607:0607.0644 06440, 2016.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Christopher Clark, Kenton Lee, Ming-Woy Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2936,, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. aee.::803..55500 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling language models with mixture-of-experts In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 5547-5569, PMLR, 17-23 Jul 2022. URL mlm www.r.press/ve 62/ 2t2.dd222mhhmmmmo
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Leurning Reseurch, 23(120):1-99 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint
2020.
Sidney Greenbaum and Gerald Nelson. The international corpus of english (ice) project, 1996.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint 2Xii:2000.00244, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge local nash equilibrium. In
11

---

Published as a conference paper at ICLR 2026
I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc.. 2017. URL https: / https: /ttpceees///pooeeeinggggggg wttp:://w.....ruu..........oceaa cc/paper files/papes
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pp. 448 456. pmlr, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, pp. 1106-1114, 2012.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVV conference on computer vision and pattern recognition, pp. 11976-11988 2022.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pp. 4969,4983, Online, July 2020. Association for Computa- tional Linguistics. doi: 10. 12t.333/1/22220.c-mmaa.a anthology/2020.ccm maaimmain..mm 447
httt::tss: www.ww.wcllee. org/
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-2391, 2018.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Jacob Morrison. Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experts language models, 2024. URL/https: www.a.avvvvo........ abs/ 2209/44990.00 00000.02000000000000
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo furious, 2024.
org/ abs www.b//55000 22002251..000 00656
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195-4205, October 2023.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1256-1274 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-6) 2020.
Prajit Ramachandran, Barret Zoph, and Quoc Le. Swish: self-gated activation function. arXiv preprint 1710v:171, 7(1):5, 2017.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106.
2021.
12

---

Published as a conference paper at ICLR 2026
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Basa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018 URL htt.hhttssp:pprcceeeeieee ngs n wtt.:urri..ccccccooeeoe paper/2018/F11 h
files/
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural 2019.
Language Processing (EMNLP-)JCCNIP), pp.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gen gated mixture-of-experr layer. In International Conference Learning Representations, 2017.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy. Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024.
Felix Stollenwerk. The mathematical relationship between layer normalization and dynamic activation functions. arXiv preprint arXiv:2503.21708. 2025.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega: A question answering challenge targeting commonsense knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149 4158, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2rXiv:2307.09288. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-6008, 2017.
Llion Jones, Aidan N Gomez, Lukasz
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generuted Text, pp. 94-106, 2017.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates. Inc.. 2019. URL https::/ http://ceeeennooeeeeiis.ps./c//e neurips..cc/pperr files/paper//2 file file/1e8a19440 2eee988888822228888883eee1112333
Defa Zhu, Hongzhi Huang, Jundong Zhou, Zihao Huang, Yutao Zeng, Banggu Wu, Qiyang Min, and Xun Zhou. Frac-comnections: Fractional extension of hyper-comnections. arXiv preprint arXiv:2503. 2025a.
Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025b..
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A THE USE LARGE LANGUAGE MODELS (LLMs)
During the writing process, use LLMs only to assist in checking English spelling and grammar, as well as to standardize academic writing.
B GRADIENT RELATIONSHIP BETWEEN RMSNORM AND DYT
Notably, inspired by Stollenwerk 2025), dynamic tanh (Zhu et 2025b) and RMSNorm exhibit profound theoretical connections with regrad to the gradient in backpropagation.
Proposition B.1. In backpropagation, DyT is an approximate element-wist operation of RMSNorm under the assumption that the norm of the input vector is constant.
Prof. Let RM)(a) where a RIIDD it has been proven Zhang Sennrich, 2019 that
Ver RMS(z)
we conduct the derivation from the (M) Since RMS(x) = vD |
perspective of gradient equivalence:
vD
x
(v (vx" (vDx")) (VDD)))
r
(9)
|
RMS(Ø)
RMS(#)
D
Given RMS(x) that is constant, let it be denoted The operation be treated independent computation, and Equation 9 can be written element-wise differential equation as follow:
RMS[2] at each position
dra 1
(10)
dx
The steps to solve this differential equation are as follows:
dra dx
1 (1
1 dx
D-
- D-
dra
-
(11)
We integrate both sides of the equation, for notational convenience, we set all integration constants in the differential equation to zero by default:
[ D
I dra
drd
VD + rd
D.
ln
2/D
VD Td
(12)
VD + Fa
2
In
VD Ta
D
VD +
VD
Since -VD < ra < VD, the left side of the Equation
12 is necessarily greater than 0, the absolute
value symbol can be removed. Then we have:
VD + rd
2F LEVD
VD
(13)
eovD -
rd VD.
CevD +
14

---

Published as a conference paper at ICLR 2026
Since t2nh(2) c'-e--
2- have:
22g
CeVD -
7d VD
2xg
eovD +
VD
CEVD
eVb
(14)
eovD +
tVD
Id
D tanh
evD
Since DyT includes a learnable scaling coefficient the constant vD can be absorbed into y Similarly, eVD
1 can also be incorporated into a.
Consequently, although DyT preserves the input norm in the forward pass, it loses the ability to dynamically adjust the gradient scale based on the magnitude of during backpropagation, compared to RMSNorm. In contrast, our method retains norm information in both the forward and backward phases, endowing the model with data-dependent, self-rescaling gradients throughout the entire optimization.
C DETAILS OF GRADIENT ANALYSIS
C.I DETAILS OF GRADIENT ANALYSIS OF
Given the standard form of SeeDNorm as presented in Equation
D x
1
SeeDNorm(z) [a( (g)) (8))aa aa G RMS(x)) where RMS(x)
(15)
we primarily investigate its gradient with respect to each token x E IRIDD, For the input sequence X = [ 222 ..., N) E NxD since the computation SeeDNorm for each token æ; does not interfere with others, the gradient calculation can be performed by simply concatenating the results computed for each token.
The gradient of the SeeDNorm output with respect to y can be expressed as:
i SeeDNorm
i (lo(x
3
xx]
RMS(
(Y
RMS(
Oy
(16)
diag(
x
RMS(x:)
Here, RM)((=) E RIID, and diag refers the generation of D D diagonal matrix, where the di-
agonal elements (i, f) correspond to the i-th element of [M)( During the actual backpropagation update of y, assuming the overall loss of the network is L, the gradient of is given by:
x
1
i SccDNorm(z)
V SeeDNorm
(17)
07
where 3 N final result V,L E I
8 L
RR. 1x à SeeDNorm(z)
DxD and denotes matrix multiplication. The
07
is the update tensor for 7, Lx
Gradient of Multihead SeeDNorm. When employing the multi-head variant of SeeDNorm, since does not participate in the per-head computation of the dynamic component a -3")) a, the gradient with respect remains identical that of the standard (single-head) SeeDNorm.
15

---

Published as a conference paper at ICLR 2026
C.2 DETAILS GRADIENT ANALYSIS
For simplicity, we denote F SecDNorm(z) E IIID, = fa(x -B²) al E R and
r RMS(w) IRI The gradient of the SeeDNorm output with respect to a can expressed as:
i SeeDNorm(
i F
FF
Ô a
da a
dou
k,l=1..D
[(5 + 7k)rk
dai
Osk
Om
+
dai
doy
k.l=1.D
(18)
das +0
BJ)
Bas
kJ==I..D
(a 33 Irk )
[o(x:: IDxD]
r
[o(x B M
RMS(#))
Here, dạ is the Kronecker delta function, which equals 1 when k 1 and otherwise; IDx D represents the D D identity matrix, FR. Sk, and Ts represent the k-th elements of F, and respectively, while at denotes the /-th element of a. Similar to the gradient of 7, the final gradient of the SeeDNorm output with respect to a is SeeDNorm(a)
E IR DxD, and during backpropagation,
8
V. = SeeDNorm
8 SoeDNorm(a) d a
RRIDD
Gradient of Multihead SeeDNorm. When adopting the multi-head formulation, the derivation in in the third line onward. We define the number of split heads as Equation begins to differ from day
dsg
n, with = [ahy ..... Jh where ah, E RRI Similarly, a = [ar ..... ot,,, J. B = [B,,, .... Br...- Assuming k-th element and 1-th element belong to the i-th head and j-th head, respectively, when asa ass = 0; when i j. 8cu
i * j. det
a BT,)). The derivation is as follows:
Osk + Tk Day k.1=1.D
87
The biga(an, BT.))
dak +0)
rk Daz
Dau k.=11.D
(ouooaa Bh, )rk Skt) 6.1=1.D
(o(an, ph rk Out k.1=1.D
Tolon, 8h, 12x2
(19)
a aa B )
Tar,, 3%h I
RMS(#) av)
(Tr. B% I
C.3 DETAILS GRADIENT ANALYSIS 3
The gradient of the SeeDNorm output with respect B can be expressed as:
16

---

Published as a conference paper at ICLR 2026
i SeeDNorm (x)
F &
OFF
0 /
033
03r k,I=1..D
[8t + 7k) 1k).,
031
0sk Fh
+ Th
03:
031
k.l=l..
(a BT)
+0
03
k.==I..
0 0
0krk:
Σ
selt
(20)
03
1=1
k.1=1.D
OATK d' 3²)
Σ 3rBe
03t
A.mmI...
(aRTk o' B²) x1) A,1=1.D
= -'x (3)) (x)))" x
= d' 3²r) ((00
x
RMS(#)
where d'(+) denotes the derivative of (-), when (-) (-) is tanh, the above expression can also be written
as:
à SeeDNorm(z) (1 tanh2 (x B²)))
(as
T
x
(21)
Ô B
RMS(x)
Similar to the gradients with respect to and the gradient of the SeeDNorm output with respect to 3 is also a D x matrix, and the final gradient for updating 33 in backpropagation is given by SeeDNorm((z) E 1
L VgL = la (ee
8.3
Gradient of Multihead SeeDNorm. Similar to the derivation for a, the main difference in the gradient / under the multi-head form also lies in dss 23, in the third line of Equation 20 We also define that the k-th element and the 1-th element belong to the i-th and j-th heads, respectively. The derivation under the multi-head form is as follows:
88k 09,
8yk
Do (
r6
Tb
08,
auradi
09,
klli..D
Oyrihis 03,
abe Pnngie
apridi o' d'(any (Tr 03:
Thy Baji
(22)
L.
(OLTIdy a (mr: Th, 81 k.I-1.D
(OLTs d.j (at, 3 21 E.IET..D
RMS(w) [o'(an) Bi, Joong
o o'an, )amm
17

---

Published as a conference paper at ICLR 2026
C.4 DETAILS GRADIENT ANALYSIS
Beyond the update of learnable parameters in SeeDNorm, in this section, we also analyze the impact of SeeDNorm as a component the overall backpropagation of the network by deriving the gradient of the SeeDNorm with respect to its input The gradient of the SeeDNorm output with respect to can be expressed as:
i SeeDNorm(
i F
OFFk Oxi
i x
a
k.1=1.D
1(8k + 7/k rk]K,
Oxi
(23)
Osk
Ōrk
Tk
(sk + 77k)
0z
k,l=1..
do 33
Ork
+ (sk +/k)=
dr
kl
For the first term, we have the following derivation:
do(x -3))
D Σ
reBr
0z1
dz,
1=1
(24)
= (x: B3
E nrBh
Oz1
1=1
d' (T))
For the second term, we have the following derivation:
ŌTk
0zi
Oxi
RMS(x))
8kS
RMSI (x)
IK (.RMS(()
(25)
RMS (x)
8k1
RMS(x)
D (RMM
By substituting Equations 24 and 25 into Equation 23 we obtain:
i Norm
bu
aurio' (a (ee + (sk + +ww)
8
(MM((a)
D RMS
21211.
+ (7+) 1ixD
= "ee (r RMS(x) diag(e
(x=
(26)
D. RM)
o 3d (a) RMS(#)
++
1
diag(s +7)
(s +) 1xxD
RMS(2)
(M)
Gradient of Multihead SeeDNorm. Consistent with the previous derivations, the main difference in the gradient of SeeDNorm with respect to æ in the multi-heat form lies in the part corresponding to Equation 24 In the multi-head form, have:
18

---

Published as a conference paper at ICLR 2026
Osk
Oo (ah; (lhg)
Ori
2
= -bis 0ml
0
Σ
The Bryge
1=1
(27)
0
= = (zh.
//h))
The. Bhyye
dxi
= = (an.
The final form of the gradient in the multi-head setting is:
(e +0++>)" "y)"""111 11x2
(ar RMB
[o'( Bi 18n,
d' BT, 38
diag(a +++)
(se" (28)
RMS(a)
D. (MMS(())
C.5 DISCUSSION ABOUT GRADIENTS
Gradients of From Equation 16 it can be observed that the gradient magnitude of SeeDNorm kue
with respect to 7 is not influenced by the scale the input æ. Because RMS(kz) E. (kx,))
2 Therefore, the gradient of exhibits scale invariance, remaining fundamentally stable without requiring additional processing.
Gradients of a. By contrast, as shown in Equation 18 the gradient of a incorporates o(x ) which introduces scale-related information from and consequently deprives the gradient of the scale invariance observed in 7. Therefore, to ensure training stability, we need to constrain its range within fixed interval using an activation function a. Ultimately, this purpose. Additionally, a directly multiplies with other terms in the gradients of both ,3 and æ without constraints. Therefore, we prefer the model to more cautious when initiating updates for a. To achieve this, we initialize (3 to O, ensuring that a starts with a smaller gradient during the update process.
adopt the lanh function for
Gradients of [3. The gradient with respect to 3 further depends on æ and a. When æ is abnormally 1. the gradient of 3 approaches 0 at this large, since - tanh²(a (a:)) is a higher-order infinitesimal x. point. When æ is abnormally small, the gradients of a and / also approaches 0. This situation is rare in practice, and even if it occurs, y can still be updated normally. And subsequent analysis also indicates that anomalous values of do have a catastrophic impact on the gradient of preceding layers during backpropagation when SeeDNorm is applied. Therefore, our primary concern is to prevent gradient explosion. Since a directly affects the gradient of B, we apply weight decay to a. Similarly, because B also influences the gradient of apply weight decay to 3 as well, control their numerical stability.
Gradients of the input æ. Regarding the gradient of æ. the first term is similar to the gradient of J it incorporates information about the norm æ and uses o' to keep the values bounded. When æ is abnormally large, d'xx)
tanh (x))2(x)) approaches O, and since infinitesimal of _. this ensures that the gradient does not explode. At this point, the gradient of SeeDNorm with respect to the input æ is primarily dominated by the last two terms. Conversely, when is abnormally small, of (zx approaches 1, and numerical stability of this term can be maintained by constraining the values of a and B3, and the gradient with respect to æ is again dominated by the latter terms.
tanh (z:) is higher-order
For the last tWO RMS(#) (diag(s 7) assuming a' kar, the expression becomes:
terms of Equation " D. (M)
26 they can be expressed as When the sample æ undergoes scaling,
(<TTx))
19

---

Published as a conference paper at ICLR 2026
1
(o(kæ R.RMS
(1kxx-8")) 33)) a a))))=1111 11xD
diag(o(ka B²) 8¹).an a +
3 (kæ" kx))
RMS(kw)
1
diag(o (²) a ao+1 +2)
(a(okk 3²) 7) 1xxD
④ (xx m) (29)
(R)S(x)
k2D RMS (x)
1
ddaag( 3²) a x7 (o(kæ B3") a "1=11x) 11xD (x .x))
(R)
D RM)*
When is abnormally large, o() approaches I when a is implemented with tanh, and the numerator and denominator of (a) are of the same order of magnitude. Therefore, the above expression is primarily influenced by kRMS(=). and it scales down by factor of k. Therefore, this achieves form of adaptive stability. When the output scale of the previous layer is abnormally large, the corresponding gradient in backpropagation decreases, thereby ensuring training stability.
_=".
When k is abnormally small, tanh(kæ B²) and k- RMS( (x) are equivalent infinitesimals. Therefore, the above expression is primarily influenced by kRMS(()) As lo decreases, 3RMS(w) increases proportionally, thus also achieving adaptive gradient stability.
Table 5: All validation datascts and downstream test datascts used in OLMoE and OLMo2. For the validation set, our primary metries are validation loss and perplexity (PPL). For downstrcam tasks, we conduct zero-shot evaluation and report answer accuracy (Acc%) as the key metric.
Validation Datasets
c4 cclen-validation (Raffel et al. 2020)
hol.mebbookks-valldaa..... (Soldamn al. 2024)
do L0nmacccommopppraa----aa1a1a1111 lon (Soldam et al.
2024)
dolma pes2o-valicationn (Soldamn al.
2024
wamaarrdalllllllattii..... (Soldaini et a
2024
dolmasstaack-vall.atttooo (Soldaini al.
2024
dolma.wikii-validattioo (oolann al.
2024
ice-validatiion (Gr & Nelson
1996
mmd2.s2orcovallcatt7. (Lo et al.
2200)
pile-validation (Gao et al. 2020)
wiki 1aa111001dditioonn (Menty al.
2016
Downstream Tasks
PIQA (Bisk et 2020
HellaSwag (Zellers et al.
2019)
ARC-Challenge CClark al.
2018
ARC-Eas: (Clark al. 2018
MMLU-Var: (Hendrycks et al.
Winogrande (Sakaguchi al.
2021
Openbook-OA (Mihaylov ct al.
2018
SCIQ (Welbl et al. 2017)
COPA (Roemmele et al. 2011
Boolq Clark
2019)
Commonsense.on (Talmor et al.
2019
Social.IQA (Sap et al.
2019
C.6 VARIANCE OF THE DOT-PRODUCT OF Two RANDOM VECTORS
Theorem 3.2. hige-diima space, the variance of the dot product of two random vectors is proportional to their dimension D.
Prof. Suppose there two D-dimensional random vectors [21, 22, ED] and y [y1, y2, .... Vn]. Their components are independent and identically distributed (i.i..) random vari- ables, and they satisfy:
20

---

Published as a conference paper at ICLR 2026
E(x;) E(y)) = 0
(30)
Var(z;) Var(U:) o²
Then Var(x y²) Var Var(E D
i= I.yh), let 3", we have:
Var(s) E(s²) (3) E(s²)
D
D
F F.
Fall
= F.
(I,n))))))))
(31)
D
Σ
E [m
DE
-
Eiy
Therefore, E|s²) Do is proportional to the dimension size.
D DETAILS OF EXPERIMENTS AND MODEL SETTINGS
In this section, will provide experimental results, detailed description of the different model configurations and hyperparameter settings used for each task, as well as the parameter settings for SeeDNorm.
D.1 OLMOE AND OL.o2 IN LANGUAGE MODELING
D.1.1 VALIDATION DATASETS AND DOWNSTREAM TASKS
The validation datasets and downstream task datasets used for OLMoE and OLMo2 in the language modeling task are presented in Table 5 For the validation datasets, we primarily focus on the validation loss and perplexity (PPI... Since the trends of PPL. and loss are consistent, we mainly report the loss results. For the downstream task datasets, we report the answer accuracy rate of the model.
D.1.2 MODEL AND TRAINING SETTINGS
OLMoE. The model configurations and
used in OLMoE-1.30 and OLMoE-7B models are presented in Table The training and optimization parameters for both models are shown
in T
Table Configuration and hyperparameters for
Table 7: Hyperparameters for optimizer of OLMoE-1.3B and OLMoE-7B.
OLMoE-113B and OLMoEE7B.
Model
OLMoE-1.3B
OLMoE-7B
Model
OLMoE-L33B OLMoE-7B
Num Layer
12
16
Optimizer
AdamW (81 0.9, B2 0.95)
Hidden Dim
1024
2048
Learning Rate
4,0c-4
4.0c-4
Num Head
16
Weight Decay
Position Embed
RoPE (0 = 10000)
LR Schedule
Cosine
Cosine
Context Length
4096
4096
Warm up Tokens
10B
10B
MoE Top-k
8
8
Balance Loss Weight
0.01
0.01
MoE Experts
64
Router z-loss Weight
0.001
0.001
Weight Tying
Yes
Yes
Gradient Clip
1.0
1.0
Micro BatchSize
Global BatchSize
768
1024
OLMoB baseline employs PreNorm-based structure, where normalization is applied at the input of both the attention and MoE layers. Additionally. there are query norm and key norm layers within the attention layer. A normalization layer is also present the Transformer output. We replaced all these normalization layers from RMSNorm to our proposed SeeDNorm. Specifically, for QueryNorm and Kormorm, perform SeeDNorm in each attention head.
21

---

Published as a conference paper at ICLR 2026
OLMo2. The model configurations and ayper-parameterys used in our OLMo2- 550M and OLMo2- 1B models are presented in T The training and optimization parameters for both models are shown in Table The application of the SecDNorm is consistent with that in OLMoE.
Table 8: Configuration and hyperparameters for
Table 9: Hyperparameters for the optimizer of OLMo2-550M and OLMo2--BB.
OL.o2-550M and OLM
Model
OLMo2-550M
OLMo2-IB
Model
OLMo2-550M OLMo2-IB
Num Layer
16
16
Optimizer
AdamW (31
0.9, 32 = 0.95)
Hidden Dim
1536
2048
Learning Rate
3.0e-4
4.0c-4
Num Head
16
32
Weight Decay
0.1
0.1
Num KV Head
LR Schedule
Cosine
Cosine
Position Embed
RoPF (0 500000)
Warm up Tokens
8B
8B
Context Length
4096
4096
Gradient Clip
1.0
1.0
Weight Tying
Yes
Yes
Micro BatchSize
8
4
Global BatchSize
1024
1024
D.1.3 EXPERIMENT RESULTS
OLMoE, Figure summarizes the downstream-takk comparison between OLMoE-1 1.Mo--1.3B equipped with SeeDNorm and the RMSNorm baseline. SeeDNorm yields consistent and often substantial gains across almost every task, delivering > 2x speed-up multiple datasets such as ARC-Fasy (Clark (ct al., 2018). ARC-Challenge (Clark et al. 2018), and Social-IQA (Sap et al. 2019). Figure reports the analysis on all validation splits and shows equally pronounced advantages. Figure 7 summarizes the downstream- task comparison between OLMoE- 7B equipped with SeeDNorm and the RMSNorm baseline. SeeDNorm also yields consistent and often substantial gains across almost every task in OLMoE-7B, delivering > 2x x speed-up on multiple datasets such as ARC-Challenge (Clark et al., 2018). Social-TQA (Sap al. 2019) and PIQA (Bisk al., 22200). Figure 8 reports the analysis all validation splits and shows equally pronounced advantages.
OLMo2. Figure Snggure/illusraattee illustrates the comparison of validation loss between SeeDNorm and the bascline RMSNorm for OLMo2-550M across all validation sets. And Figure illustraas the comparison of accuracy of downstream tasks between SceDNorm and the baseline RMSNorm for OLMo2-IB across all validation sets. Our method similarly achieves consistent improvements in most validation datasets and downstream tasks.
D.2 VIT AND CONVNEXT IN IMAGE CLASSIFICATION
In the image classification task, we conduct training the ImageNet-1K (Krizhevsky al. 2012) training set and performed evaluation on the test set. The model configurations and hyper-parameterss in our used ViT-B and ViT-L models in image classification task presented in Table T0 The training and optimization parameters for both models are shown in Table
Table 10: Configuration and hyperparameters for VIT- B and VIT-L.
Table 11: Hyperparameters for the optimizer of VIT-B and VIT-L in image classification task.
Model
VIT-B
VIT-I
Model
VIT-B
VIT-L
Num Layer
12
24
Optimizer
AdamW(Bi
0.9, 32 0.999)
Hidden Dim
768
1024
Learning Rate
4e-3
4e-3
Num Head
12
16
Weight Decay
0.05
0.1
Patch Size
16 x 16
I.R. Schedule
Cosine Schedule 20 Epochs
EMA
0.9999
0.9999
Warm up
Input Resolution
224 x 224
Gradient Clip
1.0
1.0
DropPath
0.1 0.6
Global Batch Size
4096
4096
Global Pool
Average Pooling Yes 16
Training Epochs
300
300
SecDNorm Dropout
Yes
SceDNorm Head
32
In all ViT classification experiments, we employ the multi-head variant of SeeDNorm to enhance training stability, because the classification task requires hundreds of training epochs, models prone to overfitting and can easily result excessively high gradient variance (Pezeshki et al.) 2021).
When Multihead See is not used, larger models like VIT-L cannot even converge. To further
22

---

Published as a conference paper at ICLR 2026
cons. Ass.
PIDA
300
"Skara
3C1
130
Bhara lomo)
easy
Challenge dec.
Socihl Hgp dett.
Winedrande Art.
Tokers 18 tra
PEPELL U. ather
MULLU sen, Ver Arc.
Commoneenge 9A Acc.
Doemsttrant Avey ACC.
Acuray h0
Aocur Pi
Octdd
Tohera 40
20r
300
300 500
Tokorra
Figure 5: Comparisons of the accuracy all downstream tasks from Table OLMoE-1.3B when using SeeDNorm as the normalization layer versus the default RMSNorm. The figure illustrates the evolution of downstream task accuracy as the total training tokens increase during training, with transparent lines indicating unsmoothed results and solid lines denoting 0.99 EMA amoosheothedd results.
Icee vol. Lote
Pia val, Lose
CB val, Lnss
Dolms val. Lees
2.04
30
3.20
"uhurs kuna)
Takana (ril
Tbeers crul
Slara
Delma with val,
sZore
Dolme books Lopa
Delme peazo
255
3.30
200
232
"Shors (01
Talons Frrsi
Tocere cnd
Siore bang
1.25
Doina slack val.
and
2.2
Reat 163 val.
0250
Delina reditt val.
2.6
Training
5225
242
5100
255
150
252
350
254
0.0
Maors und
Figure 6: Comparisons of the validation CrossEntropy loss of all validation datasets from Table 5 in OLMoE- 1.3B when using SeeDNorm the normalization layer versus the default RMSNorm. The figure illustrates the evolution of the validation loss as the total training tokens increase during training.
stabilize the training, we additionally divide a 33T by the dimension and apply dropout to the whole dynamic coefficient alx (²)) - of SeeINorm; the dropout rate of NermNNorm is set equal to the
23

---

Published as a conference paper at ICLR 2026
CON Ass.
PSDA
Beotd
240
304
Takers 20
Takens
1000
2005
(28 famb
easy
Challenge dec.
tgpe dic.
Winedrande Arc.
Tokees (bd kunc)
18 tred
Biaar km
Goiall
t B
PEPELL ather
MULLU Ver ACc.
Commonaenne DA Acc.
Desmstraom Avta AEC.
200r30 150
120
Touara (alliaa)
200 300
LDX
Skara
2000
300
30 "Muara (11lan)
Takera
201
1000
Figure 7: Comparisons of the accuracy all downstream tasks from Table in OLMoE-7B when using SeeDNorm as the normalization layer versus the default RMSNorm. The figure illustrates the evolution of downstream task accuracy as the total training tokens increase during training, with transparent lines indicating unsmoothed results and solid lines denoting 0.99 EMA-smoother results.
Less
Pio vell Less
val. lass
Dolmn
Loss
24
227
2.20
204
2.44
24
277
"atera (ii
Takora orat
"oeara cnl
Blara (30
214
1:00
250
230
Dolma stack vol.
Bext 203 vol.
Dolma reddit val.
Training Less mut Tokens
culal
2.50
conten
33
057
3,30
240
2.00
280
Tooare Scnel
Toone Eent
10%
Figure 8: Comparisons the validation CrossEntropy loss of all validation datasets from Table in OLMoE--B when using SeeDNorm as the normalization layer versus the default RMSNorm. The figure illustrates the evolution of the validation loss as the total training tokens increase during training.
drop-path rate of the model. Notably, VIT-I,, when augmented with SeeDNorm, exhibits strong fitting
24

---

Published as a conference paper at ICLR 2026
Cd.an vel,
Dolme
124
212
122
111
3.04
"Stars
2.20
377
13
112
157
(31 lamg
Dalena writi val,
Doline books Loct
Dalesa pa:lo
1.212
2.53
2.50
2325
Motoo trst
100ers 181 trd
1ANS 154 km))
Tranis sa11 Tokees
1.54
2,772
and use
2.00
1.54
1150
5.73
2.00
1.4
1.4
2.75
Figure 9: Comparisons of the validation Erosspy loss of all validation datasets from Tahle SahleShin OLMo2- 550M when using SeeDNorm the normalization layer versus the default RMSNorm. The figure illustrates the evolution of the validation loss as the total training tokens increase during training.
COPN Acc.
PRQA
SdQ
Boold
QUAc Bocdlon
Eoco
102
"sove
Tarns
Bioce ism
ARC eany
ABE Challeng ACC.
Stecle Act
Winedtrande Bct.
Taknr (Silera)
Toeara (B cral
Blara loma
MEIUU other
MULLU Ver Acc.
Commonaenae DA Acc.
Dowmarraan Avg.ACC.
260
Toons (383)n))
Skene
Tslons (2.1
Figure 10: Comparisons of the accuracy all downstream tasks from Table in OLMo2-1B when using SeeDNorm as the normalization layer versus the default RMSNorm. The figure illustrates the evolution of downstream task accuracy as the total training tokens increase during training, with transparent lines indicating unsmoothed results and solid lines denoting 0.99 EMA-smoothed results.
capabilities, necessitating a further increase its drop path rate to 0.6. For all models, the final classification accuracy is evaluated using the EMA model.
25

---

Published as a conference paper at ICLR 2026
Table 12: Configuration and hyperparameters for Con-NeXT-B and ConvNNeTT-L.
Table 13: Hyperparameters for the optimizer of ConvNeXT-B and ConvNeXT-L in image classifi- cation task.
Model
ConyNeXT-B ConvNeXT-LL
Depth
[3. 3, 2]
[3. 3, 27, 3]
Model
ConyNNXTTB ConyNeeTT-L
128
192
Optimizer
Adam W(31 1 = 0.9, $2 = 0.999)
256
384
Learning Rate
4e-3
4e-3
Dims
512
768
Weight Decay
0.05
0.1
1024
1536
LR Schedule
Cosine Schdule
EMA
0.9999
0.9999
Warm
20 Epochs
Input Resolution
224
224
Gradient Clip
1.0
1.0
DropPath
0.5
0.5
Global Batch Size
4096
4096
SecDNorm Dropout
No
No
Training Epochs
300
300
SccDNorm Head
16
32
Is init value
le-6
le-6
head init scale
1.0
1.0
The model configurations and hyper-parameters in used ConyNeXT-B and ConvNeXT-L models in image classification task are presented in Table [2 The training and optimization parameters for both models are shown in Table [3 The configuration of ConvNeXT largely consistent with that of ViT. The main difference lies in the ConvNeXT model, which is structured into four stages, each with varying depths and dimensions. The detailed configuration presented in Table T2
D.2.1 EXPERIMENT RESULTS
In the main text, we have already reported the accuracy results for the image classífication task. Figure 11 and Figure 12 present detailed comparisons of the loss curves during training. With the application SeeDNorm, our method demonstrates clear advantage over DyT in the training curves. When scaling up ViT-L, despite using higher drop path rate, the loss advantage becomes more pronounced. This also reflects an enhanced fitting capability of the model under the influence of SeeDNorm, although this gap is not effectively reflected in the accuracy data the test set.
Training Loss w.r.t Epochs
Training Loss w.r. Epochs
VIT-S pyT VIT STe_See
VIH
XTi_SeeDHam
Lo8
Trainin
2.6
2.2
2.4
100
125
150
175
200 Epochs
225
250
275
300
2900
125
150
175
200
225
250
275
300
Epochs
Figure 11: Comparison of the Cross Entropy loss curves for ViT-B and ViT-I the ImageNet image classifica- tion task using DyT and SeeDNorm, plotted against the number of training epochs. The loss curves have been smoothed using a 0.99 EMA.
D.3 VIT IN MAE
In the MAE pre-training and fine-tuning task, we also employed VIT-B and ViT-L as the primary research models. The model structures of ViT-B and ViT-L are consistent with those presented in Table 10 with the distinction that neither EMA is used during the pre-training nor the subsequent fine-tuning processes for MAE, drop path and dropout is also not used in the pre-training phase. The training and optimizer configurations for MAE pre- pre-training and fine-tuning are detailed in Table [4] and Table [5] respectively. During the fine-tuning process, for VIT-L, we increase the drop path rate to further prevent overfitting. For the remaining configurations, we maintained consistency with the DyT (Zhu al. 2025b) baseline.
26

---

Published as a conference paper at ICLR 2026
D.3.1 EXPERIMENT RESULTS
In Figure 13 we plot the loss comparison curves during the pre-training process. Whether for ViT-B or ViT-L., the application of SeeDNorm significantly reduces the loss. In Figure[T4] we illustrate the loss variation curves during the fine-tuning process, where SeeDNorm similarly achieves notable reduction in loss during fine-tuning.
Training Loss w.r. Epochs
Training Loss w.rt Epochs
3.6
ConNNeXT& DyT
ConNNeTT DyT
Training Loss
3.2
Training
3.0
2.8
2.6
100
125
150
175
200 Epochs
225
250
275
300
1.00
125
150
175
200 Epochs
225
250
275
300
Figure 12: Comparison of the Cross Entropy loss curves for ConvNeXT-B and ConvNeXT-L the ImageNet image classification task using DyT and SNomm plotted against the number of training epochs. The loss curves have been smoothed using 0.99 EMA.
Training Loss w.r.t Steps
Training Loss w.r.t Steps
0.435
0.430
Vit-5 Dy
VIF4. Dy
SiTe_5eettom
0.425
VTTL_SeeDNom
0.420
Training Loss
0.425
0.415
0.420
Training
0.410
0.405
0.415
0.400
0.410
0.006
0.004
0.405
0.390
150 Steps (K)
Steps (K)
Figure 13: Comparison of the MSE loss curves for ViT-B and ViT-L on the MAE self-supervised image masking reconstruction task using DyT and SeeDNorm, plotted against the number of training epochs. The loss curves have been smoothed using 0.99 EMA.
Training w.r.t Epochs
Training Loss w.r.t Epochs
6.0
VITS byT
WTFL
vira SweDkem
WFLL SeeDWom
5.0
Training Loss
5.0
Training Loss
4.0
4.0
3.5
3.0
x1.2
2.5
20
40
60
80
100
2.5
10
20
30
40
50
Epochs
Epochs
Figure 14: Comparison of Cross Entropy loss curves against the number training epochs for ViT-B B and ViT-L, using DyT and SecDNorm with full palammeteer fine-tuning initialized with MAE pre-traine rrantriind weights on the ImageNet image classification task.
27

---

Published as a conference paper at ICLR 2026
D.4 DIT IN IMAGE GENERATION
In the image generation task, we conducted experiments based on DiT-B (Peebles & Xie 2023) and DiT-XI. (Peebles & Xie 2023), with the model configurations detailed in Table T6 and training hyperparameters in Table respectively. For the image generation task, because the random noise and timestep sampling of diffusion greatly enrich sample diversity, it is harder for the model to overfit compared image classification tasks, and using the standard form of SeeDNorm is sufficient to ensure stable training of the model. Therefore, have not explored the multi-head form this stage.
Table 14: Hyperparameters for the optimizer of ViT-B and ViT-L MAE pre-training.
Table 15: Hyperparameters for the optimizer of ViT-B and ViT-L in MAE fine-tuning.
Model
ViT-B
ViT-I
Model
ViT-B
VIT-I
Optimizer
AdamW (B1 0.9. B2 = 0.95)
Optimizer
AdamW (81
0.9, 82 0.999)
Learning Rate
2.4e-3
2.4c-3
Learning Rate
2c-3
4e-3
Schedule
Cosine Schedule
LR Schedule
Cosine Schedule
Weight Decay
0.05
0.05
Weight Decay
0.05
0.05
Mask Ratio
0.75
0.75
DropPath
0.1
0.2
Warm up
40 Epochs
Warm up
Epechs
5 Epochs
Gradient Clip
No
No
Gradient Clip
No
No
Global Batch Size
4096
4096
Global Batch Size
1024
1024
Training Epochs
800
800
Training Epochs
100
50
Table 16: Configuration and hyperparameters for DřT-
Table 17: Hyperparameters for the optimizer of DiT- B/4 and DiT-XI/2 in image classification task.
B/4 and DIT-XL/2.
Model
DIT-B/4
DIT-XL/2
Model
DFT-B/4
DIT-XL/2
Num Layer
12
28
Optimizer
AdamW (3
0.9, Bg 0.999)
Hidden Dim
768
1152
Learning Rate
le-4
le-4
Num Head
12
16
LR Schedule
Constant
Image Size
256 x 256
256 x 256
Weight Decay
Latent Size
32 32
32 x 32
class drop prob
0.1
0.1
Patch Size
4
2
Global Batch Size
256
256
ML.P Ratio
E PYTORCH IMPLEMENTATION OF SEEDNORM
In Algorithm| [ and Algorithm 2 we implement our proposed SeeDNorm and its Multihead form respectively, using PyTorch-like style.
Algorithm Pseudocode SeeDNorm in a Py PyTorch-like style.
class SceDNorn(Module) def__init_ D. init):
super()..intt.))
self.a = Parameter(ones(D) init)
self,ß Parameter(zeros((D)
self. Parameter(ones(D))
def' for,,rd(self,, x):
rescale tanh(x @ self,g)
= RMS(x)
dynamic_scee scale rww.aeee.euueee (nncueeluu)q))))e( self.a
return (dynamic scale + self.r)
28

---

Published as a conference paper at ICLR 2026
Algorithm 2 Pseudocode of Multihead SeeDNorm in a PyTorch-like style.
class Moduuer
def isett_ssse__.ff D, init, num num_heads):
superr)...int_..
self.o Parameter(onex(DD) init)
self.,3 Parameter((seoos(D)
self. Parameter(ones(D))
self.num hheddddm_ = heads
def forward(self, x):
B, D xx.shape
x diyppe boyyp=xxdttpp x.dtype
= x.reshapeeBBB N, sewf.num heads, D // self.num hhw.hhh.aaansssseee transposs,, 1, 2) f selss..es self.num
D // hea.nuummeaaadeeeeeee..e 1, 1. D).transoose-1, -2) activate coa.hh(toorc.mammulll. ().float())
a self.a.reshapel,,, self.num heads, 1, self.num aef.num.heads).rgg reaes)..e 1,1,1 1)
dynamic_scale = activate * a
dynamic_scale = sca.miiiccall..((xtttpee))
x x / RMS(x)
return (dynamic scale self.y)
Liffirm
Figure 15: Input-output distribution comparison between SeeDNorm and RMSNorm on OLMoE-II IB.
Figure 16: Input-output distribution comparison between SeeDNorm and DyT on VIT-B.
F PARAMETERS AND COMPUTATION CoST
Compared to RMSNorm, SceDNorm introduces two additional D-dimensional parameters, and G. In the entire Transformer network, assuming each layer includes input normalization for the attention layer and FFN, QKNorm within the attention layer, and normalization the Transformer output, the newly introduced parameters amount to (2 x 2D + 2 x 2 x D/H) x N + 2D (4N + N + 2) x D, where H is the number of attention heads. Since N is much smaller than D, the increase in the overall parameter is much smaller than linear layer, which can be considered
29

---

Published as a conference paper at ICLR 2026
Algorithm 3 Triton Implementation of the forward process of SeeDNorm.
@triton.jit
def seednorm fwd kernel(
X. alpha, beta, stride_ml, stride_n, N. eps, BLOCK.SIZE: tl.constexpr,
}:
row tl program batch tl. (.prorram.i(()
base = row stride_m mml + batch stride_n
Y += base_idx
X += base idx
ims (lLLckss(BLLOC SIZEJ, dtype=d..loat32) _oot_product for off in range(O, N. BLOCK_SIZE)
dtype=tl.fooa322))
= tl.arange(O, BLOCK.SIZE)
a= ==tllooad(X + cols, mask=cols N, other=00.0).oo((ffoot2.)
beta eeta.clement= th.eta cols, mask=cols < ().to((l.fooa33)
.rsm += a
_dot_product += betmmet
ms tl..qqrt(t) .mms/ eps) dot = sl.sum( _dootopppp.. productt
neg.two.x = 2200 dot producct exp_neg_two. tl.exp(neg_tvv) (woo(eee__o_o))) !wo_x)
dot product (1.0 exp neg x) (1.0 exp neg x)
for off in range(0, N, BLOCK SIZE):
cols = off tlaarane(O,, BLOCK.SIZE)
mask = cols =l cols, mask=mask)
alpha = load(allh cols, mask=mask)
= tl.load(X + cols, mask=mask, oo
hah = / rms
x_hat (w + alphaaeleeee element dro_prod)ct)
d.store(Y cols, y.too..llype.eeeee wtype..lement..y... ty), mask=mask)
class Mocuule)
def (sei D. init):
super()._.nii_..0)
self.a Parameter(ones(D) init)
self.3 = Parameter((zeros(D))
self. = Parameter(ones<(D)
def forward(x):
y= forch.empty_likefff
M, L, N x.shape
grid (M, L) seednorm fwdd kernelgrrid]
x, y, weight, self.alpha, self.beta, x.stride(O), x.stride(1), L,, N, self.eps,
return y
30

---

Published as a conference paper at ICLR 2026
negligible. In terms of computational complexity, compared to RMSNorm, SeeDNorm introduces two additional matrix multiplications, one element seement-wise activation, and one element- wise addition along the channel dimension. For each token additional multiplications is 2/,, and the number of additional additions is D + 1. The total additional multiply-ade operations for the entire network are (47 1) +1)++) H - x
IIX compared RMSNorm, the number of
(8N + H + 4) x - 4N = O(D). In contrast, D D linear layer already involves computational complexity O Therefore, the additional computational overhead introduced by SeeDNorm remains negligible.
However, when using only the PyTorch implementation, SeeDNorm requires more memory access operations and these operations more fragmented compared to RMSNorm. This will affect latency and overall efficiency to a certain extent. practical applications, we recommend fusing the operations into single kernel function, thereby achieving comparable efficiency. The implementation of triton kernel of the forward process is shown Algorithm
G VISUALIZATIONS
To further understand the behavior of SeeDNorm in representation learning, we visualized the input- output distribution of SeeDNorm in Figure 15 and Figure T6 Specifically, we compared SeeDNorm with RMSNorm on the OLMoE-IB model in Figure 15 and SeeDNorm with DyT on the ViT-B model in Figure 16 respectively. The SeeDNorm layers visualized here all the final output normalization layers of the Transformer, and the two figures in the same column share the same sample imputs. In all visualizations, the x-axis denotes the input values of elements, the y-axis denotes the normalized values of SeeDNorm, and points of the same color correspond to samples from the same token. Due to the large number of points, we use JPEG images instead of vector graphics here. It can be observed that compared to DyT and RMSNorm, the outputs of SeeDNNrr show more distinct separation between different samples, instead of being clustered together. all samples are more uniformly distributed across the entire space, indicating that the features of different tokens more discriminative.
31


Reviews:
[{"review_id": "dYF9JJ3IK1", "reviewer": "Reviewer_EBT8", "summary": "The authors point out that existing normalization layers, such as RMSNorm and LayerNorm, overlook the norm and scale information of the input. In situations where there are distributional shifts, the norms of the input can provide valuable information and a consistent scaling factor. The authors argue that ignoring this aspect reduces the expressiveness of models, particularly in large language models and vision models that operate across different domains. To address this issue, the proposed SeeDNorm introduces a dynamic scaling factor for normalization that is based on the input itself. This approach helps to retain some norm and scale information while adapting to each individual input sample. When training large-scale transformers, whether for language or vision tasks, and when facing varying input distributions, it may be beneficial to consider this proposed method as it could potentially enhance performance by avoiding the limitations of fixed normalization.", "questions": "1. How beneficial would SeeDNorm be in smaller models or resource-constrained environments, such as mobile or edge devices where memory and speed are crucial? Could the authors provide some insights on this?  \n2. How stable and easy is it to integrate SeeDNorm into different network architectures?  \n3. Does SeeDNorm generalize across various tasks without requiring extensive tuning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. SeeDNorm offers improved preservation of input norm information and adaptability to shifts in input distribution. Unlike RMSNorm, which disregards input norms, SeeDNorm retains some of the \"scale\" information through a dynamic term. This scaling, being dependent on the input, allows it to better manage varying magnitudes or changes in the domain.\n\n2. Experiments conducted across language tasks demonstrate that SeeDNorm accelerates convergence and produces better final results compared to the baseline methods of RMSNorm and LayerNorm.\n\n3. The paper not only presents empirical results but also includes the derivation of gradients, discusses invariance properties, and addresses initialization and regularization choices for stability.", "weaknesses": "1. For extremely large models or those with many layers of normalization, the benefits of SeeDNorm may be limited. The paper indicates that the improvements diminish for dense models compared to Mixture of Experts (MoE) models.\n\n2. Although SeeDNorm shows consistent improvement, the gains are often subtle, typically just a few tenths of a percent in accuracy, particularly for dense models, as illustrated in Table 2.\n\n3. While the paper broadly discusses \"vision tasks,\" it does not assess tasks that heavily involve spatial features, such as detection, oriented bounding boxes (OBB), and segmentation. More targeted experiments would strengthen the validity of their claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SFSLqVknxz", "reviewer": "Reviewer_HBqF", "summary": "This paper proposes SeeDNorm (Self-Rescaled Dynamic Normalization), a dynamic normalization method that preserves input norm information and adaptively adjusts the scaling factor based on the current input. The work aims to address the limitation of RMSNorm, which discards the input norm during the forward pass and relies on a fixed scaling factor that cannot adapt to data variability or distributional shifts, leading to suboptimal performance in zero-shot scenarios. SeeDNorm introduces a self-rescaling mechanism that modulates the scaling coefficient according to the input itself, thereby retaining norm information and enabling data-dependent normalization. The paper provides theoretical analysis of scaling invariance and gradient behavior in both forward and backward propagation, showing that SeeDNorm maintains numerical stability and adaptively adjusts gradients according to input norm. Experiments demonstrate consistent improvements over RMSNorm, LayerNorm, and DyT across both language modeling (dense and MoE LLMs) and computer vision (ViT, ConvNeXt, DiT, MAE) tasks, with faster convergence and better zero-shot performance achieved with negligible additional parameters and minimal computational overhead.", "questions": "- How do $\\alpha$, $\\beta$, and $\\gamma$ evolve during training? Do they converge to stable values or continue adapting dynamically?\n- In Figure 3 (fourth subplot), SeeDNorm shows a temporary spike in loss; is this caused by instability in the rescaling term or by random training noise?\n- Why is the multi-head variant used only for vision models? Would multi-head rescaling help stabilize training in language models, or is it redundant in that setting?\n- Can the authors provide quantitative measures of computational overhead, such as runtime or FLOPs, to support the efficiency claim?\n- How does SeeDNorm interact with normalization-free or re-scaling transformer variants, and could it complement them?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is technically detailed and the motivation is clear: existing normalization layers such as RMSNorm provide stability but lose information about the input magnitude. SeeDNorm offers a straightforward extension that dynamically rescales activations conditioned on the current input, improving adaptability to data variability and distributional shifts. The theoretical analysis is comprehensive, covering forward and backward derivations as well as scaling invariance, and helps clarify why the method remains stable. The experimental section is broad and systematic, spanning both language and vision tasks, and the results consistently show better convergence and downstream performance. Ablation studies on activation choice, initialization, weight decay, and the multi-head formulation strengthen the empirical support. The method is simple to integrate and performs reliably across different architectures.", "weaknesses": "Although the results are strong, the contribution may appear incremental because SeeDNorm can be viewed as a combination of RMSNorm and DyT that merges their strengths while mitigating DyT’s vanishing gradient issue. The proposed dynamic rescaling mechanism resembles existing modulation approaches such as gating or FiLM-style conditioning, differing mainly in how the rescaling term is computed. The theoretical analysis mainly focuses on stability and gradient behavior but gives limited insight into why the dynamic adaptation improves representation learning or generalization. The paper states that the computational overhead is negligible, but there is no quantitative evidence such as runtime or FLOPs comparison. In addition, the multi-head version is only applied to vision tasks, and it is unclear whether a similar strategy could benefit language models or whether it was found unnecessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r0FGxF25U5", "reviewer": "Reviewer_gTBE", "summary": "This paper introduces SeeDNorm, a novel normalization layer for neural networks. SeedNorm is an extension of RMSNorm, that modulates the scale parameter gamma of RMSNorm by adding a term to gamma which consists of two learnable parameters. These parameters act on the unnormalized input of the normalization layer, and thus allow scale information to flow through the normalization layer. The authors prodide a gradient analysis of SeeDNorm in order to show that their normalization layer is insensitive to extremely large or small values, and to justify their regularization and initialization choices. They provide experiments with OLMo models for LLM pretraining, DiTs for Image Generation, ViTs and ConvNext for Image Classification, and ViTs for Self-supervised learning, overall reporting improvements over baselines like LayerNorm and DyT. They also provide ablations on their design choices.", "questions": "- How was the training setup for the LLM experiments chosen? Is it the default OLMo recipe? \n- in line 302 the authors write that the parameter $\\alpha$ is initialized to 1 for language experiments. I could not find if it is initialized differently for vision experiments. Can the authors clarify this?\n- How was the weight decay for $\\alpha,\\beta$ chosen? Was it the default weight decay used for other parameters, or was it tuned?\n- the numbers in Table 2 are exactly identical to the numbers in the DyT paper. Is this because the authors used the same codebase? Or were the numbers taken from that paper?\n- Overall, was there any form of hyperparameter tuning or adaption involved for SeedNorm, or were always the default recipes taken from somewhere (where?), and only the normalization layer was replaced?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I appreciate the large scope of the experiments: The paper presents extensive studies across tasks and scale, spanning LLM pretraining (with normal and MoE models), image classification, and image generation. \n- SeeDNorm either matches or outperforms other normalization variants. The strongest benefits are observed for MoE language models \n- The ablations are interesting and comprehensive. They answer several questions that came up when reading through the paper\n- the paper is well-written and easy to follow", "weaknesses": "- A few experimental details are missing, and it is unclear if default training scripts or modified scripts have been used (more below in questions)\n- For Vision experiments, especially image classification, the benefits of SeeDNorm are less clear. In Table 2, the numbers are either identical or extremely close to baselines (yet the SeedNorm variant is bold - why?). Further, the default SeedNorm variant fails to converge (Table 3), and Multihead SeedNorm is required to match or bring gains over LayerNorm or DyT, with different choices of head dimension often leading to suboptimal performance. Also the drop path rate had to be adjusted for large models “to further prevent overfitting”. This questions if SeeDNorm can be used as an easy drop-in replacement across tasks\n- as for the vision experiments, the numbers for OLMo are sometimes close. Since there are no error bars or runs over multiple seeds (which is partially understandable, given the computational cost of large scale experiments), it is hard to judge the significance of the results\n- In Theorem 3.2 the authors claim that the dot product of two random vectors is inversely proportional to their dimension D. However, they then proceed to show that it is proportional (not inversely) to D. If it were indeed inversely proportional, splitting the vectors to reduce the variance would not make sense\n\nMinor remarks:  \n- typo in line 28, neglligible\n- line 1110: “Our method similarly achieved consistent improvements” ( OLMo2-1B), but this varies across tasks, and on average, the accuracy is almost identical\n- in the abstract in line 18, the connection between a static scaling factor zero-shot setups (as compared to other setups) is not clear to me", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1OUaxq68qd", "reviewer": "Reviewer_NMms", "summary": "This paper proposes a new normalization method called SeeDNorm.\nThe purpose of this normalization method is to address the wide-ranging variations and distribution shifts in input data that a static scaling factor γ cannot handle.\nThe proposed method is simple yet effective.\nIts effectiveness is discussed in the context of large-scale language model pre-training, as well as supervised and unsupervised vision tasks (such as image generation).", "questions": "- I fundamentally believe this is a well-written paper. These are questions to better understand the paper's potential. Please see the weaknesses section for details. \n  - Is the proposed method effective with language models other than OLMo2?\n  - Regarding the possibility of combining it with other normalization techniques, if you already have insights, please share them.\n  - Is the implementation planned to be released?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and easy to understand.\n  - It strikes a good balance between theoretical explanation and experimental validation.\n- Experiments are conducted on both language models and vision tasks, discussing effectiveness across a broad range of domains.\n- References are appropriately cited.", "weaknesses": "- I consider this a well-written paper. None of these points represent fatal flaws as a paper; they are comments to better understand its potential.\n\n- While evaluating results on OLMo2 as a language model, demonstrating effectiveness on other language models would further strengthen the paper's persuasiveness.\n  \n- Regarding vision models, effectiveness has been confirmed for CNN-based models like ConvNext. I consider this a very positive point. However, for CNN-based models, techniques beyond Batch Norm exist, such as GroupNorm, Weight Normalization, and Weight Standardization. Is comparison or combined use with these possible?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)