AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
Authors: Di He
Songjun Tu
Ajay Jaiswal
Li Shen
Ganzhao Yuan
Shiwei Liu
Lu Yin
Venue: NeurIPS 2025 poster ()
Keywords: Weight decay
Large Language Models
Heavy-Tailed Self-Regularization
Primary area: deep_learning
TL;DR: We propose AlphaDecay, a per-module weight decay method guided by heavy-tailedness, improving large language model performance.

Abstract: Weight decay is a standard regularization technique for training large language models (LLMs).  While it is common to assign a uniform decay rate to every layer, this approach overlooks the structural diversity of LLMs and the varying spectral properties across modules. In this paper, we introduce AlphaDecay, a simple yet effective method that adaptively assigns different weight decay strengths to each module of an LLM.  Our approach is guided by Heavy-Tailed Self-Regularization (HT-SR) theory, which analyzes the empirical spectral density (ESD) of weight correlation matrices to quantify “heavy-tailedness.”  Modules exhibiting more pronounced heavy-tailed ESDs, reflecting stronger feature learning, are assigned weaker decay, while modules with lighter-tailed spectra receive stronger decay.  Our method leverages tailored weight decay assignments to balance the module-wise differences in spectral properties, leading to improved performance.  Extensive pre-training tasks with various model sizes from 60M to 1B demonstrate that AlphaDecay achieves better perplexity and generalization than conventional uniform decay and other adaptive decay baselines.  The code is available at https://github.com/hed-ucas/AlphaDecay.

AlphaDecay: Module-wise Weight Decay for Balancing in LLMs
Di He..233 Songjun 2.2.3, Ajay Jaiswal', Li Shen", Ganzhao Yuan", Shiwei Liu", Lu Yin i
Shenzhen Institutes of Advanced Technology. Chinese Academy of Sciences University of Chinese Academy of Sciences
Peng Cheng Laboratory University of Texas Austin Shenzhen Campus of Sun Yat-sen University Shenzhen University of Advanced Technology
University of Oxford UUniversity Surrey 1. yiiinesurrey. ac.
Abstract
Weight decay is standard regularization technique for training large language mod- els (LLMs). While it common to assign a uniform decay rate to every layer, this approach overlooks the structural diversity of LLMs and the varying spectral proper- ties across modules. In this paper, we introduce AlphaDecay, a simple yet effective method that adaptively assigns different weight decay strengths to each module of an LLM. Our approach guided by Heavy-Tailed Self-Regularization (HT-SR) theory, which analyzes the empirical spectral density (ESD) of weight correlation matrices to quantify "heavy-tailedness: Modules exhibiting more pronounced heavy-tailed ESDs, reflecting stronger feature learning, are assigned weaker decay, while modules with lighter-tailed spectra receive stronger decay. Our method lever- ages tailored weight decay assignments balance the module-wise differences in spectral properties, leading to improved performance. Extensive pre-training tasks with various model sizes from 60M IB demonstrate that AlphaDecay achieves better perplexity and generalization than conventional uniform decay and other adaptive decay baselines. The code is available https://github.com///ee ucas/AlphaDeecay
1 Introduction
Large language models (LLMs) have emerged as a core technology artificial intelligence, with extensive applications in chatbots, content generation, code synthesis, and other domains, significantly
enhancing the efficiency and user experience human-computer interaction [5] 28 |3] However,
the formidable capabilities of these models rely massive pre-training datasets and substantial
computational resources, rendering the training process fraught with challenges 44 2] Persistent
research challenges include, but are not limited to, the efficient optimization of ultra-large-scale parameters and the trade-off between training costs and model performance.
Weight decay, one of the most widely used regularization techniques for training well-generalized critically influences the convergence and performance of state-of- the-art machine learning algorithms when properly configured. Extensive prior studies (21 35
deep neural networks 25 42 K
have demonstrated its pivotal role in enhancing model generalization from diverse theoretical and empirical perspectives. Recent work 200 8 further highlights its importance in improving optimizer stability and efficacy during the training of LLMs.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pl__Appha Hill
ESD of weight matrices
More Heavy-tailed
Module Nameratt
att.q
Smaller Pl.pappaa Hill
Alphlizog
att.k
10
Larger
ESD
att.o
10"1
Figenvalnes matris
mip.
Module Name:ML.P gate
gate
Less Heavy-tailes
AlttaDeccy 2,19
Larger
Chiferrde
mlp.
Alpha Hill
10
up
ESD
Smaller
mip.
down
Uniform
AlphaDecay
160
30'
Less balanced
More balanced
bigenvaloes conclation matnis
3.0
mip.gate mulp.u
mlp.down
Alphz
2.5
Unitorm
3.0
Weight Decay
PL. Alpha Hill
2.0-
1.5
1.0
0.5
0.0
1.5
Layer Index
Figure 1: Module-wise Balance and AlphaDecay weight decay schedule. (a) Employing PL fitting to derive module-wise PL PL_Alpppppppppppp___ Alpha_ Hill values (see formula (2). AlphaDecay achieves module-wist balance by increasing the lower values (e.g.. att. Q and att more heavy-tailed) while decreasing the higher values (e.g., MLP components, less heavy-tailed). (b) Given the imbalanced module-wiss PL PL_Alpha Hill of LLaMa-60XM AlphaDecay assigns lower weight decay to modules with lower PL aalpha_Hill
The prevailing approach to weight decay assigns a globally fixed value per epoch across optimiz- ers-including SGD (39,, Adam [18), and their variants [47 -hhrre all model layers share an identical decay coefficient. However, given the scaling parameter counts and architectural complexity of modern LLMs, such uniform weight decay scheme fails to capture their intricate structural properties, making this conventional practice increasingly suboptimal. Notably, recent work has begun investigating dynamic weight decay adaptation [16 30
address this limitation. 12 observes that fxee-hhypepprrmmter weight decay fails to balance robustness and accuracy in adversarial training, causing robust overfitting. They propose Adaptive Weight Decay (AWD) to dynamically adjust decay strength via classification and regularization loss gradients, automatically enhancing robustness and adversarial performance without extra data.
12 42
Notably, prior studies on dynamic weight decay adaptation were exclusively designed for architectures like ResNet 8/34/50 R uniform decay values across all layers at each timestep) while maintaining layer-wise uniformity. This approach is reasonable for cerameter-effciien structurally simple models (e.g., ResNets) where inter-layer feature distinctions are less pronounced. However,
[14, VGG 34 and DenseNet [15), employing time-wise modulation (i.e.,
Does there exist better weight decay configuration LLMs?
Three reasons behoove us to pose the above research question: First, the prevailing consensus holds
that certain transformer components exhibit greater functional importance than others 40 4 44 26 necessitating differentiated weight decay treatment. Second, weight decay manifests fundamentally distinct roles in over-trained regimes (e.g., ResNets) versus under-trained regimes (c.g., LLMs) (8]

---

Most notably, existing research demonstrates that improper weight decay configuration for LLMs Our main contributions are as follows:
may adversely affect model performance 19 33
We identify substantial variation in the spectral properties of module-wise ESD (see figure 2 and show that these inconsistencies are a core reason for degraded model performance, as evidenced by figure 4
We propose a module-wise weight decay scheduling strategy AlphaDecay to ensure spectral alignment across modules (see figure thereby enforcing consistency in spectral properties and achieving improved training performance (see figure 3
Extensive experiments spanning models from 60M to IB parameters show that the proposed approach, AlphaDecay, consistently outperforms the Uniform baseline as well as adaptive methods such AWD 12 and AdaDecay [0] (see table 2) These results highlight the critical role of module-wise balance in achieving ofte--of-the-ar performance in LLMs.
Overall, our research provides unrecognized perspective optimizer, revealing the critical yet overlooked role of module-wise weight decay in LLM training. This novel insight can be readily applied to all state-of-the-ar optimizers and training methods, effectively enhancing their performance without structural modifications.
Related Work
Weight decay in LLM training. Weight decay is a widely adopted technique for training deep networks, spanning applications from image classification to LLMs [21]. In the context of GPT-3 training. [5] recommended incorporating weight decay primarily for its mild regularization benefits.
20 showed that weight decay promotes optimizer equilibrium in scale-invariant systems. Recent studies have provided deeper insights into weight decay's role in LLM training. [38] (7] challenged the conventional view of weight decay's generalization benefits for LLMs, and instead highlighting its critical function in reducing training loss and enhancing stability during under-training through the
lens of Effective Learning Rate. Building on these findings, B 19 established a connection between l2 regularization and spectral norms, discovering that weight decay induces low-rank attention layers. Their work further showed that employing different weight decay values for attention and MLP modules, carefully tuned via grid search, can significantly improve training outcomes. Our work presents the first formal analysis of non-uniform module-wise wise weight decay in LLM training. demonstrating its effectiveness through comprehensive empirical validation.
Dynamic weight decay. While uniform weight decay is commonly used for model training, a line of work employs gradnorm to adaptively determine weight decay settings. [T6] analyzed gradient descent with weight decay, finding that backpropagated gradients scale with upstream weights while weight decay scales with each layer's 's own weights. This mismatch in scaling causes layer-wise overfitting or underfitting, leading them propose using the gradient-to-decay magnitude ratio as a layer-wise coefficient. (30) enhanced this approach by normalizing gradients and applying a scaled sigmoid to compute the coefficient. Similarly, [12 used the ratio of gradient norms to parameter norms. 42 showed weight decay amplifies late-stage gradient norms, harming convergence. Their solution, AdamS, penalizes large gradients and outperforms both Adam and AdamW. Another line of research 10, 41 revealed that weight decay induces low-rank layer structures. [9] further showed that applying distinct weight decay values to attention and MLP modules, meticulously tuned via grid search, can substantially enhance training outcomes. Building upon these foundations, our work advances this direction by introducing the first weight decay scheduling framework for LLMs.
Heavy-tail rellaregulaiization.. HT-SR Theory examines the ESD of weight matrices and identifies its relationship with training quality based on principles from statistical physics and random matrix theory (7). HT-SR Theory posits that
neural networks exhibit strong correlations in their weights, manifesting as heavy-tailed structures in the ESD of each layer's weight matrics [27 29).
Recently, HT-SR has been applied to model selection 27 29 4 module wise adaptive training [4)] and LLM pruning (26), demonstrating its efficacy in estimating model and layer quality. However, no prior work has explored HT-SR theory in the context of weight decay configuration. Our work draws inspiration from HT-SR theory and introduces a novel technique that leverages ESD structures to guide weight decay settings.
3

---

Module Name: 4
Module Name:att&
Medule Nameratt V
Medule Nameatt
1-9
1.0
Nomallzed Siggiar valses
Nomnalzed Sitpplar valure
Nommalized Sitgolar values
Nomalizzd Siigelar values
0.4
6.4
0.5
6.-
0.3
6.4
0.4
64
04
6.2
6.-
0.2
0.2
64
64
64
600
Rank
Rank
Rank
Kank
Name:mlp.gux
Name upp.up
Module Namezmlp.de dows
1.2
1.2
1.2
Normaliznd Singniar valuns
Normalized Stngulaar valuns
0.4
8.1
Normaltor Singniaar
6.6
6-5
6.6
64-
6.4
6.4
62-
0.2
6.-
Figure Visualization of singular values from weight matrices in each layer of the pretrained -LaMa-2-133-4 hf model. For all 40 transformer layers, the plots show the sorted distribution of 5120 singular values per layer.
3 Methodology
In this section, we first present the rationale motivating our study, emphasizing the heavy-tailed singular value spectra exhibited by different modules of LLMs. We then revisit HT-SR theory and introduce key HT-SR metrics that support our analysis. Finally, we examine the AlphaDecay algorithm, which leverages "shape metrics" derived from HT-SR theory and exhibits significant improvements in LLM pretraining tasks.
3.1 Rationale
Different modules in LLMs exhibit diverse spectral properties, particularly in the distribution of their singular values. Figure 2 visualize the normalized singular value spectra of the weight matrices for each module type (att. q. att.k, att. v, att. o, mlp.gate, mlp. up, mlp.down) across all 40 transformer layers in the pretrained (LaMa-2-133-hh model. Notably, substantial variability is observed the heavy-tailedness
the singular value distributions: the attention-relater modules (att.a q and att.k, k) consistently show heavier tails, while the MLP modules (mlp. gate, mlp.up, mlp.down) exhibit lighter tails.
This phenomenon has been extensively studied within heavy-tailed random matrix theory. Specifically, heavier tails in the singular value spectra reflect greater anisotropy. with much of the module's representational power concentrated in few leading principal components pronounced in attention-related modules (att. q. att In contrast, the lighter-tailed spectra observed in MLP modules (mlp. gate, mlp.up up, mlp. down) exhibit a more uniform distribution across components. These observations suggest that different modules may benefit from tailored regularization strengths to achieve optimal performance, as attention modules could be more disrupted by excessive regularization, while MLP modules may tolerate stronger regularization.
feature especially
3.2 HT-SR Theory
The HT-SR theory provides principled framework for analyzing the empirical spectral distribution (ESD) of neural network weight matrices. Empirical evidence suggests that well-trained models exhibit more pronounced heavy-tailed ESDs, which reflect higher training quality. Building on this theoretical foundation, our method leverages the HT-SR metric to quantify spectral tail heaviness, assigning lower weight decay to heavily-tailed modules (e.g., att. q, att. and higher weight decay to less heavy-tailed ones (e.g., MI.P components), thereby aligning with spectral characteristics to potentially improve generalization and model performance (see figure I.. The degree of heavy-

---

Nameatt
Module
Module
Nameann
Ca 80)
Ca 40
Cellee c4
Ayladlwwyie 170
Alteeeo 199g
Aphachey
DSD
LSD
32
1000
Figenvalane comeiation mainis
Eigevalazee comiriin maint
Bigenvalane comietive maint
Higenvalzze combuttivn
W
Module Name MLP Par gade
33
Modale Name: M.P
18
Module Nume: ML.P down
10-
18-1
10-1
LSD
LSD
ESD
10-
10-1
12-9
10-
17
10
16-1
121
Elgesvelaes of ecconelaiin mobil
Eigeavtlues of comeliiios motis
Eiganallas ooourelaion rettii
Figure 3: Comparison of ESD distributions across modules of LLaMa- 135M under different training methods (AlphaDecay: Perplexity=22.55 vs. Uniform: Perplexity=23.1 13). Attention-relate mod- ules (e.g., att. q. att. exhibit notably heavier spectral tails contrast MLP-associated modules. Our method systematically balances the heavy-tailed properties across modules by appropriately configuring module-wise weight decay, thereby enhancing overall model performance.
tailedness is quantitatively assessed by fitting a power law (PL) to the ESD, using the resulting PL exponent (a) metric.
Given network with L modules and weight matrices {Wi) of shape n m (n S m), compute the ESD by obtaining the eigenvalues of the correlation matrix X; W W, for each module. The power law fit for the ESD takes the form:
p()) x A Amin
Amax
where p(A) denotes the density of cigenvalues À within the specified range. The PL exponent, a, serves as proxy for the degree of heavy-tailedness.
To estimate use the Hill estimator 46 24 For given module's eigenvalues (A: (sorted
in ascending order), the Hill estimator given by:
k
PL_ P1_Alpha Hill = 1
Σ In An-i41
X,
where & controls the lower cutoff for PL fitting. In our experiments, fix 3, i.e., we estimate the slope using the largest half of the eigenvalues.
PL Alpha_ Hill is key spectral descriptor for analyzing model performance. Related works 44, 24) suggest that lower PL_Alpha __ill values indicate "overtrained" layers (compared to other layers in the model), while higher values indicate "undertrained" layers. An important conclusion is that a more uniform distribution of PL Ao_Alpha_HHie nacha_Hllll across layers reflects more balanced training quality, leading to better overall model quality. While these findings highlight the importance of layer-wise training balance, our work emphasizes a complementary perspective:
Does module-wise balance matter for model performance?
We empirically demonstrate that promoting uniformity in PL PL_Alpha_ Hill across modules (c.g.. attention and MLP components) can further enhance overall model quality (see figure
3.3 AlphaDecay
Building the observed spectral diversity across modules, we introduce AlphaDecay, a simple yet effective module-wise weight decay scheduling algorithm. AlphaDecay first calculates the PL Alpha_ Hill values for all modules, and then assign larger weight decay modules with higher
5

---

23.6
Pl. Alpha Hill Mcan
st.vio - mip
Perplexity ())
23.2
2.0
1.5
3
4
10
10
Weight Decay
Weight Decay
AlphaDecay
Uniform
Figure Comparison of perplexity and module-wise PL_ PL_Alpha Hill values LLaMa-1 133M under varying weight decay settings; For each group, att. ./k shows the mean PL Alphappa______ Hill of att. and att k; v/o shows the mean for att. v and att o, mlp the mean of aalp-gate, mlp. ulp. and mlp. mlp.down. Shaded areas indicate the range between the maximum and minimum values within each group.
PL Al__appa__ Hill values, while assigning smaller weight decay those with lower PL PL_hlpha __ill values. This strategy is designed to promote module-wise PL Alpha Hill balance, thus leading to better overall model performance. We provide the details of AlphaDecay in Algorithm [ The assignment function is given by:
f((1) = 17. 0f at
a. a-min (32 81) 81)
(3)
where 77 is the initial weight decay, and (s1, 82) define the range of scaling ratios applied to of the PL_Alpha Hill value of module i step t, while at and at are the minimum and maximum PL Alpha Hill values among all modules at step I. Formula (3) guarantees that the adjusted weight decay, fi)),, remains within [$17), 827) as a scaled variant of r).
We compare AlphaDecay with the Uniform baseline under varying weight decay settings, and present the results of perplexity and module-wise PL_ S__Alpha_HHi __ill values, shown in figure Notably, AlphaDecay assigns module-wise weight decay values in accordance with the im- balance observed in module-wise PL_ PL_Alpha_ Hill metrics (see Figure [ and reallocates these weight decays every 500 update steps. This dynamic assignment adaptively moderates the module- wise PL _L_Alpha_HHin Hill imbalance present in the Uniform baseline by decreasing the elevated PL PL_Alpha_Hii Hill values in MLP modules and increasing the lower values in attention-relate modules (i.e., att. v/o and att att.q/k) q/k). As a result, our method achieves consistently lower and more stable perplexity across different weight decay configurations, thereby improving model robustness and overall performance.
4
Algorithm 1: AlphaDecay
Require: initial weight decay 7,, number of training steps T', interval i of using AlphaDecay,
minimum and maximum scaling ratio $1, $2, and az refers to leh module's
PL Alpha Hill at update step t
for t < 0 to do mod(t, i) then
Compute of for all modules using the Hill estimator; aj
Leverage all af and adopt fe(i)= = 17 (2r0n ogm (52 -1) $1) assign module-wise
weight decay between 817) and 8277;
4 Empirical results
In this section, we begin by presenting the complete experimental setup (Section 4.1 followed by comparison between AlphaDecay and several baselines (Section 4.2). Finally, we analyze the impact

---

Table I: Hyperparameters used in pre-training experiments.
Model Size
Tokens
Weight Decay
($1, 82)
60M
0.001
IB
le-5, 5e-6, le-6
(0.67.3), (0.67.5), (0.67.5)
135M
0.001
1e-5, 5c-6, le-6
(0.67.3). (0.67.5), (0.67.5)
350M
0.001
6B
Ic-5, 5c-6, lo-6
(0.67.3), (0.67.5), (0.67.5)
IB
0.0006
8.9B
1e-5, 5e-6, le-6
(0.67.3), (0.67.5). (0.67.5)
Table 2: (Main result), Comparison with various weight decay scheduling strategies on pre-training various sizes of LLaMa models C4 dataset. Validation perplexity (4) is reported. All baselines carefully tuned. 'WD=0' indicates that weight decay is disabled during model training.
LLaMa-60M
L.aMM-135MM
LLaMa-35OM
LLaMa-1B
Weight Decuy
lo-5
5c-6
le-6
le-5
5c-6
le-6
le-5
5c-6
le-6
le-5
5c-6
WD=0
33.23
24.60
18.62
16.11
Unifarm
32.39
32.56
33.03
22.99
23.14
24.14
17.12
16.74
17.50
15.36
14,66
15.03
12
33.78
33.74
33.74
24.25
24.45
24.53
18.32
18.55
18.79
16.03
16.22
16.38
Adadecay 30
32.24
32.52
33.03
23.20
23.08
23,96
17.42
17.91
17.23
18.14
15.35
AlphaDecay
31.56
31.58
32.61
22.76
22.55
23.49
17.00
16.66
16.88
15.13
14.55
14.63
of weight decay assignment functions, HT-SR module- metrics, PL fitting methods, and PL fitting time gaps through ablation studies (Section 4.4)
4.1 Experimental setup
Models and Datasets. We conduct systematic evaluation of AlphaDecay across LLaMa-based architectures spanning four model scales (60M, 135M, 350M, and 1B parameters). All experiments employ the C4 dataset (31), rigorously processed subset of Common Crawl widely adopted for language model pretraining. Our experimental design incorporates two key components: (1) non-repeating data regime with sufficient tokens for convergence, and (2) standardized preprocessing pipelines across all model scales. This multi-scale approach facilitates systematic comparison of model behaviors across different capacity regimes, while minimizing potential confounding factors in the analysis.
The detailed hyperparameter settings for all model sizes summarized in Table I All models are trained with Adam optimizer (gradient clipping at 1.0) and a cosine learning rate schedule, with 10% of the training tokens used for learning rate warmup. We conduct grid search over learning rates {0.01, 0.001, 0.0001) and report the best configuration for each scale in the table. Weight decay settings and the corresponding (#1; 82) parameter settings also detailed in the table. AlphaDecay is performed every 500 update steps throughout all experiments.
Hyperparameters.
4.2 LLM Pre-training
Table presents the main results of our study, where we evaluate the effectiveness of different weight decay scheduling strategies on the pre-training of LLaMa models with varying parameter scales (60M, 135M, 350M, and 1B) on the C4 dataset. For each model size, we conduct comprehensive experiments across three commonly used weight decay values (le-5, 5e-6, and le-6). Our proposed method is compared against several baselines, including the commonly used Uniform scheduling, adaptive global weight decay (AWD) [12) and adaptive mor-module weight decay (Adadecay) (30 All bascline methods are carefully tuned for fair comparison.
Observations. ① Weight Decay is Beneficial for Model Performance. Comparing 'WD=0' (i.c., no weight decay) and Uniform across all model sizes, applying weight decay consistently leads to substantial reductions in validation perplexity. This provides empirical support for the importance and effectiveness of weight decay in LLM pre- mre-trainnng. @ Superior and Consistent Gains Across All Weight Decay Settings. AlphaDecay consistently yields the lowest validation perplexity across all evaluated weight decay settings (le-5, 5e-6, le-6) and model sizes, surpassing both the Uniform baseline and the adaptive weight decay methods (AWD and Adadecay). This consistent superiority across various regularization strengths demonstrates the robustness of our approach and underscores its potential applicability in LLM pre-training, @ Scalability to Larger Models. The performance
7

---

improvements achieved by AlphaDecay are consistently observed from the smallest (60M) to the largest (IB) parameters, indicating the scalability and generality of our approach.
Furthermore, our experiments reveal that existing adaptive weight decay methods, originally designed for architectures without attention components, such AWD and Adadecay, do not yield optimal results for LLMs. This may be attributed to their lack of consideration for the distinct characteristics and optimization requirements of attention and MLP modules within transformer architectures. In contrast, our approach is, to the best of our knowledge, the first to demonstrate that tailored weight decay scheduling strategy can consistently enhance LLM training by explicitly accounting for the heterogeneous characteristics of different modules.
4.3 Downstream tasks & architectures
This section introduces our evaluation of downstream gains across zero-shot commonsense reasoning and fine-tuning tasks, with results on additional model architecture and image classification task (GPT-nano/C4 perplexity; ViT-tiny/lmagNeet-KK Top-1):
Zero-shot Results. We evaluate the pretrained LLaMa-1B checkpoints from Table 2 seven zero-shot commonsense reasoning tasks using on-eval-harnes with its default prompts. shown in 3 AlphaDecay delivers the best results on 6 of benchmarks, indicating that its pretraining gains transfer well to downstream reasoning tasks and support broad applicability.
Table (Zero-shot results of commonseuss-rrassoinng tasks), Zero-shot evaluation results (f) on seven commonsense reasoning benchmarks using the LLaMa- 1B model pretrained with different methods.
Method
ARC-c
ARC-e
PIQA
Hellaswag
OBQA
Winogrande
BOOLQ
Avg.
Uniform 20.22
46.72
67.68
32.94
18.8
49.41
54.74
41,50
AdaDecay
19.20
46.72
66.97
32.96
18.0
51.54
56.36
41,68
AWD
19.18
46.34
66.65
31.37
18.0
51.07
57.25
41.41
AlphaDecay 20.90
48.86
68.44
34.16
19.80
50.59
60.70
43.35
Finetuning Results. We evaluate all baselines on GLUE finetuning tasks with roberta-base, As re- ported in Table AlphaDecay attains the top result on of 7 tasks. This evidences that AlphaDecay effective not only during pretraining but also transfers well to finetuning settings.
Table 4:
(Finetuning tasks). Finetuning results (1) on eight benchmarks from the GLUE dataset using roberta-base with different methods.
Method
cola
mnli
mrpc
qnli
rte
sst2
stsb
Avg.
Uniform
59.73
86.78
87.01
92.59
89.97
70.11
93.69
90.78
83.83
AdaDecay
60.45
87.23
88.19
92.62
89.95
73.36
93.73
90.9
84.55
60.72
87.44
89.53
92.58
90.08
72.27
93.72
90.9
84.66
AlphaDecay 62.82
87.11
89.61
92.73
90.12
73.86
93.77
90.91
85.12
Across architectures and datasets. We evaluate all methods on GPT-nano/C4 (perplexity) and ViT- liny/ImageNet-IK (Top-1 accuracy). As shown in Table our method attains the lowest perplexity on GPT-nano/CC and the highest Top-1 accuracy on ViT-tny/ImageNet-Ill IKK outperforming Uni f form. AWD, and AdaDecay. These results demonstrate consistent effectiveness across different architectures and datasets, and strong generalization beyond a single setting.
Table 5: (Across architectures and datasets). Results on GPT-nano/CA (Perplexity) and ViT- tiny/ImageNet-IK KK (Top-1) 1) with different methods.
Backbone Dataset
Metric
Uniform
AWD
AdaDecay
AlphaDecay
GP'T-nano/ C4
PPL())
27.56
27.64 64.98%
27.68 66.26%
27.37
ViT-tiny/ ImageNet-1K
Top-1(t)
66.41%
67.73%

---

Uniform
Lineur
Sqri
Log2
Sigmoid-like
Perplostt
324
$2.
-0.06
-0.00
-0.19
Paplecity
124
32.0
-0.04
-0.06
-0.39
Paplasity
338
-0.49
10.05
-0.01
-0.07
31.6
-0.82
00.88
33
51.0
115
(a) Weight Decay 1c-5
(b) Weight Decay = 5e-6
(c) Weight Decay le-6
Figure 5: (Varying weight decay assignment functions). Results of using different weight decay assignment functions under different weight decay settings. All experiments conducted L.MMa-60M. The value on the top of each bar indicates the difference from the leftmost bar each plot and the same processing applied in Figure 6 Figures and Figures g
Uniform
PL Alpha Hill
GradNorm
FrobeniusNorm
SpectralNorm
+0.22
+0.36
23.10
-0.13
2438
1128
-0.25
Paplesity
Perploxity
22.00
-0.23
-0.59
-0.34
22.50
-0.55
.0.54
-0.65
23.48
(a) Weight Decay = le-5
(b) Weight Decay = 5c-6
(c) Weight Decay = 1c-6
Figure 6: (Varying HT-SR metries). Comparing PL Alpha Hill with multiple HT-SR metrics
under different weight decay settings. All experiments are conducted on 11LaMa-135M)
Goodness-of-fe
Fix-finger
Median
Time (sec)
20.03
-0.03
+0.3
Perleaiy
Time (ses)
Perplesity
-0.35
Time (soc)
Peplesity
-0.48
22.9
22.6
224
215
(a) Weight Decay = 1e-5
(b) Weight Decay = 5e-6
(c) Weight Decay 1e-6
Figure 7: (Varying PL fitting methods). Comparison of various PL fitting methods. The bar plot and left y-axis represent perplexity (lower the better), while the line piot and right y-axis indicate the time taken for AlphaDecay once (in seconds, lower the better). The computation times are averaged over all PL fitting operations throughout the model training process. All experiments conducted using LLaMa-13500
4.4 Analysis
Varying Weight Decay assignment functions. We examine the performance of PL Alpha Hill with different weight decay assignment functions, which determine the allocation ratios of weight decay across different modules. Figure presents the results obtained by different assignment functions: Uniform. Linear, Sqrt, Log2, and Si Sigmoid-1ie Among these, Linear achieves the best results across all weight decay settings, showing a notable advantage over other methods.
Varying HT-SR metrics. To investigate the impact of various HT-SR metrics regulating weight decay during model training, we conducted ablation studies comparing these metrics. While prior work has primarily utilized GradNorm [30; 20; 42 and FrobeniusNorm [12 [6] as indicators for adjusting weight decay, our study further evaluates additional metries, including PL_ PI_Alpha _illl and SpectralNorm, under the same experimental settings. Results in Fig that most HT-SR metrics outperform the uniform baseline, while PL Pl_alpha Hill achieves the lowest perplexity (lower the better) among all evaluated methods.

---

Uniform
500
GAP-100
G0P-50
GAP-1
Time (h)
25.0
23.6
24.8
Papleculy
22.1
-0.23
-0.21
-0.24
Time (b)
Paplecitt
-0.59
.0.60
-0
56
Time (b)
Perplecity
.0.65
-0.66
-0.67
-0.65
-00
2244
215
(a) Weight Decay 1c-5
(b) Weight Decay 5
(c) Weight Decay 1c-6
Figure 8: (Varying PL fitting gaps). We conduct PL. fitring at varying specified gaps of training steps. The bar plot and left y-axis represent perplexity (lower the better), while the line plot and right y-axis indicate the time required for training completion (lower the better). The computation times reflect the NVIDIA A100 hours utilized for completing model training. All experiments are conducted using LLaMa-133M
Varying PL fitting methods. In our proposed framework, the HT-SR metric Pl_Alpha__ _illl employed to guide the projection-based adjustment of weight decay during training. Since PL PL_Alpha_ Hill is derived through PL fitting, and the choice of fitting method can influence both computational efficiency and the final training effectiveness, we conduct an ablation study to sys- tematically assess its impact. Figure 7 presents presents a comparative analysís of three PL. fitting methods- Goodness-of fft [ 22 6 Fix-finger [43], and Median [46] -across multiple weight decay values. Across all settings, Median not only ensures optimal training performance but also notably decreases computation time compared to the other approaches, making it the preferred choice for PL fitting within our method.
Varying PL fitting gaps. To further analyze the stability of our proposed approach, we investigate the impact of varying update gaps for weight decay adjustments during training. It is computationally inefficient to update weight decay at every training step. Thus, we explore the performance of our method by updating weight decay at different training step gaps: 1, 50, 100, and 500 steps. Figure shows that our approach achieves stability across all gap settings. Notably, across all weight decay settings, using training step intervals from to 500 consistently outperforms the Uniform setting, including when the interval is as large as 500 training steps. This demonstrates the robustness of our method to update frequency. Therefore, select gap of 500 in all experiments because it provides substantial computational savings while maintaining stable and competitive model performance across various settings.
5
Conclusion
Weight decay is standard regularization technique in deep learning, typically implemented with single decay rate for all parameters. However, this uniform application lacks theoretical justification and may not be optimal. We present a systematic study of module-wise weight decay scheduling, an overlooked but important aspect of model regularization. The proposed AlphaDecay framework provides principled approach to module-specific decay rates based on HT-SR theory. Through extensive experiments, we demonstrate that AlphaDecay consistently improves model performance across different pretraining scales. To our knowledge, this is the first work to formally investigate and establish a framework for module-level weight decay scheduling in LLMs. Our results indicate that weight decay scheduling represents promising direction for future research. While this work represents an initial exploration, it opens new possibilities for understanding and improving weight decay in LLMs.
Limitations. While our study offers initial insights into module-wise weight decay scheduling, several limitations remain. First, evaluation Mixture-of-Fxperts (MoE) models is left for future work. Second, interactions with other regularization and optimization techniques are yet to be systematically assessed. Addressing these issues represents valuable directions for future research.
10

---

References
[1] Jeff Alstott, Ed Bullmore, and Dietmar Plenz. powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, 911)e88577 2014.
[2] Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your Ilm fully utilize the context. Advances in Neural Information Processing Systems, 37:62160-62185 2024.
[3] Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Low-rank bottleneck in multi-head attention models. In International conference on machine learning, pages 864-873. PMLR, 2020.
[4] Srinadh Bhojanapalli, Ayan Chakrabarti, Andreas Veit, Michal Lukasik, Himanshu Jain, Freder- ick Liu, Yin-Wen Chang, and Sanjiv Kumar. Leveraging redundancy in attention with reuse transformers. arXiv preprint arXiv:21 10.06821, 2021.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, (164):661-703, 2009.
[7] Romain Couillet and Zhenyu Liao. Random matrix methods for machine learning. Cambridge University Press, 2022.
[8] Francesco D' Angelo, Maksym Andriushchenko, Aditya Vardhan Varre, and Nicolas Flammar- ion. Why do we need weight decay in modern deep learning? Advances in Neural Information Processing Systems, 37:23191-23222 2024.
[9] Adriana Bermandez-Lopez, Shiwei Liu, Lu Yin, Stavros Petridis, and Maja Pantic, Full-rank no more: Low-rank weight training for modern speech recognition models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[10] Tomer Galanti, Zachary Siegel, Aparna Gupte, and Tomaso Poggio. Characterizing the implicit bias of regularized sgd in rank minimization. arXiv e-prints, pages arXiv-2206, 2022.
[11] Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, al. Openagi: When Ilm meets domain experts. Advances in Neural Information Processing Systems, 36:5539-5568, 2023.
[12] Mohammad Amin Ghiasi, Ali Shafahi, and Reza Ardekani. Improving robustness with adaptive weight decay. Advances in Neural Information Processing Systems, 36:79067-79080, 2023.
[13] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pages 1225-1234. PMLR, 2016.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700 2017.
[16] Masato Ishii and Atsushi Sato. Layer-wise weight decay for deep neural networks. In Pacific- Rim Symposium on Image and Video Technology, pages 276-289. Springer, 2017.
[17] Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. arXiv preprint arXiv:2407.11233 2024.

---

[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[19] Seijin Kobayashi, Yassir Akram, and Johannes Von Oswald. Weight decay induces low-rank attention layers. Advances in Neural Information Processing Systems, 17:4481-45111 2024.
[20] Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks. arXiv preprint arXiv:2305. 2023.
[21] Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
[22] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High- rank training through low-rank updates. arXiv preprint 2XXv:2307.05695. 2023.
[23] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy, Do we actually need dense over parameterization? ? in-time over- preeaarraettiinntoonnn in sparse training. In International Conference on Machine Learning, pages 6989-7000. PMLR, 2021.
[24] Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. Model balancing helps low-data training and fine-tuning, arXiv preprint arXiv:2410.12178, 2024.
[25] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rX1v:1711.0510,, 2017.
[26] Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. Advances in Neural Information Processing Systems, 37: 9117-9152, 2024.
[27] Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference Machine Learning, pages 4284 4293. PMLR, 2019.
[28] Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset? Advances in Neural Information Processing Systems, 37:124069 124092, 2024.
[29] Charles H Martin, Tongsu Peng. and Michael W Mahoney. Predicting trends in the quality of state-of-the-ar neural networks without access to training or testing data, Nature Communica- tions, 12(1):4122, 2021.
[30] Kensuke Nakamura and Byung- Woo Hong. Adaptive weight decay for deep neural networks. IEEE Access, 7:1 1188555118866555 2019.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[32] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithmss Cambridge university press, 2014.
[33] Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558, 2023.
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1409.1556, 2014.
[35] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19 (70):1-57, 2018.
[36] Sebastian U Stich, Jean- Man-Baatiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Advances in neural information processing systems, 31, 2018.
12

---

[37] Songjun Tu, Jingbo Sun, Qichao Zhang, Xiangyuan Lan, and Dongbin Zhao. Online preference- based reinforcement learning with self self-augmenttdd feedback from large language model. arXiv preprini arXiv:2412. 16878, 2024.
[38] Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint 17X5v0170.005550 2017.
[39] Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. Journal of Machine Learning Research, 22(213):1-50, 2021.
[40] Wenxuan Wang and Zhaopeng Tu. Rethinking the value of transformer components. arXiv preprint 10xiv:2011.0380000 2020.
[41] Zihan Wang and Arthur Jacot. Implicit bias of sgd in (_ linear dnns: One-way jumps from high to low rank. arXiv preprint arXiv:2305. 16038, 2023.
[42] Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, and Masashi Sugiyama. On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-non perspective. Advances in Neural Information Processing Systems,
1228, 2023.
[43] Yaoqing Yang. Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 301 3011-3021, 2023.
[44] Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang. et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms high sparsity. arXiv preprint 100.5:2310.05115, 2023.
[45] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficiee I1m training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
[46] Yefan Zhou, Tianyu Pang, Keqin Liu, Michael W Mahoney, Yaoqing Yang, et al. Temperature balancing, layer-wise weight analysis, and neural network training. Advances in Neural Information Processing Systems, 36:63542-63572, 2023.
[47] Juntang Zhuang. Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelicf optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806 2020.
13

---

Appendix
A Details of Experiments
A.1 Architecture
To ensure reproducibility and consistency with prior research (145 22
we adopt the LLaMa
architectural specifications and pre-training hyperparameters as detailed in Table 6 All model variants are trained with uniform maximum sequence length of 256, a batch size of 512, and an aggregate of 13K tokens per batch.
Table 6: Hyperparameters of LLaMa models used in this paper.
Params
Hidden
Intermediate
Heads
Layers
Steps
Data amount
LR
Batch Size
60M
512
1376
8
8
10K
1B
1 X 10-3
512
135M
768
2048
12
12
20K
2B
1 X 10
512
350M
1024
2736
16
24
60K
x 10-3
512
2048
5461
32
24
90K
x 10
A.2
Hyperparameter settings for reproducing our figures
We report all hyperparameters and all numerical values of experimental results shown in the main paper. First, Table reports the details of experiments shown in Figure 3 and Figure Then, Table 8 Table 9 Table I0 and Table II respectively report the details of the experiments shown in Figure
Figure Figure and Figure
Table Parameter settings of the experiment reported in Section 3.3 Figure and Figure
Method
Model Size
Weight Decay
Perplexity
Scaling Ratio
Method
Model Size
Weight Decay
Perplexity
Scaling Ratio
(8),
(5)
33.306
3c-6
4e
33.219
4e-6
23,437
33.131
5c-6
Unifforn
LLaMa
6e-6
33.157
Uniform
LLaMa 135M
6e-6
23,161
8e-6
33.028
8c-6
23.109
9c-6
33,008
9c-6
23.083
le-5
33.002
lo-5
23.111
36-6
32.557
(0.67.5)
36-6
22.895
(0.67.5)
4e-6
32,364
(0.67.5)
4e-6
22.812
(0.67.5)
Se-6
32.171
(0.67.5)
Se
22.730
(0.67.5)
AlphaDecay
LLLMMa 60M
6e-6
32.144
(0.67.5)
AlphaDecay
L.aMa 135M
6e-6
22,770
(0.67.5)
7e-6
32.122
(0.67.5)
7e
22,772
(0.67.3)
32,077
(0.67.5)
8e-6
22.830
(0.67.3)
9c-6
32.097
(0.67.5)
9c-6
22,885
(0.67.3)
(0.67.5)
(0.67.3)
Table 8: Parameter settings of the experiment reported in Section 4.4 Figure conducted on LLaMa-60M.
All experiments are
Scaling Ratio ($1,82)
Weight Decay
Uniform
Linear
Sqrt
Log2
Sigmoid-like
le-5
32.386
31.565
32.326
32.301
32.201
(0.67.3)
5e-6
32.562
31.582
32.517
32,501
32.171
(0.67.5)
le-6
33.028
32.537
33.074
33.033
33.095
(0.67,5)
Table 9: Parameter settings of the experiment reported in Section 4.4 conducted on LLMMa-135M.
Figure
All experiments
Scaling Ratio (di, 6z)
Weight Decay
Uniform
PI_Alphaa_r Hill
Gradilorm
FrobeniusNorm
SpectralNorm
22.993
22.763
22.855
23.265
23.348
(0.67.3)
5e-6
23.138
22,551
22.714
22.91
23.006
(0.67.5)
lc-6
24.142
23.488
23.801
23.596
23.601
(0.67.5)
14

---

Table 10: Parameter settings of the experiment reported in Section 4.4 Figure The computation times are averaged over all PL fitting operations throughout the model training process.
Goodness-of-it fit
Fix finger
Median
Model Size
Weight Decay
Perplexity
Computation Time (sec)
Perplexity
Computation Time (sec)
Perplexity
Computation Time (sec)
Scaling Ratio ($1
le-5
32.166
8.73+0.30
32.231
7.90+0.33
31,628
1.67400.1
(0.67.3)
LLaMa
5e-6
32.436
10.62:0.00 04
32.381
9.22:0.47
31.614
1.69#0.01
(0.67.5)
-60M
Ie-6
32.993
8.28
33,059
7.8011.67
32,703
1.66:10.01
(0.67.5)
le-5
22.9377
24.86-+00.11
23.004
21.53-+0.83
23.004
4.674-0.02
(0.67.3)
LLaMa -135M
22.937
24.43+0.11
23.090
22,00++0
22.588
4.68+00.5
(0.67.5)
Ic-6
23.924
24.6440.13
24.058
21.90:1..00
23.448
4.63#0.01
(0.67.5
Table 11: Parameter settings of the experiment reported Section 4.4 Figure times reflect the NVIDIA A100 hours utilized for completing model training.
The computation
Scaling Ratio (81,
Model Size
Weight Decay
Uniform
GAP=500
GAP=250
GAP=100
GAP=50
GAP=1
le-5
32.386
31.614
31.628
31.555
31.618
31,594
(0.67.3)
LLaMa
5e-6
32.562
31.628
31.633
31.673
31.717
31.712
(0.67.5)
-60M
le-6
33.029
32.703
32.718
32.754
32.663
32.769
(0.67.5)
Computation Time
1.4h
1.4h
1.4h
1.5h
1.6h
9.3h
le-5
22.994
22.763
22.756
22.779
22.758
22,809
(0.67,3)
LLaMa
5e-6
23.138
22.551
22.537
22.569
22.539
22.581
(0.67.5)
-135M
le-6
24.142
23.488
23.477
23.479
23.468
23.488
(0.67,5)
Computation Time
5.6h
5.7h
5.9h
6.3h
7.1h
74.5h
A.3
Assignment Function Formulas
For AlphaDecay, we selected the linear interpolation (fomula 3 for weight decay assignment
function fi, based on its superior performance in our ablation study. We provide the remaining assignment functions here:
Sqrt: (i(1) = 1) シン 0
Log2: f((1) = " " (ox)
log)(ar))
sigmoid-like. [t(i) = T-exp(--app with 03 (1951 uf)/of
Here, y7 denotes the initial weight decay, of is PL Al__lpha_a Hipha_Hillll_ Hill of the module at step f. and L is the total number of model modules. 1u is the mean of gradient norms Igğll the layer I; of the std of gradient norms |gا| in the layer 1; 3 = is control parameter for the steepness function value transition.
A.4 Derivation of Formula 2
We provide the derivation for estimating the power-law exponent from empirical singular value data, which central to our approach. We assume the empirical distribution follows a power-law (Pareto) distribution:
p(z) = cx aa
For normalization over r > Xmini
p(x)
= == (a 1)x
Smin
Thus, the probability density function (PDF) becomes:
15

---

p(z) (a - 11) xa 1
aa
2min
Fmin
Given a set of observed data æ1, 22, ........ In with Tj 2 Tmins the likelihood function is:
p 22 02.---21 In)
p(x:)
Emin
Pmin
The log- leeelikel is therefore:
In(a 1)
In Fii
In
Emin
To obtain the maximum likelihood estimate, we set the derivative L with respect to a to zero:
ac
0 a=1+n
da
Fmin
This result is known as the standard Hill estimator, and we denote the fitted exponent as PL A__Alpha__n Hill.
B More experiments
B.1
LLM Pre-training with Adam
Table 12 provides a comparison of several weight decay scheduling strategies for pre-training LLaMa- 60M and LLLMa- 1MMMM-1300 130M models with the AdamW optimizer. The results clearly demonstrate the effectiveness of applying weight decay, as all scheduling strategies outperform the baseline with no weight decay (WD==) in terms of validation perplexity.
Table 12: W.) Comparison of various weight decay scheduling strategies using AdamW optimizer for pre-training LLaMa-60M and LLaMa-130M models under different weight decay values. Validation perplexity (4) on the C4 dataset is reported. All baselines are carefully tuned. "WD=0" indicates that weight decay disabled during model training.
LLaMa-60M
ALaMa-135M
Weight Decay
0.1
0.05
0.01
0.1
0.05
0.01
WD=0
32.73
24.39
Uniform
31.95
32.31
32.66
23.32
23.81
24.28
AWD
32.58
32.67
32.67
24.30
24.35
24.41
Adadecay
31.88
32.25
32.58
23.18
23.62
24.21
AlphaDecay
31.20
31.65
32.45
22.66
23.04
23.98
AlphaDecay consistently outperforms other weight decay scheduling strategies across different model sizes and hyperparameter settings, demonstrating superior regularization and generalization when training with AdamW. These results highlight the robustness and effectiveness of AlphaDecay, supporting adoption for optimizing large-scale transformer-basee language models.
B.2 Dependent t-test for paired samples
Table provides comparison of several weight decay scheduling strategies using the Adam optimizer, evaluated through repeated experiments with different random seeds.
16

---

Table 13: (Dependent t-test with Adam.) Each method (Uniform, AWD, AdaDecay, and AlphaDecay) evaluated by conducting six repeated experiments with random seeds 5, 6, 7, 8, 9, 10 }. Validation perplexity is reported mean + standard deviation. For each weight decay setting, a dependent t-test for paired samples is performed, comparing AlphaDecay against Uniform, AWD, and AdaDecay, respectively. The resulting p-values are presented alongside perplexity scores.
Weight Decay=0 Perplexity
Weight Decay=le-5
Weight Decay=5e-6
Weight Decay=le-6
Method
Perplexity
P-value
Perplexity
P-value
Perplexity
P-value
Uniform
22,97 0.07
8.38c-4
23.12 0.03
1,47c-6
24.12 0.04
1.05c-7
AWD
24.13 + 0.15
6.94e-6
24.46 + 0.07
5.34e-8
24.53 0.03
5.34e-9
24.55 0.07
AdaDecay
23.18 0.05
1.46c-5
23.07 0.04
1.11e-7
24.00 0.03
2.69c-9
AlphaDecay
22.77 0.02
22.54 0.03
23.44 0.02
The results demonstrate the benefit of applying weight decay for improved validation perplexity, with AlphaDecay consistently exhibiting superior performance and stability across all tested settings. The dependent t-test results further substantiate these findings, with statistically sigmificant p-values supporting the advantage of AlphaDecay over Uniform, AWD, and AdaDecay in nearly all cases.
B.3
Varying assignment function hyperparameters
Uniform
(0.33,1.5)
(0.67.1.5)
(0.67.3)
(0.67,5)
Parpeess
23:31
-0.11
.0.15
+0.08
Parplenity
22.00
-0.16
-0.45
.0.59
Perploxity
-0.00
-0.38
23:48
22.48
22.00
23.88
(a) Weight Decay 1e-5
(b) Weight Decay = 5e-6
(c) Weight Decay 1c-6
Figure 9: (Hyperparameter study on (81, 82)). Search for hyperparameters (81,82) of different weight decay settings on C4. The hyperparameter choice used the paper (see table [T] performs best among all the cases. All experiments are conducted LLaMa- 135M. The value the top of each bar indicates the difference from the leftmost bar in each plot.
We provide additional results of a hyperparameter study on (51, $2). in which we consider four different settings for s2); [00.33, 1.5), (0.67, 1.5), (0.67, 3), (0.67, 5)]. We run tasks C4 with ALaMa-135M. Our results in figure (0.67, 3) (0.67, 5), yields the best performance. This hyperparameter setting the default setting used in our paper. All hyperparameters are consistent with those described in the main paper.
D show that using a larger weight decay scaling range, such as
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, highlighted our contributions, including algorithm develop- ment, theoretical analysis, and empirical study.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the Conclusion, we discuss limitations applicability MoE models, interactions with alternative regularization approaches.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[NA]
Justification:
We do no have relevant theories or proofs
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided sufficient details for reproducing the results of the paper, such as parameter settings, runtime environments, and dataset descriptions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have included all the code and data in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.com
mubllte/guudees/CoteSStmiissss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ccipsscc/public//eg http:p//lll..p.ieeeeeeeeeaaaaaia CooeeSubeeeeeeeeeeeeee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have provided sufficient details for tranining, encompassing hyperparame- ter settings and dataset generation.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: For simplicity, we prove the validity of the method only by changing the hyperparameters and the number of model parameters, Our method has obvious ad advantages over the compared methods. These results have proved their significance in the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The machines we used in the experiment have been described in the corre- sponding table.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp////eerrgssscccc/bbttttttt
Answer: [Yes]
Justification:
Our research aligns with the ethical guidelines outlined by NeurIPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper solves the problems in model training. To our knowledge, this will not cause negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The dataset used in the experiments is published open site without license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.paperssiii wwwppppppttthhhoco thcode www.coo//a/a..o datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The experiments do not involve new datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human object is involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing or human object is involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is not an important, original, or non-standard component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
24


Reviews:
[{"review_id": "3tCNPIxayd", "reviewer": "Reviewer_a9sr", "summary": "This paper is concerned with the problem  weight decay in LLMs to  improve generalization.  In particular the  authors propose a scheme based on HT-SR theory to adaptively choose the weight day for different modules. The introduced method called AlphaDecay first calculates the PL_Alpha_Hill values for all modules, and then assign larger weight decay to modules with higher\nPL_Alpha_Hill values, while assigning smaller weight decay to those with lower PL_Alpha_Hill\nvalues. This scheme balances the modules.", "questions": "In figure 3, the performance of model is not shown and only the ESD is  demostrated, therefore I am not inferring the following from figure 3:\n\n“We empirically demonstrate that promoting uniformity in PL_Alpha_Hill across modules (e.g.,\nattention and MLP components) can further enhance overall model quality (see figure 3).”", "limitations": "-The paper is purely empirical and reporting observations on balancing weight day  on models qulity.  The terms such as model quality is  very generally used and it is not clear if the methods enhances performance in terms of ID and OOD  generalization gaps or other metrics.\n\n-The method is only implemented on LLama model and it is not clear if the same improvement will be achieved on other architetures.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The strength of this paper is that it is  tackling the weight decay for improving generalization of LLMs which is awidely sought topic and interesting to the  community. The weakness is:\n\n- The effect of  promoting uniformity in PL_Alpha_Hill across modules on model quality is only shown emprically and no mathematical proof is provided to prove it. This is while  the model quality is measure only by perplexity. I am not convinced this  strategy will improve other metrics and generalization.", "was_revised": true, "final_justification": "The authors  provided additional numerical results and  clarified that the theory is already proved in the lierature.", "rebuttal": "Thank you for your insightful and helpful feedback. We respond to each point below, offering additional empirical results, clarifications of our statements, and specific changes that will be reflected in the camera-ready version.\n\n\n>**Q1: The effect of promoting uniformity in PL_Alpha_Hill across modules on model quality is only shown emprically and no mathematical proof is provided to prove it. This is while the model quality is measure only by perplexity. I am not convinced this strategy will improve other metrics and generalization.**\n\n**A1:** We agree that model quality should be demonstrated on more than one metric, and we appreciate the opportunity to strengthen the submission.\n\n- **Seven unseen commonsense‑reasoning tasks (LLaMa‑1 B)**\n\nWe took the exact checkpoints from Table 2 (trained on C4, lr = 6e-4, wd = 1e-6) and ran ```lm‑eval‑harness``` with its default prompts. The results below shows AlphaDecay delivers the best score on 6 / 7 tasks and the best overall average. Because none of these datasets are part of pre-trained C4, the gains demonstrate that the AlphaDecay could translates into stronger  generalization ability.\n\n\n\n|    Schedule         | ARC-c | ARC-e | PIQA | Hellaswag | OBQA | Winogrande | BOOLQ | Avg.  |\n|--------------|-------|-------|------|-----------|------|------------|-------|-------|\n| Uniform      | 20.22 | 46.72 | 67.68| 32.94     |18.80 | 49.41      | 54.74 | 41.50 |\n| AdaDecay     | 19.20 | 46.72 | 66.97| 32.96     |18.00 | **51.54**      | 56.36 | 41.68 |\n| AWD          | 19.18 | 46.34 | 66.65| 31.37     |18.00 | 51.07      | 57.25 | 41.41 |\n| AlphaDecay   | **20.90** | **48.86** | **68.44**| **34.16**     |**19.80** | 50.59      | **60.70** | **43.35** |\n\n\n- **Supervised fine‑tuning on GLUE (RoBERTa‑base).**\n\nWe fine‑tuned for 3 epochs with AdamW (lr = 3e-5, wd = 0.1) and chose the best checkpoint on the validation split. AlphaDecay improves every task (average +1.29 points) relative to the uniform baseline (full table under **A3**).\n\n\n- **Why no “new” mathematical proof appears in the paper.**\n\nThe formal link between the power‑law (PL) exponent α and model capacity is already established in Heavy‑Tailed Self‑Regularisation (HT‑SR) theory [1-3]. As such, we intentionally avoid reinventing the wheel by restating established theory in our manuscript. Instead, our contribution focuses on providing new empirical results and analyses that extend and support these theoretical foundations in the context of modern large language models. We will add a concise “theory recap” at the start of §3 and include a direct citation to the key theorem.\n\n\nIf you have any further concerns or requests for additional evaluation, we would be happy to discuss and perform more analyses.\n\n[1] Martin, C. H., Peng, T., & Mahoney, M. W. (2021). Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1), 4122.\n\n[2] Martin, C. H., & Mahoney, M. W. (2021). Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165), 1-73.\n\n[3] Clauset, A., Shalizi, C. R., & Newman, M. E. (2009). Power-law distributions in empirical data. SIAM review, 51(4), 661-703.\n\n>**Q2: Not inferring the following from figure 3: “We empirically demonstrate that promoting uniformity in PL_Alpha_Hill across modules (e.g., attention and MLP components) can further enhance overall model quality (see figure 3).”**\n\n**A2:** Thank you for your careful reading of our manuscript and your valuable comments regarding Figure 3, we would like to clarify:\n\n- **What Fig. 3 shows:** \n\nEach subplot overlays the empirical spectral density (ESD) of a single module under Uniform vs. AlphaDecay for the same 135 M‑parameter network. The shift of the fitted α (numbers in parentheses) illustrates how AlphaDecay raises the α of attention matrices and lowers that of MLP matrices, to provide a deeper insight into the internal representations and spectral properties of the models. \n\n\n- **Where performance evidence resides**\n\nQuantitative quality metrics are gathered in Table 2 (perplexity, **Uniform:** PPL = 23.14 ,**AlphaDecay:** PPL = 22.55), new reasoning table (above), and the GLUE table (below). We will add an explicit forward reference—“See Table 2 and §4.2 for the associated perplexity gains”—to the current caption.\n\n\nThank you again for your observations, which help us make our exposition more precise.\n\n\n>**Q3: The paper is purely empirical and reporting observations on balancing weight day on models qulity. The terms such as model quality is very generally used and it is not clear if the methods enhances performance in terms of ID and OOD generalization gaps or other metrics.**  \n\n**A3:** Thank you for your insightful feedback. Our original paper reports ID performance via perplexity on the C4 dataset using LLaMA, demonstrating improvements in this setting. To address your concerns and provide a more comprehensive evaluation, we conducted additional experiments beyond the original submission.\nThese include zero-shot OOD generalization on seven diverse commonsense reasoning tasks (detailed in our response to **Q1**), where AlphaDecay shows consistent gains. We also fine-tuned Roberta-base with AdamW on the GLUE benchmark (eight NLP tasks: CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, and STS-B) to assess ID generalization across architectures and tasks. The table below compares AlphaDecay against uniform scheduling (higher scores are better):\n\n|          | COLA  | MNLI  | MRPC  | QNLI  | QQP   | RTE   | SST2  | STSB  | Avg.  |\n|----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| Uniform  | 59.73 | 86.78 | 87.01 | 92.59 | 89.97 | 70.11 | 93.69 | 90.78 | 83.83 |\n| AlphaDecay | **62.82** | **87.11** | **89.61** | **92.73** | **90.12** | **73.86** | **93.77** | **90.91** | **85.12** |\n\nThese results, alongside the LLaMA-based **ID** (C4 perplexity) and **OOD** (commonsense) improvements, indicate that AlphaDecay enhances model quality broadly—via metrics like **perplexity, accuracy, and generalization gaps—across architectures (LLaMA, roberta-base) and datasets (C4, commonsense reasoning, GLUE)**.\nWe appreciate your input and welcome further questions or requests for additional experiments.\n\n\n>**Q4: The method is implemented only on LLaMA; it is unclear whether the same improvement holds for other architectures.**  \n\n**A4:** We have conducted extensive experiments using two additional and distinct model architectures: (1) GPT-nano on the C4 dataset, and (2) ViT-tiny on ImageNet-1K.  We performed a thorough hyperparameter search for the Adam optimizer. The results are shown below:\n\n\n\n| Backbone / Dataset | Metric | Uniform | AWD | AdaDecay | AlphaDecay |\n|---|---|---|---|---|---|\n| GPT-nano  / C4 | PPL $\\downarrow$ | 27.564 | 27.641 | 27.680 | **27.374** |\n| ViT-tiny / ImageNet-1K | Top-1 $\\uparrow$ | 66.41% | 64.98% | 66.26% | **67.73%** |\n\n\nAs shown above, AlphaDecay consistently outperforms other regularization strategies on both language and vision architectures, suggesting that module‑wise α‑balancing is architecture‑agnostic. We hope these additional results clarify the robustness of our method.\n\n\n>**Q5: All parameters and metrics should be defined before usage; for instance, PL_Hill estimator is not introduced at its first appearance in Fig. 1.**  \n\n**A5:** We will:\n\n- insert a one‑sentence definition of PL_Alpha_Hill and ESD in §3;\n- add the same definition to the captions of Fig. 1 and Fig. 3;\n- create a mini‑glossary of acronyms in Appendix A;\n- add inline comments to Algorithm 1 linking each variable to its equation.\n\nThank you for highlighting these omissions—clear notation will make the paper much easier to follow."}, {"review_id": "sSUWPS7ZA9", "reviewer": "Reviewer_fKFY", "summary": "This work investigates adaptive layer-wise weight decay method, an interesting area in the LLMs training. The authors utilize heavy-tailed self-regularization theory to guide the research direction, which analyzes the empirical spectral density of the weight correlation matrices to indicate the heavy-tailedness of the module. The proposed method assign weaker weight decay to modules with more obvious heavy-tailed ESDs reflects and stronger weight decay otherwise, since the heavy-tailed ESDs reflect the degree of feature learning. Simulation results demonstrate the effectiveness and robustness of the proposed method.", "questions": "1.\tThe authors should provide detailed description of the theory and mathematical formulation.\n2.\tThere are some typos and poorly-defined variables. HT-SR seems to be used as HR-SR in this work, which is a big misleading. Is the Hill estimator equivalent to \\alpha? If so, it is suggested to replace PL_ALPHA_HILL with variable like \\hat{\\alpha} to enhance clarity. \n3.\tSome notations and figures require further explanation and clarification. The notion of heavy-tailedness requires further clarification. Is this notion defined by metric singular values, i.e., Fig. 2, or ESD, i.e., Fig. 3. The attention layers (att.q and att.k) exhibits less heavy-tailedness in Fig.2 while more heavy-tailedness in Fig. 3.In Fig.2, why are some layers, e.g., layer 1, 2 and 39, 40, drawn with different color?  In Fig.4, the PL_ALPHA_HILL does not change significantly with weight decay, please explain why. Same in Fig. 4, is the weight decay equivalent to \\eta in the context? In Fig.8, it seems more frequent weight decay adjustment does not yield performance improvement, which requires further explanation.\n4.\tThe performance of the proposed method is only evaluated on one dataset and metric. More datasets and metrics are needed to demonstrate robustness.\n5.\tCertain parameter setting is not motivated. For example, the selection of the range of scaling ratios should be explained.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1.\tThis paper introduces an adaptive weight decay method, which differs from the traditional methods that utilizes a uniform or time-wise decay rate.\n2.\tThe authors adopt HT-SR theory to guide the weight decay rate selection. The theory provides a strong theoretical basis for the proposed method.\n3.\tThe simulation results and performance improvements across various model sizes show the effectiveness and robustness of the proposed method.\n\nWeaknesses\n1.\tThe rationale and theoretical framework is vague. The authors only provide high-level description of the theory and there is no detailed mathematical formulation.\n2.\tThe clarity of the paper requires improvement. There are some typos and poorly-defined variables. Some important notion such as heavy-tailedness and some figures are not well explained.", "was_revised": true, "final_justification": "In this work, the authors proposed an adaptive weight decay method to enhance LLM training with theoretical analysis. Several issues regarding clarity and motivation existed in the original manuscript have been well addressed. Therefore, I have raised the points from 2 to 3 for quality and clarity. The rating is also updated from 4 to 5.", "rebuttal": "Thank you for the thoughtful and constructive feedback. \n\n>**Q1: The authors only provide high-level description of the theory and there is no detailed mathematical formulation; Some important notion such as heavy-tailedness and some figures are not well explained;**  \n\n**A1:** Thank you for your valuable suggestions. Our theoretical framework is based on the HT-SR theory, which highlights the close relationship between the heavy-tailedness of a matrix's singular value spectrum and the degree to which the matrix is trained. In prior works, it is standard practice to fit the ESD of the correlation matrix using a power-law model. The resulting exponent from this fit quantitatively characterizes the degree of heavy-tailedness. According to HT-SR theory, in LLMs, a matrix with a fitted exponent PL_Alpha_Hill less than 2 is considered “overtrained” while a matrix with an exponent greater than 4 is regarded as “undertrained” [1]. Traditional Uniform Weight Decay often results in both “overtrained” and “undertrained” parameters in a model. Our method addresses this by adaptively adjusting weight decay to achieve better balance.\n\nWe provide the derivation for estimating the power-law exponent $\\alpha$ from empirical singular value data, which is central to our approach. We assume the empirical distribution follows a power-law distribution:\n\n$$\np(x) = c x^{-\\alpha}\n$$\n\nFor normalization over $x \\geq x_{min}$:\n\n$$\n\\int_{x_{\\min}}^{\\infty} p(x)  dx = 1 \\implies c = (\\alpha-1)x_{\\min}^{\\alpha-1}\n$$\n\nThus, the probability density function (PDF) becomes:\n\n$$\np(x) = (\\alpha-1)x_{\\min}^{\\alpha-1} x^{-\\alpha}\n= \\frac{\\alpha-1}{x_{\\min}} \\left(\\frac{x}{x_{\\min}}\\right)^{-\\alpha}\n$$\n\nGiven a set of observed data $x_1, x_2, \\dots, x_n$ with $x_i \\geq x_{min}$, the likelihood function is:\n\n$$\np(x_1, x_2, \\dots, x_n) = \\prod_{i=1}^{n} p(x_i) \n= \\prod_{i=1}^n \\frac{\\alpha-1}{x_{\\min}} \\left(\\frac{x_i}{x_{\\min}}\\right)^{-\\alpha}\n$$\n\nThe log-likelihood is therefore:\n\n$$\n\\mathcal{L} = \\sum_{i=1}^{n} \\left[ \\ln(\\alpha-1) - \\ln x_{\\min} - \\alpha \\ln \\frac{x_i}{x_{\\min}} \\right]\n$$\n\nTo obtain the maximum likelihood estimate, we set the derivative of $\\mathcal{L}$ with respect to $\\alpha$ to zero:\n\n\n$$\n\\frac{\\partial \\mathcal{L}}{\\partial \\alpha} = 0 \\implies\n\\alpha = 1 + n \\left[ \\sum_{i=1}^{n} \\ln \\frac{x_i}{x_{\\min}} \\right]^{-1}\n$$\n\nThis result is known as the standard **Hill estimator**, and in our paper, we denote the fitted exponent as **PL_Alpha_Hill**.\n\n**References**\n\n[1] Martin, C. H., Peng, T., & Mahoney, M. W. (2021). Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1), 4122.\n\n\n>**Q2: Typos and poorly‑defined variables … ‘HR‑SR’ vs ‘HT‑SR’; consider using $\\hat{\\alpha}$ instead of PL_ALPHA_HILL.”.**  \n\n**A2:** Thank you for catching the “HR-SR” typo; every instance will be corrected to **HT-SR**.\n- We agree that standard statistical notation improves readability. In the revision we will (i) replace **PL_ALPHA_HILL** by $\\hat{\\alpha}$ in all text, equations, and figures, and (ii) add a parenthetical remark the first time $\\hat{\\alpha}$ appears: “$\\hat{\\alpha}$ is estimated via the Hill method (see §3.1).”\n- All remaining typos flagged during our own proofreading pass will also be fixed.\n\nWe appreciate your attention to these details—they will substantially improve the paper’s presentation.\n\n\n>**Q3: Some notations and figures require further explanation and clarification. The notion of heavy-tailedness requires further clarification. Is this notion defined by metric singular values, i.e., Fig. 2, or ESD, i.e., Fig. 3. The attention layers (att.q and att.k) exhibits less heavy-tailedness in Fig.2 while more heavy-tailedness in Fig. 3.**   \n\n**A3:**   Thank you very much for your valuable comments regarding the notations and the notion of heavy-tailedness.\n\n- Clarification on “heavy-tailedness“:\n\nHeavy-tailedness in our context refers exclusively to the tail behaviour of the metric singular-value spectrum of a weight matrix. We diagnose it by fitting a power-law to the right-tail of the ESD and reporting ${\\alpha}$. A smaller ${\\alpha}$ indicates a heavier tail.\n\n \n- Why Fig. 2 and Fig. 3 differ:\n\nFig. 2 shows the sorted singular values $\\{ \\sigma_i \\}$ of each weight matrix. Fig. 3 shows the ESD, defined as:\n\n$$\nESD(x) = \\frac{1}{N} \\sum_{i=1}^N \\delta(x-\\sigma_i)\n$$\n\nwhere  $\\delta$ is the Dirac delta function and $\\{ \\sigma_i \\}$ are the singular values. For heavy-tailed analysis, we fit a power-law to the tail of the ESD.  Note that the metric plotted in Fig. 2 (sorted singular values) is different from that in Fig. 3 (ESD), so “heavy-tailedness” in Fig. 2 is not directly comparable to that in Fig. 3.\n  \n- Motivation for our analysis:\n\nFig. 2 and Fig. 3 reveal a limitation of uniform weight decay, which ignores such differences and can result in some parameters being overtrained or undertrained. Our work therefore advocates for module-wise parameterization.\n\nWe hope this clarifies our notations and the definition of heavy-tailedness.\n\n\n>**Q4: In Fig.2, why are some layers, e.g., layer 1, 2 and 39, 40, drawn with different color?**  \n\n**A4:** We colored the first and last layers differently in Fig. 2 to highlight their special importance, as recognized in prior studies [1]. This also improves the figure’s clarity and readability.\n\n[1] Song, Z., Huang, S., Wu, Y., & Kang, Z. (2024). Layer importance and hallucination analysis in large language models via enhanced activation variance-sparsity. arxiv preprint arxiv:2411.10069.\n  \n>**Q5:In Fig. 4, is the weight decay equivalent to \\eta in the context?**  \n\n**A5:** As described in the pseudocode (Algorithm 1) in our paper, all occurrences of \\(\\eta\\) in this context refer to the weight decay parameter.  \n\n\n>**Q6: In Fig.4, the PL_ALPHA_HILL does not change significantly with weight decay, please explain why.**  \n\n**A6:**  Fig. 4 only shows a narrow range of weight decay values (3e-6 to 1e-5), so PL_ALPHA_HILL changes little. When we expand the weight decay range, the effect becomes clear, as shown in the table below:\n\n| Weight Decay | att.Q/K | att.V/O |  mlp  |  PPL   |\n|--------------|---------|---------|-------|--------|\n| 0            | 1.90    | 1.87    | 2.54  | 24.53  |\n| 5e-6         | 1.71    | 1.94    | 2.47  | 23.27  |\n| 1e-5         | 1.60    | 1.97    | 2.42  | 23.11  |\n| 5e-5         | 1.31    | 1.86    | 2.19  | 26.71  |\n| 9e-5         | 1.19    | 1.77    | 2.01  | 30.87  |\n\nWith higher weight decay, PL_ALPHA_HILL drops significantly, especially for att.Q/K (a relative reduction of about 37%). We hope this clarifies the relationship.\n\n\n>**Q7: In Fig.8, it seems more frequent weight decay adjustment does not yield performance improvement, which requires further explanation.**  \n\n**A7:** Adjusting module-wise weight decay does not have an immediate effect. After each adjustment, it takes time for training to stabilize and for changes to take effect. When the adjustment GAP is small (e.g., GAP=5), monitoring the distribution over the next 100 steps usually reveals little change. Moreover, frequent adjustments and computations can disrupt convergence. We hope this clarifies the phenomenon. Please let us know if you have further questions.\n  \n>**Q8: The performance of the proposed method is only evaluated on one dataset and metric. More datasets and metrics are needed to demonstrate robustness.**  \n\n**A8:** To demonstrate the robustness of AlphaDecay, we evaluated it on multiple datasets and metrics.\n\n- Seven unseen commonsense‑reasoning tasks (LLaMa‑1 B)\n\nWe took the exact checkpoints from Table 2 (trained on C4, lr = 6e-4, wd = 1e-6) and ran ```lm‑eval‑harness``` with its default prompts. The results below shows AlphaDecay delivers the best score on 6 / 7 tasks and the best overall average. Because none of these datasets are part of pre-trained C4, the gains demonstrate that the AlphaDecay could translates into stronger  generalization ability.\n\n|              | ARC-c | ARC-e | PIQA | Hellaswag | OBQA | Winogrande | BOOLQ | Avg.  |\n|-------|-----|------|------|-----------|------|-------|-------|-------|\n| Uniform      | 20.22 | 46.72 | 67.68| 32.94     |18.80 | 49.41      | 54.74 | 41.50 |\n| AdaDecay     | 19.20 | 46.72 | 66.97| 32.96     |18.00 | **51.54**      | 56.36 | 41.68 |\n| AWD          | 19.18 | 46.34 | 66.65| 31.37     |18.00 | 51.07      | 57.25 | 41.41 |\n| AlphaDecay   | **20.90** | **48.86** | **68.44**| **34.16**     |**19.80** | 50.59      | **60.70** | **43.35** |\n\n- Supervised fine‑tuning on GLUE (RoBERTa‑base).\n\nWe fine‑tuned for 3 epochs with AdamW (lr = 3e-5, wd = 0.1) and chose the best checkpoint on the validation split. AlphaDecay improves every task (average +1.29 points) relative to the uniform baseline:\n\n|           | COLA  | MNLI  | MRPC  | QNLI  | QQP   | RTE   | SST2  | STSB  | Avg.  |\n|-------|-----|-----|-------|-----|-----|-----|-----|-----|-----|\n| Uniform   | 59.73 | 86.78 | 87.01 | 92.59 | 89.97 | 70.11 | 93.69 | 90.78 | 83.83 |\n| AlphaDecay| **62.82** | **87.11** | **89.61** | **92.73** | **90.12** | **73.86** | **93.77** | **90.91** | **85.12** |\n\nAs shown, AlphaDecay achieves consistent improvements in both zero-shot and fine-tuning settings, which demonstrates the robustness and practical effectiveness of our proposed method.\n\n>**Q9: Certain parameter setting is not motivated. For example, the selection of the range of scaling ratios should be explained.**  \n\n**A9:** Thank you for your comment on the scaling ratio range selection. Our choice was primarily empirical, based on preliminary ablation studies. For transparency, we have added detailed results and supporting experiments in Appendix B, where we systematically examine different scaling ratio settings. Please let us know if you need further clarification or additional results."}, {"review_id": "bXAL8QzqQV", "reviewer": "Reviewer_pAbe", "summary": "This paper proposes AlphaDecay, an adaptive, module-wise weight decay method for LLM training. AlphaDecay dynamically assigns different decay strengths to each module of a transformer based on the heavy-tailedness of their weight spectra, quantified via the Hill estimator of power-law exponents. The authors justify their approach using heavy-tailed self-regularization (HT-SR) theory and show empirically, on LLaMa-based models of various sizes (60M–1B), that AlphaDecay improves pre-training perplexity over standard and other adaptive baselines (AWD, Adadecay). Extensive ablations demonstrate the robustness of the method to different hyperparameters and computation intervals.", "questions": "Q1: In Figure 8, the setting with GAP=50 achieves the best performance. Could the authors provide details on the computational cost of AlphaDecay when the interval size is set to 50?\n\nQ2: The paper focuses on the pretraining stage. Is it possible to extend AlphaDecay to the finetuning stage, especially for specific downstream tasks such as question answering or text classification? Any insights or preliminary results in this direction would be valuable.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The motivation for the proposed method is simple, straightforward, and easy to grasp.  the use of HT-SR theory to analyze module-specific training behavior and justify differential regularization is well-grounded.\n- The experimental setup is solid and provides clear evidence of the method’s effectiveness and robustness.\n\nWeakness:\n- The number of compared baselines is limited. Notably, the strongest baseline cited was published in 2019, which raises concerns about whether there are more recent or advanced approaches in this research area that should be included for comparison. \n- The largest model evaluated in this work has 1B parameters, which is relatively small by current LLM standards. Evaluating on larger models, such as 7B or above, would make the results more compelling. Additionally, the experiments are limited to the LLaMa series of models, which restricts the generalizability of the findings.", "was_revised": true, "final_justification": "Following the authors’ rebuttal and the additional experiment they conducted, my main concerns have been addressed. The new results are consistent with and support my original evaluation, reinforcing the validity of the paper’s claims. Given the convincing rebuttal and supportive new evidence, I maintain my positive rating.", "rebuttal": "We appreciate the constructive comments from the reviewer.\n\n>**Q1: The number of compared baselines is limited. Notably, the strongest baseline cited was published in 2019, which raises concerns about whether there are more recent or advanced approaches in this research area that should be included for comparison.**  \n\n**A1:** Thank you for your insightful comments and interest in the completeness of baseline comparisons.\n\nTo our knowledge, few studies directly address this research direction, and our work is among the first to explore it. The two baselines in our manuscript—AWD [1] (2023) and AdaDecay [2] (2019)—are the most relevant existing methods.\n\nTo further address your concern, we added a comparison with AdamS [3], a recent optimizer using dynamic weight decay. We tuned AdamS extensively on LLaMa-60M and LLaMa-135M, searching learning rates and weight decay values. AdamS performs best with a learning rate of 0.001 and weight decay values of {5e-5, 1e-5, 5e-6}. We combined AlphaDecay with AdamS, and the results are shown below (lower is better):\n\n|       Method|            |   **LLaMa-60M**   |        |           |   **LLaMa-135M**   |        |\n|-----------------|---------|---------|----------|---------|---------|----------|\n|       **Weight Decay**          | 5e-5 | 1e-5 | 5e-6 | 5e-5 | 1e-5 | 5e-6 |\n| **WD=0**        |  \\-     |   32.73    | \\-    |   \\-    |   24.39    | \\-    |\n| **AdamS[3]**       | 32.705  | 32.715  | 32.722   | 24.380  | 24.393  | 24.393   |\n| **AdamS+AlphaDecay** | **32.699**  | **32.712**  | **32.715**   | **24.370**  | **24.380**  | **24.392**   |\n\nAdamS is robust and stable across weight decay settings, but it does not fully utilize weight decay to improve training as AlphaDecay does. Our method remains more competitive in LLM training.\n\nIf you are aware of any additional relevant methods or recent studies on weight decay, we would be more than happy to include them in our comparisons with AlphaDecay in future revisions. Thank you again for your helpful suggestions and your attention to research completeness.\n\n[1] Mohammad Amin Ghiasi, Ali Shafahi, and Reza Ardekani. Improving robustness with adaptive weight decay. Advances in Neural Information Processing Systems, 36:79067–79080, 2023.\n\n[2] Kensuke Nakamura and Byung-Woo Hong. Adaptive weight decay for deep neural networks. IEEE Access, 7:118857–118865, 2019.\n\n[3] Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, and Masashi Sugiyama. On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective. Advances in Neural Information Processing Systems, 36:1208–1228, 2023.\n\n  \n\n>**Q2: The largest model evaluated in this work has 1B parameters, which is relatively small by current LLM standards. Evaluating on larger models, such as 7B or above, would make the results more compelling. Additionally, the experiments are limited to the LLaMa series of models, which restricts the generalizability of the findings.**  \n\n**A2:** Thank you for your valuable feedback and for highlighting the importance of evaluating our method on larger models and more diverse architectures.\n\nWe fully acknowledge that our largest evaluated model (LLaMA-1B) is relatively small compared to state-of-the-art LLMs such as those with 7B parameters or above. Due to limited computational resources, we are currently unable to conduct experiments at this unprecedented scale. However, we agree that such experiments would further strengthen our findings, and we hope to pursue them in future work as additional resources become available. To further address the concern regarding the generalizability of our method, we have conducted additional experiments on a wider range of model architectures beyond the LLaMA series. Specifically, we evaluated AlphaDecay on (1) **GPT-nano** (an auto-regressive language model) trained on the C4 dataset, and (2) **ViT-tiny** (a Vision Transformer) trained on ImageNet-1K. The results are as follows:\n\n**(1). GPT-nano on C4**\n\nWe performed a detailed hyperparameter search for the Adam optimizer, selecting a learning rate of 6e-4 and weight decay values of 5e-6 and 1e-6. The metric reported is validation perplexity (lower is better):\n\n| Weight Decay | Uniform | AWD    | AdaDecay | AlphaDecay |\n|--------------|---------|--------|----------|------------|\n| 0            | 27.721 |\n| 5e-6         | 27.564  | 27.641 | 27.680   | **27.374** |\n| 1e-6         | 27.710  | 27.738 | 27.716   | **27.366** |\n\n**(2). ViT-tiny on ImageNet-1K**\n\nSimilarly, for the vision domain, we conducted a thorough hyperparameter search and selected a learning rate of 4e-3 with weight decay values of 5e-6 and 1e-6. The metric reported is top-1 validation accuracy (higher is better):\n\n| Weight Decay | Uniform | AWD    | AdaDecay | AlphaDecay |\n|--------------|---------|--------|----------|------------|\n| 0            | 65.12 |\n| 5e-6         | 66.97   | 65.21  | 65.86    | **67.40**  |\n| 1e-6         | 66.41   | 64.98  | 66.26    | **67.73**  |\n\nAs seen above, AlphaDecay consistently outperforms other regularization strategies across both language and vision tasks, demonstrating its effectiveness and generalizability beyond the LLaMA architecture.\n\nWe are committed to further validating our approach on even more diverse architectures as computational resources permit, and we welcome suggestions for additional settings that would be of interest to the community. Thank you again for your insightful comments.\n  \n\n>**Q3: In Figure 8, the setting with GAP=50 achieves the best performance. Could the authors provide details on the computational cost of AlphaDecay when the interval size is set to 50?**  \n\n**A3:** Thank you for your concern regarding the computational cost of AlphaDecay at different interval (GAP) settings. To provide a clear picture, we present below the perplexity (PPL) and the corresponding computation time (measured in NVIDIA A100 GPU hours) for various GAP values, as also visualized in Figure 8 of our main paper:\n\n| Model         | Weight Decay | Adam  | GAP=500 | GAP=250 | GAP=100 | GAP=50 | GAP=1  |\n|---------------|--------------|-------|---------|---------|---------|--------|--------|\n| LLaMa-135M    | 1e-5     | 22.99 | 22.76   | 22.76   | 22.78   | 22.76  | 22.81  |\n|               | 5e-6     | 23.14 | 22.55   | 22.54   | 22.57   | 22.54  | 22.58  |\n|               | 1e-6     | 24.14 | 23.49   | 23.48   | 23.48   | 23.47  | 23.49  |\n| **Computation Time (hours)** |              | 5.6   | 5.7     | 5.9     | 6.3     | 7.1    | 74.5   |\n\nAs shown in the last row, using GAP=50 increases the total training time to 7.1 A100 GPU hours (added more than 24% training time), while GAP=500 only added less than 3% training time. However, the improvement in perplexity with GAP=50 compared to GAP=500 is relatively marginal. Based on this trade-off between computational cost and model performance, we chose GAP=500 as the default setting in our main experiments, as it offers near-optimal performance with much lower computational overhead.\n\nWe hope this answers your question. If you need more details or further comparisons, we would be happy to provide them.\n \n\n>**Q4: The paper focuses on the pretraining stage. Is it possible to extend AlphaDecay to the finetuning stage, especially for specific downstream tasks such as question answering or text classification? Any insights or preliminary results in this direction would be valuable.**  \n\n**A4:** Thank you for your valuable question about extending AlphaDecay to the fine-tuning stage and its impact on downstream tasks. To explore this, we conducted experiments evaluating AlphaDecay during fine-tuning across various tasks. \n\n- Supervised fine‑tuning on GLUE (RoBERTa‑base).\n\nWe fine‑tuned for 3 epochs with AdamW (lr = 3e-5, wd = 0.1) and chose the best checkpoint on the validation split. AlphaDecay improves every task (average +1.29 points) relative to the uniform baseline.\n\n|           | COLA  | MNLI  | MRPC  | QNLI  | QQP   | RTE   | SST2  | STSB  | Avg.  |\n|-----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| Uniform   | 59.73 | 86.78 | 87.01 | 92.59 | 89.97 | 70.11 | 93.69 | 90.78 | 83.83 |\n| AlphaDecay| **62.82** | **87.11** | **89.61** | **92.73** | **90.12** | **73.86** | **93.77** | **90.91** | **85.12** |\n\n- Seven unseen commonsense‑reasoning tasks (LLaMa‑1 B)\n\nWe took the exact checkpoints from Table 2 (trained on C4, lr = 6e-4, wd = 1e-6) and ran ```lm‑eval‑harness``` with its default prompts. The results below shows AlphaDecay delivers the best score on 6 / 7 tasks and the best overall average. Because none of these datasets are part of pre-trained C4, the gains demonstrate that the AlphaDecay could translates into stronger  generalization ability.\n\n|            | ARC-c | ARC-e | PIQA  | Hellaswag | OBQA  | Winogrande | BOOLQ | Avg.  |\n|------------|-------|-------|-------|-----------|-------|------------|-------|-------|\n| Uniform    | 20.22 | 46.72 | 67.68 | 32.94     | 18.80 | 49.41      | 54.74 | 41.50 |\n| AlphaDecay | **20.90** | **48.86** | **68.44** | **34.16**     | **19.80** | **50.59**      | **60.70** | **43.35** |\n\nThese results indicate that AlphaDecay is effective not only during pretraining, but also in fine-tuning, and enhances model performance across both in-domain and out-of-domain tasks. Thank you again for your suggestion—we are happy to provide further insights or results if needed."}, {"review_id": "VrT2pPOZKS", "reviewer": "Reviewer_p5Tj", "summary": "After establishing that there are variations in the spectral properties of different components in transformer blocks of LLMs, i.e., the weight matrices of attention modules have singular values that follow power law distributions with heavier tails as opposed to the singular values of weight matrices attached to MLP modules, which follow power law distributions with lighter tails, the authors of this work propose a weight decay regularization scheme that aids in closing the gap between the aforementioned spectral properties of attention and MLP components. Namely, the AlphaDecay regularization strategy results in making heavier tails lighter (attention) and lighter tails heavier (MLP).\n\nExperimental evidence in the paper suggests that AlphaDecay outperforms, in terms of achieving lower perplexity score, other baselines such as the Uniform weight decay assignment, Adaptive Weigth Decay (ADW), and AdaDecay.", "questions": "1. lines 41-42: the phrasing is slightly ambiguous. What should be understood from the context is that the model uses a single, time-dependent weight decay value that is updated over training steps. At each training time-step, this value is applied uniformly across all layers.\n\n\n2. lines 52-54: In listing the first contribution of their work, the authors claim that the substantial variations in the spectral properties of attention and MLP layers, are primarily responsible for harming the generalization ability, referring to Fig. 4. However, it seems that the left subfigure in Fig. 4 does not readily yield that link. Instead, it can be more accurately said, that the AlphaDecay scheme that the authors propose, balances the variations in the module-wise spectral properties, and in comparison with applying a module-wise uniform weight decay, the proposed scheme establishes superior generalization in terms of the perplexity score.\n\n\n3. lines 55-57: The spectral alignment between the various modules is witnessed in Fig. 1 as well as in Fig. 3. There is an erroneous figure number referenced in line 57. Instead of Fig. 3, the improved training performance is reported in Fig. 4. It will be also good to include in the caption of Fig. 4 whether perplexity refers to training or validation perplexity.\n\n\n4. line 70: While the first part of the Related Work section is related to weight decay and LLMs, reference [20] is mentioned as showing the importance of weight decay in scale-invariant systems. Although, LLMs are not globally scale-invariant, one can especially refer to some of their components (more specifically layer normalization) as scale-invariant. And since this work is focusing on weight decay schemes applied solely to attention and MLP components, it might be out of scope to refer to the relevance of weight decay in scale-invariant systems.\n\n\n5. There is an overlap of information between lines 74-79 and 88-92.\n\n\n6. line 115: The reference to Fig. 3 might be unnecessary and misleading since the paragraph describes the findings of Fig. 2.\n\n\n7. line 134: L instead of N modules.\n\n\n8. line 135: Are those the correlation or covariance matrices? Are the weight matrices normalized?\n\n\n9. The last sentence in Fig. 3 is misleading. What the figure demonstrates is that the authors’ method forces balance between “overtrained” and “undertrained” modules which according to references [24, 46] contributes to overall model quality, which is rightly mentioned in l. 151. Besides that, Fig. 3 does not say anything about model performance so far.", "limitations": "The authors have addressed the limitations properly. See also weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-written, the scope and contributions clearly articulated by the authors, and the methodology elegant. Although, I would like to point out to what I see as potential weaknesses and ground for substantial improvement.\n\nWeaknesses: \n\n- A downside is that the experimental results are presented for a single family of models, i.e., LLaMa, on a single dataset, that is the C4 dataset. While C4 is a widely used corpus for pre-training, other datasets could be used to assess the capabilities of LLaMa models post-training when applying the AlphaDecay regularization strategy.\n- Besides using only the perplexity score for validation, additional metrics like accuracy, BLEU or F1 could be reported if the model was fine-tuned or evaluated on downstream tasks derived from C4.\n- When describing the hyperparameter settings, it is mentioned that all the models were trained using the Adam optimizer. Would the results remain stable under using other optimizers such as Adam, AdamW, AdamS or even SGD?\n- Finally, all the results in Figures 5-8 are discussed when fixing the scale of the LLaMa model with 135M parameter count (besides Fig. 6 which considered the LLaMa model with 60M parameters). It will be good to demonstrate the universality of the results presented in those figures when considering all the LLaMa models (with 60M, 135M, 350M, 1B parameters).", "was_revised": true, "final_justification": "The authors went in great lengths to conduct new experiments in order to address all my questions. More specifically:\n\ni) they tested their results on GPT-nano (still for the C4 dataset) and on a visual transformer for the ImageNet-1K (ViT tiny architecture). This extra effort cleared any concerns I initially had about the robustness and superiority of AlphaDecay.\n\nii) they have also added evaluations on downstream tasks and when doing supervised fine-tuning. Their proposed method provides gains despite the choice of a specific optimizer.\n\nI thereby side and agree with the consensus of the other reviewers that proposed acceptance of the paper with minimal adjustments or additions (see my Questions 4-11 and the way that the authors promised to fix the issues listed therein).", "rebuttal": "We thank the reviewer for the detailed feedback.  \n\n>**Q1: Experimental results are only presented for LLaMa on C4.**  \n\n**A1:** Thank you for your thoughtful feedback. We appreciate your concern regarding the scope of our experimental evaluation. To address this, we ran two additional sets of experiments:\n\n\n| Backbone / Dataset | Metric | Uniform | AWD | AdaDecay | AlphaDecay |\n|---|---|---|---|---|---|\n| GPT-nano  / C4 | PPL $\\downarrow$ | 27.564 | 27.641 | 27.680 | **27.374** |\n| ViT-tiny / ImageNet-1K | Top-1 $\\uparrow$ | 66.41% | 64.98% | 66.26% | **67.73%** |\n\nAlphaDecay outperforms the strongest baseline with the same optimiser and hyper-parameters in both cases, demonstrating its broad applicability and effectiveness across models and datasets.\n\n\n>**Q2:Only perplexity is used; more metrics or downstream tasks are needed.**  \n\n**A2:** We agree that a robust study should look beyond a single corpus and beyond perplexity. We have added two complementary evaluations:\n\n- Seven unseen commonsense‑reasoning tasks (LLaMa‑1 B)\n\nWe took the exact checkpoints from Table 2 (trained on C4, lr = 6e-4, wd = 1e-6) and ran ```lm‑eval‑harness``` with its default prompts. The results below shows AlphaDecay delivers the best score on 6 / 7 tasks and the best overall average. Because none of these datasets are part of pre-trained C4, the gains demonstrate that the AlphaDecay could translates into stronger  generalization ability.\n\n|              | ARC-c | ARC-e | PIQA | Hellaswag | OBQA | Winogrande | BOOLQ | Avg.  |\n|--------------|-------|-------|------|-----------|------|------------|-------|-------|\n| Uniform      | 20.22 | 46.72 | 67.68| 32.94     |18.80 | 49.41      | 54.74 | 41.50 |\n| AlphaDecay   | **20.90** | **48.86** | **68.44**| **34.16**     |**19.80** | **50.59**      | **60.70** | **43.35** |\n\n\n- Supervised fine‑tuning on GLUE (RoBERTa‑base).\n\nWe fine‑tuned for 3 epochs with AdamW (lr =3e-5, wd = 0.1) and chose the best checkpoint on the validation split. AlphaDecay improves every task (average +1.29 points) relative to the uniform baseline.\n\n\n| Schedule   | COLA (MCC) | MNLI (Acc) | MRPC (F1) | QNLI (Acc) | QQP (F1) | RTE (Acc) | SST-2 (Acc) | STS-B (Spear.) | Mean  |\n|------------|------------|------------|-----------|------------|----------|-----------|-------------|----------------|-------|\n| Uniform    | 59.73      | 86.78      | 87.01     | 92.59      | 89.97    | 70.11     | 93.69       | 90.78          | 83.83 |\n| **AlphaDecay** | **62.82** | **87.11** | **89.61** | **92.73** | **90.12** | **73.86** | **93.77** | **90.91** | **85.12** |\n\n\nThese results further demonstrate that AlphaDecay consistently improves performance across all downstream tasks and evaluation metrics in the fine-tuning setting.\n\n\n>**Q3: Are gains robust to different optimizers (AdamW, AdamS, SGD)?**\n\n**A3:** Thank you for your question. We conducted additional experiments to test robustness across optimizers:\n\n**(1) AdamW**:\n\nOn LLaMa-60M and LLaMa-135M, AlphaDecay consistently outperforms AdamW, especially at moderate weight decay values.\n\n|       Method          |            |   **LLaMa-60M**   |        |   **LLaMa-135M**        |      |        |\n|-----------------|---------|---------|----------|---------|---------|----------|\n|      **Weight Decay**           | 0.1    | 0.05   | 0.001   | 0.1    | 0.05   | 0.001   |\n| **AdamW**     | 31.95  | 32.31  | 32.66   | 23.32  | 23.81  | 24.28   |\n| **AlphaDecay** | **31.20**  | **31.65**  | **32.45**   | **22.66**  | **23.04**  | **23.98**   |\n\n**(2) AdamS**:\n\nWith AdamS, performance is stable across weight decay values, but AlphaDecay still provides marginal improvements.\n\n|       Method          |            |  **LLaMa-60M**    |        |   **LLaMa-135M**        |      |        |\n|-----------------|---------|---------|----------|---------|---------|----------|\n|        **Weight Decay**         | 5e-5 | 1e-5 | 5e-6 | 5e-5 | 1e-5 | 5e-6 |\n| **AdamS**       | 32.705  | 32.715  | 32.722   | 24.380  | 24.393  | 24.393   |\n| **AlphaDecay** | **32.699**  | **32.712**  | **32.715**   | **24.370**  | **24.380**  | **24.392**   |\n\n**(3). SGD**:\n\nSGD is not practical for LLM training due to instability and slow convergence; we could not achieve convergence with SGD in our tests.\n\n\n>**Q4: Demonstrate universality across all LLaMa scales in figures 5-8.**  \n\n**A4:** Thank you for your suggestion on demonstrating universality across LLaMa model scales. Due to resource constraints, we added experiments on LLaMa-350M (with LLaMa-1B planned for future version). Results are summarized below:\n\n**(1). Varying Weight Decay Assignment Functions**\n\n| Weight Decay | Uniform | Linear | Sqrt  | Log2  | Sigmoid |\n|--------------|---------|--------|-------|-------|---------|\n| 5e-6     | 16.74   | **16.66**  | 16.77 | 16.77 | 17.81   |\n| 1e-6     | 17.50   | **16.88**  | 17.52 | 17.44 | 17.86   |\n\n**Observation:** Linear remains superior.\n\n**(2). Varying HT-SR Metrics**\n\n| Weight Decay | Uniform | Alpha | GradNorm | Fnorm | Snorm |\n|--------------|---------|-------|----------|-------|-------|\n| 5e-6     | 16.74   | **16.66** | 16.78    | 17.45 | 17.41 |\n|1e-6     | 17.50   | 16.88 | 17.09    | **16.80** | 17.13 |\n\n**Observation:** Using the HT-SR (Alpha) metric continues to yield optimal or near-optimal results.\n\n**(3). Varying Power-Law Fitting Methods**\n\n| Weight Decay | Goodness-of-fit | Fix-finger | Median |\n|--------------|----------------|------------|--------|\n| 5e-6     | 16.68          | 16.76      | **16.66**  |\n| 1e-6     | 17.38          | 17.50      | **16.88**  |\n| Time         |      38.1s     |    34.8s  | **10.2s**   |\n\n**Observation:** The Median method best balances performance and computation time.\n\n**(4). Varying Power-Law Fitting Gaps**\n\n| Weight Decay | GAP=500 | GAP=100 | GAP=50   |\n|--------------|---------|---------|----------|\n| 5e-6     | **16.66**   | 17.52   | 17.52    |\n| 1e-6     | **16.88**   | 16.94   | 17.21    |\n| Time         | **25.9h**   | 32.0h   | 39.8h    |\n\n**Observation:** Results are stable across gap settings. The \"hours\" reported in the above table refer to NVIDIA H100 GPU hours.\n\n\nThese findings validate that our methods generalize well to larger models. Thank you again for your helpful feedback.\n\n\n>**Q5:  Lines 41-42 are ambiguous.**  \n\n**A5:**  Thank you for pointing this out. We meant that AWD keeps weight decay uniform across layers, but dynamically adjusts the overall decay rate for the model during training based on loss gradients. We will revise the sentence to: “They propose AWD, which maintains uniform weight decay across layers while dynamically updating the overall decay rate at each training step, improving robustness and adversarial performance without extra data.”\n\n\n>**Q6:The left subfigure in Fig. 4 does not clearly link spectral variations to generalization.**  \n\n**A6:** Thank you for your comment on Figure 4. We agree the left subfigure shows spectral differences but does not directly link them to generalization. Our main point is that AlphaDecay balances spectral properties and, compared to uniform weight decay, improves generalization as shown by lower perplexity. We will clarify this in the revision.\n  \n\n>**Q7: Incorrect Fig. reference and unclear perplexity description in Fig. 4.**  \n\n**A7:** Thank you for noticing the incorrect figure reference.  We will change Fig. 3 to Fig. 4 in the manuscript.  We also appreciate your suggestion and will specify in the caption of Fig. 4 that the perplexity refers to the validation set.  Thank you again for your valuable feedback.\n  \n\n>**Q8:  Reference [20] and scale-invariant systems.** \n\n**A8:** Thank you for raising this. We cited [20] for its weight decay insights, not to connect our work to scale-invariant systems. Our focus is on weight decay in attention and MLP modules, not scale-invariant components. We will clarify this and avoid overstating [20]'s relevance. Exploring weight decay in scale-invariant architectures is an interesting future direction.\n \n\n>**Q9: Redundant content between lines 74–79 and 88–92.**  \n\n**A9:** Thank you for noting the overlap between lines 74–79 and 88–92. We will consolidate these sections to remove redundancy and improve clarity in the revision.\n\n\n>**Q10: Line 115 references Fig. 3 unnecessarily; line 134 typo; line 135 matrix type.**  \n\n**A10:** Thank you for your helpful feedback. Our responses are as follows:\n\nLine 115: We agree the reference to Fig. 3 is unnecessary and will remove it, as this section discusses Fig. 2.\n\nLine 134: The typo will be corrected from “N modules” to “L modules.”\n\nLine 135:  The matrices are correlation matrices, and the weight matrices are not normalized before calculation; we will clarify this in the revision.\n\nWe appreciate your comments and will update the manuscript accordingly.\n\n>**Q11: The last sentence in Fig. 3 is misleading.**  \n\n**A11:** Thank you for your careful reading of our manuscript and your valuable comments regarding Figure 3, we would like to clarify:\n\n- What Fig. 3 shows: \n\nEach subplot overlays the empirical spectral density (ESD) of a single module under Uniform vs. AlphaDecay for the same 135 M‑parameter network. The shift of the fitted α (numbers in parentheses) illustrates how AlphaDecay raises the α of attention matrices and lowers that of MLP matrices, to provide a deeper insight into the internal representations and spectral properties of the models, \n\n\n- Where performance evidence resides\n\nQuantitative quality metrics are gathered in Table 2 (perplexity, **Uniform:** PPL = 23.14 ,**AlphaDecay:** PPL = 22.55), new reasoning table (above), and the GLUE table (above). We will add an explicit forward reference—“See Table 2 and §4.2 for the associated perplexity gains”—to the current caption.\n\n\nThank you again for your observations, which help us make our exposition more precise."}]

Author rebuttal:

Meta-review: This paper studies the effect of tuning weight decay value for each module. By using HR-SR theory, modules with higher PL_Alpha_Hill values will be assigned larger weight decay values, and modules with lower PL_Alpha_Hill values will be assigned smaller weight decay values.
In the experiments, the decreaseing of validation perplexity shows the effectiveness of the method.
By considering the suggestions of the reviewers, the authors also provide results on QA benchmarks and mathematical formulation.
Decision: Accept (poster)