Why Do We Need Weight Decay in Modern Deep Learning?

Authors: francesco  d'angelo
maksym  andriushchenko
aditya  varre
nicolas  flammarion
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:56:46

Abstract: Weight decay is a broadly used technique for training state-of-the-art deep networks from image classification to large language models. Despite its widespread usage and being extensively studied in the classical literature, its role remains poorly understood for deep learning. In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory. For deep networks on vision tasks trained with multipass SGD, we show how weight decay modifies the optimization dynamics enhancing the ever-present implicit regularization of SGD via the *loss stabilization mechanism*. In contrast, for large language models trained with nearly one-epoch training, we describe how weight decay balances the *bias-variance tradeoff* in stochastic optimization leading to lower training loss and improved training stability. 
Overall, we present a unifying perspective from ResNets on vision tasks to LLMs: weight decay is never useful as an explicit regularizer but instead changes the training dynamics in a desirable way.

Main paper: 
 introduction: 
 The training of modern neural networks broadly falls into two regimes: over-training, which involves multiple passes through a dataset and necessitates effective regularization strategies to avoid overfitting; and under-training, characterized by fewer passes due to large amounts of training data and computational constraints (Hoffmann et al., 2022) . Modern deep learning unequivocally embodies both training regimes: ResNet architectures on computer vision tasks (He et al., 2016) serve as quintessential examples of the over-training regime, while the training of large language models (Brown et al., 2020) stands as a hallmark of the under-training regime. Despite their differences, both regimes extensively adopt weight decay as a regularization technique, though its effectiveness and role remain subjects of ongoing debate. For the first regime, Zhang et al. (2016) showed that even when using weight decay, neural networks can still fully memorize the data, thus questioning its regularization properties. For the second, regularization is inherently unnecessary as the limited number of passes already prevents overfitting. These considerations raise important questions about the necessity and purpose of weight decay, introducing uncertainty about its widespread usage. To illustrate the effect of weight decay in the two regimes, we conduct a simple experiment. We train a ResNet18 on subsets of the CIFAR-5m dataset (Nakkiran et al., 2020) with sizes from 10 000 to 5 mln. The computational budget of each training session is fixed to 5 mln iterations, which amounts to a range of passes between 500 and one. In the over-training regime (left in Fig. 1 ), weight decay does not prevent the models from achieving zero training error, but its presence still improves the test error. Attempting to explain this generalization benefit, recent works (Li & Arora, 2019; Li et al., 2020) bring forth the hypothesis that it is inadequate to think about weight decay as a capacity constraint since it still bears an effect on the training of scale-invariant models. As a result, understanding the effect of weight decay on the optimization dynamics becomes crucial to understanding generalization. Nevertheless, this line of work heavily relies on an effective learning rate (ELR) which only emerges as a consequence of scale-invariance and therefore does not apply to general architectures. In the under-training regime (right in Fig. 1 ), where the generalization gap vanishes, weight decay seem to facilitate faster training for slightly better accuracy. However, a characterization of the mechanisms through which weight decay impacts the training speed in this regime remains underexplored.Our work delves into the mechanisms underlying the benefits of weight decay by training established machine learning models in both regimes: ResNet on popular vision tasks (over-training) and Transformer on text data (under-training) . Towards this goal, we make the following contributions:• In the over-training regime, we unveil the mechanism by which weight decay effectively reduces the generalization gap. We demonstrate that combining weight decay with large learning rates enables non-vanishing SGD noise, which through its implicit regularization controls the norm of the Jacobian leading to improved performance. Moreover, our investigation offers a thorough explanation for the effectiveness of employing exponential moving average and learning rate decay in combination with weight decay.• In the under-training regime, particularly for LLMs trained with one-pass Adam, we confirm experimentally that weight decay does not bring any regularization effect and is simply equivalent to a modified ELR. We explain the training curves commonly observed with weight decay: through this ELR, weight decay better modulates the bias-variance trade-off, resulting in lower loss. Additionally, we show that weight decay has another important practical benefit: enabling stable training with the bfloat16 precision.The concept of employing ℓ 2 weight penalty traces back to studies on the stability of solutions for ill-posed problems (Tikhonov, 1943) . It has since been extensively explored in statistics (Foster, 1961; Hoerl, 1962; Hoerl & Kennard, 1970) . Krogh & Hertz (1991) present one of the earliest systematic studies on weight decay tailored for neural networks. Generalization bounds, such as those by Shalev-Shwartz & Ben-David (2014) , suggest that weight decay can be sufficient for generalization, although not necessary, e.g., due to the implicit regularization of gradient methods (Soudry et al., 2018) . Zhang et al. (2016) argue that while weight decay improves test accuracy, the improvement is not substantial (≈ 1-2% on ImageNet), indicating the key role of implicit regularization. Loshchilov & Hutter (2019) highlight the distinct effects of weight decay and ℓ 2 regularization, particularly for Adam, suggesting that Adam combined with weight decay (AdamW) leads to superior regularization and simpler hyperparameter tuning. For GPT-3 training, Brown et al. (2020) suggest that they include weight decay to provide a small amount of regularization, although we believe it is not the primary reason as we discuss in Sec. 3.Multiple works have focused on weight decay as a tool influencing optimization dynamics.Van Laarhoven (2017) emphasizes that weight decay's impact on scale-invariant networks is primarily seen in terms of an effective learning rate. Zhang et al. (2018) propose three mechanisms of weight decay regularization: (1) increasing the effective learning rate for scale-invariant networks, although as we discuss, the same holds for networks beyond scale invariance (2) approximating the regularization of the input Jacobian for an optimizer inspired by second-order methods, (3) inducing a specific dampening effect in this optimizer. Li & Arora (2019) ; Li et al. (2020) explore the optimization properties of scale-invariant deep networks for which the effective learning rate can be derived. Lewkowycz & Gur-Ari (2020) suggest that the best generalization is achieved with the smallest λ although it necessitates longer training. Additionally, Lewkowycz (2021) propose a criterion for detecting when to decay the learning rate based on the evolution of the weight norm. Bjorck et al. (2021) explore the effect of decoupling weight decay, especially in the early stage of training. Li et al. (2022a) make BERT architecture scale-invariant to enhance training stability and make it more Figure 2 : Training with and w/o weight decay. We report the test error for Resnet18 on CIFAR-10 (2a) and Tiny-ImageNet (2d) trained with and without weight decay and with small and large learning rates. We also include the correspondent EMA, represented by dashed lines. After the first 250 epochs the learning rate is decayed to η = 10 -3 for all the curves. We report also the L2 norm of the parameters (2c) and Train CE (2b) which after the decay converges to the same value for all the runs with the same λ.compatible with standard SGD. Recently, Kosson et al. (2023) show a mechanism through which weight decay balances rotational updates across different layers that motivates a new optimizer.The seminal paper of Krizhevsky et al. (2012) that introduced AlexNet suggest that weight decay serves not only as a regularizer but also reduces the model's training error, functioning as an optimization tool. In recent work, Hoffmann et al. (2022) briefly observe that weight decay enhances the training performance of Adam for training LLMs, but only after ≈ 80% of the total iterations. However, they do not provide an explanation for this behavior, a point we delve into in Sec. 3. 
weight decay in the over-training regime: 
 In this section, we delve into the influence of weight decay in the over-training regime, with a specific focus on image classification tasks. We focus on the training of ResNet models (He et al., 2016) using SGD on Tiny-ImageNet (Wu et al., 2017) and report additional experiments in appendix C for VGG, Resnet32 and scale-invariant Resnet architectures on CIFAR10 and CIFAR100 (Krizhevsky & Hinton, 2009) .Notations and setup. Let (x i , y i ) n i=1 be the training inputs and labels where x i ∈ R d , y i ∈ R c , and c is number of classes. Let h : R p × R d → R c be the hypothesis class of neural network and for any parameter w ∈ R p where the function h w (•) : R d → R c represents the network predictions. The training loss L and the ℓ 2 -regularized loss L λ , for λ ≥ 0, are given by:L(w) = 1 N N i=1 ℓ (y i , h w (x i )) L λ (w) = L(w) + λ 2 w 2 .where ℓ(•, •) : R c × R c → R denotes the cross-entropy (CE) loss function. With i t ∼ U([N ]), the SGD algorithm on L λ (here with batch size 1 and with replacement) with a learning rate (LR) η isEQUATIONAlong the training we track three different iterates: (1) large-LR denoted by w t which use a large constant LR to exploit the SGD noise, (2) fine-tuning wt which starting from w t use a small LR and(3) the exponential moving average (EMA) wt along the large-LR iterates.To understand whether minimizing the regularized objective L λ alone ensures optimal generalization, we compare test errors in Fig. 2a across various settings. Although both large and small LRs minimize the regularized objective, the evidence that optimal performance is achieved exclusively with large LRs indicates that the objective alone is insufficient to explain the benefits of WD or ensure generalization.2 This experiment reaffirms the widely acknowledged consensus that implicit regularization induced by the LR is crucial (Keskar et al., 2016; Li et al., 2019; Andriushchenko et al., 2023) . Despite revealing an interplay between weight decay and large initial LR, the understanding of the corresponding dynamics remains limited. In this section, our goal is to comprehensively understand these dynamics, particularly to elucidate the difference in generalization between training with and without weight decay and using different learning rates, as observed in Fig. 2a . Given the regularization of the ℓ 2 norm of parameters, it is natural to wonder whether weight decay's improvement primarily stems from its ability to control the norm of the trained model. The experiment in Fig. 2c clearly illustrates that distinct training trajectories, while resulting in the same final ℓ 2 norm for parameters, can yield different levels of generalization stating that the ℓ 2 -norm of the learned model's parameters is inconsequential. This observation suggests that once the norm is constrained by weight decay, the critical factor influencing the model's generalization is the subsequent choice of LR. We should note that neural networks can be explicitly made scale-invariant by means of normalization layers and small architectural changes. Li & Arora (2019) have used this setting to reveal that the training dynamics has an effective learning rate which, depending on the L2-norm of the parameters, reduces the effect of WD to merely a scheduler for the learning rate. Our analysis presents a broader perspective that does not depend on scale invariance. At the core of our examination are the unique properties of exponentially tailed loss functions, such as CE: when the data is separable and WD is not applied, the infimum of the loss is at infinity, leading to the unbounded growth of the weight norm (Ji & Telgarsky, 2019; Soudry et al., 2018) . The application of WD, by inhibiting this growth, prevents the decrease of CE loss, which in turn, significantly alters the optimization dynamics.Indeed, examining the parameter norm evolution in Fig. 2c , we notice how it rapidly decreases to stabilize within a small, approximately constant interval. Similarly, the Train CE in Fig. 2b displays a stabilization effect beyond which it cannot decrease without a reduction in the learning rate. We hypothesize that WD enables an optimization dynamic akin to that on the surface of a sphere of certain radius thus triggering the following essential mechanism:Constraining the parameter norm hinders the decrease of the CE, thereby enabling non-vanishing noise in SGD. This allows SGD implicit regularization to unfold and steer the optimization trajectory.Next, we empirically characterize this implicit regularization mechanism.The long-held belief that the implicit regularization property of SGD is pivotal to the generalization capabilities of Neural Networks has been a cornerstone in the field of deep learning (Keskar et al., 2016) . Many theoretical studies (Blanc et al., 2020; Li et al., 2021b; Damian et al., 2021) , attempting to understand this phenomenon, draw upon the essential finding that, in the case of regression and when Gaussian noise is added to the labels, the shape of the covariance of the stochastic gradients matches the shape of the Hessian. This allows Damian et al. (2021) and Pillaud-Vivien et al. (2022) to show that the trajectory of the SGD iterates closely tracks the solution of a regularized problem. In our analysis, we conjecture a similar result; the dynamics of SGD with CE, closely track a regularized process. The important difference in our statement is that unlike Blanc et al. (2020) and Damian et al. (2021) , we do not need to add noise to the labels at each iteration. Instead, weight decay, in combination with large-LR induces a label noise-like behavior via loss stabilization (Andriushchenko et al., 2023) . To better understand the interplay between weight decay, loss stabilization and the noise of SGD, it is convenient to consider the binary classification case where y i ∈ {0, 1}. We also define the Jacobian of the network as J(x i , w) := ∇h w (x i ) ∈ R p and its norm averaged across the dataset: J(w)2 F = 1 N N i=1 Tr ∇h w (x i )∇h w (x i ) ⊤ .Denoting the noise of the gradient by g t = ∇ w L(w t ) -∇ w ℓ (y it , h(w t , x it )), the SGD update in equation 1 becomes:w t+1 = (1 -ηλ)w t -η∇L(w t ) + ηg t .(2)Furthermore, we consider a Gaussian approximation of the SGD noise, matching the first and second moment of g t . A substantial body of research has built upon this approximation and verified its validity (Li et al., 2020 (Li et al., , 2021a;; Smith et al., 2020; Xie et al., 2020; Li et al., 2021b) . In particular Li et al. (2021a) demonstrated how modelling the SGD noise by a Gaussian is sufficient to understand its generalization. The mean is zero due to the unbiased mini-batch gradients: ḡ(w t ) := E[g t ] = 0 whereas for the second moment:EQUATION)In equation 3, we consider ∇L(w t )∇L ⊤ (w t ) negligible compared to ∇ℓ i (w t )∇ℓ ⊤ i (w t ) as the gradient noise variance dominates the square of the gradient noise mean. This fact has been used in previous works (Mori et al., 2022; Zhu et al., 2018; Jastrzebski et al., 2017) and in particular Jastrzebski et al. (2017) and Saxe et al. (2019) empirically verify it. Finally, we assume that the first derivative is approximately constant across all datapoints: ℓ ′ i (w t ) ≈ ℓ ′ j (w t ) ∀i, j and denote this common quantity as σ η,λ (w t ). This last approximation is referred to as "decoupling approximation" and has been empirically verified for classification (Mori et al., 2022) . Furthermore, in App. C.6 we performed additional experiments to verify the decoupling approximation by comparing the spectrum of the SGD covariance with and without this approximation during the large LR phase.Altogether, these considerations lead to the following formulation of the SGD update:w t+1 ≈ (1 -ηλ)w t -η∇L(w t ) - η N σ η,λ (w t ) N i=1 ∇h wt (x i ) ξ t i , where ξ t i ∼ N (0, I). (4)This series of approximations, allows us to define the quantity σ η,λ (w t ), which has a fundamental role in the characterization of the training dynamics. We refer to it as the scale of the noise because it regulates its intensity. Although η and λ influence the noise scale indirectly through the trajectory ofw t , we explicitly highlight this dependence to emphasize our objective: to characterize the influence of WD and LR on the stochastic dynamics of SGD. We develop this characterization building upon the connection between the Jacobian of the network and the covariance of the SGD noise, which motivates the introduction of the following implicit regularization mechanism: Conjecture 1. Consider the algorithm in Eq. 1 with w 0 initialized from a distribution µ 0 R (p) . For any input x, let w t , h(w t , x) be the random variables that denote the iterate at time t and its functional value. The stochastic process (h(w t , x)) t∈N converges to the stationary distribution µ ∞ η,λ (x) with mean μη,λ (x) = h w * η,λ , x for which w * η,λ is a first-order stationary point of the following regularized loss:EQUATIONThe conjecture illustrates how varying noise levels correspond to distinct processes, wherein the mean of each solves a unique regularized problem. Moreover, it describes how each noise level determines the strength of the regularization. Using the mean to formulate the conjecture is a natural choice; even at stationarity, 3 the values of the loss L(w t ) and the regularizer J(w)F would be dominated 3 Assuming the existence of a stationary distribution, the iterates wt are eventually realizations from it. by the noise. To unveil the existence of an implicit regularization and to analyze the evolution of the distribution, we need to look at its summary statistics, in this case, the mean. While insights from Langevin dynamics suggest employing learning rate annealing to converge towards the mean of the stationary distribution, this approach introduces additional complexities which we discuss in Section 2.3. We instead consider an exponential moving average (EMA) ( wt ) t≥0 of the SGD iterates with parameter β = 0.999.The most important implication of the conjecture is that the strength of the regularization σ η,λ depends on both the LR η and the WD parameter λ. Our experiments in Fig. 3 , provide empirical validation for this conjecture. When trained with different combinations of η and λ, the EMA converges to models with different test performances. When fixing λ there exists an optimal value of learning rate η which gives the best test performance while the Jacobian norm monotonically increases. A similar picture can be drawn when fixing the learning rate.Therefore, given two solutions μη l ,λ l and μηs,λs for which Test error(μ η l ,λ l ) < Test error(μ ηs,λs ); the difference in their performances can be explained with the difference in their regularization strengths σ η,λ . The solution μη l ,λ l benefits from better regularization and therefore endows better generalization properties. Furthermore, the heatmap for test error in Fig. 3 indicates that the minimum error is not achieved by a single combination of η and λ, but rather along a contour where their product η × λ appears to be constant. This observation suggests an optimal trade-off between the learning rate and weight decay parameter, characterized by a curve in the parameter space where their product remains constant. Characterizing this relationship might reveal a useful tool which practitioners can adopt to optimally select values of weight decay and learning rate. Fig. 4c , 4d confirm that the product ηλ is the quantity controlling the regularization; combinations of η and λ with the same product show similar test performances and Jacobian norm. For Tiny-Imagenet, the test error exhibits an optimal value for ηλ ∼ 0.005 where increases beyond this point lead to over-regularization and decreases result in under-regularization. Simultaneously, the Jacobian norm J F consistently exhibits a monotonically decreasing trend.To better understand the properties of the noise scale during training, we can observe in Fig. 4a , 4b how higher values of the training loss given by larger ηλ correspond to higher values of the noise scale σ η,λ . The latter is measured by computing the Frobenius norm of the first derivative of the loss with respect to the predictions4 averaged over all training datapoints 1 N N i=1 ℓ ′ i (w) F . The scale of the noise and therefore the strength of the regularization vanish when L(w) ≈ 0 which happens for small values of ηλ. Therefore, WD combined with a large LR stabilizes the CE to a larger value, preventing the noise from vanishing and regulating the implicit regularization. This fact further confirms Conjecture 1, demonstrating that with smaller noise scales, the mean (EMA) tends towards a point where the Jacobian's norm is higher, compared to trajectories with larger noise scales. To further validate our conjecture, we created snapshot ensembles in the ResNet18 on CIFAR-10 setting, by averaging in function space along the SGD trajectory every 10 epochs for the combinations of learning rate (LR) and weight decay (WD) considered. To assess whether the mean of the stationary distribution in function space aligns closely with the EMA, where the Jacobian norm is regularized, we compared the performance of snapshot ensembles with that of the EMA. Additionally, we computed the Total Variation Distance between the softmax outputs of the ensemble and the EMA. The results are reported in App. C.7 and show a strong alignment.The role of the dynamics of the norm. As discussed at the end of previous sub-section, after a rapid initial decrease of norm, the optimization resembles the dynamics of SGD projected onto a sphere. We stress that this is the crucial phase in training and the implicit regularization induced by SGD during this spherical optimization leads to better generalization. To validate this observation and isolate it from the initial norm drop, we investigate the behavior of SGD on a sphere with scale-invariant networks (Li & Arora, 2019) . Scale invariance is chosen for its ease of LR tuning and for comparison with existing works (Kodryan et al., 2022; Li et al., 2020) . Fig. 15 depicts a similar phenomenon as Fig. 3 , where the test error vs. LR exhibits a U-shaped curve. While Kodryan et al. (2022) makes a similar observation, they do not provide an explanation. We demonstrate that the implicit regularization of the Jacobian norm is the key factor, elucidating its dependence on LR.Effective learning rate vs. high training loss. Zhang et al. (2018); Van Laarhoven (2017) explored the relationship between LR and WD, introducing the concept of effective LR η e = η/ w 2 2 . These studies highlight that WD, preventing unbounded growth of the norm, enables the training process to evolve with a higher effective LR. This hypothesis is justified only with scale-invariance (which does not hold for general architecture). Furthermore, the underlying mechanism by which a higher LR enhances generalization is understood only in limited settings (Li et al., 2019) . We propose that a high LR, combined with WD, leads to an increase in σ η,λ . This hypothesis allows us to fully characterize and understand the mechanism through which generalization is enhanced.Mixing in the function space. A simpler conjecture could have been stated in terms of the mixing of the iterates (w t ) t≥0 towards a solution of the regularized objective w * η . However, Li et al. (2020) shows empirical evidence against mixing in the parameter space, emphasizing the necessity of considering the function space. Hence, our conjecture is formulated to capture stationarity in function space.On the benefit of normalization. Our conjecture characterizes the mixing distribution but does not delve into the speed of the mixing process. In our experiments, we observe that normalization layers enable faster mixing. Li et al. (2020) observes a similar phenomenon in the case of scale-invariant networks, specifically the fast equilibrium conjecture, which is addressed by Li et al. (2022b) . We note that this phenomenon persists even when the models are not exactly scale-invariant.The large-LR phase sets the stage for SGD's inherent biases to emerge but to actually exploit such bias, reducing the stochastic noise is necessary. This can be attained in two different ways: averaging (EMA) or decaying the learning rate (fine-tuning), both strategies are widely adopted in practice. This section illustrates the relation between the two and highlights the benefits of using one over the other and the implications for our analysis. From a practical standpoint, implementing EMA is more efficient than LR-decay because it does not require additional gradient iterations or hyperparameter tuning. While both methods enhance performance, their effectiveness is contingent on being combined with loss stabilization, supporting the hypothesis that the noisy dynamics is the underlying factor for their success. Although EMA shows only a slight advantage, our experiments in Fig. 5b, 2a , 2d demonstrate that it consistently outperforms learning rate decay in various settings.When empirically validating Conjecture 1, the EMA is useful to characterize the limit point (i.e., t → ∞) but cannot adequately capture the dynamics throughout the entire trajectory. This limitation arises because different points along the trajectory are at different loss values, making the comparison of any relevant regularized quantities inconsistent. An approach to overcome this inconsistency is to project the iterate w t onto a manifold of constant loss. This can be achieved via early-stopped gradient flow (Li et al., 2021b) (SGD with small LR) on the CE loss with λ = 0 where w t is projected to a nearby point wt , such that L( wt ) ∼ const., ∀t. In practice, this corresponds to fine-tuning via LR-decay. Since after fine-tuning L( wt ) ≈ L( wt ′ ), ∀t, t ′ see Fig. 5a , we can compare J( wt ) F and J( wt ′ ) F and understand its evolution. In the experiments detailed in Fig. 5c , we report J F along the fine-tuned iterates wt and observe a decreasing trend, i.e., the sequence J( wt ) t≥0 is decreasing. This fact empirically validates that the entire trajectory of the iterates (w t ) t≥0 , closely following the trajectory of the fine-tuned iterates ( wt ) t≥0 , bias the model towards a regularized solution that might enhances generalization. This also explains why learning rate schedules, such as step-decay, which starts with a large value and then decrease it, can enhance generalization. Despite providing a straightforward methodology to analyze the trajectory, LR-decay introduces additional complexities that cause deviations from the conjecture. Indeed, in Fig. 5c we observe that the final points of the fine-tuned iterates report the opposite trend compared to the EMA (smaller η lead to larger J F ). This discrepancy is potentially due to the state-dependent nature of the SGD noise covariance in equation 3; decreasing the LR and removing WD can alter the stationary distribution and the regularized objective, leading to a different solution than anticipated by Conjecture 1.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We mention the limitations of our work at the end of Section 4.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
weight decay in the under-training regime: 
 In this section, we investigate how WD enhances optimization in the under-training regime. Although the phenomenon is more general, we focus on LLMs for which one-epoch training is typically used. 2022) but at a much smaller scale. Performing fine-tuning with a tiny LR reveals that a higher starting training loss can still be a better point in terms of the final loss after fine-tuning.Two key effects of weight decay in the under-training regime. WD is widely used in training state-of-the-art LLMs like GPT-3, Chinchilla, and Llama (Brown et al., 2020; Hoffmann et al., 2022; Touvron et al., 2023) . While Brown et al. (2020) suggest that WD offers "a small amount of regularization," its necessity remains unclear in the context of one-pass training where the population loss is directly minimized. As a sanity check, in Fig. 19 in Appendix, we verify that the generalization gap is close to zero even for models trained without WD. Instead of the regularization effect, we suggest that the two most crucial effects of WD in the under-training regime are (1) better optimization of the training loss as briefly observed by Hoffmann et al. ( 2022), (2) prevention of loss divergences under the bfloat16 weight precision. We reproduce this phenomenon at a smaller scale with 124M parameters in Fig. 6 : the final training loss is smaller for λ equal to 0.1 and 0.3 compared to 0. We study both mechanisms which stand in contrast to the over-training regime of Sec. 2, where the primary concerns are not optimization and stability, but rather generalization.Experimental setup. We use the NanoGPT repository (Karpathy, 2023) for training GPT-2 models (Radford et al., 2019) on OpenWebText. We train a 124M parameter model (known as GPT-2-Small) for 50 000 iterations with a batch size of 256. For most experiments, we reduce the default context length from 1024 to 256 to ensure practicality within an academic budget. Alternatively, we could have reduced the number of training iterations or batch size, but this would lead to insufficiently trained models. Unless mentioned otherwise, we train with AdamW using the default LR 0.0006, a short 400-iteration LR warmup, gradient clipping with the ℓ 2 -threshold 1.0, and 10× cosine LR decay. We keep all other hyperparameters at their default values (see App. B).Better optimization with WD is reproducible at a smaller scale. Training lossWD = 0.0 WD = 0.1 WD = 0.3 WD = 0.0, ELR of WD = 0.1, bf16 WD = 0.0, ELR of WD = 0.3, bf16 WD = 0.0, ELR of WD = 0.1, fp32 WD = 0.0, ELR of WD = 0.3, fp32Figure 7 : GPT-2-124M on OpenWebText. Left: The effective LR ηt/∥wt∥2 for the models reported in Fig. 6 . Middle: The LR schedule that matches the effective LR ηt/∥wt∥2 of the runs with weight decay 0.1 and 0.3. Right: Matching the effective LR is sufficient to match the whole training dynamics of the loss if we avoid the loss spikes by using full precision (float32 instead of bfloat16).in terms of the final loss after fine-tuning. Moreover, in Fig. 21 , we show that decoupling WD, as advocated by Loshchilov & Hutter (2019) , is not necessary to achieve this effect: a simple ℓ 2 penalty added to the loss suffices. Lastly, in Fig. 22 , we show that a similar improvement in training loss is observed for SGD with momentum suggesting that adaptive LRs are not key for this phenomenon.Effective LR induced by weight decay in AdamW. We hypothesize that the use of WD for LLM training results in an increased effective LR, even in the absence of scale invariance of the training loss for modern transformer architectures. Here we show that WD in combination with sign SGDutilized as a surrogate for Adam (Balles & Hennig, 2018) -is equivalent to projected SGD on the sphere, with an effective LR η eff ∝ ηt /∥wt∥2, similarly to Van Laarhoven (2017). Consider the update rule of sign SGD on loss ℓ with WD:w t+1 = (1 -η t λ t ) w t -η t sign(∇ℓ t (w t )) = (1 -η t λ t ) w t 2 w t w t 2 - η t • sign(∇ℓ t (w t )) (1 -η t λ t ) w t 2 .Considering the evolution of the direction w := w / w 2 , wt+1 ∝ wt - η t (1 -η t λ t ) w t 2• sign(∇ℓ t (w t )) .When sign(∇ℓ t (w t )) is determined solely by the direction wt , the evolution of the direction of weights becomes the primary matter. This scenario occurs when the function ℓ is scale-invariant or homogeneous. Observing the trend of the gradient norm and the parameter norm w t 2 from Fig. 26 , we note an inverse relationship, i.e., the gradient norm is higher when the parameter norm is lower. This behavior is reminiscent of scale-invariant networks for which ∇ℓ(αw) = 1 α ∇ℓ(αw), for any α ̸ = 0. Thus, controlling parameter norms with WD allows implicit changes to the LR schedule which we verify experimentally next.Matching the effective LR without weight decay. To verify the key role of the ELR ηt /∥wt∥2, we train a model without WD but with ELR corresponding to models trained with λ ∈ {0.1, 0.3} and report results in Fig. 7 . We observe that the whole training dynamics is matched which confirms our hypothesis. This fully explains the observation from Hoffmann et al. (2022) about the advantage of AdamW over Adam: there exists an LR schedule (albeit a non-standard one) shown in Fig. 7 (middle) that leads to the same loss profile as the original AdamW run. However, we note that this holds only for full float32 precision, and models trained with bfloat16 precision diverge in the middle of training. This experience suggests that WD is still necessary in practice to prevent loss divergence. We also note that matching the ELR ηt /∥wt∥2 derived above for sign SGD instead of ηt /∥wt∥ 2 2 for plain SGD (Zhang et al., 2018; Hoffer et al., 2018) is critical for AdamW. Otherwise, the runs diverge very early in training, even with float32 parameter precision.Explaining the training dynamics of AdamW. Classical optimization theory suggests that convergence of SGD-based algorithms primarily depends on two factors: the bias term that influences the rate at which initial conditions are forgotten and the variance term that results from noise in the gradient estimates (Moulines & Bach, 2011) . We argue that these two factors, together with the observation about higher ELR for WD, can explain the loss profiles from Fig. 6 . If we consider the simple case of SGD with a constant LR η applied to a linear least-squares problem, the expected excess risk after t iterations can be bounded as a sum of a bias and variance terms:Excess Risk ≲ (1 -ηµ) t w 0 -w * 2 + ησ 2 ,where σ is a uniform bound on the variance of the noise of gradient estimates, µ a lower bound on the objective function's Hessian, w 0 the initial point and w * the optimum. For linear models, it is well-established that a larger LR accelerates the contraction of the bias term but has a detrimental impact on the variance term, ultimately leading the variance term to dominate. Coming back to the dynamics in Fig. 6 , with a large ELR at the start, the convergence becomes primarily bottlenecked by the high variance term proprtional to the learning rate, leading to higher loss values in the presence of WD. Conversely, towards the end of training, when ELR and the variance term are reduced, we see that WD catches up and performs better at the end, thanks to its relatively higher ELR throughout the training and thus better bias contraction. This perspective sheds light on the observation that EMA for LLMs is most advantageous when employed with large LRs (Sanyal et al., 2023) as we also illustrate in Fig. 23 . As the variance dominates in this case, variance reduction of the averaging helps.Experiments with bfloat16. Training in reduced precision is essential for speeding up training and reducing GPU memory requirements (Kalamkar et al., 2019) . We further elaborate on the fact that WD is not fully equivalent to higher ELR and remains necessary for stable bfloat16 training. While Scao et al. (2022) observe that usage of float16 can cause loss divergences, bfloat16 is considered much more stable and is de-facto standard in LLM training. Although bfloat16 shares the same floating-point exponent size as float32 (thus, the range of possible values is the same), it offers lower precision, with only 7 bits for the fraction instead of 23. We observe that even more stable bfloat16 data type can still exhibit late-training spikes that irreparably harm model performance in standard practical settings, such as with a larger context length (e.g., 1024 instead of 256 as in the previous experiments). Therefore, we focus on this configuration for the experiments shown in Fig. 8 . Runs with a moderate LR 0.001 (the default LR of Adam) without WD exhibit late-training divergence for all random seeds when using bfloat16. By comparison, training with float32 remains entirely stable. Importantly, we observe that the model does not recover after the loss spikes which contrasts with the loss spikes described in the Edge of Stability phenomenon (Cohen et al., 2021 (Cohen et al., , 2022)) .We emphasize that all these runs use gradient clipping with the standard ℓ 2 -threshold. Finally, we observe that divergences can be prevented by reducing the LR, e.g., from 0.001 to 0.0006. However, this adjustment leads to slower training, as illustrated in Fig. 24 in the Appendix. Instead, the most effective approach is to use a higher LR of 0.001 with WD, which enables stable bfloat16 training and yields a better final training loss.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: We do have new standalone theoretical results like theorems or lemma, but we provide short derivations presented in Section 2.2, Section 3, and Appendix C.2 that support our experimental results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
conclusions: 
 In this paper, we demonstrate how weight decay, a single hyperparameter, can manifest three distinct effects across different training regimes: it offers regularization when combined with stochastic noise, improves optimization of the training loss, and guarantees stability in low-precision training environments. In the over-training regime, the scale of the noise σ η,λ is the fundamental quantity governing the implicit regularization strength of SGD. Weight decay combined with large LR enables the noisy dynamics to evolve by maintaining the scale at a constant level. Techniques such as EMA or fine-tuning work by reducing noise, thereby allowing for the effective exploitation of the accumulated hidden regularization. Coming to the under-training regime, AdamW (Loshchilov & Hutter, 2019) was introduced as a regularization method. Instead, we argue that it is effective as it modulates the ELR to attune the bias-variance tradeoff. In addition, it also improves the stability of training. In summary, weight decay is seldom valuable as an explicit regularizer; instead, its widespread adoption can be attributed to its ability to induce desirable changes in optimization dynamics. We also acknowledge limitations of our work: given our limited computational resources, we do not conduct truly large-scale experiments. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide detailed settings of our experiments in Appendix B. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?• It is to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide these details in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We confirm that our submissions complies with the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: In our opinion, this work does not directly lead to societal impacts since it focuses on a general improvement of our understanding of deep learning optimization methods.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the code, data, and models we use in our work, most of them directly when we discuss the corresponding experimental results (Section 2 and Section 3) and some of them in the appendix (Appendix B).Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: Our focus in Section 2 is on an empirical illustration of the implicit regularization phenomenon, hence we refrain from attempting to prove this general conjecture, which we believe is a challenging task. The existing theoretical works Blanc et al. (2020) ; Li et al. (2021b) ; Damian et al. (2021) present two major weaknesses; they are essentially limiting analysis and as such fail at capturing the entire optimization trajectory and they primarily target regression tasks. The powerful mathematical framework for scale-invariant networks developed by Li & Arora (2019) ; Li et al. (2020) allows them to study in detail the benefits of normalization and its interplay with weight decay. By means of this framework, they state a fast equilibrium conjecture, which gives qualitative guarantees for the speed of convergence of the stochastic process to the stationary distribution in function space.They disentangle the evolution of the norm and the direction of the parameters and show how the evolution of the direction only depends on the intrinsic LR λ i = ηλ. However, a qualitative description of the stationary distribution, its dependence on this intrinsic LR and the relationship with generalization is missing Li et al. (2020) . We attempt to fill this gap by providing a qualitative depiction of the stationary distribution and its dependence on the intrinsic LR shading some light towards understanding the relationship with generalization. The work of Kodryan et al. (2022) reports a similar observation, where the best test loss is achieved at a LR where the loss neither converges nor diverges but does not provide any explanation. Full experimental details are available in our public repository https://github.com/tml-epfl/ why-weight-decay but we also list the main training details here. All the experiments are conducted for 3 different random seeds, the error-bars report one standard deviation. CIFAR-10/100 experiments. We train a VGG network without BatchNorm and preactivation ResNet-18 on CIFAR-10 and ResNet-34 on CIFAR-100 without data augmentations. We use standard SGD without momentum for all experiments. We note that ℓ 2 regularization and weight decay are exactly the same in this case. We use the standard He initialization (He et al., 2015) for all parameters. To make ResNets scale-invariant, we follow the approach of Li et al. Tiny-ImageNet experiments. We train Resnet-18 without data augmentation. We use standard SGD without momentum in all our experiments. We use the following learning rates (0.0005, 0.0010, 0.0050, 0.0100, 0.0500, 0.1000, 0.1500, 0.2000, 0.2500) and weight decay parameter (0.0200, 0.0150, 0.0125, 0.0100, 0.0075, 0.0050, 0.0025, 0.0010, 0.0005, 0.0000). To measure the norm of the Jacobian we use a subset of the training data of 2500 examples. Each run requires approximately 5GPU hour on A100.LLM experiments. We use the NanoGPT repository (Karpathy, 2023) for training GPT-2 models (Radford et al., 2019) on OpenWebText (Gokaslan et al., 2019) . All training documents are concatenated in a single stream from which a new batch is sampled with replacement on every iteration of training. We train a 124M parameter model known as GPT-2-small for 50 000 iterations instead of the default 600 000 to make grid searches over the learning rate and weight decay parameters more accessible within an academic budget. We mostly use the context length of 256 for faster experiments except for Fig. 8 where we use the context length of 1024 since we observed that a larger context length is crucial to observe loss divergences with moderate learning rates (such as 0.001 for Adam). We train with AdamW (Loshchilov & Hutter, 2019) using batch size 256, default LR 0.0006 (unless mentioned otherwise), β 1 = 0.9, β 2 = 0.95, a short 400-iteration LR warmup, and 10× cosine LR decay. For the runs with SGD with momentum, we use the learning rate 0.3 and momentum parameter 0.9 using the same LR schedule as for AdamW. We initialize all parameters with the standard deviation equal to 0.02. We keep all other hyperparameters at their default values as in the NanoGPT repository. We perform all experiments on A100 Nvidia GPUs that support fast bfloat16 training. Each training run of GPT-2-small for 50 000 iterations takes around 12 hours on a single GPU.Here, we plot an illustrative graphic in Fig. 9 to give an idea of what happens during the fine-tuning phase. Proof. The noise in the case when the gradient is computed at(x i , y i ) is g(w) = ℓ ′ (y i , h(w, x i ))∇h(w, x i ) - 1 n i ∇ℓ ′ (y i , h(w, x i ))∇h(w, x i ),Taking the expectation over uniform sampling over i, we have,E g 2 = 1 n n i=1 ℓ ′ (y i , h(w, x i ) 2 ∇h(w, x i ) 2 - 1 n 2 i ∇ℓ ′ (y i , h(w, x i ))∇h(w, x i ) 2 (6)Upper bound: Using the self-bounding property of the binary CE, i.e., ℓ ′2 ≤ l and ∇h (w, x)2 ≤ M 2 . E g 2 ≤ M 2 1 n n i=1 ℓ(y i , h(w, x i )) = M 2 L(w).Comment on the Lower bound: Since the iterates are bound, we can assume there exists a constant c such that ℓ ′2 ≥ cl. as the second term in 6 is decreasing with O(n -2 ), we can assume that the first term is dominating and relevant and can lower bound the first term as,E g 2 ≥ cm 2 1 n n i=1 ℓ(y i , h(w, x i )) = cm 2 L(w).In this section, we report additional experimental results related to Section 2 in the main text. In Fig. 10 we report analogous results for the jacobian norm and test error of the EMA for Resnet18 on CIFAR10. In Fig. 11 and 2 we report the train CE for VGG and ResNet18 on CIFAR-10 and ResNet34 trained on CIFAR-100. We can observe how when weight decay is used in combination with large LR, the train CE stabilizes at some approximately constant level. To measure the cosine similarity (CS) between H(w) and the covariance Σ t we computeCS = E [cos (H(w)v, Σ t v)]where v is sampled from the Gaussian distribution in R p and cos(u, v) = ⟨u,v⟩ / u v . The results are reported in Fig. 13 . As reported in Table 1 previous works related with label noise, theoretically derived the connection between the trajectory of SGD and a trajectory which regularizes either the trace of the Hessian or the Jacobian norm. The Hessian of a loss function L can be decomposed as:∇ 2 L(w) = N i=1 ∇h(x i ; w) ∇ 2 h l(h(x i ; w)) ∇h(x i ; w) ⊤ Gi(w) + K c=1 [∇ h l(h(x i ; w))] c ∇ 2 h(x i ; w) Ei(w).Many works demonstrated empirically that the G i is the dominant part of the Hessian decomposition and ∇ 2 L(w) ∼ i G i (Sagun et al., 2017) . The Jacobian (J) norm instead is defined as:EQUATIONin the case of square loss, ∇ 2 h l = I where I is the identity matrix. Hence, Tr ∇ 2 L(w) ∼ J 2 F . The similarity is an exact equality at an interpolating solution since ∇ h l(h(x i ; w)) = 0 and therefore not much ambiguity is left regarding which quantity to study. However, in the case of classification, this fact does not hold. In particular, the trace of ∇ 2 h l can significantly deviate from the identity matrix and varies depending on the value of the training loss. Consequently, although the two quantities seem closely related even when using the CE, we opt for analyzing J F . This choice is motivated by its lack of explicit dependence on the training loss, enabling straightforward comparisons between different solutions.In the following, we report an empirical comparison of the two quantities along the training trajectory. In Fig. ( 14) we report a comparison between the trace of the Hessian and the Jacobian norm for both the fine-tuned iterates and the EMA. We can observe that for the finetuned iterates, both quantities display a decreasing trend; nevertheless, the ranking appears to be inverted to what is expected from previous theoretical works, i.e. larger LRs should lead to a stronger regularization effect. If we compare the EMA instead, we can see that both quantities are still decreasing along the iterations but for the Hessian we don't observe any meaningful ranking for the final solutions whereas the Jacobian norm is lower for higher LRs. In order to isolate the implicit regularization mechanism from the large initial drop and even small fluctuations in the dynamics of the ℓ 2 norm, we consider a simplified setting. We train scale-invariant networks (Li & Arora, 2019; Li et al., 2020) with projected SGD on the unitary sphere S (p-1) . We project the SGD iterates on the unitary sphere S (p-1) within the context of scale-invariant ResNet architectures (Li & Arora, 2019; Li et al., 2020) . This setup is helpful for two reasons: (a) it simplifies the selection of the LR and hence tremendously reduces the experimental overhead (b) scale-invariant networks have been extensively studied in previous works Li & Arora (2019) ; Li et al. (2020) ; Kodryan et al. (2022) . Moreover, in the context of scale invariance, the optimization on the sphere is the natural object to study as the evolution of the direction is the only quantity that matters. The projected SGD update writes asw t+1 = Π S (p-1) (w t -η∇ w ℓ (y it , h(w t , x it ))) (8)where Π S (p-1) : w → w / w 2 .(9)The training framework still consists of two phases separated by a LR decay. The primary insight from our experiments on the sphere is depicted in Fig. 15 : the test performance achieved in the fine-tuning phase depends on the LR used in the large-LR phase and, moreover, there is an optimal value. The work of Kodryan et al. (2022) reports a similar observation, where the best test loss is achieved at a LR where the loss neither converges too fast nor diverges but doesn't provide any explanation. Once again, our investigation reveals that the key to understand this behaviour and the dependence on the LR lies in the noisy dynamics in the large LR phase which closely tracks a regularized process. To summarize this idea we postulate a conjecture similar to the one reported in Section 2.2. Conjecture 3. Consider the algorithm Eq. 8 with w 0 initialized from a distribution µ 0 S (p-1) . For any input x, let w t , h(w t , x) be the random variables that denote the iterate at time t and its functional value. The stochastic process (h(w t , x)) will converge to a stationary distribution µ ∞ η (x) with mean μη (x) for which w * η is a first-order stationary point of the following regularized loss: EQUATIONWe empirically verify the validity of the "decoupling approximation" introduced in Sec. 2.2 we use Stochastic Lanczos Quadrature (Yao et al., 2020) to estimate the empirical spectral density of the SGD covariance with and without the decoupling approximation, during the large-LR phase. The experiments are performed for ResNet-18 trained on the cifar10 dataset for different combinations of learning rate and weight decay which are used in the manuscript. The results in Figures 16, 17 , and 18 illustrate a substantial overlap in the two spectra which serves as a validation of the reliability of our approximation. To further validate our conjecture, we performed additional experiments. Specifically, within the same ResNet18 on CIFAR-10 setting as in our main experiments, we created snapshot ensembles (Huang et al., 2017) by averaging in function space along the SGD trajectory every 10 epochs for the combinations of learning rate (LR) and weight decay (WD) considered in the paper.To assess whether the mean of the stationary distribution in function space aligns closely with the EMA, where the Jacobian norm is regularized, we compared the performance of snapshot ensembles with that of the EMA. Additionally, we computed the Total Variation Distance D T V between the softmax outputs of the ensemble and the EMA on the Test setD T V = 1 2N N i=1 C j=1 p (i) ensemble,j -p (i) EMA,j .The results in Table 2 show a strong alignment in test accuracies, while those in Table 3 indicate a low Total Variation across all combinations. Together, these findings offer further validation for our conjecture. We present the following additional figures related to the LLM experiments. We show that the validation loss of a GPT-2-124M model is determined by the training loss and not influenced by λ in Fig. 19 . We also show that the generalization gap stays close to zero throughout training for different λ for both 124M and 774M parameter models. We show the results for models trained weight decay on LayerNorm weights in Fig. 20 . We see that penalizing all parameters in weight decay (i.e., including the LayerNorm parameters) leads to the same effect for smaller λ (like 0.1) but underperforms on larger λ (like 0.3). Note that when WD is applied on all weights, this changes the optimal value of the objective. In Fig. 21 , we train models with ℓ 2 regularization instead of decoupled weight decay as in AdamW (Loshchilov & Hutter, 2019) . We observe that ℓ 2 regularization instead of weight decay leads to the same effect as decoupled weight decay (Loshchilov & Hutter, 2019) . We train models using SGD with momentum and show the results in Fig. 22 . We see that weight decay leads to a similar improvement in training loss for SGD with momentum as well. We show multiple metrics in Fig. 26 for the models shown in Fig. 6 : gradient variance, gradient norm, and weight norm plots that complement Fig. 7 in the main part. In Fig. 23 , we show results of weight averaging that suggests the suboptimality gap between runs with different λ is much smaller than what the loss at w t suggests. However, weight averaging is still less effective than fine-tuning with a tiny LR as in Fig. 6 . Finally, in Fig. 25 , we show results of models trained context length 1024. We see that the training loss over iterations for models trained with a range of LR and WD (all are bfloat16). All runs with LR smaller than 0.001 successfully converge but the final training loss is higher than for LR 0.001.In addition, we observe that lower learning rates prevent the weights from growing too much. ) shows that the suboptimality gap between runs with different λ is much smaller than what the loss at wt suggests. However, weight averaging is still less effective than fine-tuning with a tiny LR as in Fig. 6 . Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims in the abstract and introduction accurately reflect all our contributions presented in the paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.The red, blue and green curves have the same CE 2b and norm 2c, hence same L λ but different test error 2a.In the multi-class setting ℓ ′ i (wt) ∈ R c ; to quantify the scale, we compute its Frobenius norm. 


Decision: This paper studies the role of weight decay in the modern learning regimes of over-training computer vision models and under-training language models. The authors show that the weight decay does not act as a simple explicit regularizer, but as a component that affects the training dynamics in different ways for over-training and under-training. 

This paper provides new insights that contribute to the ongoing efforts to explain the deep learning practice. The authors base their claims by extensive experiments and their explanations use nonrigorous mathematical formulations of the SGD dynamics. All of these, sufficiently support the authors’ claims. 


The reviews reflect that the contributions of this paper are of sufficient interest and significance. The provided reviews are thorough, and the authors succeeded to improve their paper during the rebuttal and discussion with the reviewers. Accordingly, Reviewers S7kS and tBnp increased their review scores. It is worth to elaborate here on the following points that appeared in the Authors-Reviewers discussion and later in the AC-Reviewers discussion: 
1. *Actionable insights and guidelines:*  This paper adds to the foundational understanding of the deep learning practice. Therefore, the understanding gained from this paper can help experts in machine learning and optimization to further improve the practical guidelines related to weight decay. In addition, the authors' results add to the existing literature by further emphasizing the role of the product of the weight decay and learning rate hyperparameters, which is relevant for tuning these hyperparameters in practice. Following my discussion with the reviewers, I believe that these contributions are of sufficient interest for publication. 
2. *Assumptions and approximations:*  While the reviews considered some of the mathematical approximations as potential weaknesses of this work, the authors emphasize in their rebuttal and discussion that the approximations need to hold only in the stabilization phase and not throughout the entire SGD optimization trajectory. This seems adequate, considering that this paper does not intend to provide a rigorous theory but to empirically explain the practice.


Based on reading the paper, the reviews, the authors’ rebuttal and their discussion with the reviewers, as well as my discussion with the reviewers, I find this paper of sufficient quality and significance for publication. My recommendation is therefore to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper explores the role of weight decay (WD) in training modern deep neural networks, focusing on its impact on optimization dynamics in two distinct training regimes: over-training (multiple passes through the dataset) and under-training (limited passes due to large dataset sizes and computational constraints). 

In the over-training regime, the authors demonstrate that WD does not act as a capacity constraint but influences optimization dynamics by enhancing implicit regularization of SGD, as well as combines with large learning rates to maintain non-vanishing SGD noise, improving generalization by controlling the norm of the Jacobian. 

For large language models (LLMs) trained with limited data passes, the authors discuss the relationship between WD and the effective learning rate, and how WD can prevent divergence in training with reduced precision, such as bfloat16.
strength_weakness: strength_weakness: 1. While the practical implications are well-explored, the theoretical explanations could be helpful. Some claims about implicit regularization and optimization dynamics are not fully explained.

2. The study focuses on specific architectures like ResNet-18. It would be beneficial to see similar analyses on a broader range of models to strengthen the generality of the conclusions.
questions: questions: Please refer to weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: This paper studies the effect of weight decay (WD) in the over-training and under-training regimes. It hypothesizes and empirically verifies different roles that WD plays in these scenarios. Specifically, in the over-training regime, WD primarily induces the implicit regularization of SGD via controlling the noise scale $\sigma_{\eta,\gamma}$. In the under-training regime, WD serves to stabilize and improve the optimization of the training loss as well as enabling better stability in low-precision training settings.
strength_weakness: strength_weakness: 1. Some figures could be improved and(or) better explained. For example, Figure 2c presents results on the evolution of norm of weights and the correlation between the norm and the test error. It is quite difficult to cross reference which lines correspond to which test errors. It would be better if the authors could pinpoint specific lines in the figure or in the caption which lines one should look at. This would improve readability of the experimental results. Similarly for figures such as Figure 4.
2. EMA and finetuning would introduce additional hyperparameters (i.e. the momentum parameter and the smaller lr respectively). In the paper and appendix, I did not find their specific settings. The effect of these hyperparameters on the optimization is not, but should be, examined in the paper.
questions: questions: 1. The statement that EMA or fine-tuning “allows for the effective exploitation of the accumulated hidden regularization” is not entirely clear to me. Does generalization require a smaller scale of the stochastic noise and why? I feel this point is not adequately addressed or relevant prior work is not cited. In addition, if this is true, the current message seems to suggest that a large LR, combined with WD, is required to induce the stochastic noise but its scale has to be tuned down for better generalization. Is it not possible to induce the noise of appropriate scale at the very beginning with a careful selection of LR and WD? 
2. Related to the previous question, some works actually use a warm-up schedule for the WD, together with LR decay(e.g. DINO: https://arxiv.org/abs/2104.14294). This would lead to a constant noise scale $\sigma_{\eta,\gamma}$. This seems to contradict with the previous statement about the necessity of a smaller noise scale. I am interested in learning about the authors’ thoughts about this. 
3. The current work studies the training of networks from scratch. Would these conclusions/observations carry over to the fine-tuning of networks (e.g. LoRA)?
limitations: limitations: The authors note the limitations of the current work as the experiments reported are of smaller scale due to computational constraints.
paper_summary: paper_summary: In the paper the authors investigate WD in current deep learning practices. The authors argue that for deep networks used in vision tasks, WD enhances implicit regularisation by modifying the optimisation through loss stabilisation. They also argue that for LLMs, WD helps balance the bias-variance trade-off, leading to improved training stability. The study offers a unified perspective on the role of WD across different training regimes, suggesting that WD’s benefits are more related to changes in training dynamics than to explicit regularization effects.
strength_weakness: strength_weakness: The weaknesses are as follows:
- The study primarily focuses on ResNets and GPT-2. It would have been interesting to see how the results scale to larger models, such as the open source Llama-3. This fact makes it harder to generalise the results.
- No comparison with other regularisation methods, like batch norm or dropout. I'm not saying this is necessary for the paper to be accepted but it would be a considerable added strength to have at least some of the experiments on other regularising techniques for comparison
- The study assumes fixed hyperparameters in many experiments, which may not reflect the dynamic tuning often required in real-world training
- Lack of justification and explanation around assumptions (see questions)

Minor comments:
- Font in figures is far too small and thin, making it very hard to read
- The use of "weight decay" vs "WD" in the main text is mixed and inconsistent
questions: questions: - There is an assumption that the first derivative is approximately constant across all datapoints (eq 3), what is this assumption based on? Will this hold across all types of data and data complexity, including multimodal data?
- Related to the previous question, does the data have to be homogenous? 
- In the sections discussing the benefits of weight decay with bfloat16 precision, could the authors elaborate on the mechanisms through which weight decay interacts with lower precision calculations to enhance stability?
limitations: limitations: The discussion on limitations is severely lacking.

There is one mention of a limitation of computational resources which hinder large scale experiments but I cannot find any other mention of limitations. This is not sufficient. The authors make a number of assumptions in their experimental setups and in the interpretation of the findings (I mention only a few in my weakness list). The authors must provide a more thorough discussion of the limitations of their work.
paper_summary: paper_summary: This paper investigates the role of weight decay (WD) in modern deep learning, differentiating its effects in over-training and under-training regimes. Through experiments with ResNets on vision tasks (over-training) and Transformers on text data (under-training), the authors demonstrate that WD’s primary benefit is not explicit regularization, but rather its influence on optimization dynamics.  In the over-training regime, WD with large learning rates leads to non-vanishing SGD noise, controlling the Jacobian norm and improving generalization. In the under-training regime, WD functions as a modified effective learning rate, improving training loss and stability, particularly with bfloat16 precision.
strength_weakness: strength_weakness: * Although the paper provides a compelling narrative connecting WD's influence to optimization dynamics, it doesn't offer concrete, actionable guidelines for practitioners on how to tune WD for different model architectures or datasets. Insights partially boil down to "instead of weight decay, other regularizer that avoid loss-collapse would have a similar effect".  
* The theoretical analysis, while providing some interesting insights, rely heavily on approximations and lacks formal mathematical proofs for the proposed conjectures. This leaves room for inaccuracies not covered by the simplified theoretical framework.
* Especially the language experiments in the under-training regime use tiny models by today's standard. One can indeed hope though, that the results generalize to bigger architectures.
questions: questions: From Figures 3 and 10, but also my own experience: A meaningful improvement in test-performance often requires carefully tuned WD, within at most a factor of 2. Would you agree with that observation? Many of the experiments show ablations with significantly larger differences in WD strength. Is that an issue for any of the conclusions described in the paper?
limitations: limitations: See above.
paper_summary: paper_summary: In this work, the role of weight decay (WD) in contemporary deep learning is investigated. The authors note that interpreting weight decay as a classical $L_2$ regularizer seems less relevant for modern deep learning. Instead, it is suggested to separately regard WD from positions of over- and under-training. In the over-training regime, typical for common image classification with ResNets on CIFAR-like datasets, weight decay paired with learning rate acts as an implicit regularizer on the Jacobian of the model, controlling the scale of the noise defined by the derivative of the loss. Optimal combination of weight decay and learning rate thus yields the best generalization due to a balanced regularization. In the under-training regime, attributed to a single-pass LLMs training, WD takes a different role controlling the effective learning rate of the training process and saving it from numerical instabilities.
strength_weakness: strength_weakness: **Over-training regime**

My personal expertise is the over-training regime, hence most questions and weaknesses are attributed to the first part of the work.
1. The mechanism, by which WD helps in the over-training regime via constraining the optimization on a sphere, as described in lines 137-138, is unconvincing. Would similar reasoning be applicable to other loss functions not requiring infinite increase of parameters norms, such as MSE, or fully scale-invariant models, which also benefit from the use of WD during training but do not depend on the parameters norm by definition?
2. Equation 3 involves a series of unrealistic approximations. First of all, where did the total loss gradient go after the first approximation? According to the previous assumption, the loss is separated from its minimum value by WD, thus its gradient must be substantially non-zero. Second, the assumption that the first derivative is approximately constant across all data points is too stringent as the loss on individual objects has a heavy-tailed distribution in practice, where the mean and median values may be very distinct.
3. The Gaussian approximation of the SGD noise is questionable [1-3] and must be justified experimentally or at least by citing related work.
4. Conjecture 1 is formulated in a concrete form, however only proxy values are measured to justify it. (see Question 3 below for further detail). 
5. It is unclear how regularization of model Jacobian and generalization are related in the first place, as trends plotted in Fig. 3 and 4 could be mere correlations caused by other intrinsic factors [4]. In particular, in Figure 5, the authors note that the Jacobian norm trends for EMA and fine-tuning are inverse, making it hard to conclude whether the Jacobian norm is regularized or actually fostered during the large-LR stage of training.
6. Dependence of the training dynamics on the product of learning rate and weight decay is not novel and was particularly studied, for example, in [5] where this value is called intrinsic learning rate.

**Under-training regime**

Overall, I like this part of the work more, especially the derivation of the effective learning rate for Adam and its empirical examination. However, I still have a few concerns.

7. The informal explanation of the AdamW training dynamics using bias-variance reasoning is vary vague and not supported by any experiments. I suppose that similar linear least-squares arguments could be similarly applied to the over-training regime, where WD is expected to play a different role according to the claims of the paper.
8. Although empirically validated, no explanation is offered as to how the use of WD allows avoiding numerical instabilities in LLM training.

[1] Simsekli, Umut, Levent Sagun, and Mert Gurbuzbalaban. "A tail-index analysis of stochastic gradient noise in deep neural networks." International Conference on Machine Learning. PMLR, 2019.

[2] Panigrahi, Abhishek, et al. "Non-Gaussianity of stochastic gradient noise." arXiv preprint arXiv:1910.09626 (2019).

[3] Zhou, Pan, et al. "Towards theoretically understanding why sgd generalizes better than adam in deep learning." Advances in Neural Information Processing Systems 33 (2020): 21285-21296.

[4] Andriushchenko, Maksym, et al. "A Modern Look at the Relationship between Sharpness and Generalization." International Conference on Machine Learning. PMLR, 2023.

[5] Li, Zhiyuan, Kaifeng Lyu, and Sanjeev Arora. "Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate." Advances in Neural Information Processing Systems 33 (2020): 14544-14555.
questions: questions: 1. Figure 2: How come the huge spikes of the dark blue line in the training loss and test error are absolutely not reflected in the weight norm?
2. Equation 4: How exactly does this formula approximate the SGD update in eq. (2)? Where is the WD factor $(1 - \eta \lambda)$ before $w_t$? Why is a single $g_t$ term replaced with a sum of normally distributed values? 
3. According to the formulation of Conjecture 1, it seems more natural to perform MCMC ensembling of the models in the training trajectory to approximate the mean value $\bar{\mu}_{\eta, \lambda} (x)$ rather than doing EMA or fine-tuning (the authors discuss limitations of these methods in Sec. 2.3). I would suggest to conduct such experiment to more accurately validate the conjecture.  
4. Does Fig. 4 also report the EMA results? 
5. How does the authors derivation of the effective learning rate for Adam compare with a similar prior result obtained for Adam in the scale-invariant case [1]? I acknowledge that the authors result could be more general, albeit it still requires independence of the gradient sign from the parameters scale akin to the scale-invariant case. 

[1] Roburin, Simon, et al. "Spherical perspective on learning with normalization layers." Neurocomputing 487 (2022): 66-74.
limitations: limitations: The authors partly discuss the limitations in the conclusion of the work.

Comments: title: Reviewer's response to additional experiments
Additional experiments with Snapshot Ensembles
Additional experiment suggested by reviewer tBnp
Reviewer's response
comment: Thank you for experimentally validating my idea!

I find the results quite interesting and convincing. I would like to increase my score be one more point and suggest the authors including these results in the next revision, as they significantly strengthen the claims of the paper.
Thank you for your responses and clarification. Overall, I find the paper and the additional experiments to be insightful. I will maintain my score and recommend for its acceptance.
We appreciate your suggestion for the additional experiment and your consideration of increasing your score.  We conducted the experiment as you suggested, and the results have been posted as a general comment above. We hope these new findings help to further clarify the doubts you raised.
To further validate our conjecture, we performed the experiment suggested by reviewer tBnp. Specifically, within the same ResNet18 on CIFAR-10 setting as in our main experiments, we created snapshot ensembles by averaging in function space along the SGD trajectory every 10 epochs for the combinations of learning rate (LR) and weight decay (WD) considered in the paper.

To assess whether the mean of the stationary distribution in function space aligns closely with the EMA, where the Jacobian norm is regularized, we compared the performance of snapshot ensembles with that of the EMA. Additionally, we computed the Total Variation Distance (TVD) between the softmax outputs of the ensemble and the EMA on the Test set $TVD = \frac{1}{2N} \sum_{i=1}^{N} \sum_{j=1}^{C} \left| p^{(i)}_{\text{ensemble}, j} - p^{(i)}\_{EMA,j} \right|$
. The results in **Table 1** show a strong alignment in test accuracies, while those in **Table 2** indicate a low Total Variation across all combinations. Together, these findings offer further validation for our conjecture.

We believe these additional experiments contribute to strengthening our contributions. While time constraints limited us to running these experiments on CIFAR-10, we plan to extend them to TinyImageNet and include results from both datasets in the revised version of the manuscript.



| WD     |**LR=0.001**|**LR=0.001**|**LR=0.005**|**LR=0.005**|**LR=0.01**|**LR=0.01**|**LR=0.025**|**LR=0.025**|**LR=0.05**|**LR=0.05**|**LR=0.1**| **LR=0.1**  | **LR=0.15** |**LR=0.15**|
|--------|-----------|--------|-----------|--------|----------|--------|-----------|--------|----------|--------|---------|--------|----------|--------|
|        | Ensemble  |  EMA   | Ensemble  |  EMA   | Ensemble |  EMA   | Ensemble  |  EMA   | Ensemble |  EMA   | Ensemble|  EMA   | Ensemble |  EMA   |
| 0.0000 | 0.32      | 0.33   | 0.27      | 0.26   | 0.24     | 0.25   | 0.17      | 0.17   | 0.17     | 0.17   | 0.13    | 0.13   | 0.13     | 0.13   |
| 0.0005 | 0.32      | 0.32   | 0.29      | 0.29   | 0.24     | 0.24   | 0.18      | 0.17   | 0.13     | 0.16   | 0.13    | 0.13   | 0.11     | 0.13   |
| 0.0010 | 0.32      | 0.33   | 0.25      | 0.27   | 0.21     | 0.21   | 0.13      | 0.19   | 0.10     | 0.13   | 0.10    | 0.11   | 0.11     | 0.11   |
| 0.0015 | 0.32      | 0.34   | 0.23      | 0.22   | 0.22     | 0.25   | 0.11      | 0.14   | 0.10     | 0.12   | 0.10    | 0.10   | 0.09     | 0.09   |
| 0.0025 | 0.30      | 0.30   | 0.22      | 0.22   | 0.19     | 0.20   | 0.09      | 0.10   | 0.10     | 0.11   | 0.10    | 0.10   | 0.10     | 0.11   |
| 0.0050 | 0.33      | 0.34   | 0.21      | 0.20   | 0.12     | 0.16   | 0.10      | 0.10   | 0.10     | 0.10   | 0.09    | 0.09   | 0.10     | 0.09   |
| 0.0075 | 0.35      | 0.37   | 0.15      | 0.16   | 0.10     | 0.11   | 0.11      | 0.11   | 0.09     | 0.08   | 0.10    | 0.09   | 0.12     | 0.10   |
| 0.0100 | 0.31      | 0.34   | 0.13      | 0.15   | 0.11     | 0.11   | 0.10      | 0.10   | 0.11     | 0.10   | 0.11    | 0.09   | 0.13     | 0.13   |

**Table 1:** Test Error for Snapshot ensemble and EMA for different values of LR and WD

| WD     | **LR=0.001** | **LR=0.005** | **LR=0.01** | **LR=0.025** | **LR=0.05** | **LR=0.1** | **LR=0.15** |
|--------|--------------|--------------|-------------|--------------|-------------|------------|-------------|
| 0.0000 | 0.03         | 0.02         | 0.01        | 0.01         | 0.01        | 0.01       | 0.01        |
| 0.0005 | 0.04         | 0.02         | 0.02        | 0.03         | 0.09        | 0.08       | 0.07        |
| 0.0010 | 0.04         | 0.05         | 0.04        | 0.10         | 0.09        | 0.07       | 0.07        |
| 0.0015 | 0.04         | 0.07         | 0.07        | 0.08         | 0.08        | 0.07       | 0.07        |
| 0.0025 | 0.04         | 0.11         | 0.10        | 0.06         | 0.08        | 0.08       | 0.09        |
| 0.0050 | 0.06         | 0.15         | 0.11        | 0.09         | 0.08        | 0.10       | 0.11        |
| 0.0075 | 0.08         | 0.12         | 0.10        | 0.10         | 0.10        | 0.11       | 0.12        |
| 0.0100 | 0.10         | 0.09         | 0.10        | 0.10         | 0.11        | 0.12       | 0.13        |

**Table2:** Total variation Distance between softmax output of Ensemble and EMA. 

[17] Huang, Gao, et al. "Snapshot ensembles: Train 1, get m for free."
Thank you. I have read the other reviews and your replies carefully. Like some of the other reviewers, I too consider the analysis and discussion of the over-parametereized regime the more interesting and novel one in this work. 

I believe this work contributes to the already vast body of literature on weight decay -- even though maybe not all aspects the the analysis will withstand the test of time in the long run. But your rebuttal and the additional experiments have strengthened the paper, which is interesting and well executed.


Overall, I maintain my score and hope the paper will be accepted.
Thank you for the detailed discussion and additional experiments!

> Sampling from the stationary distribution of SGD would be interesting, but we lack the analytical form of this distribution, making MCMC sampling inapplicable. We argue that the SGD trajectory at stationarity corresponds to samples from a stationary distribution, whose mean is a stationary point of the regularized loss in Equation 5.

We do not need the analytical form of the distribution, if we reached stationarity. In that case, simple SGD iterates would behave similarly to the Langevin dynamics on the desired distribution. I would suggest conducting the following experiment: after SGD reached stationarity (according to some proxy metrics, e.g. training loss stabilization), try to ensemble consequent iterates, as is done in Snapshot Ensembles. The ensembled model would closely approximate the mean value of the distribution in the function space, if we truly reached stationarity.  
I wonder, what is the discrepancy between such a model and the EMA/fine-tuned network? If all three are somewhat similar, then the claims made in the paper would be much better supported, to my view.

---

After reading the author's rebuttal and other reviews, I would like to increase my score by one point.

To recommend acceptance, I would expect better justification of theoretical approximations and more clear evidence supporting the specific formulation of the main conjecture made in the paper.  
The cited literature is legit, however, I wanted to see more evidence related to the specific setup considered in this work, e.g. I still doubt that training with sufficiently large LR values for the SGD dynamics to stabilize at some constant loss level would ensure that the total loss gradient is negligible (compared to a near-converged training state considered in prior works). Also, there seem to be as many papers supporting the Gaussian approximation as there are against it (I provided a few references in the review), therefore I was interested why this approximation is relevant for this particular study with its specifics.