Revisiting Random Weight Perturbation for Efficiently Improving Generalization
Authors: Tao Li
Qinghua Tao
Weihao Yan
Yingwen Wu
Zehao Lei
Kun Fang
Mingzhen He
Xiaolin Huang
Venue: Accepted by TMLR ()

Abstract: Improving the generalization ability of modern deep neural networks (DNNs) is a fundamental challenge in machine learning. Two branches of methods have been proposed to seek flat minima and improve generalization: one led by sharpness-aware minimization (SAM) minimizes the worst-case neighborhood loss through adversarial weight perturbation (AWP), and the other minimizes the expected Bayes objective with random weight perturbation (RWP). While RWP offers advantages in computation and is closely linked to AWP on a mathematical basis, its empirical performance has consistently lagged behind that of AWP. In this paper, we revisit the use of RWP for improving generalization and propose improvements from two perspectives: i) the trade-off between generalization and convergence and ii) the random perturbation generation. Through extensive experimental evaluations, we demonstrate that our enhanced RWP methods achieve greater efficiency in enhancing generalization, particularly in large-scale problems, while also offering comparable or even superior performance to SAM. The code is released at https://github.com/nblt/mARWP.

Published in Transactions on Machine Learning Research (03/2024)
Revisiting Random Weight Perturbation for Efficiently Im- proving Generalization
Tao Li
B.taa@ssjuueddu edu.co
Shanghai Jiao Tong University
Qinghua Tao
Caaoottsat.kulewweee
KU Leuven
Weihao Yan
poth92200332200s$$tuccc..mm
Shanghai Jiao Tong University
Yingwen Wu
yingwen_ www.een___@sttuuuu...r
Shanghai Jiao Tong University
Zehao Lei
labsjstudyy@sjuu.ddu.cc
Shanghai Jiao Tong University
Kun Fang
fanghenshaooBsjuu.euu...
Shanghai Jiao Tong University
Mingzhen He
mingahen mingghen__helkssuu.dd... eeeeisstuuedu.cr
Shonghai Jino Tong University
Xiaolin Huang
ricokinbuung@Bsjuu.ddu.k
Shanghai Jino Tong University
Reviewed on OpenReview: https: openreview. net/ forum? id= Wobg0HpoPX
Abstract
Improving the generalization ability of modern deep neural networks (DNNs) is fundamental challenge machine learning. Two branches of methods have been proposed to seek flat. minima and improve generalization: one led by sharpness-aware minimization (SAM) minimizes the worst-case neighborhood loss through adversarial weight perturbation (AWP), and the other minimizes the expected Bayes objective with random weight perturbation (RWP). While RWP offers advantages in computation and is closely linked to AWP mathematical basis, its empirical performance has consistently lagged behind that of AWP. In this paper, we revisit the use of RWP for improving generalization and propose improvements from two perspectives: i) the trade-off between generalization and convergence and ii) the random perturbation generation. Through extensive experimental evaluations, we demonstrate that our enhanced RWP methods achieve greater efficiency enhancing generalization particularly in large-scale problems, while also offering comparable even superior performance to SAM. The code is released at htttpp///ggttuu..oommbl/////////
1 Introduction
Modern deep neural networks (DNNs) are commonly characterized by their over-parameterization, boasting millions even billions of parameters. This extensive model capacity empowers DNNs explore vast.
"Corresponding author

---

Published in Transactions on Machine Learning Research (03/2024)
hypothesis space and achieve state-of-the-art performance across various domains Tan & Le 2019 Kolesnikov et al.
2020: Liu et al. 2021 Radford et al. p.21p. However, this abundance of parameters relative to the number of training samples makes DNNs prone to memorizing the training data, leading to overfitting issues. Consequently, it becomes crucial to develop effective training algorithms that facilitate generalization beyond the training set (Neyshabur et al. 2017).
Numerous studies have been dedicated to improving the generalization ability of DNNs (Szegedy al. 2016
Izmailov et al., 2018; Zhang et al.) 2018 2019 Forct et al., 2020). Building upon the notion that flat minima are better suited to adapt to potential distribution shifts between training and test data, leading to improved generalization (Hochreiter & Schmidhuber 1997 Dinh et al.) 2017 Li ct al., 2018). two prominent branches of methods have emerged to identify and exploit such flat minima for effective generalization improvement. The first branch formulates the optimization objective as a min-max problem, aiming to minimize the training loss under the worst-case adversarial weight perturbation (AWP). This approach, also known as sharpness-aware minimization (SAM) (Foret et al., 2020), seeks to find flat minima that reside in neighborhoods characterizod by consistently low loss values. The second branch, exemplified by LPF-SGD (Bisla al.) 2022 aims to recover flat minima by minimizing the expected training loss through random weight perturbation (RWP). Notably, these two approaches share similaritics in their formulations and can be mathematically connected MMllenhoff & Khan 2023
Despite achieving state-of-the-art generalization performance, AWP, represented by SAM (Foret et al. 2020 suffers from a significant drawback in terms of computation and training time. This due to the involvement of two gradient steps, which doubles the computational requirements. Consequently, applying AWP to large-scale problems becomes a prohibitive challenge. On the other hand, RWP offers more computationally efficient alternative, requiring only negligible additional computational overhead compared to regular training. However, it is commonly believed that RWP exhibits inferior empirical performance when compared to AWP (Zheng et al. 2021 Liu et al. 2022b). This discrepancy can be attributed to the fact that RWP perturbs the model with less intensity than AWP, which benefits from leveraging procise gradient information.
In this paper, we revisit the use of RWP for improving generalization and aim to bridge the performance gap between these two types of perturbations. We start by illustrating trade-off between generalization and convergence in RWP: it requires perturbations with orders of magnitude larger than those needed in AWP, to effectively enhance generalization; however, this can lead to convergence issues in RWP. To tackle this challenge, we propose a simple approach called RWP, or m-RWP, which incorporates the gradient of the original loss objective to improve convergence and simultaneously guide the network towards better minima. Notably, although both SAM and m-RWP require two gradient steps per iteration, m- m-RWP is more efficient in improving generalization, which lies in two aspects: 1) in m-RWP, these two steps are separable and can be efficiently computed in parallel, enabling the same training speed as regular SGD, In contrast, the two gradient steps in SAM are successive, resulting in doubling of the training time. 2) the two separable gradient steps in m- m-RWP allow simultaneous use two different botches of data, further accelerating the convergence, especially on large-scale datascts. In contrast, SAM does not allow for using two different batches and may even negatively impact generalization performance. The improved convergence of m-RWP also allows for a larger perturbation variance to confer a better trade-off between generalization performance and convergence. Furthermore, we improve the generation of random weight perturbation by incorporating historical gradient information as guidance. This improvement enables more stable and adaptive weight perturbation generation, leading to enhanced performance. As result, we significantly boost the generalization performance of RWP and introduce two improved RWP approaches: 1) ARWP which achieves competitive performance to AWP but requires only half of the computation and 2) m- ARWP which achieves comparable or even superior performance while benefiting from parallel computing of the two gradient steps.
In summary, we make the following contributions:
We analyze the convergence propertics of SGD with RWP in non-convex settings and identify a potential trade-off between generalization and convergence in RWP. We then propose a simple method to enhance such trade-off and improve the generalization performance of RWP.
2

---

Published in Transactions on Machine Learning Research (03/2024)
We propose an improvement to the generation of random weight perturbation by utilizing historical gradient information. This enhancement enables more stable and adaptive generation of weight perturbations, leading to improved performance.
We present a comprehensive empirical study showing that our improved RWP approaches can achieve more efficient generalization improvements compared to AWP, especially on largo-scale problems.
2 Related Work
Flat Minima and Generalization. The connection between the flatness of local minima and generalization
has been extensively studied (Dinh et al. 2017 Keskar et al. 2017 Izmailov et al., 2018 Li et al. 2018
Jiang* et al., 2020). Hochreiter et al. (Hochreiter Schmidhuber) 1994 1997) are among the first to reveal the connection between flat minima and the generalization of model. Keskar et al. (Keskar et al., 2017) observe that the performance degradation of large batch training is caused by converging to sharp minima. More recently, Jiang el al. (Jiang* el al. 2020) present large-scale study of generalization in DNNs and demonstrate a strong connection between the sharpness and generalization error under various settings and pyper-parameters. Keskar et al. (Keskar et al. 2017) and Dinh et al. (Dinh et al. 2017) state that the flatness can be characterized by Hessian's eigenvalues and provide computationally feasible method to measure it.
Sharpness-aware Minimization (SAM). SAM (Foret al. 2020 is a recently proposed training scheme that seeks flat minima by formulating min-max problem and utilizing adversarial weight perturbation (AWP) to encourage parameters to sit in neighborhoods with uniformly low loss. Tt. has shown power to achieve statc-of-the-art performance. Later, line of works improves the SAM's performance from the perspective of
the neighborhood's s geometric measure (Kwon et al. 2021 Kim el al. 2022 Liu et al. 2022b) or surrogate loss function Zhuang et al. 2022). Several methods have been developed to improve training efficiency (Du
ct al.
2022a b Liu et al.) 2022a Mi ct al. 2022 Zhao et al. 2022b Jiang et al.. 2023 Li al. 2024
Random Weight Perturbation (RWP). RWP is widely used in deep learning. Multiple weight noise injection methods have been shown to effectively escape spurious local optimum (Zhou et al. 2019) and saddle points (Jin et al., 2021). Upon generalization, Zhang et al. (Zheng et al.] [021] discuss that RWP is much less effective for generalization improvement than AWP. Wen et al.. (Wen et al. 2018 propose SmoothOut framework to smooth out the sharp minima. WWang & Mao (2021) propose Gaussian model perturbation (GMP) as a regularization scheme for SGD training, but it remains inefficient due to the need of multiple computation budgets for noise sampling. Bisla et al. (Bisla al.) [202)) connect the smoothness of the loss objective to generalization and adopt filter-wise random Gaussian perturbation generation to improve the performance. However, the performance of RWP still lags behind that of AWP (Bisla et al.. 2022 Liu et al. 2022b). Notably, recent Möllenhoff et al. (Möllenhoff & Khan 2023) mathematically connect the expected Bayes loss under RWP with the min-max loss in SAM and suggest that RWP can be viewed as a 'softer' version of AWP. We significantly lift the performance of RWP from the convergence perspective and fill the performance gap to that of AWP.
3 Preliminary
Let f(;;w) be the neural network function with trainable parameters w E IR,, where d is the number of parameters. The loss function over a pair of data point (æ,, yı) is denoted as L(I(x; w), yi) (shorted for L.ww). Given the datasets S = {xx,, y:) drawn from data distribution D with i.i.d. condition, the empirical loss can be defined as L(w) E L,(w).
Two branches of methods are proposed to pursue flat minima and better generalization ability. The first, known as sharpness-aware minimization (Foret et al. 2020), tries to minimize the worst-case loss in a neighborhood (defined by norm ball) to bias training trajectorics towards flat minima, i.c.,
I, SAM (wn) = max L(w + ts),
(1)
||c,
3

---

Published in Transactions on Machine Learning Research (03/2024)
where p is the radius that controls the neighborhood size. Instead of posing the strict 'max-loss' over the neighborhood, the second, represented by LPFF-GD (Bisla et al. 2022), adopts 'expected-loss and minimizes the posterior (typically an isotropic Gaussian distribution) of the following Bayes objective (Möllenhoff & Khan 2023
I Bayess (w) = - t,),
(2)
where o² is the variance that governs the magnitude of the random weight perturbation. Such expected loss objective can effectively smooth the loss landscape and thereby recover flat minima (Bisla et al.) 2022
The above two objectives resemble formulation except that the maximum in Equ. 1 is replaced by an expectation in Eqn. (2) Intuitively, the expectation could be viewed as 'softer' version of the maximum.
Möllenhoff & Khan (2023) demonstrates that the above two objectives can be bridged mathematically, leveraging the tools of Fenchel biconjugate in convex optimization Efiriart-Urruty & Lemaréchal 2004 Specifically, let T.relaxed(11 (w) be the Fenchel biconjugate of (1)) defined in the dual spaces of exponential- family distributions (Wainwright et al. 2008). which is a convex relaxation w.r.t. original Bayes objective. There exists an equivalence such that
arg min SAM (w)) arg min I relaaxdd (w),
(w)),
(3)
assuming the SAM-perturbation satisfies ||c p at a stationary point.
Assumptions. Before delving into analysis, first make some standard assumptions in stochastic optimization which are typical as in Duchi et al. (2012); Ghadini & Lan| (2013); Karimi et. al. (2016);
Andriushchenko & Flammarion (2022): Jiang et al. (2023) that will be used in our theoretical analysis.
Assumption 1 (Bounded variance) There exists a constant M > 0 for any data batch B such that
E [IVV((()) [ S M, Vw Rdd.
Assumption (a-Lipschitz continuity) Assume the loss function L TR. + be a-Lipschitz continu- ous. There erists a > 0 such that
[[(w)) - L(v)lla S al/w 0112, Vw, v E Rdd
(5)
Assumption 3 (S-smoothness) Assume the loss function L Rd + IR to be 8-smooth. There erists 3 > 0
such that
[L(w)- VL(v)lla < B v112, Yw, R".
(6)
4 Random Weight Perturbation v.s. Adversarial Weight Perturbation
Despite the theoretical connection established between the 'max-loss' and 'expected-losss objectives, the performance of the latter still empirically lags behind the performance of the former, as the biconjugate function could not be attained in real neural network functions and there are gaps between the original objectives and its approximation. In this section, we work practical analysis for solving the above two objectives through the lens of weight perturbation.
Adversarial Weight Perturbation (AWP). To optimize SSAM (w), we first have to find the worst-case perturbations e for the max problem. Foret el al. (Foret et al. 2020 practically approximate Eqn. via the first-order expansion:
V Tww
R max e, [w 0 V (w)ll2
Then the gradient at the perturbed weight w + E is computed for updating the model:
SAM VL(w))wte;-
(8)
Due to the two sequential gradient calculations involved for each iteration, the training speed of SAM is 2x that of regular SGD training.
4

---

Published in Transactions on Machine Learning Research (03/2024)
Method
Perturbation Radius
Gradient Information
Computation
Time
Accuracy (%)
AWP
Small
Yes
2x
2x
77.15
RWP
Medium
No
1x
1x
76.77
ARWP
Medium
Yes
1x
1x
77.02
m-RWP
Large
No
2x
1x
77.82
m-ARWP
Large
Yes
2x
1x
78.04
Table An overall comparison of different methods. RWP and ARWP utilize much larger perturbation radius than AWP, while m-RWP and m-ARWP can utilize even larger perturbation radius due the improved convergence. The "gradient information" denotes whether utilizing gradient. information for generating weight, perturbation. RWP and ARWP requires only half of computation budget than AWP, while m-RWP and m-ARWP are able to paralleling the computation of the two gradient steps to half the training time. For performance comparison on ImageNet with ResNet-50, RWP and variants can achieve much more efficient generalization improvement than AWP.
Random Weight Perturbation (RWP). For optimizing I Bayess()) we similarly sample a random perturbation from a given distribution for each iteration and calculate the gradient at the perturbed weight w + 6, for updating the model:
V Bayes (wy) s V
99)
Note that the selection the distribution plays an crucial role the effectiveness of RWP. For modern DNNs, the loss function does not change with parameter scaling when Rel.U-nonlinearilies and batch normalization structure. Following the
[Loffe & Szegedy 2015) are applied. Hence, is essential to consider the approach in (Bisla et al. 2022), we practically generate the RWP from filter-wise Gaussian distribution, ( [11] k ))=)),, with controlling the perturbation magnitude.
i.e. 6, ~- N (0, ddaggl
)
RWP requires much larger magnitude. As the precise gradient direction is known, AWP is much more "effective" at perturbing the model compared to RWP. Consequently, in order to achieve a similar level of perturbation strength, the magnitude of RWP needs to be considerably larger than that of AWP. We carry out comparative experiments using a model un* that has been well-trained with SGD and apply different perturbation magnitudes for both RWP and AWP. As dipicted in Figure I to attain a similar expected perturbed training loss E [L(w" + +)] (we calculate the mean perturbed loss over the entire training set with a batch size of 256), the perturbation radius of RWP would need to be roughly two orders of magnitude larger than that of AWP. Such large perturbations can introduce instability in training and cause convergence issues that degrade the performance.
1.5
RWP
Expected training loss
AWP
1.0
0.5
0.0
10-3
10-2
10-1
10 10¹
Perturbation radius
Figure Expected training loss E[[(w" E)] under different perturbation radii ||e||a. The experiments employ well-trained model w" using SGD on CIFAR-10 with ResNet-18. Note that the x-axis is in logarithmic coordinates.
5 Improving Random Weight Perturbation
In this section, we propose to improve the performance of random weight perturbation from two primary perspectives. Firstly, we integrale the original loss objective to facilitate the trade-off between generalization and convergence. This enables enhanced convergence and meanwhile allows for larger perturbation magnitudes with improved generalization performance. Secondly, we focus on refining weight perturbation generation by incorporating historical gradient information. This enables more adaptive and effective perturbation generation process.
5

---

Published in Transactions on Machine Learning Research (03/2024)
5.1 Trade-off between Generalization and Convergence
In this subsection, we showcase that there exists trade-off' belween convergence and generalization for RWP. We first analyze the smoothness properties of the expected loss function under RWP.
Theorem (Smoothness of RWP, Bisla et al. (2022) With L BByyes(w) defined in Equ. 2 is ( 8] smmooth.
Assumption
and the function
The above theorem indicates that a larger variance for RWP will provide better guarantee for smoothness, consequently leading to smaller generalization error [Bisla et al. 2022). A minor observation is that a small magnitude of perturbation may not effectively contribute to the enhancement of smoothness when o is small, as in such cases _ 3.
We then investigate the convergence properties of the following SGD optimization under RWP:
W++1 ut 7/ LLs +,),
(10)
where B denotes the mini-batch data.
Theorem 2 (Convergence of RWP in non-convex setting) Assume Assumption | and 3 hold. Let G, N(0,002 Iaxa) be the random perturbation and b be the batch size. Consider the sequence (we))e generaled by Eqn. 10 with a stepsize salisfying 7t-
2a and 7 < 3. Then we have VI
1 T
2 ([ - E[[(w")) E V
log T + 232
+23M + 0
(11)
TO
VT
We provide the proof Appendix A and make several remarks: (1) Different from prior theoretical analysis lAndriushchenko Flammarion, 2022 Mi et al.] 2022) in SAM that assumes decreasing magnitude perturbation strength for achieving good convergence properties (e.g. Pt Si
po in Andriushchenko &
Flammarion 2022 ), we consider more realistic setting with non-decreasing a that is adopted in practice. (2) There are three terms that upper bound the expected training loss. The first two terms decrease at a rate of O ) and O Ol(o)()), respectively, which is consistent with the convergence rate of regular SGD. ododd is the additional variance term introduced by random weight perturbation. It can remain large when the perturbation radius (||e,Il) is large, of which the expected radius is |/]] odd, thereby slowing down convergence. The third term is positive constant, proportional to the perturbation variance o², which prevents the effective reduction of the gradient norm beyond a certain point.
trade-off. Based on the above two aspects, we can conclude that there exists a trade-off between gencralization and convergence: for the sake of good smoothness properties and smaller generalization error, need to increase the variance o² for random weight perturbation. However, this can significantly increase the magnitude of the perturbation, and on the contrary, pose convergence challenge.
5.2 Improving the Trade-off by Incorporating Original Loss
From the analysis in the last subsection, know that there exists trade-off between generalization and convergence in RWP. To achieve good generalization ability brought by large perturbation variance while enjoying good convergence properly, propose to combine the original loss with the expected Bayes loss to improve the convergence of RWP, resulting in our mired-RWt RWW (m-RWP) loss objective:
L.t ("www) XL, (w))
A)L(w)
(12)
where X E [0, 1] is a pre-given balance coefficient. The two loss terms in our m-RWP objective are complemen- tary to each other: the first L (Bayes() provides a smoothed landscape that biases the network towards flat region, while the second L(w) helps recover the necessary local information and better locates the minima that contributes high performance. These two together could provide a both "local" and "global" viewing of the landscape- by optimizing T.m(wv), a good solution be expected.

---

Published in Transactions on Machine Learning Research (03/2024)
In practical Implementation, we adopt particular procedure for optimizing the objective function L Bayys( using RWP, i.e., in each training iteration, we sample one single random weight perturbation vector denoted as Ey. To enhance the optimization process, we utilize two distinct batches of data, namely Bi and B2, for the two gradient steps involved. Note that this effectively enlarges the virtual batch size by a factor of two. The optimization process of our m-RWP can be lormulated as follows:
W wt >x [XV LB, (w; E,) + (1 (V:)),(..).
(13)
In the subsequent analysis, we examine the smoothness and convergence properties of m-RWP. The proofs for these properties can be found in Appendix B and C
respectively.
Theorem (Smoothncss of m-RWP) With Assumption
and the function ("www) defined in
Eqn. 12 min Arx + A)3, 3] -smooth.
Thcorem (Convergence of m-RWP in non-convex sctting) Assume Assumption and hold. Let NO, laxa) be the random perturbation and b be the batch size. Consider the sequence (w;)re generated by Eqn. 13), with stepsize satisfying "t- Vi and y < B. Then we have
log T B. [l: ([B[L[wn)] E[((w"))) + [23M 21+1)+
VT + 2,3
T
Improved convergence. Compared to the convergence properties of RWP as stated in Theorem m-RWP offers immediate improvements in two aspects. Firstly, it reduces the variance term introduced by random weight perturbation (33 12 w and the positive constant term (2.3² by a factor of X2. Secondly, m-RWP enables the use of two different data batches to compute the two gradient steps. This effectively T doubles the batch size for each iteration training and thus reduces the gradient variance term (2,9M% /T by a factor of 2X² 2λ- + 1. As result, the convergence is significantly improved. The convergence of RWP and m-RWP is compared in Figure 2
Better trade-off between generalization and convergence. With the significant enhancement in convergence achieved by m-RWP, it becomes feasible to employ larger perturbation variance. This allows to enjoy the benefits from the improved smoothness properties offered by the increased perturbation variance while maintaining good convergence, potentially resulting in better trade-off between generalization and convergence. Specifically, for attaining similar convergence condition, denoted as foadd in Equ. [11. the perturbation variance employed by m- RWP can be increased to o' = o/X. The subsequent lemma demonstrates that, under certain conditions, the mixed loss function could provide better guarantees of smoothness compared to the original Bayes objective under similar convergence propertics.
Lemma 1 Assuming that a < 30 < 2a, with Assumption I Bayes (w)) with a perturbation variance a is smoother than ((w) and 2) I. (ww) with a balance coefficient Xe (
2 and 3. we have the following conclusions: 1)
(Bo-a,1) 1) and a perturbation variance a is smoother than L Bayess(w)
Please refer to Appendix D for the proof. In practical scenarios where constants such as the Lipschitz constant a and smoothness constant / are unknown, we thus further validate our findings empirically. In Figure 3 we observe that m-RWP indoed achieves a better tradc-off between generalization and convergence. Morcover, it demonstrates significantly improved performance by utilizing larger perturbation variances.
Efficient parallel training. Both SAM and m-RWP indeed involve two gradient steps for each iteration: VL(w) and VL( + +). However, the key distinction lies in the separability of these steps. In m-RWP, they are scparable, and can be calculated independently and in parallel, whereas, in SAM, they need to be computed sequentially. This parallel computing capability of m-RWP allows for halving of the training time, making highly efficient approach for large-scale problems.
7

---

Published in Transactions on Machine Learning Research (03/2024)
SAM
RWP
Training loss
m-RWP
Accuracy (9)
m-RWP
0
50
100 Epochs
150
200
50
100
150
200
Epochs
(a) Training loss
(b) Test accuracy
Figure 2: Training performance comparison of RWP and m-RWP. m-RWP significantly improve the conver- gence over RWP and leads to much better performance. The experiments are conducted on CIFAR-100 with ResNet-18. The perturbation variance a is set to 0.01.
Generalization Error (%)
m-RWP
RWP
20
Accuracy
18
m-RWP
RWP
0.02
0.04
0.06
0.08
0.10
0.12
0.005
0.010
0.015
0.020
Training Loss
Perturbation Variance o²
(b)
Figure 3: The mixed loss objective achieves a better trade-off between generalization and convergence. In (a), we conducted multiple runs of RWP and m-RWP with varying perturbation variances (a) and recorded the final training losses and generalization errors (difference between training accuracy and test accuracy) for each trial. Our observations reveal that m-RWP achieves an improved trade-off between generalization error and convergence compared RWP. In (b), m-RWP is capable of utilizing larger perturbation variances achieve better generalization performance. The experiments performed on CIFAR-100 with ResNet-18 and the balance cocfficient À is set to 0.5.
5.3 Adaptive Random Weight Perturbation Generation
One limitation of previous methods for generating RWP is their exclusive reliance on weight norm, without utilizing any gradient information. This approach creates disconnect between perturbation generation and the actual loss objective, resulting in overly "rough" perturbations.
Despite that for the efficiency considerations, can not adopt an additional gradient step to assist the generation of RWP, the historical perturbed gradients are available and can be leveraged. We thus propose to utilize the historical gradient to better guide the generation clement-wise scaling of the gradient, taking into account the historical running sum of squares in cach dimension. It important to that this technique is widely used in adaptive gradient descent optimizers such as Adam (Kingma & Ba 2015) and RMSprop (Tieleman et. al. 2012). The generation form of our
RWP. To achieve this, we employ an
8

---

Published in Transactions on Machine Learning Research (03/2024)
proposed method, called Adaptive Random Weight Perturbation (ARWP), is as follows:
0²I
Er ~ N
0,
(14)
+ 9 E ,
g
where 1) and are two cyper-parameters. The rationale behind our generation approach is to move away from assigning a uniform perturbation magnitude (o) to all parameters. Instead, we aim to introduce an adaptive perturbation strength based the historical perturbed gradients. Tf the historical gradients are large, we want to assign smaller perturbation magnitude, and vice versa. By incorporating such adaptability, we can tailor the perturbation magnitudes to the specific characteristics of cach parameter. This allows us to account for variations in sensitivity and importance across different dimensions, leading to a more adaptive and effective generation of random perturbations. In practice, the perturbation variance a is progressively increasing (i.e., cosine increasing strategy in Section [6. during the training RWP/ARWP for gradually recovering the flat minima, as suggested by Bisla et al. (2022). Therefore, the primary enhancement of our ARWP lies in its adaptive generation of perturbations, rather than annealing the variance. Regarding convergence, we note that the variance of ARWP in Eqn. (14) is upper bounded by the corresponding variance of RWP in Eqn. (2). Therefore, the convergence properties of RWP in Theorem 2 and 4 still hold for ARWP.
Then combing with the filter-wise perturbation generation techniques from Bisla et al. approach can be expressed as follows:
2022). our generation
(5) o² dida 11 |2] - i=1 Biai- {1l (1) 12)55
Cry ~ N
O,
(15)
+
Then combining the mixed loss objective in Eqn. 12 propose our m- m-ARWP approach, which takes the same computational cost as SAM while being capable of halving the training time through parallel computation of the two gradient steps.
6 Experiments
In this section, we present extensive experimental results to demonstrate the efficiency and effectiveness of our proposed methods. We begin by introducing the experimental setup and then evaluate the performance over three standard benchmark datasets: CIFAR-10, CIFAR-100 and ImageNet. We also conduct ablation studies the hyper-parameters and visualize the loss landscape to provide further insights.
6.1 Results
Datasets & Models. We experiment over three benchmark image classification tasks: CIFAR-10, CIFAR- 100 (Krizhevsky& & Hinton 2009), and ImageNet (Deng et al. 2009). For CIFAR, apply standard random horizontal flipping, cropping, normalization, and Cutout angmentation (DeVries & Taylor 2017 (except for ViT, for which we use Rand Augmentt (Cubuk et al. 2020)). ). For ImageNet, we apply basic data preprocessing and augmentation following the public Pytorch example (Paszke et al. 2017). We evaluate across a variety of representative architectures, including VGG (Simonyan & Zisserman, 2014). ResNet (He et al. 2016),
WideResNet (Zagoruyko & Komodakis 2016), and ViT (Dosovitskiy et al. 2021
Training Settings. We compare the performance of six training methods: SGD, SAM, RWP, ARWP, m-RWP, and m-ARPP. It is worth noting that SGD, RWP, and ARWP share the same computation (1x), while SAM, m-RWP and ARWP require twice the computational resources (2x). For CIFAR experiments. we set the training epochs to 200 with batch size 256, momentum 0.9, and weight decay 0.001 (Du et al. 2022a Zhao el al. 2022a), keeping the same among all methods for a fair comparison (except for ViT, we adopt a longer training schedule and provide the details in Appendix D For SAM, we conduct a grid search for p over {0.005, 0.01, 0.02, 0.05, 0.1, 0.1, 0.5) and find that for CIFAR-10, p = 0.05 gives the best results for VGG16-EN, p 0.1 suits best for ResNet-18 and WRN-28-10, while for CIFAR-100, = 0.1 works best for VGG-16BN and p 0.2 is optimal for ResNet-18 and WRN-28-10, which also coincides
9

---

Published in Transactions on Machine Learning Research (03/2024)
Table 2: Results on CIFAR-10/100. We set the computation (FLOPs) and training time of SGD as 1 1x. The best accuracy is bold and the second best underlined.
Model
Method
CIFAR-10
CIFAR-100
FLOPs
Time
SGD
94.96=0.15
75.43=0.29
1x
1x
SAM
95.43=0.11
76.74=0.22
2x
2x
RWP
94.97=0.07
76,56=0.10
1X
1x
VGGI6-BN
ARWP
95.07=0.28
77.07=0.21
1x
1x
m-RWP
95.49=0.16
77.68=0.14
2x
1X
m-ARWP
95.6110.23
77.98
2x
1x
SGD
96.10=0.08
78, 18.03.939
1x
1X
SAM
96.56-0.11
80.48
2x
2X
RWP
96.01
80.21
1x
1X
ResNet-18
ARWP
96.30=0.08
80.71=0.21
1x
1X
m RWP
96.58=0.09
81.18=0.00
2X
1X
n-ARWP
96.68
$1.38:0.25
2x
1x
SGD
96.85=0.05
82.51=0.24
1x
1x
SAM
97.35
84.6810.21
2x
2x
RWP
96.73=0.12
83.67=0.14
1x
1x
WRN-28-10
ARWP
96,89=0.11
83.96=0.09
1x
1X
m-RWP
97.21
81.37
2X
1X
m-ARWP
97.27
84.62200.15
2X
1X
Adam 86.60=0.03
63.66=0.29
1x
1X
SAM
87.48
04.83
2X
2X
RWP
86.53=0.04
63.67=0.41
1X
1X
ViT-S
ARWP
86.88=0.09
64.12-0.22
1X
1X
m-RWP 87.71
65.67=0.09
2x
1x
m-ARWP
88.18#0.19
66.13
2x
1X
with the hyper- pyper-parameters reported by Mi al. (2022): Li & Giannakis (2023). For RWP and ARWP, we search over (0.005, 0.01, 0.015, 0.02] and set a 0.01 for both CIFAR-10 100 it gives the optimal performance. We also adopt cosine increasing schedule for described in (Bisla et al. 2022). For m-RWP and m-ARWP, We set 0.1 and 8 = 0.99 as default choice. For ImageNet experiments, we set the training epochs to 90 with batch size 256, weight decay 0.0001, and momentum 0.9. We use 0.05 for SAM which aligns with (Forct et al. 2020 Kwon et al. )021), o = 0.003 for RWP and ARWP, and o = 0.005, X = 0.5 for m-ARWP. We employ m-sharpness with m 128 for SAM as (Foret et al. 2020 Kwon al. 2021). For all experiments,
0.015 and À 0.5 as it gives moderately good performance across different models.
adopt cosine learning rate decay [Loshchilov & Hutter 2016) with an initial learning rate of 0.1 and record the final model performance the test set. Mcan and standard deviation calculated over three independent trials.
CIFAR. We begin by focusing on the CIFAR-10 and CIFAR-100 datasets. We evaluate the final test accuracy, total computation (in FLOPs), and training time for different methods. Detailed comparisons are presented in Table 2 We observe that ARWP consistently improves the performance RWP by 0.1-0.3% on CIFAR-10 and 0.3-0.5% on CIFAR-100, and m-ARWP consistently outperforms ARWP, improving performance by 1.6% on CIFAR-10 and 3.1% on CIFAR-100, confirming the effectiveness of our improvements on enhancing generalization. It is also worth mentioning that ARWP achieve competitive, and sometimes even better, generalization performance compared to SAM, particularly with VGG16-BN (+0.3%) and ResNet-18 (+0.5% models on CIFAR-100. Remarkably, ARWP achieves this with only half of the computation required by SAM. Additionally, m- m-ARWP oulperforms SAM in most cases on CIFAR-10, and achieves improvements ranging from 0.1% to 1.3% on CIFAR-100. We can also observe that the mixing strategy alone contributes the most performance improvement, c.g. with a notable increase of +0.97 % on CIFAR-100 with ResNet-18, as it entails a doubled computational overhead. In contrast, the adaptive perturbation strategy contributes an increase of +0.50%. Lastly, it is important to note that although both SAM and m-ARWP require twice the
10

---

Published in Transactions on Machine Learning Research (03/2024)
Table Results on ImageNet. We set the computation (FLOPs) and training time of SGD as 1x. The best accuracy is in bold and the second best underlined.
Model
Method
Accuracy (%)
FLOPs
Time
SGD
73.11
1x
1x
SAM
74.65
2x
2x
RWP
74.01
1X
1x
VGG16-BN
ARWP
74.28
1x
1x
m-RWP
75.45
2x
1X
m-ARWP
75.79
SGD
70.32
SAM
70.77
2X
2X
RWP
70.46
1x
1X
ResNet- 18
ARWP
70.71
1X
1X
m-RWP
71.42
2X
1X
m-ARWP
71.58
2X
1X
SGD
76.62
SAM
77.15
2X
2X
RWP
76.77
ResNet-50
ARWP
77.02
1X
1X
m-RWP
77.82
m-ARWP
78.04
2X
1X
AdamW
68.12
1X
1X
SAM
68.98
2x
2X
RWP
68.40
ViT-S/32
ARWP
68.74
1X
1X
m-RWP
69.42
m-ARWP
69.76
2X
1X
computation of SGD, the training time of m- RWP can be reduced by half compared to SAM through parallel computing. This reduction in training time is a valuable advantage of m-ARWP.
ImageNet. Next, we evaluate our proposed methods on ImageNet dataset, which has a substantially larger scale than CIFAR. We evaluate over four different architecturos, namely VGG16-BN, ResNet-18, ResNct-50, and VIT-S/32, and present the results in Table 3 We observe that ARWP can achieve very competitive performance against. SAM. For instance, with ResNet-18, ARWP achieves accuracy of 70.71% compared to SAM's 70.77%, while with ResNet-50, ARWP achieves 77.02% accuracy compared to SAM's 77.15%. Notably, ARWP accomplishes this while requiring only half of the computational resources. Furthermore, m-ARWP significantly outperforms SAM, achieving 75.79% accuracy (+1.14%) with VGG16-BN, 71.58% accuracy (+0.81%) with ResNet-18, 78.04% accuracy (+0.89%) with ResNet-50, and 69.76% accuracy (+0.78%) with ViT-S/32. We note that models trained on ImageNet are typically under-trained. Therefore, the improved convergence of m-ARWP offers even more significant advantages over SAM. Moreover, the capability of parallel computing in m-ARWP is especially advantageous the 2x longer training time in SAM would be prohibitively slow for large-scale problems.
6.2 Ablation Study and Visualization
Table 4: Effects of /dif/eifferent data batches for two gradient steps. The experiments are conducted on CIFAR-100 with ResNet-18.
Impact of different data batches. We conducted further investigation into the impact of using the same or different data batches for the two gradient steps in m- m-ARWP and SAM. The results are presented in Table and we made the following observations. For m-ARWP, both choices of using the same or different data batches yield comparable performance, with the of different.
Training
Same
Different
SAM
80.48
78.30 (1 2.18)
m-ARWP
81.14+0.10
81.38+0.12 (1 0.24)
11

---

Published in Transactions on Machine Learning Research (03/2024)
batches resulting in slightly better performance. On the other hand, for SAM, the two approaches yield starkly different results. Adopting different data batches for the adversarial attack and gradient propagation in SAM would undesirably degrade its generalization performance to that of plain SGD. This finding suggests that applying the same data batch for adversarial attack and gradient propagation is a crucial aspoct for SAM's generalization improvement. We attribute this difference to the unique characteristic of AWP SAM, which specific to a particular batch of data. The perturbation computed over one batch in SAM can degenerate into meaningless noise when applied to another batch. In contrast, the perturbation used in m-ARWP is not associated with specific data instances, allowing for the use of different data batches to accelerate convergence with better efficiency.
Sensitivity of hyper-parameters. There are four in ARWP, and additionally À and in m-ARWP. We note that for the first three hyper-parameterrs, we use default values as L, B = 0.99, À = 0.5 across different experiments and only a needs to be tuned, thus virtually having the same number of hyper-parameters as SAM and RWP. To better understand their effects on performance, we test the performance under different choices of values. Specifically, we test on CIFAR-10/100 10/100 datasets with ResNet-18, and vary 7) in {0.01,0.1 0,,,, 1}; B in {0.9, 0.99, 0.999), a {0.005, 0.01, 0.015, 0.02) and A in {0.1, 0.3, 0.5, 0.7, 0.93. The results are in Figure 4 We observe that 0.1, 8 0.99, a 0.015, and A 0.5 are robust choice that achieves moderately good performance on both CIFAR-10 and CIFAR-100.
involved in approaches, 7j and B
R8 CIFAR-10
ReeNeil18 CIFAR-100
ResNei-18 CIFAR-10
ResNel- CIFAR-100
1.0
96.36
80.7
0.22
0.22
81.0
96.30
00.6
0.016
96.4
0.015
98.25
96.20
-0.5
0.01
36.2
0:01
80.0
0.1
96.15
80.
0.005
96.0
0.005
79.0
00,3
0.59 g
0.999
0.9
0.5
0.7
0.9
(a) ARWP
(b) m-ARWP, 7 0.1, B = 0.99
Figure 4: Performance under various hyper-parameter configurations.
Loss landscape and Hessian spectrum. Finally, we compare the loss landscape and Hessian spectrum of SGD, RWP, ARWP, SAM, and m-ARWP. Following the plotting technique in (Li et al. 2018)), we uniformly sample 50 50 grid points in the range of [-1, 1] from random tiiter-normalized" direction (Li et al.) 2018
and for Hessian spectrum, approximate it using the Lanczos algorithm (Chorbani et al.) 2019). In F Figure we observe that RWP, ARWP, SAM, and m-ARWP all achieve flatter loss landscape and smaller dominant eigenvalue (Ai) compared to SGD. Additionally, ARWP demonstrates improved flatness and dominant eigenvalue compared to RWP. Moreover, m-ARWP exhibits even better flatness and a smaller dominant cigenvalue than SAM. Interestingly, from the perspective of the loss landscape, RWP and ARWP appear to have flatter landscapes compared to SAM and m-ARWP. However, they exhibit much larger Hessian dominant eigenvalues and worse generalization performance. This discrepancy may be attributed to the fact that the minima found by RWP and ARWP may not converge well and are even "over-smoothed" due to the large perturbation radius involved. Conversely, with the improved convergence of m- m-ARWP, we were able to reach a more precise minimum with better generalization performance.
7 Conclusion
In this work, we revisit the use of random weight perturbation for improving generalization performance. By uncovering the inherent trade-off between generalization and convergence in RWP, we propose mixed loss objective that enables improved generalization while maintaining good convergence. We also introduce an adaptive strategy that utilizes historical gradient information for better perturbation generation. Extensive experiments on various architectures and tasks demonstrate that our improved RWP is able achieve more efficient generalization improvement than AWP, especially large-scale problems.
12

---

Published in Transactions on Machine Learning Research (03/2024)
RWP Assu 8.40
ARWP Arou - 7.40
190
Aj/A, =3.72
12.20
Ag/ly
Ag/2g - 2.24
Ascu
1.76
Arsr
Denatty
Dondtt
Ougakt
Bgelwslue
Bigensaicc
Eggnsshic
Figure 5: Loss landscape (Up) and the corresponding Hessian spectrum visualization (Down) of different methods. Models are trained on CIFAR-10 with ResNet-18.
Acknowledgments
We thank the anonymous reviewers for their thoughtful comments that greatly improved the paper. This work was partially supported by National Key Research Development Project 22023FFF1044200 National Natural Science Foundation of China (62376155), Shanghai Municipal Science and Technology Research Program (22511105600) and Major Project (2021SHZDZX0102),
References
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-awarr minimization. In International Conference on Machine Learning (ICML), 2022.
Devansh Bisla, Jing Wang, and Anna Choromanska. Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape. In International Conference on Artificial Intelligence and Statistics (AISTATIS), 2022.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2009.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. or Xiv preprint arXiv:1 045504 2017.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning (ICML.), 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Representations (ICLR), 2021.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware aware minimization for improved training of neural networks. International Conference on Learning Representations (ICLR), 2022a.
Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent YF Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In Advances in Newral Information Processing Systems (NewrIPS), 2022b.
13

---

Published in Transactions on Machine Learning Research (03/2024)
John C Duchi, Peter L Bartlett, and Martin J Wainwright. Randomized smoothing for stochastic optimization. STAM Journal on Oplimization, 22(2):674-701, 2012.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-oware minimization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2020.
Saced Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Oplimization, 23())22341 2368, 2013.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jean-Baptise Hiriart- Hiriart-Urruty and Claude Lemaréchal. Fundamentals of convex analysis. Springer Science & Business Modia, 2004.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances in Newral Information Processing Systems (NeurIPS), 1994.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Newral computation, 1997.
Sergey loffe and Christian Szegedy. Batch normalization: internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
Accelerating deep network training by reducing
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. orXiv preprint arXiv:1803. 05407, 2018.
Weisen Jiang, Hansi Yang, Yu Zhang, and James Kwok. An adaptive policy to employ sharpness-aware minimization. In International Conference on Learning Representations (ICLR), 2023.
Yiding Jiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic gen- eralization measures and where to find them. In International Conference on Learning Representations (ICLR), 2020.
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael Jordan. On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points. Journal of the ACM (JACM), 2021.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal -rraiient methods under the polyak-lojasiewic/ condition. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang, On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations (ICLR), 2017.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning (ICML), 2022.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision (ECCV), 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning (ICML), 2021.
Bingcong Li and Georgios B Giannakis. Enhancing sharpness-aware optimization through variance suppression. In Advances in Newral Information Processing Systems (NeurIPS), 2023.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Tao Li, Pan Zhou, Zhongbao Hc, Xinwen Cheng, and Xiaolin Huang, Friendly sharpness-aware minimization. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognilion (CVPR), 2024.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsich, and Yang You. Towards efficient and scalable sharpness- aware minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), 2022a.
Yong Liu, Siqi Mai, Minhao Cheng, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Random sharpness-aware minimization. In Advances in Neural Information Processing Systems (NeurIPS), 2022b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE International Conference on Computer Vision (CVPR), 2021.
Lya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint ar 1rXiv:1606 03988, 2016.
Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness- aware minimization stronger: A sparsified perturbation approach. arXiv preprint w:Xin:2210.01177 2022.
Thomas Möllenhoff and Mohammad Emtiyaz Khan. SAM as an optimal relaxation of Bayes. In Interna- tional Conference on Learning Representations (ICLR), 2023. URL attp::://ppaxeeiii..nnefffooooo k4fevFqS0cX
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization deep learning. In Advonces in Neural Information Processing Systems (NeurIPS), 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation in pytorch. 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 140X 1556, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), 2019.
Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Martin J Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trendse in Machine Learning, 1(1-2):1 -305,, 2008.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Ziqiao Wang and Yongyi Mao. On the generalization of models trained with sgd: Information-theoreti bounds and implications. In International Conference on Learning Representations (ICLR), 2021.
Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Chumpeng Wu, Yiran Chen, and Hai Li. Smoothout: Smoothing out sharp minima improve generalization in deep learning. arXiv preprint arxi 07898, 2018
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), British Machine Vision Conference (BMVC), 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E. Hinton. Lookahcad optimizer: k steps forward, step back. In Advances in Neural Informalion Processing Systems (NeurIPS), 2019.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning (ICML), 2022a.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. SS-SAM: Stochastic scheduled sharpness-await aware minimization for efficiently training deep neural networks. arXiv preprint Ttw:22030009666 2022b.
Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tho Zhao. Toward understanding the importance of noise in training neural networks. In International Conference on Machine Learning (ICML), 2019.
Juntang Zhnang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-awen aware training. In Interna- tional Conference on Learning Representations (ICLR), 2022.
A Proof of Theorem 2 Proof
Denote Wx+1/2 = we + Cr,t- From Assumption 1, it follows that
1(w(+1) <L(w,, + ((w,,) (( - -w;) + 6 [1++1 w. w,
(16)
=L(w;) Y VL(W:) VLo(W++1/2)+ 1.33
VLg(
(17)
-L(w))
74 VL(W;) VL8(W0+1/2)
+ (VVL(WW(11/2 - +2V (V/
2VL
(LB(W(+1/22)
(18)
2
=L(w.) 728
[VL(w:) 212
2 -
- (1 B%) VL(W.) (LB(W(++122
(19)
<T(w.:) 723
[L( + + V -
- (LW
+ 7 B V vL(w;)] 2 (1 3 TVL((W(41)))
(20)
The last step using the fact that ||a b112 < 2/ c|2 + 2||c b112. Then taking the expectation both sides gives:
7333 VL(w:)]l
B[L(w++1)] )) S B[L(w:)]
+ 338 +
V4 (Lr((w-1))) (21)
2
16

---

Published in Transactions on Machine Learning Research (03/2024)
For the last term, have:
E[VL(w)) VLB(WL+122) =E [L(w)) (VLs(wi+1/2) - (L + VLg(w)))
(22)
=E
[[ +C.
where C = E [V (VLB(We+1/2)
VLB(W:))] Using the Cauchy-Schwarz inequality, we have
+ 1 [L( - VL(w;)]l?
C
E
V
([(w)))l
2
(23)
3Ro2d
E
Plugging Eqn. (22 and
23 into Eqn.
we obtain:
3
E|L(W(+11) ]]] sB(L(w.))
E L( (w:) + 3/3M
3-33++100 + +2 - (1- 3nhi [VV(()))
BBo²d
+ (1 - 3/
El V
(24)
7t [L( + BBM + (1 + 8y038880 2
<B|L(w))]
(25)
2
Taking summation over 7' iterations, we have:
T
T
[W:VL( B[L(w))] [[[(wr)]+ (BM
3o o3
1
+
1
32
o
dru
Σ
1
(26)
2
2
2
vt
t=1
t=1
This gives:
2 (E]]( [[(((w)))) L[(;)))
log T
+ (23M + o2
+ 2822
(27)
To
VT
2 (IB[L(wo)] [ww)(w))]
log T T
(23M + 000
2,3
(28)
T
where w" the optimal solution.
Proof of Theorem 3
Proof For all w, u E Rd,
[V (L"( s A VL Bayes (w) V Bayes (v) + (1 A) ( - vL(1)]
a
< Amin
3) B}|l - v/|
- + (1 - A)B w l
= min
+ (1-1)88, / w||-
This shows that (ww) is min( da + (1 - A)3, B) 3--moothh
C Proof of Theorem 4
Proof
I(w441) SL(W:) VL(w:) (wwes - wT) + 88 /110041 well
(29)
=L(w;) (LW):) LAVLB,WWLLL. + + +11- (1-1)V (we))
723
+ 2
LVVL (W0+1/2)
(30)
17

---

Published in Transactions on Machine Learning Research (03/2024)
3/3 + 1 (1- (1-)) + +12 B[L(w[+1)l sB(((w;))
2
(1 11-3 VL || + (1- (1- VL(w;))] 2| 2 x
(31)
[[[(w:)] [VL(w, || + 3 -2
22 +
/ (1 B 3
(32)
(33)
Taking summation over T iterations, we have:
T
70
E VL(w))]
[[[( (B(((u)))
(BM 22 21+1)+ 1 288800d 70 Σ
2
(34)
T
+ 2 82 dr Σ
vt
This gives:
L[(;))) 2 [[][((W))] [[ (23M 22 +1 83x008 70
log VT + 2,8822022
D Proof of Lemma 1
Proof From Theorem and and the condition a < 3r < 2a, we can conclude that L (aaye((w)) is
g-smooth since < So. Then leveraging X (S0-a, 1), oblain
A B0 -a
(35)
a
+ +a(A+1) 3o
(36)
+ o(1 < (A-12 1)80
(37)
+ X²a
(1 A)B
(38)
The Eqn.
37
is derived by multiplying both sides of the equation by (A 1). Note that L (m((w) a/A is
min La + 1)8.oh-smooth. Thus, we can conclude that L ("W))aa/x is smoother than L Bayes (w))s(()leo (w)
E ViT training
To train the vision transformer model (Dosovitskiy et al. 2021 we adopt Adam Kingma & Ba 2015 as the base optimizer and train the models on 10000- datasets from scratch with Adam, SAM, RWP, and m-RWP. Specifically, we select the ViT-S model with input size 32 x 32, patch size 4, number of heads 8, and dropout rate 0.1. We train the models for 400 epochs with a batch size of 256, an initial learning rate of 0.0001, and cosine learning rate schodule. For ARWP, we set o 0.001, and for m-ARWP, we set 0.002 and 0.5. For SAM, perform grid search for p over [0.001, 0.002, 0.005, 0.01, 0.05, 0.1, 0.2, 0.5 and finally select 0.05 for optimal. For ImageNet, we follow the implementation of Du al. (2022b) Li & Giannakis 2003)). where train the model for 300 epochs with batch size of 4096. The baseline optimizer is chosen as AdamW with weight decay 0.3. SAM relies on p = 0.05. We use a = 0.005 for RWP/ARWY and a = 0.01 for m-RWP m-RWP/m-ARWZ
18


Reviews:
[{"review_id": "kzrH6w8zvr", "reviewer": "Reviewer_iFGJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well-written and easy to follow.\n- The proposed algorithm is simple and easy to implement, has some benefits over SAM (in terms of computational complexity), and yet outperforms SAM.\n- The experiments are diverse, and done for large-scale architectures. The proposed method works well for every case. Ablation studies on the hyperparameters are also provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an improvement of the random weight perturbation-based optimization algorithm. It has widely been known that optimizing the worst-case error around a parameter reduces the sharpness of the loss surface near the local optima, and leads to a better generalization. The most popular approach to implement this idea is SAM, where the adversarially chosen noises perturb the parameters. On the other hand, SAM has a loose connection with the Bayes objective based on the random weight perturbation (RWP), but the corresponding performance is not on par with SAM. This paper empirically and theoretically analyzes why the RWP may be ineffective, and proposes ways to fix it. Specifically, the mixed RWP is proposed, where the Bayes objective with the RWP is combined with the ordinary SGD objective. The theory shows that mixing can reduce the expected gradient norm, leading to a faster convergence. An additional contribution is to adaptively set the standard deviation of the Gaussian weight perturbation based on the empirical gradients, similar in spirit to the adaptive optimization methods. The proposed method, entitled m-ARWP, is demonstrated on various image classification benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The main experimental results do not include m-RWP. It would be good to have it as well to see the net effect of the mixing.\n- Theorem 2 only guarantees improved convergence, but it does not say anything about the flatness. It would be good to have a related theory, but if not feasible, at least, the experiments to see the effect of the hyperparameter lambda on the flatness of the loss surface would be helpful; the current experiments only include the comparison to baselines with a fixed lambda value.\n- While m-RWP is theoretically backed up, ARWP and m-ARWP are only empirically validated. Would it be hard to incorporate the modified variance term into the Theorem 2?", "broader_impact_concerns": "I don't see any ethical issues or implications related to the contribution in this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SJtgIOfyOh", "reviewer": "Reviewer_HxDU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* I like the high-level organisation of the paper. Starting with identification of the drawbacks of existing approaches, followed by suggesting how they can be overcome is a useful way to frame new methodological developments.\n* Unfortunately the instantiation of this organisation is not convincing. The analysis experiment presented by and in Figure 1 does not seem to directly influence the methodological development in the next section.\n* The theoretical analysis does not add any new insights over previous work. Classic results on the convergence of SGD already take the variance of the gradient estimator into account, so it is already known that reducing the variance of gradient noise leads to better convergence. The proof of the theorems in this paper simply add the already known variance of the noise injected by RWP to the usual noise considered by typical SGD analysis.\n* The argument used to support the proposed method is that, by sampling weight perturbations from a distribution with smaller variance, convergence can be improved. However, the convergence analysis suggests that the best convergence is achieved by not adding a perturbation at all. This is obviously at odds with the original motivation for using AWP and RWP, which is to improve generalisation rather than convergence. The theoretical and empirical analyses do not explore this tradeoff.\n* The empirical results are unreliable, as the authors tuned hyperparameters for their own methods but not others. There are also few details of how the hyperparameters were systematically tuned. Moreover, even with the unfair empirical comparison it is not clear that the results are actually statistically or practically significant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new distribution for sampling weight perturbations when training neural networks. While previous work just makes use of Gaussian distributions, this paper uses a mixture of a delta function placed on zero and a Gaussian. Moreover, they reuse the moment estimation techniques from adaptive optimisers to set the variance of the Gaussians dynamically.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The authors should provide a more detailed description of the hyperparameter tuning process, and should tune hyperparameters for all methods considered in the empirical evaluation.\n* The authors should demonstrate the practical and statistical significance of their experimental findings.\n* The paper should be revised to make it clear how the experiment in Section 4 is related to the new method. Alternatively, the authors could provide a more detailed discussion/demonstration of the tradeoff between convergence and generalisation of existing methods as the magnitude of perturbations are varied. A subsequent demonstration that the proposed perturbation distribution enables a different/better set of tradeoffs would greatly strengthen the paper.\n* The paper claims that an advantage over previous theoretical analysis is that the result in this paper does not require the variance to be annealed throughout training. However, the proposed adaptive variance method will anneal the variance throughout training. Can the authors add some additional discussion to resolve the inconsistency?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QquRC8bpDs", "reviewer": "Reviewer_AyRx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- A clever idea to bring RWP and AWP by finding what’s the important factors of the perturbations (maybe it is only the scale)\n- they justify the propose approach with a theoretical analysis, showing that perturbation size has a large effect on convergence\n- the experiments are impressive, delivering good performance with much less compute\n\nWeakness:\n\n- Theoretical results. I am not an expert in optimization and I didn’t check the details of the theorems. Looking at the final theorem, I am wondering why in Thm 1, a larger peturbation always imply worse convergence. If so, why would we adopt pertuabtions at all. I believe that there should be some tradeoffs there, and it’s good to include it as a whole picture.\n- It seems that the mixed training delivers most of the empirical improvement, and the improvement of the adaptive scheme ARWP seems to have marginal improvement, sometimes inferior to SAM (see Table 2). It is good to have an ablation on the contributions of the two techniques.\n- No ViT result on ImageNet, which is strange (Table 2 has ViT results on CIFARs). It would be good to show how it works for ViTs as well, which is the  trending method now.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new version of random weight perturbation (RWP) that mitigates the gap between adversarial weight perturbation (SAM) and RWP. They find that random weight perturbation require orders of larger perturbation sizes and propose to use other batches or historical gradient to help adjust the perturbation size for each parameter. Experiments show that the proposed algorithm can acheive comparable or even better performance without using as much compute as SAM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness.\n\nMinor:\n\np2 - lie two aspects -> lies in two aspects", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: - The paper is well-written and easy to follow.
- The proposed algorithm is simple and easy to implement.
- The proposed algorithm has some benefits over SAM (in terms of computational complexity), and yet outperforms SAM.
- The experiments are diverse and done for large-scale architectures.
- The proposed method works well in all experiments. 
- Ablation studies on the hyperparameters are also provided.

The authors were active during the rebuttal period.
Decision: Accept (poster)