Improving robustness to corruptions with multiplicative weight perturbations
Authors: Trung Trinh
Markus Heinonen
Luigi Acerbi
Samuel Kaski
Venue: NeurIPS 2024 spotlight (2024)
Keywords: covariate shift
corruption robustness
generalization
regularization
training method
deep learning
Primary area: optimization_for_deep_networks
TL;DR: We show that simply perturbing weights during training with random multiplicative noises can improve robustness of neural networks to a wide range of corruptions.

Abstract: Deep neural networks (DNNs) excel on clean images but struggle with corrupted ones. Incorporating specific corruptions into the data augmentation pipeline can improve robustness to those corruptions but may harm performance on clean images and other types of distortion. In this paper, we introduce an alternative approach that improves the robustness of DNNs to a wide range of corruptions without compromising accuracy on clean images. We first demonstrate that input perturbations can be mimicked by multiplicative perturbations in the weight space. Leveraging this, we propose Data Augmentation via Multiplicative Perturbation (DAMP), a training method that optimizes DNNs under random multiplicative weight perturbations. We also examine the recently proposed Adaptive Sharpness-Aware Minimization (ASAM) and show that it optimizes DNNs under adversarial multiplicative weight perturbations. Experiments on image classification datasets (CIFAR-10/100, TinyImageNet and ImageNet) and neural network architectures (ResNet50, ViT-S/16, ViT-B/16) show that DAMP enhances model generalization performance in the presence of corruptions across different settings. Notably, DAMP is able to train a ViT-S/16 on ImageNet from scratch, reaching the top-1 error of 23.7% which is comparable to ResNet50 without extensive data augmentations.

Improving robustness to corruptions with multiplicative weight perturbations
Trung Trinh'
Markus Heinonen'
Luigi Acerbi² Samuel Kaski 1.3
DDpartmeett of Computer Science, Aalto University, Finland 2 DDpartment of Computer Science, University of Helsinki, Finland 3 Department of Computer Science, University of Manchester, United Kingdom (trung. trinh, markus. o heinonen, samuel luigi acerbi Seeerbi@elsiink.fff
faallo... ff. fi.
Abstract
Deep neural networks (DNNs) excel on clean images but struggle with corrupted ones. Incorporating specific corruptions into the data augmentation pipeline can im- prove robustness to those corruptions but may harm performance on clean images and other types of distortion. In this paper, improves the robustness of DNNs to a wide range of corruptions without compro- mising accuracy clean images. We first demonstrate that input perturbations be mimicked by multiplicative perturbations in the weight space. Leveraging this, we propose Data Augmentation via Multiplicative Perturbation (DAMP), training method that optimizes DNNs under random multiplicative weight perturbations. We also examine the recently proposed Adaptive Sharpness-Aware Minimization (ASAM) and show that it optimizes DNNs under adversarial multiplicative weight perturbations. Experiments on image classification datasets CUFFAR-10/100, Tiny- ImageNet and ImageNet) and neural network architectures (ResNet50, ViT-S/16, ViT-B/16) show that DAMP enhances model generalization performance the presence of corruptions across different settings. Notably, DAMP is able to train a ViT-S/16 on ImageNet from scratch, reaching the top- error of 23.7% which is comparable to ResNet50 without extensive data augmentations.
introduce an alternative approach that
1
Introduction
Deep neural networks (DNNs) demonstrate impressive accuracy in computer vision tasks when evaluated on carefully curated and clean datasets. However, their performance significantly declines when test images are affected by natural distortions such as camera noise, changes in lighting and weather conditions, or image compression algorithms (Hendrycks and Dietterich, 2019). This drop in performance is problematic in production settings, where models inevitably encounter such perturbed inputs. Therefore, it is crucial to develop methods that produce reliable DNNs robust common image corruptions, particularly for deployment in safety-critich systems (Amodei et al., 2016).
To enhance robustness against specific corruption, one could simply include it in the data aug- mentation pipeline during training. However, this approach can diminish performance on clean images and reduce robustness other types of corruptions (Geirhos et al., 2018). More advanced data augmentation techniques (Cubuk et al., 2018; Hendrycks et al., 2019; Lopes et al., 2019) have been developed which effectively enhance corruption robustness without compromising accuracy on clean images. Nonetheless, a recent study by Mintun et al. (2021) has identified a new set of image corruptions which models trained with these techniques remain vulnerable. Besides data
Our code is available at https://gitae comm/brungtrinn44/DDa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

æ1
01
61
W1
#1
I01
wi t1/m1))
W161
E2
W2
w2(1 + e2/its)
22
#2 + C2
z
2²2
:
6n
1x (c/aa))
Wn5n
In
En + Cn
fn
In
(a) w (x + cc)
(b) = = (w
(c) = (w E)_ x, E~ <>P(E)
w x in the presence of (a) covariate shift (b) a multiplicative weight perturbation (MWP) equivalent to 6, and (c) random MWPs denotes the Hadamard product. Figs. (a) and (b) show that for covariate shift one can always find an equivalent MWP. From this intuition, we propose inject random MWPs & to the forward pass during training as shown in Fig. (c) to robustify DNN to covariate shift.
Figure I: Depictions of a pre-activation neuron 2 == w
augmentation, ensemble methods such as Deep ensembles and Bayesian neural networks have also been shown improve generalization in the presence of corruptions (Lakshminarayanan et al., 2017; Ovadia et al., 2019; Dusenberry et al., 2020; Trinh et al., 2022). However, the training and inference costs of these methods increase linearly with the number of ensemble members, rendering them less suitable for very large DNNs.
Contributions In this work, show that simply perturbing weights with multiplicative random variables during training can significantly improve robustness to a wide range of corruptions. Our contributions are as follows:
We show in Section and Fig. that the effects of input corruptions can be simulated during training via multiplicative weight perturbations.
From this insight, we propose new training algorithm called Data Augmentation via Multiplicative Perturbations (DAMP) which perturbs weights using multiplicative Gaussian random variables during training while having the same training cost as standard SGD.
In Section 3, we show connection between adversarial multiplicative weight perturbations and Adaptive Sharpness- Minimization (ASAM) (K woon et al., 2021).
Through a rigorous empirical study in Section 4, we demonstrate that DAMP consistently im- proves generalization ability of DNNs under corruptions across different image classification datasets and model architectures.
Notably, we demonstrate that DAMP can train a Vision Transformer (ViT) (Dosovitskiy et al., 2021) from scratch ImageNet, achieving similar accuracy to ResNet50 (He et al., 2016a) in 200 epochs with only basic Inception-style preprocessing (Szegedy et al., 2016). This is significant as VIT typically requires advanced training methods or sophisticated data augmentation to match ResNet50's performance when being trained on ImageNet from seratch (Chen et al., 2022; Beyer et al., 2022). We also show that DAMP can be combined with modern augmentation techniques such as MixUp (Zhang et al., 2018) and RandAugment (Cubuk et al., 2020) further improve robustness of neural networks.
2 Data Augmentation via Multiplicative Perturbations
In this section, we demonstrate the equivalence between input corruptions and multiplicative weight perturbations (MWPs), shown in Fig. 1, motivating the use of MWPs for data augmentation.

---

x
x (1)
E(w, v)
(10) (()
Els,
Figure 2: Depiction of how a corruption affects the output of a DNN. Here Xg = g(x). The corruption g creates a shift ö 5gx(0) Xg x in the input which propagates into shifts 6 ggx(() in the output of each layer. This will eventually cause a shift in the loss Ege. This figure explains why the model performance tends to degrade under corruption.
2.1 Problem setting
Given a training data set S = {(x,, y:]]k=1 C X' x y drawn i.i.d. from the data distribution D.
we seek to learn model that generalizes well on both clean and corrupted inputs. We denote G as a set of functions whose each member : X - X' represents an input corruption. That is, for each x X, g(x) ) is corrupted version of x.² We define g(S) == (g(xx) )))))) ]== as the training set corrupted by g. We consider a DNN f X + Y parameterized by w E W. Given
per-sample loss é W X' x y Rt, the training loss is defined as the average loss over the samples L)w;S)= == N Ek=1 f(w, Xk: Mk). Our goal is to find w which minimizes:
L(w; G(S)) == Eg-g|[(w;g(() g())]
(I)
without knowing exactly the types of corruption contained in G. This problem crucial for the reliable deployment of DNNs, especially in safety-critical systems, since it is difficult to anticipate all potential types corruption the model might encounter production.
2.2 Multiplicative weight perturbations simulate input corruptions
To address the problem above, we make two key assumptions about the corruptions in G: Assumption (Bounded corruption). For each corruption function g X' in 9, there exists constant M >> 0 such that lg(x)) -x|2 S M for all x E X.
Assumption 2 (Transferable robustness). A model's robustness to corruptions in G can be indirectly enhanced by improving its resilience more easily simulated set of inpul perturbations.
Assumption implies that the corrupted versions an input x must be constrained within a bounded neighborhood of in the input space. Assumption 2 is corroborated by Rusak et al. (2020), who demonstrated that distorting training images with Gaussian noise improves a DNN's performance against various types of corruption. We further validate this observation for corruptions beyond Gaussian noise in Section 4.1. However, Section 4.1 also reveals that using corruptions as data augmentation degrades model performance on clean images. Consequently, we need identify method that efficiently simulates diverse input corruptions during training, thereby robustifying DNN against a wide range of corruptions without compromising its performance on clean inputs.
One such method involves injecting random multiplicative weight perturbations (MWPs) into the forward pass of DNNs during training. The intuition behind this approach is illustrated in Fig. 1. Essentially, for pre-activated neuron z = w x in DNN, given a corruption causing a covariate always find equivalent MWP g(e,
shift in the input x, Figs. la and l show that
(x+ T
(w x() x, E(c,x)
+ e/x
where o denotes the Hadamard product. This observation suggests that input corruptions can be simulated during training by injecting random MWPs into the forward pass, depicted in Fig. 1c. resulting in model more robust to corruption. We thus move the problem simulating corruptions from the input space to the weight space.
Here we provide theoretical arguments supporting the usage of MWPs to robustify DNNs. To this end, we study how corruption affects training loss. We consider feedforward neural network f(x; w)
For instance, if is clean image then gxx) could be corrupted by Gaussian noise.

---

of depth H parameterized by w (
M= E W. which we define recursively as follows: (()(x)) ==x, z! (x) == ((1)p(h-)(xx))
(x) == (z (x)), vh = I.... H (3)
where f(x: w) == (x) and (*a) is the non-lineat activation of layer h. For brevity, we use x x(h)
and X as shorthand notations for f(h) (x) and f()) (gx)) respectively. Given corruption function
g. Fig. 2 shows that creates a covariate shift 0gx(0) := Xg (0) x() in the input x leading to shifts 6gx(1) := =x x(h) in the output of each layer. This will eventually cause a shift in the per-sample loss 8gl(w.X,) x, == (g 8) E(W, x, y). The following lemma characterizes the connection between 6 6gl(w, x. y) and (gx(h);
Lemma 1. For all I,..., H and for all x E X, there exists a scalar
C.M (x)
> 0 such that:
C((h)
8gl(wxxU1) x, (V (annn x, v) 8 8 kgx(h) w(tt))
K
2
Here @ denotes the outer product of two vectors, (>-) denotes the Frobenius inner product of two matrices of the same dimension, || ||r is the Frobenius norm, and Ve0ns x, y) is the Jacobian of
the per-sample loss with respect to the pre-activation output (x) at layer h. To prove Lemma 1, we use Assumption I and the following assumption about the loss function:
Assumption 3 (Lipschitz-contimuuus objective input gradients). The inpul gradient of the per-sample loss V Vxl(w, x, is Lipschitz continuous.
Assumption 3 allows us define a quadratic bound of the loss function using second-order Taylor expansion. The proof of Lemma is provided in Appendix A. Using Lemma 1, we prove Theorem 1, which bounds the training loss in the presence of corruptions using the training loss under multiplicative perturbations in the weight space:
Theorem 1. For function g : X + X satisfying Assumption 1 and loss function L satisfying Assumption 3, there exists Eg E and C2 > O such that:
CB 2
((w;g(8)) S Llw Egi +
(5)
We provide the proof of Theorem in Appendix B. This theorem establishes an upper bound for the target loss in Eq. (I):
C(w; G(S)) S Ex-g [(w o Eg: + Cs 2
|
(6)
This bound implies that training a DNN using the following loss function:
L S) == - [C(w E: {; + Wwl
(7)
where the expected loss is taken with respect to a distribution E of random MWPs &, will minimize the upper bound of the loss L(w; (SS) of hypothetical set of corruptions simulated by & This approach results in model robust to these simulated corruptions, which, according Assumption 2, could indirectly improve robustness to corruptions in g.
We note that the second term in Eq. (7) is the Læ-regularization commonly used in optimizing DNNs. Based on this proxy loss, we propose Algorithm which minimizes the objective function Eq. (7) when E is an isotropic Gaussian distribution M(1, I²)). We call this algorithm Data Augmentation via Multiplicative Perturbations (DAMP), as it uses random MWPs during training to simulate input corruptions, which can be viewed as data augmentations.
The standard method to calculate the expected loss in Eq. (7), which lacks closed-form solution, is the Monte Carlo (MC) approximation. However, the training cost of this approach scales linearly with the number of MC samples. To match the training cost of standard SGD, Algorithm I divides each data batch into M equal-sized sub-batches (Line 6) and calculates the loss on each sub-batch with different multiplicative noises from the noise distribution (Lines 7-9). The final gradient is obtained by averaging the sub- bub-batch gradients (Line 11). Algorithm 1 is thus suitable for data parallelism in multi-GPU training, where the data batch is evenly distributed across M > 1 GPUs. Compared to SGD, Algorithm I requires only two additional operations: generating Gaussian samples and wise multiplication, both of which have negligible computational costs. In our experiments, we found that both SGD and DAMP had similar training times.
Remark

---

Algorithm DAMP: Data Augmentation via Multiplicative Perturbations
1: Input: training data ((xx; yk )
k==1 neural network f(::w) parameterized by RR..
number of iterations T, step sizes {ne 17 number of sub-batch M, batch size B divisible by M,
a noise distribution E (1, IIIp). weight decay coefficient X, a loss function RRP + R+.
2: Output: Optimized parameter w
3: Initialize parameter (0)
4: for t = 1 to I do
5: Draw a mini-batch B = ((x,, yb) ~.
6: Divide the mini- batch into M disjoint sub-batches {Bm 1m=1 of equal size.
7:
for m to M in parallel do
8:
Draw a noise sample &. E.
9:
Compute the gradient gm V C(w, Bm) wiltfe" end for
11: Compute the average gradient: g M7 Em= Bm.
12: Update the weights: ((+1) (() 1)t (g Aw(())
13:
end for
3 Adaptive Sharpness-Aware Minimization optimizes DNNs under adversarial multiplicative weight perturbations
In this section, we demonstrate that optimizing DNNs with adversarial MWPs follows a similar update rule to Adaptive Sharpness-Awen Aware Minimization (ASAM) (Kwon et al., 2021). We first provide a brief description of ASAM and its predecessor Sharpness- Aware Minimization (SAM) (Foret et al., 2021):
SAM Motivated by previous findings that wide optima tend generalize better than sharp ones (Keskar et al., 2017; Jiang et al., 2020), SAM regularizes the sharpness of an optimum by solving the following minimax optimization:
min max L(w +) 1wwl 1
(8)
which can be interpreted as optimizing DNNs under adversarial additive weight perturbations. To efficiently solve this problem, Foret al. (2021) devise two-step procedure for each iteration t:
$ ( Vwl(w() (IV
((++)) = (()) The (VwC(W)) + () Xw(
(9)
where is the learning rate. Each iteration of SAM thus takes twice as long to run than SGD.
Kwon et al. (2021) note that SAM attempts to minimize the maximum loss over rigid sphere of radius p around an optimum, which is not suitable for ReLU networks since their parameters can be freely re-scaled without affecting the outputs. The authors thus propose ASAM an alternative optimization problem to SAM which regularizes the adaptive sharpness an optimum:
ASAM
min
max I E112SP
L(w + E: + 2 |w
(10)
where To an invertible linear operator used to reshape the perturbation region (so that is not necessarily sphere in SAM). Kwon et al. (2021) found that Tu, = |w produced the best results. Solving Eq. (10) in this case leads the following two-step procedure for each iteration
( (w) o (. ((++)) (())) ni (vo + ;2 ) (11)
wltt) ((wltt)) S)
Similar to SAM, each iteration of ASAM also takes twice as long to run than SGD.
5

---

ASAM and adversarial multiplicative perturbations Algorithm minimizes the expected loss in Eq. (7). Instead, we could minimize the loss under the adversarial MWP:
m (w: S) max L(w + wo 2 1
(12)
IEllaS
Following Foret et al. (2021), we solve this optimization problem by using a first-order Taylor expansion of C(w + S) to find an approximate solution of the inner maximization:
arg max (ww w o E; S) S
arg max ()w; + (w of &, Vul(w;))
(13)
The maximizer of the Taylor expansion is:
Vwl(w;:) ((w) P ll SWW 2
(14)
Subtituting back into Eq. (12) and differentiating, we get:
mmxm (w; 8) = VwL(O; + === Vol(ū: + Au
(15)
Vol((; + Vo Vo (wo((w)) Va(((;;)) da
(16)
where @ the perturbed weight:
w² [wb(w; e = w w [ = w + ww VwL(w; S)ll2
(17)
Similar to Foret et al. (2021), we omit the second summand in Eq. (16) for efficiency, it requires calculating the Hessian of the loss. We then arrive at the gradient formula the update rule of ASAM in Eq. (11). We have thus established a connection between ASAM and adversarial MWPs.
4 Empirical evaluation
In this section, we assess the corruption robustness of DAMP and ASAM in image classification tasks. We conduct experiments using the CIFAR-10/100 (Krizhevsky, 2009), TinylmageNa (Le and Yang, 2015), and ImageNet (Deng et al., 2009) datasets. For evaluation on corrupted images, we utilize the CIFAR-10/100- TinylmageNet-C, and ImageNet-C datasets provided by Hendryeks and Dietterich (2019), as well as ImageNet-C (Mintun et al., 2021), ImageNet-D (Zhang et al., 2024), ImageNet- (Hendrycks et al., 2021), ImageNNt-SSkech (Wang et al., 2019), ImageNett(Drawing Cartoon) (Salvador and Oberman, 2022), and ImageNet-Hard (Taesiri et al., 2023) datasets, which encapsulate wide range of corruptions. Detail descriptions of these datasets are provided in Appendix E. We further evaluate the models on adversarial examples generated by the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014). In terms of architectures, we use ResNet18 (He al., 2016a) for CIFAR- 10/100, 00,, NreActResNer118 8 (He et al., 2016b) for TinylmageNe ViT-S/16, and ViT-B/16 (Dosovitskiy et al., 2021) for ImageNet. We ran all experiments on a single machine with 8 Nvidia V100 GPUs. Appendix F includes detailed information for each experiment.
ResNet50 (He et al., 201 6a),
4.1 Comparing DAMP to directly using corruptions as augmentations
In this section, compare the corruption robustness of DNNs trained using DAMP with those trained directly on corrupted images. To train models on corrupted images, we utilize Algorithm described in the Appendix. For given target corruption g. Algorithm 2 randomly selects half the images in each training batch and applies g to them. This random selection process enhances the final model's robustness the target corruption while maintaining its accuracy on clean images. We use the imagecorruptions library (Michaelis et al., 2019) to apply the corruptions during training.
We use the corruption error CE (Hendrycks and Dietterich, 2019) which measures the predictive error of classifier ʃ in the presence of corruption e. Denote Efe- as the error of classifier of under corruption e with corruption severity s, the corruption error CEA is defined as:
Evaluation metric
5
5
CE!-
BlC
Σ
(18)
8,C

---

zoom, Blur
Impulse, noie
shet,
marse moise
geussian neice
medin plur
pizelate
brightness
snow
Hoat
contrest
109
detocud, Mutt
glase Stur
breg wwwprroaeeeeeo pevgreccion
elestic, transform
none
Arg
ISCD, nonw)
1,00
1.00
1.00
1.00
(S6O, zoom_blur
0.54
0.80
0.80
9.99
0.88
0.06
86
(SCD, impulse neise)
-0.95
0.30
8.83
0.85
0.95
0.86
1.04
0.93
9.91
0.09
8.08
0.98
0.92
6.79
0.98
1.06
0.91
1.0
(SCD, shot_ shot_neise)
0:90
0.44
0.46
0.42
0.97
0.75
0.93
0:68
1.29
4.4
0.99
0.58
6.
1.05
0.92
(SGD, gaussian, neise)
-0.9
0.64
0.47
0.42
0.90
0.76
1.39
0.96
0.89
1.32
50
1.00
0.59
0.73
1.05
0.40
0.93
0.9
150D, motion blud
0.85
0.94
0.94
0.86
.000
0.98
Figure 3: DAMP improves robustness to all corruptions while preserving accuracy on clean images. Results of R
experiments averaged over 5 seeds. The heatmap shows CE described in Eq. (18) (lower is better), where each row corresponds a tuple of training (method, corruption), while each column corresponds to the test corruption. The Avg column shows the average of the results of the previous columns. none indicates no corruption. We use the models trained under the SGD/nons setting (first row) as baselines to calculate the CE:- The last five rows are the 5 best training corruptions ranked by the results in the Avg column.
For this metric, lower is better. Here /bsaeline is baseline classifier whose usage is to make the error more comparable between corruptions as some corruptions can be more challenging than others (Hendrycks and Dietterich, 2019). For each experiment setting, we use the model trained by SGD without corruptions as foaseline-
Results We visualize the results for the ResNet18/CIFAL 100 setting in Fig. 3. The results for the ResNet18/CIFAR-- and PreActResNet 8/TinyyImma ImageNet settings are presented in Figs. and in the Appendix. Figs. 3, 5 and 6 demonstrate that DAMP improves predictive accuracy over plain SGD across all corrppiinn without compromising accuracy on clean images. Although Fig. 3 indicates that including zoom_ blurrruuuuuuru as augmentation when training ResNet18 8 on CIFAR-100 100 yields better results than DAMP on average, it also reduces accuracy on clean images and the bri brightness corruption. Overall, these figures show that incorporating specific corruption as data augmentation during training enhances robustness to that particular corruption but may reduce performance on clean images and other corruptions. In contrast, DAMP consistently improves robustness across all corruptions. Notably, DAMP even enhances accuracy on clean images in the PreActResNet 8/ setting, as shown in Fig. 6.
4.2 Comparing DAMP to random additive perturbations
In this section, we investigate whether additive weight perturbations can also enhance corruption robustness. To this end, compare DAMP with its variant, Data Augmentation via Additive Perturbations (DAAP). Unlike DAMP, DAAP perturbs weights during training with random additive Gaussian noises centered presents the results of DAMP and DAAP under different noise standard deviations, alongside standard SGD. Overall, Fig. shows that across different experimental settings, the corruption robustness of DAAP is only slightly better than SGD and is worse than DAMP. Therefore, we conclude that MWPs are better than their additive counterparts at improving robustness to corruptions.
as detailed in Algorithm 3 in the Appendix. Fig. in the Appendix
4.3 Benchmark results
In this section, we compare DAMP with Dropout (Srivastava et al., 2014), SAM (Foret et al., 202 1), and ASAM (Kwon et al., 2021). For SAM and ASAM, we optimize the neighborhood size p by using 10% of the training set as validation set. Similarly, we adjust the noise standard deviation o for DAMP and the drop rate p for Dropout following the same procedure. For hyperparameters and additional training details, please refer to Appendix F.
CIFAR- 10/100 and TinyImageNet. Fig. 4 visualizes the predictive errors of DAMP and the baseline methods on 1IFAR-10/100 and TinylmageNet, with all methods trained for the same number of epochs. demonstrates that DAMP consistently outperforms Dropout across various datasets and corruption severities, despite having the same training cost. Notably, DAMP outperforms SAM under
7

---

ResNet18 ResNet18/
PrehstResNet18 TinylmageNet.
Commston Intondyy
Dropout
Figure 4: DAMP surpasses SAM on corrupted images in most cases, despite requiring only half the training cost. We report the predictive errors (lower is better) averaged over 5 seeds. A severity level of indicates no corruption. We use the same number of epochs for all methods.
Table DAMP surpasses the baselines on corrupted images in most cases and on average. We report the predictive errors (lower is better) averaged over 3 seeds for the ResNet50 / ImageNet experiments. Subscript numbers represent standard deviations. We evaluate the models on IN-(C. C, A, D, Sketch, Drawing, Cartoon, Hard], and adversarial examples generated by FGSM. For FGSM, we use 2/224. For IN-(C, C), we report the results averaged over all corruption types and severity levels. We use 90 epochs and the basic Inception-style preprocessing for all experiments.
Method
Clean Eror (%)
Comupted Eroo (9)
FGSM
A
C
C
Cartoon
D
Drawing
Sketch
Hard
Avg
Dropout
23.6p
90.7p.2
05.7.co.1
61.7oz
61 61.6.c0.1
19. 19.6p.2
88.9co.1
77.41s
78.3gs
85.8
76.6
DAMP
23.8.go.1
88 3..3
96.2.m.1
58.7..01
44.4.0.
88.7.20
71.1oa
76.3o
85.3p.2
74.2
SAM
23.2.o.1
90.4p.2
96.6u.1
60.2
60.7p.1
47.60.1
88 3o.1
74.8c0a
77.5
85.8p
75.8
ASAM
22.8o.1
89.7p.2
96.8p.
58.96
59.2a
45.5.g0.
88.7a.
72.3
76.4.
85.2o.
74.7
most corruption scenarios, even though SAM takes twice as long to train and has higher accuracy on clean images. Additionally, DAMP improves accuracy on clean images over Dropout on CIFAR- 100 and TinylmageNet. Finally, ASAM consistently surpasses other methods on both clean and corrupted images, it employs adversarial MWPs (Section 3). However, like SAM, each ASAM experiment takes twice as long as DAMP given the same epoch counts.
ResNet50 / ImageNet Table presents the predictive errors for the ResNet50 / ImageNet setting on variety of corruption test sets. It shows that DAMP consistently outperforms the baselines in most corruption scenarios and on average, despite having half the training cost of SAM and ASAM.
VIT-S16 ImageNet/ / Basic augmentations Table presents the predictive errors for the ViT-S16 / ImageNet setting, using the training setup from Beyer et al. (2022) but with only basic Inception-style preprocessing (Szegedy et al., 2016). Remarkably, DAMP can train ViT-S16 from scratch in 200 epochs to match ResNet50's accuracy without advanced data augmentation. This is significant as ViT typically requires either extensive pretraining (Dosovitskiy et al., 2021). comprehensive data aug- mentation (Beyer et al., 2022), sophisticated training techniques (Chen et al., 2022), or modifications to the original architecture (Yuan et al., 2021) to perform well on ImageNet. Additionally, DAMP consistently ranks in the top 2 for corruption robustness across various test settings and has the best corruption robustness on average (last column). Comparing Tables and 2 reveals that VIT-S16 more robust to corruptions than ResNet50 when both have similar performance clean images.
VIT / ImageNet/ / Advanced augmentations Table 3 presents the predictive errors of ViT-S16 and ViT-B on ImageNet with MixUp (Zhang et al., 2018) and RandAugment (Cubuk et al., 2020). These results indicate that DAMP can be combined with modern augmentation techniques to further improve robustness. Furthermore, using DAMP to train a larger model (ViT-B16) yields better results than using SAM/ASAM to train a smaller model (ViT-S16), given the same amount of training time.
8

---

Table 2: ViT-S16/ ImageNet (IN) with basic Inception-style data augmentations. Due to the high training cost, we report the predictive error (lower is better) of a single run. We evaluate corruption robustness the models using IN- C, A, D, Sketch, Drawing, Cartoon, Hard), and adversarial examples generated by FGSM. For IN-(C, C), we report the results averaged over all corruption types and severity levels. For FGSM, we use = 2/ 2/224. We also report the runtime of each experiment, showing that SAM and ASAM take twice long run than DAMP and AdamW given the same
number of epochs. DAMP produces the most robust model on average.
Method
#Epochs
Runtime
Clean Error (%)
Comppted Emur (%)
L
FGSM
Cartoon
Drawing
Sketch
Hard
Avg
Dropout
100
20.60
28.35
93.47
93.44
65.87
64.52
50.37
91.15
79.62
88.06
87.19
79.30
200
41.1h
28.74
90.95
93.33
66.90
64.83
51.23
92.56
81.24
87.99
87.60
79.63
DAMP
100
20.7h
25.50
92.76
92.92
57.85
57.02
44.78
88.79
69.92
83.16
85.65
74.76
200
41.1h
23.75
84.33
90.56
55.58
55.58
41.06
87.87
68.36
81.82
84.18
72.15
SAM
100
23.91
87.61
93.96
88.23
85.54
73.42
ASAM
100
41.lh
24.01
85.85
92.99
55.13
55.64
40.74
89.03
67.80
81.47
84.31
72.55
Table 3: VIT / ImageNet (IN) with MixUp and RandAugment. We train ViT-S16 and ViT-B 16 on ImageNet from scratch with advanced data augmentations (DAs). We evaluate the models on IN- C, A, D, Sketch, Drawing, Cartoon, Hard], and adversarial examples generated by FGSM. For FGSM, we use c = 2/224. For IN-(C, C). we report the results averaged over all corruption types and severity levels. These results indicate that: (i) DAMP can be combined with modern DA techniques to further enhance robustness; (ii) DAMP is capable of training large models like ViT-B 16; (iii) given the same amount of training time, it is better to train large model (ViT-B16) using DAMP than to train a smaller model (ViT-S16) using SAM/ASAM.
Model Method
#Ppochs
Runtime
Clean Error (56)
Comupted Eror (%) 4
RGSM
C
A
C
Carloon
D)
Drawing
Sketch
Hard
Avg
Drapout
500
111h
20.25
44.72
VIT S16
DAMP
500
111h
20.09
59.87
39.30
83.12
43.18
34.01
84.74
54.16
68.03
80.05
60.72
SAM
300
123h
20.17
59.92
40.05
83.91
44.04
34.34
85.99
55.63
70.85
80.18
61.66
ASAM
300
123h
20.38
39.44
83.64
43.41
85.41
69.13
61.02
Drapout
275
123h
20.41
56.43
39.14
$2.85
43.82
33.13
87.72
56.15
71.36
79.13
61.08
WT
DAMP
275
124h
19.36
55,20
80.49
41.6
31.63
87.06
$2.32
67.91
78.69
59.19
B16
SAM
150
135h
19.84
61.85
39.09
82.69
43.53
32.95
$8.38
55.33
71.22
79.48
61.61
ASAM
150
136h
19.40
58.87
37.41
82.21
41.18
30.76
$8.03
51.84
69.54
78.83
59.85
5 Related works
Perhaps most relevant to our method is Dropout (Srivastava et al., 2014) and its many variants, such as DropConnect (Wan et al., 2013) and Variational Dropout (Kingma et al., 2015). These methods can be viewed as DAMP where the noise distribution E is a structured multivariate Bernoulli distribution. For instance, Dropout multiplies all the weights connecting to node with binary random variable ~ Bernoullipp). While the main motivation of these Dropout methods is to prevent co-adaptations of neurons to improve generalization is to improve robustmess to input corruptions without harming accuracy on clean data. Nonetheless, experiments show that DAMP improve generalization on clean data in certain scenarios, such as PreActResNet 88
Dropout
clean data, the motivation of DAMP
and ViT-S116/mageNet..
Ensemble methods Ensemble methods, such as Deep ensembles (Lakshminarayanan et al., 2017) and Bayesian neural networks (BNNs) (Graves, 2011; Blundell et al., 2015; Gal and Ghahramani, 2016; Louizos and Welling, 2017; Izmailov et al., 2021; Trinh et al., 2022), have been explored as effective defenses against corruptions. Ovadia et al. (2019) benchmarked some of these methods, demonstrating that they are more robust to corruptions compared to a single model. However, the training and inference costs of these methods increase lincarly with the number of ensemble members, making them inefficient for use with very large DNNs.
Data augmentation Data augmentations aim at enhancing robustness include AugMix (Hendrycks et al., 2019), which combines common image transformations; Patch Gaussian (Lopes et al., 2019), which applies Gaussian noise to square patches; ANT (Rusak et al., 2020), which uses adversarially learned noise distributions for augmentation; and AutoAugment (Cubuk et al., 2018), which learns

---

augmentation policies directly from the training data. These methods have been demonstrated to improve robustness to the corruptions ImageNet-C (Hendrycks and Dietterich, 2019). Mintun et al. (2021) attribute the success of these methods to the fact that they generate augmented images perceptually similar to the corruptions ImageNet-C and propose ImageNet-C, test set of 10 new corruptions that are challenging to models trained by these augmentation methods.
Test-time adaptations via BatchNorm One effective approach to using unlabelled data to improve corruption robustness is to keep BatchNorm (loffe and Szegedy, 2015) on at test-time to adapt the batch statistics to the corrupted test data (Li et al., 2016; Nado et al., 2020; Schneider al., 2020; Benz et al., 2021). A major drawback is that this approach cannot be used with BatchNorm-free architectures, such Vision Transformer (Dosovitskiy et al., 2021).
6 Conclusion
In this work, we demonstrate that MWPs improve robustness of DNNs wide range of input corruptions. We introduce DAMP, a simple training algorithm that perturbs weights during training with random multiplicative noise while maintaining the same training cost as standard SGD. We further show that ASAM (Kwon et al., 2021) can be viewed as optimizing DNNs under adversarial MWPs. Our experiments show that both DAMP and ASAM indeed produce models that are robust to corruptions. DAMP is also shown to improve sample efficiency of Vision Transformer, allowing it to achieve comparable performance to ResNet50 on medium size datasets such as ImageNet without extensive data augmentations. Additionally, DAMP can be used in conjunction with modern augmentation techniques such as MixUp and RandAugment to further boost robustness. As DAMP is one future direction is to explore its effectiveness in domains other than computer vision, such as natural language processing and reinforcement learning. Another direction is to explore alternative noise distributions to the Gaussian distribution used in our work.
Limitations Here we outline some limitations of this work. First, the proof of Theorem assumes simple feedforward neural network, thus it does not take into accounts modern DNN's components such as normalization layers and attentions. Second, we only explored random Gaussian multiplicative perturbations, and there are likely more sophisticated noise distributions that could further boost corruption robustness.
Broader Impacts
Our paper introduces a new training method for neural networks that improves their robustness to input corruptions. Therefore, believe that our work contributes towards making deep leading models safer and more reliable to use in real-world applications, especially those that are safety-critical. However, with other methods that improve robustness, our method could also be improperly used in applications that negatively impact society, such as making mass surveillance systems more accurate and harder to fool. To this end, we hope that practitioners carefully consider issues regarding fairness, bias and other potentially harmful societal impacts when designing deep learning applications.
Acknowledgments
This work was supported by the Research Council of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI and decision no. 359567, 345604 and 341763), ELISE Networks of Excellence Centres (EU Horizon: 2020 grant agreement 951847) 147) and UKRI Turing Al World-Leading Researcher Fellowship (EP/W002973/1), We acknowledge the computational resources provided by Aalto Science-IT project and CSC-IT Center for Science, Finland.
References
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv: 1203, 2019.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, arXiv preprint arXiv:1606.06565. 2016.
10

---

Robert Geirhos, Carlos RM Temme. Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data, arXiv preprint arXiv:1805.09500 2018
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty.
arXiv preprint arXiv: 19/2.0278, 2019.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint arXiv:1906.02611, 2019.
Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and corruptions in natural corruption robustness. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors. Advances in Neural Information Processing Systems, 2021. URL ittps://openreviier peeee/forummid-LOyyyieett
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evalu- ating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: h/tp://://///ceeeeeeeeee.. meurips.... cc/ http:///ppppe/..//..///////..0.0
Paper pdf.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snock, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank- 1 factors. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2782-2792. PMLR, 13-18 Jul 2020. URL https://prroceeding: mlr. mwr.mrrreeee hat.ppees//v1/ dusenberry20a. html.
Trung Trinh, Markus Heinonen, Luigi Acerbi, and Samuel Kaski. Tackling covariate shift with node-based Bayesian neural networks. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 21751- 21775. PMLR, 17-23 Jul 2022.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905-5914 PMLR, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https: /ttp:///eee/.....e///m// net/ net/fforum?dd-YiicaaaaTT"
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE conference on Computer Vision and Pattern Recognition, 2016a.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 2818-2826, 2016.
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations, 2022, 2022, URL

---

Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- 1k. arXiv preprint arXiv:2205.01580, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018. URL https: // //openreiie www.://ttt/teeeeeem //pppa/eriss..eet/ooootttatrrrrr Adrroddp-
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. arXiv preprint arXiv:2001.0603 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https: http::////pperreeeeeee.eeee///// met//forum//t--011mmmmmlll
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL https http:p::///eerreiew..e...
Horumiid-HLoyRLYgg
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where find them. International Conference on Learning Representations, 2020. URL attps://openrewe
www./eeeuffiiiiie htt JJgII.BVV....
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Ya Le and Xuan S. Yang. Tiny ImageNet visual recognition challenge. 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009, doi:
20w.CV 52008888.
Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. Imagenet-d: Bench- marking neural network robustness on diffusion synthetic object. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 21752-21762, June 2024.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262-15271, June 2021.
Haohan Wang, Songwei Ge, Zachary Lipton, and Erie P Xing. Learning robust global represen- tations by penalizing local predictive power. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL paper_files foppe__flees//pprr/2001 /ttp:///////////////////e/////// http://///...///otto0.00o00/////
wwwpiiiinnnnn....ieieeeeeeeeeeee neurips.. ips. cc/ .dff
Tiago Salvador and Adam M Oberman. Imagenet-cartoon and imagenet-drawing: two domain shift datasets for imagenet. In JCML 2022 Shift Happens Workshop, 2022. URL https:///ppeeerr......
jaQQt
Mohammad Reza Taesiri, Giang Nguyen, Sarra Habchi, Cor-Paul Bezemer, and Anh Nguyen. Imagenet-hard: The hardest images remaining from study of the power of zoom and spatial biases image classification. 2023.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1126521 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference Computer Vision, 2016b.
12

---

Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann. Alexan- der S. Ecker. Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprini arXiv: 1907.07484, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 19266)1199,-1958 2014. URL http::/ttmmiii wwt.eorgggpppppp.eee wwggpaapes/vlle erivastava14a. html.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pages 558-567, 2021.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropeonnect. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1058-1066 Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https:/ httpss////occceo.....oo mlr. press/ paress/v2/wwant /v2////aw////..// html.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparametcrization trick. Advances in neural information processing systems, 28, 2015.
Alex Graves. Practical variational inference for neural networks. In Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc 2011. URL https::/ppocceediiiiii Paper, .epeer.pd. pdf. paper_files files/ www.ffelllpppee////e//aee
neurips. pisicccc cc/
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pages 1613-1622. PMLR, 2015.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncer- tainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https: /ptp://////occco....cooo mlr. paress/v4B) ga116. html.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pages 2218-2227. PMLR, 2017.
Pavel Izmailov, Patrick Nicholson, Sanae Lotfi, and Andrew G Wilson. Dangers of bayesian model averaging under covariate shift. Advances in Neural Information Processing Systems, 34:3309- 3322, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference International Conference on Machine Learning Volume 37, ICML'15, page 448-456. JMLR.org. 2015.
Yanghao Ii,, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.0 04779, 2016.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 109663 2020.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Betbge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33:1 1539-115511 2020.
Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Revisiting batch normalization for improving corruption robustness. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 494-503, 2021.
Dankmar Böhning and Bruce G Lindsay. Monotonicity of muadratie-approximaaio algorithms. Annals of the Institute of Statistical Mathematics, 40(4):641-663, 1988.
13

---

A Proof of Lemma 1
Proof. Here we note that:
(b) (x)
(19)
((k) = P(M) (g(x))
(20)
8
8g (w, x, y) = E(w, xg. 11) - f(w, x, y)
(21)
8 8gx = x(( x((1)
(22)
We first notice that the per-sample loss f(w.x, x, 3) can be viewed as a function of the intermediate activation x(k) of layer (see Fig. 2). From Assumption 3, there exists a constant LA > such that:
V (n) l(w (g: Xg. y) V x(x: ((w y)ll2 I 1 112
(23)
which gives us the following quadratic bound:
f(w Xg. v) S ((w,x,
(v E y), )gx(")
La [18gx
11
(24)
where (:,-) denotes the dot product between two vectors. The results in the equation above have been proven in Böhning and Lindsay (1988). Subtracting l(w, x, v) from both side of Eq. (24) gives us:
( x, < ( (xins x. v), 6 xx)(())
+
La 16 x 2
(25)
Since the pre-activation output of layer h 1 is 2 (1+1) can rewrite the inequality above as:
(()+ (x)
w(++1)
8gf(w, < (V (ith
& ) w(h+1)
La 18
(26)
2
where denotes the outer product of vectors and ( :) denotes the Frobenius inner product of two matrices of similar dimension.
From Assumption 1, we have that there exists constant M 0 such that:
16gx (0) 12 = |x (0) = llg(x) x) x < M
(27)
Given that x1) = (1) (W (0)), we have:
||6gx (1) = ||x - x 1 < ( x (0)
(28)
Here we assume that the activate satisfies ||o(x) - (yy)ll S ||x - yl12. which true for modern activation functions such as ReLU, Since that:
10 bounded, there exists a constant ( (x) such
c00) (x)
| x
(1) 1/ [1x (1) x (2) W (1)
(0)
(
(29)
where || || denotes the Frobenius norm. Similarly, as we have proven that there exists constant ((22 (x) such that:
|| || is bounded,
(12)) (x)
| x (2)
(2)
(2) W (2)
"
(2) 2
(30)
2
Thus have proven that for all h 1,... , H. there exists constant (/( (x) such that:
(66x(()) < 2
(h)
O
(x)
(31)
By combining Eqs. (26) and (31) and setting 0gh (x) Lb, Lg ((b)) (.,1))))))))))))))))))))) (x),
arrive at Eq. (4).
14

---

B Proof of Theorem 1
Proof. From Lemma 1, have for all =0...,,P H
N
N
[(w;g(S))
Σ
(w. ye) + dgllw, M1:
(32)
N
N
k=
k=
((h)
S [((w;8) +
1
V 12 Xk. uk) @ 0g*% (h)
w
||
(33)
N
2
where C(.)) maxxes Chec (x).
Since this bound is true for all h, we can take the average:
[(w;g(8)) S ((w;S) +
1
Vain)(Ws vx) @ dg (h-1) w
1=1
Cg
(34)
2
where Cg= 1 DH -
((h)
Cg
The right-hand side of Eq. (34) can be written as:
H
L(w;S) +
1
(i) llw, Xk. xk. s0 8 x (b-1) w(N)
C5
(35)
H
N
F
= S) +
H
wwn L) W()) o (B)
+
C 1ااام
(36)
1=1
$ C(w + E(g): + CR
C
[wll = C(w o (1 + E(g) S) + 2
|
(37)
where (g) is matrix of the same dimension as (h)) whose each entry is defined as:
1 V Veinellus xx. 9b og x H
N
(h-1)
5 (h) (g)
45
(38)
Σ V Taon)((w X 8k X (h- 1)
is
The inequality in Eq. (37) is due the first-order Taylor expansion and the assumption that the training loss is locally convex at w. This assumption is expected to hold for the final solution but does not necessarily hold for any w. Eq. (5) obtained by combining Eq. (34) and Eq. (37). O
C Training with corruption
Here we present Algorithm 2 which uses corruptions as data augmentation during training, as well as the experiment results of Section 4.1 for settings in Figs. 5 and 6.
and PreAAcRRsSNel////TnnInngeee
D Training with random additive weight perturbations
Here, we present Algorithm 3 used in Section 4.2 which trains DNNs under random additive weight perturbations and Fig. 7 comparing performance between DAMP and DAAP.
E Corruption datasets
CIFAR-10/100-C (Hendrycks and Dietterich, 2019) These datasets contain the corrupted versions of the CIFAR-10/100 test sets. They contain 19 types corruption, each divided into levels of severity.
TinyImagen MeaylmageNet-C (Hendryeks and Dietterich, 2019) This dataset contains the corrupted versions of the TinylmageNet test set. It contains 19 types of corruption, each divided into 5 levels of severity.
15

---

Algorithm 2 Training with corruption
1: Input: training data S {(x;; ) k=1 neural network f(::w) parameterized by w RRP, number of iterations T, step sizes (7p -t=1= batch size B, corruption g such Gaussian noise, weight decay coefficient \, a loss function L RRP R+.
2: Output: Optimized parameter w
3: Initialize parameter (0)
4: for t = 1 to I do
5: Draw a mini-batch B = {(xb, yb)] S.
6: Divide the mini-batch into two disjoint sub-batches of equal size B1 and B2-
7: Apply the corrptiion to all samples in B1: g(B,) =
8: Compute the gradient g V (ww; g(B)) U B2).
9: Update the weights: w (1+1) w()) 11 (g + \w))).
10: end for
zoom, blur
shot, noise
gaussian roise
motion blur pixelate
glass blur
impulse naise
brightness
defocus bur elastic transfor contrast
Ipeg. compression none
tog
snow
frest
Avg
(SGD, none)
-1.00 1.00 1.00 1.00 1.00
1.00
1.00
1.00
1.00
1.00
1.00 1.00 1.00 1.00 1.00
1.00
1.00
(DAMP, none)
0.71
(5GD, zoom_blur)
0.27
0.76
7.76
0.49
0.
0.89
0.82
0.70
7.
8.82
1.07
0.37
0.69
0.56
0.92
1.11
0.73
(SGD, noise)
0.21
0.78
1.0
(SGD, gaussian _
0.70
0.21
0.17
8.82
0.46
1.77
0.86
0.64
0
0.
1.46
0.80
0 88
25
0.49
1.51
0.79
(SGD, motion _
blur)
0.58
0.83
0 77
0.35
0.70
0.
0.87
0.82
0.71
0.91
1.24
0.58
0.39
0.93
1.36
0.80
0.8
(SGD, pixelate)
0.49
0.73
0.69
0.58
0.31
1.33
0.77
0.64
0.33
0.94
1.55
0.58
0.71
.09
0.80
1.55
0.82
Figure 5: DAMP improves robustness to all corruptions while preserving accuracy clean images. Results of ResNer18/CIFAR- exxerimentt averaged over 3 seeds. The heatmap shows CF/ described in Eq. (18), where each row corresponds to tuple of of training (method, corruption on), while each column corresponds the test corruption. The Avg column shows the average of the results of the previous columns. none indicates no corruption. We use the models trained under the SGD/ /one/ setting (first row) as baselines to calculate the CE. The last five rows are the best training corruptions ranked by the results in the Avg column.
zoom, blur
Jpeg compressionn gaussian, noise shet noise
motion blur brightness
defocus blur
Impulse noise contrast
elastic, transform show frost
pixelate
fog
none
Avg
1.10
(SGD, none) 11.00 1.00 1.00 1.00 1.00 1.00
1.00
1,00
1.00
1.00 1.00 1.00 1.00 1.00
1.00
1.00
(DAMP, none) 0.90
0.67 0.88 0.93
0.91
0.90
90
0.91
0.91
0.98 0.93 0.89
0.88 0.92
0.93
0.91
1.05
(SGD, 20om_bbott blur) 0.70
0.99
0.99
0.95
94
0 7.
89
0.76
0.95
0.95
0.91
7.
0.92
0.89
1.05
0.90
(SGD, pixelate)
- 0.77 0.81
0.85
0.89
87
0 88
95
0.81
0.94
1.01
1.01
8.
0.91
0.93
1.15
0.90
1.00
(SGD, jpeg_ compreso)))))))
0. .0 0.97 0.86
7.79
80
0 9.92
1.03
0.92
0.83
1.05
111
0:93
0.96
0.99
1.17
0.95
(SGD, gaussian nolse)
0 91 0.96
0.94
7.79
80
99
04
0.93
0.85
1.04
1.10
0.95
0.97
1.00
0008
0.95
(SGD, shot_noise noise)
0. 0.92 0.96
0.93
0.81
0 0.94
.3
0,
0.84
1.05
1.09
0.96
0.96
0.99
1.10
0.96
0.90
Figure 6: DAMP improves robustness to all corruptions while preserving accuracy on clean experiments averaged over seeds. The heatmap shows CE! described in Eq. (18), where each row corresponds a tuple of training (method, corruption), while each column corresponds to the test corruption. The Avg column shows the average of the results of the previous columns. none indicates no corruption. We use the models trained under the SGD/none setting (first row) as baselines to calculate the CE.- The last five rows are the 5 best training corruptions ranked by the results in the Avg column.
images. Results of B/
ImageNet-C (Hendrycks and Dietterich, 2019) This dataset contains the corrupted versions of the ImageNet validation set, as the labels of the true ImageNet test set was never released. It contains 15 types of corruption, each divided into 5 levels of severity.
16

---

Algorithm 3 DAAP: Data Augmentation via Additive Perturbations
1: Input: training data S ((xx; )=1 neural network f(::w) parameterized by w RR..
k=
number of iterations T, step sizes {ne tt= number of sub-batch M, batch size B divisible by M, a noise distribution E (0, o pp), weight decay coefficient X, a loss function L RRP + R+.
2: Output: Optimized parameter w
3: Initialize parameter (0)
4: for t = 1 to T do
5: Draw a mini-batch B = ((x,,
S.
6: Divide the mini- batch into M disjoint sub-batches {Bm 1m=1 of equal size.
7:
for m to M in parallel do
8:
Draw a noise sample &.. ~ E. Compute the gradient gm V C(w Bm)) WitH end for
9:
11: Compute the average gradient: g 17
1 Em= Bm.
12:
Update the weights: (1+1) (t) 1)t (g \w)
13: end for
ResNet18 CIFAR-10
RedNet18 / CIFAR-100
PreActResNeet18 B/ Tinylmag eNe
Comppton Intensty
0.200 DAAP, o-
DAMP,
SCD
DAAP. 0,805
DAAP, 5.020
DAAP 0.040
Figure 7: DAMP has better corruption robustness than DAAP. We report the predictive errors (lower is better) averaged over seeds. None indicates no corruption. Mild includes severity levels 2 and 3. Severe includes severity levels 4 and 5. We evaluate DAMP and DAAP under different noise standard deviations a. These results imply that the multiplicative weight perturbations of DAMP are more effective than the additive perturbations of DAAP in improving robustness to corruptions.
ImageNet-C (Mintun et al., 2021) This dataset contains the corrupted versions of the ImageNet validation set, as the labels of the true ImageNet test set was never released. It contains 10 types of corruption, each divided into 5 levels of severity. The types of corruption ImageNet-C differ from those in ImageNet-C.
ImageNet-A (Hendrycks et al., 2021) This dataset contains natural adversarial examples, which are real-world, unmodified, and naturally occurring examples that cause machine learning model performance to significantly degrade. The images contain in this dataset, while differ from those in the ImageNet validation set, stills belong to the same set of classes.
ImageNet-D (Zhang et al., 2024) This dataset contains images belong to the classes of ImageNet but they are modified by diffusion models to change the background, material, and texture.
ImageNet eet-Cartonn and ImageNet-Drawee (Salvador and Oberman, 2022) This dataset con- tains the drawing and cartoon versions of the images in the ImageNet validation set.
Image (Wang et al., 2019) This dataset contains sketch images belonging to the classes of the ImageNet dataset.
ImageNet- (Taesiri et al., 2023) This dataset comprises an array of challenging images, curated from several validation datasets of ImageNet.
17

---

F
Training details
For each method and each setting, we tune the important hyperparameters (a for DAMP, P for SAM and ASAM) using 10% of the training set as validation set.
CIFAR-10/100 For each setting, we train a 8 for 300 epochs. We use batch size of 128. We use learning rate of 0.1 and weight decay coefficient of 5 x 10-4. We use SGD with Nesterov momentum as the optimizer with a momentum coefficient of 0.9. The learning rate is kept at 0.1 until epoch 150, then is linearly annealed to 0.001 from epoch 150 to epoch 270, then kept at 0.001 for the rest of the training. We use basic data preprocessing, which includes channel-wise normalization, random cropping after padding and random horizontal flipping. On CIFAR-10, 10, we set a = 0.2 for DAMP, p = 0.045 for SAM and p = 1.0 for ASAM. On CIFAR-I00, we set o = 0.1 for 2.0 for ASAM. Each method is trained on single host with 8 Nvidia V100 GPUs where the data batch is evenly distributed among the GPUs at each iteration (data parallelism). This means we use the number of sub-batches M 8 for DAMP.
DAMP, p 0.06 for SAM and
TinyImageNet For each setting, we train a PreActResNet18 for 150 epochs. We use a batch size of 128. We use learning rate of 0.1 and weight decay coefficient of 2.5 x 10 We use SGD with Nesterov momentum as the optimizer with a momentum coefficient of 0.9. The learning rate is kept at 0.1 until epoch 75, then is linearly annealed 0.001 from epoch 75 epoch 135, then kept at 0.001 for the rest of the training. We use basic data preprocessing, which includes whan normalization, random cropping after padding and random horizontal flipping. We set 0.2 for DAMP, p = 0.2 for SAM and = 3.0 for ASAM. Each method is trained on single host with 8 Nvidia V100 GPUs where the data batch is evenly distributed among the GPUs each iteration (data parallelism). This means we use the number of sub-batches M 8 for DAMP.
ResNet50 / ImageNet We train each experiment for 90 epochs. We use batch size of 2048. We use a weight decay coefficient of 1 x 10-4, We use SGD with Nesterov momentum as the optimizer with momentum coefficient of 0.9. We use basic Inception-style data preprocessing. which includes random cropping, resizing to the resolution of 224 x 224, random horizontal flipping and channel-wise normalization. We increase the learning rate linearly from 10 4 to 0.8 for the first 5 epochs then decrease the learning rate from 0.8 to 8 x 10 using a cosine schedule for the remaining epochs. All experiments were run on a single host with 8 Nvidia V100 GPUs and we set M = 8 for DAMP. We use p = 0.05 for Dropout, o = 0.1 for DAMP,
0.05 for SAM, and p 1.5 for ASAM. We also use the image resolution of 224 x 224 during evaluation.
VTT-S16/ / ImageNet / Basic augmentations We follow the training setup of Beyer et al. (2022) with one difference is that we only use basic Inception-style data processing similar to the ResNee50//mageNee experiments. We use AdamW as the optimizer with 81 = 0.9, B3 0.999 and <= 10-5, We clip the gradient norm to 1.0. We use weight decay coefficient of 0.1. We use a batch size of 1024. We increase the learning rate linearly from 10-6 to 10-3 for the first 10000 iterations, then we anneal the learning rate from 10-3 3 to 0 using a cosine schedule for the remaining iterations. We use the image resolution of 224 x 224 for both training and testing. Following Beyer et al. (2022), we make 2 minor modifications to the original ViT-S16 architecture: (1) We change the position embedding layer from learnable to sincos2d; (2) We change the input of the final classification layer from the embedding of the [cls] token to global average- weeage-poollnng. All experiments were run on single host with 8 Nvidia V000 GPUs and we set M 8 for DAMP. We use p 0.10 for Dropout, o 0.25 for DAMP, o = 0.6 for SAM, and = 3.0 for ASAM.
VTT-S16 and B16 / ImageNet/ / MixUp and Ragment Most of the hyperparameters identical to the ViT-S16 / ImageNet/ / Basic augmentations setting. With ViT-S16, we use p = 0.1 for Dropout, 0.10 for DAMP, p 0.015 for SAM, and p 0.4 for ASAM. With VIT-B16, use p = 0.1 for Dropout, a = 0.15 for DAMP, = 0.025 for SAM, and = 0.6 for ASAM.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state our contributions and claims the abstract and introduction and back these claims and contributions with theoretical justifications and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work in Section 6, which outlines the shortcomings of our theoretical proofs and our experiments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: We provide full set of assumptions and complete proofs of the theoretical result in Section and Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all the details regarding our experiments in Appendix F. We also describe the new algorithm that we propose in detail in Algorithm 1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide our code on a public GitHub repository with instruction on how to reproduce the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips. ptiisscc.. wwt.cc/pppcccppeeeeeeeeeec gwwpce/gg/e///eeeeeeeeooecees
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the details regarding our experiments in Appendix and specify sufficient details to understand the results in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For Figs. 4 and 7, the error bars display 95% confidence intervals. For Table 1, report the standard deviation. For Figs. 3, 5 and 6, cannot display the error bars since these figures show heatmaps. For Tables 2 and 3, we does not report the error bars since we ran each experiment once due the high training cost and our limit in computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We state clearly in the paper that run all experiments on single machine with 8 Nvidia V100 GPUs.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://neet /ttpe://nneui....eee http://uurrps.cccpuu..ltt cc/ppblic//aieecccccccccc http//ullc//tthsseeeeeeeese sthicssGuiellnee
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and confirm that the research in this paper conforms with these guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss at the end of the paper the possible societal impact of the work in
this paper.
Guidelines:
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper does not release any new datasets or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all the papers that produced the datasets and model architec- tures used in this work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com conddatasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide our code public GitHub repository with documentation on how run the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "WwHQ3mtUzD", "reviewer": "Reviewer_XEeL", "summary": "This paper works on improving robustness by multiplying a random Gaussian variable on weights during training.", "questions": "1. What is the novelty of this work?\n\n2. How does DAMP compare to modern augmentation techniques such as those used in DeiT and subsequent works? DAMP's augmentation does not appear to be simpler than these methods. Furthermore, DAMP is evidently more unstable, particularly in large-scale training, due to the stochasticity introduced during training, a common issue observed in many Bayesian Neural Network (BNN) related works.", "limitations": "See the above sections.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The writing is easy to read and follow.", "weaknesses": "1. The novelty is quite limited. This concept has been proposed and explored for at least a decade. For instance, variational dropout already suggested this approach, using a Bernoulli distribution. Another example can be found in [1], \"Multiplicative or Additive Perturbation?\", where they used a Gaussian distribution.\n\n2. The theoretical foundation is weak. The theory section consists of trivial analysis by adding constrained corruption to inputs, which is not significantly related to this work.\n\nReference:\n\n1. Dusenberry, Michael, et al. \"Efficient and scalable Bayesian neural nets with rank-1 factors.\" International Conference on Machine Learning. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. On the novelty of this work and its connections to previous works:**\n\nThe novel contributions are:\n1. Showing the theoretical connection between perturbations in the input space and perturbations in the weight space, in particular, that one can simulate perturbations in the input space via multiplicative weight perturbations.\n2. From this connection, we propose a new method, Data Augmentation via Multiplicative Perturbations (DAMP), which multiplies each individual weight of a network with its own Gaussian random variables during training.\n3. We provide a new theoretical interpretation of Adaptive Sharpness Aware Minimization (ASAM) [3], a variant of SAM [2], that ASAM actually minimizes the loss under adversarial *multiplicative* weight perturbation as opposed to SAM which minimizes the loss under adversarial *additive* perturbations.\n4. We show that DAMP works well with modern architectures as demonstrated by our large-scale experiments on training ResNet50 and ViT-S16 on ImageNet.\n\nThe differences between DAMP and previous works like Dropout and its variants are:  \n1. Dropout multiplies *all the weights connecting to the same input node* with a *random Bernoulli variable*, which has the disadvantage that it reduces the update frequency of the weights during training (since dropped weights receive no gradient update from backpropagation). On the other hand, DAMP multiplies *each individual weight* with *its own random Gaussian variable*, which perturbs the weights while allowing them to receive non-zero gradients at each iteration.\n2. Dropout focuses on improving generalization on clean test data, while DAMP focuses on improving generalization on corrupted test data.  \n3. The work in [1] multiplies *all the weights connecting to the same input node* with a *Gaussian random variable*. Their motivation is to introduce Rank-1 BNNs, an efficient Bayesian neural network (BNN), and they attribute the improvements on corrupted data to the epistemic uncertainty of the Rank-1 BNNs. Our work demonstrates that *multiplicative weight perturbations* are actually the main reason for the improved robustness.\n\nWhile the concept of weight perturbations is not new, to the best of our knowledge, no earlier work specifically studies this technique to robustify models to input corruptions. Here we shed light on a simple algorithm that could be applied to any training setting to enhance model robustness with zero additional cost, and we believe that this new knowledge is beneficial to the machine learning community.\n\n**2. \"The theoretical foundation is weak. The theory section consists of trivial analysis by adding constrained corruption to inputs, which is not significantly related to this work.\"**\n\nWe appreciate your perspective on our theoretical analysis. However, we believe that our analysis, while simple, provides a direct link between input corruptions and multiplicative weight perturbations. Specifically, Theorem 1 proves that the training loss under input corruptions is upper bounded by the training loss under multiplicative weight perturbations plus an $L2$-regularization term. This connection serves as the foundation for our DAMP algorithm, offering a clear rationale for its design and effectiveness. We also note that many analyses, when presented in a clear and concise manner, may seem obvious in hindsight.\n\n**3. Comparing DAMP to modern augmentation techniques:**\n\nThe modern augmentation techniques directly modify the inputs in the input space, while DAMP perturbs the weights in the weight space to simulate input corruptions. As they operate on two different spaces, DAMP and the augmentation techniques can be combined together to further enhance corruption robustness. We demonstrate this idea with new experiments training the ViT-S16 and ViT-B16 on ImageNet with MixUp and RandAugment (Table D in the PDF of our global response), showing that DAMP works in tandem with these augmentations to boost robustness.\n\nDAMP is simpler than these augmentation techniques since it is domain-agnostic and can be applied to any tasks and any neural network architectures. This is actually an advantage as demonstrated through our new experiments finetuning Mistral-7B on MedMCQA dataset using DAMP and LoRA (Table A in the PDF of our global response). By applying DAMP on the low-rank weights, we improve the test accuracy on the MedMCQA from 49.05% to 50.25%.\n\n**4. \"DAMP is evidently more unstable, particularly in large-scale training, due to the stochasticity introduced during training...\"**\n\nWe respectfully disagree. DAMP multiplies each individual weight with its own Gaussian random variable with distribution $\\mathcal{N}(1, \\sigma^2)$, and thus we can control the strength of the perturbations via the variance $\\sigma^2$. Of course, if we set $\\sigma^2$ too high then the training cannot converge due to high stochasticity, while setting $\\sigma^2$ too low reduces the effectiveness of the perturbations. In this sense, our method is no different than other regularization techniques, as setting the dropout rate or the weight decay coefficient too high also prevents convergence. In our experiments, we never observe any instability when the variance is properly tuned using a validation set, and we were able to successfully train ResNet50, ViT-S16, and ViT-B16 on ImageNet, as well as finetuning Mistral-7B. Finally, we note that earlier studies such as [4] showed that introducing stochasticity in neural network training acts as regularization which improves generalization.\n\n[1] Dusenberry et al. \"Efficient and scalable Bayesian neural nets with rank-1 factors.\" ICML, 2020.\n\n[2] Foret et al. “Sharpness-Aware Minimization for Efficiently Improving Generalization”. ICLR, 2021\n\n[3] Kwon et al. “ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks”. ICML, 2022\n\n[4] Keskar et al. “On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima”, ICLR 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h4ULVzz9jS", "reviewer": "Reviewer_dM3x", "summary": "The paper presents Data Augmentation via Multiplicative Perturbations (DAMP), a novel method to enhance DNN robustness against image corruptions by optimizing with random multiplicative weight perturbations. This approach improves generalization on corrupted images without compromising accuracy on clean ones. The authors show that input perturbations can be mimicked by multiplicative perturbations in the weight space. The authors demonstrate DAMP's effectiveness across multiple datasets and architectures, and explore its connection to Adaptive Sharpness-Aware Minimization (ASAM).", "questions": "Why is DAMP trained for 200 epochs compared to other methods trained for 100 epochs in Table 2?\nHow does the model perform when tested on other types of distribution shifts beyond those included in the corruption benchmarks?\nHow do the methods perform without basic Inception-style preprocessing?", "limitations": "The results could be better explained, particularly why the accuracy on clean images and lower severity corruptions is lower than the ASAM method in larger-scale experiments for ResNet/ImageNet.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel perspective on enhancing neural network robustness by leveraging multiplicative weight perturbations. The experiments are comprehensive and well-designed, covering multiple datasets and model architectures. The results consistently show that DAMP improves robustness against a wide range of corruptions without sacrificing performance on clean images. The method achieves these improvements without incurring additional computational costs.", "weaknesses": "It would be beneficial to see how the method performs with different hyperparameter values, as the reported numbers for different metrics are close to each other. Assumption 2 is not explained very well and could benefit from a clearer, more detailed explanation. While DAMP works well for small models and datasets, it would be interesting to see the results with larger models and datasets, as it seems to show some instability in results on these larger models and datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. \"It would be beneficial to see how the method performs with different hyperparameter values, as the reported numbers for different metrics are close to each other.\"**\n  \nThank you for your suggestion. We provide Table C in the PDF file attached to our global response showing the accuracy of ViT-S16 on ImageNet under different variances of the random multiplicative weight perturbations of DAMP. Specifically, as we increase the perturbation variance, the accuracy improves up to a maximum value then degrades afterwards. We will include this table in the revised paper.\n  \n**2.  \"Assumption 2 is not explained very well and could benefit from a clearer, more detailed explanation\"**\n\nAssumption 2 states that the gradient with respect to the input $\\mathbf{x}$ of the loss function $\\ell(\\boldsymbol{\\omega}, \\mathbf{x}, y)$ is Lipschitz continuous. This simply means that there exists a constant $C > 0$ such that for all $\\mathbf{x}\\_1, \\mathbf{x}\\_2$, we have:  \n$$||\\nabla\\_{\\mathbf{x}}\\ell(\\boldsymbol{\\omega},\\mathbf{x}\\_1,y)-\\nabla\\_{\\mathbf{x}}\\ell(\\boldsymbol{\\omega},\\mathbf{x}\\_2,y)||\\_2^2\\leq C||\\mathbf{x}\\_1-\\mathbf{x}\\_2||\\_2^2$$\nWe will clarify assumption 2 as shown here in the revised paper.\n\n**3. \"While DAMP works well for small models and datasets, it would be interesting to see the results with larger models and datasets, as it seems to show some instability in results on these larger models and datasets.\"**\n\nPerhaps there is a misunderstanding here. We have shown that DAMP works well on both ResNet50 and ViT-S16 on ImageNet in our paper, and we experienced no instability when using DAMP to train these models. In our new experiments (see Table D in the PDF file of our global response), we use DAMP to train a ViT-B16, a larger version of ViT-S16, showing that DAMP also leads to improved performance in this case. Thus we believe that DAMP is perfectly capable of training large models on large datasets.\n\n**4. \"Why is DAMP trained for 200 epochs compared to other methods trained for 100 epochs in Table 2?\"**\n\nIn Table 2, we compare models trained with DAMP and AdamW for 200 epochs to models trained with SAM and ASAM for 100 epochs. This is because SAM and ASAM require two forward-backward passes per training iteration, while DAMP and AdamW require only one forward-backward pass per iteration. Therefore, the time it takes to run 200 epochs of DAMP is roughly the same as 100 epochs of SAM/ASAM. We have already emphasized this in the paper but will add a further comment. We also have included the runtime of these experiments in Table B and Table D of the PDF attached to our global response.\n\n**5. \"How does the model perform when tested on other types of distribution shifts beyond those included in the corruption benchmarks?\"**\n\nWe provide new results in Table B and Table D in the PDF of our global response where we evaluate models trained with DAMP and baseline methods on ImageNet-D [1], ImageNet-A [2], ImageNet-Sketch [3], ImageNet-Cartoon [4], ImageNet-Drawing [4], and adversarial corruptions generated by FGSM [5], demonstrating that models trained with DAMP are more robust to various types of input perturbations than the baseline training methods.\n\n**6. \"How do the methods perform without basic Inception-style preprocessing?\"**\n\nWe provide new results in Table D in the PDF of our global response where we train ViT-S16 and ViT-B16 with MixUp and RandAugment, showing that DAMP is able to work in tandem with these modern data augmentations to further enhance model performance.\n\n**7. \"The results could be better explained, particularly why the accuracy on clean images and lower severity corruptions is lower than the ASAM method in larger-scale experiments for ResNet/ImageNet.\"**\n\nAs we explain in Section 3 of the paper, ASAM actually minimizes the training loss under adversarial multiplicative weight perturbations (MWPs), as shown by the inner maximization over the MWPs $\\boldsymbol{\\xi}$ in its objective function (Eq. 21 in the paper):\n$$\\mathcal{L}\\_{\\mathrm{ASAM}}(\\boldsymbol{\\omega};\\mathcal{D})=\\max\\_{||\\boldsymbol{\\xi}||\\_2 \\leq \\rho}\\mathcal{L}(\\boldsymbol{\\omega}\\circ(1+\\boldsymbol{\\xi});\\mathcal{D})+\\frac{\\lambda}{2}||\\boldsymbol{\\omega}||\\_2^2$$\nand thus it is also able to produce robust models. This is why Table 1 in the paper (ResNet50 / ImageNet) shows that ASAM is able to outperform DAMP on clean images and on corruption severity levels 1, 2 and 3 of ImageNet-C. However, Table 1 also shows that DAMP outperforms ASAM on severity levels  4 and 5 of ImageNet-C as well as on all 5 severity levels of ImageNet-$\\bar{\\mathrm{C}}$. Furthermore, as we have stated above, the training time of DAMP is half that of ASAM, and since we use 90 epochs for all experiments in Table 1, this means that DAMP is able to outperform ASAM on the majority of test sets while being more efficient. We will add a further pointer to these for clarity.\n\n[1] Zhang et al. \"ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object\". CVPR 2024.\n  \n[2] Hendrycks et al. \"Natural Adversarial Examples\". CVPR 2021.\n\n[3] Wang et al. \"Learning Robust Global Representations by Penalizing Local Predictive Power\". NeurIPS 2019.\n\n[4] Salvador et al. \"ImageNet-Cartoon and ImageNet-Drawing: two domain shift datasets for ImageNet\". ICML 2022 Shift Happens Workshop.\n\n[5] Goodfellow et al. \"Explaining and Harnessing Adversarial Examples\". arXiv preprint arXiv:1412.6572, 2014.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "451aNQQJmz", "reviewer": "Reviewer_Vu1M", "summary": "This submission tackles the problem of generalization of image classifiers to corrupted images. The paper shows a link between data augmentation as well as previous methods such as Adaptive Sharpness-aware Minimization (ASAM) and multiplicative weight perturbations. Effectively, ASAM works as an adversarial weight multiplication which requires a second gradient pass per batch. The proposed DAMP method relaxes this constraint by using random weight perturbations. DAMP is benchmarked on CIFAR-10/100, TinyImageNet, and ImageNet under common corruptions (and follow-up works) with ResNet and ViT where it often matches or even exceeds the original SAM at lower cost.", "questions": "See above.", "limitations": "The authors are very upfront about Limitations and I have no other suggestions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good writing.\n- A theoretical motivation for DAMP is provided (arguably with under-simplified relaxation).\n- A connection between ASAM and multiplicative weight perturbations is shown.\n- Good evaluation: DAMP is benchmarked on CIFAR-10/100, TinyImageNet, and ImageNet under common corruptions (and follow-up works) with ResNet and ViT. It is also compared to SAM and ASAM. All experiments (except ViT) use multiple runs!\n- DAMP outperforms SAM in most cases at a lower cost; it allows to training of ViTs without excessive augmentation techniques.\n- DAMP is theoretically domain-agnostic and could be applied beyond image classification or vision.", "weaknesses": "My only concern about this work is the contextualization of this work. It seems like DAMP is mostly an approximation of ASAM which allows faster training (and that is great) but at the same time, it is not as effective as ASAM. So effectively, this method introduces a trade-off between performance (accuracy) and training time. This raises the following detailed concerns:\n\n- If multiplicative weight perturbations are connected to data augmentation then how does DAMP rank against them? Augmentations are mostly cheap so they could achieve even better performance at the same overhead cost. One indication of this is in the ViT experiments on ImageNet: DAMP achieves a 23.7% clean error in 200 epochs without augmentations. This may sound impressive at first, however, the baseline that the authors rely on (Beyer, 2022) even exceeds this (23.5% error) in just 90 epochs. The only difference is that (Beyer, 2022) utilizes some simple augmentations. That is only clean accuracy but usually, it is indicative of performance under distribution shift [1]. Ideally, the authors would demonstrate that on a train time vs. mCE or clean acc curve, DAMP exceeds the current Pareto frontier (by benchmarking a few modern augmentation techniques in addition to SAM/ASAM). \n- Additionally, it would be great to understand if DAMP is compatible with augmentations. I don't want to use the clearly stated Limitations against the authors but this is a very straightforward question that I had during reading. \n\n\nOther than that the theory in Eq.1ff assumes bias-free MLP architectures without non-linearities which obviously does not scale to modern architectures. (The authors mention this in their limitations and this is not something that needs to be addressed, but it is a weakness nonetheless)\n\nNitpicks:\n- L118: I agree that random perturbations introduce almost no overhead, but you shouldn't state \"we found that ... had similar training times\" in a scientific paper without backing it up with hard numbers.\n- It would be great to add averages over all severities in Tab. 1 and 2.\n\nSuggestion: \n- A simple way to kill all my concerns is to show performance on non-vision tasks (since DAMP is in theory data-agnostic). In that case, it would exceed the value of domain-specific augmentations. This would greatly enhance this paper but of course, I do not expect this from a rebuttal.\n\n[1] Taori et al., \"Measuring Robustness to Natural Distribution Shifts in Image Classification\", NeurIPS 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. \"...It seems like DAMP is mostly an approximation of ASAM which allows faster training (and that is great) but at the same time, it is not as effective as ASAM. So effectively, this method introduces a trade-off between performance (accuracy) and training time...\"**\n\nThere is a misunderstanding here. DAMP is actually **not an approximation of ASAM** and can even outperform ASAM. The objective function of DAMP (Eq. 16 in our paper) is:\n$$ \\mathcal{L}\\_{\\mathrm{DAMP}}(\\boldsymbol{\\omega}; \\mathcal{D}) = \\mathbb{E}\\_{\\boldsymbol{\\xi}\\sim\\mathcal{N}(\\boldsymbol{1},\\sigma^2\\mathbf{I})}\\left[\\mathcal{L}(\\boldsymbol{\\omega}\\circ\\boldsymbol{\\xi}; \\mathcal{D})\\right] + \\frac{\\lambda}{2}||\\boldsymbol{\\omega}||\\_2^2$$\nwhile the objective function of ASAM (Eq. 21 in our paper) can be written as:\n$$\\mathcal{L}\\_{\\mathrm{ASAM}}(\\boldsymbol{\\omega};\\mathcal{D})=\\max\\_{||\\boldsymbol{\\xi}||\\_2 \\leq \\rho}\\mathcal{L}(\\boldsymbol{\\omega}\\circ(1+\\boldsymbol{\\xi});\\mathcal{D})+\\frac{\\lambda}{2}||\\boldsymbol{\\omega}||\\_2^2$$\nwhere $\\boldsymbol{\\omega}$ is the model's parameters, $\\mathcal{D}$ is the training data, $\\mathcal{L}$ is the loss function such as cross-entropy, and $\\circ$ denotes the Hadamard product. Thus DAMP minimizes the **expected loss** under multiplicative weight perturbations (MWPs), while ASAM minimizes the loss under the **worse-case MWP**. This is a crucial distinction, since minimizing the expectation allows us to devise an efficient version of DAMP (Algorithm 1 in the paper), while ASAM needs two forward-backward passes for each iteration (the first one to approximate the worse-case MWP, the second one is to minimize the loss under such MWP). Thus given the same number of training iterations, ASAM takes **twice as long** to train as DAMP. In Table D of the PDF attached to our global response, training a ViT-B16 (86M params) with DAMP leads to better accuracy than training ViT-S16 (22M params) with ASAM, yet both settings take roughly the same amount of time. Thus given the same training budget, it is better to train a large model with DAMP than to train a smaller model with ASAM.\n\n**2. \"If multiplicative weight perturbations are connected to data augmentation then how does DAMP rank against them?...\"**\n\n**\"Additionally, it would be great to understand if DAMP is compatible with augmentations...\"**\n\nModern data augmentations (DAs) (such as MixUp, RandAugment) are cheap and contain informative prior knowledge, and thus they could greatly improve performance of a model. However, they are specific to computer vision and not applicable to other domains like natural language processing. On the other hand, multiplicative weight perturbations (MWPs) are less informative than DAs but they are domain-agnostic and thus can be applied to a wide range of tasks. For instance, DAMP improves the test accuracy of Mistral-7B finetuned by LoRA on the MedMCQA dataset from 49.05% to 50.25%, as shown in Table A in the PDF attached to the global response.  Furthermore, since MWPs operate on the weight space while DAs operate on the input space, they can be combined together to further enhance performance, as shown in Table D of the global response combining DAMP with MixUp and RandAugment to train ViT-S16 and ViT-B16.\n\n**3.  \"the theory in Eq.1ff assumes bias-free MLP architectures without non-linearities which obviously does not scale to modern architectures.\"**\n\nThere is a misunderstanding here. In our theoretical analysis, we use a bias-free MLP architectures with **non-linear activations**, as shown in Eq. 1-3 in our paper. It would be pointless for us to analyze an MLP without non-linear activations since such model is equivalent to a simple one-layer linear model. We note that other papers (e.g. [1, 2]) also assume a bias-free MLP for their analyses.  Finally, we want to note that Theorem 1 in our paper motivates the design of the DAMP algorithm which, as we demonstrate through our large-scale ImageNet experiments, does scale well to modern architectures.\n\n**4. \"I agree that random perturbations introduce almost no overhead, but you shouldn't state \"we found that ... had similar training times\" in a scientific paper without backing it up with hard numbers.\"**\n\n**\"It would be great to add averages over all severities in Tab. 1 and 2.\"**\n\nThank you for your suggestions. We now added the runtimes and the results averaged over severities for each setting in our global response (Table B and D), and will include them in the revised paper.\n\n**5. \"A simple way to kill all my concerns is to show performance on non-vision tasks (since DAMP is in theory data-agnostic). In that case, it would exceed the value of domain-specific augmentations. This would greatly enhance this paper but of course, I do not expect this from a rebuttal.\"**\n\nThank you for your suggestions. We now added results of using DAMP with LoRA to finetune a Mistral-7B on the MedMCQA dataset in our global response (Table A), demonstrating that it improves the test accuracy from 49.05% to 50.25%.\n\n[1] Dusenberry et al. \"Efficient and scalable Bayesian neural nets with rank-1 factors.\" ICML, 2020.\n\n[2] Andriushchenko et al. \"Sharpness-Aware Minimization Leads to Low-Rank Features\". NeurIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ih9JtBjXTs", "reviewer": "Reviewer_dwU7", "summary": "To achieve robustness against corruptions, most of existing works relying on incorporating some corruptions during the training. In contrast, this paper takes a different perspective and propose a weight-based approach called DAMP to achieve model robustness against corruptions without compromising accuracy on clean examples. \n\nThis work also studies Sharpness-Aware Minimisation (SAM) methods and point out the relationship between their proposed DAMP and SAM-based methods.", "questions": "What is the effectiveness of DAMP on deeper/modern neural networks, such as Resnet-152 EfficientNet, ConvNext, MaxViT? \n\nWhat is the effectiveness of DAMP on other corruption (such as Stylised ImageNet, ImageNet-D), natural corruption (such as ImageNet-A, ImageNet-Sketch), and adversarial corruption?\n\nWhat is the advantage of DAMP over SAM/ASAM?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper takes a new perspective compared to existing methods to achieve robustness against corruptions.\n\nThe proposed method is simple with nearly similar complexity to SGD and achieve competitive results.\n\nTheoretical results is provided to justify the hypothesis.\n\nThe presentation is clear and easy to follow.", "weaknesses": "There could be a related work [1]  focusing on (adversarial) corruptions, where it also approaches the model robustness problem from model weights perspective. I suggest the authors to take a look at [1] and compare with their work.\n\nThe experimental results relying on only algorithmically corruption (ImageNet-C, CIFAR-C) could be limited. I suggest the author to conduct the results on other corruption (such as Stylised ImageNet, ImageNet-D), natural corruption (such as ImageNet-A, ImageNet-Sketch), and adversarial corruption. I believe this assessment will further improve the experimental results.\n\nThe results for SAM/ASAM in Tables 1 and 2 are relatively comparable to those for DAMP when considering the effect of accuracy on clean images. As we know, improving accuracy on clean images generally leads to better accuracy on corrupted images. While I understand that this work takes a different approach from SAM/ASAM, I am curious to know what advantages DAMP offers over SAM/ASAM.\n\nAs the theory based on a simple feedforward neural network, I wonder if the DAMP can improve the robustness on deeper/modern neural networks, such as Resnet-152, EfficientNet, ConvNext, MaxViT? \n\n[1] Ha, Hyeonjeong, Minseon Kim, and Sung Ju Hwang. \"Generalizable lightweight proxy for robust NAS against diverse perturbations.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. \"There could be a related work [1] focusing on (adversarial) corruptions, where it also approaches the model robustness problem from model weights perspective....\"**\n\nThank you for the suggestion. This is indeed an interesting related paper which proposes a score called CRoZe that can quickly evaluate the robustness to input perturbations of an architecture using only one batch of training data for zero-shot neural architecture search (NAS). We think that the multiplicative weight perturbations (MWPs) of DAMP can also be used inside the calculation of the CRoZe score to evaluate the robustness of an architecture, which could be a good future research direction. We will include this mention of [1] in the Related Works section of our revised paper.\n\n**2. \"What is the effectiveness of DAMP on deeper/modern neural networks, such as Resnet-152 EfficientNet, ConvNext, MaxViT?\"**\n\nIn our paper, we have demonstrated the effectiveness of DAMP on ResNet50 and ViT-S16 on ImageNet. We now further provide new results with ViT-B16 in our global response (Table D in the PDF). These results together demonstrate that DAMP is capable of improving the robustness of deep and modern neural networks. Furthermore, as modern networks get larger and have bigger learning capacity, we believe that they could benefit from the additional implicit data augmentations/perturbations induced by the random multiplicative weight perturbations of DAMP, even more so than their smaller counterparts since they have the capacity to learn to output the correct predictions even when the inputs are corrupted by these implicit perturbations.\n\n**3. \"What is the effectiveness of DAMP on other corruption (such as Stylised ImageNet, ImageNet-D), natural corruption (such as ImageNet-A, ImageNet-Sketch), and adversarial corruption?\"**\n\nThank you for your suggestions. We provide new results in our global response (Table B and D in the PDF) where we evaluate models trained with DAMP and baseline methods on ImageNet-D [2], ImageNet-A [3], ImageNet-Sketch [4], ImageNet-Cartoon [5], ImageNet-Drawing [5], and adversarial corruptions generated by FGSM [6], demonstrating that models trained with DAMP are more robust to various types of input perturbations than the baseline training methods.\n\n**4. \"What is the advantage of DAMP over SAM/ASAM?\"**\n\nAs stated in our paper, the main advantage of DAMP over SAM/ASAM is that it requires only one forward-backward pass per training iteration, while SAM/ASAM requires two consecutive forward-backward passes per iteration. Therefore, given the same number of iterations, SAM/ASAM takes twice as long to finish training than DAMP. Furthermore, our experimental results show that on corrupted inputs, DAMP outperforms SAM in the majority of cases and is competitive with ASAM. Finally, our new results in the global response (Table D in the attached PDF) indicate that training a ViT-B16 (86M params) with DAMP leads to better accuracy than training ViT-S16 (22M params) with SAM/ASAM, yet both settings take roughly the same amount of time. Thus given the same training budget, it is better to train a large model with DAMP than to train a smaller model with SAM/ASAM.\n\n[1] Ha et al. \"Generalizable lightweight proxy for robust NAS against diverse perturbations.\" NeurIPS 2024.\n\n[2] Zhang et al. \"ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object\". CVPR 2024.\n\n[3] Hendrycks et al. \"Natural Adversarial Examples\". CVPR 2021.\n\n[4] Wang et al. \"Learning Robust Global Representations by Penalizing Local Predictive Power\". NeurIPS 2019.\n\n[5] Salvador et al. \"ImageNet-Cartoon and ImageNet-Drawing: two domain shift datasets for ImageNet\". ICML 2022 Shift Happens Workshop.\n\n[6] Goodfellow et al. \"Explaining and Harnessing Adversarial Examples\". arXiv preprint arXiv:1412.6572, 2014.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank the reviewers for their time and for providing us with thoughtful comments which help us improve our work. In this global response, we first provide a brief summary of our paper. We then present additional experiment results, which are included in the attached PDF file.
## Paper summary:

The aim of our paper is to introduce a novel, efficient training method that enhances neural network (NN) robustness against input perturbations without incurring additional costs. We begin by presenting theoretical analyses demonstrating that input space corruptions can be simulated using multiplicative weight perturbations (MWPs) in the weight space. We argue that training under these simulated corruptions would allow models to become more robust. From these insights, we propose Data Augmentation via Multiplicative Perturbations (DAMP), which optimizes NNs under random Gaussian MWPs. At each training iteration, DAMP multiplies **each individual weight with its own random variable $\mathcal{N}(1, \sigma^2)$**. Additionally, we offer a new perspective on ASAM [1], revealing that it also produces robust models by optimizing under adversarial MWPs. Notably, while DAMP maintains the same training time as standard optimizers like Adam, ASAM *doubles* this duration. Our experimental results, conducted on both small datasets (CIFAR, TinyImageNet) and large-scale scenarios (ResNet50 and ViT-S16 on ImageNet), demonstrate that DAMP outperforms baselines in robustifying neural networks against a wide range of corruptions.
## New results:

The additional results in the PDF file demonstrate the following properties of DAMP:
* Domain-agnostic:  DAMP can be used in non-computer-vision domains as it can be used to finetune a large language model.
* Compatible with modern augmentations: DAMP can be combined with modern data augmentations to further enhance robustness.
* Capable of training large models: DAMP has no problem training ViT-B16 on ImageNet and achieves better results than AdamW.
* Same training costs as standard optimizers: We back up the claim that DAMP and standard optimizers like AdamW have the same training time by providing concrete numbers.
* Capable of improving robustness to various corruptions: We evaluate DAMP on additional types of corruptions including adversarial corruptions to verify that it is indeed effective on a wide range of corruptions.

We believe that these results will resolve all of the concerns that you had. Below, we outline the contents of the PDF file:
- Table A includes the results of using DAMP with LoRA [2] to finetune Mistral-7B [3] on the MedMCQA dataset [4]. These results show that DAMP consistently improves the test accuracy of the finetuned models, which answer concern of Reviewer **Vu1M** about the benefit of DAMP on non-computer-vision tasks, as well as concerns of Reviewer **XEeL** and **Vu1M** about the advantages of DAMP over domain-specific augmentations like MixUp [5] and RandAugment [6].
- Table B extends the results of ViT-S16 / ImageNet with basic Inception-style augmentations in our paper with additional evaluation results on ImageNet-D [7], ImageNet-A [8], ImageNet-Sketch [9], ImageNet-Cartoon [10], ImageNet-Drawing [10], and adversarial corruptions generated by FGSM [11]. These results show that DAMP produces the most robust model on average, which address the concerns of Reviewer **dM3x** and **dwU7** about robustness of DAMP to other types of input perturbations.
- Table B also includes the results averaged over all severity levels of ImageNet-C and ImageNet-$\bar{\mathrm{C}}$ and the runtime of each experiment as requested by Reviewer **Vu1M**. These runtimes show that ASAM and SAM indeed take roughly twice as long to train than DAMP and AdamW, and that DAMP and AdamW have the same runtime.
- Table C shows the behavior of DAMP in the ViT-S16 / ImageNet experiments with basic Inception-style augmentations under different standard deviations $\sigma$ of the Gaussian random multiplicative weight perturbations (MWPs) as requested by Reviewer **dM3x**. These results show that as $\sigma$ increases, the accuracy of DAMP improves up to its maximum value and then degrades afterwards.
- Table D presents results of DAMP and baselines when training ViT-S16 and ViT-B16 on ImageNet with MixUp [5] and RandAugment [6], demonstrating that: (i) DAMP can be combined with modern data augmentations (DAs) to further enhance robustness; (ii) DAMP is capable of training large models like ViT-B16; (iii) given the same amount of training time, it is better to train a large model (ViT-B16) using DAMP than to train a smaller model (ViT-S16) using SAM/ASAM. This addresses the concerns of Reviewer **dwU7** and **Vu1M** regarding the advantages of DAMP over SAM/ASAM, and concern of Reviewer **Vu1M** about the compatibility of DAMP with modern DA techniques.

[1] Kwon et al. "ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks". ICML 2022.

[2] Hu et al. "LoRA: Low-Rank Adaptation of Large Language Models". ICLR 2022.

[3] Jiang et al. "Mistral-7B". arXiv preprint arXiv:2310.06825, 2023.

[4] Pal et al. "MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering". CHIL 2022.

[5] Zhang et al. "mixup: Beyond Empirical Risk Minimization". ICLR 2018.

[6] Cubuk et al. "RandAugment: Practical automated data augmentation with a reduced search space". NeurIPS 2020.

[7] Zhang et al. "ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object". CVPR 2024.

[8] Hendrycks et al. "Natural Adversarial Examples". CVPR 2021.

[9] Wang et al. "Learning Robust Global Representations by Penalizing Local Predictive Power". NeurIPS 2019.

[10] Salvador et al. "ImageNet-Cartoon and ImageNet-Drawing: two domain shift datasets for ImageNet". ICML 2022 Shift Happens Workshop.

[11] Goodfellow et al. "Explaining and Harnessing Adversarial Examples". arXiv preprint arXiv:1412.6572, 2014.

Meta-review: The paper tackles the problem of "robustness" of deep neural networks (DNN) in the presence of "corrupted" images. The main intuition/contribution is a method that improves the robustness of DNN to corrupted images by manipulating the weights of the DNN during its training phase---thereby suggesting that there is some sort of correlation between perturbations in the "input space" (i.e., in the images analysed by the DNN) and those in the "weight space" (i.e., in the weights of the DNN). Experiments on various benchmark datasets, as well as various theoretical proofs, support the effectiveness of the proposed method. Moreover, **the rebuttal phase also allowed to shed light on the fact that the proposed method is not specific to DNN analysing images** (as demonstrated in an experiment on the MedMCQA dataset), thereby further showcasing the generality and "improvement" of the proposed solution over existing alternatives.


Four reviews were submitted for this paper. Two reviewers (with confidence 3 and 4) originally recommended a "5-Borderline Accept" and then increased to a "6-Weak Accept" after interacting with the authors. Another reviewer (with a confidence of 2) proposed a "6-Weak Accept" and maintained their score after the rebuttal phase. A fourth reviewer recommended a "4-Borderline Reject" (with a high-confidence of 4), rating the paper with 1-2-1 for Soundness, Presentation and Contribution, but the exchanges with the authors led such a reviewer to improve their Soundness and Contribution ratings (to a 3 and to a 2) and also to recommend a 5 "Borderline Accept". Hence, it can be said that the author-reviewer interactive phase played a pivotal role in clarifying lingering doubts as well as in increasing the quality of this submission. This is a positive sign of a constructive peer-review.

No factual errors were found by the reviewers. The code is also made public. Moreover, all reviewers agreed in that the paper provides a meaningful contribution to the state of the art. Some praised the extensive experiments, or the theoretical foundations, or even the presentation of the paper---all of which being aspects that I also believe to be significant strengths of this submission. 

Due to the above, I recommend acceptance of this work---with a "spotlight". I could not identify glaring weaknesses, and the combination of sound experiments, theoretical analyses, good presentation, nice intuition, and generalizability make me believe that this paper can spearhead many discussions in the field.
Decision: Accept (spotlight)