Improving robustness to corruptions with multiplicative weight perturbations

Authors: trung  trinh
markus  heinonen
luigi  acerbi
samuel  kaski
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 10:09:52

Abstract: Deep neural networks (DNNs) excel on clean images but struggle with corrupted ones. Incorporating specific corruptions into the data augmentation pipeline can improve robustness to those corruptions but may harm performance on clean images and other types of distortion. In this paper, we introduce an alternative approach that improves the robustness of DNNs to a wide range of corruptions without compromising accuracy on clean images. We first demonstrate that input perturbations can be mimicked by multiplicative perturbations in the weight space. Leveraging this, we propose Data Augmentation via Multiplicative Perturbation (DAMP), a training method that optimizes DNNs under random multiplicative weight perturbations. We also examine the recently proposed Adaptive Sharpness-Aware Minimization (ASAM) and show that it optimizes DNNs under adversarial multiplicative weight perturbations. Experiments on image classification datasets (CIFAR-10/100, TinyImageNet and ImageNet) and neural network architectures (ResNet50, ViT-S/16, ViT-B/16) show that DAMP enhances model generalization performance in the presence of corruptions across different settings. Notably, DAMP is able to train a ViT-S/16 on ImageNet from scratch, reaching the top-1 error of 23.7% which is comparable to ResNet50 without extensive data augmentations.

Main paper: 
 introduction: 
 Deep neural networks (DNNs) demonstrate impressive accuracy in computer vision tasks when evaluated on carefully curated and clean datasets. However, their performance significantly declines when test images are affected by natural distortions such as camera noise, changes in lighting and weather conditions, or image compression algorithms (Hendrycks and Dietterich, 2019) . This drop in performance is problematic in production settings, where models inevitably encounter such perturbed inputs. Therefore, it is crucial to develop methods that produce reliable DNNs robust to common image corruptions, particularly for deployment in safety-critical systems (Amodei et al., 2016) .To enhance robustness against a specific corruption, one could simply include it in the data augmentation pipeline during training. However, this approach can diminish performance on clean images and reduce robustness to other types of corruptions (Geirhos et al., 2018) . More advanced data augmentation techniques (Cubuk et al., 2018; Hendrycks et al., 2019; Lopes et al., 2019) have been developed which effectively enhance corruption robustness without compromising accuracy on clean images. Nonetheless, a recent study by Mintun et al. (2021) has identified a new set of image corruptions to which models trained with these techniques remain vulnerable. Besides data (a) z = w ⊤ (x + ϵ) augmentation, ensemble methods such as Deep ensembles and Bayesian neural networks have also been shown to improve generalization in the presence of corruptions (Lakshminarayanan et al., 2017; Ovadia et al., 2019; Dusenberry et al., 2020; Trinh et al., 2022) . However, the training and inference costs of these methods increase linearly with the number of ensemble members, rendering them less suitable for very large DNNs.(b) z = (w • (1 + ϵ/x)) ⊤ x (c) z = (w • ξ) ⊤ x, ξ ∼ p(ξ)Contributions In this work, we show that simply perturbing weights with multiplicative random variables during training can significantly improve robustness to a wide range of corruptions. Our contributions are as follows:• We show in Section 2 and Fig. 1 that the effects of input corruptions can be simulated during training via multiplicative weight perturbations.• From this insight, we propose a new training algorithm called Data Augmentation via Multiplicative Perturbations (DAMP) which perturbs weights using multiplicative Gaussian random variables during training while having the same training cost as standard SGD.• In Section 3, we show a connection between adversarial multiplicative weight perturbations and Adaptive Sharpness-Aware Minimization (ASAM) (Kwon et al., 2021) .• Through a rigorous empirical study in Section 4, we demonstrate that DAMP consistently improves generalization ability of DNNs under corruptions across different image classification datasets and model architectures.• Notably, we demonstrate that DAMP can train a Vision Transformer (ViT) (Dosovitskiy et al., 2021) from scratch on ImageNet, achieving similar accuracy to a ResNet50 (He et al., 2016a) in 200 epochs with only basic Inception-style preprocessing (Szegedy et al., 2016) . This is significant as ViT typically requires advanced training methods or sophisticated data augmentation to match ResNet50's performance when being trained on ImageNet from scratch (Chen et al., 2022; Beyer et al., 2022) . We also show that DAMP can be combined with modern augmentation techniques such as MixUp (Zhang et al., 2018) and RandAugment (Cubuk et al., 2020) to further improve robustness of neural networks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We clearly state our contributions and claims in the abstract and introduction and back these claims and contributions with theoretical justifications and experimental results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
data augmentation via multiplicative perturbations: 
 In this section, we demonstrate the equivalence between input corruptions and multiplicative weight perturbations (MWPs), as shown in Fig. 1 , motivating the use of MWPs for data augmentation.Figure 2 : Depiction of how a corruption g affects the output of a DNN. Here x g = g(x). The corruption g creates a shift δ g x (0) = x g -x in the input x, which propagates into shifts δ g x (h) in the output of each layer. This will eventually cause a shift in the loss δ g ℓ. This figure explains why the model performance tends to degrade under corruption.Given a training data set S = {(x k , y k )} N k=1 ⊆ X × Y drawn i.i.d. from the data distribution D, we seek to learn a model that generalizes well on both clean and corrupted inputs. We denote G as a set of functions whose each member g : X → X represents an input corruption. That is, for each x ∈ X , g(x) is a corrupted version of x. 2 We define g(S) := {(g(x k ), y k )} N k=1 as the training set corrupted by g. We consider a DNN f : X → Y parameterized by ω ∈ W. Given a per-sample loss ℓ : W × X × Y → R + , the training loss is defined as the average loss over the samples L(ω; S) := 1 N N k=1 ℓ(ω, x k , y k ). Our goal is to find ω which minimizes:EQUATIONwithout knowing exactly the types of corruption contained in G. This problem is crucial for the reliable deployment of DNNs, especially in safety-critical systems, since it is difficult to anticipate all potential types of corruption the model might encounter in production.To address the problem above, we make two key assumptions about the corruptions in G: Assumption 1. For each corruption function g : X → X in G, there exists a constant M > 0 such that ||g(x) -x|| 2 ≤ M for all x ∈ X . Assumption 2. A model's robustness to corruptions in G can be indirectly enhanced by improving its resilience to a more easily simulated set of input perturbations.Assumption 1 implies that the corrupted versions of an input x must be constrained within a bounded neighborhood of x in the input space. Assumption 2 is corroborated by Rusak et al. (2020) , who demonstrated that distorting training images with Gaussian noise improves a DNN's performance against various types of corruption. We further validate this observation for corruptions beyond Gaussian noise in Section 4.1. However, Section 4.1 also reveals that using corruptions as data augmentation degrades model performance on clean images. Consequently, we need to identify a method that efficiently simulates diverse input corruptions during training, thereby robustifying a DNN against a wide range of corruptions without compromising its performance on clean inputs.One such method involves injecting random multiplicative weight perturbations (MWPs) into the forward pass of DNNs during training. The intuition behind this approach is illustrated in Fig. 1 .Essentially, for a pre-activated neuron z = w ⊤ x in a DNN, given a corruption causing a covariate shift ϵ in the input x, Figs. 1a and 1b show that one can always find an equivalent MWP ξ(ϵ, x):z = w ⊤ (x + ϵ) = (w • ξ(ϵ, x)) ⊤ x, ξ(ϵ, x) = 1 + ϵ/x (2)where • denotes the Hadamard product. This observation suggests that input corruptions can be simulated during training by injecting random MWPs into the forward pass, as depicted in Fig. 1c , resulting in a model more robust to corruption. We thus move the problem of simulating corruptions from the input space to the weight space.Here we provide theoretical arguments supporting the usage of MWPs to robustify DNNs. To this end, we study how corruption affects training loss. We consider a feedforward neural network f (x; ω)of depth H parameterized by ω = {W (h) } H h=1 ∈ W, which we define recursively as follows:f (0) (x) := x, z (h) (x) := W (h) f (h-1) (x), f (h) (x) := σ (h) (z (h) (x)), ∀h = 1, . . . , H (3)where f (x; ω) := f (H) (x) and σ (h) is the non-linear activation of layer h. For brevity, we use x (h) and x (h) g as shorthand notations for f (h) (x) and f (h) (g(x)) respectively. Given a corruption function g, Fig. 2 shows that g creates a covariate shift δ g x (0) := x (0) g -x (0) in the input x leading to shifts h) in the output of each layer. This will eventually cause a shift in the per-sample loss δ g ℓ(ω, x, y) := ℓ(ω, x g , y) -ℓ(ω, x, y). The following lemma characterizes the connection between δ g ℓ(ω, x, y) and δ g x (h) : Lemma 1. For all h = 1, . . . , H and for all x ∈ X , there exists a scalar C (h) g (x) > 0 such that:δ g x (h) := x (h) g -x (δ g ℓ(ω, x, y) ≤ ∇ z (h+1) ℓ(ω, x, y) ⊗ δ g x (h) , W (h+1) F + C (h) g (x) 2 ||W (h) || 2 F (4)Here ⊗ denotes the outer product of two vectors, ⟨•, •⟩ F denotes the Frobenius inner product of two matrices of the same dimension, || • || F is the Frobenius norm, and ∇ z (h) ℓ(ω, x, y) is the Jacobian of the per-sample loss with respect to the pre-activation output z (h) (x) at layer h. To prove Lemma 1, we use Assumption 1 and the following assumption about the loss function: Assumption 3. The input gradient of the per-sample loss ∇ x ℓ(ω, x, y) is Lipschitz continuous.Assumption 3 allows us to define a quadratic bound of the loss function using a second-order Taylor expansion. The proof of Lemma 1 is provided in Appendix A. Using Lemma 1, we prove Theorem 1, which bounds the training loss in the presence of corruptions using the training loss under multiplicative perturbations in the weight space: Theorem 1. For a function g : X → X satisfying Assumption 1 and a loss function L satisfying Assumption 3, there exists ξ g ∈ W and C g > 0 such that:L(ω; g(S)) ≤ L(ω • ξ g ; S) + C g 2 ||ω|| 2 F (5)We provide the proof of Theorem 1 in Appendix B. This theorem establishes an upper bound for the target loss in Eq. (1):L(ω; G(S)) ≤ E g∼G L(ω • ξ g ; S) + C g 2 ||ω|| 2 F (6)This bound implies that training a DNN using the following loss function:L Ξ (ω; S) := E ξ∼Ξ [L(ω • ξ; S)] + λ 2 ||ω|| 2 F (7)where the expected loss is taken with respect to a distribution Ξ of random MWPs ξ, will minimize the upper bound of the loss L(ω; Ĝ(S)) of a hypothetical set of corruptions Ĝ simulated by ξ ∼ Ξ. This approach results in a model robust to these simulated corruptions, which, according to Assumption 2, could indirectly improve robustness to corruptions in G.We note that the second term in Eq. ( 7) is the L 2 -regularization commonly used in optimizing DNNs.Based on this proxy loss, we propose Algorithm 1 which minimizes the objective function in Eq. ( 7) when Ξ is an isotropic Gaussian distribution N (1, σ 2 I). We call this algorithm Data Augmentation via Multiplicative Perturbations (DAMP), as it uses random MWPs during training to simulate input corruptions, which can be viewed as data augmentations.Remark The standard method to calculate the expected loss in Eq. ( 7), which lacks a closed-form solution, is the Monte Carlo (MC) approximation. However, the training cost of this approach scales linearly with the number of MC samples. To match the training cost of standard SGD, Algorithm 1 divides each data batch into M equal-sized sub-batches (Line 6) and calculates the loss on each sub-batch with different multiplicative noises from the noise distribution Ξ (Lines 7-9). The final gradient is obtained by averaging the sub-batch gradients (Line 11). Algorithm 1 is thus suitable for data parallelism in multi-GPU training, where the data batch is evenly distributed across M > 1 GPUs. Compared to SGD, Algorithm 1 requires only two additional operations: generating Gaussian samples and point-wise multiplication, both of which have negligible computational costs. In our experiments, we found that both SGD and DAMP had similar training times.Algorithm 1 DAMP: Data Augmentation via Multiplicative Perturbations 1: Input: training data S = {(x k , y k )} N k=1 , a neural network f (•; ω) parameterized by ω ∈ R P , number of iterations T , step sizes {η t } T t=1 , number of sub-batch M , batch size B divisible by M , a noise distribution Ξ = N (1, σ 2 I P ), weight decay coefficient λ, a loss function L : R P → R + . 2: Output: Optimized parameter ω (T ) . 3: Initialize parameter ω (0) . 4: for t = 1 to T do 5:Draw a mini-batch B = {(x b , y b )} B b=1 ∼ S.Divide the mini-batch into M disjoint sub-batches {B m } M m=1 of equal size.for m = 1 to M in parallel do 8:Draw a noise sample ξ m ∼ Ξ.Compute the gradientg m = ∇ ω L(ω; B m ) ω (t) •ξ . 10:end for 11:Compute the average gradient:g = 1 M M m=1 g m .12:Update the weights: ω (t+1) = ω (t) -η t g + λω (t) . 13: end for 3 Adaptive Sharpness-Aware Minimization optimizes DNNs under adversarial multiplicative weight perturbationsIn this section, we demonstrate that optimizing DNNs with adversarial MWPs follows a similar update rule to Adaptive Sharpness-Aware Minimization (ASAM) (Kwon et al., 2021) . We first provide a brief description of ASAM and its predecessor Sharpness-Aware Minimization (SAM) (Foret et al., 2021) :SAM Motivated by previous findings that wide optima tend to generalize better than sharp ones (Keskar et al., 2017; Jiang et al., 2020) , SAM regularizes the sharpness of an optimum by solving the following minimax optimization:min ω max ||ξ||2≤ρ L(ω + ξ; S) + λ 2 ||ω|| 2 F (8)which can be interpreted as optimizing DNNs under adversarial additive weight perturbations. To efficiently solve this problem, Foret et al. (2021) devise a two-step procedure for each iteration t:EQUATIONwhere η t is the learning rate. Each iteration of SAM thus takes twice as long to run than SGD.ASAM Kwon et al. (2021) note that SAM attempts to minimize the maximum loss over a rigid sphere of radius ρ around an optimum, which is not suitable for ReLU networks since their parameters can be freely re-scaled without affecting the outputs. The authors thus propose ASAM as an alternative optimization problem to SAM which regularizes the adaptive sharpness of an optimum:EQUATION)where T ω is an invertible linear operator used to reshape the perturbation region (so that it is not necessarily a sphere as in SAM). Kwon et al. (2021) found that T ω = |ω| produced the best results. Solving Eq. ( 10) in this case leads to the following two-step procedure for each iteration t:EQUATION; S) + λω (t) (11)Similar to SAM, each iteration of ASAM also takes twice as long to run than SGD.ASAM and adversarial multiplicative perturbations Algorithm 1 minimizes the expected loss in Eq. ( 7). Instead, we could minimize the loss under the adversarial MWP:L max (ω; S) := max ||ξ||2≤ρ L(ω + ω • ξ; S) + λ 2 ||ω|| 2 F (12)Following Foret et al. (2021) , we solve this optimization problem by using a first-order Taylor expansion of L(ω + ω • ξ; S) to find an approximate solution of the inner maximization:EQUATIONThe maximizer of the Taylor expansion is:ξ(ω) = ρ ω • ∇ ω L(ω; S) ||ω • ∇ ω L(ω; S)|| 2 (14)Subtituting back into Eq. ( 12) and differentiating, we get:EQUATION)where ω is the perturbed weight:ω = ω + ω • ξ(ω) = ω + ρ ω 2 • ∇ ω L(ω; S) ||ω • ∇ ω L(ω; S)|| 2 (17)Similar to Foret et al. (2021) , we omit the second summand in Eq. ( 16) for efficiency, as it requires calculating the Hessian of the loss. We then arrive at the gradient formula in the update rule of ASAM in Eq. ( 11). We have thus established a connection between ASAM and adversarial MWPs.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of our work in Section 6, which outlines the shortcomings of our theoretical proofs and our experiments. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: We provide all the details regarding our experiments in Appendix F and specify sufficient details to understand the results in the main paper. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
empirical evaluation: 
 In this section, we assess the corruption robustness of DAMP and ASAM in image classification tasks.We conduct experiments using the CIFAR-10/100 (Krizhevsky, 2009) , TinyImageNet (Le and Yang, 2015) , and ImageNet (Deng et al., 2009) datasets. For evaluation on corrupted images, we utilize the CIFAR-10/100-C, TinyImageNet-C, and ImageNet-C datasets provided by Hendrycks and Dietterich (2019) , as well as ImageNet-C (Mintun et al., 2021) , ImageNet-D (Zhang et al., 2024) , ImageNet-A (Hendrycks et al., 2021) , ImageNet-Sketch (Wang et al., 2019) , ImageNet-Drawing (Salvador and Oberman, 2022) , and ImageNet-Cartoon (Salvador and Oberman, 2022) datasets, which encapsulate a wide range of corruptions. Detail descriptions of these datasets are provided in Appendix E. We further evaluate the models on adversarial examples generated by the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) . In terms of architectures, we use ResNet18 (He et al., 2016a) for CIFAR-10/100, PreActResNet18 (He et al., 2016b) for TinyImageNet, ResNet50 (He et al., 2016a) , ViT-S/16, and ViT-B/16 (Dosovitskiy et al., 2021) for ImageNet. We ran all experiments on a single machine with 8 Nvidia V100 GPUs. Appendix F includes detailed information for each experiment.In this section, we compare the corruption robustness of DNNs trained using DAMP with those trained directly on corrupted images. To train models on corrupted images, we utilize Algorithm 2 described in the Appendix. For a given target corruption g, Algorithm 2 randomly selects half the images in each training batch and applies g to them. This random selection process enhances the final model's robustness to the target corruption while maintaining its accuracy on clean images. We use the imagecorruptions library (Michaelis et al., 2019) to apply the corruptions during training.Evaluation metric We use the corruption error CE f c (Hendrycks and Dietterich, 2019) which measures the predictive error of classifier f in the presence of corruption c. Denote E f s,c as the error of classifier f under corruption c with corruption severity s, the corruption error CE f c is defined as: 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 For this metric, lower is better. Here f baseline is a baseline classifier whose usage is to make the error more comparable between corruptions as some corruptions can be more challenging than others (Hendrycks and Dietterich, 2019) . For each experiment setting, we use the model trained by SGD without corruptions as f baseline .EQUATIONWe visualize the results for the ResNet18/CIFAR-100 setting in Fig. 3 . The results for the ResNet18/CIFAR-10 and PreActResNet18/TinyImageNet settings are presented in Figs. 5 and 6 in the Appendix. Figs. 3, 5 and 6 demonstrate that DAMP improves predictive accuracy over plain SGD across all corruptions without compromising accuracy on clean images. Although Fig. 3 indicates that including zoom_blur as an augmentation when training ResNet18 on CIFAR-100 yields better results than DAMP on average, it also reduces accuracy on clean images and the brightness corruption. Overall, these figures show that incorporating a specific corruption as data augmentation during training enhances robustness to that particular corruption but may reduce performance on clean images and other corruptions. In contrast, DAMP consistently improves robustness across all corruptions. Notably, DAMP even enhances accuracy on clean images in the PreActResNet18/TinyImageNet setting, as shown in Fig. 6 .In this section, we investigate whether additive weight perturbations can also enhance corruption robustness. To this end, we compare DAMP with its variant, Data Augmentation via Additive Perturbations (DAAP). Unlike DAMP, DAAP perturbs weights during training with random additive Gaussian noises centered at 0, as detailed in Algorithm 3 in the Appendix. Fig. 7 in the Appendix presents the results of DAMP and DAAP under different noise standard deviations, alongside standard SGD. Overall, Fig. 7 shows that across different experimental settings, the corruption robustness of DAAP is only slightly better than SGD and is worse than DAMP. Therefore, we conclude that MWPs are better than their additive counterparts at improving robustness to corruptions.In this section, we compare DAMP with Dropout (Srivastava et al., 2014) , SAM (Foret et al., 2021) , and ASAM (Kwon et al., 2021) . For SAM and ASAM, we optimize the neighborhood size ρ by using 10% of the training set as a validation set. Similarly, we adjust the noise standard deviation σ for DAMP and the drop rate p for Dropout following the same procedure. For hyperparameters and additional training details, please refer to Appendix F. CIFAR-10/100 and TinyImageNet. (Szegedy et al., 2016) . Remarkably, DAMP can train ViT-S16 from scratch in 200 epochs to match ResNet50's accuracy without advanced data augmentation. This is significant as ViT typically requires either extensive pretraining (Dosovitskiy et al., 2021) , comprehensive data augmentation (Beyer et al., 2022) , sophisticated training techniques (Chen et al., 2022) , or modifications to the original architecture (Yuan et al., 2021) to perform well on ImageNet. Additionally, DAMP consistently ranks in the top 2 for corruption robustness across various test settings and has the best corruption robustness on average (last column). Comparing Tables 1 and 2 reveals that ViT-S16 is more robust to corruptions than ResNet50 when both have similar performance on clean images.ViT / ImageNet / Advanced augmentations  
related works: 
 Dropout Perhaps most relevant to our method is Dropout (Srivastava et al., 2014) and its many variants, such as DropConnect (Wan et al., 2013) and Variational Dropout (Kingma et al., 2015) . These methods can be viewed as DAMP where the noise distribution Ξ is a structured multivariate Bernoulli distribution. For instance, Dropout multiplies all the weights connecting to a node with a binary random variable p ∼ Bernoulli(ρ). While the main motivation of these Dropout methods is to prevent co-adaptations of neurons to improve generalization on clean data, the motivation of DAMP is to improve robustness to input corruptions without harming accuracy on clean data. Nonetheless, our experiments show that DAMP can improve generalization on clean data in certain scenarios, such as PreActResNet18/TinyImageNet and ViT-S16/ImageNet.Ensemble methods Ensemble methods, such as Deep ensembles (Lakshminarayanan et al., 2017) and Bayesian neural networks (BNNs) (Graves, 2011; Blundell et al., 2015; Gal and Ghahramani, 2016; Louizos and Welling, 2017; Izmailov et al., 2021; Trinh et al., 2022) , have been explored as effective defenses against corruptions. Ovadia et al. (2019) benchmarked some of these methods, demonstrating that they are more robust to corruptions compared to a single model. However, the training and inference costs of these methods increase linearly with the number of ensemble members, making them inefficient for use with very large DNNs.Data augmentation Data augmentations aim at enhancing robustness include AugMix (Hendrycks et al., 2019) , which combines common image transformations; Patch Gaussian (Lopes et al., 2019), which applies Gaussian noise to square patches; ANT (Rusak et al., 2020) , which uses adversarially learned noise distributions for augmentation; and AutoAugment (Cubuk et al., 2018) , which learns augmentation policies directly from the training data. These methods have been demonstrated to improve robustness to the corruptions in ImageNet-C (Hendrycks and Dietterich, 2019) . Mintun et al. (2021) attribute the success of these methods to the fact that they generate augmented images perceptually similar to the corruptions in ImageNet-C and propose ImageNet-C, a test set of 10 new corruptions that are challenging to models trained by these augmentation methods.Test-time adaptations via BatchNorm One effective approach to using unlabelled data to improve corruption robustness is to keep BatchNorm (Ioffe and Szegedy, 2015) on at test-time to adapt the batch statistics to the corrupted test data (Li et al., 2016; Nado et al., 2020; Schneider et al., 2020; Benz et al., 2021) . A major drawback is that this approach cannot be used with BatchNorm-free architectures, such as Vision Transformer (Dosovitskiy et al., 2021) . 
conclusion: 
 In this work, we demonstrate that MWPs improve robustness of DNNs to a wide range of input corruptions. We introduce DAMP, a simple training algorithm that perturbs weights during training with random multiplicative noise while maintaining the same training cost as standard SGD. We further show that ASAM (Kwon et al., 2021) can be viewed as optimizing DNNs under adversarial MWPs. Our experiments show that both DAMP and ASAM indeed produce models that are robust to corruptions. DAMP is also shown to improve sample efficiency of Vision Transformer, allowing it to achieve comparable performance to ResNet50 on medium size datasets such as ImageNet without extensive data augmentations. Additionally, DAMP can be used in conjunction with modern augmentation techniques such as MixUp and RandAugment to further boost robustness. As DAMP is domain-agnostic, one future direction is to explore its effectiveness in domains other than computer vision, such as natural language processing and reinforcement learning. Another direction is to explore alternative noise distributions to the Gaussian distribution used in our work.Limitations Here we outline some limitations of this work. First, the proof of Theorem 1 assumes a simple feedforward neural network, thus it does not take into accounts modern DNN's components such as normalization layers and attentions. Second, we only explored random Gaussian multiplicative perturbations, and there are likely more sophisticated noise distributions that could further boost corruption robustness. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: For Figs. 4 and 7, the error bars display 95% confidence intervals. For Table 1 , we report the standard deviation. For Figs. 3, 5 and 6, we cannot display the error bars since these figures show heatmaps. For Tables 2 and 3 , we does not report the error bars since we ran each experiment once due to the high training cost and our limit in computing resources. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss at the end of the paper the possible societal impact of the work in this paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not release any new datasets or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cite all the papers that produced the datasets and model architectures used in this work.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We provide our code in a public GitHub repository with documentation on how to run the code.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This work does not involve crowdsourcing nor human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This work does not involve crowdsourcing nor human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our paper introduces a new training method for neural networks that improves their robustness to input corruptions. Therefore, we believe that our work contributes towards making deep leading models safer and more reliable to use in real-world applications, especially those that are safety-critical. However, as with other methods that improve robustness, our method could also be improperly used in applications that negatively impact society, such as making mass surveillance systems more accurate and harder to fool. To this end, we hope that practitioners carefully consider issues regarding fairness, bias and other potentially harmful societal impacts when designing deep learning applications.Proof. Here we note that:EQUATIONδ g ℓ(ω, x, y) := ℓ(ω, x g , y) -ℓ(ω, x, y) (21)EQUATIONWe first notice that the per-sample loss ℓ(ω, x, y) can be viewed as a function of the intermediate activation x (h) of layer h (see Fig. 2 ). From Assumption 3, there exists a constant L h > 0 such that:||∇ x (h) g ℓ(ω, x g , y) -∇ x (h) ℓ(ω, x, y)|| 2 ≤ L h ||δ g x (h) || 2 (23)which gives us the following quadratic bound:EQUATION)where ⟨•, •⟩ denotes the dot product between two vectors. The results in the equation above have been proven in Böhning and Lindsay (1988) . Subtracting ℓ(ω, x, y) from both side of Eq. ( 24) gives us:δ g ℓ(ω, x, y) ≤ ∇ x (h) ℓ(ω, x, y), δ g x (h) + L h 2 ||δ g x (h) || 2 2 (25)Since the pre-activation output of layer h) , we can rewrite the inequality above as:h + 1 is z (h+1) (x) = W (h+1) f (h) (x) = W (h+1) x (EQUATION)where ⊗ denotes the outer product of two vectors and ⟨•, •⟩ F denotes the Frobenius inner product of two matrices of similar dimension.From Assumption 1, we have that there exists a constant M > 0 such that:||δ g x (0) || 2 2 = ||x (0) g -x (0) || 2 2 = ||g(x) -x|| 2 2 ≤ M (27)Given that x (1) = σ (1) W (1) x (0) , we have:||δ g x (1) || 2 2 = ||x (1) g -x (1) || 2 2 ≤ ||W (1) δ g x (0) || 2 2 (28)Here we assume that the activate σ satisfies ||σ(x) -σ(y)|| 2 ≤ ||x -y|| 2 , which is true for modern activation functions such as ReLU. Since ||δ g x (0) || 2 2 is bounded, there exists a constant Ĉ(1) g (x) such that:EQUATION)where || • || F denotes the Frobenius norm. Similarly, as we have proven that ||δ g x (1) || 2 2 is bounded, there exists a constant Ĉ(2) g (x) such that:EQUATION)Thus we have proven that for all h = 1, . . . , H, there exists a constant Ĉ(h) g (x) such that:||δ g x (h) || 2 2 ≤ Ĉ(h) g (x) 2 ||W (h) || 2 F (31)By combining Eqs. ( 26) and ( 31) and setting C (h)g (x) = L h Ĉ(h) g (x), we arrive at Eq. ( 4).Proof. From Lemma 1, we have for all h = 0, . . . , H -1:EQUATION)whereĈ(h) g = max x∈S C (h) g (x). Since this bound is true for all h, we can take the average:EQUATION)whereC g = 1 H H h=1Ĉ(h) g . The right-hand side of Eq. ( 34) can be written as:EQUATION)where ξ (h) (g) is a matrix of the same dimension as W (h) whose each entry is defined as:EQUATIONThe inequality in Eq. ( 37) is due to the first-order Taylor expansion and the assumption that the training loss is locally convex at ω. This assumption is expected to hold for the final solution but does not necessarily hold for any ω. Eq. ( 5) is obtained by combining Eq. ( 34) and Eq. ( 37).Here we present Algorithm 2 which uses corruptions as data augmentation during training, as well as the experiment results of Section 4.1 for ResNet18/CIFAR-10 and PreActResNet18/TinyImageNet settings in Figs. 5 and 6 .Here, we present Algorithm 3 used in Section 4.2 which trains DNNs under random additive weight perturbations and Fig. 7 comparing performance between DAMP and DAAP.CIFAR-10/100-C (Hendrycks and Dietterich, 2019) These datasets contain the corrupted versions of the CIFAR-10/100 test sets. They contain 19 types of corruption, each divided into 5 levels of severity.TinyImageNet-C (Hendrycks and Dietterich, 2019) Draw a mini-batch B = {(x b , y b )} B b=1 ∼ S.Divide the mini-batch into two disjoint sub-batches of equal size B 1 and B 2 .7:Apply the corruption g to all samples in B 1 : g(B 1 ) = {(g(x), y)} (x,y)∈B1 .8:Compute the gradient g = ∇ ω L(ω; g(B 1 ) ∪ B 2 ).Update the weights: ω (t+1) = ω (t) -η t g + λω (t) . 10: end for 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.90 0.87 0. ImageNet-C (Hendrycks and Dietterich, 2019) This dataset contains the corrupted versions of the ImageNet validation set, as the labels of the true ImageNet test set was never released. It contains 15 types of corruption, each divided into 5 levels of severity.Our code is available at https://github.com/trungtrinh44/DAMPFor instance, if x is a clean image then g(x) could be x corrupted by Gaussian noise. 


Decision: The paper tackles the problem of "robustness" of deep neural networks (DNN) in the presence of "corrupted" images. The main intuition/contribution is a method that improves the robustness of DNN to corrupted images by manipulating the weights of the DNN during its training phase---thereby suggesting that there is some sort of correlation between perturbations in the "input space" (i.e., in the images analysed by the DNN) and those in the "weight space" (i.e., in the weights of the DNN). Experiments on various benchmark datasets, as well as various theoretical proofs, support the effectiveness of the proposed method. Moreover, **the rebuttal phase also allowed to shed light on the fact that the proposed method is not specific to DNN analysing images** (as demonstrated in an experiment on the MedMCQA dataset), thereby further showcasing the generality and "improvement" of the proposed solution over existing alternatives.


Four reviews were submitted for this paper. Two reviewers (with confidence 3 and 4) originally recommended a "5-Borderline Accept" and then increased to a "6-Weak Accept" after interacting with the authors. Another reviewer (with a confidence of 2) proposed a "6-Weak Accept" and maintained their score after the rebuttal phase. A fourth reviewer recommended a "4-Borderline Reject" (with a high-confidence of 4), rating the paper with 1-2-1 for Soundness, Presentation and Contribution, but the exchanges with the authors led such a reviewer to improve their Soundness and Contribution ratings (to a 3 and to a 2) and also to recommend a 5 "Borderline Accept". Hence, it can be said that the author-reviewer interactive phase played a pivotal role in clarifying lingering doubts as well as in increasing the quality of this submission. This is a positive sign of a constructive peer-review.

No factual errors were found by the reviewers. The code is also made public. Moreover, all reviewers agreed in that the paper provides a meaningful contribution to the state of the art. Some praised the extensive experiments, or the theoretical foundations, or even the presentation of the paper---all of which being aspects that I also believe to be significant strengths of this submission. 

Due to the above, I recommend acceptance of this work---with a "spotlight". I could not identify glaring weaknesses, and the combination of sound experiments, theoretical analyses, good presentation, nice intuition, and generalizability make me believe that this paper can spearhead many discussions in the field.

Reviews: review: paper_summary: paper_summary: This paper works on improving robustness by multiplying a random Gaussian variable on weights during training.
strength_weakness: strength_weakness: 1. The novelty is quite limited. This concept has been proposed and explored for at least a decade. For instance, variational dropout already suggested this approach, using a Bernoulli distribution. Another example can be found in [1], "Multiplicative or Additive Perturbation?", where they used a Gaussian distribution.

2. The theoretical foundation is weak. The theory section consists of trivial analysis by adding constrained corruption to inputs, which is not significantly related to this work.

Reference:

1. Dusenberry, Michael, et al. "Efficient and scalable Bayesian neural nets with rank-1 factors." International Conference on Machine Learning. PMLR, 2020.
questions: questions: 1. What is the novelty of this work?

2. How does DAMP compare to modern augmentation techniques such as those used in DeiT and subsequent works? DAMP's augmentation does not appear to be simpler than these methods. Furthermore, DAMP is evidently more unstable, particularly in large-scale training, due to the stochasticity introduced during training, a common issue observed in many Bayesian Neural Network (BNN) related works.
limitations: limitations: See the above sections.
paper_summary: paper_summary: The paper presents Data Augmentation via Multiplicative Perturbations (DAMP), a novel method to enhance DNN robustness against image corruptions by optimizing with random multiplicative weight perturbations. This approach improves generalization on corrupted images without compromising accuracy on clean ones. The authors show that input perturbations can be mimicked by multiplicative perturbations in the weight space. The authors demonstrate DAMP's effectiveness across multiple datasets and architectures, and explore its connection to Adaptive Sharpness-Aware Minimization (ASAM).
strength_weakness: strength_weakness: It would be beneficial to see how the method performs with different hyperparameter values, as the reported numbers for different metrics are close to each other. Assumption 2 is not explained very well and could benefit from a clearer, more detailed explanation. While DAMP works well for small models and datasets, it would be interesting to see the results with larger models and datasets, as it seems to show some instability in results on these larger models and datasets.
questions: questions: Why is DAMP trained for 200 epochs compared to other methods trained for 100 epochs in Table 2?
How does the model perform when tested on other types of distribution shifts beyond those included in the corruption benchmarks?
How do the methods perform without basic Inception-style preprocessing?
limitations: limitations: The results could be better explained, particularly why the accuracy on clean images and lower severity corruptions is lower than the ASAM method in larger-scale experiments for ResNet/ImageNet.
paper_summary: paper_summary: This submission tackles the problem of generalization of image classifiers to corrupted images. The paper shows a link between data augmentation as well as previous methods such as Adaptive Sharpness-aware Minimization (ASAM) and multiplicative weight perturbations. Effectively, ASAM works as an adversarial weight multiplication which requires a second gradient pass per batch. The proposed DAMP method relaxes this constraint by using random weight perturbations. DAMP is benchmarked on CIFAR-10/100, TinyImageNet, and ImageNet under common corruptions (and follow-up works) with ResNet and ViT where it often matches or even exceeds the original SAM at lower cost.
strength_weakness: strength_weakness: My only concern about this work is the contextualization of this work. It seems like DAMP is mostly an approximation of ASAM which allows faster training (and that is great) but at the same time, it is not as effective as ASAM. So effectively, this method introduces a trade-off between performance (accuracy) and training time. This raises the following detailed concerns:

- If multiplicative weight perturbations are connected to data augmentation then how does DAMP rank against them? Augmentations are mostly cheap so they could achieve even better performance at the same overhead cost. One indication of this is in the ViT experiments on ImageNet: DAMP achieves a 23.7% clean error in 200 epochs without augmentations. This may sound impressive at first, however, the baseline that the authors rely on (Beyer, 2022) even exceeds this (23.5% error) in just 90 epochs. The only difference is that (Beyer, 2022) utilizes some simple augmentations. That is only clean accuracy but usually, it is indicative of performance under distribution shift [1]. Ideally, the authors would demonstrate that on a train time vs. mCE or clean acc curve, DAMP exceeds the current Pareto frontier (by benchmarking a few modern augmentation techniques in addition to SAM/ASAM). 
- Additionally, it would be great to understand if DAMP is compatible with augmentations. I don't want to use the clearly stated Limitations against the authors but this is a very straightforward question that I had during reading. 


Other than that the theory in Eq.1ff assumes bias-free MLP architectures without non-linearities which obviously does not scale to modern architectures. (The authors mention this in their limitations and this is not something that needs to be addressed, but it is a weakness nonetheless)

Nitpicks:
- L118: I agree that random perturbations introduce almost no overhead, but you shouldn't state "we found that ... had similar training times" in a scientific paper without backing it up with hard numbers.
- It would be great to add averages over all severities in Tab. 1 and 2.

Suggestion: 
- A simple way to kill all my concerns is to show performance on non-vision tasks (since DAMP is in theory data-agnostic). In that case, it would exceed the value of domain-specific augmentations. This would greatly enhance this paper but of course, I do not expect this from a rebuttal.

[1] Taori et al., "Measuring Robustness to Natural Distribution Shifts in Image Classification", NeurIPS 2020.
questions: questions: See above.
limitations: limitations: The authors are very upfront about Limitations and I have no other suggestions.
paper_summary: paper_summary: To achieve robustness against corruptions, most of existing works relying on incorporating some corruptions during the training. In contrast, this paper takes a different perspective and propose a weight-based approach called DAMP to achieve model robustness against corruptions without compromising accuracy on clean examples. 

This work also studies Sharpness-Aware Minimisation (SAM) methods and point out the relationship between their proposed DAMP and SAM-based methods.
strength_weakness: strength_weakness: There could be a related work [1]  focusing on (adversarial) corruptions, where it also approaches the model robustness problem from model weights perspective. I suggest the authors to take a look at [1] and compare with their work.

The experimental results relying on only algorithmically corruption (ImageNet-C, CIFAR-C) could be limited. I suggest the author to conduct the results on other corruption (such as Stylised ImageNet, ImageNet-D), natural corruption (such as ImageNet-A, ImageNet-Sketch), and adversarial corruption. I believe this assessment will further improve the experimental results.

The results for SAM/ASAM in Tables 1 and 2 are relatively comparable to those for DAMP when considering the effect of accuracy on clean images. As we know, improving accuracy on clean images generally leads to better accuracy on corrupted images. While I understand that this work takes a different approach from SAM/ASAM, I am curious to know what advantages DAMP offers over SAM/ASAM.

As the theory based on a simple feedforward neural network, I wonder if the DAMP can improve the robustness on deeper/modern neural networks, such as Resnet-152, EfficientNet, ConvNext, MaxViT? 

[1] Ha, Hyeonjeong, Minseon Kim, and Sung Ju Hwang. "Generalizable lightweight proxy for robust NAS against diverse perturbations." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: What is the effectiveness of DAMP on deeper/modern neural networks, such as Resnet-152 EfficientNet, ConvNext, MaxViT? 

What is the effectiveness of DAMP on other corruption (such as Stylised ImageNet, ImageNet-D), natural corruption (such as ImageNet-A, ImageNet-Sketch), and adversarial corruption?

What is the advantage of DAMP over SAM/ASAM?
limitations: limitations: N/A

Comments: title: Official Comment by Reviewer dwU7
comment: Thank you for your response.

**1. On the connection between DAMP and variational inference methods such as variational DropConnect [1] and SSVI [2]**

DAMP indeed could be interpreted as performing variational inference. This can be seen from the objective function of DAMP (Eq. 16 in our paper), which we restate below:
$$\mathcal{L}\_{DAMP}(\boldsymbol{\omega};\mathcal{S})=\underbrace{\mathbb{E}\_{\boldsymbol{\xi}\sim p(\boldsymbol{\xi})}\left[\mathcal{L}(\boldsymbol{\omega}\circ\boldsymbol{\xi};\mathcal{S})\right]}\_{\text{expected loss}} + \underbrace{\frac{\lambda}{2}||\boldsymbol{\omega}||\_F^2}\_{\text{$L\_2$-regularization}}$$
where:
* $\boldsymbol{\omega}$ is the model's weights.
* $\boldsymbol{\xi}$ is the vector of the multiplicative random variables with the same dimension as $\boldsymbol{\omega}$ whose distribution is $p(\boldsymbol{\xi})=\mathcal{N}(\boldsymbol{1},\sigma^2\mathbf{I})$.
* $\circ$ denotes the Hadamard product.
* $\mathcal{S}$ is the training data.
* $||\cdot||\_F^2$ denotes the Frobenius norm.
* $\mathcal{L}$ is **the original loss function which is the negative log-likelihood loss** for classification tasks.

Therefore, the **expected loss** in the above equation can be interpreted as the expected log-likelihood term of the evidence lower-bound (ELBO) (the loss function used in variational inference) where the variational distribution is the distribution of the multiplicative random variables $p(\boldsymbol{\xi})$. In fact, since we don't optimize the variational distribution $p(\boldsymbol{\xi})$ and keep it fixed at $\mathcal{N}(\boldsymbol{1},\sigma^2\mathbf{I})$ throughout training, **minimizing the loss function $\mathcal{L}\_{DAMP}$ is equivalent to minimizing the negative ELBO plus an $L\_2$-regularization term** (since the KL divergence typically presented in the ELBO has no effect on training when we don't optimize the variational distribution.)

From this perspective, DAMP is indeed related to [1] and [2] as you have suggested. Furthermore, our work presents directions to scale these variational approaches to large models and datasets, as you have concluded above. We will include the main points of this  discussion in the revised version of our paper.

We hope that this response is helpful to further understand our method.

[1] Rimella, Lorenzo, and Nick Whiteley. "Dynamic Bayesian Neural Networks." arXiv preprint arXiv:2004.06963 (2020).

[2] Li, Junbo, et al. "Training Bayesian Neural Networks with Sparse Subspace Variational Inference." arXiv preprint arXiv:2402.11025 (2024).
I appreciate the authors for their prompt response. It appears that DMAP is more similar to variational DropConnect [1], as it involves multiplying random variables with weights rather than activations. This approach also seems akin to variational inference in Bayesian Neural Networks (BNNs), where independent Gaussian variables are multiplied or added to weights in each forward process in practice, as seen in recent methods like SSVI [2].

I would be inclined to support acceptance if the authors could clarify these 2 points further in future revisions. This work seems to provide valuable guidance and aims to scale these techniques to larger models and datasets, with a specific benefit in robustness.

1. Rimella, Lorenzo, and Nick Whiteley. "Dynamic Bayesian Neural Networks." arXiv preprint arXiv:2004.06963 (2020).

2. Li, Junbo, et al. "Training Bayesian Neural Networks with Sparse Subspace Variational Inference." arXiv preprint arXiv:2402.11025 (2024).
Thank you for engaging in the discussion on our work.

We will demonstrate the differences between DAMP and Dropout/Variational Dropout (VD) below using a simple linear layer:
* Original calculation of the forward pass without DAMP, Dropout or VD: $$\mathbf{y} = \mathbf{W}\cdot\mathbf{x}$$ where $\mathbf{y} \in \mathbb{R}^{m}, \mathbf{x} \in \mathbb{R}^{n}$ and $\mathbf{W} \in \mathbb{R}^{m \times n}$ and $\cdot$ is the matrix-vector product.
* DAMP: Calculating the forward pass of the linear layer above with DAMP is as follows: $$\mathbf{y}\_{DAMP} = (\mathbf{W} \circ \boldsymbol{\xi})\cdot\mathbf{x}=\underbrace{\begin{bmatrix}W\_{1,1}\xi\_{1,1} & W\_{1,2}\xi\_{1,2} & \cdots & W\_{1,n}\xi\_{1,n} \\\ W\_{2,1}\xi\_{2,1} & W\_{2,2}\xi\_{2,2} & \cdots & W\_{2,n}\xi\_{2,n} \\\ \vdots & \vdots & \ddots & \vdots \\\ W\_{m,1}\xi\_{m,1} & W\_{m,2}\xi\_{m,2} & \cdots & W\_{m,n}\xi\_{m,n}\end{bmatrix}}\_{\mathbf{W}\circ\boldsymbol{\xi}}\cdot\mathbf{x}$$ where $\mathbf{W},\boldsymbol{\xi} \in \mathbb{R}^{m\times n}$ are two matrices of the same dimension and $\circ$ denotes the Hadamard product. Here each element $\xi\_{i,j}$ of $\boldsymbol{\xi}$ is a **different** Gaussian random variable with distribution $\mathcal{N}(1, \sigma^2\_{i,j})$. In DAMP, for simplicity, we set $\sigma\_{i,j}^2 = \sigma^2$ for all $i\in[1,m]$ and $j\in[1,n]$. Therefore, DAMP **multiplies each individual weight with its own Gaussian random variable**.
* Dropout/VD: Calculating the forward pass of the linear layer above with Dropout/VD is as follows: $$\mathbf{y}\_{Dropout/VD}=\mathbf{W}\cdot(\mathbf{x}\circ\mathbf{r})=\mathbf{W}\cdot\begin{bmatrix}x\_1r\_1 \\\ x\_2r\_2 \\\ \vdots \\\ x\_nr\_n\end{bmatrix}=\begin{bmatrix}W\_{1,1}r\_1 & W\_{1,2}r\_2 & \cdots & W\_{1,n}r\_n \\\ W\_{2,1}r\_1 & W\_{2,2}r\_2 & \cdots & W\_{2,n}r\_n \\\ \vdots & \vdots & \ddots & \vdots \\\ W\_{m,1}r\_1 & W\_{m,2}r\_2 & \cdots & W\_{m,n}r\_n\end{bmatrix}\cdot\mathbf{x}$$ where $\mathbf{r}\in\mathbb{R}^n$ has the same dimension as input $\mathbf{x}$. For Dropout, each element $r\_i$ of $\mathbf{r}$ is a random variable following a Bernoulli distribution, while for VD, they follow the Gaussian distribution. As the equation has demonstrated, Dropout and VD **share the same multiplicative random variable for all weights connecting to the same input node**, i.e., all the weights in each column $i$ of matrix $\mathbf{W}$ share the same random variable $r\_i$.

We hope that these equations are helpful to further understand the differences between DAMP and Dropout/VD, and we are happy to answer any follow-up questions that you might have.
Thank you to the authors for their response. Could you please elaborate further on the section regarding the same/different input nodes mentioned in point 1? It might be helpful to illustrate this with a specific mathematical formula.
Thank you for your comments.

**1. On the differences between DAMP, Variational Dropout (VD) and Dropout**

We need to emphasize that there is an important difference between DAMP, VD and Dropout. More details below:

While it is true that DAMP is closely related to the family of Dropout methods (which includes VD) as we have stated in the Related works section of our paper, we also highlight one crucial difference: DAMP **multiplies each individual weight with its own Gaussian random variable (RV)** $\mathcal{N}(1, \sigma^2)$, while VD and Dropout **share the same multiplicative RV for all weights connecting to the same input node** (VD uses Gaussian RVs while Dropout uses Bernoulli RVs). This difference is very important because we have shown that DAMP **can train a ViT-S16 from scratch on ImageNet to match performance of a ResNet50 without strong data augmentions**, while Dropout is unable to do so as stated in Section 3 of [1]  which we directly quote below:

>  Existing works report that ViTs yield inferior accuracy to the ConvNets of similar size and throughput when trained from scratch on ImageNet without the combination of those advanced data augmentations, despite using various regularization techniques (e.g., large weight decay, Dropout (Srivastava et al., 2014), etc.).

This phenomenon suggests that the weight perturbations of DAMP are more expressive than VD and Dropout.

The reason we did not mention VD further in our rebuttal is because we already mentioned [2] which is a more recent work that uses Gaussian multiplicative weight perturbations. We also emphasize that this work also **shares the same multiplicative Gaussian RVs for all weights connecting to the same input node** just like VD and therefore is different from DAMP.

Regarding Bayesian neural networks (BNNs), which you mention as the reason why VD improves robustness, they achieve robustness by ensembling multiple predictions from samples drawn from their weight posteriors via Bayesian model averaging [3]. Thus at test time, BNNs need to make multiple forward passes to make a prediction on each test input. Our method directly improves the robustness of deterministic neural networks, which only need one forward pass to make a prediction on each test input.

[1] Chen et al. "When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations". ICLR 2022.

[2] Dusenberry et al. "Efficient and scalable Bayesian neural nets with rank-1 factors." ICML, 2020.

[3] Wilson et al. “Bayesian Deep Learning and a Probabilistic Perspective of Generalization.” 	arXiv:2002.08791. 2022

**2. On the novelty of our paper**

In addition to introducing a new training method (DAMP), the novelty of our work also includes providing a new perspective on Adaptive Sharpness Aware Minimization (ASAM) [4], showing that it optimizes neural networks under adversarial **multiplicative** weight perturbations, while its predecessor Sharpness Aware Minimization (SAM) [5] optimizes under adversarial **additive** weight perturbations. This explains why both DAMP and ASAM outperform SAM on corrupted test sets as we showed in our paper.

Furthermore, while we agree that Dropout-like weight-based perturbation approach to robustness is not new, most prior works we could find only consider Bernoulli random variables. This is also evidenced by the fact that the training recipes of large vision models such as ViT only consider using Dropout and not the alternatives. Here we shed light on DAMP which is possibly a better alternative of Dropout and we demonstrate that DAMP greatly improves performance and robustness of ViT, regardless of whether strong data augmentation techniques are applied.

[4] Kwon et al. “ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks”. ICML, 2022

[5] Foret et al. “Sharpness-Aware Minimization for Efficiently Improving Generalization”. ICLR, 2021

**3. "Just using Mixup and Randaugment seems not to be a strong baseline. What about stronger baseline such as the commonly-used DeiT ? Actually they mentioned that dropout hurts performance in their settings."**

We use MixUp and RandAugment following the training recipe from [6] and we believe it is not a weak baseline since it contains half the augmentation techniques used by DeiT (DeiT uses MixUp, RandAugment, CutMix and random erase). In fact, the left panel in Figure 4 of [6] shows that with only MixUp and RandAugment, ViT-B16 can reach up to 83% test accuracy on ImageNet, which is similar to the 83.1% test accuracy achieved by DeiT-B (Table 7 of [7]).

Due to the differences between DAMP and Dropout stated above, the fact that Dropout hurts performance of DeiT does not imply that DAMP would as well.

[6] Steiner et al. "How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers". TMLR 2022.

[7] Touvron et al. "Training data-efficient image transformers & distillation through attention". ICML 2021.
Thank you to the authors for their response. Unfortunately, it does not address my concerns regarding this paper.

Is the technique new? No. This is simply variational dropout (VD). While the authors cite the paper by Kingma et al. (2015) in the related work section, their comments are somewhat misleading. The VD series of works has already established that traditional dropout involves multiplying Bernoulli variables, and that these can also be Gaussian variables, which is exactly what this work does. However, the authors only cite Kingma’s paper but overlook this point, merely mentioning that dropout involves multiplying a Bernoulli variable.

It's well-established that this is just a variant of dropout. Is using dropout to improve robustness new? Certainly not. A simple search for "dropout improve robustness" yields numerous papers and blog posts on this topic. Even for variational dropout, this approach isn't novel. Variational dropout is a standard technique in training Bayesian neural networks (BNNs), where testing robustness is a common metric, as BNNs are well-known for their enhanced robustness.

Is a weight-based approach to improve robustness new? No. Dropout itself can be considered a weight-based approach to improving robustness, and it is standard practice to combine dropout with other techniques applied to inputs.

Just using Mixup and Randaugment seems not to be a strong baseline. What about stronger baseline such as the commonly-used DeiT [1]? Actually they mentioned that dropout hurts performance in their settings.

1. Training data-efficient image transformers & distillation through attention
Thank you for your comments. We will definitely include results of SAM/ASAM+ViT-B16 in the revised version of the paper.
Thank you for the clarifications provided in your rebuttal. After reviewing the rebuttal and considering other comments, I will maintain my current rating. Including SAM/ASAM+ViT-B16 in Table D would further improve the paper.
Thank you for the comments and for the increased confidence in the proposed method.

We note that there are cases where DAMP does not need more training iterations to surpass ASAM/SAM. For instance, in the ResNet50 / ImageNet experiments (Table 1 in the paper), we use 90 epochs for all training methods, yet DAMP is able to perform competitively with ASAM and surpass SAM.

Furthermore, we show in Table D of our global response that training ViT-S16 / ImageNet using 500 epochs of DAMP (which took 111hrs) outperforms 300 epochs of ASAM/SAM (which took 123hrs). Therefore, even in the cases where DAMP requires more iterations to outperform ASAM/SAM, it still ends up requiring less overall runtime than ASAM/SAM.
I thank the authors' effort in conducting new experiments, which have made the proposed method more convincing to me.

However, the performance of DAMP does not appear to significantly surpass that of ASAM/SAM. I agree with the authors that DAMP generally offers a x2 speedup in iteration training time compared to ASAM/SAM. With the same training iterations (i.e., faster training time), ASAM/SAM still outperforms DAMP. DAMP only (marginally) surpasses ASAM/SAM when it is trained with more iterations, resulting in the same training time.

Consequently, I appreciate the authors for their thorough rebuttal and have raised my score to 6.
Thank you for the comments. We will definitely include these new results in the revised version of the paper.
I thank the authors for their rebuttal. The newly added experiments address my concerns - I'd suggest including them in the paper. I am raising my score to 6.