Maximizing Feature Distribution Variance for Robust Neural Networks

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:22:13

Abstract: The security of Deep Neural Networks (DNNs) has proven to be critical for their applicabilities in real-world scenarios. However, DNNs are well-known to be vulnerable against adversarial attacks, such as adding artificially designed imperceptible magnitude perturbation to the benign input. Therefore, adversarial robustness is essential for DNNs to defend against malicious attacks. Stochastic Neural Networks (SNNs) have recently shown effective performance on enhancing adversarial robustness by injecting uncertainty into models. Nevertheless, existing SNNs are still limited for adversarial defense, as their insufficient representation capability from the fixed uncertainty. In this paper, to elevate feature representation capability of SNNs, we propose a novel yet practical stochastic neural network that maximizes feature distribution variance (MFDV-SNN). In addition, we provide theoretical insights to support the adversarial resistance of MFDV, which primarily derived from the stochastic noise we injected into DNNs. Our research demonstrates that by gradually increasing the level of stochastic noise in a DNN, the model naturally becomes more resistant to input perturbations. Since adversarial training is not required, MFDV-SNN does not compromise clean data accuracy and saves up to 7.5 times computation time. Extensive experiments on various attacks demonstrate that MFDV-SNN improves adversarial robustness significantly compared to other methods.

Main paper: 
 introduction: 
 Despite the promising performance of deep learning models, recent studies have shown that Deep Neural Networks (DNNs) are vulnerable to adversarial attacks [5, 10] . The typical attacks, such as adding minor, carefully crafted perturbations to model inputs, can potentially cause misguided and incorrect decisions for DNNs, which is imperceptible to human recognitions. This poses a serious security threat to the practical application of deep models. To tackle the challenge, researchers have intensified efforts to develop defense methods to enhance the robustness of these models against adversarial attacks. Recently, Stochastic Neural Networks (SNNs) have demonstrated significant potential for model robustness by introducing uncertainty into model feature activations, such as in Variational Autoencoders [18] , or into model weights, as exemplified by Bayesian Neural Networks [13] . However, current implementations of SNNs predominantly utilize Gaussian distributions with constant variance to model this uncertainty and how the magnitude of the injected noise affects adversarial robustness tends to be ignored.However, we assume that this limitation restricts the potential to achieve higher levels of adversarial robustness. The intuitive motivation behind our approach is illustrated in Figure 1 , which conceptually represents how our noise injection method is designed to improve the model resistance against adversarial attacks.It is crucial to emphasize that the essence of adversarial robustness lies in minimizing the chances of a model misclassifying an input, especially when faced with adversarial perturbations. To correctly classify these perturbed instances, the model must adjust its internal parameters. Hence, we promote it to establish decision boundaries that are more robust to input variations. To be specific, at the beginning of the training, the model learns the primary patterns from the data. As training progresses and the noise intensity increases, the model becomes increasingly resilient to these perturbations. This ensures that foundational learning isn't overshadowed by the noise. In this way, the model leverages noise as a regularizer. As a result, it not only enhances both the model's robustness and generalization capabilities but also maintaining a stable and controlled training process. Benefited from these, the new boundaries exhibit increased resilience to adversarial perturbations. We have named this model the Maximizing Feature Distribution Variance Stochastic Neural Network (MFDV-SNN).In this work, our primary contribution is three-fold:• To the best of our knowledge, we are the first to explore how the magnitude of the injected noise affects adversarial robustness in SNNs. We conduct experiments on various white-and black-box attacks to demonstrate the effectiveness of the proposed MFDV-SNN compared to stochastic or non-stochastic defenses. 
related work 2.1 adversarial attack: 
 Researchers divide adversarial attacks into white-box and blackbox attacks according to whether they could obtain the gradient information.White-box attack: White-box attacks mean that the attacker knows the model gradient information. A simple yet effective whitebox attack method is called Fast Gradient Sign Method (FGSM) [10] , which adds a small perturbation in the direction of the sign of the gradient updates. It can be formulated asEQUATIONwhere ì 𝑥 denotes the input image, 𝜖 denotes the perturbation strength, L denotes the loss function and ℎ(•) denotes the target model. Kurakin further updated the one-step attack FGSM to a multi-step attack named Basic Iterative Method (BIM) [19] . Compared to FGSM, BIM uses a smaller step size to explore the possible adversarial direction. Madry further updates BIM by randomly initializing the input point, which is one of the strongest first-order attacks named Projected Gradient Descent (PGD) [23] . It can be formulated asEQUATIONwhere Π ì 𝑥+𝑠 is the projection operation that force the adversarial example in the ℓ 𝑝 ball 𝑠 around ì 𝑥, and 𝛼 is the step size. Another strong first-order attack algorithm is called C&W attack [5] which finds adversarial examples by solving the following optimization function formulated asEQUATION)where 𝑝 is the norm distance, commonly choosing from {0, [27] . Among them, zero-order optimization [6] algorithm aims to estimate the gradients of the target model directly.In this study, we use well-known white and black box attacks to evaluate our method, including FGSM, PGD 10 , C&W, PGD 100 , n-pixel attacks, and Auto attack [7] .To improve the model's robustness against unseen attacks. Stochastic defenses are proposed which mainly introduce stochastic noises into model weights/activations to simulate possible model parameters and the related probability distribution. Typically stochastic neural networks, such as Bayesian neural networks, introduce randomness into model weights and transform the parameter point estimates into distribution estimates. Unlike introducing randomness to model weights, methods such as Variational auto-encoders (VAE) [18] improve model robustness by introducing randomness to the model feature activations [32] . Our proposed method belongs to the latter, and similar defense methods are as follows.RSE [21] uses ensemble tricks to improve model robustness. Specifically, they inject standard Gaussian noise into multi-layers during training and then perform multiple forward passes to test it. It means they only need training once and can be considered an ensemble model. Adv-BNN [22] improves the robustness of the model by introducing a standard isotropic Gaussian noise prior to the model weights and further using adversarial training to find the best model distribution. However, the above methods are all use fixed noise parameters, and hence the uncertainty of the model is fixed. Parameter noise injection (PNI) [12] further propose learning a sensitive parameter to control the variance by adding trainable Gaussian noise to each layer of features or weights of the model. Moreover, L2P [16] updates PNI by introducing a perturbation injection module and alternating training the perturbation injection module and the neural network module, which they called "alternating back-propagation, " to improve the robustness of the model.Our method is based on a stochastic neural network framework and aims to explore the effect of noise injection on adversarial robustness, and the results of the study can be extended to existing stochastic neural network methods. 
methodology: 
 In this section, we detail the implementation of the proposed MFDV-SNN. The critical point is shown in Figure 2 . sampling the same dimension with the feature dimension from the uniform distribution. Then, we establish a Gaussian distribution 𝑧 and sample from it. Finally, we take the plus of the sample and the original hidden representation ℎ 𝑙+1 to get the logit of the network.In this section, we show that introducing additional stochastic noise into a deep neural network (DNN) can naturally create an effective regularization mechanism to counteract input perturbations.Consider a standard DNN used for supervised learning. For a single sample, the loss function L : R 𝑑 in → R is defined. The loss depends on model parameters 𝜃 = 𝑊 (ℓ ) , 𝑥 (ℓ ) , 𝑏 (ℓ ) |ℓ = 0, ..., 𝑁 𝐿 -1, where 𝑁 𝐿 is the number of layers, with weights, features, and biases of a given layer being𝑊 (ℓ ) ∈ R 𝑑 ℓ ×𝑑 ℓ +1 , 𝑥 (ℓ ) ∈ R 𝑑 ℓ , 𝑏 (ℓ ) ∈ R 𝑑 ℓ +1 , respectively. In each optimizer iteration, a mini-batch B includes a set of labeled examples, 𝒙 𝑖 , 𝒚 𝑖 𝑖 = 1 | B | ∈ R 𝑑in × R 𝑑 label .Adding stochastic noise (SN) in a given layer ℓ SN corresponds to a random scalar input, 𝜖 ∈ R. The batch average loss function with SN can be written as a series expansion of the noise shift parameter 𝜖𝑊 SN .Introducing noise injection in a specific layer ℓ SN transforms the activation to 𝑧 (ℓ SN ) + 𝑊 SN 𝜖. The single sample loss function for transformed activation is expressed asEQUATIONUsing the definition of the translation operator 𝑓 (𝑥 + 𝑎) = 𝑒 𝑎∇ 𝑓 (𝑥), we can explicitly compute the batch average loss functionEQUATIONHere, 𝐿(𝜃 ) is the loss function without any stochastic noise (SN), and R 𝑘 represents the batch average derivative of the loss function with respect to activations prior to the noise injection layer.EQUATIONThese functions are the products of moments of the injected noise, values of W SN , and derivatives of the loss function's activation without noise injection.Considering noise sampled from a zero-mean distribution, under this assumption, the first two terms simplify to the following formsR 1 = 𝑊 𝑇 SN • 𝜖𝑔 ℓ SN , R 2 = 1 2 𝑊 𝑇 SN 𝜖 2 H ℓ SN 𝑊 SN (7)Here, batch average is denoted by ⟨• • •⟩, andEQUATIONare the network-dependent local gradients and local Hessian, respectively.Under this condition, R 1 and R 2 act as follows: R1 induces constrained random walks on the norms of noise injection weights and on the data weights in layers ℓ > ℓ SN , with step sizes varying according to local gradients during training. On the other hand, R 2 can be understood as a direct regularization term on the local Hessian, striving to reduce its eigenvalues, thereby reducing local curvature between layers. These results imply that when noise is large, regularization through R 2 becomes the dominant effect, which is also a benefit of increasing stochastic noise.Next, we show why pushing the local Hessian towards smaller eigenvalues is beneficial for reducing sensitivity to noise perturbations.Consider a non-random network but with noisy input, represented as 𝑥 → 𝑥 + 𝛿, where 𝛿 is a random vector. Similar to the above, we can transform the pre-activation as [12] , Adv-BNN [22] and Learn2Perturb (L2P) [16] . Results show that the proposed MFDV-SNN is effective in training robust model and achieves state-of-the-art results.EQUATIONAdv-BNN L2P MFDV-SNN (Ours)Assuming the vector 𝛿 is sampled from the distribution N 0, 𝜎 2 𝛿 , similar to the aforementioned, the first two terms of the Taylor expansion can be expressed asEQUATIONIt can be seen that if the noise injection reduces H 0 , it will also correspondingly decrease the sensitivity to data perturbations. Therefore, in the mechanism of stochastic noise injection, when 𝜎 is initially small, R 1 >> R 2 , and at this point, noise injection mainly follows a random walk with small steps without significantly affecting the network's behavior. As the injected noise 𝜎 gradually increases, R 2 begins to surpass R 1 and dominates in the training process. During this phase, the model's robustness against adversarial defenses to input perturbations significantly increases.As mentioned above, we need to improve the model uncertainty during model training. In practice, we do not assign parameter variances directly because we do not know the exact uncertainty required by the model, which is closely related to the model architecture and dataset. Instead, we initialize the Gaussian variance with a non-informative standard uniform prior. When the network trains, the variance gradually increases in small steps. Also, the unbounded variance does not collapse because the gradient is -1 𝜎 , and once the variance is too large, it will not back-propagate the gradient information.Thus, the loss function can be formulated asEQUATIONwhere L 𝐶 is the cross-entropy loss, and D denotes the feature dimension of the penultimate layer. Furtherly, we adopt the ln ( ì 𝜎 𝑖 ) operation in practice, which will facilitates the computation of gradient derivatives, and slows down the numerical change of variance. The final 𝐿 2 regularization term penalizes over large weights. In addition, 𝜆 1 and 𝜆 2 control the power of variance and weight penalty, respectively. The algorithm is shown in supplementary material.  
experimental setting: 
 In this subsection, we detail the experimental setting, specifically, the attacks setting partically following the literature [12, 16] . the attacks strength 𝜖 of FGSM and PGD are set as 8/255. For PGD 10 attack, the steps 𝑘 is set as 10 and the 𝛼 is set as 𝜖/10. For C&W attack, the learning rate 𝛼 is set as 5 • 10 -4 , the number of iterations 𝑘 = 1000, initial constant 𝑐 = 10 -3 , and maximum binary steps 𝑏 𝑚𝑎𝑥 = 9. For the n-Pixel attack, the population size 𝑁 = 400 and maximum number 𝑘 𝑚𝑎𝑥 = 75 same as [16] , and we conduct a stronger 5-pixel attack which is not implemented in their setting, to validate the strong adversarial robustness of proposed method.Besides, we choose PGD 100 and Auto-Attack as stronger whiteand black-box attack methods which are not implemented in their experiments and apply different attack strength. Note that Auto-Attack contains both white-and black-box attacks. We evaluate them individually. For PGD 100 attack, we set 𝑘 = 100 and the 𝛼 is set as 𝜖/100. For Auto-Attack, we refer to the implementation from [7] . All experiments are performed on the Pytorch platform and partial Clean FGSM PGD Clean FGSM PGD AT [23] 85.5 52.5 43.9 58.0 25.0 20.5 Adv-BNN [22] 82.2 60.0 53.6 50.0 30.0 27.0 PNI [12] 87.2 58.1 49.4 61.0 27.0 22.0 L2P [16] 85.3 62.4 56.1 58.0 30.0 26.0 SE-SNN [33] 92.3 74. Baselines: The stochastic defense baselines are as follows. Note that all these stochastic defenses need adversarial training. No defense [11] : models without stochastic injection served as baseline. AT [23] : models with adversarial training. Adv-BNN [22] For Table 1 , we compare the proposed method with PNI, Adv-BNN, and L2P on various network sizes and capacities. The proposed MFDV-SNN consistently achieves state-of-the-art results on clean accuracy and robust accuracy.For Table 2 (Upper) C&W attack, we compare the common confidence level setting 𝑘 from [0,0.1,1,2,5]. The results show that the proposed MFDV-SNN significantly exceeds the stochastic defense methods even when the confidence level 𝑘 is 5. Specifically, compared to existed state-of-the-art defense PNI [12] and L2P [16] , we have about 6.2%, 4.6%, 14.1%,30.2% and 140% improvement, respectively.For Table 3 , we compare the state-of-the-art SNNs for FGSM and PGD 10 attacks on CIFAR-10 and CIFAR-100 datasets with a ResNet-18 backbone. Specifically, for the CIFAR-10 dataset, we have about 1.5%, 15.3%, and 41.9% improvement against prior state-of-theart stochastic defenses SE-SNN [33] and L2P [12] . For the CIFAR-100 dataset, which has richer classes than CIFAR-10, the proposed MFDV-SNN also achieves state-of-the-art results. Compared to existed defense methods IAAT [31] , L2P [12] and Adv-BNN [22] , we have about 8.6%, 57.0%, and 38.1% improvement, respectively.For Table 4 , we evaluate the proposed MFDV-SNN on stronger attacks, PGD 100 attack, and Auto-Attack (white-box). We divide Auto-Attack into white-and black-box to better present our results. The results demonstrate that the proposed MFDV-SNN maintains great performance even on the stronger attack. Black-box Attacks Results: The black-box attacks include n-Pixel attack and Auto-Attack (black-box), which termed as Square attack, as reported in Table 2 (Bottom) and Table 4 . For n-Pixel attack, which relies on evolutionary optimization and is derivativefree. The attack strength is controlled by the number of pixels it comprises. From the results in Table 2 (Bottom), we can see that the proposed MFDV-SNN outperforms other state-of-the-art methods in all attack strengths. Specifically, for 1-, 2-, 3-pixels attacks, compared with the best defense method Adv-BNN, we have improvements of about 24.5%, 24.5%, and 27.3%, respectively. Furtherly, we add a stronger 5-pixel attack to evaluate the proposed MFDV-SNN's efficiency. Moreover, Auto-Attack (black-box) results also confirm the strong adversarial robustness of MFDV-SNN against black-box attacks.Overall, various white-and black-box attacks show that the proposed MFDV-SNN is strong enough. It is worth emphasizing that the proposed MFDV-SNN does not need adversarial training, while the other state-of-the-art defense methods all need adversarial Backbone AT Clean PGD PGD AT [23] ResNet-20 (4x) ✓ 87.0 46.1 RSE [21] ResNext × 87.5 40.0 DP [20] 28-10 Wide ResNet × 87.0 25.0 TRADES [34] ResNet-18 ✓ 84.9 56.6 PCL [24] ResNet-110 ✓ 91.9 46.7 PNI [12] ResNet-20 (4x) ✓ 87.7 49.1 Adv-BNN [22] VGG We compare MFDV-SNN with state-of-the-art defense methods proposed in recent years. Among these defense methods, some methods are SNNs, and some are not. We present the results in Table 5 . All experiments are conducted on CIFAR-10 with the PGD attack. AT means using adversarial training. The results show that many defense methods need adversarial training, which requires a high computational cost. The proposed MFDV-SNN outperforms defense methods listed in Table 5 , even with deeper network architectures, and achieves the highest accuracy on clean data and strong adversarial robustness. 
additional analysis 5.1 inspection of gradient obfuscation: 
 Athalye et al. [2] claimed that some stochastic algorithms are false defense methods. These methods mainly obfuscate the gradient information to improve the model's robustness, and they propose a checklist to identify whether a defense is of an obfuscation gradient. Once the defense belongs to gradient obfuscation, it can be attacked by the proposed EOT attack [3] . In this section, we thoroughly inspect whether the proposed method belongs to gradient obfuscation by the two-stage inspection. In the first stage, we conduct experiments to check the gradient obfuscation list proposed by Athalye [2] , and experiment details follow [16] . In the second stage, we directly evaluate our method by the EOT attack.IRU3*'DWWDFNZKHQN step 7HVW$FF 0)'96112XUV 1XPEHURI$WWDFN6WHSN step ZKHQ 7HVW$FF 0)'96112XUVFirst Stage Inspection: Checklist Criterion 1: One-step attack performs better than iterative attacks. Refutation: The PGD attack is an iterative attack, and FGSM is a one-step attack. From Table 1 , we can see that the accuracy of MFDV-SNN against FGSM attack is consistently higher than that of PGD attack. Criterion 2: Black-box attacks are better than white-box attacks.Refutation: With the development of adversarial attack methods, there are strong white-box attacks and black-box attacks, making it difficult to have a common result. Nevertheless, we can partially confirm in Table 4 that stronger black-box attacks perform worse than strong white-box attacks. Criterion 3: Unbounded attacks do not reach 100% success. Refutation: Following [12] , as drawn in Figure 3 , we run experiment by increasing the distortion bound-𝜖. The results show that the unbounded attack leads to 0% accuracy. Criterion 4: Random sampling finds adversarial examples. Refutation: Following [12] , the prerequisite is that the gradientbased (e.g., PGD and FGSM) attack cannot find adversarial examples. However, Figure 3 indicates that when we increase the distortion bound, our method can still be broken. Criterion 5: Increasing the distortion bound does not increase the success rate.Refutation: The experiment in Figure 3 shows that increasing the distortion bound improves the attack success rate.Following [16, 25] , we use a Monto Carlo method which expects the gradient over 80 simulations of different transformations on the CIFAR-10 dataset with the backbone ResNet-18. Experiment Moreover, we adopt 15 MC sampling expectations of the Gaussian noise in the test phase, verified in the experiments to be stable enough to ensure the experiment results are not stochastic gradient. Overall, the proposed method passes the two-stage inspection of gradient obfuscation, ensuring that the proposed MFDV-SNN is not of gradient obfuscation.We conduct experiments on different datasets and network architectures to evaluate the proposed MFDV-SNN generalization ability. The results are reported in Table 6 and Table 7 . In Table 6 , we mainly explore the datasets' influence on the proposed MFDV-SNN. Six different datasets are used in this experiment. The experiments are based on the backbone ResNet-18, except for the MNIST dataset.We evaluate MNIST dataset performance on the LeNet architecture. The result shows that MFDV-SNN has a great generalization to different datasets. In Table 7 , we mainly explore the impact of the network architectures on the proposed MFDV-SNN. Specifically, we adopt various architectures, including VGG, GoogLeNet, DenseNet, and ResNets. The results demonstrate that MFDV-SNN generalizes well on various network architectures. We also confirm that MFDV-SNN generalizes well on different network widths and depths in Table 1 .Overall, we explore the generalization of the proposed MFDV-SNN from two perspectives: network architectures and datasets. For network architectures, we further explore the effect of different types of networks and the effect of different widths and depths. Both results show the proposed MFDV-SNN generalizes well to the mainstream DNN models. 8 . Specifically, we report the average epoch time consumption as the evaluation metric. We have the following observations: 1) Our proposed MFDV-SNN is computationally inexpensive and close to standard defense-free training. We can observe this phenomenon in all the comparison experiments. 2) Compared to AT, we save computation time from 6.5 to 7.5 times, which makes sense since we do not need to generate adversarial examples. 3) Although different architectures are compared, the basic rules are almost identical. As the network structure becomes more complex, the computation time increases proportionally. i.e., From ResNet-18 to ResNet-50 to WRN-34-10, the complexity of the model increases sequentially, and the WB-SNN computation time increases accordingly. However, the ratio of adversarial training to our proposed WB-SNN method varies relatively steadily.Effectiveness of regularization loss. We conduct experiments to evaluate the effectiveness of the proposed regularization module on the FGSM attack and PGD attack trained on the CIFAR-10 The proposed loss function is effective; it shows that MFDV outperforms the Gaussian distribution by a large margin, while the only difference in this experiment is the proposed regularization loss.Where to apply stochastic layer? We build a shallow and a deep model to explore the location to apply stochastic layers. The shallow model consists of two Conv and three linear layers, in which we implement the stochastic layer to different layers. The model robustness is evaluated on the FGSM attack. For notation, the linear1-3 denoted different layers before the last classifier layer. For example, linear1 means we implement the penultimate layer as a stochastic layer, linear2 means we implement the third layer from the classifier layer as a stochastic layer, linear12 means we implement both two layers as stochastic layers, and so forth. The result is shown in Table 9 . Furtherly, we evaluate on a deeper model ResNet-20 [4x] under PGD attack, in which linear12(1) means we perform regularization at the penultimate layer, and linear12( 12) means we perform regularization on both layers. We have the following observations. 1) Applying stochastic layer to multiple high-level layers generally achieves a better performance, for instance, in shallow model, linear12 and linear13 are better than linear1, and linear123 achieves the best results. 2) Applying stochastic layer to the penultimate layer is the key to achieving strong adversarial robustness, for instance, in shallow model, linear2, linear3 and linear23 are worse than linear1, but linear12, linear13, and linear123 perform well. This makes sense because linear1 is the closest to model prediction and tends to capture label-related information. 3) Regularization on multilayers is not suitable for large network, i.e., in ResNet-20[4x] model, linear12(1) achieves the highest robust accuracy, linear12 (12) is worse than linear12(1) and linear1. 
conclusion: 
 In this paper, we have theoretically demonstrated that by gradually increasing the level of stochastic noise within a deep neural network, the model inherently enhances its resistance to input perturbations. This principle guided the development of our MFDV-SNN framework, which efficiently integrates noise intensity into the model training process. Our comprehensive experiments validate the superior adversarial robustness of MFDV-SNN against established white-box and black-box attacks. Notably, the MFDV-SNN framework achieves this robustness without depending on adversarial training methods and maintains high accuracy on clean data inputs. Furthermore, the computational overhead introduced by our method is minimal, closely aligning with the costs of standard model training which underscores its practicality and effectiveness.  
 
 Appendix:  


Decision: This work is appreciated by the reviewers in terms of the technical novelty and the performance improvement comparing with the state-of-the-art methods. Some concerns are related to a theoretical proof of the proposed method, which requires a more detailed explanation to be provided in the camera ready. Thus, I recommend the acceptance as a poster work.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel method, MFDV-SNN, that improves the adversarial robustness of DNNs by gradually increasing the stochastic noise during training. This method does not require adversarial training, thus preserving the model's accuracy on clean data and significantly reducing computation time. The effectiveness of MFDV-SNN is validated through extensive experiments against various adversarial attacks.
strength_weakness: strength_weakness: The introduction of stochastic noise to maximize feature distribution variance is a novel method that enhances adversarial robustness.
It provides theoretical insights into how the stochastic noise injection enhances robustness and avoids the computationally expensive adversarial training, making it more efficient while maintaining clean data accuracy. Comprehensive experiments on multiple datasets and various attacks demonstrate the effectiveness of MFDV-SNN. Gradient obfucation is also inspected.
limitations: limitations: 1. **Insufficient Evaluation**: The evaluation using AutoAttack is only compared to the no-defense baseline. The improvement over other baseline defenses is difficult to assess, as PGD and FGSM are relatively weaker attacks and both suffer from the gradient vanishing issue.
2. **Outdated Baseline**: To demonstrate superiority, the baseline should be updated. For instance, the selected adversarial training method is outdated and simplistic. Many recent adversarial training methods focus on improving training efficiency[a,b].
[a] Adversarial Training for Free!
[b] Fast is Better than Free: Revisiting Adversarial Training.
paper_summary: paper_summary: This paper theoretically shows that by gradually increasing the level of stochastic noise in a DNN, the model naturally becomes
more resistant to input perturbations. The proposed method does not require adversarial training and does not sacrifice clean data accuracy.
strength_weakness: strength_weakness: 1. Impressively, I have carefully checked the theoretical derivation and it's indeed convinced me from a theoretical standpoint. 
2. The method surpasses the performance of current state-of-the-art (SOTA) methods, showcasing its potential. 
2. The proposed method is underpinned by rigorous mathematical proofs, adding to its credibility.
3. The paper is commendably articulated and easy to follow.
4. This work commits the source code and increases its repeatability.
limitations: limitations: 1. The theoretical derivation of this paper is confusing with some symbols, and the meaning of some symbols is unclear, i.e. $𝑊_{SN}$, $\epsilon𝑊_{SN}$, $ 𝑊_{SN} \epsilon$. I recommend that the author clarify the theoretical section; nevertheless, this does not detract from my view that it represents a significant and valuable contribution. 
2. For what I want to point out specifically,  I suggest the author add some background on **translation operator** as it is a critical point in the derivation and may take a lot of extra time to understand the principles behind this if the reader does not have a relevant background in advanced linear algebra. 
2. Ablation studies on $ln(\sigma)$ and $L_{2}$ regularization penalty term in Eq.11 should be further conducted as  $L_{2}$ regularization can also enhance the robustness of the model in principle. This can help us understand the true robustness gained from **stochastic noise (SN)** introduced in the training phase.

Overall, after carefully checking the theoretical derivation of this paper, I believe that this paper demonstrates its superiority both theoretically and experimentally. It is a work with a clear motivation, and coherent theory, and achieves new SOTA performance.
paper_summary: paper_summary: This paper proposes a method to improve model adversarial robustness, by introducing random Gaussian noise into extracted features during training and gradually increasing the noise magnitude, which controls the uncertainty injected into the training process. The paper has evaluated the proposed method with different deep neural network architectures, datasets, and several representative adversarial attacking methods.
strength_weakness: strength_weakness: 1.The proposed method is relatively easy to implement while providing desirable adversarial robustness, as shown in the experiments.

2.The proposed method has been evaluated from multiple perspectives, crossing datasets and models with different scales, and adversarial attacks with different attacking strengths.
limitations: limitations: 1. The proposed method is not well presented. In Figure 2, it appears that the noise is injected before reaching to part for computing logits, while the analysis in section 3.2 implies that the noise injection can be applied to an arbitrary layer. Moreover, the analysis of noise injection as a regularization approach is not well presented. First of all, the underlying assumption for applying the translation operator is not specified, implying that the analysis can be applied to different layers, including layers with linear activation and various non-linear activation.

2. While there are many diverse evaluation perspectives considered in this paper, the evaluation, however, is not comprehensive enough to support the claimed performance improvement. Specifically, 
(1) although one of the major contributions is that the proposed method does not require adversarial training, the experiment results on comparing the computation time required for adopting the proposed method and adversarial training do not sufficiently convince the reviewer about this claim. Specifically, to adopt the proposed method, a model needs to be trained from the very beginning with all data (the same as training a model with no defense method adopted), while for adversarial training, a previously stored checkpoint can be adopted and trained only with augmented adversarial examples. Also, it is not clear how is the computation time calculated, what is the detailed adversarial training process? How many adversarial examples are used for training? Are all models trained to obtain similar performance, or they may converge with different performances? 

(2) While FGSM, PGD, C&W, and EOT attacks are mentioned in this paper, only the first two are mostly evaluated, C&W is occasionally evaluated, and EOT is only used for inspecting if the proposed method obfuscates gradients.

(3) The selection of models for evaluation appears to be quite arbitrary, for example, for evaluating computation time, a WRN-34-10 model is used yet has never been mentioned before, while previously evaluated googlenet, vgg19, and other models are ignored.

(4) As shown in Tables 2 and 4, the proposed MFDV method performs better than No Defense on clean data, it would be better if more discussion or analysis for these results could be provided, as this phenomenon has not been observed for other stochastic methods. 

(5) The evaluation results shown in Table 5 cannot well justify that the proposed method outperforms SOTA defense methods, as the models adopted for comparison do not even have the same architecture.