Watch Out!! Your Confidence Might be a Reason for Vulnerability

Authors: Ayush Pandey, Akshay Agarwal
Primary area: interpretability and explainable AI
Keywords: Confidence, Robustness, Natural Adversaries, Object Recognition

Abstract:
The tremendous success of deep neural networks (DNNs) in solving `any' complex computer vision task leaves no stone unturned for their deployment in the physical world. However, the concerns arise when natural adversarial corruptions might perturb the physical world in unconstrained images. It is widely known that these corruptions are inherently present in the environment and can fool DNNs. While the literature aims to provide safety to DNNs against these natural corruptions they have developed two forms of defenses: (i) detection of corrupted images and (ii) mitigation of corruptions. So far, very little work has been done to understand the reason behind the vulnerabilities of DNNs against such corruption. We assert that network confidence is an essential component and ask whether the higher it is, the better the decision of a network is or not. Moreover, we ask the question of whether this confidence itself is a reason for their vulnerability against corruption. We extensively study the correlation between the confidence of a model and its robustness in handling corruption. Through extensive experimental evaluation using multiple datasets and models, we found a significant connection between the confidence and robustness of a network.

Decision: reject
Reviewer scores: 3, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730253841208

thread_type:
official_review

<thread_object>
creator: reviewer_qqpm

<content>
summary:
This paper investigates the correlation between the confidence of deep neural networks and their vulnerability to natural corruptions. Specifically, the authors leverage the model calibration method SWAG to construct a smoothed model. The parameter of this model is sampled and averaged from the estimated Gaussian distribution of several versions of model parameters recorded during training. The evaluation on the widely used natural corruption benchmark CIFAR-10-C for VGGNet and PreActResNet has shown the robustness of the smoothed model against natural corruptions.

strengths:
- **Novel insight**: this paper is the first to leverage model calibration method to mitigate natural corruptions. 
- **Promising experiment results**: the leveraged SWAG method substantially improves the robustness of CNNs against natural corruptions. 
- **Well-written paper**: the paper is well-organized and easy to follow.

weaknesses:
- **Limited technical contribution**: the methodology in Section 3 is originally proposed in SWA and SWAG. This paper has not introduced further adjustment or improvement when applying the method to mitigating natural corruptions. 
- **Lacking theoretical analysis**: the experiments have shown the effectiveness of SWAG in improving the robustness against corruptions. However, no theoretical analysis is provided to help better understand the source of the robustness.

questions:
- For Figure 1, how is the reliability plot plotted? Specifically, which hyper-parameter is adjusted to control the confidence of the model, and how is it adjusted? 
- How does the proposed method perform on larger datasets like ImageNet-C? Can this method survive different set of natural corruptions other than those in CIFAR-10-C, e.g., ImageNet-P, ImageNet-$\bar{C}$ [1]? 



[1] On interaction between augmentations and corruptions in natural corruption robustness. NeurIPS 2021.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730602668456

thread_type:
official_review

<thread_object>
creator: reviewer_obon

<content>
summary:
This paper investigates the vulnerability of deep neural networks (DNNs) when facing natural corruptions (such as noise, blur, etc.) and proposes that the model's confidence could be an important factor contributing to this vulnerability. Experiments demonstrate a significant correlation between a model’s confidence and its robustness in handling corruption. The study primarily focuses on calibrating model confidence and employs the Stochastic Weight Averaging Gaussian (SWAG) method to enhance model robustness.

strengths:
Proposing that high confidence might lead to model vulnerability in naturally corrupted environments is a novel perspective, differing from traditional defense methods.

weaknesses:
1. This paper only conducts experiments on convolutional neural networks (CNNs), lacking tests on other network architectures such as ViT-B/16 or DeiT, to validate the conclusions about confidence and robustness across different model types. This would provide a more comprehensive demonstration of the method's applicability and effectiveness.

2. This paper validates the robustness of the model to natural corruptions and its relationship with confidence using CIFAR-10 and CIFAR-100 datasets. However, the complexity of these datasets is relatively low, making it difficult to fully reflect the model's performance in real-world complex scenarios. It is recommended to conduct further experiments on more challenging datasets such as ImageNet-C or ImageNet-A, which include a broader range of corruptions (e.g., Gaussian noise, motion blur, weather-induced degradation, digital transformations) and better reflect the diversity and complexity of real-world applications. 

3. The paper mainly focuses on confidence calibration without an in-depth comparison with other advanced defense methods (such as adversarial training), which may weaken the practical applicability of this approach.

questions:
Although the proposed method addresses natural corruption, its effectiveness against gradient-based adversarial attacks remains unclear. It is recommended that the authors conduct experiments involving FGSM, PGD, and C&W attacks to evaluate the method's performance under adversarial attacks. For example, performance under different noise magnitudes and different numbers of attack iterations could be assessed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613622979

thread_type:
official_review

<thread_object>
creator: reviewer_gzh2

<content>
summary:
The paper explores the challenges DNNs face from natural adversarial corruptions, which can undermine their robustness. While past work has focused on detecting and mitigating these corruptions, this study examines whether a model’s confidence may contribute to its vulnerability.

strengths:
1. It explores the vulnerability of DNNs from the perspective of model overconfidence.
2. The article is well-structured and relatively clear in its presentation.

weaknesses:
1. What exactly is the novelty of this paper? SWAG is not your contribution; merely using it to derive some results for analysis does not suffice.
2. The phenomenon of model overconfidence appears to be only a description in your paper. Do you have specific examples or experimental results to substantiate this claim?
3. Is your method limited to CNN architectures? Given the prevalence of transformer-based models, a method solely applicable to CNNs may have limited relevance, and it appears you tested on a very small set of CNN models.
4. Based solely on the text, I cannot appreciate the superiority of your method. Please provide comparative experiments with adversarial training methods, covering dimensions such as effectiveness and cost. Furthermore, does your method apply only to natural corruptions? How would it perform against adversarial samples?
5. The experiments lack depth: (1) In terms of models, this paper tests only on VGG-16 and ResNet, which seems rather limited. Where are the tests on more advanced models? (2) In terms of datasets, you only used CIFAR-10 and CIFAR-100, yet experiments on ImageNet are also necessary.

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730767859140

thread_type:
official_review

<thread_object>
creator: reviewer_iz8w

<content>
summary:
This paper proposes using Stochastic Weight Averaging Gaussian (SWAG) as a method for calibrating neural networks, aiming to improve their performance and robustness against natural corruptions. The approach leverages SWAG's capacity to model uncertainty and enhance prediction reliability, asserting that better-calibrated confidence scores contribute to robustness in challenging real-world conditions.

strengths:
The paper presents a detailed investigation into the impact of calibration on model robustness, especially under naturally occurring corruptions. By systematically exploring the role of confidence in model predictions, the authors contribute to understanding the relationship between calibration and robustness, reinforcing SWAG’s potential in addressing natural corruption without introducing additional computational burden associated with adversarial training.

weaknesses:
1. Limited Novelty: The paper largely relies on the established SWAG technique without introducing new calibration methods or adaptations specific to the architecture or the problem of natural corruption.

2. Experimental Scope: The experimental evaluations are confined to small datasets (CIFAR-10 and CIFAR-100) and convolutional architectures like VGG and ResNet, lacking analysis on larger-scale datasets and modern architectures like Transformers.

questions:
1. Applicability Across Architectures: The proposed method seems tailored primarily for convolutional neural networks (CNNs). A major gap lies in assessing how well SWAG might generalize to other architectures, such as Transformers, which have become prevalent in vision tasks. Expanding the discussion on generalizability or including Transformers in the experimental setup could enhance the study's relevance and adaptability to current deep learning trends.

2. Novelty in Approach: While the study reinforces known concepts around calibration and robustness, these insights are not novel, particularly within the Bayesian deep learning community, where calibration’s role in improving robustness under adversarial scenarios is well-understood [refA, refB]. This limits the paper's contribution, as it primarily confirms existing knowledge rather than pushing the boundaries with a novel calibration approach. Introducing an innovative calibration technique, or a modified variant of SWAG tailored for robustness, would provide a more substantial contribution.

3. Experimental Limitations: The experiments focus on CIFAR datasets and CNNs, which are both limited in size and scope. A broader evaluation involving larger datasets like ImageNet, and a wider range of architectures, including Transformer-based models, would offer a stronger validation of SWAG's effectiveness. This could also strengthen the paper’s generalizability claims and its relevance for real-world deployment.

References

[refA] Wicker, Matthew, et al. "Bayesian inference with certifiable adversarial robustness." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[refB] Stutz, David, Matthias Hein, and Bernt Schiele. "Confidence-calibrated adversarial training: Generalizing to unseen attacks." International Conference on Machine Learning. PMLR, 2020.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732973781836

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Responses and Update in Manuscript

comment:
First, we want to thank each reviewer for providing constructive comments and highlighting that this is the first work aiming to provide an explainability in understanding the vulnerability of deep models against natural (common) corruption. It is observed from the literature that several efforts have been started to mitigate the impact of corruption or detect the images as clean or corrupted but no research effort has been made to highlight the reason for its sensitivity in the first place itself. We assert that such interpretability can help build a better and more robust model than later developing one model (extra, and incur additional cost, probably heavy to deploy on computationally limited devices) that can mitigate the impact of corruption or detect them which will further require the purification of corruption.

To further strengthen our findings, we have performed experiments with large-scale ImageNet corruption datasets, adversarial noises, and analysis with different training optimizers.

**Updates in Manuscript:** The modifications can be found on pages 8-9 under section 4.1.4 and in Figures 4 and 5.

**Results on ImageNet-C:**

To address the need for evaluating the proposed method on a broader and more challenging benchmark, we have included experiments on the ImageNet-C dataset. These results align with our observations on CIFAR-10 and CIFAR-100, further demonstrating that SGD-trained models exhibit significant overconfidence when exposed to natural corruption. Additionally, SWAG-trained models consistently outperform SGD in terms of accuracy and robustness under these corruptions, reinforcing the generality of our findings across diverse datasets.

**Analysis with Different Optimizers:**

In response to the suggestion to explore the impact of different optimizers, we extended our experiments to include Adam, alongside SGD. The results reveal that while SGD and ADAM optimizers show varying degrees of overconfidence under natural corruptions, the patterns of improvement achieved by SWAG remain consistent across these optimizers. This highlights the adaptability of SWAG in addressing overconfidence issues, regardless of the underlying optimizer.

**Analysis with different adversarial perturbations are also provided as the responses to individual reviewers who asked for such observations.**

**Results on Transformers** Our preliminary findings reveals that the transformer models are equally vulnerable to common corruption and overconfidence through SGD is a primary reason. To verify that, have performed experiments with SWAG and observed significant reductions in overconfidence and boost in classification performed on each dataset including ImageNet-C.

**Comparison with Adversarial Training (AT):** AT is one of the strongest defense against adversarial perturbation; however, its effectiveness against corruptions are not adequately studied. Although, based on the suggestion of the reviewers, we have performed the comparison of the proposed work with AT. The comparison can be performed on atleast three following perspectives: (i) computational cost, (ii) accuracy on clean images, and (iii) handling of corruptions. The proposed SWAG model is found computationally lighter as compared to AT in terma of training time. For example, the PGD AT model on the CIFAR-10 dataset took approximately 250 minutes; whereas, the computational time of SWAG is 170 minutes on the similar GPU machine. Further, as well known, the AT show significant reduction in clean accuracy. The proposed model not only maintain clean accuracy but even improve it as compared to traditional SGD trained models. The similar effectiveness and strength can be seen in handling corruption, where, the accuracy of the proposed model is atleast 15\% better than the PGD AT model.

**Since now the upload of the revised pdf is not possible, we aim to add all these new findings in the camera ready paper**. We are hopeful that we addressed all the comments of the reviewers and hence looking forward for an updated rating and *acceptance of such a critical work which can pave a way in developing a secure deep learning era of models*. We will be happy to address any remaining comments. Thanks
</content>
</thread_object>

</thread>
<thread>
creation date:
1733124984874

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Awaiting Reviewer's Acknowledgement

comment:
We want to again thank each reviewer for providing the valuable feedback which significantly improved our finding and can pave a way in developing robust system which can be deployed in the real world.

We are actively looking forward to hearing back from the reviewers and meta reviewers in acknowledging our detailed response to each comment raised and would be happy to resolve any remaining comments.

Thanks
</content>
</thread_object>

</thread>
<thread>
creation date:
1733221927772

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Adversarial Attack Analysis and Novel Extensions

comment:
Thanks again for your comments. As mentioned earlier and asserted that the proposed approach can provide defense against adversarial attacks as well, we have extensive experiments with different adversarial parameters. The results of two popular adversarial attacks under varying perturbation norms are reported below.

Table 1: Accuracy with different perturbation values under PGD attack using CIFAR-10 dataset and VGG network.

| Epsilon (Max Perturbation) | SWAG    | SGD    |
|----------------------------|---------|--------|
| 1/255                      | **75.68**%  | 45.21% |
| 2/255                      | 72.50%  | 39.50% |
| 3/255                      | 68.74%  | 34.89% |
| 4/255                      | 62.08%  | 29.08% |
| 5/255                      | 57.56%  | 24.37% |
| 6/255                      | 52.67%  | 19.12% |
| 7/255                      | 46.87%  | 17.54% |
| 8/255                      | **41.56**%  | 14.24% |

Table 2: Accuracy with different perturbation values under FGSM attack using CIFAR-10 dataset and VGG network.

| Epsilon (Max Perturbation) | SWAG    | SGD    |
|----------------------------|---------|--------|
| 1/255                      | **61.24**%  | 46.78% |
| 2/255                      | 55.89%  | 41.24% |
| 3/255                      | 51.24%  | 37.89% |
| 4/255                      | 47.89%  | 35.29% |
| 5/255                      | 44.36%  | 32.56% |
| 6/255                      | 41.15%  | 30.21% |
| 7/255                      | 38.19%  | 29.43% |
| 8/255                      | **35.24**%  | 27.17% |

**From the results it can be observed that the SWAG model is not only effective in handling common corruptions but also the adversarial perturbation with a significantly higher margin than SGD.** We believe such universality and extensive analysis can help in building a **universal** defense architecture.

---------------------------------------
**Extension and Novelty:** We have already begun to expand and enhance the methodology to address the limitation of SWAG. One key limitation of SWAG, which we acknowledge, is its unimodal nature. By approximating the posterior distribution as a single Gaussian, SWAG inherently struggles to capture more complex uncertainty landscapes, particularly in cases where the true weight distribution may exhibit multimodality or other non-Gaussian characteristics. Recognizing this constraint, we are actively working on extending the framework to incorporate more flexible and expressive distributions.

Specifically, we are exploring the use of mixtures of Gaussians to model multimodal posterior distributions. This extension allows the framework to represent multiple modes in the uncertainty space, thereby providing a richer understanding of model uncertainty. Sampling from this mixture of Gaussians implicitly generates an ensemble of models, each corresponding to weights from different modes of the loss landscape, which will improve the generalization. 

**Our preliminary results using the VGG model on the CIFAR-10 dataset showcase the improvement of at least 5\% across corruptions.** As said, we are actively working in this direction to improve SWAG and are committed to adding these findings to the camera-ready paper.

We believe that these extensions will significantly improve the ability of the framework to address overconfidence and provide a more robust approach to uncertainty modeling.

**We hope all these new results address the concerns of the reviewers and look forward to the upgrade to the rating of the paper.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper focused on studying the underlying reason behind the vulnerabilities of deep neural networks to adversarial corruptions. It found that model confidence could be a key factor. Based on the observation, the paper further proposed using Stochastic Weight Averaging Gaussian (SWAG) for DNN calibration. The experiments on multiple datasets prove the effectiveness of the method.

Different from prior work that tried to detect corrupted images or mitigate corruptions, this paper tried to understand the vulnerabilities and developed a method based on the motivation. Drawing the connection between model confidence and corruption robustness is novel and provides new insights. The paper is well-written. However, there are few weaknesses identified by the reviewers. First, the technical contribution is not enough, as the paper simply applies the Stochastic Weight Averaging Gaussian (SWAG) with some necessary modifications. Second, the initial paper lacks in extensive experiments on larger dataset (e.g., ImageNet) and models (Transformers). The authors made an effort to provide more experiments on larger datasets, more attacks, and new architectures. There are still some concerns about the novelty and comprehensiveness. 

After thorough discussions, the reviewers reached a consensus that the paper needs further improvements to address the technical contribution. Therefore, the AC considers that the paper falls short of the ICLR acceptance threshold and recommends rejection.