Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness

Authors: Erh-Chung Chen, Pin-Yu Chen, I-Hsin Chung, Che-Rung Lee
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Adversarial training, AI safety, Lipschitz Continuity

Abstract:
The security and robustness of deep neural networks (DNNs) have become increasingly concerning. This paper aims to provide both a theoretical foundation and a practical solution to ensure the reliability of DNNs. We explore the concept of Lipschitz continuity to certify the robustness of DNNs against adversarial attacks, which aim to mislead the network with adding imperceptible perturbations into inputs. We propose a novel algorithm that remaps the input domain into a constrained range, reducing the Lipschitz constant and potentially enhancing robustness. Unlike existing adversarially trained models, where robustness is enhanced by introducing additional examples from other datasets or generative models, our method is almost cost-free as it can be integrated with existing models without requiring re-training. Experimental results demonstrate the generalizability of our method, as it can be combined with various models and achieve enhancements in robustness. Furthermore, our method achieves the best robust accuracy for CIFAR10 and CIFAR100 datasets on the RobustBench leaderboard.

Decision: reject
Reviewer scores: 5, 5, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730375834118

thread_type:
official_review

<thread_object>
creator: reviewer_erqp

<content>
summary:
This work aims to enhance the adversarial robustness of DNNs by reducing the Lipschitz constant. Unlike existing adversarially trained models, this method is a cost-free method to enhance adversarial robustness by introducing a forged function in the DNNs. Experimental results on CIFAR10, CIFAR100, and ImageNet datasets have demonstrated the proposed method outperforms the adversarial training method.

strengths:
1. The idea of enhancing the adversarial robustness by reducing the Lipschitz constant is reasonable and straightforward.

2. This paper is well structured.

weaknesses:
1. The concept of utilizing the Lipschitz constant to certify the adversarial robustness of DNNs has been explored in many previous works [1-3]. This Lipschitz constant is utilized to provide a provable adversarial robustness. However, this paper does not discuss the main differences and improvements from those provable adversarial defense methods.

2. The proposed forged function seems to be a variation of ReLU, just with a different threshold. Additionally, as illustrated in Figure 2, the Forged function is concatenated with ReLU. How does this differ from simply applying a shifted ReLU?

3. Given Equation (3), if the model becomes larger and more complex, would this increase the Lipschitz constant, potentially worsening the model's robustness?


[1] Certified adversarial robustness via randomized smoothing, In ICML2019
[2] Provably robust deep learning via adversarially trained smoothed classifiers, In Neurips 2019
[3] Scaling provable adversarial defenses, In Neurips 2018.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730440905264

thread_type:
official_review

<thread_object>
creator: reviewer_q4ed

<content>
summary:
The paper proposes a model agnostic method for improving the robustness of the network. Essentially, the method prune the activation of small values of the pretrained models. The pruning threshold is determined by the data and a hyperparameter. The paper performs a series experiment on different kinds of adversarially-trained models and shows that the proposed method can improve the robust accuracy by about 0.5%-2% while keeping the accuracy on the natural images. The paper discuss the possibility of gradient masking. However, the experiments cannot fully eliminate such possibility.

strengths:
1. The paper proposes a model agnostic method that can be easily applied to various robust models and improve the robustness of these method. Such methods have good potential for wide applications.

2. While the proposed method is simple, the pruning technique shows to improve the robust accuracy of various models. Considering the newly introduced hyperparameters are limited, the improvement is far above the margin of errors.

weaknesses:
1. While the paper discuss the possibility of gradient masking, the experiment is not enough to completely rule out such possibilities. Specially, the paper uses forging function to cut specific activation. Therefore, attacking methods like BPDA should be combined into AutoAttack and the robust accuracy under such attacks should be reported.

2. The motivation of the paper is not fully elaborated in the experiment. For example, whether the empirical Lipschitz has truly been lowered by such method should be reported.

3. Despite the simplicity the method, the novelty of  the proposed method is limited.

questions:
1. As $c^r$ is very small, I wonder the proportion of activation that be pruned in the experiment. 

2. As the proposed method change the activation even in the natural images, why the natural accuracy is improved in some cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622433627

thread_type:
official_review

<thread_object>
creator: reviewer_exhp

<content>
summary:
The author proposes a plug-play method to enhance the adversarial robustness of deep neural networks by focusing on Lipschitz continuity. This work reduces the Lipschitz constant by remapping the input domain to a constrained range, thus improving robustness without requiring retraining. This approach is almost cost-free, and experimental results combined with the previous work show the effectiveness of the proposed method.

strengths:
1.	The method is highly cost-effective as it does not require model retraining or additional data. Moreover, due to its plug-and-play nature, the proposed method can be easily integrated with existing algorithms. 
2.	The authors provide detailed theoretical insights into the relationship between the Lipschitz constant and adversarial robustness. 
3.	The paper is well-organized and clearly presented, with extensive experiments across diverse datasets that validate the robustness of the proposed method.

weaknesses:
1.	The experimental results, along with the authors' own analysis, indicate that the proposed method may reduce the classification accuracy on clean samples in some cases. While the authors discuss possible reasons, the paper could further analyze this trade-off and explore strategies to mitigate accuracy loss. 
2.	The choice of parameter $c^r$ is crucial to performance, but finding a single $c^r$ that performs well across all tasks is challenging based on the results presented. For example, $c^r=2^{-6}$ performs well on most tasks, while $c^r=2^{-7}$ works better for RST-AWP in Table 4. Adjusting this parameter for new tasks could be time-consuming.

questions:
1.	In the evaluation using $acc_{AA}$, are the adversarial examples generated using the original neural network architecture, or using the architecture with the proposed forged function?
2.	Can this method be extended to other tasks, such as NLP applications? Alternatively, could the proposed forged function improve the generalization ability of the model in standard training?
3.	Could the authors elaborate on the selection process of parameter $c^r$ and explore a broader range of $c^r$ values across additional datasets?
4.	The method is efficient for certain datasets, but for large datasets like the full ImageNet, it still be computationally expensive. Do the authors have any future plans to further reduce computational requirements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650097485

thread_type:
official_review

<thread_object>
creator: reviewer_tjwf

<content>
summary:
This paper proposes a robustness enhancement method based on Lipschitz continuity, aiming to improve the security of deep neural networks (DNNs) under adversarial attacks. The authors introduce a forged function that constrains the input domain of the model to reduce its Lipschitz constant, potentially enhancing robustness. This method optimizes inference without retraining the model, thus reducing computational costs. Experiments on CIFAR10, CIFAR100, and ImageNet datasets tested various model architectures like WRN and Swin Transformer, evaluating performance under both white-box and black-box attacks.

strengths:
The proposed forged function based on Lipschitz continuity is implemented during the inference phase, eliminating the need for retraining or model parameter adjustments. Compared to traditional adversarial training, this method offers a significant computational cost advantage, making it more efficient for applications with limited resources.

weaknesses:
1. Please distinguish between the use of `\cite{}` and `\citep{}`.
2. The authors are encouraged to open-source their code.
3. In the "Related Work" section, the authors should mention the names of methods alongside author names to aid reader comprehension.
4. In Algorithm 1, Forged Function, please add a description of the hyperparameter $c^r$ in the `require` section.
5. AutoAttack is not the latest attack method; the authors are encouraged to use more advanced black-box attack methods, as referenced in [1].
6. Although the paper emphasizes low computational overhead during the inference phase, it is recommended that the authors provide specific experimental data or quantitative analysis to compare computational overhead.

**Reference**:

[1] https://github.com/Trustworthy-AI-Group/TransferAttack

questions:
1. How effective is the proposed method in more complex tasks and models, such as object detection and multimodal tasks?
2. Although the authors validated against gradient obfuscation through AutoAttack, the forged function suppresses inputs below a threshold to zero, potentially introducing gradient obfuscation under untested attack methods. Are there further theoretical or experimental supports to ensure that this method does not introduce gradient obfuscation under various attack strategies?
3. How generalizable is this method? Is it applicable to non-vision tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704144168

thread_type:
official_review

<thread_object>
creator: reviewer_y1db

<content>
summary:
This work starts from Lipschitz continuity and enhances adversarial robustness by minimizing the empirical Lipschitz constant. Specifically, the authors propose a plug-in forged function that can be inserted before each convolutional layer and MLP layer to remap the input domain of each layer into a constrained set. The theoretical guarantee of this paper primarily demonstrates that the largest singular value of the parameter matrix can serve as a loose bound for the Lipschitz constant. Experimental results show that the forged layers can bring some improvement to existing adversarial training methods.

strengths:
The logic flow of this work is coherent, and the writing is clear and easy to understand. The idea of optimizing the Lipschitz constant to achieve certified adversarial robustness is also a classic approach. The implementation is very straightforward, and the proposed forged function can serve as a plug-in module that integrates with any CNN or transformer-based model.

weaknesses:
Theoretical side: The bound is somewhat too loose. This paper derives the final optimization objective through the Gershgorin Circle Theorem, but this bound lacks guarantees due to the multiple assumptions made.

Empirical side: On one hand, from Tables 1 and 2, it seems that the forged function does not show significant improvements; in fact, when combined with other robust methods, the accuracy under AutoAttack even decreases. On the other hand, the robustness of the forged function itself lacks experimental support, such as how much robustness it can provide without any robustness designs (including adversarial training). An important contribution of this paper is that it incurs lower costs compared to adversarial training, so a comparison of performance with adversarial training is necessary.

Meanwhile, I suggest conducting adaptive attack against forged function to test the white-box robustness.

questions:
Please see weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732270850483

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kindly Reminder: Review and Discussion of Manuscript Revision

comment:
Dear reviewers,

We would like to kindly remind you that we have submitted point-by-point responses to each of the comments provided in your review, and the revised manuscript has been uploaded for your consideration. We are truly grateful for the thoughtful feedback and constructive suggestions you have shared with us.

As the deadline for the review phase has now passed, we sincerely hope that all reviewers can participate in the discussion during the rebuttal phase, if possible. We greatly appreciate your time and effort in reviewing our manuscript.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a method to enhance adversarial robustness in deep neural networks by reducing the empirical Lipschitz constant through a novel forged function applied during inference. While the work aims to address a critical issue in AI safety and robustness, it falls short of meeting the standard for acceptance due to several significant weaknesses. The theoretical foundation is undermined by overly loose bounds derived under multiple assumptions, which limits the generalizability and rigor of the approach. Empirically, the method demonstrates marginal improvements in robustness, with results that are inconsistent across different datasets and adversarial scenarios. The robustness gains, often within 0.5%-2%, are accompanied by trade-offs in clean accuracy, which are neither systematically analyzed nor mitigated. Additionally, the novelty of the approach is questionable, as it builds upon well-established techniques such as ReLU-like thresholding without substantial innovation.

The reviewers have highlighted critical gaps, including insufficient motivation, incomplete experimental support for robustness claims, and unresolved concerns about potential gradient masking under adversarial attacks. For example, while the authors argue that their method is computationally efficient, they do not convincingly compare its performance against state-of-the-art adversarial training methods in terms of robustness-to-cost ratio. Furthermore, the paper lacks a thorough investigation of its applicability beyond vision tasks, which limits its impact and generalizability. The claims of cost-effectiveness, while appealing, are undercut by the method's limited scalability to larger datasets and complex tasks. Moreover, the rebuttal failed to adequately address concerns about empirical validation and theoretical soundness, leaving doubts about the reliability and significance of the results.

The paper does present a coherent structure and a clear narrative, with some potential for practical application due to its plug-and-play nature. However, these strengths are overshadowed by the method's incremental contribution and lack of robustness against adaptive attacks. In light of these limitations, the submission does not meet the bar for acceptance at ICLR, where originality, rigor, and impact are paramount. The work would benefit from a stronger theoretical underpinning, comprehensive empirical evaluations, and a clearer articulation of its contribution relative to existing literature.