Random Logit Scaling: Defending Deep Neural Networks Against Black-Box Score-Based Adversarial Example Attacks

Authors: Hamid Dashtbani, Mehdi Dousti Gandomani, AmirMahdi Sadeghzadeh, Rasool Jalili
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Adversarial examples, Query-based black-box attacks, Randomized defenses

Abstract:
Machine learning models are increasingly adapted in various domains, such as autonomous driving, facial recognition, and malware detection, achieving state-of-the-art results. However, adversarial example attacks pose a significant threat to the reliable deployment of machine learning models in such applications. In recent years, some powerful adversarial example attacks have been proposed for the fast and query-efficient generation of adversarial examples, even in black-box scenarios where attackers only have an oracle access to the target model, highlighting the need for scalable, low-cost, and powerful defenses. In this work, we present two contributions to the domain of black-box attacks and defenses. First, we propose Random Logit Scaling (RLS), a randomization-based defense against black-box score-based adversarial example attacks. RLS is a plug-and-play, post-processing defense that can be implemented on top of any existing ML model with minimal effort. The idea behind RLS is to confuse an attacker by outputting falsified scores resulting from randomly scaled logits while maintaining the model accuracy. We show that RLS significantly reduces the success rate of state-of-the-art black-box score-based attacks while preserving the accuracy and minimizing confidence score distortion compared to state-of-the-art randomization-based defenses. Second, we introduce a novel adaptive attack against AAA, a SOTA non-randomized black-box defense against black-box score-based attacks that also modifies output logits to confuse attackers. With this adaptive attack, we demonstrate the vulnerability of AAA, establishing RLS as the effective SOTA defense against black-box score-based attacks.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730504424097

thread_type:
official_review

<thread_object>
creator: reviewer_gqxn

<content>
summary:
Based on the idea that a defender could blind an attacker by not producing true logit values, resulting in falsified confidence scores, this paper proposes Random Logit Scaling (RLS) as a novel defense mechanism. While random logit scaling could degrade model performance, RLS mitigates this by scaling output logits with a carefully chosen positive factor $m$, effectively reducing attack success rates, and preserving clean accuracy. Experimental results demonstrate the effectiveness of this approach.

strengths:
1. Motivation and observation are both clear.
2. This work is a post-processing method. doesn't need any overhead during training. This is important for larger models and datasets.
3. The experiments span a range of attacks and datasets, providing strong empirical support for the robustness and effectiveness of RLS.

weaknesses:
1. In comparison with other baselines, it’s straightforward in your framework to explain why they lead to performance degradation. However, it’s less clear why your method achieves a lower attack success rate than others. Could you clarify the limitations of the other methods?
2. The choice of using a uniform distribution for logit scaling, though effective, appears somewhat arbitrary. I think more discussion on potential alternative distributions would strengthen the approach.
3. In the experiments, a few fixed settings are explored for the range of $m$. I am curious about how varying the range of $m$ impacts model robustness and accuracy. An empirical ablation study on optimal parameter selection would add valuable insights.
4. I'm just curious if this method can be applied in the scenario of a white box attack.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589572030

thread_type:
official_review

<thread_object>
creator: reviewer_wbxu

<content>
summary:
This paper explores a new randomization-based method Random Logit Scale (RLS) to defend against black-box score-based adversarial attacks. RLS is a post processing defense which confuses an attacker by randomly scaling the output logit. While maintaining the model clean accuracy and confident score distortion, RLS can significant reduce the attack success rate of SOTA black-box score-based attacks.

strengths:
- The paper's motivation is clear and convincing. The idea behind why randomly scaling the model logits helps fortify against black-box score-based attacks sounds intuitively reasonable.
- Most figures are easy to understand and the writing is clear and easy to follow. 
- The experiments are done against popular black-box attacks and the empirical improvements compared to baselines seem significant and no performance drop is desirable.

weaknesses:
My main concerns are:
1. Lack of theoretical analysis
2. Some results in the experiments are not convincing
3. Some parts of the paper are not well written.

Please see the section on questions for a list of my concerns.

questions:
## Main questions:
1. The method is very simple and straightforward. Randomly scaling the output logits does not hurt the accuracy because it maintains the rank of predicted labels if $m$ is larger than 0 and is able to falsify the output scores and confuse an adversary. This approach is also applicable for output softmax, so how does it work if the output softmax is randomly scaled instead of output logits?
2. Although the authors provided an analysis of the impact of RLS on the confidence scores, a theoretical analysis of the efficiency and robustness of RLS against score-base attacks, as in [1, 2], is missing. I encourage the authors to provide a similar theoretical analysis.
3. $l_2, l_\infty$ and $l_0$ attacks are different, and a defence designed for score-based attacks should be robust against different $l_p$ attacks. I would suggest the authors should conduct more comprehensive experiments across different $l_p$ score-based attacks [3, 4].
4. The larger the perturbation is, the higher the attack success rate of an attack is, and there is no guarantee RLS is better than other defence mechanisms under larger perturbation attacks. Therefore, a strong defense should work well across different perturbation budgets and I encourage the authors demonstrates the robustness of RLS vs others across different perturbation budgets.
5. The average number of queries of different attacks against different defense methods is very low if compared with the query budget used in this study (10,000). This means that these attacks successfully yield adversarial examples for most of the selected samples. The average number of queries should have a correlation with the ASR. For instance, lower ASR attacks should have an approximate or higher average number of queries than higher ASR attacks. However, results in Tables 3 and 4 show that the average number of queries of Bandit and Square attacks against RLS is significantly low and even lower than the non-defensive method, while ASR is extremely low if compared with the non-defensive method (~100% ASR). 
6. The experiment setup for RFD method reported in Table 3 and 4 is not consistent. The values of σ are varied and the accuracy drop is not consistent around 1% or 2% as reported in [2]. For a fair comparison, the authors should evaluate RFD with the same settings as done in [2] when the clean accuracy drop is around 1% or 2%.
7. As some results in the paper are not convincing and evaluation settings are not comprehensive mentioned, I suggest the authors release your implementation for replication and verification purposes?  
8. In this study, EOT is employed as an adaptive attack and it can be incorporated with gradient-estimation or random search attack method. Although the authors use k queries for each sample at each attack iteration and take an average over the outputs to cancel out the randomness but it is unclear what method is used to determine attack direction e.g. gradient-based or random search approach. What is the perturbation budget used to attack against the adaptive attack? Again, the query average is very low if compared to the query budget but the ASR is not high. Can the author report the median query number instead and provide a ridiculous explanation?
9. The authors discussed AAA defense which is designed for score-based attacks and this defense does not induce an accuracy drop as RLS. For completeness, I encourage the author to evaluate and compare this defense with RLS.

## Minor:
1. There are some contributions that are not actually contributions, e.g., 2 and 3. The authors should reorganize and summarize contributions 1 and 4  more thoroughly. 
2. Is minimizing distortion to the confidence score (in the introduction and section 5.3) really worth achieving? If so, the authors should cite papers or provide existing applications that rely on the accuracy of the gap between the model’s top and runner-up classes’ confidence scores to support your argument.
3. The experimental setup is not well explained and lacks some crucial information. For instance, what is the $l_p$ norm of the attacks? What is the perturbation budget used in this study? How many samples are selected for evaluation on CIFAR-10? How did the author select these samples? The author should clearly mention how samples are selected for evaluation on ImageNet in the experimental setup, e.g. 1000 samples are randomly selected, as mentioned in the caption of Table 4 in the experimental setup. Similarly, 2000 samples from CIFAR-10 are in the caption of Table 5. 

[1] Zeyu Qin, Yanbo Fan, Hongyuan Zha, and Baoyuan Wu. Random noise defense against query-based black-box attacks. NeurIPS, 2021.

[2] Nguyen Hung-Quang, Yingjie Lao, Tung Pham, Kok-Seng Wong, and Khoa D. Doan. Understanding the robustness of randomized feature defense against query-based adversarial attacks. ICLR, 2024

[3] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square Attack: a query-efficient black-box adversarial attack via random search. ECCV, 2020

[4] Quoc Viet Vo, Ehsan Abbasnejad, and Damith C. Ranasinghe. Brusleattack: Query-efficient score-based sparse adversarial attack. ICLR, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719374288

thread_type:
official_review

<thread_object>
creator: reviewer_ye5x

<content>
summary:
Existing works show that randomizing the prediction mitigates the threat of query-based attacks. However, defenses that inject noise into the input or to the feature lead to accuracy drops on clean data. Instead, this paper proposes random logit scaling (RLS), a defense that rescales the returned logit by a random variable $m$, therefore remains clean accuracy. Experimental results show that RLS is robust to score-based black box attacks without the tradeoff in accuracy.

strengths:
- RLS is simple and plug-and-play, and can be applied to any model.
- RLS is robust against score-based attacks and does not change the accuracy of the target model.
- The experiments show that RLS is also robust against adaptive attacks that query the victim multiple times and average the prediction.

weaknesses:
- Since RLS does not change the prediction, it will not be effective against decision-based attacks. The argument that decision-based attacks are inefficient is not convincing; recent attacks like RayS [1] can find adversarial samples with ~100 queries.
- The paper should compare to AAA [2], another defense that changes the logit to fool the attacker. AAA also guarantees the reliability of the returned probability by optimizing the calibration error. 
- The experiments are only conducted on CIFAR10, which is insufficient to conclude the effectiveness of RLS.


[1] Chen, Jinghui, and Quanquan Gu. "Rays: A ray searching method for hard-label adversarial attack." Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.
[2] Chen, Sizhe, et al. "Adversarial attack on attackers: Post-process to mitigate black-box score-based query attacks." Advances in Neural Information Processing Systems 35 (2022): 14929-14943.

questions:
- What is the performance of AAA? 
- What is the calibration error of RLS?
- Experimental results on other datasets such as ImageNet would be helpful to demonstrate the robustness of RLS.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732792782541

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Revision Summary

comment:
We have submitted our revised rebuttal revision and would like to express our gratitude to the reviewers for their valuable comments and feedback, which have significantly contributed to enhancing our submission. In the revised version, all major changes and additions are highlighted in red. Below, we outline the changes made to our submission.

- **The AAA Defense**:
    - We have added a new section (Section 4) to **introduce our adaptive attack against AAA-sine, a state-of-the-art defense against black-box score-based attacks that previous works, including RFD [1], have consistently failed to outperform**. Consequently, we have updated the Introduction, Abstract, and Conclusion sections to reflect this new contribution. The experimental results of our adaptive attack are detailed in Section 6.2. All changes are highlighted in red.
    - We have included a discussion of AAA to provide a more in-depth comparison between AAA and RLS (see Section 2.2).
    - Additionally, we have included the ECE comparison of our defense with AAA and other defenses (see Section 6.3 and Table 6).
    - We have also added a robustness comparison between AAA and RLS in Appendix F.4, demonstrating that RLS outperforms AAA.

- **Additional Experiments**:
    - We have provided extensive experiments and evaluations in **$\ell_0$ space, using the SOTA black-box sparse attack, BruSLeAttack [2] on both CIFAR-10 and ImageNet** and have updated our Related Work and Experiments sections to highlight this. The results are presented in Appendix F.2. We also included references to other sparse attacks for decision-based attacks, for example. All of these changes are highlighted in red.
    - Additional experiments in $\ell_2$ space on both ImageNet and CIFAR-10 have been added in Appendix F.1.
    - Further experiments for $\epsilon = 0.1$ in $\ell_\infty$ space are included in Appendix F.3.

- **Improvement to Experimental Setup**:
    - We edited our original submission to provide a more detailed description of our experimental setup (see Section 6.1).
    - Additionally, we have expanded Appendix A to discuss the attack and defense configurations in greater detail.
    - Minor changes have been made to the table captions to clarify the setup and configuration.

- **Reorganization of Sections**:
    - To comply with the page limit guidelines following the highlighted additions, we have moved Sections 4.3 (The Method Formulation) and 5.2 (Defense Components) from our original submission to Appendices B and D, respectively.

- **Code**:
    - We have added a more comprehensive README file to our code, along with the implementation of AAA.


[1] Nguyen Hung-Quang, Yingjie Lao, Tung Pham, Kok-Seng Wong, and Khoa D. Doan. Understanding the robustness of randomized feature defense against query-based adversarial attacks. ICLR, 2024

[2] Quoc Viet Vo, Ehsan Abbasnejad, and Damith C. Ranasinghe. Brusleattack: Query-efficient score-based sparse adversarial attack. ICLR, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733060385342

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We thank you for your valuable comments and feedback which helped us to improve our work substantially.

We would like to take full advantage of the extended discussion period to address any remaining concerns you might have and look forward to further discussions. We also **encourage all reviewers to re-evaluate our work considering the revised version**, which incorporates **two key changes**. First, we introduce **a new contribution: the Pendulum attack, our adaptive attack against AAA**—the state-of-the-art defense against score-based attacks that prior works have not been able to surpass. Our adaptive attack **reveals a critical weakness in AAA**. Second, we have included **a more comprehensive set of experiments** to strengthen our analysis. These additions include evaluations in $\ell_0$ and $\ell_2$ spaces, larger perturbations in $\ell_\infty$, robustness comparison of AAA and RLS, the performance of our adaptive attack against AAA, and the expected calibration error of defenses.

We greatly value your input and thank you once again for your thoughtful feedback.

Best regards,

The authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Random Logit Scaling (RLS), a randomization-based defense against black-box score-based adversarial example attacks. RLS is a plug-and-play postprocessing defense that applies to any existing ML model. The reviewers acknowledge that RLS is lightweight, easy to use, and does not sacrifice clean accuracy. However, reviewers think the work lacks theoretical analysis, and some of the design choices seem arbitrary. The defense only works for score-based black-box attacks and is not applicable against decision-based and transfer-based black-box attacks or white-box attacks. Given the limitations of the proposed method, I recommend reject.