Democratic Training Against Universal Adversarial Perturbations

Authors: Bing Sun, Jun Sun, Wei Zhao
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Neural network adversarial attack; Universal adversarial perturbation; Adversarial attack defense

Abstract:
Despite their advances and success, real-world deep neural networks are known to be vulnerable to adversarial attacks. Universal adversarial perturbation, an input-agnostic attack, poses a serious threat for them to be deployed in security-sensitive systems. In this case, a single universal adversarial perturbation deceives the model on a range of clean inputs without requiring input-specific optimization, which makes it particularly threatening. In this work, we observe that universal adversarial perturbations usually lead to abnormal entropy spectrum in hidden layers, which suggests that the prediction is dominated by a small number of ``feature'' in such cases (rather than democratically by many features). Inspired by this, we propose an efficient yet effective defense method for mitigating UAPs called \emph{Democratic Training} by performing entropy-based model enhancement to suppress the effect of the universal adversarial perturbations in a given model. \emph{Democratic Training} is evaluated with 7 neural networks trained on 5 benchmark datasets and 5 types of state-of-the-art universal adversarial attack methods. The results show that it effectively reduces the attack success rate, improves model robustness and preserves the model accuracy on clean samples.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729252116954

thread_type:
official_review

<thread_object>
creator: reviewer_etkh

<content>
summary:
The author aims to improve the robustness of universarial adversarial by fine-tuning with a small amount of data, mainly by performing entropy based data augmentation to suppress the influence of general adversarial perturbations in a given model. Then the experiment was presented.

strengths:
1, The setting is reasonable, author want to resist universal adversarial samples through small cost, this may be useful in some situation.
2, Good experiment results.

weaknesses:
1, The method provided in the article is not novel, overall, it is still based on the 'min-max' methods. And in my opinion, it seems to be a weakened version of adversarial training.

2, The symbols are somewhat confused. For example, in euqation (4), author use $L_{cce}$, I think this means cross entropy loss, but author did not introduce what is $L_{cce}$ before the euqation (4); in algorithm line 3, author define $I_b^{en}$ which is got by SampleGenerator, but do not use it in the following of algorithm, or maybe it have been wirtten as $i_{en}$ in line 4?

3, I try to test the author's algorithm on CIFAR10(VGG16, budget 8/255), but I didn't get such a good result shown in table2, the SR has only decreased by about 10%. Author did not submit the code, so I hope the author can provide a detailed introduction to the settings for Algorithms 1 and 2 (For example, how to select the epoch, learning rate, hyperparameters), and it's best to provide a detailed explanation of each symbol and steps in the algorithm.

questions:
1: The author obtained the idea of Democratic Training by studying the performance of UAP between different layers in network. I want to know why author said 'Democratic Training does not rely on generating UAPs and are thus not limited to specific UAP attacks.', As mentioned in the article, the UAP analyzed by the author is produced based on FA-UAP, so the properties analyzed should mainly for such kind of UAP, and the algorithm guided by this should also mainly target on FA-UAP. If it cannot be argued that all UAPs have such  properties, then this statement seems unreasonable?

2: The author did not provide a detailed explanation for $H(i)$ in algorithm 2 line 2, according to the equation (3), I think that the author is  trying to say $H(i)$ is an entropy loss. Less strictly speaking, maximizing H(x) means that the various components of x are directly averaged as much as possible. So, is the goal of SG(SampleGenerator, Algorithm 2) equivalent to finding an adversarial sample with each component average? Is this a form of weakened PGD? If so, why is finding weaker adversarial samples beneficial for improving robustness? Why not directly target finding UAP for SG?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730060142481

thread_type:
official_review

<thread_object>
creator: reviewer_haa5

<content>
summary:
To improve neural network robustness to targeted UAPs, this paper proposed an adversarial training-like method that fine-tunes a pretrained model to reduce middle-layer entropy.

strengths:
1. The experiments are comprehensive.
2. The proposed defense is attack-agnostic which is more practical and efficient.
3. The proposed defense largely reduced the targeted attack success rate.

I tend to accept this pape. However, since I'm not familiar with UAP attack and defense baseline methods, I will listen to other reviewers and public comments and then decide.

weaknesses:
1. UAP attacks evaluated in the paper were published in 2018,2019,2020 and seem out-of-date.
2. After democracy training, there is still a gap between ``AAcc.'' and clean accuracy. I wonder about the effectiveness of democracy training against non-targeted UAPs.
3. Average results in Table 4\&5 are ambiguous since there can be a large bias among different networks.

questions:
1. For RQ3: did you adversarially train a model from scratch or just fine-tune a pretrained model with an adversarial training objective?
2. I'm not very convinced by the claim in Lines 239-243 that says middle-layer feature entropy suggests the model's classification confidence.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730438865166

thread_type:
official_review

<thread_object>
creator: reviewer_1bzp

<content>
summary:
The paper presents a novel defense method called Democratic Training to mitigate the impact of Universal Adversarial Perturbations (UAPs) on deep neural networks.  The authors observed that UAPs lead to an abnormal entropy spectrum in hidden layers, which shows the model's prediction is dominated by a small subset of features. Democratic Training mitigates this issue by increasing entropy to ensure model predictions rely on a wider range of features. The approach was evaluated on multiple models and datasets and was found to be effective in reducing attack success rates while preserving accuracy on clean data.

strengths:
1. The use of entropy to reveal the dominance of UAPs and the concept of Democratic Training as a defense mechanism is innovative.
2. The method was evaluated across various neural network architectures and benchmark datasets, which strengthens the claim of its general applicability.
3. Unlike other defense methods, Democratic Training does not require architectural modifications, which makes it easy to integrate into existing systems

weaknesses:
1. The evaluation focused primarily on benchmark datasets and common UAP generation methods. It would be beneficial to see how this approach performs on more sophisticated and adaptive attacks, such as adversarial examples generated in dynamic environments.
2. The proposed method mainly works well on CNN. Authors should validate it in more types of networks, such as transformers.
3.  The method requires access to a small set of clean data for entropy measurement and training, which might not always be practical

questions:
1. How about performances of the method on ViT?
2. What's the time cost of the method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730530280197

thread_type:
official_review

<thread_object>
creator: reviewer_xsaf

<content>
summary:
The paper presents an investigation into the defense against universal adversarial perturbations (UAPs), with a particular focus on targeted UAPs. The authors have made a notable observation regarding the entropy spectrum in hidden layers when UAPs are introduced, which serves as the cornerstone for their proposed 'democratic training' approach. This method aims to enhance the adversarial robustness of neural network models. The empirical results provided in the paper demonstrate the efficacy of the approach, as well as its ability to maintain the clean accuracy of the model.

In general, this paper is well-structured and presents a novel approach, which meets the basic criteria of ICLR conference. However, there are some aspects that could be improved or expanded upon to enhance the overall quality and impact of the paper.

strengths:
1. This paper is well-written and easy-to-follow.
2. The paper makes a commendable observation concerning the entropy spectrum in deep neural network layers, which is a significant contribution to the field and forms the basis for the proposed defense mechanism.
3. The efficiency of the proposed democratic training method is noteworthy. It circumvents the need to generate UAPs during training, instead utilizing a limited number of epochs to identify low-entropy examples, which is a resourceful approach.

weaknesses:
1. The threat model employed in the experiments primarily utilizes gradient-based attack methods. These methods presuppose access to the model's parameters, aligning with white-box attack scenarios. This appears to be at odds with the assertion in Section 2.3 that adversarial knowledge does not extend to the internal parameters of the model. Clarification on this point would be beneficial.

2. The comparison with adversarial training methods may require further refinement. Adversarial training aims to bolster adversarial accuracy by integrating adversarial examples with clean examples during training. Constraining the number of training epochs could result in an underfit model, which may not provide a fair benchmark. Additionally, it would be advantageous to include a comparison with the widely recognized TRADES method[1], which is absent from the current manuscript.

3. The potential for adaptive attacks warrants consideration. If adversaries are aware of the defense strategy, they could tailor adversarial examples to bypass the defense. I know that in the threat model, no adaptive attacks are considered since the attackers do not know the internal details of the models. However, the chosen attack methods in the experiments inherently rely on gradient information. So I would suggest that the authors should consider the potential for adaptive attacks.

4. The scope of the experiments is largely limited to datasets comprising natural images. It would be beneficial to extend the evaluation to smaller-scale datasets, such as CIFAR-10, to complement the findings and potentially leverage open-source robust models for further exploration of the neuron entropy spectrum concept.

5. While the paper discusses various existing defensive methods against UAPs and includes experimental comparisons, a direct comparison with state-of-the-art methods is missing. It is recommended to condense the background section and incorporate a more thorough comparison with leading-edge techniques.

6. Minor Issues
  (1) Please consider to reduce the margin between Figure 1 and the text.
  (2) Suggesting to add necessary notations (SR) from the main test to the Table 2 for better understanding.


[1] Zhang, Hongyang, et al. "Theoretically principled trade-off between robustness and accuracy." International conference on machine learning. PMLR, 2019.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732687769326

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Change Summary on Revised Paper

comment:
Dear All,

We deeply appreciate your thorough evaluation and insightful feedback. We have updated our paper accordingly. Below is a summary of the changes:

1. Update Abstract: Democratic training is evaluated on one more model, one more benchmark dataset and one more UAP attack method
2. Add Section 2.2 Evaluation Metrics: add definitions of SR and AAcc.
3. Update Section 2.4 Threat Model: correct descriptions on adversarial capabilities and knowledge
4. Update Section 3.2 Entropy Analysis: improve descriptions on empirical study
5. Update Section 3.3 Entropy-based Repair: improve descriptions with symbols used in Algorithm 1, Algorithm 2 and Equation 6 properly introduced
6. Update Section 4.1 Experimental Setup: new dataset evaluatedL CIFAR-10; update Table 1
7. Update RQ1: add experimental results for $NN_7$ trained on CIFAR-10 with widsresnet architecture; update Table 2 and Figure 3
8. Update RQ2: add experimental results for UAP attack method SGA; update Table 3
9. Update RQ3: add experimental results for TRADES; update Table 4
10. Update RQ4: add experimental results for DensePure; update Table 5
11. Update Section 5 Related Works: improve background on defense against adversarial attacks
12. Add Future Works section in Appendix 7.1
13. Update Datasets Used in Our Experiments in Appendix 7.3: add descriptions on CIFAR-10
14. Update Adaptive Attacks in Appendix 7.5: add experimental results for advanced adversaries; add Table 8
15: Add Entropy Analysis on Other UAPs in Appendix 7.6: add Figure 4
16. Add Non-targeted UAP Attacks in Appendix 7.7: add results for non-targeted UAP defense performance; add Table 9; add Figure 5
17. Address comments on formatting



Thanks!

BR.
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a novel defense method called Democratic Training to mitigate the impact of Universal Adversarial Perturbations (UAPs) on deep neural networks. The authors observed that UAPs lead to an abnormal entropy spectrum in hidden layers, which shows the model's prediction is dominated by a small subset of features. Democratic Training mitigates this issue by increasing entropy to ensure model predictions rely on a wider range of features. The approach was evaluated on multiple models and datasets and was found to be effective in reducing attack success rates while preserving accuracy on clean data. There are some merits in this paper. For example, the use of entropy to reveal the dominance of UAPs and the concept of Democratic Training as a defense mechanism is innovative. The method was evaluated across various neural network architectures and benchmark datasets, which strengthens the claim of its general applicability. Unlike other defense methods, Democratic Training does not require architectural modifications, which makes it easy to integrate into existing systems. Moreover, this paper is well-written and easy-to-follow. The paper makes a commendable observation concerning the entropy spectrum in deep neural network layers, which is a significant contribution to the field and forms the basis for the proposed defense mechanism. The efficiency of the proposed democratic training method is noteworthy. It circumvents the need to generate UAPs during training, instead utilizing a limited number of epochs to identify low-entropy examples, which is a resourceful approach. While the reviewers had some concerns about reproducibility, the authors did a particularly good job in their rebuttal. Therefore, all of us have agreed to accept this paper for publication! Please include the additional discussion in the next version.