Democratic Training Against Universal Adversarial Perturbations
Authors: Bing Sun
Jun Sun
Wei Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Neural network adversarial attack; Universal adversarial perturbation; Adversarial attack defense
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: A novel method to mitigate the effect of UAPs via democratic training.

Abstract: Despite their advances and success, real-world deep neural networks are known to be vulnerable to adversarial attacks. Universal adversarial perturbation, an input-agnostic attack, poses a serious threat for them to be deployed in security-sensitive systems. In this case, a single universal adversarial perturbation deceives the model on a range of clean inputs without requiring input-specific optimization, which makes it particularly threatening. In this work, we observe that universal adversarial perturbations usually lead to abnormal entropy spectrum in hidden layers, which suggests that the prediction is dominated by a small number of ``feature'' in such cases (rather than democratically by many features). Inspired by this, we propose an efficient yet effective defense method for mitigating UAPs called \emph{Democratic Training} by performing entropy-based model enhancement to suppress the effect of the universal adversarial perturbations in a given model. \emph{Democratic Training} is evaluated with 7 neural networks trained on 5 benchmark datasets and 5 types of state-of-the-art universal adversarial attack methods. The results show that it effectively reduces the attack success rate, improves model robustness and preserves the model accuracy on clean samples.

Published as a conference paper at ICLR 2025
DEMOCRATIC TRAINING AGAINST UNIVERSAL AD- VERSARIAL PERTURBATIONS
Bing Sun
Jun Sun
Singapore Management University bing. sun. 2020@phdcs smu sg
Singapore Management University
junsun edu.sg
Wei Zhao
Singapore Management University wzhaolsma smu. edu. sg
ABSTRACT
Despite their advances and success, real-world deep neural networks are known vulnerable to adversarial attacks. Universal adversarial perturbation, an input- agnostic attack, poses serious threat for them to be deployed in security-sensitive systems. In this case, a single universal adversarial perturbation deceives the model range of clean inputs without requiring input-specific optimization, which makes it particularly threatening. In this work, we observe that univer- sal adversarial perturbations usually lead abnormal entropy spectrum in hidden layers, which suggests that the prediction is dominated by small number of "fea- ture" in such cases (rather than democratically by many features). Inspired by this, we propose an efficient yet effective defense method for mitigating UAPs called Democratic Training by performing entropy-based model enhancement suppress the effect of the universal adversarial perturbations in given model. Democratic Training is evaluated with 7 neural networks trained on 5 benchmark datasets and types of ofate-of-the-art universal adversarial attack methods. The results show that effectively reduces the attack success rate, improves model robustness and preserves the model accuracy on clean samples.
INTRODUCTION
Advances and success in deep learning have enabled the widespread use of Deep Neural Networks (DNNs) based machine learning models. DNNs become the algorithm of choice for wide range of applications (Fu et al. 2016; [Schroff et al. 2015 Bojarski et al.) 2016 Vieira et al.) 2017). How- ever, despite their success, DNNs are found to make erroneous predictions when a carefully crafted, small magnitude human-imperceptible perturbation is added to an input (Goodfellow et al.) 2015) Dong et al. 2018 Madry et al.) 2018 Moosavi-Dezzooll et al. (017). One can easily conduct adversarial attacks against the target network by generating adversarial examples utilizing such per- turbations. The existence of adversarial examples has become a serious concern to systems based on DNNs especially in safety-critica applications. Neural network adversarial attacks can be input- specific (Goodfellow et al. 2015 Dong et al. 2018 Madry et al., 2018 Wang et al. 2021b Zhang (Moosavi-Dezfool)
al. 2017 Shafahi al. 2020 Zhang et al.) 2020b Mopuri et al. 2018; Poursaeed et al.) 2018 Hayes & Danezis) 2018 Liu et al.] 2019). In the case of ipput-specific attacks or per-instance attacks, perturbations are dividually optimized for each input to produce the corresponding adversarial example. In contrast, in input-agnostic attacks, a single perturbation optimized for a set of inputs to produce an uni- versal perturbation to generate set of adversarial examples. Such perturbations are often referred to as universal adversarial perturbations (UAP), where the same perturbation applied to a range of clean inputs will cause the model to misclassify. Compared to input-specific adversarial attacks, UAPs could be considered more threatening since they are more efficient in terms of computation cost from the attack point of view. Furthermore, defending against UAPs poses significant chal- lenge, as it is hypothesized that they exploit and amplify legitimate features essential to the model's performance (Moosavi-Dezfooli et al. 2017 Zhang et al. 2020b; Borkar al. 2020).
al. Ganeshan
(019)

---

Published as a conference paper at ICLR 2025
Comes Cluss
Democratic Training
LANP
Tran
Fineturing
Orignall mode
Democrati Trassing
model
Sumple Canterts
Cumples
Class
Asstul chaniian
campiis
Comrzzt
Figure 1: An overview of our framework
A number of practical and realistic attacks based on UAPs have been successfully conducted in var- ious scenarios, i.e., image classification Deztooi al. 2017). facial recognition (Sharif et al. 2016), object detection (Eykholt et al., 2018 Song et al. 2018). etc. Moosavi-Dezfore et al. (2017) first explore the existence UAPs. For given set of training imputs, the proposed algorithm iteratively computes the perturbation to make an adversarial example across the decision boundary of the expected predicted category. Following this work, several different approaches have been proposed to generate UAPs from different aspects, utilizing different loss functions. These can be categorized into two main groups (Weng et al.] 2024):: 1) noise-based (Moosavi-Dezfooli et al.)
2017 Zhang et al. 22200b Mummadi et al.] 2019), and 2) generator-based methods (Poursaeed et al. 2018 Mopuri et al. 2018; Naseer et al. 202I). J). Noise-based methods directly update the perturbation through optimization. On the other hand, generator-basee methods train generative network in prior to obtain the perturbation indirectly. UAP becomes a relevant threat in practice and it is important attacks. A range of existing works have been proposed to address the problem of defending machine learning models against UAPs. These include finetuning given model's 's parameters with UAP perturbed samples (Moosavi-Dezitee
manage such security risk and ensure neural networks are robust against such
et al. 2017 Mummadi et al. 2019), inserting feature regen- eration layers (Borkar al.] [2020), applying feature norm clipping techniques YYu et al.) 2021), ) etc. However, existing methods mainly focus on non-targett tor-ttegeeed attacks (Moosavi-Dezioon et al. 2017;
Shafahi et al. 2020; Benz al. (02T) and often require craft a large number of UAPs (Moosavi-
Dezfooli et al.) [017] Mummadi et al. 2019 Shafahi et al. 20200 Benz et al.] 2021 Borkar et al.)
2020 or change the architecture of the original model (Borkar et al. [0200] Yu et al. 2021: 2023).
In this work, focus on targeted universal adversarial attack which is both more relevant from an attacker point of view (i.e., so that the attacker can trigger specific target outcome) and more challenging from defender point of view. Our approach does not require constructing UAPs or modifying the model architecture. We propose scalable algorithm that mitigates the effect of UAPs through entropy based model enhancement. Specifically, described in Figure I Democratic Training with the key idea of enhancing given neural network by adjusting the weights of hidden neurons towards the correct predictions in the presence of UAPs. We first analyze the distribution of hidden neuron activation when an input perturbed with UAP is supplied to model and compare that with the activation when clean samples are supplied. We study the entropy such hidden neuron activation and our empirical results suggest that the presence of UAP causes layer- wise entropy drop and such effect becomes more severe at deeper layers. We conjecture that this is because the UAP enforces the "power" of certain features, which subsequently dominates the prediction. Base on such result, we propose to mitigate the effect of UAPs through adversarial finetuning guided by hidden layer entropy, philosophically speaking, enforcing democracy in the decision making. We compare the performance of our work with existing solutions on UAP defense and show that Democratic Training improves existing approaches significantly.
we propose
PRELIMINARIES
2.1 UNIVERSAL ADVERSARIAL PERTURBATION
We start with introducing the notation for targeted UAP attacks. Given a trained neural network N, a test dataset X and let y/ represent the attacker-chosen target class. A targeted UAP is perturbation
2

---

Published as a conference paper at ICLR 2025
6 that satisfies the following:
Ne 8) yt
(I)
6 11 S
where E X and [X] is sufficiently large and 5 is bounded by certain I, norm (I) 8 1 S e). We remark that here focus definition of vicinity based on lp norm. In general, it can be defined other forms as well. For instance, a UAP can take the form of a patch that small in size but applying to a range of clean inputs, the model will classify the perturbed inputs the target class.
The existence of UAP shows that there are systemic vulnerabilities in the model which can be ex- ploited by attacker regardless of the input. Hence, UAP attacks pose serious threats in real-world applications of neural networks such as attacking facial recognition systems where incorrect iden- tity returned (Amada al. 2021 Zolfi al. 2022). autonomous driving systems where wrong traffic sign or road condition is misidentified (Benz et al. 22200 Eykholt et al. 20188) speech recog- nition systems which may cause various systems to interpret human commands wrongly (Sun al.) malware detection systems where suspicious programs may bypass the detection (Castro 2021) and many others (Moosavi-Dezfool et al. 2017 Zhang et al. 2020b; Metzen et al.,
2024).
al.
2017
Wallace et al. 2019).
2.2
EVALUATION METRICS
Attack Success Rate (SR): This metric measures the percentage of adversarial samples (except the samples of the target class yr) classified to the target class y/:
[N + 6) y/ X - Xel
SR =
FE Xx)
where æ E X represents a clean input from dataset X, X, C X represents a set of samples from target class y/-
Adversarial Accuracy (AAcc.): This metric measures the accuracy of adversarial examples (where y represents the label of sample x):
[N + +5) yzl
AAc.
Σ BEX
X
2.3 ENTROPY
Shannon Entropy:
information theory, the entropy random variable represents the average amount of "information" "uncertainty" associated with the variable's possible outcomes. The concept of information entropy was introduced by Claude Shannon (Shin & Kim 1949), where the Shannon entropy proposed quantify the amount of information carried by variable. For a random variable v. which takes values from the set V that follows the probability distribution [0, 1] the entropy of is defined as:
p V
H(v) -- Σr(v) log
p(v) -()==-p(( VEV
where the summation denotes the sum over the variable's possible values.
Measure the Entropy of A Neural Network: The concept of entropy can be applied in neural networks for different purposes. Appendix 8.2]shows two methods proposed existing works mea- suring the entropy of given neural network. In this work, propose measure layer-wise entropy to understand how UAP fools a given neural network. The details are provided in Section 3.1
2.4 THREAT MODEL.
Our approach aims to mitigate the effect of UAPs for trained neural networks. In this work, assume evasion threat where set of clean data available to the adversary.
3

---

Published as a conference paper at ICLR 2025
Adversary goals. The goal of the adversary is generate UAPs such that applied to range clean imputs, the model will classify the perturbed inputs wrongly.
Adversarial capabilities. We assume the adversary has white-box access to the model and is capable of crafting UAPs.
Adversarial knowledge. We assume that the adversary has the information on the target model's architecture, inner parameters and optimization algorithms.
Our goal is mitigate the effect of UAPs on given model with minimum assumptions. Specifically, we assume the defender has the following knowledge about the neural network:
Defense goals. We aim to design strategy that remove the effect of UAPs from the model by adjusting the model parameters.
Defender's 's capabilities. We assume the defender has white-box access to the neural net- work model. The defender has information about the model architecture but cannot inter- fere with the training process.
Defender's knowledge. We assume small clean data is available (as it usually the case in practice), either given by the model provider or collected by the defender, to test the model's performance.
2.5 OUR PROBLEM
Problem. Let N be neural network which is assumed to be obtained from third party; x is an input and a small positive threshold. The UAP defense problem to mitigate the effect of UAPs on N such that the predictions of inputs patched with UAPs stay robust. Furthermore, the UAPs are bounded by Lp norm where lpp Se e. We would also require that the model's performance on clean data is minimally affected after the miligation process.
3 OUR APPROACH
To understand how UAPs deceive model, first conduct systematic analysis of model behav- iors from the lens of entropy. We study layer-wise entropy of given model with and without the presence of UAPs. As shall show in Section 3.2 the presence of UAPs will cause the layer- wise entropy to be abnormally lower than that clean inputs. Furthermore, such effect becomes more severe at deeper layers. Based on these findings, we propose Democratic Training which con- ducts entropy- bbased model enhancement to repair the given model such that the effect of UAPs is mitigated.
3.1 ENTROPY MEASUREMENT
Firstly, we present how entropy is measured in this work. For given neural network N, con- sisting of n layers, we treat each layer as single random variable, characterized by its in- put in and output 11+1- Thus, for layer I containing di neurons, given input to this layer I (zf, 2t, zt di d-11-13, its layer-wise entropy calculated as:
= (Win + b1) Pt soflmar((i)
(5)
k=
H
(i(k) log
where W; and by are the weight and bias parameters of layer / and is the activation function of layer I. Intuitively, we treat the activated value (k) of each neuron in layer as the activation probability for neuron k. and calculate the Shannon entropy of pi following Equation For given input to layer higher layer entropy H, indicates higher ambiguity and lower entropy H, indicates higher certainty.
4

---

Published as a conference paper at ICLR 2025
Layeewiee Entropy
Staple Sample with UAP
UAP
12.89
12.10
12.20
12,00
13.57
LAP
Shalkow Layer 1
Shallow Layer
Misdlle Layer 1
Midlle Layer
Doep Layer I
Doep Layer
Figure 2: Layer-wise entropy. Enlarged view for shallow and middle layers is provided.
3.2 ENTROPY ANALYSIS
To understand how UAP fools trained neural network, we conduct empirical study the layer- wise entropy of the model as follows:
Step 1.
Given a pretrained neural network, generate UAP such that the model classifies samples perturbed with the UAP as the target class.
Analyze the layer-wise entropy with clean samples only, i.e., we randomly select a set of clean samples and calculate their layer-wise entropy defined in Equation
Step 2.
Step 3. Apply the UAP generated in Step to the same set of clean samples selected in Step
2 and analyze the layer-wise entropy.
Step Take the UAP itself as an input to the model and analyze the layer-wise entropy.
We conduct the analysis on all models shown in Table I and for each model, multiple targeted UAPs are generated using a state-of thrthee UAP attack method DF-UAP (Zhang et al., 2020b). We study the entropy of the pooling layers and the last layer of each stage. We observe similar results across all models and we show the results on NN for shallow 1 (max pooling layer of stage 1), shallow 2 (end of stage 2), middle 1 (end of stage 3), middle 2 (end of stage 4), deep (end of stage 5) and deep 2 (last average pooling layer) layers for illustration purpose. As shown in Figure 2 at shallow layers, the entropy spectrum for clean and UAP infected samples are quite similar. At middle layers, the entropy of some UAP infected samples becomes smaller than that of the clean sample, but there is no clear boundary for separating the two for all samples. At deep layers, we observe clear separation between entropy from clean and perturbed samples, where UAP infected samples show abnormally small entropy compared to that of clean samples. These results suggest that the presence of UAP will cause layer-wise entropy drop and such effect becomes more severe at deeper layers. We interpret entropy as an indicator of the neural network's uncertainty in classifying the intermediate features. High entropy suggests the features are ambiguous while low entropy indicates the model is more certain on classifying the features. Our analysis results show that the presence of UAP will cause the layer-wiss wise entropy to drop significantly, and such lower entropy indicates the model is more certain on its classification at the same layer. As shown in Figure the entropy distribution of UAP perturbed samples leans towards the entropy of the UAP, i.e., the UAP dominates layer-wise entropy rather than the original image. At deeper layers, the entropy of UAP itself drops and is much lower than the entropy of clean samples, while the entropy of UAP perturbed samples follow such trend closely. We believe that, UAPs contain dominant features that cause the model to be certain on the prediction class at earlier layers, i.c., instead of features from the original sample, features from UAP lead the model to predict the target class. Similar findings reported in existing work (Zhang et al. 2020b) that suggests UAPs contain dominant features and original images behave like noise to them. We argue that such dominant features cause the layer-wise entropy drop which dominates the model prediction.
3.3 ENTROPY-BASER REPAIR
Based on our analysis results presented in Section 3.2 UAPs called Democratic Training, which aims to finetune the model such that it learns to predict low-entropy samples (by effectively reducing the presence of certain dominate features in these samples). During this process, we introduce Sample Generator that will craft low entropy samples from clean samples to mimic the effect of UAPs and guide the model to the correct prediction. Note that, the Sample Generator does not require information about the attack target class unlike existing
we design a general approach for mitigating
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Remove m.
Algorithm 2: Sample N, m,
for n epochs do
for m iterations do
for each batch do
L(i) H()); + i i sign((.LLi)) i= Clampli,
1
SampleGenerator N, m.
L(i,ben
return i;
aL [mn) + a Leon (1):
((0) aL..
0 C 0 70 (G)(
return 8:
works that rely on pre-computed perturbations (Moosavi-Deziooll et al. 2017). As described in Algorithm [ Democratic Training requires a small set of clean sample i E I ( 5% of training set) to finetune the original model N. For each epoch and each batch during finetuning, the Sample Generator transforms batch of clean inputs (Ip) into low entropy samples (I5")) as described in Algorithm Starting from clean sample i, the perturbation is updated based on the projection of the layer-wise entropy loss (i) = - Hi)) iteratively. At each step, a Clamp operation applied to the perturbed sample to keep it within the perturbation bound. Next, Democratic Training calculates the loss of clean and low entropy samples as below:
(6) where Leee represents the categorical cross entropy loss, í represents clean sample and ien rep- resents low entropy example generated. In Equation 6 (fLnce(een) aims to guide low entropy samples towards the correct prediction by minimizing their cross entropy loss and (1 Lose aims to keep the loss on clean samples low. Parameter a E (0, 1) controls the trade-off between the effectiveness of UAP removal and performance unperturbed inputs during the optimization pro- cess. To make sure the loss on the low-entropy samples is low, the model must learn to ignore those dominating features present in the low-entropy samples, i.c., learn to predict based on many features rather than a small number of dominating features. Democratic Training. Bacck-propggtion is adopted using the projected gradient descent (PGD) method (Madry et al., 2018). Finally, Demo- cratic Training returns the updated model parameter as the result. Different from exiting methods (either generate UAPs in prior or on-the-fly), Democratic Training does not rely on generating UAPs and are thus not limited to specific UAP attacks.
L(i, ien) = aLoce (ien) +1 Lcce (i)
The overall time complexity of Algorithm iis O(n-m m [Z]), where [[] is the size of the clean dataset used, n is the number of epochs to finetune and m represents number of iterations required to gen- erate low entropy samples. Although Democratic Training requires multiple iterations to transform clean samples into low entropy samples, converting clean samples into low entropy samples is much simpler than generating UAPs and the total amount of samples to transform (n [7]) does not de- pends the number of classes in given dataset since the Sample Generator does not require any information on the target class. This is a clear advantage over multiple existing UAP defense meth-
ods relying generating UAPs (e.g.. (Akhtar et al.) 2018; Mummadi et al. 2019 Borkar et al.) 2020) etc.), for which, perturbations are generated for each target class. For datasets that contains large number of classes (e.g., ImageNet dataset contains 1000 classes, JFT-300M dataset (Sun al., 2017) contains 18k classes), a large number of perturbations shall be generated order to achieve acceptable defense performance. Unlike these methods, Democratic Training transforms some clean samples into low-entropy samples and and it does not require the size of clean set to be large (S 5% of training set).
IMPLEMENTATION AND EVALUATION
In the following, we conduct multiple experiments to evaluate the effectiveness of Democratic Train- ing by answering multiple research questions (RQs). All experiments are conducted on a machine with 96-Core 1.4GHz CPU and 60GB system memory with NVIDIA 24GB RTX 4090 GPU. Our approach has been implemented as a self-contained toolkit Python and is open-sourced (https: //t::////itt ab. com http://niinnn///cccc/ccii...

---

Published as a conference paper at ICLR 2025
Table 2: UAP Defense Performance.
Table 1: Neutral Networks Used.
Model
Before SR
After
AAcc.
AAce.
SR
ACC
Time
Net
Datasct
Architecture
Acc
NN1
0.134
0.714
0.617
0.002
-0.02
8
2N
0.067
0.701
0.431
0.077
-0.04
NN;
ImageNNet
ResNet50
0.73
NN3
0.195
0.584
0.549
0.004
-0.03
7
N.
ImageNet
VGG19
0.70
NN.
0.035
0.997
0.894
0.004
-0.02
18
N.
ImageNet
GoogleNet
0.69
NN,
0.059
0.842
0.715
0.018
-0.01
30
N.
ASL
MobileNet
0.99
NN;
0.236
0.784
0.786
0.048
-0.01
16
NN.
CalTech101
ShufflcNetV2
0.85
NN; 0.154
0.933
0.860
0.031
-0.03
21
NNo
EuroSAT
ResNet50
0.89
Avg 0.126
0.794
0.693
0.028
-0.02
NN;
CIFAR-10
WideResNNt
0.93
We report attack success rate (SR), adversarial accuracy (AAcc.), change accuracy clean inputs (AC.Ace.) and execution time (Time) in minutes.
.NN,
N.N
NN
Figure 3: Change in layer-wise entropy.
4.1 EXPERIMENT SETUP
We conduct our experiments with 7 neural network models trained over benchmark datasets:
1) ImageNet (Deng et al. 2009) 2) ASL Alphaber (Sau 2018), 3) Caltech/01 (Li er al. (022),
4) EuroSAT [elber er al 2079 and 5) CIFAR-I0 (Krizhevsly 2009). Details can be found in Appendix 8.3 For experiments with the ImageNet dataset, we adopt the pretrained models from 2019). For experiments with ASI., 1a1tech101, EuroSAT and CIFAR-10 datasets, we train CNN models following standard model training process. Details of the models summarized in Table | When applying Democratic Training, we focus on the last pooling or dense layer for the entropy calculation since the effect of UAP layer-wise entropy is stronger in deep layers as shown in Section model enhancement.
PyTorch Paszke al.)
2 A small set clean data (S 5% of the training set) is used during the
4.2 RESEARCH QUESTIONS AND ANSWERS
RQI: Is Democratic Training effective in defending against UAP attacks?
For each neural network, we train eight UAPs for randomly selected targets. The details of the UAP attacks are summarized in Table 2 We systematically apply Democratic Training to all the above-mentioned models and return the repaired models N N1 to NN:
Firstly, we measure the layer-wise entropy of clean and UAP perturbed inputs on the repaired mod- els. Figure 3 shows the box plot for the layer-wise entropy of clean samples and UAP infected samples before and after applying Democratic Training. On average, across all original models and attack target classes, the layer-wise entropy difference between clean inputs and those perturbed by UAPs is 16.7%. After applying Democratic Training, such difference is reduced 0.2%. Thus, Democratic Training is able to reduce the effect of UAPs in terms of layer-wise entropy effectively.
Next, we show the change in UAP attack success rate (SR) and model accuracy (on clean inputs (Clean Acc.) and perturbed inputs (AAcc.)). As shown in Table on average, across all original models and attack target classes, the attack success rate reduced from 79.4% to 2.8% after apply- ing Democratic Training. In addition, the adversarial accuracy is improved from 12.6% to 69.39 69.3%. Hence, by reducing the effect of UAPs on wise entropy, the effectiveness of UAPs is reduced significantly. In terms of clean sample accuracy, it is minimally affected. On average, the model accuracy is reduced by about 2%. Thus, Democratic Training is able to focus on removing the effect of UAPs while the model functionality is maintained.
7

---

Published as a conference paper at ICLR 2025
Table 3: Performance of Democratic Training on UAPs generated using sPGD, LaVAN, GAP and SGA.
sPGD
LaVAN
GAP
SGA
Model
Hefore
After
Before
After SR
Before
After
Refore
After
AAce.
SR
AAcc.
SR
AAcc.
SR
AAcc.
AAoc.
SR
AAcc.
SR
AAce.
SR
|Acc.
SR
N.N,
0.231
0.594
11629
0.0017
0.152
0.790
0.677
0.005
(.2299
0.512
0.575
0.002
0.133
0.722
0.592
0.004
N.Nz
0.248
0,484
0.016
0.506
0.343
0.144
0.482
0.001
0.067
0,415
0.096
NNN5
0.388
0.281
0.592
0.000
0.986
0.348
0.621
0.002
0.226
0.460
0.614
0.900
0.147
0.640
0.510
0.010
N.N
0.045
0.980
0.981
0,000
0.537
0.271
0.678
0.033
0.031
0.921
0.984
0.034
0.034
0.999
0.904
0.009
0.270
0.008
0.725
0.106
0.020
NNo
0.401
0.375
0.861
0.022
0.224
0.743
0.925
0.004
0.274
0.757
0.900
0.106
0.227
0.776
0.811
0.033
Avg
0.273
0.497
0.734
0,009
0.232
0,601
0.685
0.014
0.172
0.632
0.713
0.025
0.119
0.790
0.662
0.029
Hence, answer RQI, Democratic Training able to reduce the attack success rate of UAP attacks and improve the robustness against adversarial samples effectively, and at the same time, the model accuracy is maintained at high level.
RQ2: Is Democratic Training effective in mitigating UAPs crafted from different attack methods?
There are many UAP generation algorithms proposed in existing works and we further evaluate Democratic Training against another four types of UAP attacks: 1) sPGD (Mummadi et al.) 2019)
which adopts PGD) to update the perturbations iteratively to generate UAPs, 2) LaVAN (Karmon) et al. 2018) which is proposed as a method to generate image-agnostic localized adversarial noise that covers only 2% of the image but fool the neural network, 3) GAP (Poursaeed et al., 2018) as generator-based UAP attack method which adopts generative model for crafting UAPs and 4) SGA (Liu et al.] 2023) which alleviates the gradient vanishing and escapes from poor local op- tima when generating UAP. For each method, we randomly select eight attack target classes and train UAPs for NN,, to NN, and evaluate their attack success rate and model accuracy before and after applying Democratic Training- The average results across all models and target classes summarized in Table 3 For all models, on average, sPGD attack achieves 49.7% targeted success rate and the adversarial accuracy is below 27.3%. LaVAN attack achieves 60.1% success rate with adversarial accuracy of 23.2%. GAP attack achieves 63.2% success rate and the adversarial accu- racy is below 17. 77. SGA attack achieves 79.0% success rate and the adversarial accuracy is below 11.9%. When tested on Democratic Training enhanced models, the adversarial accuracy is improved to 73.4%, 68.5%, 71.3% and 2% for sPGD, LaVAN, GAP and SGA attacks respectively. and the attack success rate is below 0.9%. 1.4%, 2.5% and 2.9%. Thus, models enhanced by Democrutic Training are robust against UAPs generated different ways. We further analyze the change in layer-wise entropy before and after applying Democratic Training. The results summarized in Appendix 8.6 These results suggest that, regardless of the generation method, the effect of targeted UAP model can be revealed by layer-wise entropy and such effect can be suppressed via our entropy based model enhancement effectively.
Thus, to answer RQ2, Democratic Training is effective defending UAPs generated with various algorithms.
RQ3: How does Democratic Training compare with adversarial training?
Adversarial training can be a useful method to improve model robustness against UAPs (Mummadi et al. 2019 Shafahi et al. 2020 Benz et al.) 2021). We evaluate the effectiveness of low-entropy samples and adversarial samples in finetuning a given model three settings: 1) non-targeted adver- sarial training, i.e., adversarial examples are not targeted and are generated on-the-fly 2) targeted adversarial training, i.e., adversarial examples are targeted and are generated on-the-fly and 3) fine- tuning with pretrained targeted UAP. While there are various options of adversarial training algo- rithm for the first two settings, we adopt PGD based adversarial training (Madry et al. (0188) as it provides a good trade-off between being computationally efficient and powerful (Mummadi et al,) 2019). We finetune the model with adversarial samples generated with the same number of iterations as in Democratic Training for a fair comparison. Furthermore, we assume the attack target class is known for aaggted-advvessaiill finetuning (which gives the defender some unrealistic advantage). For finetuning with pretrained UAPs, similarly we assume the target class is known and train a set of 10 UAPs to be used together with set of clean samples. During the finetuning, we add randomly chosen pretrained UAP to a clean sample with 50% probability. We keep the number of clean exam- ples used in finetuning the same as Democratic Training well. The average performance over all
8

---

Published as a conference paper at ICLR 2025
Table 4: Performance of adversarial training.
Table 5: Performance of existing methods.
Setting
AAcc.
SR
ACAce.
Method
AAcc.
SR
ACAcc.
Targeted
0.464
0.167
-0.104
SFR
0.468
0.011
-0.022
Non-targeted
0.295
0.455
-0.168
CFN
0.150
0.559
-0.073
Known UAP
0.476
0.223
0.0
FNS
0.149
0.623
-0.013
TRADES 0.816
0.022
-0.110
DensePure
0.802
0.010
-0.121
6 models is shown in T All three methods are not that effective in UAP defense, i... the attack success rate remains high (> 16%) and adversarial accuracy is lower than 50% after the finetuning. In comparison, Democratic Training is able to reduce the attack success rate < 3% and improve the adversarial accuracy to 69% on average. We believe this is due to the fact that adversarial train- ing aims direct adversarial examples towards their correct predictions while Democratic Training focuses on guiding low- low-entropy samples. Based on our experimental results, low-entropy samples are more efficient in guiding the model enhancement process.
Moreover, we evaluate the performance of a well-recognized adversarial training method TRADES (Zhang et al., (019) on UAP defense. As shown in the last row of Table 4 TRADES is effective in defending against UAPs but sacrifices model accuracy for over 10%.
Hence, to answer this RQ3, Democratic Training is more effective in defending against UAPs when compared to adversarial training with equivalent parameter settings.
RQ4: How does Democratic Training compare with other existing neural network UAP defense methods?
We further compare the performance of Democratic Training with four state-of-the-art UAP defense methods, i.e., selective feature regeneration (SFR) (Borkar et al.) (020), clipping feature norms (CFN) eYu al. (02I), feature norm suppressing (FNS) (Yu et al. (023) and DensPure (Xiao et al.) 2023). SFR is an approach proposed to defend against UAPs from feature-level. It deploys feature regeneration units in a given model aiming to transform vulnerable features into resilient features against UAPs. CFN is proposed based on the fact that universal adversarial patches usually lead to deep feature vectors with very large norms (Yu et al. 202T). It introduces a feature norm clipping layer to be inserted into the original model that aims to adaptively suppress the generation of large norm deep feature vectors. Similarly, FNS is designed on top of CFN which is able to renormalize the feature norm by non-inereasing functions. FNS can be adaptively inserted in to a given model to achieve multistage suppression of the generation of large norm feature vectors. No training is required for such feature norm suppressing layer. DensPure employs iterative denoising to an input image to get multiple reversed samples with different random seeds. Next, the samples are given to the model to make final decision via majority voting. The results of the average performance for SFR, CFN, FNS and DensPure are summarized in Table 3 Both CFN and FNS are not effective in defending against UAPs, i.e., the attack success rates remain above 50%. Thus, suppressing or clipping feature norms of given model has limited effect weakening the impact of targeted UAPs. Furthermore, both CFN and FNS modify the original neural network architecture by inserting an additional feature norm clipping / suppression layer. Although SFR achieves comparable UAP defense performance as Democratic Training, it modifies the architecture of the original model which is often not preferred in real-life application (as this might prolong development cycles and bring in integration challenges (Hutter et al.) ) Furthermore, SFR requires to pretrain 25 UAPs (and 2000 synthetic UAPs) to train the additional layers, which is rather time consuming (it takes > 40 min to train one UAP based on method proposed in (Moosavi-Deztooll et al. 2017) following open-source implementation]] while Democratic Training repairs the same model within 10 min). DensePure is effective in improving the model robustness against UAPs but model accuracy affected which drops by 12.1%. Moreover, it introduces overhead in inference time for reversed samples.
Hence, to answer RQ4, Democratic Training is more effective in mitigating the impact of UAPs on trained neural networks, which does not require to change the original model architecture
http::/:/hwa.ctttooooooooteetaee Uniiverallaadversarialppppn

---

Published as a conference paper at ICLR 2025
5 RELATED WORKS
Adversarial attacks. Neural networks are highly vulnerable to adversarial attacks, which are small, deliberately crafted perturbations to input data that can fool the model into making incorrect pre- dictions. Such perturbation can be 1) image-specific where the attacker computes perturbation for every clean input and 2) image-agnostic where a single perturbation will cause majority of clean samples to fool given model. In recent years, many input-specific adversarial attacks are proposed to generate disruptive perturbations, Gondfelloo et al. (2015) introduce Fast Gradient Sign Method (FGSM) that generates adversarial examples. Subsequently. Basic Iterative Method (BIM) is pro- posed (Kurakin et al., 2017) as an extension of FGSM which applies small FGSM steps iteratively aiming generate higher quality perturbations, MMddry al. (2018) propose PGD which optimizes the perturbation at each iteration based on gradient of loss function. Furthermore, Carliiii & Wagner (2117) propose C&W attack which formulates the adversarial example generation as optimiza- tion problem aiming at minimal perturbation. Together with many others. e.g. (Wang et al. 202
Wu et al. 2020: Zhong & Deng 2021; Dong et al.) 2219 WWng et al.] [0222] Peng et al. 202
adversarial attacks pose significant threat to real-world applications in different domains.
Universal adversarial attacks. Unlike per-instance perturbations, UAPs work for the majority of clean samples, i.e., adding a single perturbation to majority of clean samples, the neural network will response with incorrect predictions. Such attacks can be broadly classified in to noise-based and generator-based attacks. Noise-based attack methods directly train UAP that can be applied to all inputs while generator-based methods train an extra generative model as a bridge to craft he perturbation indirectly (Weng al. 2024). Moosavi-Dezfoolii et al. (2017) first explore the exis- tence of such input-agnostic adversarial perturbations. Furthermore, Khrulkov & Oseledets (2018) propose to craft UAPs by maximizing the difference between the activations of hidden layer for clean and perturbed inputs. Later on, many noise-based methods are proposed with good perfor- mance (Mopuri al. 2017 Zhang al. 2020b). On the other hand, Poourseed al. (2018) firstly apply generative model for crafting UAPs. NAG is proposed (Mopuri et al. 20188) with a novel loss function for training the perturbation generator. Beyond above mentioned methods, there are many other UAP attacks, e.g.. (Benz et al. 2020; Khrulkov & Oseledets 2018 /Zhang et al.) |2020a| Amaada al. 202T, Sun al., (002)). Compared to input-specific perturbations, UAPs are more efficient in terms of computation cost and become a more significant threat in practice.
Defense against adversarial attacks. Defense against adversarial attacks can be grouped into six domains (Costa et al.) 2024): 1) adversarial training which augments the training data with adver- sarial examples to make the model more robust (Goodfellow et al. 2015; Madry et al. 20188) Wong
cet al. 2020; Zhang et al. 2019 Mummaai et al. 2019 Chen et al. 2022). 2) modifying the train-
ing process which adjusts the traming process to improve robustness Papernot et al. 2016 Shafahi
2020 Huang et al., 2020; Chen & Lee 1021 Pang et al. 2022 Akhtar et al. 2018), 3) use of supplementary networks which add extra networks on top of the original model to remove the effect of adversarial perturbations (Liu al.
et al.
Liao al. 2018 Li et al. 2021 Abusnaina al. 2021 (020), 4) changing network architecture which modifies the architecture of the original model for robustness (Xie et al. 2019) Guo et al.) [2020] Xie al. 2018
2020
Ho & Vasconcelos] 2022 Borkar et al.
Atzmon et al., 2019 Yu et al., |021 2023). 5) performing network validation which validates and
certifies the robustness of a given model (Pei et al., 2019 Ma et al. 2018: Kim et al., 2019) and 6) adversarial purification which removes adversarial perturbations of input samples and recovers the clean image (Gowal al. [02TT: Ho et al.) 22200 Sehwag et al.) 2022 Nie et al., 2022. Xiao al.)
2023). Among them, there are multiple works proposed to defense against UAPs MMooavi-Dezzooli
ct al.
2017 Akhtar et al. 2018 Mummadi et al. 2019 Borkar et al. 2020
Yu et al. 202 2023).
6 CONCLUSION
In conclusion, we propose Democratic Training as an efficient and effective defense method against targeted UAP attacks for neural networks. Democratic Training first analyzes the layer-wise entropy to understand how UAP deceive the model and conducts entropy-based model enhancement to mitigate the effect of UAP. Our experimental results show that Democratic Training is effective in removing the effects of UAPs from given model and it outperforms existing oate-of-the-an UAP attack defense methods.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This research is supported by Singapore Ministry of Education under its Academic Research Fund Tier (Award ID: MOET32020 0004).
REFERENCES
Ahmed Abusnaina, Yuhang Wu, Sunpreet Arora, Yizhen Wang, Fei Wang, Hao Yang, and David Mohaisen. Adversarial example detection using latent neighborhood graph. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7687-7696, 2021.
Naveed Akhtar, Jian Liu, and Ajmal Mian. Defense against universal adversarial per- turbations. In 2018 IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3389-3398. Com- puter Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.201 00357. URL oppta::/ppenaccess..o twecff..ecc..ccccm com content_ _ecpe 2018/htm1/ Akhtar
fense Aww.eeeeeeeaaaaeeeeeeeeeeae
Uni wtw.iieeeeaa_eeee_0010 papppe htmpperrmmll]
Takuma Amada, Seng Pei Liew, Kazuya Kakizaki, and Toshinori Araki. Universal adversarial spoof- ing attacks against face recognition. In International IEEE Joint Conference on Biometrics, IJBB 2021, Shenzhen. China, August 4-7, 2021, pp. 1-7. IEEE, 2021. doi: 10. 0001009//CB53358222222 9484380. URL https: /oii .rgg//g/g. 0 1109/ JJCB5338 2021 9484380
Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman. Controlling neural level sets. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' AlechheBBu,, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 2032-2041, 2019. URL htttps://proceed:: weweeeddinnnnuuuereeeeeeeee wwwpaurips..cppppeeeeeeeee wccppapper/219//haaeee b2C 6d9 www.afddddaa99886eoe.eo.
Abstract html
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy-based logic explanations of neural networks. In Thirty-Sixth AAAI Confer- ence on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposiun Educational Advances in Artifi- cial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 6046-6054, AAAI Press, 2022. doi: 10.1609/AAALVV366622200 V3616. 20551
URL http::ttp///lttt http:::///o...og//11..15533/
Philipp Benz, Chaoning Zhang, Tooba Imtiaz, and In So Kweon. Double targeted universal adversar- ial perturbations. In Hiroshi Ishikawa, Cheng-Lin Liu, Tomás Pajdla, and Jianbo Shi (eds.), Com- puter Vision - ACCV 2020 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 December 4, 2020, Revised Selected Papers, Part IV, volume 12625 of Lecture Notes in Computer Science pp. 284 300. Springer, 2020. doi: 10.1007/978 0-077-88--30--538-551- https://doi. . .rg. 10. 1007/978 3-000-333--0000 69538-5_15
URL
Philipp Benz, Chaoning Zhang, Adil Karjauv, and So Kweon. Universal adversarial training with class- wise perturbations. In 2021 IEEE International Conference on Multimedia and Expo, ICME 2021, Shenzhen, China. July 5-9, 2021, pp. 1-6. IEEE, 2021. doi: 10.1109//CME512007 9428419. URL Inttps://don hww.ddo..o.....ooooooooooooooeoe 0 20009//MM511000 2021 9428419
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End end learning for self-driving cars. CoRR, 2016. URL/attp://arxit
org/ abs/ arrg/bs//0 604 07316
Tejas S. Borkar, Felix Heide, and Lina J. Karam. Defending against universal attacks through selective feature regeneration. In 2020 IEFE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2020, Seattle. WA. USA, June 13-19, 2020, pp. 706-716. Com- puter Vision Foundation IEEE, 2020. doi: 10.1009CVPR4260..200000000
URL
https:://ppnacceesss twe.vf.................. htt.aa/cccccccccooeeeetttttttttt hwtpp//w22/ee/////e22222

---

Published as a conference paper at ICLR 2025
Defending Aggendddnn__giistttt Universal Attacks Ve Featureeeeaeeeeeeee Regeneration, Cep_iioon__P__P__ 2020 paper. .tmml
Selective
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pp. 39-57. IEEE Computer Society, 2017. doi: 10.1 URL htttps://do
org/10. 1109/S2 2017.49
Raphael Labaca Castro, Luis Muñoz-Gonzälee Feargus Pendlebury, Gabi Dreo Rodosek, Fabio Pierazzi, and Lorenzo Cavallaro. Universal adversarial perturbations for malware. CoRR, abs/2102.06747,, 2021. URL htllhttss://aaaee httppss:/aaxxiv.or//a/.a.s 22tps///12..0000 06747
Erh-Chung Chen and Che-Rung Lee. Ltd: Low temperature distillation for robust adversarial train- ing. arXiv preprint
2021.
Jinghui Chen, Yu Cheng, Zhe Gan, Quanquan Gu, and Jingjing Liu. Efficient robust training via backward smoothing. In Thirty-Sixth AAAI Conference Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium Educational Advances Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pp. 6222-6230. AAAI Press, 2022. doi: 10.1609/AAAL V3666 20571. URL https: doi wrg. 10. 110.0.1.999// aaai. v3616..6..2... 20571
Joana Cabral Costa, Tiago Roxo, Hugo Proença, and Pedro Ricardo M. Inácio. How deep learn- ing sees the world: A survey on adversarial attacks & defenses. ICEE Access, 12:61113- 61136, 2024. doi: 10.1 D0.100/AACEESSS.22443399 ACCESS. 2024 3395118
URL https:://ai..oroe http///.....0/10.11100001
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009). 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE htRp:t/::: doi org/10.
Computer Society, 2009. doi: 10.
11C9/CVPR. 2009. 5206848
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp- 9185- 9193. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR. thevvf..teeeecc.ccc wwweeecccccooceeoooooemmeeeeeeee cot.c//ot///c_ccpprrrr.rcc...... 2018/htm1/
2018.00957. URL Dong Boost ing Adversarial LttecaalAAttcccae SecpicccssccPP_PP_ 2at.0188ppppeehtaaa pttpaaaaac.aaac........a.pa.a...
Yinpeng Dong, Hang Su, Baoyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu. Efficient decision- based black-box adversarial attacks face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 717777-7772 7722. Computer Vision Foundation / IEEE, 2019. doi: 10. 10.1109/CVPR.20199000 URL h
20t9/22t1//0/tmt thecvf. com coton//oonettt.m Batpateett_________eeeeee Cwtp__219/tta__eeeee Box Bowe-BoooAAvvveraaaa.... Altaallctstootonttooocoo
Sfficient_Deci.eee Recognition_cvP 2019 2aapa__200_pp_________ html
Based
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 1625-1634, 2018.
Kang Fu, Dawei Cheng, Yi Tu, and Liqing Zhang. Credit card fraud detection using convo- lutional neural networks. In Neural Information Processing 23rd International Conference (ICONIP 2016). Kyoto, Japan, pp. 483-490, 2016. URL https::/ /ttp//cc..oo..100000001 978-3-319-46675-1 53]
Aditya Ganeshan, Vivek B. S., and Venkatesh Babu Radhakrishnan. FDA: feature disruptive at- tack. In 2019 JEEE/CVF International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 8068 8078. IEEE, 2019. doi: 10. 1099/ 00816. URL https::tp//............ http:s//ddo. /tt. .rgg 00. 09/ TCCV, 2019. 00816
12

---

Published as a conference paper at ICLR 2025
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http:: httt:://axxiv.or///ea. / 1412. 6572
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A. Mann. Improving robustness using generated data. In Marc Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.). Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 4233,, 2021. URL ott.::/ceo...caoaooo.ooooaa.ooa.
neurips......................... wwtpaper///ppc 22wee//2221//2 1/hash/
21
Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 631-640, 2020.
Jamie Hayes and George Danezis. Learning universal adversarial perturbations with generative models. In 2018 IEEE Security and Privacy Workshops, SP Workshops 2018, San Francisco, CA, USA, May 24, 2018, pp. 43-49, IEEE Computer Society, 2018. doi: 10.1 10.100//PW.000.00000
URL https::t//// //doi org 11t.://..1111//.11. 11t.:1119//PPPP 2018. 00015
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Chih-Hui Ho and Nuno Vasconcelos. DISCO: adversarial defense with local implicit functions. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL httt:tp://papers nips. wttpppppppppppppppppppppppppeeep 2ttp:pppper...lleeeepee///22222/ httppp/attttthhhhh/h
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http:///ooeedddeeeeeee neur $c5bcfec8584a 400d9667 ab - html
neur ips. cC/ /p0p/////////2hash
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems, 33:19365-19376, 2020.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning Meth- ods, Systems, Challenges. The Springer Series on Challenges in Machine Learning. Springer, 2019. ISBN 978 978-3-030-05317-88 doi:
URL htttps://doo
org/10. 1000010007778880
05318-5
Danny Karmon, Daniel Zoran, and Yoav Goldberg. Lavan: Localized and visible adversarial noise. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmüsssn, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2512-2521 PMLR, 2018. URL agoooeedings.oo mlr. press/ vBOss/vB//e iaaamoonn 8a html
Valentin Khrulkov and Ivan V. Oseledets. Art singular vectors and universal adversar- ial perturbations. In 2018 IEEE Conference Computer Vision and Pattern Recogni- tion, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pp. 8562-851 8570. Com- puter Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.201 00893. URL Khrulkov _tt____t__oo Sart_of_Singuler.te
thecvf..ttee....cc com www/conteet__epp..tc.cccoootoeee 2018 8/htm1/ 201 1a1888_ppper__e html
13

---

Published as a conference paper at ICLR 2025
Jinhan Kim, Robert Feldt, and Shin Yoo. Guiding deep learning system testing using surprise adequacy. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.). Proceedings of the 41st International Conference Software Engineering. ICSE 2019, Montreal, QC, Canada. May 25- 31, 2019, pp. 1039-1049. IEEE/ ACM, 2019. doi: 10.1 11100/9SEE222190000006 URL https:
7/doi. 0rg/0.0rg/100 100 1109/ TCSE. 2019 2019000188
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL /tLL:ttts:///w..cccoom www.::/wwwww..rooo....t.w... www.dddd/d.iiaaaaaeaaeee html
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL /tttps:://ppnneeiier net/ pett/oose
Fei-Fei Li, Marco Andreeto, Marc' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022.
Yao Li, Martin Renqiang Min. Thomas Lee. Wenchao Yu, Erik Kruus, Wei Wang, and Cho-Jui Hsich. Towards robustness of deep neural networks via regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7496-7505, 2021.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1778-1787, 2018.
Hong Liu, Rongrong Ji, Jie Li, Baochang Zhang, Yue Gao, Yongjian Wu, and Feiyue Huang. Uni- versal adversarial perturbation via prior driven uncertainty approximation. 2019 IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pp. 2941-2949, 2019.
Shuqi Liu, Mingwen Shao, and Xinping Liu. Gan-based classifier protection against adversarial attacks. Journal of Intelligent & Fuzzy Systems, 39(5):7085-7088 2020.
Xuannan Liu, Yaoyao Zhong, Yuhang Zhang. Lixiong Qin, and Weihong Deng. Enhancing gen- eralization universal adversarial perturbation through gradient aggregation. IEEE/CVF In- ternational Conference Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 4412 4421. IEEE, 2023. doi: 10. 10010099CCC55100022223000 1109/ ICCV 2023 00409
URL htttps://///t o
Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. Deepgauge: multi multi- granularity testing criteria for deep learning systems. In Marianne Huchard, Christian Kästner, and Gordon Fraser (eds.), Proceedings of the 33rd ACM/IEEE International Conference on Automated Software En- gineering, ASE 2018, Montpellier, France, September 3-7, 2018, pp. 120 131. ACM, 2018. doi: D0.1145/323814..0 37.322202. URLpttps:: /dottp:://oi.... org./0.......oo..oooooo//.111/.o 11 / / 323814338777 22213333111111222222222 3238202
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track P Proceedings. OpenReview.net, 2018. URL htttps://pperrevieees forum? 22
Jan Hendrik Metzen, Mummadi Chaithanya Kumar, Thomas Brox, and Volker Fischer. Universal adversarial perturbations against semantic image segmentation. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 pp. 2774-2783 IEEE Com- puter Society, 2017. doi: 00.1109/CCV.2017..300
URL https:/ /ttl:///o....oo .rr.oorg//1
ICCV. 300
Seyed-Mohsen Goossvi-Dezzooll Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA. July 21-26, 2017, pp. 86-94 94, IEEE Computer Society, 2017. doi: 10.1109/CVPR.2007.11 URL httpt: /oi org 1119//.....11./........ 2017.11
14

---

Published as a conference paper at ICLR 2025
Konda Reddy Mopuri, Utsav Garg, and Venkatesh Babu Radhakrishnan. Fast feature fool: A data independent approach to universal adversarial perturbations. In British Machine Vision Conference 2017, BMVC 2017, London, UK, September 4-7, 2017. BMVA Press, 2017. URL fittps://wer wwwpbox..b.oooooooo com/ 8 q8 Puak 9ww35tkkk / /0058.0...... pdd
Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and R. Venkatesh Babu. NAG: net- work for adversary generation. In 2018 IEEE Conference on Computer Vision and Par- tern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, Pp. 742- 751. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00084. WtRLhttp::// hptpa////sppccccssssss thecch..cccc.......e cto///cccoo//coc/ccooo content_cvpr (atpeeen__cpr__cB11//_ 10tppp_211//tm111 Mopuri NAG Network for_ 2018 paper. html
Chaithanya Kumar Mummadi, Thomas Brox, and Jan Hendrik Metzen, Defending against universal
perturbations with shared adversarial training. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 4927- 4936. IEEE, 2019. doi: 10. N0.110//CCV.201900000
URL https:///cii org 10 1109/
ICCV. 2019.2019.000 00503
Muzammal Naseer, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. On generating transferable targeted perturbations. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pp. 7688 7697. IEEE, 2021. doi: (0.1109/CCCV49922.22222 200.922220020007766 URL https: //tps::://doi /eii org 10. 1109/
ICCV48922 2021. 00761
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandku- mar. Diffusion models for adversarial purification. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 16805-16827. PMLR, 2022. URL https: 7/proceedings wtwp:///www.......e.esss.s.....e
www.ppresssvv... wwtp///2ll//ne.a.a.maaaaaa www./a..eaaa...aa..a html
Tianyu Pang, Min Lin, Xiao Yang. Jun Zhu, and Shuicheng Yan. Robustness and accuracy could
be reconcilable by (proper) definition. International Conference on Machine Learning, pp- 17258-17277. PMLR, 2022.
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. IEEE Symposium on Se- curity and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016, pp. 582 IEEE Computer Society, 2016.
doi: 10. 109/SP.2016.411 URL httph:tt////..... http::://dooiooo http:////o.o.....o./oo///. 1109/SP 2016.
41
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetoto Martín Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. arXiv preprint arXiv: 19 1912017703 2019.
Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: automated whitebox testing of deep learning systems. Commun. ACM, 62(11):137-145, 2019. doi: 10.1145/336156.. URL
hhttps://doi.org/// 1tte://....1133111111
Wenyu Peng, Renyang Liu, Ruxin Wang, Taining Cheng, Zifeng Wu, Li Cai, and Wei Zhou. En- semblefool: A method generate adversarial examples based on model fusion strategy. Comput. Secur. 107:102317, 2021. doi: 10. 10.1016J..OOS..201... 102317. URL https://doi.org
10. 10.1016/5. 2021. 102317
Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge J. Belongie. Generative adversar- ial perturbations. In 2018 IEEE Conference Computer Vision and Pattern Recognition,
CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018 pp. 4422 4431. Computer Vi- sion Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.200 201 200200081.0055. http://open.ccss.taaer.
URL
GeaeerativeAdvversee ctw.r/aarraaaeeeetrriiiiiieeese
www.pppppp.............. aaper.hte html
15

---

Published as a conference paper at ICLR 2025
Debashish Sau. Asl alphabet, 2018. URL /ttptttss//ww..aaooe....///aa/a
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), Boston, MA, USA, pp. 815-823, 2015. URL EVPR.
Vikash Schwag, Saced Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang. Mung Chiang, and Prateck Mittal. Robust learning meets generative models: Can proxy distributions im- prove adversarial robustness? In The Tenth International Conference on Learning Represen- tations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https:
//openzeview. hetpi/ewweeeeeeee.eeee..ee.eee.e wott///fommiiiiom WW WVXONNVBBKV
Ali Shafahi, Mahyar Najibi, Zheng Xu, John P. Dickerson, Larry S. Davis, and Tom Goldstein. Universal adversarial training. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Inteligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. v34i04.6017. WtRL:ttts::///
AAAI Press, 2020. doi: 10.1609/aaai.
org 10. www.16109aaaaaa v34104.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime:
Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 acm sigsac conference on computer and communications security. pp. 1528-1548h 2016.
Jin W Shin and Sang Joon Kim. mathematical theory of communication. University of Ulinois Ress, 1949.
Dawn Song, Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, and Tadayoshi Kohno. Physical adversarial examples for object detectors. In 12th USENIX workshop on offensive technologies (WOOT 18), 2018.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable fectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 843 2017.
Zheng Sun, Jinxiao Zhao, Feng Guo, Yuxuan Chen, and Lei Ju. Commanderuap: practical and transferable universal adversarial attacks on speech recognition models. Cybersecur. (1):38, 2024. doi: 10.11866542400-0244000222 s 242400-024-00211
URL httpps://oii. http://ii...r/00111111//1
Sandra Vieira, Walter H.L. Pinaya, and Andrea Mechelli. Using deep learning to investigate the neu- roimaging correlates of psychiatric and neurological disorders: Methods and applications. Neu- roscience & Biobehavioral Reviews, 74:58-75, 2017. URL htttps::////tt /ttp://:///a...0/1110110.0. 10161
j.reubiorev. 2017 01.00..0000 002]
Eric Wallace, Shi Feng. Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversar- ial triggers for attacking and analyzing NLP. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing, EMNIP-IICNILP 2019, Hong Kong, China, November 3-7, 2019, pp. 2153-2162. As- sociation for Computational Linguistics, 2019. doi: 10.18653/V1/D19- 1221. URL https:
www..1163/vvvbb 1221
Weitao Wan, Jiansheng Chen, Tianpeng Li, Yiqing Huang, Jingqi Tian, Cheng Yu, and Youze Xue. Information entropy based feature pooling for convolutional neural networks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2019, pp. 3404-3413, IEEE, 2019. doi: N0.110//CC2.2209.00000 URL https: 7/doi. wlo.oogg1100. 2W/////CC/CVV 2019. 00350
Qian Wang, Baolin Zheng, Qi Li, Chao Shen, and Zhongjie Ba. Towards query-efficient adversarial attacks against automatic speech recognition systems. IEEE Trans. Inf. Forensics Secur, 16:896- 908, 202 doi: 1011110019S.200000000000 D http::////ob...og//00111/////.. 2020 3026543
URL http::ttps///t
16

---

Published as a conference paper at ICLR 2025
Zhen Wang, Yitao Zheng, Hai Zhu, Chang Yang, and Tianyi Chen. Transferable adversarial exam- ples can efficiently fool topic models. Comput. Secur. 118:102749, 2022. doi: 10.1016J.COSE. 2022.102749.
cose. 2ttp://...oooe.2..1222222 107799
Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature importance-awee aware transferable adversarial attacks. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 7619-7628 IEEE, 2021b. doi: 10. 11910//C48892220210000000 URL. https: //oci TCCV48922. 2021 00754
Juanjuan Weng, Zhiming Luo, Dazhen Lin, and Shaozi Li. Comparative evaluation of recent universal adversarial perturbations in image classification. Comput. Secur, 136:103576, 2024. doi:
103576. URL htt.https:///
1/.1111166.... 6/ cose. 2023.
103576
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htttt:://pppneeeee//s
net forum?id
BUX040EFVE VF
Junqi Wu, Bolin Chen, Weiqi Luo, and Yanmei Fang. Audio steganography based on iterative adversarial attacks against convolutional neural networks. IEEE Trans. Inf. Forensics Secur, 15: 2282 2294, 2020. doi: 10. 10.1119//TFS 296977963764. URL htttpts:://// wtt.:/ei...r//...........00 1109/
TIFS. 2019. 2963764
Chaowei Xiao, Zhongzhu Chen, Kun Jin, Jiongxiao Wang, Weili Nie, Mingyan Liu, Anima Anand- kumar, Bo Li, and Dawn Song. Densepure: Understanding diffusion models for adversarial robustness. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htt/ttts::// neet Forum?id=p Serummid=p/hvOJJ6@@0
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan L. Yuille. Mitigating adversarial effects through randomization. In 6th International Conference on Learning Representaatinss ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:: /tttps://openeevieet wtt.:///oooommmddd/slyyeeee
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 501-509, 2019.
Cheng Yu, Jiansheng Chen, Youze Xue, Yuyang Liu, Weitao Wan, Jiayu Bao, and Huimin Ma. Defending against universal adversarial patches by clipping feature norms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. (6434-16442 2021.
Cheng Yu, Jiansheng Chen, Yu Wang, Youze Xue, and Huimin Ma. Improving adversarial robust- ness against universal patch attacks through feature norm suppressing. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Chaoning Zhang, Philipp Benz, Tooba Imtiaz, and In-So Kweon. CD-UAP: class discriminative universal adversarial perturbation. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Secom Innovative Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 6754-6761 AAAI Press, 2020a. doi:
htthstss:: //ottp:://oi.... org. 10. 1 1609 hel/aaaii.em awa...33i0i...... www.04....... 6154
Chaoning Zhang, Philipp Benz, Tooba Imtiaz, and In So Kweon. Understanding adversar- ial examples from the mutual influence of images and perturbations. Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 14518. Computer Vision Foundation IEEE, 2020b. doi: 00.1099/CVPR42600.2000..111 content_cVEP 2020/htttttttt CBA_e___CCERR222//mmmmm/ From_the_Me chh__utualllllc vence. Tmages CVPR 2020 B0c.000pppee____ html
2020 IEEE/CVF
URL
www.theeeeecee com/
Zhang
Examples
17

---

Published as a conference paper at ICLR 2025
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Confer- ence on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, vol- 97 of Proceedings of Machine Learning Research, pp- 7472-7482. PMLR, 2019. URL
http:::/proceediinge mlr. wwwssressss.eee /97 2017/2haae w whwngl9p.htseeee.o. 1tml
Jaaniing Zhang, Weibin Wu, Jen-tse Huang. Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R. Lyu. Improving adversarial transferabilityy via neuron attribution-ben saariod attacks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 14973-14982. IEEE, 2022. doi: 10.110//CYPS22288.22222 01457. / http:///....111/////eeeee
doi /ttp:///////ooooo.//////////////
2022. 01457
Yaoyao Zhong and Weihong Deng. Towards transferable adversarial attack against deep face recog- nition. IEEE Trans. Inf. Forensics Secur, 16: 146614452466666 2021. doi: 10. 1010//0FS..000.33300000 URL
2020. 3036801
Alon Zolfi, Shai Avidan, Yuval Elovici, and Asaf Shabtai. Adversarial mask: Real-world uni- versal adversarial attack on face recognition models. In Massih-Reza Amini, Stéphane Canu, Asja Fischer, Tias Guns, Petra Kralj Novak, and Grigorios Tsoumakas (eds.). Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2022, Greno- ble, France, September 19-23, 2022, Proceedings. Part III. volume 13715 of Lecture Notes in Computer Science, pp. 60443200 Springer, 2022. doi: 10. 100000007788-3-01--66699999 https: www.ddi..rggggoooo 1007 978 00000079778--0011224-----2--
19. URI.
APPENDIX
8.1
FUTURE WORKS
In our future works, would like extend Democratic Training other model architectures (e.g., transformer based models) and non-vision tasks (e.g.. language models, audio tasks). Moreover, would like integrate Democratic Training with adversarial training, i.e., apply low-entropy samples in adversarial training. We would like to explore if the performance can be further improved and whether the method be extended to other types of adversarial attacks.
8.2 MEASURING ENTROPY NEURAL NETWORKS
Barbiero et al. 2022 proposed apply entropy-based layer to conduct logic explanations of neural networks. For a concept-based classifier where human-understandable input concepts are mapped to output predictions, the relevance of an input concept to a prediction class i can be approximated by the weight connecting jth input to its class embedding, i.c.,
3
| 11
(7)
35
Σ e
where W' represents the weight matrix and 7 is a user-defined temperature parameter tune the softmax function. The entropy of distribution 8x
H(S)) = Σ 83 log 53
(8)
minimized when single input concept dominates the prediction and it is maximized when all concepts are equally important.
Wan et al. (2019) proposed entropy- pooling for CNNs that helps the network concentrate on semantically important image regions. In CNN architecture, a global averaging pooling (GAP) layer is typically connected fully connected (FC) layer with softmax activation to produce the
18

---

Published as a conference paper at ICLR 2025
class scores. The input to GAP layer is the last convolutional feature maps U Rixxxe consisting of local feature vectors U E Ri = 1,2, hw, and the final prediction scores are computed as
fGAr(U) hw Σ 2
F --
(9)
1 WT lew
The entropy of the localized class probability for location i is then measured by
P: = softmax (W2
((m)) = Σ pi log
(10)
where TT,, E RK For feature location i, if its receptive field is centered on specific object, the localized class prediction of V should probably be highly confident leading to a low entropy value measured using Equation TO] Otherwise, if its receptive field is centered image textures or patterns that frequently occurred in other image classes, the corresponding entropy should generally be high (Wan et al. 2019).
8.3 DATASETS USED IN OUR EXPERIMENTS
ImageNet (Deng et al. 2009: The ImageNet 2012 dataset, also known the ILSVRC 2012 (ImageNet Large Scale Visual Recognition Challenge), is a large-scale dataset used for visual object recognition tasks. contains over 1.2 million images for training, 50,000 for validation, and 100,000 for testing. There are 1,000 different classes, which include wide variety of objects, animals, and scenes. Each class has hundreds thousands of images. We focus on image classification task in this work.
ASL Alphubel (Sau 2018): This dataset is collection of images of alphabets from the American Sign Language. consists of 87K 200 x 200 images of 29 classes, including 26 letters (A to 7) and 3 classes for "SPACE", "DELETE" and "NOTHING". The task is to identify the 29 alphabets.
Caltech101 [Li et al. 2022): This dataset contains of 9k pictures of objects belonging to 101 categories. There are 40 to 800 images per category. Images are of variable sizes with typical edge lengths of 200 to 300 pixels. The task is to recognize the 101 different objects.
EuroSAT (Helber et al. 2019): This dataset is a benchmark dataset in the field of remote sensing and geospatial analysis for the classification of land use and land cover from satel- lite imagery. It contains 27k 64 x 64 labeled images of 10 different classes representing various land use and land cover types, including: forest, highway, river etc. The task is to classify the land usage types.
CIFAR-10 10 (Krizhevsky 2009): This dataset is widely used benchmark dataset for image classification in machine learning. contains 60k color images, each with resolution of 32x32 pixels. The task is for image recognition of 10 catagories.
8.4 PERFORMANCE ON UAPS GENERATED WITH DIFFERENT
To further evaluate Democratic Training. we generate UAPs with different settings. We train UAPs with e 5/255 and eight with < 15/255 255 for NN, with the same set of target classes selected in RQI and evaluates the attack success rate and model accuracy on the Democratic Training repaired model (repaired with
10/25 10/255). The average results are summarized in Table [ For a smaller perturbation budget (e 5/ 5/255) the repaired model stays robust against the generated UAPs. The attack success rate is below 1% for all targeted classes. For a larger perturbation budget (e is larger 15 15/255, the repaired model still
than the value used during the finetuning process) where remains robust to a certain level. The average attack success rate drops from 91.3% to 11.5%. For the adversarial examples to be human-imperceptiblee the shall not be large. Hence, by setting
19

---

Published as a conference paper at ICLR 2025
Table 6:
Performance of Democratic Training on UAPs generated with different
Table 7: Adaptive attack performance.
Model
AAcc.
SR
Before AAcc.
After
NNÇ
0.480
0.115
SR
AAcc.
SR
NN
0.344
0.288
NN;
0.491
0.058
5/255
0.468
0.274
0.699
0.000
NN
0.655
0.174
10/255
0.134
0.714
0.617
0.002
N.
0,385
0.409
15/255
0.027
0.913
0.364
0.115
NONG
0.559
0,369
reasonable value, e.g., 10, the Democratic Training repaired model will be robust against UAPs generated with various perturbation budgets.
Thus, Democratic Training repaired model stays robust against UAP attacked samples generated with different perturbation budgets (e).
8.5 ADAPTIVE ATTACKS
In this section, we evaluate Democratic Training two types of adaptive UAP attacks: 1) secondary white-box attacks, where the attacker has full access to the Democratic Training repaired model, and 2) advanced attacks where the attacker is capable of tailoring the UAP trying to bypass our defense.
Firstly, for secondary attacks, for pretrained model NN, to N No described Table [ we apply Democratic Training to repair it as in RQI to mitigate the effect of UAPs and obtain the repaired models. Next, we apply the method DF-UAP proposed in (Zhang et al. (020b) on all the repaired models (NI), NN NN, NN1. NN and NN..) We keep all the attack parameters the same as the initial attack including the attack target classes. The secondary attack performance is show in Table 7 As described in Section 4 before applying Democratic Training, the UAP attack (Zhang et al. 2020b) can easily achieve an average of 81.3% targeted attack success rates and 14.7% adversarial accuracy. After applying Democratic Training, a subsequent attack can achieve an average attack success rate of 23.6% (with highest attack success rate of 40.9% on N N₂ and lowest of 5.8% on N N2). The average adversarial accuracy on the sub- sequent attack is 48.6%. Furthermore, we apply sPGD and GAP attacks on NN, as well. Adaptive sPGD is able to achieve 16.5% attack success rate and 48.7% adversarial accuracy. Adaptive GAP only manages to achieve an average success rate of 1.1% and the adversarial accuracy stays above 50%. Hence, similar to DF-UAP, subsequent UAP attacks such as sPGD and GAP are no longer effective on Democratic Training repaired models. Based such result, we believe that, UAPs exploit large correlations and redundancies in the decision boundary of a given model (Moosavi- Dezfooli et al., (2177). Democratic Training is able to reduce such correlations and redundancies so that it is much more difficult to find highly effective UAPs on the Democratic Training enhanced models.
generate
sets of UAPs accordingly.
Thus, although secondary UAP attacks on Democratic Training repaired models can still generate UAPs that successfully fool the models, our defense keeps the secondary attack success low level while keeping the adversarial accuracy high. Hence, based on above results, Democratic Training repaired model able to stay robust against adaptive UAP attacks.
very
Secondly, for advanced attacks we conduct experiments such that when generating UAPs, the at- tacker further controls the change in layer-wise entropy. Based on DF-UAP, the optimization loss function used for crafting an UAP is modified as:
L(i) (1 p) Lene ve) p H(()
(11)
where i represents training sample, Ve represents the attack target class and H(t) represents the layer-wise entropy loss for i. We use H(i) to control the entropy change caused by the UAP and parameter p is used to control the importance of H()) over attack success rate. We conduct such advanced attack on model NN, to N Nr with P set to 0.1 to 0.9, All models show similar results and for illustration purpose, results on NN are summarized Table 3 Increasing causes the attack performance to drop, i.c., the attack success rate starts drop when below 60% when p = 0.9. Our defense stays effective across different settings where the attack SR is reduced to 1 % for all scenarios. Hence, knowing how Democratic Training enhance the
0.5 and the attack SR is
20

---

Published as a conference paper at ICLR 2025
Table Advanced attack performance on N N1. We report the Adversarial accuracy (AAcc.), attack success rate (SR) and layer-wise entropy (Entropy) of UAP infected samples. Note that clean sample layer-wise entropy is 7.1
Before
After
AAcc.
Entropy
AAcc.
Entropy
0.118
0.764
5.62
0.619
0.001
7.39
0.121
0.775
6.07
0.619
0.001
7.39
0.2
0.118
0.759
6.29
0.619
0.001
7.40
0.3
0.128
0.7644
6.35
0.613
0.001
7.39
0.4
0.127
0.761
6.89
0.612
0.001
7.40
0.759
7.07
0.608
7.39
0.6
0.141
0.745
7.13
0.618
0.0
7.39
0.7
0.161
0.693
7.16
0.599
0.001
7.39
0.8
0.185
0.657
7.33
0.609
0.001
7.40
0.207
0.568
0.628
sPGD
LaVAN
GAP
SGA
Figure 4: Change in layer-wise entropy of other UAPs.
model and control the change in layer-wise entropy during attack process, the adversary still not able to bypass defense effectively.
8.6 ENTROPY ANALYSIS ON OTHER UAPS
As part of RQ2, we further analyze the change in layer-wise entropy of clean and UAP infected samples for other types of UAP attacks, i.e., sPGD, LaVAN, GAP and SGA. The results are summa- rized in Figure which show that similar to DF-UAP, UAPs generated with above mentioned four methods also cause the layer-wise entropy to drop and Democratic Training is able to mitigate such effect effectively.
8.7 NON-TARGETER UAP ATTACKS
We further evaluate Democratic Training on non-targeted UAP attacks. We generate non-targeted UAPs following DF-UAP for NN, N Ng and report the adversarial accuracy and attack success rate on original models (N to N.)) and repaired models (NN) to NN For non-targeted attacks, the attack success rate (SR) is calculated SR Σxex ((++)))+N+a) where E X
represents a clean sample, is the UAP. The results are summarized in Table
Although not designed for non-targeted UAPs, Democratic Training manages to reduce the attack SR from over 90% to 30% on average. This is indeed not as effective targeted UAP defense and we believe this is due to the different entropy spectrum caused by the two types of UAPs. Figure 5 shows the entropy spectrum of clean and non-targeted UAP infected samples for N₂ where no clear separation of the two observed.
21

---

Published as a conference paper at ICLR 2025
Table 9: Performance on non-targeted UAP attacks.
Before
After
Model
AAcc.
SR
AAcc.
SR
ACc.c.
N
0.057
0.939
0.594
0.267
-0.047
N 0.056
0.943
0.369
0.559
-0.066
N
0.098
0.888
0.469
0.408
-0.035
NN
0.002
0.981
0.918
0.066
-0.031
NN;
0.053
0.958
0.607
0.374
-0.019
NN:
0.289
0.737
0.801
0.129
-0.008
Avg
0.092
0.907
0.626
0.300
-0.034
Laydtwiee Ensopy
Sample wil UAP a Sample with t1AP
Autii wnii Middle Layer
ni aa Middlle Layer
Shilkow Layer
Shallow Layer?
Deep Layer
Deep Layer?
Figure 5: Layer-wise entropy of N N2.
22


Reviews:
[{"review_id": "5NVRLKSgFI", "reviewer": "Reviewer_xSaF", "summary": "The paper presents an investigation into the defense against universal adversarial perturbations (UAPs), with a particular focus on targeted UAPs. The authors have made a notable observation regarding the entropy spectrum in hidden layers when UAPs are introduced, which serves as the cornerstone for their proposed 'democratic training' approach. This method aims to enhance the adversarial robustness of neural network models. The empirical results provided in the paper demonstrate the efficacy of the approach, as well as its ability to maintain the clean accuracy of the model.\n\nIn general, this paper is well-structured and presents a novel approach, which meets the basic criteria of ICLR conference. However, there are some aspects that could be improved or expanded upon to enhance the overall quality and impact of the paper.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy-to-follow.\n2. The paper makes a commendable observation concerning the entropy spectrum in deep neural network layers, which is a significant contribution to the field and forms the basis for the proposed defense mechanism.\n3. The efficiency of the proposed democratic training method is noteworthy. It circumvents the need to generate UAPs during training, instead utilizing a limited number of epochs to identify low-entropy examples, which is a resourceful approach.", "weaknesses": "1. The threat model employed in the experiments primarily utilizes gradient-based attack methods. These methods presuppose access to the model's parameters, aligning with white-box attack scenarios. This appears to be at odds with the assertion in Section 2.3 that adversarial knowledge does not extend to the internal parameters of the model. Clarification on this point would be beneficial.\n\n2. The comparison with adversarial training methods may require further refinement. Adversarial training aims to bolster adversarial accuracy by integrating adversarial examples with clean examples during training. Constraining the number of training epochs could result in an underfit model, which may not provide a fair benchmark. Additionally, it would be advantageous to include a comparison with the widely recognized TRADES method[1], which is absent from the current manuscript.\n\n3. The potential for adaptive attacks warrants consideration. If adversaries are aware of the defense strategy, they could tailor adversarial examples to bypass the defense. I know that in the threat model, no adaptive attacks are considered since the attackers do not know the internal details of the models. However, the chosen attack methods in the experiments inherently rely on gradient information. So I would suggest that the authors should consider the potential for adaptive attacks.\n\n4. The scope of the experiments is largely limited to datasets comprising natural images. It would be beneficial to extend the evaluation to smaller-scale datasets, such as CIFAR-10, to complement the findings and potentially leverage open-source robust models for further exploration of the neuron entropy spectrum concept.\n\n5. While the paper discusses various existing defensive methods against UAPs and includes experimental comparisons, a direct comparison with state-of-the-art methods is missing. It is recommended to condense the background section and incorporate a more thorough comparison with leading-edge techniques.\n\n6. Minor Issues\n  (1) Please consider to reduce the margin between Figure 1 and the text.\n  (2) Suggesting to add necessary notations (SR) from the main test to the Table 2 for better understanding.\n\n\n[1] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" International conference on machine learning. PMLR, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WWJR8C7Nld", "reviewer": "Reviewer_1Bzp", "summary": "The paper presents a novel defense method called Democratic Training to mitigate the impact of Universal Adversarial Perturbations (UAPs) on deep neural networks.  The authors observed that UAPs lead to an abnormal entropy spectrum in hidden layers, which shows the model's prediction is dominated by a small subset of features. Democratic Training mitigates this issue by increasing entropy to ensure model predictions rely on a wider range of features. The approach was evaluated on multiple models and datasets and was found to be effective in reducing attack success rates while preserving accuracy on clean data.", "questions": "1. How about performances of the method on ViT?\n2. What's the time cost of the method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of entropy to reveal the dominance of UAPs and the concept of Democratic Training as a defense mechanism is innovative.\n2. The method was evaluated across various neural network architectures and benchmark datasets, which strengthens the claim of its general applicability.\n3. Unlike other defense methods, Democratic Training does not require architectural modifications, which makes it easy to integrate into existing systems", "weaknesses": "1. The evaluation focused primarily on benchmark datasets and common UAP generation methods. It would be beneficial to see how this approach performs on more sophisticated and adaptive attacks, such as adversarial examples generated in dynamic environments.\n2. The proposed method mainly works well on CNN. Authors should validate it in more types of networks, such as transformers.\n3.  The method requires access to a small set of clean data for entropy measurement and training, which might not always be practical", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PRbnLkYBZH", "reviewer": "Reviewer_hAa5", "summary": "To improve neural network robustness to targeted UAPs, this paper proposed an adversarial training-like method that fine-tunes a pretrained model to reduce middle-layer entropy.", "questions": "1. For RQ3: did you adversarially train a model from scratch or just fine-tune a pretrained model with an adversarial training objective?\n2. I'm not very convinced by the claim in Lines 239-243 that says middle-layer feature entropy suggests the model's classification confidence.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experiments are comprehensive.\n2. The proposed defense is attack-agnostic which is more practical and efficient.\n3. The proposed defense largely reduced the targeted attack success rate.\n\nI tend to accept this pape. However, since I'm not familiar with UAP attack and defense baseline methods, I will listen to other reviewers and public comments and then decide.", "weaknesses": "1. UAP attacks evaluated in the paper were published in 2018,2019,2020 and seem out-of-date.\n2. After democracy training, there is still a gap between ``AAcc.'' and clean accuracy. I wonder about the effectiveness of democracy training against non-targeted UAPs.\n3. Average results in Table 4\\&5 are ambiguous since there can be a large bias among different networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H808TvFjr8", "reviewer": "Reviewer_etKh", "summary": "The author aims to improve the robustness of universarial adversarial by fine-tuning with a small amount of data, mainly by performing entropy based data augmentation to suppress the influence of general adversarial perturbations in a given model. Then the experiment was presented.", "questions": "1: The author obtained the idea of Democratic Training by studying the performance of UAP between different layers in network. I want to know why author said 'Democratic Training does not rely on generating UAPs and are thus not limited to specific UAP attacks.', As mentioned in the article, the UAP analyzed by the author is produced based on FA-UAP, so the properties analyzed should mainly for such kind of UAP, and the algorithm guided by this should also mainly target on FA-UAP. If it cannot be argued that all UAPs have such  properties, then this statement seems unreasonable?\n\n2: The author did not provide a detailed explanation for $H(i)$ in algorithm 2 line 2, according to the equation (3), I think that the author is  trying to say $H(i)$ is an entropy loss. Less strictly speaking, maximizing H(x) means that the various components of x are directly averaged as much as possible. So, is the goal of SG(SampleGenerator, Algorithm 2) equivalent to finding an adversarial sample with each component average? Is this a form of weakened PGD? If so, why is finding weaker adversarial samples beneficial for improving robustness? Why not directly target finding UAP for SG?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, The setting is reasonable, author want to resist universal adversarial samples through small cost, this may be useful in some situation.\n2, Good experiment results.", "weaknesses": "1, The method provided in the article is not novel, overall, it is still based on the 'min-max' methods. And in my opinion, it seems to be a weakened version of adversarial training.\n\n2, The symbols are somewhat confused. For example, in euqation (4), author use $L_{cce}$, I think this means cross entropy loss, but author did not introduce what is $L_{cce}$ before the euqation (4); in algorithm line 3, author define $I_b^{en}$ which is got by SampleGenerator, but do not use it in the following of algorithm, or maybe it have been wirtten as $i_{en}$ in line 4?\n\n3, I try to test the author's algorithm on CIFAR10(VGG16, budget 8/255), but I didn't get such a good result shown in table2, the SR has only decreased by about 10%. Author did not submit the code, so I hope the author can provide a detailed introduction to the settings for Algorithms 1 and 2 (For example, how to select the epoch, learning rate, hyperparameters), and it's best to provide a detailed explanation of each symbol and steps in the algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)