Is Adversarial Training Really a Silver Bullet for Mitigating Data Poisoning?
Authors: Rui Wen
Zhengyu Zhao
Zhuoran Liu
Michael Backes
Tianhao Wang
Yang Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Data poisoning
adversarial training
indiscriminative features
adaptive defenses
robust vs. non-robust features

Abstract: Indiscriminate data poisoning can decrease the clean test accuracy of a deep learning model by slightly perturbing its training samples.
There is a consensus that such poisons can hardly harm adversarially-trained (AT) models when the adversarial training budget is no less than the poison budget, i.e., $\epsilon_\mathrm{adv}\geq\epsilon_\mathrm{poi}$. This consensus, however, is challenged in this paper based on our new attack strategy that induces \textit{entangled features} (EntF). The existence of entangled features makes the poisoned data become less useful for training a model, no matter if AT is applied or not. We demonstrate that for attacking a CIFAR-10 AT model under a reasonable setting with $\epsilon_\mathrm{adv}=\epsilon_\mathrm{poi}=8/255$, our EntF yields an accuracy drop of $13.31\%$, which is $7\times$ better than existing methods and equal to discarding $83\%$ training data. We further show the generalizability of EntF to more challenging settings, e.g., higher AT budgets, partial poisoning, unseen model architectures, and stronger (ensemble or adaptive) defenses. We finally provide new insights into the distinct roles of non-robust vs. robust features in poisoning standard vs. AT models and demonstrate the possibility of using a hybrid attack to poison standard and AT models simultaneously. Our code is available at~\url{https://github.com/WenRuiUSTC/EntF}.

Published as a conference paper at ICLR 2023
Is ADVERSARIAL TRAINING REALLY A SILVER BULLET FOR MITIGATING DATA POISONING?
Rui Wen', Zhengyu Zhao', Zhuoran Liu", Michael Backes', Tianhao Wang', Yang Zhang' CISPA Helmholtz Center for Information Security, "Radboud University, University of Virginia {rui wen, zhengyu. zhao, director, zhang) / @ccispa.d,
2 1iiuccc...... ru.nl, tianhaoévirginia.
ABSTRACT
Indiscriminate data poisoning can decrease the clean test accuracy of deep learning model by slightly perturbing its training samples. There is a consensus that such poisons can hardly harm adversarially-traineee (AT) models when the adversarial training budget no less than the poison budget, i.e., Cadv 2 "poi- This consensus, however, is challenged in this paper based that induces entangled features (EntF). The existence of entangled features makes the poisoned data become less useful for training model, no matter if AT is applied or not. We demonstrate that for attacking CIFAR-10 AT model under a reasonable setting with Eadv Cpoi = 8/255, our EntF yields an accuracy drop of 13.31%, which is 7x better than existing methods and equal to discarding 83% training data. We further show the generalizability of EntF to more challenging settings, e.g., higher AT budgets, partial poisoning, unseen model architectures, and stronger (ensemble or adaptive) defenses. We finally provide new insights into the distinct roles of non-robust robust features in poisoning standard vs. AT models and demonstrate the possibility of using a hybrid attack to poison standard and AT models simultancously. Our code is available htttpp:: //tpt//gttttt/t com/ CinRuuUSSCCEEnt.
attack strategy
INTRODUCTION
Indiscriminate data poisoning aims to degrade the overall prediction performance of machine learning model at test time by manipulating its training data. It has been increasingly important to understand indiscriminate data poisoning as web scraping becomes a common approach to obtaining large-scale data for training advanced models (Brown al. 2020 Dosovitskiy et al. 202T). Although slightly perturbing training samples has been shown to effectively poison deep learning models, there consensus that such poisons can hardly harm adversarially-traien
model when the perturbation
budget in adversarial training, Cadv, is no less than the poison budget, épois i.e., eadv 2 ppoi (Fowl
al. 2021a b. Huang et al. [021 Tao al., 2021 Wang al., 2021 Fu al., 2022 Tao al.
2022). In particular, Tao et al. (2021) have proved that in this setting, adversarial training can serve as principled defense against existing poisoning methods.
However, in this paper, we challenge this consensus by rethinking data poisoning from a fundamentally new perspective. Specifically, we introduce new poisoning approach that entangles the features of training samples from different classes. In this way, the entangled samples would hardly contribute to model training no matter whether adversarial training is applied or not, causing substantial performance degradation of models. Different from our attack strategy, existing methods commonly inject perturbations as shorteuts, as pointed out by Yu et al. (2022). This ensures that the model wrongly learns the shortcuts rather than the clean features, leading to low test accuracy (on clean samples) Segura et al. 2022a Evtimov et al. 2022 Yu et al. 2022).
Figure illustrates the working mechanism of our new poisoning approach, with a comparison to a reverse operation that instead aims to eliminate entangled features. Our new approach is also inspired by the conventional, noisy label- poisoning approach (Biggio et al. 2012) 2011 et al. (2077), where entangled labels are introduced by directly flipping labels (c.g.. assigning "dog" (or "cat") label to both the "dog" and "cat" images) under strong assumption that

---

Published as a conference paper at ICLR 2023
(a) Test Acc: 84.88%
(b) Test Ace: 72.99% + (c) Test Acc: 71.57% 4 (d) Test Ace: 88.72%
Figure 1: The t-SNE feature visualizations for (a) clean CIFAR-10 vs. poisoned CIFAR-10 10 achieved by (b) EntF-pull and (c) EntF-push, which aim induce entangled features. As comparison, (d) uses a reverse objective of EntF-push and instead increases the model accuracy. Different from our EntF, existing methods lead to well-separable features (see Appendix [A]. All t-SNE visualizations in this paper are obtained from the same clean reference model with éref
4.
the labeling process of the target model can be manipulated. However, due to the imperceptibility constraint in the common clean-label setting, we instead propose to introduce entangled features represented in the latent space. Our work mainly makes three contributions:
We demonstrate that, contrary to the consensus view, indiscriminate data poisoning can actually decrease the clean test accuracy of adversarially-traine (AT) models to substantial extent. Specifically, we propose EntF, a new poisoning approach that is based on inducing entangled features the latent space of pre-trained reference model.
We conduct extensive experiments to demonstrate the effectiveness of EntF against AT in the reasonable setting with Eadv = Epol and also its generalizability to a variety of more challenging settings, such as AT with higher budgets, partial poisoning, unseen model architectures, and stronger (ensemble or adaptive) defenses.
We further highlight the distinct roles of non-robust vs. robust features in compromising standard vs. AT models and also propose hybrid attacks that are effective even when the defender is free to adjust their AT budget Cadv-
RELATED WORK
Data poisoning. Data poisoning aims to compromise a model's performance test time by manipu- lating its training data. Related work on poisoning DNNs has mainly investigated targeted, backdoor.
and indiscriminate poisoning. Different from backdoor (Gu al. 2017 Liu al. 2018 Salem et
2022) and targeted poisoning MMuñoz-Gonzálke et al. 2017 Shafahi et al. 2018 Geiping et al. 2021), which aim to degrade the model on specific (targeted) test samples, indiscriminate poisoning aims at arbitrary clean test samples. Traditional indiscriminate poisoning is based on injecting noisy labels (Biggio et al., 2012. 201T] Munoz--Gnzázlee
et al. 2017); however, they can be easily de- tected (Shafahi al. 2018 Song et 2022). Recent methods instead pursue "clean-label" poisons by adding imperceptile perturbations. These methods mainly use the error-minimizane
(Huang arror-maximizaion loss (Fowl et al.) (2021b). with (Huang Fu al. (202)) reference model. However, these methods known to be vulnerable to
al. 2021 Tao et al.
2021 Fu et al. 2022
pre-trained Fowl et al. [202bb] Wang et al. 2021 Tao et al. 2021) or sainne-froo--sraadd
al. |021
adversarial training (AT). In particular, two concurrent methods, ADVIN (Wang et al. 2021) and
REM (Fu al. 2022), also attempt to poison AT models, but under easy settings with Epoi Deady- Generating poisons using a feature-space space loss has also been explored (Shafahi et al. 2018 Zhu et al. 2019 Geiping et al., 202T), but without considering AT and in the field of targeted poisoning.
Adversarial training, Adversarial training (AT) was recognized as the only promising solution so far to provide robustness against (test-time) adversarial examples (Athalye et al. 2018 Tramer
et al. 2020). It was also recently proved to be principled defense against indiscriminate poison- ing (Tao al. 2021). The general idea of AT to simply augment training data with adversarial examples generated in each training step. The single-step approach, FGSM, was initially used by the seminal work of Goodfellow et al.) (2015) but has been found to be ineffective against multi-step attacks (Tramrr et al.) 2017: Kurakin et al. (201).. To address this limitation, |adry et al. (2018) have proposed the WGD-based AT, which uses the multi-step optimization to further enhance the robustness. Other ofth-of-the-rtt methods have been focused improving this PGD-based AT by,
2

---

Published as a conference paper at ICLR 2023
incorporating an explicit differentiation of misclassified examples (Wang et al. 2020). identifying a bag of training tricks (Pang et al. 202T), accelerating the training process via gradient recycling (Shafahi et al. 2019). In this paper, we consider three different state-of-the-art AT techniques that adopt the standard PGD and also varied adversarial training budget Cadv-
for example, training on both clean and adversarial examples (Zhang al. 2019),
3 ENTANGLED FEATURES (ENTF) FOR POISONING AT MODELS
3.1 PROBLEM STATEMENT
We formulate the problem in the context of image classification DNNs. There are two parties involved, the poisoner and the victim. The poisoner has full access to the clean training dataset
D. {)) and is able to add perturbations gnoi to each sample and release the poisoned version D, {( 36) where x I; + 5,.00 Once the poisoned dataset is generated and released, the poisoner cannot further modify the dataset. Moreover, the poisoner has no control over the target model's training process and the labeling function of the victim. The victim only has access to the poisoned dataset and aims to train a well-generalized model using this dataset. As the victim is aware that the obtained dataset may be poisoned, they decide to deploy adversarial training to secure their model. The goal of the poisoner is to decrease the clean test accuracy of the adversarially-trainne
model by poisoning its training dataset.
When perturbing the clean dataset, the poisoner wants to ensure that the perturbation is imperceptible and can escape any detection from the victim. To this end, the poisoner constrains the generated perturbations 8poi by a certain poison budget Epol. i.c., ||8 |o I Epol. Take the iidely-adopted adversarial training framework (Madry et al.) (018) as an example, the victim trains a target model F on the poisoned dataset D, by a certain adversarial training budget Cadv following the objective:
argmin 9
E () max C(F(x' + (.ddd), s.t. [| 1loo S Cadv.
where x'' denotes the poisoned input, gadv denotes the adversarial perturbations, 0 denotes the model parameters, and L is the classification loss (e.g., the commonly used cross-entropy loss).
In this paper, we focus on the reasonable setting with Spoi S Cadv- In contrast, the two concurrent studies, ADVIN (Wang et al.] (02I) and REM (Fu al. 2022), focus on much easier settings with 'poi 2 2eadv. in which it is not surprising that AT would fail because the clean samples are already out of the cady-ball of the poisoned samples TToo al. 2021).
3.2 METHODOLOGY
In this section, we introduce EntF, our new poisoning approach to compromising adversarial training. The key intuition of EntF is to cause samples from different classes to share entangled features and then become useless for model training, including adversarial training. Specifically, we propose two different variants of EntF, namely EntF-push and EntF-pull. For EntF-push, all training samples in each of the original classes y are pushed away from the corresponding class centroid μ in the latent feature space (i.e., the output of the penultimate layer Fi of reference model F which has totally layers. The objective function can be formulated as:
Cpush
max gpes
|7 + groi) Mylla s.t. ||8 1loo S Epoi.
For EntF-pull, each training sample is pulled towards the centroid of its nearest class y/:
Lpull
min Pi-1(2 gmoi) - My ll2 s.t.
|| | S Cpoi-
8ooi
The above class centroid is computed as the average features of all clean samples X' in that class:
1
Σ Fz))(z).
(4)
|X
xEN
3

---

Published as a conference paper at ICLR 2023
We find this simple, average- baaege method works well in case, and we leave the exploration of other, metric learning methods (Kaya and Bilge 2019) to future work.
In order to learn similar representation space that of an AT target model, the reference model F is also adversarially trained with a certain perturbation budget éref. We discuss the impact of cref on the poisoning performance in Section 5] Following the common practice, adopt the Projected Gradient Descent (PGD) (Madry et al. 2018) to solve the above poison optimization.
Why adversarial training be compromised. TTo et al. (2021) have proved that adversarial training can serve as a principled defense against data poisoning based on the following theorem.
Theorem Given a classifter f : - y. for any data distribution 'D and any perturbed distribution D such that D E B (D,),, we have
Rnat (J,D) S
max D'EBwa (D.c) Rnat Rnat(1 (S, - Radv(SDD)
where Rust denotes natural risk and Rady denotes adversarial risk. Detailed definitions can be found in Appendix B Theorem guarantees that adversarial training the poisoned data distribution D optimizes upper bound of natural risk the original data distribution DD if D is within the 0o- So-Wasserstein ball of 'D (Tao er al. (021). That is to say, achieving a low natural risk on DD (i.e., high clean test accuracy) requires low adversarial risk on D. This guarantee is based on an implicit assumption that adversarial training is capable of minimizing the adversarial risk on the poisoned data distribution D. which holds for existing attacks. However, for our EntF, the poisoned data that share entangled features become not useful even for adversarial training, and as result, the assumption required for the proof is broken. Our experimental results Appendix B validate this claim.
Important note on the cross-entropy loss. The key novelty of our EntF over existing methods lies in not only the attack strategy (entangled features vs. shortcuts) but also the specific loss (feature-level vs. output-level). Existing methods on poisoning standard models have commonly adopted the cross-entropy (CE) loss and concluded that the targeted optimization, either with an incorrect (Fowl al. b0221b Tao et al. 2021; Wang et al. 2021) or original (Huang et al. 2021) class as the target, is generally stronger than the untargeted CE. This conclusion somewhat leads to the fact that the two concurrent studies on poisoning AT models (i.e., ADVIN (Wang al. 2021 and REM (Fu et al.
2022 have completely ignored the untargeted CE as their baseline.
However, we find that the above conclusion does not hold for poisoning AT models. Specifically, we notice that the untargeted CE can also lead to entangled features to some extent and as result yield substantial accuracy drop (12.02%), while its targeted counterpart (i.e, ADVIN (Wang et al. 2021) shown in our Table [ completely fails. Note that the untargeted CE still performs worse than our EntF-push, especially in the more complex tasks, i.e., CIFAR-100 and TinylmageNet (see Appendix [or details). This indicates that using the CE loss is not an ultimate solution in practice.
EXPERIMENTS
In this section, we conduct extensive experiments to validate that adversarial training (AT) can be compromised by our new poisoning attack. In particular, we consider challenging scenarios with high AT budgets, partial poisoming, unseen model architectures, or strong (ensemble or adaptive) defenses.
4.1 EXPERIMENTAL SETTINGS
We use three image classification benchmark datasets: CIFAR-10 (CIF). CIFAR-100 (CIF). and TinyImageNet (Tin). These datasets have been commonly used in the poisoning literature. We adopt the perturbation budget Gref = 4/255 for adversarially training the reference model and find that other values also work well (see results in Section 5). PGD-300 with step size of 0.4/255 and differentiable data augmentation (Fowl et al. 2021b) is used for poison optimization. If not explicitly mentioned, we focus on the reasonable setting with Epoi Cady 8/255 and adopt ResNet- 18 for both the reference and target models. Additional experimental settings can be found in Appendix [ All experiments are performed on NVIDIA DGX-A100 server.
4

---

Published as a conference paper at ICLR 2023
Table Comparison of different poisoning methods against adversarial training. ADVIN, Hypocritical", and REM also adopt an adversarially-trained reference model, as our methods.
POISON METHOD
CLEAN TEST ACCURACY (%,
NONE (CLEAN)
84.88
HYPOCRITICAL TAO ET AL.
2021
84.96
UNLEARNABLE HUANG ET AL.
2021
84.91
CLASS-WISE, KANDOM NOISE
84.06
ADVPOISON (FOWL AL., 2021B
83.11
ADVIN (WANG ET AL. 2021)
86.76
HYPOCRITICAL (TAO ET AT.
2022)
86.56
REM (Ft ET 2022
84.21
ENTE-PULL (OURS)
72.99
ENTF-PUSH (OURS)
71.57
4.2 ENTF COMPARED TO EXISTING ATTACKS UNDER Eadv Epoi
We first evaluate the performance of different state-of-the-art poisoning methods against adversarial training the basic setting with époi Eadv on CIFAR-10. As can be seen from Table I all existing methods can hardly decrease the model accuracy. Specifically, although ADVIN (Wang et al. 2021
and REM (Fu et al.. 2022)) have claimed effectiveness in the unreasonable settings with Epoi 2 2eadv they fail in our reasonable setting. In some cases, the poisons may even slightly increase the model accuracy, which is also noticed in the concurrent work (Tao et al. 2022).
In contrast to existing methods, both our EntF-push and EntF-pull can substantially decrease the model accuracy. Note that decreasing the model accuracy to 71.57% is dramatic because it equals the performance achieved by directly discarding 83% of the original training data (see more relevant discussions in Section|4.5). In addition, EntF-push and EntF-pull achieve similar results but obviously, EntF-pull is less efficient because it needs to calculate and then rank the distance between each sample and class centroid. Moreover, the class selection strategy in EntF-pull may have an impact on the final performance (see more analysis in Appendix E). For these reasons, if not specifically mentioned, we choose to use EntF-pusee in the following experiments.
Table 2: Evaluating EntF on different datasets.
POISON METHOD \ DATASET
CIFAR-10
CIFAR-100
TINYIMAGENET
NONE (CLEAN)
84.88
59.50
51.95
ENTF (OURS)
71.57
47.29
41.32
A0
Clean Test Accuracy
Clean Test Accuracy
Clean Test Accuracy
20
20
20
rod moining Mudda Mapp
60
60
500
50
50
60
Clean Data
30
Clean Data
10
Clean Data Polson Daca
Polson Data
Polson Data
103
50
100
Training Epochs
Training Epochs
Training Epochs
(a) Madry
Madry et al. 2018
(b) TRADES (Zhang et al.
2019
(c) MART Wang et al. 2020
Figure 2: Evaluating EntF against three different well-known adversarial training frameworks,
4.3 ENTF FOR LARGER DATASETS AND OTHER AT FRAMEWORKS UNDER fadv 6poi
The results shown Table 2 further validate the general effectiveness of our EntF on larger datasets, where the model accuracy consistently drops by more than 10%. We also evaluate EntF against
5

---

Published as a conference paper at ICLR 2023
different uidely-used AT frameworks. Figure 2 shows the learning curves of the poisoned AT target model that is trained with Madry (Madry et al. 2018), TRADES (Zhang et al. 2019). or MART (Wang et al. 2020p. As can be seen, our EntF largely decreases the clean test accuracy in all cases. We can also observe that all three frameworks exhibit a relatively steady learning process, i.e., the model accuracy monotonically increases lower than that of the model trained on clean data. This pattern is different from that in poisoning standard models, where the model accuracy is found increase at few early epochs, and then start to decrease dramatically to the final low accuracy (Huang et al. 2021 Liu et al. [021 [Segura et al. 2022a). This fundamental difference indicates that early stopping cannot be used as effective defense against poisoning for AT models.
epochs, and finally reaches accuracy that is still
Table 3: Evaluating EntF under different Epol VS. Cadv-
POISON BUDGET ADVTRAIN BUDGET
Cadv 4/255
Cadv 8/255
Cadv 16/25
NONE (CLEAN)
90.31
84.88
73.78
Epo 4/255
84.37
79.25
69.35
Epo = 8/255
75.39
71.57
63.73
poi 16/255
50.27
60.29
53.03
4.4 ENTF UNDER HIGHER AT BUDGETS
We further test EntF under higher adversarial training budgets and also consider different poison budgets. As can be seen from Table 3 even with an overwhelming budget, adversarial training can still be largely degraded by our EntF. For example, when the AT budget is Eady 16/255, which is 2x larger than the poison budget Epoi 8/255, our EntF still yields substantial accuracy drop of 10.05%. In addition, under the same setting with eadv = <pois larger poison budget leads to better poison performance. Specifically, for cadv fpo = 4/255, model accuracy drops by 5.94% (90.31% % + 84.37%), while for a larger poison budget, 8/255 or 16/255, the accuracy drops by 13.31% (84.88% + 71.57%) 20.75% (73.78% specific poison budget, the model accuracy and AT budget do not have a clear correlation. This reasonable because although enlarging the AT budget increase clean accuracy due to higher robustness to poisons, it also inevitably leads to an accuracy drop compared to standard training. We leave detailed explorations for future work.
53.03 03... We can also observe that under
Table 4: Effects of adjusting the poison proportion. "None (Clean)" shows the baseline results where the rest clean data is used without poisoning.
POISON METHOD POISON PROPORTION
0.2
0.4
0.6
0.8
NONE (CLEAN)
83.66
81.82
79.16
73.23
CLEAN+ENTF
81.41
78.67
75.84
73.56
CLEAN+ENTF (WORSE CASE)
81.84
78.83
75.92
74.01
Poison Proportion: 0.2 Poison Proportion: 0.4
Poison Proportion: 0.6
Poison Proportion: 0.8
Figure 3: The t-SNE visualizations for different poison proportions.
4.5 ENTF UNDER PARTIAL POISONING
We also examine EntF in challenging scenario where only partial training data allowed to be poisoned. We consider two different poisoning settings where different data are used for calculating the class centroids. Specifically, the first setting is based the whole original (clean)

---

Published as a conference paper at ICLR 2023
dataset but the second, worse-case one is based on only the partial clean data that are allowed to be poisoned. As can be seen from Table our EntF is still effective in this challenging scenario. Recall when the whole dataset is poisoned.
that other attacks can hardly decrease the model accuracy In particular, EntF decreases the model accuracy from 84.88% to 81 81.41% % by only poisoning 0.2 of the training data, and this result is also lower than the baseline that is achieved by directly discarding these poisoned data (83.66%)
We can also observe that the two different poisoning settings yield very similar results. This indicates that the calculation of class centroids in our EntF is not sensitive to the amount of data, and as a result, efficiency can be potentially improved by using fewer data for the centroid calculation. The fact that poisoning more data yields better performance can also be explained by Figure 3 where a larger poison proportion leads to a larger number of entangled features.
Table 5: Transferability of EntF poisons from ResNet-18 to other model architectures.
POISON METHOD \ TARRGT
RESNET-18
RESNET-34
VGG-19
DENSENET-121
MOBILENETV2
NONE (CLEAN)
84.88
86.58 73.05
75.99
87.22 74.35
80.11
ENTF
71.57
64.66
67.21
4.6 TRANSFERABILITY OF ENTF TO UNSEEN MODEL ARCHITECTURES
The latent feature space that is used for generating poisons is specific certain reference model. For this reason, one natural question to ask is whether the poisons generated on one model architecture still effective when the target model adopts different architecture. T5 demonsttates that the poisoning effects of our EntF optimized against a ResNet-18 reference model can transfer to other target model architectures. Specifically, for the four different (unseen) architectures, the generated poisons are able to degrade the model accuracy to almost the same extent, indicating the strong generalizability of our EntF.
Table 6: Evaluating EntF against defenses that apply both data augmentations and AT.
DEFENSE
CLEAN TEST ACCURACY (%)
NONE (CLEAN)
84.88
ADVERSARIAL TRAINING
71.57
+RANDOM NOISE
71.88
+JPEG COMPRESSION
70.40 70.40
+MIXUP ZHANG ET AL. 2018
71.84
+CUTOUT DEVRIES AND TAYLOR 2017
69.81
+CUTMIX YUN AL. 2019)
68.85
+GRAYSCALE (LI ET AL. 2021
68 68.67
4.7 ENTF AGAINST OTHER DEFENSES
Ensemble defenses with data augmentations. Applying additional data augmentations before stan- dard training has been shown to be able to mitigate the effects of perturbation- based poisons (Huang ct al. 2021 Fowl et al. 2021b Liu et al. [002]] Tao et al. 2021). Here we study if data augmentations can complement adversarial training when facing our EntF. Following previous work, we test diverse set of data augmentations, including random noise, Mixup (Zhang et al.) 2018), Cutmix Yuun al. 2019), and Cutout (Devries and Taylor] 2017). We also test gray-scale pre- filtering (Liu et
2021 which shows strong performance in mitigating unlearnable examples (Huang et al. 2021) Table shows that all the data augmentation methods fail to help AT to mitigate our EntF.
Adaptive defenses by filtering out entangled samples. In addition to existing defense methods, we also consider stronger, adaptive defenses that the victim may design based on a certain level of knowledge about our EntF. It is worth noting that, in realistic scenarios, the victim can only leverage a poisoned model, and so it also has no access to a clean AT reference model, which available to the poisoner, including our EntF. This is reasonable because if it is indeed feasible for the victim to get a clean AT (reference) model, there no need for dealing with the poisoned data in the first place, and the clean AT model can already be used as an effective target model.
7

---

Published as a conference paper at ICLR 2023
When the victim knows that entangled features have been introduced by EntF, they would filter out the "overlapped samples", which are located close in the feature space but from different classes. We test by removing different proportions of such "overlapped samples" and find that the best setting can only recover the accuracy from 71.57% to 72.43%. We go a step further by considering a stronger victim who even knows the specific algorithm of EntF-push (i.e., Equation 2 In this case, the vietim would recover the data by pulling the poisoned samples back toward their original class centroids. We find that this defense stronger than the above but still can only recover the accuracy 75.32% (about 10% lower than the clean AT accuracy). We also consider practical scenario in which the victim can leverage an ST AT CIFAR- model as a general purposs model. In this scenario, the adaptive defense only recovers the model accuracy to 72.97%.
Adaptive defenses through feature perturbation-base AT. We further test new defense that uses Equation 2 for generating the adversarial examples in AT instead of the common, cross-entropy loss. When trained on clean data, this new AT variant yields an accuracy of 86.8 86.84, similar that achieved by the conventional AT. However, when trained still substantially drops to 72.99%. indicating that it is not a satisfying defense. This defense even worse than the above with pre-trained AT model (72.99% vs. 75.32%). This might be because the class centroids calculated when the model is not well trained (in the early AT training stage) cannot provide meaningful guidance compared to those based the pre-trained model. As sanity check, we also try another AT variant that uses a reverse loss of Equation 2 and find that as expected, it causes the model accuracy to drop a lot (to 47 17.26))
our poisoned data, the model accuracy
80-
ST Aazuracy (blue)
& 28 Accuracy (orange)
60-
ST Clean
AT Poison
38 78
AT Clean
Robustness of the reference model (frez)
Clean
Crof
0
Grof
2 Crof 4 Cyof 8
(b)
Figure 4: Impact of the robustness of the reference model on poisoning standard (ST, Gref = 0) vs. adversarially-trained (AT) target model. (a) Clean test accuracy for both ST and AT; (b) Perturbation visualizations for different cref = 0. More visualizations can be found in Appendix G
5 POISONING AT VS. ST MODELS
All our experiments have so far been focused on poisoning AT models. However, it is also valuable study the problem in the context of standard training and figure out the difference. To this end, we analyze the poisons against a standard (ST) model vs. an AT model. Specifically, we adjust the perturbation budget efef for adversarially training the reference model and analyze the poisons in terms of both the poisoning strength and the visual characteristics of perturbations.
As can be seen from Figure 4a for poisoning the AT model, the poisoning strength is gradually improved as the reference model becomes more robust (i.e., Gzef is increased). In contrast, for poisoning the ST model, the poisoning gradually degraded. Concurrent work (Tao et al.) 2022) also discusses the impact of fref poisoning AT models but focuses on a fundamentally different task where the attack aims to degrade the adversarial robustness of AT models (rather than the clean accuracy here). It is also important to note that our EntF can work well under different ezef between 2 and 8, where their approach is much more sensitive to the choice éref in their task (see their Figure 2(a)). We further visualize the perturbations generated with different Gref in Figure 4b) As can be seen, when using a standard reference model (i.e., Gref = 0), the perturbations exhibit noisy patterns, but the Gee is gradually increased, the perturbations tend be more aligned with image semantics.
These observations suggest that poisoning ST and AT models requires modifying different types of features. More specifically, modifying the robust (semantic) features is the key to poisoning AT
8

---

Published as a conference paper at ICLR 2023
models, while modifying the non-robust features works for ST models. This conclusion also supports the well-known perspective that non non-robust features can be picked up by models during standard training, even in the presence of robust features, while adversarial training tends to utilize robust features (Ilyas et al., 2019). Figure also confirms that our EntF is effective in poisoning ST models since it can decrease the model accuracy the random guess level (i.e., 10% for CIFAR-10).
Hybrid attacks. The distinct roles of robust and non-robust features in poisoning AT and ST models inspire us search for a hybrid attack to poison the ST and AT models simultaneously. This is particularly relevant for practical defenders who may want to adjust their AT budget to achieve optimal accuracy. straightforward idea to simply average the perturbations generated using an ST reference model and those using an AT reference model. In this case, the poison budget remains unchanged. Alternatively, we propose a hybrid attack that is based on jointly optimizing the perturbations against both ST and AT reference models, balanced by factors X; that correspond to different AT models. This modifies Equation Q to the following Equation 5
Llybrid
max gooi
F 1i. 6 My,TTll2 + Σ Aul F.llFT.- L.AT, (x 8 My,AT, 112- (5)
Here we adopt two AT reference models with Eye! = 2/255 and 4/255. Table 7 shows that both hybrid attacks substantially decrease the model accuracy across different AT budgets cadv varying from (i.e., ST) to 16. Specifically, the optimization- average- based method. Table 9 Appendix [ further shows the much worse performance of other attacks and confirms the general effectiveness of our hybrid attack strategy, see ours (Hybrid) vs. AdvPoison (Hybrid).
method performs much better than the
Table 7: Evaluating two hybrid attacks (average/opptiiizationn under different Epoi VS. Eadv.
Epoi Endy
0/255
4/255
8/255
16/255
OPTIMAL
NONE (CLEAN)
94.59
90.31
84.88
73.78
94.59
4/255
51.98/
19.51
86.87/84.48
81.24/80..5
71.13/70.26
86.87 86.87/84.48
/8.448
8/255
22.86/ / 11.89
82.30/76.55
77.19/74.30
66. /5..75
82.30/76.55 76..5
16/255
5.34/6.59
75.32/59.55
69.41/66.25
59.40/60.73
75.32/66.25
6 CONCLUSION
In this paper, we have proposed EntF, a new poisoning approach to decreasing the deep learning classifier's accuracy even when adversarial training is applied. This approach is based on new attack strategy that makes the features of training samples from different classes become entangled. Extensive experiments demonstrate the effectiveness of EntF against adversarial training in different scenarios, including those with more aggressive AT budgets, unseen model architectures, and adaptive defenses. We also discuss the distinct roles of the robust non-robust features in poisoning standard vs. ad adversarially-tre trained models and demonstrate that our hybrid attacks can poison standard and AT models simultaneously.
We encourage future research to analyze EntF in more comprehensive settings and compare it to the current, shortcut-based methods from different angles. In particular, it is important to come up with new defenses against EntF, possibly based on advanced techniques of learning from noisy labels (Song et al., 2022). It is also worth noting that most of the current poisoning studies, including ours, have assumed that the poisoner has access to the training dataset of the target model. This assumption is realistic in specific threat models (e.g., secure dataset release (Fowl et al.] (021a]) but may not be plausible for sensitive/private data. Therefore, it would be promising to extend EntF to addressing data-free free poisons YYu et al. 2022 Segura et al. 2022b).
On the one hand, data poisoning could be potentially leveraged by malicious parties attacks. In this case, we hope our work can inspire the community to develop stronger defenses based on our comprehensive analysis. On the other hand, when data poisoning is directly used for social good, e.g.. for protecting personal data from being misused (Fowl et al. 2021a Huang et al. 2021 Fu et al. 2022), our approach for generating stronger poisons leads to stronger protective effects.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank all anonymous reviewers for their constructive comments. This work is partially funded by the Helmholtz Association within the project "Trustworthy Federated Data Analytics" (TFDA) (funding number ZTI-0O1 4), by the European Health and Digital Executive Agency (HADEA) within the project "Understanding the individual host response against Hepatitis Virus develop personalized approach for the management of hepatitis D (D-Solve) (grant agreement number 101057917), and by NSF grant number 2217071.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020. D
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition Scale, In International Conference on Learning Representations (ICLR), 2021.
Liam Fowl, Ping- Yeh Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing Unauthorized Use of Proprietary Data: Poisoning for Secure Dataset
Release. CoRR
2021a. a.
9
Liam Fowl, Micah Goldblum, Ping- Yeh Chiang, Jonas Geiping, Wojtek Czaja, and Tom Goldstein. Adversarial Examples Make Strong Poisons. In Annual Conference Neural Information Processing Systems (NeurIPS), pages 30339-30351, NeurlPS, 2021
Hanxun Huang, Xingjun Ma. Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable Examples: Making Personal Data Unexploitable. In International Conference on Learning Representations (ICLR), 2021.
Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 16209-16225. NeurIPS, 2021.
Zhirui Wang. Yifei Wang, and Yisen Wang. Fooling Adversarial Training with Inducing Noise. CoRR 5
abs/2111.101338 2021.
Shaopeng Fu, Fengxiang He. Yang Liu, Li Shen, and Dacheng Tao. Robust Unlearnable Exam- ples: Protecting Data Against Adversarial Learning. In International Conference on Learning Representations (ICLR), 2022.
Lue Tao, Lei Feng, Hongxin Wei, Jinfeng Yi, Sheng-Jun Huang, and Songean Chen. Can Adversarial Training Be Manipulated By Non- Nob-Ro Features? In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2022. 5
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie Yan Liu. Availability Attacks Create Shortcuts. In ACM Conference on Knowledge Discovery and Data Mining (KDD), pages 2367-2376. ACM, 2022.
9
Pedro Sandoval Segura, Vasu Singla, Liam Fowl, Jonas Geiping, Micah Goldblum, David Jacobs, and Tom Goldstein. Poisons that are learned faster are more effective. CoRR abs/2204.08615, 2022a.
14
Ivan Evtimov, Ian Covert, Aditya Kusupati, and Tadayoshi Kohno. Disrupting Model Training with Adversarial Shortcuts. CoRR abs/2106.0665 2021. [4
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Machines. In International Conference on Machine Learning (ICML). icml.ce/ / Omnipress, 2012.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Support Vector Machines Under Adversarial Label Noise. In Asian Conference on Machine Learning (ACML), pages 97-112. JMLR, 2011.
Luis Ganizzzzzzzálzz, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, and Fabio Roli. Towards Poisoning of Deep Learning Algorithms with Back- gradient Optimization. In Workshop on Security and Artificial Intelligence (AlSec), pages 27-38. ACM, 2017.
11

---

Published as a conference paper at ICLR 2023
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Grag. Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. CoRR fls/1708.06633, 2017.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chure Lee, Juan Zhai, Weihang Wang, and Xiangyu
Zhang. Trojaning Attack Neural Networks. In Network and Distributed System Security
Symposium (NDSS). Internet Society, 2018. 2
Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. Dynamic Backdoor Attacks Against Machine Learning Models. In IEEE European Symposium on Security and Privacy (Euro
S&P). IEEE, 2022. 2
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 6103-6113. NeurIPS, 2018. 2
Jonas Geiping, Liam H. Fowl, W. Ronny Huang. Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching, In International Conference on Learning Representations (ICLR), 2021.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae- Gil Lee. Learning from Noisy Labels with Deep Neural Networks: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
9
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable Clean-label Poisoning Attacks Deep Neural Nets. In International Conference on Machine Learning (ICML), pages 7614-7623. JMLR, 2019. 2
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated Gradients Give a False Sense of Security: Cireumventing Defenses Adversarial Examples. In International Conference on Machine Learning (ICML), pages 274-283 PMLR, 2018. 2
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On Adaptive Attacks to Adversarial Example Defenses. In Annual Conference on Neural Information Processing Systems
(NeurIPS), NeurIPS, 2020. 2
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learming Representations (ICLR), 2015. 2
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick Mc-
Daniel. Ensemble Adversarial Training: Attacks and Defenses. In International Conference Learning Representations (ICLR), 2017. 2
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial Machine Learning at Scale. In International Conference Learning Representations (ICLR), 2017. 2
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on
Learning Representations (ICLR), 2018. 2 5
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing. Laurent El Ghaoui, and Michael I. Jordan. Theoretically Principled Trfde-off between Robustness and Accuracy. In International Conference on Machine Learning (ICML), pages 1472-7482. PMLR, 2019. 3
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving Adversarial Robustness Requires Revisiting Misclassified Examples. In International Conference on Learning Representations (ICLR), 2020.
6
Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of Tricks for Adversarial Training. In International Conference on Learning Representations (ICLR), 2021.
Ali Shafahi, Mahyar Najibi. Amin Ghiasi, Zheng Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 3353-3364. NeurIPS, 2019.
12

---

Published as a conference paper at ICLR 2023
Mahmut Kaya and Hasan Sakir Bilge. Deep Metric Learning: A Survey. Symmetry, 2019. 4
http:://www...c...oootccco/////d 4
Zhuoran Liu, Zhengyu Zhao, Alex Kolmus, Tijn Bems, Twan van Laarhoven, Tom Heskes, and
Martha A. Larson. Going Grayscale: The Road to Understanding and Improving Unlearnable
Examples. CoRR abv/2111. 13244, 2021. 7
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond Empirical
Risk Minimization. In International Conference Learning Representations (ICLR), 2018.
Terrance Devries and Graham W. Taylor. Improved Regularization of Convolutional Neural Networks with Cutout. CoRR abs/1708. 04552, 2017. 7
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In IEEE International Conference Computer Vision (ICCV), pages 6022-6031. IEEE, 2019, 7
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial Examples Are Not Bugs, They Are Features. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 125-136. NeurIPS, 2019.
Pedro Sandoval Segura, Vasu Singla, Jonas Geiping, Micah Goldblum, Tom Goldstein, and David W. Jacobs. Autoregressive Perturbations for Data Poisoning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurlPS, 2022b. 9
13

---

Published as a conference paper at ICLR 2023
A WELL-SEPARABLE FEATURES IN EXISTING METHODS
Instead of introducing entangled features, most existing poisoning methods rely on generating certain shorteut perturbations that be more easily learned by the target model than the actual image content (Too et al. 2021] Segura et al., 2022a Evtimov et al. [02T]] Yu et al. 2022). Figure S visualizes the poison representations of different existing methods on an adversarially-rre hetrrnnll-- model. As can be seen, although there exist certain differences between the patterns of different methods (e.g.. regarding the relative positions of different classes), these methods indeed yield well-separable data. This observation also confirms that causing entangled features is sufficien condition to degrade adversarial training.
Hypocritical#*
Hypocritical
Unlearnable
AdvPoison
ADVIN
REM
Figure 5: The t-SNE visualizations of the feature representations for existing poisoning methods.
B NATURAL AND ADVERSARIAL RISKS
2.2
2.0
Adversaria Risk
1.4
Clean Data
0.8
Poison Data
Training Epochs
Figure 6: Adversarial risk curves for clean and poison data.
According to Tao et al. (2021), natural risk and adversarial risk are defined follows.
Definition (Natural Risk)
Ruad (J, D) BL [e(s(a))u))
[
Definition (Adversarial Risk)
Radi (f,)), Th (D
max ((((x').)) se (BB(())))
For the whole training process, we calculate the adversarial risk based on Definition The adversarial risks for both clean and poison data are shown in Figure. 6 It validates our claim that adversarial training faces difficulties when optimizing poisoned data because the risk decreases more slowly than the clean case.
14

---

Published as a conference paper at ICLR 2023
C DETAILED RESULTS FOR ENTF vs. THE CE LOSS
Table 8: EntF vs. the CE loss using the same robust reference model.
POISON METHOD \
DATASET CIFAR-10
CIFAR-100
TINYIMAGENET
POSIONNNNR ENTF-PUSH
72.86
50.45
45.47
71.57
47.29
41.32
(a) Clean (84.8%%
(b) Pull-pair (78.21%)
(c) Pull (72.99%)
(d) Push (71.57%)
Figure 7: The t-SNE visualizations for EntF-pull--pa vs. our original EntF (Pull and Push).
D ADDITIONAL EXPERIMENTAL SETTINGS
All reference and target models are trained for 100 epochs using SGD optimizer with an initial learning rate of 0.1 that is decayed by factor of 0.1 at the 75-th and 90-th training epochs. The optimizer is set with momentum 0.9 and weight decay 10-1 The inner maximization (i.e., generation of adversarial examples) of the adversarial training is solved by 10-step PGD with step size of 2/255.
E ADDITIONAL ANALYSIS OF ENTANGLED FEATURES
Our experimental results have demonstrated the effectiveness of EntF various scenarios. Here we provide additional analysis better understand the property of entangled features. To this end, adjust the class selection strategy in EntF-pull to be simply based on pairwise entangled features. Specifically, each pair consists of two classes that have minimal centroid distance. For optimization, we calculate the centroids of each class pair and then minimize the distance between samples in one class and the centroid of the other class. We denote this attack variant targeting pairwise entangled features as EntF-pull-pair
We find that Enul--uul-pair can decrease the model accuracy from 84.88% 78.21%, which indicates that introducing pairwise entangled features can already substantially compromise adversarial training. However, there is still large performance gap between Entl--ull-pair and our original EntF. This can be explained by the fact that EntF-pull exploits more diverse pulling directions based on the sample-class distance, and EntF-push makes samples from multiple classes become overlapped. The visualizations in Figure clearly confirm the above observation that EntF-pulll pplll-pair yields entangled features to some extent but still fewer than the original EntF-pull and EntF-push.
F ADDITIONAL EXPERIMENTAL RESULTS
15

---

Published as a conference paper at ICLR 2023
60
50
Clean Test Accuracy
100/ Mar
Clean Test Accuracy
50
Clean Test Accuracy
60
60
30
50
20
Clean
Clean
10
Clean
Posson
Posson
Poison
30
Training Epochs
Training Epochs
Training Epochs
(a) CIFAR-10
(b) CIFAR-100
(c) TinyImageNet
Figure 8: Learning curves of our EntF on different datasets.
14
32
800
4G3
28
50
800
28
35
158
13
32
400
400
33
892
16
50
39
2
527
23
196
86
25
732
30
400
6
306
33
14
16
23
14
410
12
34
31
20
825
10
35
$40
817
Clean (84.88%)
EntF (71.57%)
Figure 9: Classification confusion metries for clean vs. EntF on CIFAR-10. Our EntF causes misclas- sification from one true class to multiple wrong classes, while previous work causes misclassification from one class dominant wrong class (see Figure in (Fowl et al. (021b)).
16

---

Published as a conference paper at ICLR 2023
G ADDITIONAL PERTURBATION VISUALIZATIONS
Clean
Cref 8
Clean
Cref
4
(a) CIFAR-100
(b) TinylmageNet
Figure 10: Perturbation (normalized to [0,1]) visualizations for CIFAR-100 and Tiny
H OPTIMAL ACCURACY OF DIFFERENT ATTACKS
Table 9: Attack performance when the defender adjusts AT budget Eady to obtain the optimal accuracy.
POISON METHOD (Epol 8/255)
0/255
4/255
8/255
16/255
OPTIMAL
NONE (CLEAN)
94.59
90.31
84.88
73.78
94.59
ADVPOISON REM
9.91
88.98
83.11
71.31
88.98
25.59
46.57
84.21
85.76
85.76
ADVIN UNLEARNABLE HYPOCRITICAL
77.31
90.08
86.76
72.16
90.08
25.69
90.47
84.91
79.81
90.47
74.06
91.18
84.96
73.33
91.18
HYPOCRITICAL*
75.22
84.82
86.56
82.26
86.56
ADVPOISON (HYBRID)
4.16
82.74
78.65
67.17
82.74
OURS (HYBRID)
12.93
76.55
74.30
65.75
76.55
17


Reviews:
[{"review_id": "tyaI1J1Fqaq", "reviewer": "Reviewer_ffZU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper challenges the defense ability of adversarial training against clean-label availability poisoning attacks. It was believed that these attacks can hardly harm adversarially trained models. However, the proposed attack method in the paper substantially degrades the test accuracy of adversarially trained models from 84.88% to 71.57% on CIFAR-10, which is seven times better than existing methods, resulting in a new SOTA.\n", "summary_of_the_review": "This paper establishes a new SOTA in degrading the test accuracy of adversarial training under clean-label availability attacks. Overall, the proposed method is adequately evaluated, and the results are very impressive. Thus, the paper successfully challenges the consensus that clean-label availability attacks can hardly harm adversarial training.", "strength_and_weaknesses": "Strengths:\n- The setting of $\\epsilon_{adv}\\ge \\epsilon_{poi}$ considered in this paper is indeed more reasonable than the setting of $\\epsilon_{poi}\\ge 2\\epsilon_{adv}$.\n- The paper points out an implicit assumption behind the defense ability of adversarial training against clean-label availability poisoning attacks: adversarial training is capable of minimizing the adversarial risk on the poisoned data distribution. When this assumption is broken by the proposed attack, adversarial training would fail.\n- Experimental results are strong. As far as I know, no previous attack can degrade the test accuracy of adversarially trained models by more than 2% on CIFAR-10. In contrast, the proposed attack can degrade the test accuracy by more than 10%. This is impressive.\n\n\nWeaknesses:\n\n- The authors claim that the proposed attack method makes some features become not useful even for adversarial training, and thus the assumption behind the defense ability of adversarial training is broken. This means that when the proposed method induces indiscriminative features in the training data, adversarial training becomes not capable of minimizing the adversarial risk on the poisoned data. Are there any experimental results that validate that adversarial training faces difficulties in minimizing the adversarial risk of such poisoned data?\n\n- It is interesting to see that in Table 3, when the poison budget $\\epsilon_{poi}=8/255$, adversarial training with $\\epsilon_{adv}=4/255$ performs better than adversarial training with $\\epsilon_{adv}=8/255$. Could you provide any insight about this?\n\n- The reference model adopted in this paper is adversarially trained. Figure 4 shows that this performs better than adopting naturally trained models. The effectiveness of the previous attack methods may also be improved by adopting an adversarially trained model as the reference model. Have the previous works tried this? Do you have any results to share?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iSy_AtO6Ohp", "reviewer": "Reviewer_DCRE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work the authors attack in models in the following adversarial threat model: \n* Problem considered: CIFAR-10 image classification task\n* Attacker threat model: can modify each training image by up to $\\epsilon=8/255$ in $\\ell_\\inf$ norm\n* Attacker goal: minimize clean test accuracy on models trained on this dataset \n\nThe authors develop a new algorithm, INF, for this kind of data poisoning task, specifically aimed at the defense of performing adversarial training at train time. The authors find that they can decrease accuracy by 13.31% (from 84.88% under no poisoning to 71.57% under INF).", "summary_of_the_review": "The paper presents a new method that yields state-of-the-art poisoning on adversarially trained models, which were previously seen as an effective defense against $\\ell_p$-based data poisoning methods. The paper has some issues around claims and contextualization of work,. ", "strength_and_weaknesses": "Strengths:\n* The method is novel and obtains state-of-the-art levels of poisoning success on CIFAR-10 adversarially trained models. The authors' attack is interesting: given a learning algorithm, it exploits the model class' idiosyncrasies to poison newly trained models.\n\nWeaknesses (mostly in writing and unnecessary claims):\n* Claims about the mechanism of INF (and of other attacks). Any claims about the claims need to be properly defined or removed. For example, from the introduction \"Different from our attack strategy, existing methods commonly inject perturbations as shortcuts, which ensures that the model wrongly learns the shortcuts rather than the actual image content, leading to low test accuracy (Segura et al., 2022a; Evtimov et al., 2021; Yu et al., 2022)\" - there is no definition of what a shortcut or indiscriminate feature here is. The authors should describe concretely in what sense the given attack is different from previous work. These poorly defined claims around shortcuts vs indescriminate features are throughout the paper and should be rectified (or removed) before the work is accepted.\n* Poor contextualization: Is adversarial training currently considered the best possible defense in the given $\\ell_p$-based threat model? It would be good to provide this kind of context.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TY14AUQIRz", "reviewer": "Reviewer_VP8n", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new approach, indiscriminative features (INF), to the problem of poisoning a data set so as to make predictive models learned from the data less successful, even if those models are trained using adversarial training.  Typically, adversarial training has been a successful defense against indiscriminative data poisoning.  To overcome this, the paper introduces INF-push and INF-pull.  INF-push permutes the data points so their points' features in a learned model are maximally moved further from the centroid of their class's centroid, within a permutation budget.  In INF-pull, data points are permuted so their points' features are moved closer to the nearest centroid of an incorrect class.\n\nThe paper demonstrates these poisons effectively evade a defense of adversarial training, as well as several other reasonable defenses.", "summary_of_the_review": "A very good and interesting paper, which moves the ball forward on indiscriminate poisoning.", "strength_and_weaknesses": "**Strengths**\n\n- The paper is very clear and easy to read.\n- The approach is intuitive and effective.\n- The problem is relevant.\n- Visualizations are clear and informative.\n\n**Weaknesses**\n\nThe weaknesses are few. It's impact will not be world-changing, but it is a professionally written paper.  The description of Theorem 1 is awkward, as it uses notation and terms from another paper that is not introduced in this paper (for example, \"risk\" and $\\mathcal{R}$).  This should be easily remedied.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FVwtM64_YB", "reviewer": "Reviewer_VPK4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel technique to poison training data by crafting perturbations to either repel or attract training data to the corresponding class' mean feature vector. The authors demonstrate that this poisoning technique is superior to known availability attacks when defended against with adversarial training.", "summary_of_the_review": "I find the work interesting and well motivated, but I have serious questions about the setting for comparisons. I am willing to raise my score if these questions are addressed.", "strength_and_weaknesses": "### Pros:\n1) The attack is intuitive, and the results appear promising.\n2) The paper is clear and well written.\n3) Experimentation is fairly thorough, and a large number of related works are compared to.\n\n### Cons/Random things to address:\n1) I strongly encourage the authors to re-name their method and rephrase its positioning. The authors claim that their poisoning method is successful because it perturbs \"indiscriminative features\". However, in my opinion, there is little basis for this claim. It is also an unnecessary claim as the paper's results ostensibly justify the proposed method. In fact, I would argue that the poisoning technique does in fact perturb discriminative features - especially in the INF pull case as the perturbations by construction center the features of the poisoned data around the class mean - a descriminative quantity... Observing the visualizations of your perturbations would seem to confirm this. To their credit, the authors do try to justify this claim (briefly) in section 6, but TSNE visualizations are not convincing evidence of this claim. It's also not clear from what models the plots in Figure 5 were generated. Are they the features according to the reference model or the victim model after poisoning?\n2) Table 1 presents very encouraging results for your method. However, from Table 3, it appears that your attack success is reduced by using a smaller adversarial training $\\varepsilon$-budget. To present a fair comparison, Table 1 should instead show the *maximum* adversarial training accuracy a practitioner could achieve against each method - a practitioner would not use a higher than necessary adversarial training budget if a smaller one produces better results.\n3) If I understand correctly, there is another slightly unfair aspect to the comparisons to previous works. Your method crafts the perturbations using an adversarially trained reference model, and you show the superiority of your method compared to other methods in the face of an adversarial training defense. Did you adapt the other attacks and use an adversarially trained model for those results? If not, Table 1 doesn't present an apples-to-apples comparison. As you state \"More specifically, modifying the robust (semantic) features is the key to poisoning the AT models\". Maybe this is key for other methods as well.\n4) While I understand your point that defenders might not have access to a clean-trained model, or a clean AT model, I think this is a bit of a stretch. What if a defender could use a general purpose clean-trained feature extractor (widely available) to defend against your attack?\n5) Potentially the biggest issue I have is Figure 4a. If I'm reading this correctly, if the defender simply trained  in a standard fashion (not AT), the results in Table 1 would jump well into the 80's? Why would the defender choose to adversarially train then? \n\nOverall, I think the paper is interesting, and I'm willing to raise my score if my concerns are sufficiently addressed.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%