APBench: A Unified Availability Poisoning Attack and Defenses Benchmark
Authors: Tianrui Qin
Xitong Gao
Juanjuan Zhao
Kejiang Ye
Cheng-zhong Xu
Venue: Accepted by TMLR ()

Abstract: The efficacy of availability poisoning, a method of poisoning data by injecting imperceptible perturbations to prevent its use in model training, has been a hot subject of investigation. Previous research suggested that it was difficult to effectively counteract such poisoning attacks. However, the introduction of various defense methods has challenged this notion. Due to the rapid progress in this field, the performance of different novel methods cannot be accurately validated due to variations in experimental setups. To further evaluate the attack and defense capabilities of these poisoning methods, we have developed a benchmark — APBench for assessing the efficacy of adversarial poisoning. APBench consists of 9 state-of-the-art availability poisoning attacks, 8 defense algorithms, and 4 conventional data augmentation techniques. We also have set up experiments with varying different poisoning ratios, and evaluated the attacks on multiple datasets and their transferability across model architectures. We further conducted a comprehensive evaluation of 2 additional attacks specifically targeting unsupervised models. Our results reveal the glaring inadequacy of existing attacks in safeguarding individual privacy. APBench is open source and available to the deep learning community.

Published in Transactions on Machine Learning Research (08/2024)
APBench: A Unified Benchmark for Availability Poisoning Attacks and Defenses
Tianrui Qintabd, Xitong Juanjuu Zhao", Kejiang Ye Cheng-Zhong Xu
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
University of Chinese Academy of Sciences, Beijing. China.
Shenzhen University of Advanced Technology, Shenzhen, China.
Tencent Security Platform, Tencent, Shenzhen, China.
University of Macau, Macau S.A.R., China.
Reviewed on OpenReview:
Abstract
The efficacy of availability poisoning, a method of poisoning data by injecting imperceptible per- turbations to prevent its use in model training, has been hot subject of investigation. Previous research suggested that was difficult to effectively counteract such poisoning attacks. However, the introduction of various defense methods has challenged this notion. Duc to the rapid progress in this field, the performance of different novel methods cannot be accurately validated due to variations in experimental setups. To further evaluate the attack and defense capabilities of these poisoning methods, we have developed a benchmark - APBench for assessing the efficacy of adversarial poi- soning. APBench consists of 11 state-of-the-arr availability poisoning attacks, 8 defense algorithms, and 4 conventional data augmentation techniques. We also have set up experiments with varying different poisoning ratios, and evaluated the attacks multiple datasets and their transferability across model architectures. We further conducted comprehensive evaluation of additional attacks specifically targeting unsupervised models. Our results reveal the glaring inadequacy of existing attacks in safeguarding individual privacy. APBench is open source and available to the deep learning community
Introduction
Recent advancements of deep neural networks (DNNs) (LeCun et al., 2015 Schmidhuber 2015 He et al. 2016) heavily
rely on the abundant availability of data resources (Deng et al. 2009 Russakovsky al. 2015 Karras et al.
2020).
However, the unauthorized collection of large-scale data through web scraping for model training has raised concerns regarding data security and privacy. In response, new paradigm of practical and effective data protection methods
has emerged, known as availability poisoning attacks (APA)
Tao et al.
2021
Yuan & Wu 202T Fowl et al.
2021
Huang et al. 2021 Wu et al.) 2023 Fu et al.
2022 Ren et al. 2023 He et al. 2023 Sandoval- -Sgullr-Sraaaaa et al. 2022b
2023 or unlearnable example attacks. These poisoning Feng et al. 2019 Yu et al. 20022] He et al. 2023; Ren et al.
methods inject small perturbations into images that are typically imperceptible to humans, in order to hinder the model's 's ability to learn the original features of the images. Recently, the field of deep learning has witnessed advancements in
defense strategies (Liu al. 2023] Qin et al. 2023 Dolatabadi et al. 2023 (Hang et al. 202T) that hold the potential to challenge APAs, thereby undermining their claimed effectiveness and robustness. These defenses reveal the glaring inadequacy of existing APAs in safeguarding individual privacy in images. Consequently, anticipate an impending arms race between attack and defense strategies in the near future.
However, evaluating the performance of these new methods across diverse model architectures and datasets poses a significant challenge due to variations in experimental settings recent literatures. In addition, researchers face the
"qqual contribution.
T CCoresponneecee Xitong Gao (xt www.ggaatttaac..oomoo ht tps: : //git thub. com/ feat fest//

---

Published in Transactions on Machine Learning Research (08/2024)
daunting task of staying abreast of the latest methods and assessing the effectiveness of various competing attack-defense combinations. This could greatly hamper the development and empirical exploration of novel attack and defense strategies.
To tackle these challenges. we propose APBench, a benchmark specifically designed for availability poisoning attacks and defenses. involves implementing poisoning attack and defense mechanisms under standardized perturbations and training hyperparameters, in order to ensure fair and reproducible comparative evaluations. APBench comprises range of availability poisoning attacks and defense algorithms, and commonly used data augmentation policies. This comprehensive suite allows to evaluate the effectiveness of the poisoning attacks and the robustness of the defense strategies across multiple datasets and model architectures.
Our contributions can be summarized as follows:
An open source benchmark for state-of-te of-ttheart availability poisoning attacks and defenses, including 9 supervised and unsupervised poisoning attack methods, 8 defense strategies and 4 common data augmentation methods.
We conduct comprehensive evaluation competing pairs of poisoning attacks and defenses under standardized experimental settings.
We carried out experiments across publicly available datasets, and also extensively examined scenarios of partial poisoning, increased perturbations, the transferability of attacks to 4 CNN and 4ViT models under various defenses, and unsupervised learning. We provide visual evaluation tools such t-SNE, Shapley value map and Grad-CAM to qualitatively analyze the impact of poisoning attacks.
The aim of APBench is to serve as a catalyst for facilitating and promoting future advancements in both availability poisoning attack and defense methods. By providing platform for evaluation and comparison, we aspire to pave the way for the development of future availability poisoning attacks that effectively preserve utility and protect privacy.
2
Related Work
2.1 Availability Poisoning Attacks
Availability poisoning attacks (APAs) belong to a category of data poisoning attacks (Goldblum et al.) 2022) that adds a small perturbation images, that is often imperceptible humans. The objective contrasts with the malicious intent of traditional data poisoning instead, the purpose of these perturbations is to protect individual privacy from deep learning algorithms, preventing DNNs from effectively learning the features present in the images. The attacker's goal is to render their data unlearnable with perturbations, hindering the unauthorized trainer from utilizing the data to learn models that can generalize effectively to the original data distribution. The intent of APAS is thus benign rather than malicious as generally assumed of data poisoning attacks. We typically assume that the attacker publishes (a subset of) the images, which get curated and accurately labeled by the defender to train on, typically without consent from the attacker.
Formally, consider a source dataset comprising original examples Detean [xx), .I....., "X),....,,,,,, Bn)3 where x; E X' denotes input image and y, Y' represents its label. The objective of the attacker is thus to construct set of availability perturbations 5, such that models trained on the set of availability poisoned examples (p) {(x + 8x, y) (x, y) E Petemn are expected to perform poorly when evaluated on a test set Drest sampled from the distribution S:
max E (x (C (C(o 9:]],
(1)
s.t. (8) argmin E E (D..) L ((fo
where L denotes the loss function, usually the softmax cross-entropy loss. In order to limit the impact on the original utility of images, the perturbation d, is generally constrained within small e-ball of é, distance.
In the realm of adversarial machine learning, key challenge is to enforce a small perturbation budget while maximizing the disruptive impact the target model. To address this challenge, recent methods have typically constrained their
2

---

Published in Transactions on Machine Learning Research (08/2024)
perturbations within 4p perturbation, where p E {0, 2, oo). This constraint ensures that the perturbations relatively imperceptible to humans but can still have significant impact on the behavior of the target model.
First, we introduce the APAs examined in APBench below:
DeepConfuse (DC) (Feng et al. 2019) DC is a precursory APA which uses autoencoders generate unlearning perturbations. Its key idea involves employing autoencoder-Hi like network to capture the training trajectory of the target model and adversarially perturbing the training data.
Neural tangent generalization attacks (NTGA) Yuan & Wu 2021 x NTGA approximates the targeted model training as a Gaussian process (Jacot et al. 2018), and leverages this surrogate to find better local optima with improved transferability.
Error-minimizing attack (EM) (Huang et al., 2021): EM trains surrogate model by minimizing the error of images relative to their original labels, generating perturbations that minimize the errors and thus render the perturbed images unlearnable. The authors of EM introduce the threat model of availability poisoning attacks, highlighting their role as a mechanism for privacy protection.
(Fu et al. 2022;: REM is a natural extension of EM to generate attack perturbations with adversarial training of the surrogate model. This approach aims enhance attack robustness against adversarial training-based based defenses.
Robust error-minimizing attacks (REM)
Hypocritical (HYPO) (Tao et al. 2021): HYPO, similar to EM, generates error-minimiiing perturbations akin to those used in EM but employs a pretrained surrogate model.
Targeted adversarial poisoning (TAP) (Fowl et al. 2021): TAP achieves availability poisoning by generating targeted adversarial examples of non-ground-trute labels of pre-trained models. shows that adversarial attacks can also be effective APAs.
Linear-separable poisoning (LSP) (Yu et al., 2022): LSP is prescription-based APA which algorithmically generates randomly initialized linearly separable color block perturbations, enabling effective APAs without requiring surrogate models or excessive computational overhead.
Autoregressive Poisoning (AR) (Sandoval-Segura al. 2022b): AR, similar to LSP, is prescription-based and does not require additional surrogate models. It fills the initial rows and columns of each channel with Gaussian noise and uses autoregressive process fill the remaining pixels, generating random noise perturbations that are linearly separable.
Ahe-PixelSShorteu (OPS) (Wu et al. 2023): OPS is a targeted availability poisoning attack that perturbs only one pixel per image, generating an effective availability poisoning attack against traditional = bbounded adversarial training methods.
Contrastive Poisoning (UCL) (He et al., 2023) uses the contrastive loss (e.g., g., the InfoNCE loss) to generate perturbations that are effective under unsupervised learning settings.
Transferable Unlearnable Examples (TUE) (Ren et al. 2023) boosts the transferability of unlcarnable examples with linear separable poisons produced with the class-wise separability discriminant.
These various attack strategies underscore the complexity and multi-faceted nature of adversarial machine learning, where the interplay between attack and defense mechanisms continues to evolve. Finally, Table provides catego- rization of the characteristics of the above APAs, and the visual examples provided in Figure highlights the visual differences across availability poisoning attacks.
2.2 Availability Poisoning Defenses
The goal of the defender is successfully train model with good generalization abilities (e.g., test accuracies on natural unseen images) protected data. Generally, the defender can control the training algorithm, and only have
3

---

Published in Transactions on Machine Learning Research (08/2024)
Table I: Availability poisoning attack algorithms implemented in APBench. "Type" and "Budget" respectively denotes the type of perturbation and its budget. "Mode" denotes the training mode, where "S" and "U" and respectively mean supervised and unsupervised training. "No surrogate" denotes whether the attack requires access to a surrogate model for perturbation generation. "Class-wise" and ""Samplewise" indicate if the attack supports class-wise and sample-wise perturbation generation. "Stealthy" denotes whether the attack is stealthy to human.
Attack Method
Type
Budget
Mode
No surrogate
Class-wise
Sample-wise
Stealthy
DC (Feng et al.
2019
NTGA Yuan & Wu 2021
HYPO Tao et al.
2021
S
EM (Huang et al.
2021
f,0
8/255
REM [uu al. 22
TAP
Fowl et 202
S
UCL
He et al.
2023)
TUE
Ren
2023
LSP
Yu et al.
2022)
1.30
8
AR
Sandoval Segura al. 2022b)
1.00
OPS
Wu et
2023)
DC
NTGA
EM
REM
TAP
НУPО
LSP
AR
OPS
Clean
Ex -//255
8 = 1.30
8 1.00
bn -
Figure 1: Visualizations of unlcarnable CIFAR-10 images with corresponding perturbations. Perturbations are normal- ized for visualization.
access to a training data set with data poisoned either partially or fully. The objective of the defender is thus to find a novel training algorithm (Dppi) that trains models to generalize well to the original data distribution:
ming B(x (...).DDDD.. [C(So (x:), v:)], s.t. 0* ((Dpi)).
Notably, the method employs the standard training loss but performs novel image transformations h, then g can further specialized as follows:
g (Poii) argming ((E...)) ((fe(h()))) v:).
(3)
Currently, defense methods against availability poisoning can be mainly classified into two categories: preprocessing and thining-phasse defenses. Data preprocessing methods preprocess the training images to climinate the poisoning perturbations prior to training. Here, we summarize the defense methods implemented in APBench, and Table 2
categorizes the above defense methods in terms of their time costs, types, and descriptions:
Adversarial training (AT) (Madry et al. 2017; AT a widely-recognizz effective approach against availability poisoning Huang al. (2021): Fu al. (2022). Small maximizing adversarial perturbations are applied the training images during training, in order improve the robustness of the model against APA perturbations.
4

---

Published in Transactions on Machine Learning Research (08/2024)
Image Shorteut Squeezing (ISS) (Liu et al. 2023):
ISS uses traditional image compression techniques such as grayscale transformation, bit-depth reduction (BDR), and JPEG compression, as defenses against availability poisoning.
Early stopping (ES) (Sandoval-Segura et al. 2022a): As training on availability poisons typically leads to early overfitting on availability poisons, early stopping, i.e., stopping the training process when the model reaches peak validation accuracy, may be practical as defense mechanism.
Adversarial augmentations (UEraser-Lite, UEraser-Max) (Qin et al. 2023): APBench implements two flavors of UEraser. First, similar ISS, UEraser-Lite uses effective augmentation pipeline suppress availabiliity poisoning shorteuts. UEraser-Max further improves the defense against availability poisoning with emoor-mximizing adversarial augmentations. Specifically, UEraser-Max samples multiple augmentations per image, and train models on the augmented images with maximum loss to prevent learning from poisoning shortcuts.
AVATAR (Dolatabadi et al. 2023 Inspired by DiffPure (Ni et al., 2022) which improves adversarial robustness by using diffusion models to remove adversarial perturbations, AVATAR similarly cleans the images of the unlearnable perturbations with diffusion models.
Standard data augmentations: For referential baselines, APBench also includes commonly used data augmentation techniques such as Gaussian blur, random crop and flip (standard training), CutOut (DeVries & Taylor, 2017). CutMix (Yun et al. (201)), and MixUp (Zhang al. 2018), and show their (limited) effect in mitigating availability poisons.
Table 2: Availability poisoning defense algorithms implemented in APBench.
Defense Method
Type
Time Cost
Description
Standard
Low
Random image cropping and flipping
CutOut
DeVries & Taylor
2017)
Low
Random image erasing
Data augmentations
CutMix Yun et al. 2019)
Low
Random image cutting and stitching
MixUp Zhang et al.
2018
Low
Random image blending
Gaussian (used in LLu al.
2023
Low
Image Gaussian blurring
BDR (used (Liu et al. 202.1
Low
Image bit-depth reduction
Grayscale (used in (Liu al. 2023
Data preprocessing
Low
Image grayscale transformation
JPEG (used in (Liu al. 202.1
Low
Image compression
AVATAR (Dolatabadi et al. 2023
High
Image corruption and restoration
Early stopping Seindoval--eguree et al.
2022a
Low
Finding peak validation accuracy
UEraser-Lite Lite (Qin 2023
Low
Stronger data augmentations
Training-phan defense
UEraser- Max (Qin ct al
2023
High
Adversarial augmentations
AT (Madry al. 2017)
High
Adversarial training
2.3 Related Benchmarks
Availability poisoning is closely connected to the domains of adversarial and backdoor attack and defense algorithms. Adversarial attacks primarily aim to deceive models with adversarial perturbations during inference to induce misclassi- fications. There are several libraries and benchmarks available for evaluating adversarial attack and defense techniques,
such as Foolbox (Rauber et al. 2020). AdvBox (Goodman et al.) 2020), and RobustBench (Croce et al.) 2020).
Backdoor data poisoning attacks (Cinà et al. 2023) focus injecting backdoor triggers into the training algorithm or data respectively, causing trained models to misclassify images containing these triggers while maintaining or minimally impacting clean accuracy. Benchmark libraries specifically designed for backdoor attacks and defenses include TrojanZoo (Pang et al. 2020)), Backdoorbench (Wu et al., 2022), and Backdoorbox (Li et al.) 2023). Moreover, Schwarzschild et al. (202T) also introduced benchmarks for data poisoning and backdoor attacks, which explores their ability to generalize to different testing settings. Interestingly, it examines the "triggerless" attack scenario, where it forces the model to misclassify specific target image, opening backdoor in the model for post-training attack
5

---

Published in Transactions on Machine Learning Research (08/2024)
opportunities. In contrast, attacks in APBench aims prevent the model from learning from features present in the entire poisoned set of images to preserve user privacy. Geippng et al. (202T) extends the benchmark to include additional data poisoning attacks. To summarize, the above benchmark suites all examine malicious attacks in order to control the model's behavior, whereas the goal APAs considered in APBench is benign, aiming to protect individual privacy by perturbing their own personal data.
There currently lack and an urgent need of a dedicated and comprehensive benchmark that standardizes and evaluates availability poisoning attack and defense strategies. To the best of our knowledge, APBench is the first benchmark that fulfills this purpose. It offers extensive library of recent attacks and defenses, explores various perspectives, including the impact of poisoning rates and model architectures, well as attack transferability. We hope that APBench can make significant contributions to the community and foster the development of future availability attacks for effective privacy protection.
3 A Unified Availability Poisoning Benchmark
Input Module
Attack
Module
pertebatoos
Survogate model
Clean data
Searched parturludions
Poisoned data
Clean data
Pattern search
Defense
Evaluation Module
Module
Preprocessing based defense
Purifled data
Poisens evluation
Training basod defense
Model training
Model Architecture
ESNE
Poisoned data
Q Loss objective
Grad-CAM
& Shapley value map
Figure 2: The overall system design of APBench.
As shown in Figure 2 APBench consists of three main components: (a) The availability poisoning attack module. This library includes set of representative availability poisoning attacks that can generate unlearnable versions of given clean dataset. (b) The poisoning defense module. This module integrates a suite state-of-the of-the-art defenses that focus on mitigating the unlearning effect and restore clean accuracies to a certain extent. (c) The evaluation module. This module can efficiently analyze the performance of various availability poisoning attack methods using accuracy metrics and visual analysis strategies.
We built extensible codebase as the foundation of APBench. In the attack module, provide total of availability poisoning attacks of 3 different perturbation types (ep) for supervised learning, and 2 attacks for unsupervised learning. For each availability poisoning attack method, we can generate their respective poisoned datasets. This module also allows us to expand to different perturbations budgets and poisoning ratios, and easily extend to future poisoning methods. Using the poisoned datasets generated by the attack module, we can evaluate defenses through the defense module, which currently includes defense algorithms and 4 conventional data augmentation methods. The goal of this module is to ensure that models trained on unlearnable datasets still generalize well clean data. The defense

---

Published in Transactions on Machine Learning Research (08/2024)
module primarily achieves poisoning mitigation through data preprocessing taining-phasee defenses. Finally, the evaluation module computes the accuracy metrics of different attacks and defense combinations, and can also perform qualitative visual analyses to help understand the characteristics of the datasets.
4 Evaluations
4.1 Setups
Datasets We evaluated our benchmark on 4 commonly datasets (CIFAR-10 Krizhevsky et al. 2009). CIFAR-
100 Krizhevsky et al. 2009), SVHN (Netzer et al. 201T). and an ImageNet (Deg et al. 2009) subset) and 7 mainstream models, comprising 5 CNN-based models (ResNet-18 18 (He et al. 2016). ResNet-55 (He et al. 2016)
MobileNetV2 (Sandler et al. 2018), and DenseNet-121 (Huang et al. 2017)) and ViT models (ViT-small (Dosovitskiy at 202T) and CaiT-small TTourron et al., 2021)). To ensure a fair comparison between all methods, we evaluate the methods on the general and standardized model architectures. Appendix [ further describes the detail specifications of the datasets and the respective test accuracies achievable through standard training clean training data.
Attacks and defenses For complete introduction of the APA attacks and defense methods, please refer to Section
Types of Threat Models We classify APAs based on three distinct perturbation types: fx- attacks (DC, NTGA, EM, REM, TAP, and HYPO); l fa-bounded attacks (LSP and AR); an to lo-bounded attack (OPS). Notably, the ly-bounded OPS attack demonstrates remarkable resilience against a wide range of defenses, resisting disruption from image preprocessing augmentations and remaining unaffected by foo adversarial training. Conversely, in terms of stealthiness, the to attacks are less subtle than their for and f2 counterparts, illustrated in Figure 1 Visually, perturbations bounded by both loc and & are comparable w.r.f. the degree of visual stealthiness and effectiveness. Finally, as the two fz-bounded attacks (LSP and AR) do not require surrogate model training, they are thus more efficient in the unlearnable examples synthesis.
Training settings We trained the CIAFR-IO, CIFAR-I00 and ImageNet-subset models for 200 epochs and the SVHN models for 100 epochs. We used the stochastic gradient descent (SGD) optimizer with momentum of 0.9 and learning rate of 0.1 by default. As for unsupervised learning, all experiments are trained for 500 epochs with the SGD optimizer. The learning rate is 0.5 for SimCLR (Chen et al. 2020a) and 0.3 for MoCo-v2 (Chen et al., 2020b). Please note that we generate sample-wise perturbations for all APAs. Specific settings for each defense method may have slight differences, and detailed information can be found in the Appendix A
Evaluation objectives In this section, we first evaluated the effectiveness of APA attacks against the defense methods (Section 4.2). Moreover, we explored challenging scenarios with real-world constraints, such as varying data poisoning rates and perturbation magnitudes on different models, assessed the transferability of attacks across different model architectures (Section 4.3). and APAs under unsupervised learning (UCL (He al., (023) and TUE (Ren al. 2023 The implementation details of the algorithms and additional results can be found in Appendix A and Appendix B respectively. Below, we present the key takeaways from our evaluations:
Key Takeaways
1. The efficacy of availability poisoning attacks in terms of hindering the model's generalization ability is suboptimal when the poisoning rate is relatively low. However, it can still ensure the effectiveness of privacy protection for data that has been poisoned, when no defense is employed.
2. Nearly all availability poisoning attacks exhibit high degree of transferability across diverse model architectures, with the exception of AR, which appears to be ineffective on CaiT-small. This suggests that current attacks may exploit common training algorithmic flaws rather than model-specific vulnerabilities. Furthermore, all defenses demonstrate similar defense performance on different architectures.
3. For defenses, the most promising method is based on pretrained diffusion generative models, e,g., AVATAR. Without the availability of additional data or pretrained models, UEraser variants stand out as the most effective defense methods.
7

---

Published in Transactions on Machine Learning Research (08/2024)
4.2 Standard Scenario
To start, we consider a common scenario where both the surrogate model and target model are ResNet-18 and the poisoning rate is set to 100%. We first evaluate the performance of the supervised poisoning methods against the defense mechanisms and commonly used data augmentation strategies. Table 3 and Table present the evaluation results on CIFAR-IO from our benchmark.
It is evident that the conventional data augmentation methods appear be ineffective against all poisoning methods. Yet, even simple image compression methods (BDR, grayscale, and JPEG corruption from ISS (Liu al. 2023 demonstrate notable effect in mitigating the poisoning attacks, but fails achieve high clean accuracy.
Despite requiring more computational cost or additional resources (pretrained diffusion models for AVATAR), methods such as UEraser-Max (Qin et al. 2023) and AVATAR (Dolatabadi et al. 2023), generally surpass the image compression methods from ISS terms effectiveness. While AVATAR is inferior to UEraser-Max in gaining accuracy, it decouples the defense into an independent data sanitization phase, making more universally applicable, but requiring a pretrained diffusion model. While the early stopping (ES) method can be somewhat effective defense, is not usually considered good one, with subpar peak accuracies. Adversarial training appears effective but in many cases is outperformed by JPEG compression. Notably, it also fails against OPS, the Lo perturbation budget cannot mitigate to threats.
Table Test accuracies (%) of models trained on poisoned CIFAR-10 datasets. The model trained on a clean CIFAR-10 dataset attains accuracy of 94.32%. We highlight best defenses in bold (highest aceuracy for the same attack), and best attacks with underline (lowest accuracy for the same defense). "ND" No Defense, "ST" - Standard training, "CO" CutOut, "CM" CutMix, "MU" MixUp, "Gauss." Gaussian, "BDR" Bit-Depth Reduction, "GS" Grayscale, "JPEG" JPEG Compression, "AVA" AVATAR, "ES" Early Stopping. "U-Max" UEraser-Mar Maxx and "AT" adversarial training.
Method
ND
ST
CO
CM
MU
Gauss.
BDR
GS
JPEG
AVA
ES
U-Max
AT
DC
14.17
15.19
19.94
17.91
25.07
16. 10
67.73
85.55
83.57
82.10
26.08
92.17
76.85
EM
19.68
20.78
18.79
22.28
31.14
14.71
37.94
92.03
80.72
75.62
25.39
93.61
82.51
REM
17.81
17.47
21.96
26.22
43.07
21.80
58.60
92.27
85,44
82.42
31.32
92.43
77.46
НУРО
69.04
70.38
69.04
67.12
74.25
62.17
74.82
63.35
85.21
85.94
70.52
88.44
81.49
NTGA
17.82
22.76
13.78
12.91
20.59
19.95
59.32
70.41
68.72
86.22
28.19
86.78
69.70
TAP
4.34
6.27
9.88
14.21
15.46
7.88
70.75
11.01
84.08
87.75
39.54
79.05
79.92
LSP
10.19
13.06
14,96
17.69
18.77
18.61
53.86
64.70
80.14
76.90
29.10
92.83
81.38
AR
12.63
11.74
10.95
12.60
14.15
13.83
36.14
35.17
84.75
88.60
44.29
90.12
81.15
OPS
11.94
14.69
52.98
64.72
49.27
13.38
37.32
19.88
78.48
66
38.20
77.99
14.95
Table 4: Test accuracies (%) of ViT-small models trained on poisoned CIFAR-I0 datasets. The model trained from scratch on a clean CIFAR-10 dataset attains an accuracy of 84.66%. Value highlight rules and method abbreviations are the same as Table
Method
ND
ST
CO
CM
MU
Gauss.
BDR
GS
JPEG
AVA
U-Max
AT
DC
20.37
22.94
26.03
32,80
33.64
26.06
55.47
72.45
71.82
79.16
26.49
82.89
77.26
EM
36.82
37.14
38.79
40.30
46.91
36.29
54.59
59.38
70.69
72.81
41.29
81.64
78.03
REM
30.61
31.72
29.90
34.68
44.99
33.14
50.59
80.29
74.63
75.52
42.88
82.50
79.32
НYPO
74.47
75.23
75.76
74.60
76.16
75.12
79.21
63.62
75.77
83.15
67.29
83.50
78.62
TAP
22.38
22.93
22.65
24.86
27.82
23.47
40.03
37.75
67.26
73.26
31.17
69.83
76.55
NTGA
26.40
28.27
22.14
23.39
39.58
25.57
52.38
56.92
53.94
74.88
33.26
76,39
77.08
LSP
27.14
29.06
30.17
32.66
28.38
33.07
41.17
59.34
68.07
66.74
32.69
87,01
76.94
AR
22.37
25.04
26.92
21.18
30.67
25.48
37.04
38.90
74.77
78.64
45.54
63.90
75.62
OPS
18.16
20.84
61.27
76.59
34.58
32.39
45.71
31.60
69.31
59.26
22.63
66.72
24.30
8

---

Published in Transactions on Machine Learning Research (08/2024)
We further conduct experiments representative poisoning methods (EM, REM, LSP and AR) the CIFAR-100 SVHN, and ImageNet-suubet datasets, and the results are shown in Table
Table 5: Test accuracies (%) poisoned CIFAR- 100, SVHN and ImageNet-subseet (IN-100) datasets. Value highlight
rules and method abbreviations are the same as in Table 3
Dataset
Method
ND
ST
CO
CM
MU
Gauss.
BDR
GS
JPEG
ES
U-Max
EM
2.94
3.03
4.15
3.98
6.46
2.99
34.10
59.14
58.71
7.06
68.81
CIFAR-
REM
3.57
3.73
4.00
3.71
10.90
3.59
29.16
57.47
55.60
10.99
67.72
100
LSP
2.39
2.56
2.33
4.52
4.86
1.71
27.12
39.45
52.82
9.52
68.31
AR
2.04
1.87
1.63
3.17
2.35
2.62
31.15
16.13
54.73
26.58
55.95
EM
11.07
10.33
13.38
10.77
12.79
8.82
36.65
65.66
86.14
13.47
90.24
REM
12.95
14.02
18.92
9.55
19.56
7.54
42.52
19.59
90.58
19.61
88.26
SVHN
LSP
11.38
12.16
12.98
8.17
18.86
7.15
26.67
16.90
84.06
12.91
90.64
AR
17.69
19.23
14.92
6.71
13.52
7.75
39.24
10.00
92.46
89.32
90.07
EM
1.90
2.94
4.05
4.73
4.15
3.15
6.45
12.20
31.73
8.80
44.07
IN- 100
REM
3.82
3.66
4.13
4.78
3.94
4.28
4.03
3.95
40.98
17.19
42.14
LSP
30.64
38.52
40.56
29.78
7.85
42.68
26.58
25.18
36.83
39.52
63.28
To summarize, our findings indicate that perturbations constrained by traditional &p norms are ineffective against adversarial augmentation (UEraser-Max), and image restoration by pretrained diffusion models (AVATAR), as they break free from the assumption of lp constraints. Even simple image compression techniques (JPEG, Grayscale, and BDR) can effectively remove the effect of perturbations. At this moment, APAS that rely on ep-uuneded perturbations may not as effective as initially suggested by the relevant publications.
4.3 Challenging Scenarios
To further investigate the effectiveness and robustness of APA attacks and defenses, we conducted evaluations in more challenging scenarios with real-world constraints. Namely, we considered scenarios include partial poisoning, larger perturbation budgets, and the attack transferability to different models.
Privacy protection under partial poisoning In realistic scenarios, it is difficult for attacker achieve modification of the entire dataset. We thus investigate the impact of partial poisoning rates, where only portion of the training data is perturbed by APAs. This raises the following question: Are APAs effective in protecting only portion of the training data? To answer, we introduce poisoning perturbations with APAS to a varying portion of the training data, and investigate how well the models learn the origin features that exist in the poisoned images for different poisoning rates. Specifically, we explore different poisoning rates (20% 40%, 60%, 80%). Figure evaluaaee and compares the mean losses of the unlearnable images used during training ("Unlearnable"), the origin images of the unlearnable part ("Clean"), and for reference, the mean losses of images unseen by the model from the test set ("Test"), and "Train" means the loss of the clean part of the training set. For similar view the accuracies, please refer to Figure 3 We find that the losses on the original images of the unlearnable part is similar to that of the test set, or even lower. This suggests that APAS can reasonably protect the private data against undefended learning. For results under defenses, please refer to Appendix B
Larger perturbations We increased the magnitude of APA perturbations to further evaluate the performance of attacks and defenses. Table presents the results of APAs with larger perturbations on CIFAR-10. We note that larger perturbations indeed have more pronounced impact on suppressing defense performance, leading significant accuracy losses for all defense methods. There exists a trade-off between perturbation magnitude and accuracy recovery. Considering that at larger perturbations, APAs generate images that are perceptually of lower quality, yet some defense methods are still effective, which may make such attacks less practical in real-world scenarios.
Attack transferability across models In real-world scenarios, APAs can only manipulate the data and do not have access to specific details of the defender. Therefore, we evaluated the transferability of APAs by using ResNet- 18 as the surrogate for the surrogate-base based attacks, and transfer the perturbed data be trained on other models, including CNN-based models (ResNet-50, SEN MobileNerV2, DenseNet-121) 121), and 4 VIT models (ViT-small, and
9

---

Published in Transactions on Machine Learning Research (08/2024)
Table 6: Test accuracies (%) on poisoned CIFAR-10 datasets with increased perturbations. We only highlight the best defenses per attack in bold, as the attacks may have different perturbation budgets.
Method
Type
Budget
Standard
Grayscale
JPEG
ES
UEraser-Max
AT
8/255 16/255
20.78
92.03
80.72
25.39
93.61
82.51
EM
18.74
76.76
55.96
27.39
88.09
77.82
8/255
17.47
92.27
85.44
31.32
92.43
77.46
REM
60
16/255
19.80
83.65
80.07
33.07
80.36
75.64
1.30
13.06
64.70
80.14
29.10
92.83
81.38
LSP
l
1.74
15,83
37.60
42.83
27.30
87.20
77.92
1.00
11.74
35.17
84.75
44.29
90.12
81.15
AR
1.50
11.20
26.10
78.24
20.96
68.42
70.14
CaiT-small). The results are shown in Table 7 is evident that all surrogate-basee based and -free poisoning methods exhibit strong transferability, while the three recently proposed defenses also achieve successful defense across different model architectures. The only exception the AR method, which fails against CaiT--smal.
Adaptive poisoning We evaluated strong adaptive poisons against various defenses using two poisoning methods, EM (Huang al. 2021) and REM (Fu et al. 2022). We assume that the defenders can be adapted to three defenses (Grayscale, JPEG, and UEraser), by using the attack in the perturbation generation process. From Tables 8 and it can be seen that adaptive poisoning significantly affects the performance of the Grayscale defense, but has less effect on JPEG and UEraser.
Unsupervised learning We evaluated the APAs targeting unsupervised models on CIFAR-10. We considered 2 popular unsupervised learning algorithms: SimCLR (Chen al., 2020a) and MoCo-v2 (Chen al., 2020b). All defense methods applied by preprocessing images prior to the unsupervised learning. Therefore, used UEraser-L Lite data preprocessing method by sampling the augmented images instead of as training- defense. The results of all experiments are shown in Table 10
4.4 Other Evaluations
Adversarial robustness of trained models Table II additionally compared the robustness of the trained models CIFAR-10 under the projected gradient descent (PGD) adversarial attack with a perturbation budget of 8/255. Note that all experiments considered both scenario including white-box attacks and black-box transferability from similarly-trained model with different initialization. The PGD attack used 20 steps with step size of 2/ 2/255. As
Unlearnahle
Original
Unseen
Train
100
100
100
100
95
85
85
85
80
80
80
80
75
75
75
75
60
40
60
20
40
20
40
80
(a) EMM
(b) REM.
(c) L.P..
(d) AR.
Figure 3: The accuracies (%, vertical axes) indicate that original features unlearnable examples are not learned by the model. All evaluations consider partial poisoning scenarios (poisoning rates from 20% to 80%, horizontal axes). Note that "Unleamable" and "Original" respectively denote the set of unlearnable examples and their original clean variants, and "Unseen" denote images from the test set unobserved during model training.
10

---

Published in Transactions on Machine Learning Research (08/2024)
Unlcarnable
Original
Unseen
Train
10
10
10
10
10
10
10
10
10
(a) EM.
(b) REM.
(c) LSP.
(d) AR.
Figure 4: The mean losses (vertical axes) indicate that original features in unlearnable examples are not learned by the model. The evaluation setting follows Figure 3
Table Clean test accuracies of different CIFAR-10 10 target models, where attacks are oblivious to the model architectures. Note that AR and LSP are surrogate- free, and for EM and REM the surrogate model is ResNet-18.
Model
Clean
Method
Standard
Grayscale
JPEG
ES
UFraser-Max
AVATAR
EM
14.41
83.40
76.88
26.69
85.89
77.64
REM
16.26
87.26
75.79
31.37
92.69
83.68
ResNet-50
94.47
LSP
19.23
68.94
73.24
32.73
93.08
76.47
AR
11.83
27.51
80.24
28.66
81.40
86.39
EM
13.60
86.03
79.35
16.35
83.27
74.22
REM
20.99
84.50
78.92
22.85
93.17
84.37
SENet-18
94.83
LSP
18.54
65.06
76.51
26.38
92.53
75.19
AR
13.68
34.26
79.29
37.04
75.06
84.37
EM
15.62
77.21
70.96
16.71
82.71
75.62
REM
20.83
80.81
72.27
21.92
91.03
82.77
MobileNetV2
94.62
LSP
16.82
61.07
72.03
28.12
92.10
76.81
AR
13.36
28.54
68.14
39.45
73.40
81.63
EM
13.89
82.49
78.42
15.68
82.37
76.69
REM
21.45
85.47
78.42
22.35
93.09
83.04
DenseNet-121 121
95.08
LSP
18.94
67.95
74.90
26.86
93.47
78.22
AR
13.43
25.51
81.12
36.51
82.36
89.92
EM
21.47
80.42
72.64
30.91
74.29
54.84
REM
32.17
79.65
74.92
43.07
83.27
73.57
ViT-small
84.66
LSP
29.06
59.34
68.07
32.69
87.01
66.74
AR
25.04
38.90
74.77
45.54
63,90
78.64
EM
17.01
64.76
63.75
39.69
63.37
41.94
REM
26.11
65.05
66.43
47.39
72.05
62.53
CaiT-small
71.96
LSP
25.08
63.06
57.15
37.95
70.92
51.39
AR
68.63
66.27
69.30
67.41
70.04
62.77
Table 8: Test accuracies (%) of adaptive poisoning with
Table 9: Test accuracies (%) of adaptive poisoning with REM on ResNet-18.
EM on ResNet-13 18.
EM
Standard
Grayscale
JPEG
U-Max
REM
Standard
Grayscale
JPEG
U-Max
Grayscale
19.48
21.64
78.39
90.52
Grayscale
16.70
56.33
82,47
91.37
JPEG
20.67
90.29
76.25
93.22
JPEG
19.45
91.71
75.84
92.53
UBraser
35.24
88.62
80.46
89.55
UEraser
21.61
89.26
77.51
91.84
11

---

Published in Transactions on Machine Learning Research (08/2024)
Table 10: Performance APA attacks and defenses on different unsupervised learning algorithms and datasets. Note that "U-Lite" denotes LEreser-Lite.
Algorithm
Method
Standard
Grayscale
JPEG
AVATAR
UEraser-Lite
UCL
47.25
46.91
66.76
83.22
68.42
SimCLR
TUE
57.10
56.37
67.54
84.24
66.59
UCL
53.78
53.34
65.44
83.08
72.13
MoCo-v2
TUE
66.73
64.95
67.28
82.48
74.82
Table 11: Test accuracies (%) on CIFAR-10 PGD-20 adversarial examples.
Type
Method
Standard
Grayscale
JPEG
AVATAR
UEraser-Max
AT
EM
16.83
19.77
73.85
20.07
71.43
69.77
REM
22.53
19.16
73.70
17.36
80.46
70.74
Black-box
LSP
14.29
24.34
67.01
15.42
80.40
71.39
AR
15,82
16.55
75.21
16.84
76.97
68.81
EM
0.00
0.00
3.74
0.00
1.37
28.60
REM
0.00
0.00
3.35
0.00
3.11
32.52
White-box
LSP
0.00
0.00
0.39
0.00
3.83
34.45
AR
0.00
0.00
2.59
0.00
2.83
32.04
expected, adversarial training provides the best defense against white-box attacks. While standard training, Grayscale and AVATAR failed gain adversarial robustness, interestingly. JPEG compression and UEraser-Mas demonstrated small improvements against white-box attacks, and notable robustness against black-box attacks, even surpassing that of adversarial training.
Visual analyses We provide visualization tools (Grad-CAM (Selvaraju et al. 2017) and Shapley Value Maps (Lundberg & Lee, 2017)) facilitate the analysis and understanding of APAs. We also use t-SNE (Van der Maaten Hinton 2008) visualize the availability poisons (Figure 7 Although t-SNE cannot accurately represent high-dimensional spaces, it aids in the global visualization of feature representations, allowing to observe specific characteristics of availability poisons. For additional discussions the visualizations, please refer to Appendix B.1
5 Discussions
Future outlook of APAS Future research directions on APAs should explore methods that enhance the resilience of perturbations. Promising future directions of APAs include generalizable attacks, which can simultancously target the DNNs being trained, leveraging diffusion models for stronger attacks, and how remain robust against color distortions- based defenses. Moreover, semantic- based perturbations that breaks away from the constraints of lp norms may offer an alternative strategy to enhance the effectiveness of APAs while preserving the perceptual quality of images. Such modifications to images can be challenging to remove by a wide-range of defenses.
Limitations APBench has mainly focused on providing algorithms and evaluations related to image data. However, such availability poisoning methods may also be applicable to text, speech, or video domains. the future, we plan to expand APBench to include more domains, aiming to establish more comprehensive benchmark for personal privacy protection across various modalities against deep learning.
Reproducibility Statement We provide open-source implementation of all attacks and defenses in the supplementary material. Following the README file, users should be able to run all experiments on their own machines to reproduce the results shown in paper.
Ethics Statement Similar to many other technologies, the implementation of APAs can used for both beneficial and malicious purposes. We understand that APAs were originally proposed to protect privacy, but they can also be used to generate maliciously data introduce model backdoors. The benchmark aims to promote understanding of
12

---

Published in Transactions on Machine Learning Research (08/2024)
various APA attack and defense methods, as well encourage the development of new algorithms in this field. is also important for us to raise awareness of the false sense of security provided by APAS However, we emphasize that the use of these algorithms and evaluation results should comply with ethical guidelines and legal regulations. We encourage users to be aware of the potential risks of the technology and take appropriate measures to ensure its beneficial use for both society and individuals.
6 Conclusions
We have established the first comprehensive and up-to-date benchmark for the field of availability poisoning, covering a diverse range of availability poisoning attacks and state-of-the-art defense algorithms. We have conducted evaluations and analyses of different combinations attacks and defenses, as well as additional challenging scenarios. Through this new benchmark, our primary objective to provide researchers with an in-depth empirical and comparative analysis of the current progress in the field of availability poisoning attacks and defenses. We hope it can enable rapid comparisons between existing methods and new approaches, while also inspiring fresh ideas through our comprehensive benchmark and analysis tools. We believe that our benchmark will contribute to the advancement of availability poisoning research and the development of more effective attacks to safeguard privacy in the future.
Acknowledgements
This work is supported in part by National Natural Science Foundation of China (Nos. 62376263 and 62372443), Guangdong Basic and Applied Basic Research Foundation (Nos. 2023B15151 Natural Science Foundation of Guangdong (Nos. 2024 and 2024A1 1970), and Shenzhen Science and Technology Innovation Commission (Nos. JCYI200230807144000015 and 10002000222211000000000000000000
References
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlci Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive learning. arXiv preprint Xiiv:2003.04297, 2020b.
Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A. Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, and Fabio Roli. Wild patterns reloaded: A survey of machine learning security against training data poisoning. ACM Comput. Surv. 55(13s), jul 2023. ISSN 0360-0300. doi: 10.1 155.1455/88885. URL htt.hhttt::/ddee http:///i..og////////// wwtp:/......11151111111111
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
Jia Deng, Wei Dong, Richard Socher, and et al. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pultern recognition, pp. 248-255. IEFE, 2009.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout, arXiv preprint arXiv:1708.0 04552, 2017.
Hadi M Dolatabadi, Sarah Erfani, and Christopher Leckie. The devil's advocate: Shattering the illusion of unexploitable data using diffusion models. arXiv preprint xxxv:2303.08500, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Ji Feng, Qi-Zhi Cai, and Zhi-Hua Zhou. Learning to confuse: generating training time adversarial data with encoder.
Advances in Neural Information Processing Systems, 32, 2019.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojeiech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. Advances in Neural Information Processing Systems, 34:30339-00351, 2021.
Shaopeng Fu, Fengxiang He, Yang Liu, Shen, and Dacheng Tao. Robust unlearnable examples: Protecting data against adversarial learning. In International Conference on Learning Representations, 2022.
Jonas Geiping, Liam H Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In International Conference on Learning Representations, 2021. URL http::://opepeeeeeteeee ew. net ftttt/oorrmmiddlllllll1
Micah Goldblum, Dimitris Tsipras, Chulin Xie, and al. Dataset security for machine learning: Data poisoning backdoor attacks, and defenses. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2):1563-1580,
2022.
Dou Goodman, Hao Xin, Wang Yang. Wu Yuesheng. Xiong Junfeng, and Zhang Huan. Advbox: a toolbox to generate adversarial examples that fool neural networks. arXiv preprint h0l::20011005744 2020.
Hao He, Kaiwen Zha, and Dina Katabi. Indiscriminate poisoning attacks on unsupervised contrastive learning. In International Conference on Learning Representations, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Hanxun Huang, Xingjun Ma. Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlcarnable examples: Making personal data unexploitable. In International Conference Learning Representations, 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kerne:: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF conference computer vision and pattemn recognition, pp. 81 10-1811 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553) 444, 2015.
Yiming Li, Mengxi Ya, Yang Bai, Yong Jiang, and Shu-Tao Xia. BackdoorBox: A Python toolbox for backdoor learning. In International Conference on Learning Representations Workshop, 2023.
Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Image shorteut squeezing: Countering perturbative availability poisons with compression. In International conference on machine learning, 2023.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint wr.iiv:706..6083. 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Ng, et al. Reading digits in natural images with unsupervised feature learning. 201 22):44 2011.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In International conference on machine learning, 2022.
Ren Pang, Zheng Zhang, Xiangshan Gao, and et al. Trojanzoo: Everything you ever wanted know about neural backdoors (but were afraid ask). arXiv preprint, 2020.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Tianrui Qin, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Cheng-Zhong Xu. Learning the unlearnable: Adversarial augmentations suppress unlearnable example attacks. arXiv preprint arXiv:2303.15127. 2023.
Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. Foolbox native: Fast adversarial attacks to benchmark the robustness machine learning models pytorch, tensorflow, and jax. Journal of Open Source Software, 5(53):2607, 2020.
Jie Ren, Han Xu, Yuxuan Wan, Xingjun Ma, Lichao Sun, and Jiliang Tang. Transferable unlearnable examples. International Conference on Learning Representations, 2023.
Olga Russakovsky, Jia Deng, Hao Su, and et al. ImageNet large scale visual recognition challenge. International journal of computer vision, 115:211-252 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chigg Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Pedro Sandoval-Seen -Sguura, Vasu Singla, Liam Fowl, Jonas Geiping, Micah Goldblum, David Jacobs, and Tom Goldstein. Poisons that are learned faster are more effective. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 198-205, 2022a.
Pedro Sandoval-Segura, Vasu Singla, Jonas Geiping, Micah Goldblum, Tom Goldstein, and David Jacobs. Autore- gressive perturbations for data poisoning. Advunces in Neural Information Processing Systems, 35:27374-27386,
2022b.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John Dickerson, and Tom Goldstein. Just how toxic is data poisoning? a unified benchmark for backdoor and data poisoning attacks. In International Conference on Machine Learning, pp. 9389-9398. PMLR, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, and al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pp. 1188626,220 2017.
Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Preventing delusive adversaries with adversarial training. Advances in Neural Information Processing Systems, 34:16229916225,, 2021.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. Proceedings of the JEEE/CVF international conference on computer vision, pp. 32 42, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9
(11), 2008
Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning, Advances in Neural Information Processing Systems, 35: 2022.
Shutong Wu, Sizhe Chen, Cihang Xic, and Xiaolin Huang. One-pixel shortcut: on the learning preference of deep neural networks. In International Conference on Learning Representations, 2023.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Availability attacks create shortcuts. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 2367-2376, 2022.
Chia-Hung Yuan and Shan-Hung Wu. Neural tangent generalization attacks. In International Conference Machine Learning, pp. 12230-12240, PMLR, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, and et al. CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. MixUp: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
15

---

Published in Transactions on Machine Learning Research (08/2024)
A Experimental Settings
A.1
Datasets
Table 12 summarizes the specifications of datasets and the respective test accuracies on ResNet-18 architectures under
standard training settings.
Table 12: Dataset specifications and the respective accuracies on ResNet-18.
Datasets
#Classes
Training
Test Size
Image Dimensions
Clean Accuracy (%)
CIFAR-10 KKrzzevskky
2009
50,000/ 10,000
32x32x3
94.32
CIFAR-100 (Krizhevsky et
2009
100
50,000/ 10,000
32x32x3
75.36
SVHN (Netzer et al. 2011)
73,257 26,032
32x32x3
96.03
InageNet-subsel (Deng et al.
2009
100
20,000 4,000
224x224x3
64.18
Table 13
presents the default hyperparameters for all availability poisoning attacks implemented APBench.
Table 13: Default hyperparameter settings of attack methods.
Methods
Hyperparameter
Settings
Perturbation
8/255
DC
Pre-trained model
Official pretrained
Perturhation
fx 8/255
NTGA
Poisoned dataset
Official pretrained CIFAR- 10 CNN (best)
Perturbation
8 8/255
Perturbation lype Stopping error rate Learning rate
Sample-wise
0.01
EM
0.1
Batch size
128
Optimizer
SGD
Perturbation Step size
fx 8/255
HYPO
fox 155/25
TAP
Perturbation
& 8/255
Perturbation
& 8/255
Perturbation type
Samplc-wisc
Stopping error rale Learning rate
REM
0.1
Batch size
Optimizer
SGD
Adversarial training perturbation
Ex 4/255
Perturbation
1.30 (Project from &a 6/255)
LSP
Patch size
for CIFAR- 10/100 and SVHN; ImageNet
Perturbation
i 1.00
AR
Default hyperparameters
GitHub: puadooaallgguuuuuuuurrsseeooonnn
Perturbation
OPS
Perturbation type
Sample-wise
Default hyperparameters
GitHub: sychomaticcOOnn--ieeeShttoo
Perturbation
Ex 8/255
UCL
Poisoned dataset
Gutl kuiven/haloontrasiiiepppuonnn
Perturhation
f.e. 8/255
TUE
Poisoned dataset
GitHub: renjie3/TUE
B
Additional Results
Results with statistical errors Tables 16]and extends upon Table showing the detailed test accuracies of models trained poisoned CIFAR-10 datasets, including statistical errors of independent for each experiment.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Table 14: Default training hyperparameter settings.
Datascts
Hyperparameter
Settings
Optimizer
SGD
Momentum
0.9
Weight-decay
0.0005
Batch size
128
CIFAR-10-1000
Standard Augmentations
Random crop, random horizontal flp
Initial learning
0.1
Optimizer
SGD
Momentum
0.9
Weight-docay
0.0005
Batch size
128
SVHN
Standard augmentations
Nome
Initial learning rate
0.1
Optimizer
SGD
Momentum
Weight-decay
0.0005
Batch size
256
ImageNet-100
Standard augmentations
Random crop, horizontal flip, and color jitter
Initial learning rale
0.1
Table 15: Default hyperparameter settings of defenses.
Methods
Hyperparameter
Settings
CIFAR-10/100: 50
Standard
Training epochs
SVHN: 40
ImageNet-1000; 100
Perturhation
for 8/255
Steps size
Ex 2/255
Adversarial training (Madry al.
2017
PGD steps
10
Training epochs
200
CutOut DDeriss Taylor
2017
CuiMix
Training epochs
200
MixUp Zhang ct
2018)
Kernel size
Gaussian Liu ct al.
2023
Standard deviation
0.1
Training epochs
200
Quality
JPEG
Liu al.
2023
Training epochs
200
Number of bits
BDR Liu et
2023
Training epochs
200
PlasmaBrightness PlasmaContrast
p= 0.5
UEraser-Lite Qin et al.
2023
ChannelShuflle Training epochs
p 0.5
200
PlasmaBrighttess PlasmaContrast ChannelShuffl
UFraser-Max Qn et al.
2023
p=
Number of Repeats K
Training epochs
300
Diffusion sampler
Score-SDE
Starting step / Total diffusion steps
60 1000
AVATAR Dolatabadi et
2023
Pre-trained model
GitHub: monggsgoo/screessde_po
Training epochs
200
Partial poisoning In addition to the discussion on partial poisoning in Section accuracies under defenses in Figure
we provide the results in terms of
17

---

Published in Transactions on Machine Learning Research (08/2024)
Table 16: Detailed test accuracies (%) of models trained on poisoned CIFAR-I0 datasets. This table an extension of Table 3 and further includes an error range of separate runs for each experiment. The model trained on clean CIFAR-10 10 dataset attains an accuracy of 94.32%. "ST" Standard training. "CO" CutOut, "CM" CutMix, "MU"-
MixUp, and "Gauss." Gaussian.
Method
ST
CO
CM
MU
Gauss.
DC
15.19
19.94
17.91
25.07=2.74
16. 1077
EM
20.78
18.79+2.66
22.28
31.14+3.51
14.71 +0.66
REM
17.47
21.9611.72
26.22.12.6
43.07
21.80.10.4
HYPO
1..38+1.79
69. 1.33
67.12+3.27
74.25=2.60
62. 1.:00
NTGA
22.76
13.78
12.91
20.59:22.41
19.95
TAP
6.27
9.88+
14.21+2.14
15.46=1.77
7.88
LSP
13.06+0.4 14,96+1.02
17.69+1.37
18.77
+0.82
AR
11.7440.7 10.95
12.60+1.02
14.15
13.83=+1700
OPS
14.69
52.98
64.72+2.70
49.27
13.38
Table 17: Table 16 continued. "BDR" Bit-Depth Reduction, "GS" Grayscale, "JPEG" JPEG Compression,
"U-Max" UEra "AVA" "AVA--A AVATAR, and "AT" adversarial training.
Method
BDR
GS
JPEG
U-Max
AVA
DC
67.73=2.77
85.55
83.57+261
97
82.10+2.00
76.85+1179
EM
2..9
92.03
80.72.1.22
93.61 10.48
75.62.1216
1..51
REM
58.60 38.60=2.35
92.27=0029
85.4442.10
92.43
82.42+1147
77.46+1.66
HYPO
1.
63.35 1.088
85.21
88, 18.74
85.94
81.4942.37
NTGA
59.32=1.96
70,41 22667
68.72+3.37
16.78=1164
86.22++2.07
69. 69.70+2.66
TAP
7.475
11.01 11:6.77
84.08
79.05
87.75+12.52
79.92+1.87
LSP
53.86_2.40
64.70.1.229
80.14
92.83
76.90#
11.38:11.92
AR
36.14=2.04 35.17
84.75++227
90. 12:
88.60+:2.33
81.15++194
OPS
37.32=187
19.88
78.484094
77.99
66.16+2.13
14.95+0.67
No defonse
Crayscale
JPEG
UErascr-Max
90
90
85
8.5
85
80
80
80
75
(a) EM.
(b) REM.
(c) LSP.
(d) AR.
Figure 5: The efficacy in test accuracies (%, vertical axes) of defenses (No defense, Grayscale, JPEG, and UEraser-Max) against different partial poisoning attacks including EM (a). REM (b). LSP and AR (d) with poisoning ratios
(horizontal axes) ranging from 20% to 80%.
B.1 Visualizations
Grad-CAM and Shapley visualizations Gradient- Gradient-weighted class activation mapping (Grad-CAM) (Selvaraju et al. 2017) and Shapley value map (Lundberg & Lee| (017) are commonly used image analysis tools that visualize the contributions of different pixels in an image the model's 's predictions. From the Shapley value map and Grad- Grad-CAM visualizations (Figure 6), observe discernible changes in activation features in the poisoned model relative
18

---

Published in Transactions on Machine Learning Research (08/2024)
to the clean model. AVATAR showed activation features most similar to the clean model compared other defense mechanisms, as it aims to restore the original features while disrupting the APA perturbations. In contrast, models applying the other defense strategies typically have different activation features than clean models. This discrepancy implies that image preprocessing or augmentations may modify the inherent feature extraction from the original images instead of restoring them.
Grad-CAM
Shapley value map
EM
REM
LSP
AR
Clean
No-defense
ISS
U-max
AVATAR
Clean
No-defense
ISS
U-max
AVATAR
Figure 6: Grad-CAM and Shapley value map visualizations of regions contributed to model decision under different attack methods and defense methods with ResNet- 18. (Left) Grad-CAM visualizations of EM and LSP attacks. (Right) Shapley value map visualizations of REM and AR attacks.
T-SNE visualizations Figure 8 shows the t-SNE visualization Figure of the models' feature representations on the clean test for CIFAR-10. Notably, models without defenses struggle create coherent class clusters, although there exist spatial variations in class frequency. contrast, models equipped with defenses display a feature distribution akin to the clean baseline. Models with higher clean accuracies also often exhibit better separated clusters.
No defense
ISS
U Urraser-max
AVATAR
ТАР
OPS
Figure 7:
The t-SNE visualization of the models' feature representations on the clean test set. Note that without defenses, the feature representations of the poisoned models are mostly scrambled as the models struggle to learn useful features.
C Licenses
APBench is open source, and the source code will be made available upon publication. Table 18 provides the licenses of the derived implementations of the original algorithms and datasets.
19

---

Published in Transactions on Machine Learning Research (08/2024)
EM/No defense
EM/ISS
EM/UEraser
EM/AVATAR
REM/No defense
REM/ISS
REM/UErase
VATAR
Clean
LSP/No defense
LSP/ISS
LSP/UE
LSP/AVATAR
R/No defense
AR/ISS
AR/UEraser
AR/AVATAR
Figure 8: The t-SNE visualization of the models' feature representations on the clean test set under additional attacks for CIFAR-10. 10. "UEraser" denotes "UEraser-axx".
20

---

Published in Transactions on Machine Learning Research (08/2024)
Table 18: Licenses of the datasets and codebases used in this paper.
Name
License
URL
PyTorch
BSD
GitHub: pytorch/pytorch
Attacks
GitHub: kingfongji/DeeppContui
NTGA
Apache-2.0
GitHub: lionclmessi64 (intga
EM
MIT
Gub:: Taanxnl I/U Unlearaable- Exampees
HYPO
MIT
Gitllub: LLMMchaee/Delusive-Ateee
TAP
MIT
GitHub: Iatowlladversarial poisons]
REM
MIT
GitHub: http:////mmuu----aaaaaeeeec
LSP
GitHub:
Shortetss
AR
MIT
GitHub: posntooweeeuuruuuueeessssssseeuu
OPS
Apache-2.0
Gitl: Cww.oomatiiccc/mm----eesststut
UCL
MIT
GitHub: kaiienzhuuconrraattv-poooo
TUE
GitHub: renjic3/TUE
Defenses
GitHub: huzree/lmageShorruuuuuzzzzng
DifiPure
NVIDIA
GitHub: NVlabs/DiftPure
Datasets
CIFAR-10
https:
CIFAR-100
SVIIN
hcco://rldd..ttafoordd. ott
ImageNet-100
Gilluub TengLowel///mggeeeeeelllaaasss
21


Reviews:
[{"review_id": "oIIW0FCM4l", "reviewer": "Reviewer_2T9D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Standardization and Reproducibility: APBench standardizes the experimental setup, ensuring fair and reproducible evaluations. This addresses the issue of varying experimental setups in previous research, providing a consistent platform for comparison.\n- Wide Range of Attacks and Defenses: The benchmark covers 9 availability poisoning attacks, 8 defense algorithms, and 4 conventional data augmentation techniques. This extensive coverage provides a broad perspective on the effectiveness of different methods.\n- Focus on Privacy: The research highlights the inadequacy of existing attacks in safeguarding individual privacy, emphasizing the importance of developing more effective methods. This focus on privacy protection aligns with the growing concerns regarding data security.\n\nWeaknesses:\n- Limitations of datasets: The datasets used in the paper are mainly traditional natural image datasets, such as CIFAR-10, CIFAR-100, SVHN, and a subset of ImageNet. Although these datasets are widely used in visual tasks, their diversity and complexity may not be sufficient to comprehensively evaluate various data poisoning attacks and defense methods. Increasing the use of mixed (natural + generated images) datasets may provide a more comprehensive evaluation.\n- It appears that the APA's effectiveness in privacy protection is gauged by the extent of reduction in model performance, akin to untargeted poisoning attacks that aim to degrade model performance by causing specific samples to be misclassified. Perhaps this paper can be compared with the relevant benchmark [1].\n\n[1]Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks abstract", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses benchmarks for availability poisoning attacks, which involve injecting perturbations into training samples to prevent unauthorized trainers from using these data to train models. Specifically, this paper presents APBench, an open-source benchmark that encompasses a range of poisoning attacks and defense strategies. The experiments reveal limitations in the current attack methods' effectiveness at safeguarding privacy. APBench is designed to foster equitable and reproducible evaluation, propel the advance of privacy-preserving availability poisoning attacks and defense mechanisms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness above.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BNOroLozck", "reviewer": "Reviewer_ZpQS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe topic is interesting and this paper provide a Benchmark, which might be useful for the researchers. \nThis paper also conducts many experiments, some of which are valuable. \n\n\n**Weaknesses:**\n\n1.This paper overclaims the contributions. This paper uses strong terms to describe the proposed benchmark, e.g., “this comprehensive suite allows us to evaluate the effectiveness of the poisoning attacks thoroughly”, “with a clear understanding of the current progress…”, “An open source benchmark for state-of-the-art availability poisoning attacks and defencses.”. However, I cannot find enough evidence to support these terms:  \n(1)\tWhy this paper uses these availability poisoning attacks and defenses methods? Why they are comprehensive for evaluation? By the way, how does the implementation (or the results) of the methods related to the original implementation (results) of the corresponding papers? Better or worse? Or not comparable? I believe this paper should provide these information to support its claims.   \n(2)\tThis paper claims “the experiments are conducted on 5-mainstream models , ResNet-18, ResNet-50, MobileNetV2 and DenseNet-121”, why not conduct experiments on ViTs here? Apparently, Vits have significant differences compared to these traditional CNNs. Besides, why this paper claims in Section 1“also extensively examined scenarios of partial poisoning, increased perturbations, the transferability of attacks to 4 CNN and 2 ViT models under various defenses, and unsupervised learning…”?   \n(3)\tIf this paper claims “with a clear understanding of the current progress…”, I would understand this paper will describe the related work like a survey, but it not. \n\n2.This paper should re-organize the description of the related work in page 2 (the last paragraph). In the current description, this paper only describes each method, but I believe this paper should well organize them with better logic. \n\n\n3.This paper should provide more clarification in its description. For example:  \n(1) in Section 2.1, this paper states that “The attacker’s goal is to thus render their data unlearnable with perturbation”, What is “their data unlearnable”? How to define “unlearnable”? Does this definition depends on tasks?   \n(2) In section 4, it states “To ensure a fair comparison between attack and defense methods, we used only the basic version of training for each model”, what does the meaning of “he basic version of training for each model”? why this can ensure “a fair comparison between attack and defense methods”. Please clarify it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper develops a benchmark--APBench for assessing the efficacy of adversarial poisoning, including Attacks and Defenses. APBench consists of 9 availability poisoning attacks, 8 defense algorithms, and 4 conventional data augmentation techniques. It conducts experiments with varying different poisoning ratios, and evaluates the attacks on multiple datasets and their transferability across model architectures. The results reveals the glaring inadequacy of existing attacks in safeguarding individual privacy.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tProvide more evidence to support the claims or reword the claims.   \n2.\tReword the related work with better presentation.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RgMhu7dAQb", "reviewer": "Reviewer_uKf3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n1. The paper is well-motivated, and the benchmark covers many existing attacks and defenses.\n2. The authors present an extensive amount of evaluation, and the evaluation demonstrates the authors’ own findings.\n3. The authors explore different challenge scenarios that can happen in real-world cases.\n\n### Weaknesses\n1. The writing of the paper can be improved further, making it more readable and organized. Please see **Requested Changes** for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is mainly about the authors’ development of a benchmark that evaluates the effectiveness of availability poisoning. Specifically, the authors focus on the availability poisoning attack (APA) that is used to improve data privacy against unauthorized training attempts and the defense against APA that tries to circumvent such privacy protection.\n\nThe authors formally describe the availability poisoning attack and the defense against it, with examples of existing attacks and defenses. They also point out the lack of benchmarks for availability poisoning. Then, they introduce the APBench benchmark, which can assess the performance of APA against existing defenses.\n\nThen, the authors present a set of experimental results evaluated from the APBench benchmark. They explore both standard scenarios (commonly assumed by the researchers) and challenging scenarios (that modify the common scenarios to reflect more realistic variations). From these experimental explorations, the authors demonstrate three non-trivial findings about adversarial poisoning attacks and defenses.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Evaluations section\n1. The Evaluations section should explain why the authors want to run experiments. I suggest writing a few paragraphs describing the experiments' goals. It is also better to present the **Key Takeaways** at the beginning of this section.\n2. Because the experiments involve many attack-defense combinations, the tables are difficult to interpret. To facilitate interpretation, highlight some key values (that demonstrate the authors' desired results) in boldface letters.\n3. In Tables 3 and 4, it is better to match the attack orderings to those of Table 1 and the defense orderings to those of Table 2. This makes it easier to look up the original papers and follow chronological order so readers can check whether or not recent attacks/defenses perform better. Also, if the vertical line in Tables 3 and 4 (between columns “MixUp” and “Gaussian”) is intended to follow the separation in Table 2, there should be another vertical line separating the Data preprocessing category and the Training-phase defense category.\n4. IIUC, the authors regard “Standard” data augmentation as one of the defense methods in Table 2. If so, the authors should present evaluation results for “No defense” (that does not do any augmentation/preprocessing) as the baseline in Tables 3 and 4.\n\n### Minor changes\n1. Overall, the paper writing is good, but there are small mistakes. I suggest the authors proofread the paper once again.\n   * In Section 2.3, the last sentence of the second paragraph should use in-text citations. For details about the citation styles, please read the [TMLR LaTeX stylefile and template](https://github.com/JmlrOrg/tmlr-style-file/archive/refs/heads/main.zip) in [the guidelines for authors](https://www.jmlr.org/tmlr/author-guide.html).\n   * On Page 6, “... is not usually considered a good one. mainly due to the fact…” should not have a period in the middle of the sentence.\n2. While I don’t think that there is a clear standard for writing the **Reproducibility Statement** and the **Ethics Statement**, they look too short to be separate Sections. One possible suggestion is to write a dedicated section for miscellaneous **Discussions**, containing subsections for **Future research direction** (in the last paragraph of Section 4), **Limitations** (Section D), **Reproducibility Statement** (Section 6), and **Ethics Statement** (Section 7).", "broader_impact_concerns": "I don’t see a particular concern about the broader impact of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers unanimously think the paper should be accepted after the modifications made by the authors (2 x Accept, 1 x Leaning Accept). Also, there has been a lengthy discussion with Reviewer ZpQS on the experiments.
Decision: Accept (poster)