Robust Unlearnable Examples: Protecting Data Privacy Against Adversarial Learning
Authors: Shaopeng Fu
Fengxiang He
Yang Liu
Li Shen
Dacheng Tao
Venue: ICLR 2022 Poster (2022)
Keywords: unlearnable examples
adversarial training
privacy

Abstract: The tremendous amount of accessible data in cyberspace face the risk of being unauthorized used for training deep learning models. To address this concern, methods are proposed to make data unlearnable for deep learning models by adding a type of error-minimizing noise. However, such conferred unlearnability is found fragile to adversarial training. In this paper, we design new methods to generate robust unlearnable examples that are protected from adversarial training. We first find that the vanilla error-minimizing noise, which suppresses the informative knowledge of data via minimizing the corresponding training loss, could not effectively minimize the adversarial training loss. This explains the vulnerability of error-minimizing noise in adversarial training. Based on the observation, robust error-minimizing noise is then introduced to reduce the adversarial training loss. Experiments show that the unlearnability brought by robust error-minimizing noise can effectively protect data from adversarial training in various scenarios. The code is available at \url{https://github.com/fshp971/robust-unlearnable-examples}.

Published as a conference paper at ICLR 2022
ROBUST UNLEARNABLE EXAMPLES: PROTECTING DATA AGAINST ADVERSARIAL LEARNING
Shaopeng Fu', Fengxiang He', Yang Liu", Li Shen' & Dacheng Tao JJD Explore Academy, JD.com Inc, China Institute for AI Industry Research, Tsinghua University, China (shaopengfu15, fengxiang f. h,e,, mathshenli, dacheng. tao @gmai com Suuy0e@air.tsime tsinghuu....nnnn edu.cn cn
ABSTRACT
The tremendous amount of accessible data in cyberspace face the risk of being unauthorized used for training deep learning models. To address this concern, methods are proposed to make data unlearnabee for deep learning models by adding type of error-minimizing noise. However, such conferred unlearnabil- ity is found fragile to adversarial training. In this paper, we design new methods to generate robust unlearnable examples that are protected from adversarial train- ing. We first find that the vanilla error-mingn mror-mmnimizzig noise, which suppresses the informative knowledge of data via minimizing the corresponding training loss, could not effectively minimize the adversarial training loss. This explains the vul- nerability of error-minimizae monommiiiiiing noise in adversarial training. Based on the obser- vation, robust error- prror-miniiziin noise is then introduced to reduce the adversarial training loss. Experiments show that the unlearnability brought by robust error- minimizing noise effectively protect data from adversarial training in vari- ous scenarios. The code available at ht / com/ fshp971/ robust mollearnable examllles
INTRODUCTION
Recent advances in deep learning largely rely on various large-scale datasets, which are mainly built upon public data collected from various online resources such as Flickr, Google Street View, and search engines (Lin et al. 2014 Torralba et al.] 2008 Netzer et al.) (011). However, the process of data collection might be unauthorized, leading to the risk of misusing personal data for training deep learning models (Zhang & Tao 2020 He & Tao 2020). For example, recent work demonstrates that individual information such as name or email could be leaked from pre-trained GPT-2 model (Carlini al. 2020). Another investigation reveals that company has been continuously collecting facial images from the internet for training commercial facial recognition models (Hill 2020).
To prevent the unauthorized perceptible noise such that the performances of models trained on the modified data be significantly downgraded (Fowl et al., 202 lb al Yuan & Wul 2021] Huang et al., [202I). This work focuses on that by Huang et al. (2021 Specifically, they introduce unlearnable examples that are crafted from clean data via adding type of imperceptible error-minimizing noise and show impressive protec- tion ability. The error-minee mrormmmnimizzng noise is designed based on the intuition that an example of higher training loss may contain more knowledge be learned. Thereby, the noise protects data from being learned via minimizing the corresponding loss to suppress the informative knowledge of data.
of the released data, recent studies suggest poisoning data with im-
However, such an unlearnability conferred by error-minimizing noise is found vulnerable to adver- sarial training (Huang et al. 2021 Tao et al.) (021), standard approach towards training robust deep learning models (Madry et al. 2018). adversarial training, a model trained on the unlearn- able data can achieve the same performance that trained on the clean data. Similar problems are also found in other poisoning-bar based data protection methods (Fowl et al. [021b] Yuan & Wu (021) based based data protection, and further raises the following question: Can we still make data unlearnable by adding imperceptible noise even under the adversarial training?
(see Fig. [].. These findings bring great challenges to poisoning

---

Published as a conference paper at ICLR 2022
CIFAR-10
CIFAR-100
ImageNet (Subsot)
60
60
Text Aocuracy (76)
Test Aoturate (N)
105 Nnturor/
60
10
30
Cken
TAP
TAP KIBAA REV [Oun)
TAP
NMAA 10M
NIBAA
Rodux (235)
Redixx (2266)
Radiu Raduu
Figure 1: We conduct adversarial training on data protected by different types of noise with varied adversarial perturbation radius Pa- EM denotes error-minimizing noise, TAP denotes targeted adver- sarial poisoning noise, NTGA denotes neural tangent generalization attack noise, and REM denotes robust error- miro-mminimizin noise. The curves of test accuracy vs. radius Pa are plotted. The results show that as the radius Pa increases, (1) the protection brought by EM, NTGA and TAP gradually become invalid, and (2) the proposed REM can still protect data against adversarial learners.
In this paper, we give an affirmative answer that imperceptible noise can indeed stop deep learning models from learning high-level knowledge in adversarial manner. We first analyze the reason why the unlearnability conferred by prror-mniimzzing noise fails under adversarial training. Ad- versarial training improves the robustness of models via training them on adversarial examples. However, adversarial examples usually correspond to higher loss, which according to Houangg et al.) (202I) may contain more knowledge to be learned. Though the error- minimiiing noise is designed to minimize the training loss, ad adversarial examples crafted from unlearnable data could still signif- Therefore, adversarial training can break the protection
icantly increase the loss scale (see Fig. 2 noise and still learn knowledge from these data.
brought by error-
Based on the aforementioned analysis, we further deduce that the invalidation of unlearnable exam- ples in adversarial training might be attributed to the error-minimizing noise could not effectively reduce the adversarial training loss. To this end, a new type of noise, named robust error-minimizing noise, is designed to protect data against adversarial learners via minimizing the adversarial train- ing loss. Inspired by the strategy of adversarial training and the generation of error- error-miniman minimizing noise, miin-min-maa optimization is designed to train the robust error- arror-mmnimizing noise generator. Examples crafted by adding robust error miroommiiiizzing noise are called robust unlearnable examples.
In summary, our work has three main contributions: (1) We present robust error- mrrommminiiizing noise, which to the best of our knowledge, is the first and currently the only method that can prevent data from being learned in adversarial training. (2) We propose solve min-min-max optimization problem to effectively generate robust error-minimizing noise. (3) We empirically verify the effec- tiveness of robust error- minimizini noise under adversarial training.
RELATED WORKS
Adversarial attacks. Adversarial examples carefully crafted from normal data and can fool ma-
chine learning models to behave unexpectedly. It has been found that machine learning models are
vulnerable to adversarial examples (Szegedy al. 20133] Goodfellow et al.) 2015 Carlini & Wag-
ner] |er||2017] 2017). Though minor data transformations can mitigate the adversaries of adversarial examples
(Lu al. 2017, [uo
Luo et al. 2015), some studies have shown that adversarial examples can further be
made resistant to these transformations (Evtimov et al. 2017 Eykholt et al. 2018 Wu et al. 2020b
Athalye et al. 2018 Liu et al. 2019 2020a). This makes adversarial attacks realistic threatens.
To tackle adversarial attacks, adversarial training is proposed to improve the robustness of machine learning models against adversarial examples. Similar to GANs (Goodfellow et al., [014]] Wang
et al. 2017 2019), standard adversarial training algorithm aims to solve a minimax problem that minimizes the loss function on most adversarial examples (Madry et al.) 2018). Other advances
include TRADE (Zhang et al. (2019), FAT (Zhang et al. 2020)) GAIRAT (Zhang et al. 2021)) (Song) et al. 2020), Jiang et al. (2021 ). Stutz et al. 220,), Singla la & Feizi 2020 Wu et al. (2020a). and Wu
et al. 2021). Tang et al. (2021) propose adversarial robustness benchmark regarding architecture
design and training techniques. Some works also attempt to interpret how machine learning models
gain robustness (Ilyas al. 2019 Zhang & 7hu 2019).
2

---

Published as a conference paper at ICLR 2022
Poisoning attacks. Poisoning attacks aim to manipulate the performance of machine learning model via injecting malicious poisoned examples into its training set (Biggio et al. 2012 Koh & Liang 2017; Shafahi et al.) 2018 Liu et al. 2020b; Jagielski al. 2018 Yang al. 2017 Steinhardt et al. 2017). Though poisoned examples usually appear different from the clear ones (Biggio al.j 2012, Yang al., 2017), however, recent approaches show that poisoned examples can also be crafted imperceptible to their corresponding origins (Koh & Liang| 2017 Shafahi et al, 2018). A special case of poisoning attacks is the backdoor attack. model trained on backdoored data may perform well on normal data but wrongly behave on data that contain trigger patterns
(Chen al.) 2017 Nguyen & Tran 2020 2021] Li et al., 202T Weng et al., 2020). Compared to
other poisoning attacks, backdoor attack is more covert and thus more threatening (Gu et al. 2017).
Recent works suggest employ data poisoning to prevent unauthorized model training. Huang et al. (221) modify data with error-minimizigg noise to stop deep models learning knowledge from the modified data. Yuan & Wu (202T) generate protective noise for data upon an ensemble of neural networks modeled with neural tangent kernels (Jacot al., 2018), which thus enjoys strong transfer- ability, Fowl et al. (2021a) a)) and and[owl et al. (202 lb) employ gradient alignment (Geiping et al. 2020) 2021) and PGD method (Madry al. (0118) to generate adversarial examples poisoned data to downgrade the performance of models trained on them. However, Tao et al. (2021) find that existing poisoning- -baedd protection be easily broken by adversarial training.
PRELIMINARIES
Suppose D = {(a11 v1),.. (Zin Mn)) is dataset consists of n samples, where I; E X is the feature of the i-th sample and y E y is the corresponding label. A parametcrized machine learning model is fe X' - V/ where 0 θ is the model parameter. Suppose & y x y + [0, 1] is loss function. Then, the empirical risk minimization (ERM) aims to approach the optimal model by solving the following optimization problem,
min n
((fo(x))))).
Adversarial training (Madry et al. 2018 improves the robustness of models against adversarial attacks via training them on adversarial examples. A standard adversarial training aims to solve the following min-max problem,
min max E(fo(z; 6.). 11..1S
(2)
where Pa > 0 is the adversarial perturbation radius and (xx + 5.) is the most adversarial example within the ball sphere centered at 2; with radius Pa. Usually, larger radius Pa implies stronger adversarial robustness of the trained model.
Unlearnable examples (Huang et al. 202T) type of data that deep learning models could not effectively learn informative knowledge from them. Models trained on unlearnable examples could only achieve severely low performance. The generation unlearnable examples contains two steps. Firstly, train an error-minimzing noise generator S via solving the following optimization problem,
min- 0
min E(S(a, + +.), 10:
where Pa is the defensive perturbation radius that forces the generated error-minimizing noise to be imperceptible. Then, an unlearnable example (x', y) crafted via adding error-minimizing noise generated by the trained noise generator fb to its clean counterpart (x,z), where z' x + arg min Sp.. f(ff + 6), y). The intuition behind this is, a smaller loss may imply less knowledge that could be learned from example. Thus, error-minimizing noise aims to make data unlearnable via reducing the corresponding training loss. It worth noting that once the unlearnable data is released publicly, the data defender could not modify the data any further.
Projected gradient descent (PGD) (Madry et al. 2018) is a standard approach for solving the inner maximization and minimization problems in Eqs. and (3). It performs iterative projection
3

---

Published as a conference paper at ICLR 2022
Acversaria Training, Pa 2/255
Adversarta Training, Pa 4/255
Cloan
Clean
2.0
2.0
2.0
Tainin Lote
Traning Lntes
1.6
1.6
1.5
50
0.6
0.0
0.0
0.0
10000
20000
30000 40203
10000
20000
30000
40303
10000
20000
30000
40200
Bwoston
Iwnston
Iacalon
Figure 2: The training loss curves of ERM training and adversarial training on CIFAR-IO. Lower training losses suggest stronger unlearnability of data. The results show that: (1) error- noise could not reduce the training loss as effectively as that in ERM training; (2) robust error- minimizing noise can preserve the training loss in significantly low levels across various learning scenarios. These observations suggest that robust error-minimziing noise more favorable in pre-
venting data from being learned via adversarial training.
updates to search for the optimal perturbation as follows,
0 8(h-1) 1) +c. a sign 00 "(fol (f 8(k-1)).
g(h) = II
where is the current iteration step (K steps all), 8(k) is the perturbation found in the k-th iteration, E 1-1,1] } is a factor for controlling the gradient direction, a is the step size, and I[ sp means the projection is calculated in the ball sphere {8 ||ô|| I p). The final output perturbation is g(K), Throughout this paper, the coefficient e is set as 1 when solving maximization problems and -1 when solving minimization problems.
4 ROBUST UNLEARNABLE EXAMPLES
In this section, first illustrate the difficulty of reducing adversarial training loss with error- minimizing noise, and then introduce robust unlearnable examples to tackle the raised challenge.
4.1 PRROR-MINIMLZN NOISE IN ADVERSARIAL TRAINING
Error-miniminn mirormmiiimizing noise prevents unauthorized model training via reducing the corresponding loss to suppress the learnable knowledge of data. However, the goal of reducing training loss may be at odds with that of adversarial training. Recall Eq. (2). adversarial examples usually correspond to higher training losses than that of the clean data, which according to Huang et al.) (202T) may contain more knowledge to be learned. Even if error-minimizing noise presents, crafting adversarial examples from unlearnable data could still significantly increase the training loss.
To better illustrate this phenomenon, conduct adversarial training clean data and unlearnable data with different adversarial perturbation radius Pa. The training losses along the training process collected and plotted in Fig. 2 From the figure, it can be found that mntommmmnimizing noise could not reduce the training loss as effectively as that ERM training. Furthermore, as the adversarial perturbation radius Pa increases, the adversarial training loss curves unlearnable data eventually coincide with that on clean data. These observations suggest that adversarial training can effectively recover the knowledge of data protected by arrormmnnmizzing noise, which makes the conferred unlearnability invalid.
4.2 ROBUST MIROR-MINIIIZING NOISE
Motivated by the empirical study in Section 4. we propose a type of imperceptible noise, named robust error-minimmn prroo-minimizing noise, to protect data against adversarial learners. Robust error-mnnmiizing noise is designed tuitively can barrier the knowledge learning process in adversarial training. Examples crafted via adding robust error-minimizing noise named robust unlearnable examples.
minimize the adversarial training loss during adversarial training, which in-
4

---

Published as a conference paper at ICLR 2022
Similar to that in Huang al. (2021), the generation of robust error-minimizing noise also needs to first train a noise generator fg. Inspired by the adversarial training loss in Eq. (2) and the objec-
tive function for training error-mininn minimiiiiizzng noise generator in Eq. 3 a min-min- min-max optimization process is designed to train the robust error-mininizing noise generator S& as follows,
1 min
max E(fol2y + 6 81). TI P 1
min
where the defensive perturbation radius Pe. forces the generated noise to be imperceptible, and the adversarial perturbation radius Pa controls the protection level of the noise against adversarial train- ing. The idea behind Eq. 44) is, aim find "safer" defensive perturbation 8t" such that the adversarial example crafted from the protected sample point (I; + 65"*) would not increase the train- ing loss much. As result, the adversarial learner could not extract much knowledge from the data protected by this safer defensive perturbation.
For the choices of the perturbation radii Pu and Pa. we notice that when Pu S Pas the following inequality holds for each summation term in Eq. 4
min max ( + 0 + +.). 2 ((f)(z;).
The above inequality suggests that when Pu. Pa. the generated defensive noise of could suppress any learnable knowledge of data even in ERM training. Thereby, to help the generated noise gain remarkable protection ability, the radius Pe should be set larger than Pa-
4.3 ENHANCING THE STABILITY OF THE NOISE
However, the unlearnability conferred by the noise generated via Eq. (4 is found to be fragile to mi- nor data transformation. For example, standard data augmentation (Shorten Khoshgoftaary 2019)
easily make the conferred unlearnability invalid (see Section 5.2). To this end, we adopt the expectation over transformation technique (EOT; Athalyy et al. 2018) into the generation of robust error-minimizing noise. EOT is stability-enhancing technique that was first proposed for adver- sarial examples (Evtimov et al., 2017] Eykholt et al. 2018; AAthayy et al., (018). Since adversarial examples and robust unlearnable examples are similar some extent, is likely that EOT can also help to improve the stability of robust arrormminmizinn noise.
Suppose T' a given distribution over set of some transformation functions (t X' + X). Then, the objective function for training robust error-minimizing noise generator with EOT is adapted from Eq. as follows,
min-
ELT max SP [1SLIS
min
E(b(((x; + 5 + 8,),
(5)
Eq. (5) suggests that, when searching for the defensive perturbation 8. one needs to minimize several adversarial losses of set of the transformed examples rather than only minimize the adver- sarial losses of single examples. After finishing training the noise generator via Eq. (5). the robust unlearnable example (a', y) for given data point (x.,) y) is generated as
L' x + arg min EL-T max
((S6((( + +
||
[|80|| Spc
4.4 EFFICIENTLY TRAINING THE NOISE GENERATOR
To train the robust error- arror-minmizzing noise generator with Eq. employ PGD to solve the inner minimization and maximization problems. The main challenge this approach is the gradient calculation of the inner maximization function.
Specifically, the inner maximization function in Eq. 5 usually does not have an analytical solution. As result, its gradient calculation could not be directly handled by modern Autograd systems such as PyTorch and TensorFlow. Toward this end, we approximate the gradient via first calculating the optimal perturbation 8t* arg max 69 Sp. ((t((t(z, + 6,²) + 8 y1) with PGD, and then B approximating the gradient of the maximization function by 08 (( + 81) + 68"). vs).
5

---

Published as a conference paper at ICLR 2022
Algorithm Training robust error-minimizing noise generator with Eq. (5)
Input: Training data set D, training iteration M,
1: PGD parameters Pu, ds and K, for solving the minimization problem,
2: PGD parameters Pa. Oa and K, for solving the maximization problem,
3: The data transformation distribution T,
4: the number of sampling time J when approximating the gradient of the expectation function.
Output: Robust arrormmnimiziig noise generator fg.
5: Initialize source model parameter 6.
6: for in 1.... MM do
7: Sample a minibatch (z, v) D.
8: Initialize 8".
9:
for in 1.... K2 do
10:
for j in 1, ... J do Sample transformation function 13 ~ 89 <- PGD((1, 6²), S Pa: Oa. Ka) end for
12:
Finding adversarial perturbation.
13:
14:
9k J E 8 II 1811SP. (6"h as sign(gs)
j==1 J8 (f + 80) + 89),
a
end for
17:
Sample a transformation function T.
18:
8" PGD((t(r d²)),,, y, Sy Pay Oay Ka)
19:
Update source model parameter & based on minibatch (f(x 60) + 00" 8),
end for
21:
return //
Therefore, when searching for the optimal defensive perturbation 6. with PGD, one can follow Athalye al. (2018) to approximate the gradient of the expectation of transformation via first sam- pling several transtormation functions and then averaging the gradients of the corresponding inner maximization function. The gradients of the inner maximization function are approximated via the aforementioned gradient approximation method. Finally, the overall procedures for solving Eq. (5) are presented as Algorithm[] The effect of the robust mirommmiiiizing noise generated based on Al- gorithm is illustrated in Fig, which verifies the ability of error-ming minimizing noise of suppressing training loss in adversarial training.
EXPERIMENTS
In this section, conduct comprehensive experiments to verify the effectiveness of the robust error-minimziing noise in preventing data being learned by adversarial learners.
5.1 EXPERIMENT SETUP
Three benchmark computer vision datasets, CIFAR-10, CIFAR-100 (Krizhevsky et al. 2009) and an ImageNet subset (consists of the first 100 classes) (Russakovsky al. (2015), are used in our experiments. The data augmentation technique (Shorten & Khoshgoftaar (019) is adopted in every experiment. For the detailed settings the data augmentation, please see Appendix A.2
Datasets.
Robust error-minimizing noise generation. Following Huang et al. 22021) we employ ResNet-18 18 (He al. 2016) as the source model I' for training robust error-minimizing noise generator with Eq. (5. The Lee-nnunded noises ||8, llo < Pu and || lsc Pa. are adopted in our experiments. The detailed training settings are presented in Appendix A.3.
Baseline methods. The proposed robust error-minimizing noise (REM) is compared with three -fate-of-the--rt data protection noises, the error- mroommmnimiziig noise (EM) (Huang et al.] 2021). the targeted adversarial poisoning noise (TAP) (Fowl et al. 2021b), and the neural tangent generaliza- tion attack noise (NTGA) (Yuan & Wu 2021). See Appendix A.3.2 for the detailed generation procedures of every type of noise.

---

Published as a conference paper at ICLR 2022
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(a) CIFAR-IO.
(b) ImageNet subset.
Figure 3: Visualization results of different types of defensive noise well the correspondingly crafted examples. EM denotes aroo-mmnimizing noise, TAP denotes targeted adversarial poisoning noise. NTGA denotes neural tangent generalization attack noise, and REM denotes robust error- minimizing noise.
Table 1: Test accuracy (%) of models trained on data protected by different defensive noises via adversarial training with different perturbation radii. The defensive perturbation radius is set as 8/ 8/255 for every type of noise, while the adversarial perturbation radius Pa of REM noise takes various values.
Dalasel
Adv. Train.
Clean
EM
TAP
NTGA
Pu
Pu
1/255
2/255
3/255
4/255
0
94.66
13.20
22.51
16.27
15.18
13.05
20.60
20.67
27.09
1/255
93.74
22.08
CIFAR-IH 10
2/255
92.37
71.43
90.53
85.13
75.42
29.78
25.41
27.29
30.69
3/255
90.90
87.71
4/255
89.51
88.62
88.02
88.96
85.15
86.34
75.14
47.51
48.16
0
76.27
1.60
13.75
3.22
1.89
3.72
3.03
8.31
10.14
1/255
71,90
71.47
70.03
65.74
9.45
4.47
5.68
9.86
11.99
2/255
3/255
66,45
65.66
64.30
64.80
66.27
44,93
27.29
17.55
17:74
4/255
64.50
63.45
62.39
62.44
64.17
61,70
61.88
41.43
27.10
80.66
1.26
9.10
8.42
2.54
4.52
6.20
8.88
13:74
ImageNet
1/255
76.20
74.88
75.14
63.28
12.80
13,42
14.92
Subset
2/255
72.52
71.74
70.56
66.96
49.58
33.14
27.06
23.76
29.40
3/255
4/255
66.62
63.40
63.56
63.06
64.80
59.32
51.78
41.52
41.66
conduct adversarial training (Madry al.) 2018 data protected by different defensive noises with different models, including VGG- (Simonyan & Zis- serman 2014), ResNet-18, ResNet-50 (He al., 2016), DenseNet-122 121 (Huang et al. (017)) and wide ResNet-34-10 (Zagoruyko & Komodakis (016).. Similar to that in training noise generator, also focus on L -bouded noise I|Pallso S Pa in adversarial training. Note that when Pa takes O, the adversarial training in Eq. (2) degenerates to the ERM training in Eq. (T). The detailed training settings are given in Appendix A.4
Model training. We follow Eq.
Metric. We use the test accuracy to assess the data protection ability of the defensive noise. A low test accuracy suggests that the model learned little knowledge from the training data, which thus implies strong protection ability of the noise.
5.2 EFFECTIVENESS OF ROBUST ERROR- NOISE
In this section, study the effectiveness of robust error-minimizing noise from different aspects. We first visualize different defensive noises and the correspondingly crafted examples in Fig. More visualization results can be found in Appendix B
Different adversarial training perturbation radius. We first add different defensive noises to the entire training set. The defensive perturbation radius p.. of every noise is set 8/255. Adversarial training is then conducted on both clean data and modified data with ResNet-18 models and different adversarial training perturbation radii Pa- Table I reports the accuracies of the trained models clean test data. We have also conducted experiments on noises that generated with larger defensive perturbation radius Pre 16/255, and the results are presented in Table 7 in Appendix C

---

Published as a conference paper at ICLR 2022
Table 2: Test accuracy (%) on CIFAR-IO and CIFAR-100 with different protection percentages. For EM, TAP, and NTGA noises, the perturbation radius Pu is set 8/255. For REM noise, the perturbation radii Pu and Pa are set as 8/255 and 4/255, respectively.
Duta Protection Perceninge
Adv.
Noise Type
Datasel
Train.
20%
40%
60%
80%
0%
100%
Pa
Mixed
Clean
Mixed
Clean
Mixed
Clean
Mixed
Clean
EM
92.26
91.94
91.81
91.14
71:43
2/255
TAP
92.37
92.17
91.30
91.62
90.31
91.32
88.65
83.37
90.53
NTGA
92.41
92.19
92.23
91.74
85.13
REM
92.36
90.22
88.45
30.69
CIFAR-I
EM
89.60
89.40
89.49
89.10
88.62
4/255
NTGA REM
TAP
89.51
89.01
89.56
88. 17
89.35
88,
86.76
88.40
89.22
85.07
88 04.04
89.17
79.41
88, 88,02
88. 96
89.60
89.34
89.61
EM
68.51
68.72
67.96
44
68.
2/255
TAP
68.91
68.05
66.54
67.83
67.75
67.27
47.99
66.91
NTGA
68 68.52
68. 82
68.36
68.71
66.53
REM
69.00
68.20
60.75
$2.33
14.15
CIFAR-100
EM
63.86
64.24
63.62
63.37
63.43
4/255
TAP
64.50
63.21
61.73
63.01
57.61
62.95
$3.86
62.90
44.79
62,39
NIGA
63.48
63.59
63.64
62.83
62.44
REM
63.75
63.99
64.36
63.05
27.10
As shown in Table D when the adversarial training perturbation radius increases, the test accuracy of the model trained on unlearnable data and targeted adversarial poisoned data rapidly increases. On the other hand, the robust mrrommmiiiizzig noise can always significantly reduces the test accuracy even when adversarial training presents. Furthermore, in the extreme case when the model trained on unlearnable data and targeted adversarial poisoned data achieve the same performance as that trained on clean data, the robust error-minimizing noise can still reduce the test accuracy of the model trained on robust unlcarnable data by around 20% 40%. These results demonstrate the effectiveness of the robust urror-minimizing noise against adversarial learners.
Different protection percentages. We then study more challenging as well as more realistic learning scenario, where only a part of the data are protected by the defensive noise, while the others are clean. Specifically, we randomly select part of the training data from the whole training set, adding defensive noise to the selected data, and conduct adversarial training with ResNet- 18 on the mixed data and the remaining clean data. The defensive perturbation radius for every noise is set as 8/ 8/255, while the adversarial perturbation radius Pa of the robust error-minimziing noise is set as 4/255. The accuracies on clean test data are reported in Table The difference between the test accuracies on mixed data and clean data reflects the knowledge gained from the protected training data. We have also conducted experiments with noises that generated with a larger defensive perturbation radius P. 16/ 26/255. The results reported in Table in Appendix C
Table 2 shows that when the adversarial training perturbation radius is small, robust error- minimizing noise can effectively protect the selected part of the data. However, when large adver- sarial training perturbation radius presents, the protection becomes worthless. This suggests that to protect data against adversarial training with a perturbation radius Pas one has set the defensive perturbation radius Pu of the robust error-minimizing noise to a value that is relatively larger than Pa- Table shows that the data protection percentage decreases, the performance of the trained model increases. This suggests that the model can learn more knowledge from more clean data, which coincides with intuition. Nevertheless. Table 2 further shows that the data protection ability of the robust error-minimizing noise is stronger than that of the error-minimizin noise and the tar- geted adversarial poisoning noise. This demonstrates that the robust grrimiiin noise still more favorable than other types of defensive noise when adversarial training presents.
Different model architectures. So far, we have only conduct adversarial training with ResNet- 18, which is as same the source model in the defensive noise generation. We now evaluate the effectiveness of the robust error- mroo-mmiimizzig noise under different adversarial learning models. Specifically, we conduct adversarial training with perturbation radius 4/255 and five different types of model. including VGG-16. 16, ResNet- 18, ResNet-50, DenseNet-121, and wide ResNet-34-1 on data that is protected by noise generated via ResNet-18. Table presents the test accuracies of the
8

---

Published as a conference paper at ICLR 2022
Table 3: Test accuracy (%) of different types of models on CIFAR-I0 and CIFAR-100 datasets. The adversarial training perturbation radius is set as 4/255. The defensive perturbation radius Pa of every type of defensive noise is set as 8/255.
REM
Datasel
Model
Clean
EM
TAP
NIGA
2/255
4/255
VGG- 16 RN-18
87.51
86.48
86.27
86.65
75.97
65.23
89.51
88.62
88.02
88.96
75.14
48.16
CIARR-10
89.28
40.65
DN. 121
83.27
82.44
81.72
80.73
77.82
81.48
91.21
90.05
90.23
73.98
VGG. 16
$7.14
56,94
55.24
55.81
56.50
48.85
RN-18
63.43
CIFAR 100
RN 50
66.93
66.43
64.44
64.91
61.30
26.03
53.73
53.52
WRN-34-10
68.64
68.27
65.80
67.41
64.11
25.04
trained models CIFAR-10 and CIFAR-100. We have also conducted experiments noises that generated with a larger defensive perturbation radius 16/ 16/255. See Table 9 in Appendix C
Table 3 shows the robust error-minimizing noise generated from ResNet-18 can effectively protect data against various adversarially trained models. However, when the DenseNet- 121 model presents, the robust error-minet mrrormmiiiiizing noise could not achieve the same protection performance as that in other scenarios. This may partially be attributed to the limitation of DenseNet- 121 itself in adversarial training. as could not achieve the same generalization ability as those other models on clean data. We will leave further studies on this phenomenon in future works. Nevertheless, the experiment results still demonstrate the effectiveness of robust error- error-mininge moi-mmziiiimggggee noise in most cases.
Ablation study on the EOT tech- nique. In Section 4.3) we pro- pose employ the EOT technique to enhance the stability of the ro- bust error-minimiiing noise in ad- versarial training. Here, we em- pirically investigate the effect of EOT. Specifically, we conduct ad- versarial training with ResNet-18 on datasets protected by robust er- ror noises that are generated with and without the EOT technique, re- spectively. The accuracies of the trained models on clean test data are reported in Table The results show that without EOT, the gener- ated noise gains no protection ability, which in turn justifies the necessity employing EOT during the robust emror- error- mroo-mmiimiziig noise generation.
Table 4: Ablation study on the EOT technique. Test accura- cies (%) of models trained on CIFAR-10 and CIFAR- 100 are reported. The defensive perturbation radius Pu of the REM noise is set 8/255. "EOT" and "None" denote that the noise is generated with and without the EOT technique, respectively.
Adv. Train.
REM
Datasct
Clean
Pa.
2/255
4/255
Pa
EOT
None
FOT
None
0/255
94,66
20.60
29.15
27.09
18.55
CIFAR-10
2/255
92.37
25.41
91.69
30.69
91.76
4/255
75.14
48.16
89.41
0/255
3,03
CIFAR-100
2/255
68.91
7.03
68.85
14.15
69.98
4/255
64.50
61.88
63.51
27.10
63.79
CONCLUSION AND FUTURE WORKS
This paper proposes robust error minimizing noise, the first defensive noise that can protect data from being learned by unauthorized adversarial training. We deduce that the prroo-miiimizing noise proposed by Huang et al. (2021) could not prevent unauthorized adversarial training mainly because can not suppress the training loss in adversarial training based on empirical study. Motivated by this, the robust error-minn pitoo-mmnimizing noise is introduced to effectively reduce the adversarial training loss and thus suppress the learnable knowledge of data in adversarial training. Inspired by the adversarial training loss and the vanilla error-minimizing noise generation, min-min-max problem is designed for training the robust error-minimizinn noise generator, where the expectation over transform tech- nique is adopted enhance the stability of the generated noise. An important future direction is to establish theoretical foundations for the effectiveness of robust-error minimizing noise. Another interesting direction is to design more efficient robust mirror-mnimizing noise generation methods.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work is supported by the Major Science and Technology Innovation 2030 "New Generation Artificial Intelligence" key project (No. 20212D0111700). Yang Liu is supported in part by the Tsinghua (AIR)-Asiainfo Technologies (China) Research Center under grant No. 20203910074. The authors sincerely appreciate the anonymous ICLR reviewers for their helpful comments.
REFERENCES
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International Conference on Machine Learning, pp. 284-293. PMLR, 2018.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector ma-
chines. In International Conference on Machine Learning. pp. 1807-1814, 2012.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXiv:2012.078005 2020.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:/712.05526, 2017.
Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. Robust physical-worid attacks on machine learning models. arXiv preprint arXiv:1707.0880 2017.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018
Liam Fowl, Ping-yeh Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use of proprietary data: Poisoning for secure dataset release. arXiv preprint (it.:2200302282. 2021a.
Liam H Fowl, Micah Goldblum, Ping yeh Chiang, Jonas Geiping, Wojeiech Czaja, and Tom Gold- stein. Adversarial examples make strong poisons. In Advances in Neural Information Processing Systems, 2021b.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients How easy is to break privacy in federated learning? In Advances in Neural Information Processing Systems, volume 33, pp. 16937-16947, 2020.
Jonas Geiping, Liam H Fowl, W. Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In International Conference Learning Representations, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.0673:: 2017.
Fengxiang He and Dacheng Tao. Recent advances in deep learning theory. arXiv preprint
109331 2020.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kashmir Hill. Meet Clearview AI, the secretive company that might end privacy as we know it. The New York Times, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang- Unlearn- able examples: Making personal data unexploitable. In International Conference on Learning Representations, 2021.
Andrew Ilas,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples not bugs, they are features. In Advances in Neural Information Processing Systems, volume 32, 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita Nita-Rotaru, and Bo Li. Ma- nipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP), pp. 19-35, IEEE, 2018.
Haoming Jiang, Zhehui Chen, Yuyang Shi, Bo Dai, and Tuo Zhao. Learning to defend by learning to attack. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pp. 577-585. PMLR, 2021.
Pang Wei Koh and Perey Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning, pp. 1885-1894, PMLR, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects context. European Conference Computer Vision, pp. 740-755. Springer, 2014.
Aishan Liu, Xianglong Liu, Jiaxin Fan, Yuqing Ma, Anlan Zhang, Huiyuan Xie, and Dacheng Tao. Perceptual-sensitive GAN for generating adversarial patches. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 33, pp. 1028-1035, 2019.
Aishan Liu, Jiakai Wang, Xianglong Liu, Bowen Cao, Chongzhi Zhang, and Hang Yu. Bias-based universal adversarial patch attack for automatic check-out. Proceedings of the European Con- ference Computer Vision, pp. 395-410, Springer, 2020a.
Sijia Liu, Songtao Lu, Xiangyi Chen, Yao Feng, Kaidi Xu, Abdullah Al-Dujaili, Mingyi Hong, and Una-May O' O'Reilly. Min-mat optimization without gradients: Convergence and applications to box evasion and poisoning attacks. In International Conference on Machine Learning, pp- 0282-6293. PMLR, 2020b.
Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. No need to worry about adversarial examples in object detection in autonomous vehicles. arXiv preprint arXiv: 17070030000 2017.
Yan Luo, Xavier Boix, Gemma Roig. Tomaso Poggio, and Qi Zhao. Foveation- mechanisms alleviate adversarial examples. arXiv preprint 10Xv::551.00622 2015.

---

Published as a conference paper at ICLR 2022
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In Advances in Neural 2020.
Information Processing Systems, volume 33, pp.
Tuan Anh Nguyen and Anh Tuan Tran. Waner Imperceptible warping-based backdoor attack. In International Conference Learning Representations, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255 2015.
Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! Targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018.
Connor Shorten and Taghi M Khoshgoftaar, survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Sahil Singla and Soheil Feizi. Second-order provable defenses against adversarial attacks. In Inter- national Conference Machine Learning, pp. 8981-8991. PMLR, 2020.
Chuanbiao Song, Kun He. Jiadong Lin, Liwei Wang, and John Hoperoft. Robust local features for improving the generalization of adversarial training, In International Conference on Learning Representations, 2020.
Jacob Steinhardt, Pang Wei Koh, and Percy Liang. Certified defenses data poisoning attacks. In Advances in Neural Information Processing Systems, pp. 3520-3532, 2017.
David Stutz, Matthias Hein, and Bernt Schiele. Confidence-calirrec adversarial training: Gen- eralizing to unseen attacks. In International Conference on Machine Learning, pp. 9155-9166. PMLR, 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Shiyu Tang, Ruihao Gong, Yan Wang, Aishan Liu, Jiakai Wang. Xinyun Chen, Fengwei Yu, Xiang- long Liu, Dawn Song, Alan Yuille, Philip H.S. Torr, and Dacheng Tao. RobustART: Benchmark- ing robustness architecture design and training techniques. arXiv preprint arXiv:2109.05221,, 2021.
Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Pre- venting delusive adversaries with adversarial training. In Advances in Neural Information Pro- cessing Systems, 2021.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 30(1 1911889998-1970 2008.
Chaoyue Wang, Chaohui Wang, Chang Xu, and Dacheng Tao. Tag disentangled generative ad- versarial network for object image re-rendering- Proceedings of the 26th International Joint Conference Artificial Intelligence (ICCAI), pp. 2901-2907, 2017.
Chaoyue Wang, Chang Xu, Xin Yao. and Dacheng Tao. Evolutionary generative adversar- ial networks. IEEE Transactions on Evolutionary Computation, 23(6):2119344 2019. doi: 10.1 1091199E..22211,2888999
12

---

Published as a conference paper at ICLR 2022
Cheng-Hsin Weng, Yan-Ting Lee, and Shan- (Brandon) Wu. On the trade-off between ad- versarial and backdoor robustness. In Advances in Neural Information Processing Systems, vol- ume 33, pp. 11973- 1983, 2020.
Boxi Wu, Heng Pan, Li Shen, Jindong Gu, Shuai Zhao, Zhifeng Li, Deng Cai, Xiaofei He, and Wei Liu. Attacking adversarial attacks as defense. arXiv preprint vrXiv:2106.04955 2021.
Kaiwen Wu, Allen Wang, and Yaoliang Yu. Stronger and faster Wasserstein adversarial attacks. In International Conference on Machine Learning, pp. 10377-10387. PMLR, 2020a.
Tong Wu, Liang Tong, and Yevgeniy Vorobeychik. Defending against physically realizable attacks image classification. In International Conference Learning Representations, 2020b.
Chaofei Yang, Qing Wu, Hai Li, and Yiran Chen. Generative poisoning attack method against neural networks. arXiv preprint arXiv: 770.003340 2017.
Chia-Hung Yuan and Shan-Hung Wu. Neural tangent generalization attacks. In International Con- ference on Machine Leurning, pp. 12230-12240. PMLR, 2021.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide
residual networks.
arXiv preprint
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.
Jing Zhang and Dacheng Tao. Empowering things with intelligence: survey of the progress, challenges, and opportunities in artificial intelligence of things. IEEE Internet of Things Journal, 8(10):7789-8877, 2020.
Jingfeng Zhang. Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. In International Conference on Machine Learning, pp. 11278-1122 1287. PMLR, 2020.
Jingfeng Zhang, Jiaming Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. International Conference on Learn- ing Representations, 2021.
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural net- works. In International Conference on Machine Learning, pp. 7502-7511. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2022
A EXPERIMENT DETAILS
This section provides the experiment details omitted from Section S
A.1 HARDWARE DETAILS
The experiments CIFAR- 10 and CIFAR-100 are conducted on 1 GPU (NVIDIA R Tesla V100 16GB) and 10 CPU cores (Intel Xeon Processor E5-2650 v4 @ 2.20GHz).
The experiments on ImageNet are conducted on 4 GPU (NVIDIA @ Tesla V100 16GB) and 40 CPU cores (Intel Xeon Processor E5-2650 v4 @ 2.20GHz).
A.2 DATA AUGMENTATION
We use different data augmentations for different datasets. For CIFAR-10 and CIFAR-100, we perform data augmentation via random flipping, padding 4 pixels on each side, random cropping to 32 x size, and rescaling per pixel to [-0.5, 0.5) for each image. For the ImageNet subset, we perform data augmentation via random cropping. resizing to 224 x 224 size, random flipping, and rescaling per pixel to [-0,5, 0.5] for each image.
A.3 DEFENSIVE NOISE GENERATION
Our experiments involve three types of defensive noise, the proposed robust error- monn-miiiiizinggg noise, and two baseline methods, error-minimizing noise and adversarial poisoning noise.
Table The settings of PGD (see Eq. (3) for the noise generations of error- crror-minimizing noise (EM), targeted adversarial poisoning noise (TAP), neural tangent generalization attack noise (NTGA), and robust error-minimizing noise (REM) different experiments. Pu. denotes the defensive perturba- tion radius of different types of noise, while Pa denotes the adversarial perturbation radius of the robust error- prrorminimizzng noise.
Datasets
Noise Type
Ka
Ka
FM
Pu/5
10
CIFAR-I0
TAP
Pu. 125
250
CIFAR-100
Pu/10 1.1
REM
Pa/5
10
0a/6
10
EM
Pn/5
ImageNet
TAP
Ps /50
100
Subset
NTGA
Pu/8
REM
P../
Pa/5
10
A.3.1 ROBUST ERROR MINIMIZING NOISE
Following Huang et al. 2021) we employ ResNet-18 (He et al. 2016) as the source model f' for the robust mrrormminiizzig noise generation. The -ounnde noises | lo. S Ps, and || S Pe are adopted in our experiments, in which the defensive perturbation radius Pa. and adversarial per- turbation radius Pa can take various values. The settings of PGD for solving the inner minimization and maximization problems in Eq. (5) are presented in Table
For CIFAR-10 10 and CIFAR-100, each source model is trained with SGD for 5, 000 iterations, with batch size of 1 128, momentum factor of 0.9, weight decay factor of 0.0005, an initial learning rate of 0. and learning rate scheduler that decay the learning rate by a factor of every 2, 000 iterations. For EOT, the data transformation T is set as the data augmentation of the corresponding dataset, and the repeatedly sampling number for expectation estimation is set as 5.
Besides, for the ImageNet subset, each source model is trained with SGD via Eq. (5) for 3, 000 iterations, with batch size of 128, a momentum factor 0.9, a weight decay factor of 0.0005, an initial learning rate of 0.1, and a learning rate scheduler that decay the learning rate by factor of 0.1 every 1, 200 iterations. For EOT, the data transformation corresponding dataset, and the repeatedly sampling number for expectation estimation is set 4.
set as the data augmentation the
14

---

Published as a conference paper at ICLR 2022
A.3.2 BASELINE METHODS IMPLEMENTATIONS
Two baseline methods are adopted in our experiments comparisons, including the error- minimizing noise method and the targeted adversarial poisoning noise method. Every method is reproduced on our own.
Brror-minimizing noise (Huang et al. 2021). We follow Eq. (3) to train the moiimmmiii noise generator. The mror-mmnimizing noise then generated with the trained noise generator. ResNet- 18 model is used as the source model f". PGD is employed for solving the inner minimization problem in Eq. (3), where the settings PGD is presented in Table 5 Other hyperparameters for training the noise generator are set the same as that for the robust error-minimizing noise generator in the previous section.
Targeted adversarial poisoning noise (Fowl et al. 2021b). This type of noise is generated via conducting targeted adversarial attack to the model that is trained on clean data, in which the gen- erated adversarial perturbation is used the adversarial poisoning noise. Specifically, given fixed model fo and a sample (x, y). the targeted adversarial attack will generate noise via solving the problem arg max {(fa(x 8u), g(U)), where is permutation function on the label space ).. PGD with differentiable data augmentation (Geiping et al. 2021) is employed for solving the above problem. The pareeeerrsmetesss for the PGD is given in Table
Neural tangent generalization attack noise Yuan & Wu, 2021). This type of protective noise aims to weaken the generalization ability of the model trained on the modified data. To do this, an ensemble of neural networks is modeled based on neural tangent kernel (NTK) (Jacot et al, 2018), and the NTGA noise is generated upon the ensemble model. As a result, the NTGA noise enjoys remarkable transferability. We the official code of NTGA D to generate this type of noise. Specifically, we employ the FNN model in Yuan & Wu (2021) as the ensemble model. For CIFAR-10 and CIFAR-100, the block size for approximating NTK is subset. the block size is set as 100. The byper-parameters for the PGD are given in Table 5 Other experiment settings follow Yuan & Wu (2021), please refer accordingly.
4, 000. For the ImageNet
A.3.3 TIME COSTS OF NOISE GENERATIONS
We calculate the time costs for training different types of noise generators different datasets. The results are reported in the following Table 6
Table 6: Time costs for training different defensive noise generators on different datasets.
Dataset
EM
TAP
NTGA
REM
CIEAR-
22.6h
CIFAR-100 100
0.4b
0.5h
5.2h
22.6h
ImageNNll Subeet
3.9h
5.2h
14.6h
51.2h
A.4 MODEL. TRAINING DETAILS
We follow Eq. to perform adversarial training (Madry et al.) 2018). Similar to that in training noise generator, we also focus on bouunded noise I| |o S Pa in adversarial training.
In every experiment, the model is trained with SGD for 40, 000 iterations, with a batch size of 128, a momentum factor of 0.9, a weight decay factor of 0.0005, an initial learning rate of 0.1, and learning rate scheduler that decays the learning rate by a factor of 0.1 every 16, 000 iterations. For CIFAR-10 10 and CIFAR-100, the steps number K, and the step size 0a in PGD are set as 10 and Pa/5. 5. For the ImageNet subset, the steps number K, and the step size Oa
8 and Pa/4.
'The official code of NTGA is available htttts:://iithhu.ccom/// messi 641 nt
15

---

Published as a conference paper at ICLR 2022
B NOISE VISUALIZATION
This section presents more visualization results different defensive noises.
B.1 CIFAR-10
Clean
Clean
FM
TAP
NTGA
REM
FM
TAP
NTGA
REM
(a) Defensive perturbation radius Pu 8/255.
(b) Defensive perturbation radius p... 16/255
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(c) Defensive perturbation radius
8/255.
(d) Defensive perturbation radius 16/255.
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(e) Defensive perturbation radius 8/255.
(I) Defensive perturbation radius Pu = 16/255.
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
Defensive perturbation radius 8/225..
(h) Defensive perturbation radius 16/255.
Figure Visualization results of CIFAR-10. Examples of data protected by mroorminnmizzig noise (EM), targeted adversarial poisoning noise (TAP), neural tangent generalization attack noise (NTGA), and robust error-minimizing noise (REM). When Pu. is set as 8/255 or 16/255, the adver- sarial perturbation radius Pa of REM is set 4/255 or 8/255.
16

---

Published as a conference paper at ICLR 2022
B.2 CIFAR-100
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(a) Defensive perturbation radius 8/255.
(b) Defensive perturbation radius 16/255.
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(c) Defensive perturbation radius 8/255.
(d) Defensive perturbation radius p. 16/255.
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(e) Defensive perturbation radius Pu 8/255.
(f) Defensive perturbation radius P.. 16/255.
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(g) Defensive perturbation radius 8/255.
(h) Defensive perturbation radius Ps. 16/255.
Figure 5: Visualization results CIFAR-100. Examples data protected by grror-mnimizing
noise (EM), targeted adversarial poisoning noise (TAP), neural tangent generalization attack noise (NTGA), and robust error-minimizing noise (REM). When Pu. is set as 8/255 or 66/25,, the adver- sarial perturbation radius Pa. of REM is 4/255 8/255.
17

---

Published as a conference paper at ICLR 2022
B.3 IMAGENET SUBSET
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(a) Defensive perturbation radius 8/255.
(b) Defensive perturbation radius 16/255.
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(c) Defensive perturbation radius 8/255.
(d) Defensive perturbation radius p. 16/255.
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(e) Defensive perturbation radius Pu 8/255.
(f) Defensive perturbation radius Pu. 16/255.
Clean
Clean
EM
TAP
NTGA
REM
EM
TAP
NTGA
REM
(g) Defensive perturbation radius 8/255.
(h) Defensive perturbation radius Ps. 16/255.
Figure 6: Visualization results of ImageNet subset. Examples data protected by error- minomiiinizzzng noise (EM), targeted adversarial poisoning noise (TAP), neural tangent generalization attack noise (NTGA), and robust error-minimizing noise (REM). When Pu. is set as 8/255 or 66/25,, the adver- sarial perturbation radius Pa. of REM is 4/255 8/255.
18

---

Published as a conference paper at ICLR 2022
C MORE EXPERIMENTS RESULTS
This section collects the additional experiment results for Section 5.2
Different adversarial training perturbation radius.
Table 7: Test accuracy (%) of models trained data that protected by different defensive noises via adversarial training with different perturbation radii. The defensive perturbation radius Pr. is set 16/255 for every type of noise, while the adversarial perturbation radius Pa. of REM noise takes various values.
REM
Dataset
Adv. Train.
Clean
TAP
NTGA
Pa
Pa
2/255
4/255
8/255
8/255
94.66
16.84
11.29
10.91
13.69
13.15
19.51
24.54
24.08
2/ 2/255
92.37
22.46
78.01
19.96
21.17
17.73
20.46
21.89
26.30
CIFAR-10
4/200
89.51
41,95
87.60
32.80
45.87
31.18
23,52
26.11
28.31
6/255
86.90
66.82
8/255
84.79
64.71
82.56
74.50
77.08
72.87
63.49
46.92
36.37
0
76.27
1.44
4.84
1.54
2.14
4.16
4.27
5.86
11.16
2/255
68,
5,21
64.59
5.21
6.68
5.04
4.83
7.86
13,42
CIRAR-100
4/ 4/255
64,50
35.65
61.48
18.43
28.27
9.80
6.87
14.46
6/ 6/255
60.86
56.73
57.66
46.30
47.08
35.25
30.16
14.41
17.67
8/255
80.66
1.10
3.96
3.42
2,90
3.40
3.86
5.90
9.72
ImageNet Subset
2/255
72.52
70.94
70.80
19.90
16.00
8.50
5.66
14.74
4/255
66.62
62.16
62.54
41.08
44.12
20.58
11.20
12.46
19.38 18.38
6/255
$8.80
55,46
44.32
23.14
8/255
53.12
46.44
46.90
42.78
47.96
44.54
28.20
21.66
26.64
Different data protection percentages.
Table 8: Test accuracy (%) of models trained on CIFAR- 10 and CIFAR-100 with different protection percentages. The defensive perturbation radius of every noise is set as Pa 16/255. The adversarial perturbation radius of robust error-minimin aroormiiiiizzin noise is set as 8 255.
Protection Percentage
Adv. Noise Type
Dataset
Train.
20%
40%
60%
0%
100%
mixed
clean
mixed
clean
mixed
clean
mixed
clean
86.53
41,95
4/255
TAP
89.51
88.80 88.80
89.63
88.17
88 64
88.70
86.76
87,50
87.11
85.07
87.83
83.32
79.41
87.60
32.80
REM
89.08
87.00
85.05
79.00
28.31
CIFAR
EM
84,20
84.37
83,70
82,91
64.71
TAP
84.24
83.58
83.42
83.09
82,56
8/255
84.79
83.25
81.54
79.29
73.21
REM
84.85
83.99
83.15
80.05
36.37
EM
64.27
63.226
63.86
61.45
35.65
4/255
TAP
64.50
63.00
61.73
62.69
57.61
62.65
53.86
60.99
44.79
61.48
NTGA
63.47
63.09
63.14
58.72
18.43
REM
63.01
$9.90
$4.52
47.27
14.46
CIFAR-100
EM
57.44
57.70
57.14
$7.27
56.66
8/255
$8.27
57.75
$5.00
57.15
50.72
$6.32
46.29
55.73
39.65
$5.30
NTGA
57.49
$6.35
$5.69
53.92
50.81
REM
$7.78
58.12
$7.69
$5.30
23.29
19

---

Published as a conference paper at ICLR 2022
Different model architectures.
Table Test accuracy (%) of different types of models trained on CIFAR-10 10 and CIFAR- 100. The defensive perturbation radius of every defensive noise is set as 16/255. The adversarial training perturbation radius set as 8/255.
REM
Model
Clean
NTGA
4/255
8/255
VGG-16
41.17
RN-18
84.79
64,71
82:56
74.50
63.49
36.37
DN-121
75.75
67.70
74.35
66.56
62.39
58.70
WRN. 34 10
86.36
65.42
84.19
76.42
64.91
37.51
VGG--1
45.67
45.56
43.42
38.83
48.44
22.54
RN-18
CIFAR-100
RN-50
60.15
58.34
56.67
51.88
54.68
21.19
DN-121
48.28
47.12
46,74
41.05
47.36
42.87
WRN-34-10
60.95
58.50
57.92
54.47
58.22
22.67
The evolution of the train and test accuracies along the adversarial training process. The curves of train and test accuracies data protected by different defensive noises drawn in Fig.
7
The figure shows that robust error- mror-mmnimizing noise can effectively protect data from being learned along the adversarial training process, while other existing approaches could not. This result further justifies the effectiveness of our proposed method.
CIFAR-10
CIFAR-100
ImageNet (Subeet)
100
1.0
133 NSUU (N1)
Tas Accuroo (h4)
NITIA HDM
40
80
40
20
TAP
8
TAP
20
10000
20000
30000
40000
10000
20000
30000
40003
10000
20000
30000
40000
(a) Evolution the train accuracy.
ImageNitt
(34
Test Nocuesyy (%1
002
60
60
60
NTGA
TXP
30
NTGA
20
TAP
NTGA
RBM (0un)
RBM (Dun)
HEM (Oun)
10000
20020
30000
40202
10000
20000
30000
40303
10000
20020
30000
40200
Benston
Banstin
Iecaton
(b) Evolution of the test accuracy.
Figure 7: The curves of train and test accuracies to training iteration on data protected by differ- ent defensive noises. The defensive perturbation radius Pu. for every noise is set as 8/255, while the adversarial perturbation radius Pa. for REM is set as 4/255. Besides, the adversarial training perturbation radius is set as 4/255 in every experiment.
20

---

Published as a conference paper at ICLR 2022
The selections of adversarial perturbation radius Pa in robust error-minimizing noise gener- ation. We conduct adversarial training against the robust error-minimizing noises generated with fixed defensive perturbation radius Pa and different adversarial perturbation radii Pa- The results are presented Fig. g which suggests that setting the adversarial perturbation radius to be half of the defensive perturbation radius would help the noise achieve consistent protection ability against adversarial training with different perturbation radii.
REM with 8/255 CIFAR-10
RFM with 8/255 on CIFAR-100
102
60
Pp- 22555
Tac
A 21255
R- 42255
41215
6235
A.156 6/25
Abvarsarial Traning Radks
Advesari Trakiing Rodia
Figure 8: We conduct adversarial training on data protected by different robust error- proo-mmnnmmizing noise (REM). The curves test accuracy vs. adversarial training radius are plotted. The defensive perturbation radius Pr. for every REM noise is set as 8/255.
D RESISTANCE TO LOW-PASS FILTERING
This section analyzes the resistance of different protective noises against low-pass filters. We first process each image from the training dataset (can be clean dataset or protected dataset) with three low-pass filters, mean filter, median filter, and Gaussian filter, respectively. For every filter, the shape of the window is set as 3 x 3. We then conduct adversarial training on the processed data with various adversarial training perturbation radii. The experiment results are presented in Table I0
From the table, we find that when the adversarial training perturbation is small, robust error- minimizing noise can effectively prevent data from being learned. However, when large adversarial training perturbation presents, less, compared with other types of noise, the protection brought by robust error-minimizing noise is stronger in most situations. These results demonstrate that robust error-minimizing more favorable in protecting data against adversarial learning.
robust error- mror-mmnimizzig noise becomes ineffective. Neverthe-
Table 10:
Test accuracy (%) of different types of models trained on CIFAR-10 10 and CIFAR-100 that are processed by different low-pass filters. The defensive perturbation radius Pu. of every defensive noise is set as 16/ 25/5 The adversarial training perturbation radius is set as 8/255.
Dataset
Filter
Clean
EM
TAP
NTGA
REM
Pa
2/255
84.25
34.87
82.53
40.26
28.60
Mcan
4/255
80.47
56.41
78.87
57.42
40.32
8/255
2/255
87.04
85.10
30.87
27.36
CIEAR-IU
Modian
4/255
83.87
49.33
82.31
48.50
34,06
8/255
78.34
74.02
76.66
67.63
62.14
2/255
86.78
29.71
83.44
41.85
28.70
Gaussian
4/255
35.13
8/255
77.30
73.84
75.98
70.06
68.19
2/255
52.42
$3.07
$1.30
26.49
13,89
Mean
4/255
50.89
50.61
50.35
35.46
24.52
8/255
47.73
46.98
46.62
42.03
44,96
2/255
CIFAR-100 Median
4/255
54.77
53.50
53.31
33.05
19.14
8/255
51.09
49.64
48.86
45.13
40.45
2/255
56.64
56.49
$5.19
29.05
13.74
Gaussian
1/255
53.44
53.62
37.39
22.92
8/255
49.75
49.61
48.59
44.55
47.03
21

---

Published as a conference paper at ICLR 2022
E RESISTANCE TO ADVERSARIAL TRAINING WITH DIFFERENT PERTURBATION NORMS
This section studies the resistance of different protective noises against adversarial training with various perturbation norms. Specifically, when perform adversarial training (i.e., Eq. (2) on the protected dataset, we the perturbation norm Eq. 2 as Lı-norm or L2-norm. Other training settings are as same as that presented in Appendix A.4 The experiment results are collected and
presented in Table [T] which shows that robust error- moor-mmnimizzig noise can effectively protect data from adversarial training in these situations.
Table 11: Test accuracy (%) of different types of models trained CIFAR-I0 and CIFAR-100. The adversarial training is conducted with different types of perturbation norms. The defensive perturba- tion radius Pu of every defensive noise is set 16 /255.. The adversarial training perturbation radius set as 8/255.
Adv. Train.
Norm Type
Clean
TAP
NTGA
REM
Pa
1000/255
93.61
18.49
53.74
46.78
29.07
Li-norm
3000/255
CIFAR-10
30
92.66
44.11
91.30
73.00
32.90
L2-norm
100/255
89.91
82.55
88.70
86.75
62.76
1000/255
72.32
72.03
65.13
47.55
14.49
Li-nor
3000/265
67.37
66.77
63.64
64.83
40.84
CIFAR-100
50/256
70.05
69.52
67.45
65.31
17.43
100/255
65.65
65.25
63.86
63.31
47.16
22


Reviews:
[{"review_id": "hi4j2jqEF9a", "reviewer": "Reviewer_XwVk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The paper is well motivated as being able to securely release the dataset to the public is an important problem that people care about\n- The authors proposed a sensible remedy (robust error minimization noise) for the weakness identified by the prior authors.\n- The authors are thorough with the experiments to illustrate the effectiveness of the proposed approach compared to the prior methods\n- The authors have provided sufficient details for the experiments to make me feel comfortable reproducing their results\n\nWeakness\n- The proposed method seems quite straightforward. While the method is effective, the results are not that surprising.\n\nQuestions\n- In table 1, what are the differences between REM with radius 0 and EM? It seems like the numbers between the two columns are quite different\n\n", "summary_of_the_paper": "Error minimization noise was first proposed so that when a dataset is released to the public, conventional empirical risk minimization cannot learn a good model from the perturbed dataset. However, error minimizing noise is only effective to the extent that adversarial training is not used. Adversarial training was identified as an effective method of overcoming the error minimization noise in prior work. In this paper, the authors proposed a variant of the error minimization noise, which continues to be robust even in the presence of adversarial training. Specifically, they generat the adversarial noise with a similar method as the original error minimization paper, but calculate the gradients with respect to the attacked input as opposed to the clean input. ", "summary_of_the_review": "While the novelty of the method is weaker, I do find the study of error minimizing noise to be important. The proposed method is sensible, and the experiments are quite thorough to convince me of the effectiveness of the proposed method. Overall, I like the submission, and would recommend weak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Yihb7baYjEg", "reviewer": "Reviewer_ntwj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The paper is clearly written and is easy to follow. \n\n2. The paper considers the adversarial learning setting, which might be useful in practice. The proposed min-min-max framework is reasonable in principle.\n\n3. Extensive experiments are shown to demonstrate the effectiveness of the generated robust error-minimization noise in terms of reducing test performance (but it doesn't show how data privacy can be protected as discussed in the weakness).\n\n## Weaknesses\n1. The paper majorly builds upon the previous work on unlearnable examples which only consider normal training. Although this work improves the reliability of the noise generation by replacing the original loss with adversarial loss and considering the expectation of data augmentation, the proposed method is quite straightforward and a bit trivial. Therefore, the novelty of the paper is a bit limited. \n\n2. The paper aims to protect data privacy but there is a lack of measurement of how privacy is protected. In all experiments, the paper focuses on showing how effectively the manipulated data degrades the test performance of the trained model. As far as I am concerned, this goal is the same as data poison attacks. However, it is unclear why and how the test performance relates to data privacy. From my understanding, the fact that a model performs badly on some samples doesn't necessarily imply the data privacy of those samples is protected. A thorough discussion on this issue will be necessary since protecting data privacy is the major goal of this work. Otherwise, it might be more appropriate to put it in the context of data poison attacks.\n\n3. The paper claims that the unlearnability conferred by the noise generated via Eq. (4) is fragile to minor data transformation and proposes to adopt the expectation-over transformation technique (EOT) into the generation process. However, the paper doesn't present evidence to support this claim. Moreover, it is expected to include more ablation studies about the effect of the EOT technique in both the normal setting and the adversarial setting. \n\n4. There is a lack of discussion of the data poison percentage that can effectively reduce the model performance.\n\n", "summary_of_the_paper": "The paper proposes a min-min-max formulation to generate robust unlearnable examples in order to protect data privacy in adversarial learning. The basic idea of this paper builds upon a previous paper \"Unlearnable examples: Making personal data unexploitable\" where the error-minimization noise is generated to reduce the training loss such that the model performs badly on the clean data. This paper finds that such unlearnability is fragile in adversarial training and fragile to minor data augmentations. Therefore, it proposes to generate more robust unlearnable samples by considering adversarial loss and data augmentation in the noise generation process, and it shows the generated noise can effectively reduce the quality of the trained model through extensive empirical study.\n\n", "summary_of_the_review": "Overall, the paper has shown that the proposed method can successfully reduce the test performance of the model (adversarially) trained by generating robust-error-minimization noise. However, the novelty is a bit limited and more justification of the data privacy protection is necessary. Therefore, I would like to recommend rejection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UmhA_9d9uHw", "reviewer": "Reviewer_c7i6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written, and the experiments are extensive. However, the proposed REM has the following issues:\n\n* Unrealistic assumptions. The authors assume that a \"data protector\" knows the hyperparameters (such as the loss function (l_{1}, l_{2}, or l_{inf}) and perturbation budget (rho_{a})) of adversarial training to be used by a data consumer, which is unlikely to hold in practice. Although the author used a large perturbation budget to evaluate REM, the budget may differ across different applications for many reasons (e.g., due to different degradation in clean performance introduced by adversarial training). Also, recent studies (e.g., (Tramèr and Boneh 2019)) have demonstrated that adversarial training with a single type of perturbation cannot provide well defense against other types of adversarial attacks, yet adversarial training with multiple perturbation types is still an active research problem (Maini, Wong, and Kolter 2020; Madaan, Shin, and Hwang 2020; Zhang, Zhang, and Wang 2021; Stutz, Hein, and Schiele 2020). Furthermore, there is a recent study (Yuan and Wu, \"Neural Tangent Generalization Attacks,\" 2021) that makes a dataset unlearnable using blackbox settings. The authors should 1) cite the work (Yuan et al. 2021) and discuss whether the problem targeted by this paper still exists for blackbox methods, and 2) discuss in more detail how REM perform against adversarial training with multiple perturbations, and 3) discuss how to choose the perturbation budget when running REM for different applications in practice.\n \n* Time complexity. The REM finds unlearnable examples under adversarial training by approximating a min-min-max problem, which can be very time-consuming and limits the practicability of REM. How long does take to make a full MNIST, CIFAR10, CIFAR100, and ImageNet unlearnable, respectively?\n\nFurthermore, exiting experiments lack some critical information and more should be conducted:\n\n* None of an experiment shows how training and test accuracy evolve as the training epoch increases. There’s no evidence to prove that \"the poor accuracy is not come from overfitting on the training data.\"\n\nTypos:\n1. Page 6 line 2: the loss has a typo  \n2. Table 7: \"AP\" should be \"TAP\"\n\nEdit after rebuttal:\n\nThe authors have partially addressed my concerns, so I am willing to raise my score provided that the authors agree to further improve the paper in the following aspects:\n1. Show the actual time cost of your approach and different baselines (including the new NTGA). Please describe the hardware (CPU/GPU) and settings you use in detail. \n2. Discuss in more detail how to set the epsilon in practical situations, especially for the black-box settings.  \n", "summary_of_the_paper": "Based on a previous work (Huang et al. 2021), this paper proposed a method, called REM, that generates makes a dataset \"unlearnable\" to adversarial training. Experiments are conducted to validate the effectiveness of REM under various settings, including (1) different adversarial training perturbation radiuses, (2) different proportions of unlearnable data, (3) different model architectures, on different common datasets, including CIFAR-10, CIFAR-100, and ImageNet.", "summary_of_the_review": "Overall, this paper studies an important and interesting problem. The paper is well-written and the experiments are extensive. However, the assumption used by the proposed REM is not realistic, and the authors missed an existing blackbox method that could entirely destroy the motivation of this work. Furthermore, some critical information, such as runtime, is missing from the current experiments. Therefore, I give a vote for rejection before the above concerns can be addressed.  \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OTiNVvNMMHk", "reviewer": "Reviewer_z2Tr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I lean towards recommending accepting this paper. \n\n---\n\nStrengths:\n- The proposed robust unlearnable example is technically sound and empirically demonstrated it can be effective when used against adversarial training. Experiments are comprehensive. \n- As suggested in the paper, none of the existing work can effectively be used to protect the data against adversarial training, this work demonstrated adversarial training could also be vulnerable to training-time perturbation/data poisoning. \n- Paper's presentation is easy to follow and most related works have been discussed.\n\n---\n\nQuestion/weakness: \n- How important is the EOT for the robust error-minimizing noise? An ablation study would be very helpful. Why is the REM on ImageNet can show the content of the image while not on CIFAR-10? Is this because of different data augmentations between CIFAR/ImageNet for EOT? \n- Is the proposed method only effective if the epsilon for the defensive perturbation is larger than epsilon for the adversarial training? For table 1, if adversarial training uses larger epsilon, at what point does the REM become ineffective?\n- What is extra overhead (time) compared to EM/TAP?\n- Need some clarifications for Algorithm 1. Line 10-13 finds the adversarial perturbation for J steps, and line 14-15 finds the unlearnable perturbation for a single step, is that right? For the adversarial perturbation, then the maximum change is 4/255, but for a single step of the unlearnable perturbation, the step size is $\\alpha_u$, wouldn't the adversarial perturbation always overwrite the unlearnable perturbation? ", "summary_of_the_paper": "This paper studies how to make data unlearnable towards adversarial training. In prior works, including dataset poisoning, no methods can prevent data from being learned in adversarial training. This paper proposed a robust error-minimizing noise to solve this task. ", "summary_of_the_review": "Existing works have demonstrated adversarial training (AT) can remove the effect of both data poisoning using targeted adversarial attacks and unlearnable examples. This paper demonstrated an effective method that can prevent the data from being used for AT. Despite the potential limitation on the perturbation radius difference between AT and the unlearnable, this work reveals a weakness of adversarial training. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: To address the problem of unauthorized use of data, methods are proposed to make data unlearnable for deep learning models by adding a type of error-minimizing noise. Based on th fact that the conferred unlearnability is found fragile to adversarial training, the authors design new methods to generate robust unlearnable examples that are protected from adversarial training. In addition, considering the vulnerability of error-minimizing noise in adversarial training, robust error-minimizing noise is then introduced to reduce the adversarial training loss.
The authors have tried to respond to reviewers' comments along with adding more experiments.
Overall, this manuscript finally gets three positive reviews and one negative review, where the possible vulnerability or robustness of error-minimizing noise against (simple) image processing operations was not verified.
In comparison with other manuscripts I'm handling that got consistent positive comments, this manuscript is still recommended to be accepted (poster) with a further study of robustness under simple image transformations in the final version.
Decision: Accept (Poster)