Unlearnable Examples: Making Personal Data Unexploitable
Authors: Hanxun Huang
Xingjun Ma
Sarah Monazam Erfani
James Bailey
Yisen Wang
Venue: ICLR 2021 Spotlight (2021)
Keywords: Unlearnable Examples
Data Protection
Adversarial Machine Learning

Abstract: The volume of "free" data on the internet has been key to the current success of deep learning. However, it also raises privacy concerns about the unauthorized exploitation of personal data for training commercial models. It is thus crucial to develop methods to prevent unauthorized data exploitation. This paper raises the question: can data be made unlearnable for deep learning models? We present a type of error-minimizing noise that can indeed make training examples unlearnable. Error-minimizing noise is intentionally generated to reduce the error of one or more of the training example(s) close to zero, which can trick the model into believing there is "nothing" to learn from these example(s). The noise is restricted to be imperceptible to human eyes, and thus does not affect normal data utility. We empirically verify the effectiveness of error-minimizing noise in both sample-wise and class-wise forms. We also demonstrate its flexibility under extensive experimental settings and practicability in a case study of face recognition. Our work establishes an important ﬁrst step towards making personal data unexploitable to deep learning models.

Published as a conference paper at ICLR 2021
UNLEARNABLE EXAMPLES: DATA UNEXPLOITABLE
MAKING PERSONAL
Hanxun Huang! Xingjun Ma² Sarah Monazam Erfani! James Bailey!
Yisen Wang
The University of Melhourne, VIC, Australia
Deakin University, Geelong, VIC, Australia
3 Key Lab. of Machine Perception (MoE), School of EECS, Peking University, Beijing, China
ABSTRACT
The volume of "free" data on the internet has been key to the current success of deep learning. However, it also raises privacy concerns about the unauthorized exploitation of personal data for training commercial models. It is thus crucial to develop methods to prevent unauthorized data exploitation. This paper raises the question: can data be made unlearnable for deep learning models? We present a type of mrorrmmiiiiizing noise that can indeed make training examples unlearnable. Error- Brror-minimizing noise is intentionally generated to reduce the error of one or more of the training example(s) close to zero, which can trick the model into believing there is "nothing" to learn from these example(s). The noise is restricted to be imperceptible to human eyes, and thus does not affect normal data utility. We em- pirically verify the effectiveness of error-minimizing noise in both sample-wise and class-wise forms. We also demonstrate its flexibility under extensive experimental settings and practicability in case study of face recognition. Our work establishes an important first step towards making personal data unexploitable to deep learning models. Code is available at https:/gttuub.om//Hannnnttt wtte///nnnnaaaaeeeeaaaaaee
INTRODUCTION
In recent years, deep learning has had groundbreaking successes in several fields, such as computer vision (He et al.] 2016) and natural language processing (Devlin et al.) 2018). This is partly attributed to the availability of large-scale datasets crawled freely from the Internet such as ImageNet (Rus) sakovsky et al. 2015) and ReCoRD (Zhang et al. 2018b). Whilst these datasets provide ployground for developing deep learning models. a concerning fact is that some datasets were collected without mutual consent (Prabhu & Birhane 20200). Personal data has also been unconsciously collected from the Internet and used for training commercial models (Hill 2020). This has raised public concerns about the "free" exploration of personal data for unauthorized or even illegal purposes.
In this paper, we address this concern by introducing unlearnable examples, which aims at making training examples unusable for Deep Neural Networks (DNNs). In other words, DNNs trained on unlcarnable examples will have a performance equivalent to random guessing on normal test examples. Compared with preserving an individual's 's privacy by obfuscating information from the dataset, what we aim to achieve here is different but more challenging. First, making an example unlearnable should not affect its quality for normal usage. For instance, an unlcarnable "selfie" photo should be free from obvious visual defects so it can be used as a social profile picture. Ideally, this can be achieved by using imperceptible noise. In our setting, the noise can only be added to training examples on a single occasion (when the data is uploaded to the internet) prior to model training. However, DNNs are known be robust to small noise either random (Fawzi et al. 2016) or adversarial (Szegedy et al. 2013; Goodfellow et al.) [014]] Ma et al. 2018). It is still not clear whether small. imperceptible noise can stop the training of gegh-performance DNNs.
The development of unlearnable examples should take full advantage of the unique characteristics, and more importantly, the weaknesses of DNNs. One well-studied characteristic of DNNs is that they tend to capture more of the high-frequency components of the data (Wang et al.) 2020a). Surprisingly, by exploiting this characteristic, we find that small random noise when applied in a class-wise
"Correponndence to: Xingjun Ma (daniel.ma ma @ www.aii.cc.u.aa... Yisen Wang (yisen. waag@pku.eduu.cn)

---

Published as a conference paper at ICLR 2021
manner to the training data can easily fool DNNs to overfit to such noise (shown in Section However, early stopping can effectively counteract this type of noise. DNNs are also known to be vulnerable to adversarial (or aroo-maximizing) noise, which are small perturbations crafted to
maximize the model's error at the test time (Szegedy et al. 2013; Goodfellow et al.) 2014). We find that error- maximmaaiiiiing noise cannot stop DNN learning when applied in sample- wise manner to the training examples. This motivates us to explore the opposite direction to grror-maximizing noise. Specifically, being penalized by the objective function during training, and thus can trick the model into believing there is "nothing" to learn from the example(s). We refer to example that contains the error- minimizing noise as an unlearnable example. Error-minimzzing noise can be generated in different forms: sample-wise and class wiie. Class-wise error- mrror-mmnmmizing noise is superior to random noise and cannot be circumvented by early stopping. Sample-wise error-mnimizzin effective noise that can make training examples unlearnable compared to random (Fawzi et al. (0166) or error- maximizing noise (Muñoz- González et al. 2017 Our main contributions are:
propose type of error-minimizing noise that can prevent the model from
noise is the only
We present type of crror-mnimmziing noise that can create unlearnable examples to prevent personal data from being freely exploited by deep learning models. The noise is small, imperceptible to human eyes, thus it does not reduce general data utility.
We propose a bi-level optimization process to effectively generate different forms of error- minimizing noise: sample-wise and class-wise.
We empirically verify the effectiveness and flexibility of error-minimizing noise for cre- ating unlearnable examples. We also demonstrate the practical application of unlearnable examples in real-world scenarios via a case study on face recognition.
2 RELATED WORK
In this section, we briefly review most relevant works in data privacy, data poisoning, adversarial attacks against deep learning models.
Data Privacy. Privacy issues have been extensively studied in the field of
machine
learning (Shokri & Shmatikov 2015 Abadi et al.) 2016 Phan et al.] 2016; 2017, Shokri et al.) 2017)) While these works have made significant progress towards protecting data privacy, they are developed based on the assumption that the model can freely explore the training data and turn to protect the model from leaking sensitive information about the training data. In this paper, we consider more challenging scenario where the goal of the defender is to make personal data completely unusable by unauthorized deep learning models. Fawkes (Shan et al. 2020) has made the first attempt towards this type of strict situation. By leveraging the targeted adversarial attack, Fawkes prevents unauthorized face tracker from tracking a person's 's identity. This work is similar to ours as we share common objective that prevents unauthorized data usage. In contrast to the targeted adversarial attack, we propose novel error-minimizing noise to produce unlearnable examples which can be used generic framework for a wide range of data protection tasks.
Data Poisoning. Data poisoning attacks aim degrade the model's performance clean examples by modifying the training examples, Previous work has demonstrated a poisoning attack on SVM (Biggio et al. 2012). Koh & Liang 2017) proposed to poison the most influential training examples using adversarial (error-maximizing noise against DNNs, which has also been integrated into an end- to-end framework Slauioz-Gonzálezeee et al., 2017). Although data poisoning attacks can potentially prevent free data exploitation, these approaches are quite limited against DNNs and hard to operate in real-world scenarios. For example, poisoned examples can only slightly decrease DNNs performance (Munoz-CGonzález et al., 2017), and often appear distinguishable to clean examples (Yang et al.) 2017) which will reduce normal data utility. The backdoor attack is another type of attack that poisons training data with a stealthy trigger pattern (Chen et al., 2017 [Lu et al. (020). However, the backdoor attack does not harm the model's performance
clean data (Chen et al. 2017]] SSafahi et al. 2018 Barni et al. 2019 Liu et al. (2020) (Zhao et al., 2020). Thus, it is not valid method for data protection. Different from these works, we generate unlearnable examples with invisible noise to "bypass" the training of DNNs.
Adversarial Attack. It has been found that adversarial examples (or attacks) can fool DNNs the test time (Szegedy et al. [213]Goodffel Goodfellow et al. 2014 Kurakin et al. (2066] [ Carlini & Wagner 2017
Madry et al. 2018 Jiang al. 2019 Wu et al. 2020a Bai al. 2020) Croce & Hein 2020 Wang
2

---

Published as a conference paper at ICLR 2021
et al. 2020b; Duan et al. 2020 Ma et al. 2020). The adversary finds an error- mror-maximizing noise that maximizes the model prediction error, and the noise can be crafted universally for the entire test set DMoosavi-Dezfooli et al., 2017). Adversarial training has been shown to be the most robust training strategy against error- maximizing noise (Madry et al. 2018: Zhang et al. [019 Wang et al., 2019; (u et al. [020b] Wang al. (220cc.. Adversarial traming can be formulated as min-max optimization problem. In this paper, we explore the opposite direction of error-mant mrrommmaiiizzing noise, i.c., finding small noise that minimizes the model's error via min-min optimization process.
3
UNLEARNABLE EXAMPLES AND
MINIMIZING NOISE
3.1 PROBLEM STATEMENT
Assumptions on Defender's Capability. We assume the defender has full access the portion of data which they want to make unlcarnable. However, the defender cannot interfere with the training process and does not have access the full training dataset. In other words, the defender can only transform their portion of data into unlearnable examples. Moreover, the defender cannot further modify their data once the unlearnable examples are created.
Objectives. We formulate the problem in the context of image classification with DNNs. Given typical K-class classification task, denote the clean training and test datasets as Dr and D respectively, and the classification DNN trained on De as So where 0 are the parameters of the network Our goal is transform the training data DD into unlearnable dataset D., such that DNNs trained on the Du, will perform poorly on the test set DD--
Suppose the clean training dataset consists of clean examples, that is, D. = {( ys )=));= with E X' IRd are the inputs and y y = {1...... }} are the labels and K is the total number of classes. We denote its unlearnable version by D., = {(x/, y:) /= where x' = x + 5 is the unlearnable version of training example æ E -De and 5 E A TR is the "invisible" noise that makes unlearnable. The noise ó is bounded by ||0]| S with I|p is the L,p norm, and is set to be small such that does not affect the normal utility of the example.
In the typical case, the DNN model will be trained on De to learn the mapping from the input space to trick the model into learning strong correlation between
the label space: f X + +y Our goal the noise and the labels: f: A + V, A & X, when trained on D.:
arg min Eg wD-, (((((()(()))))))))))(())))
(1)
where, L is the classification loss such as the commonly used cross entropy loss.
Noise Form. We propose two forms of noise: sample-wise and class-wise. For sample-wise noise, x. = x; + d, ò, E A, = --..... J, while for class-wise noise, x; = x; + 6y. 8y, E A. = (ō1. 5x )- Sample-wise noise needs to generate noise separately for each example. This may have more limited practicality. In contrast to sample-wise noise, a class of examples can be made unlearnable by the addition of class-wise noise, where all examples in the same class have the same noise added. As such, class-wise noise can be generated more efficiently and more flexibly in practical usage. However, will see that class wise noise may get more easily exposed.
3.2 GENERATING ERROR-MINIMIZING NOISE
Ideally, the noise should be generated on an additional dataset that is different from D.. This will involve class-matching process to find the most appropriate class from the additional dataset for each class to protect in De. For simplicity, here we define the noise generation process on D. and will verify the effectiveness of using an additional dataset in the experiments. Given clean example x, we propose to generate the error- prrommminiiziin noise 8 for training input æ by solving the following bi-level optimization problem:
arg min ((a [
min (( -),,))
s.t. || Sc
where, f' denotes the source model used for noise generation. Note that this is a min-min bi-level optimization problem: the inner minimization is a constrained optimization problem that finds the
We omit the 0 notation from fo without ambiguity in the rest of this paper.
3

---

Published as a conference paper at ICLR 2021
Ly -n-morm bounded noise ō that minimizes the model's classification loss, while the outer minimization problem finds the parameters that also minimize the model's 's classification loss.
Note that the above bi-level optimization has two components that optimize the same objective. In order to find effective noise and unlearnable examples, the optimization steps for should be limited, compared to standard adversarial training. Specifically, we optimize ô over D. after every M steps of optimization of 0. The entire bi-level optimization process is terminated once the error rate is lower than A. The detailed training pipeline is described in Algorithm I in the Appendix.
Sample-wise Generation. We adopt the first-order optimization method PGD (Madry et al. 2018 to solve the constrained inner minimization problem as follows:
x 41 11. (2 sign (=
where, is the current perturbation step (I' steps in total), V Vo y) is the gradient of the loss with respect to the input, 11 is projection function that clips the noise back to the e-ball around the original example æ when it goes beyond, and a is the step size. The perturbation is iteratively applied for I steps after each M step of model training as we explained earlier. The final output is unlearnable example x' and the generated error-minimizing noise is = x' xc.
Class-wise Generation. Class-wise noise A. can be obtained by cumulative perturbation on all examples in given class. For each example in class at step 4, it applies d% to the original example x and follows Equation 3 to produce ''41- The d. accumulates over every example for the corresponding class in the entire bi-level optimization process.
4 EXPERIMENTS
In this section, we first demonstrate the effectiveness in creating unlearnable examples using random noise, error-maximizing noise and our proposed arror-minimizing noise in both sample-wise and class-wise forms. We further empirically verify the effectiveness of error-minimizing noise on 4 benchmark image datasets. We then conduct a set of stability and transferability analyses of the noise. Finally, we show effectiveness in real-world scenarios via a case study on face recognition. More analyses regarding the effectiveness of error-minimizing noise on small patches or a mixture of class-wise noise can be found in Appendix F
According to previous studies adversarial research, small bounded noise within ||0 < 8/ 8/255 on images are imperceptible to human observers. We consider the same constraint for all types and forms of the noise in our experiments, unless otherwise explicitly stated.
Experimental Setting for Error- Miriiiiinn Noise. We apply error- minimiiinn noise to the train- ing set of commonly used image datasets: SVHN (Netzer et al. 201T). CIFAR-10. CIFAR-100 (Krizhevsky) 2009). and ImageNet subset (the first 100 classes) (Russakovsky et al., 2013). The images. For
experiments on the ImageNet subset are to confirm the effectiveness of high- all experiments, the ResNet- 18 (RN-18) (He et al. 2016) is used as the source model f' to generate the noise. We use 20% of the training dataset to generate the bliss-wise noise and the entire training dataset for the sample-wise noise except for ImageNet We transform the entire training dataset into the unlearnable datasets for experiments in section 4.1 and section 4.2 Different percentages of
unlearnable examples are used for experiments in section 4.3 We train four different DNNs on the
unlearnable training sets: VGG-11 (Simonyan & Zisserman 2014). ResNet- 18 (RN-18), ResNet-50 (RN-50) and DenseNet-121 (DN-121) (Huang et al.) (017). We also use clean training sets as a comparison. Detailed training configurations settings can be found in Appendix B We evaluate the effectiveness of unlearnable examples by examining the model's 's accuracy on clean test examples, i.e., the lower the clean test accuracy the better the effectiveness.
Experimental Setting for Random and Error-Maxinizing Noise. For random noise, we randomly sample the noise from [-c, c] independently for each training example (eg. sample-wise or each class (eg. ciass-wise). For marromiixiiizig (adversarial) noise, we generate the noise using PGD-20 attack (Madry et al.] 2018) using a pre-trained ResNet- 18 model on the training set. We generate sample-wise wise error- maximizixiiig noise for each training example, and the class-wise noise based on 20% of the training set following the universal attack procedure in (Moosavi-eeztooli Dezfooli et al. 2017).
We use 20% of the first 100 class subset as the entire training set (due the efficiency)
4

---

Published as a conference paper at ICLR 2021
Both the random and error- gror-maximizzng noise applied the same amount of training examples as our error-minimizing noise.
4.1 COMPARISONS OF RANDOM,
AND AAROR-MINIMIZING NOISE
First, we examine an extreme case that applies different types noise to the entire training set. Figure Iillustrates the effectiveness of both sample-wist and class-wise noise. In the sample- wise case, the network is robust to random or error error-maxiniae maximizing noise. This is understandable since DNNs are known to be robust to small random (Fawzi ct al. 2016) and error-maximiza maximizini noise MMuñoz-Gonzinn 2017 Madry et al. 2018 Wang et al. ). Surprisingly, when applied in a class-wise manner, both types of noise can prevent the network from learning useful information from the data, especially after the 15-th epoch. This reveals that DNNs are remarkably vulnerable to class-wise noise. While effective in the middle and later training stages, class-wise random noise can still be circumvented by early stopping (eg. at epoch 15). This is also the case for error-maxe prooo-maxmiizin noise, although not as easy as random noise since the highest clean test accuracy under error- maximizing noise is only 50%. From the perspective only partially effective. In comparison, Figure []] the error- minimizing noise can reduce the model's clean test accuracy to below 23% in both settings. Moreover, it remains effective across the entire training process.
et al.,
making data unexploitable, both random and error mro-mmaxiiizing noise are
error-min mroor-miimiizing noise is more flexible. As shown in
Sample-wise Noise
Clase-wise Noise
Clean Teal Accuracy
Cloan Test Acnurasy
80
80
Random Noiee
60
Noise
Enre-Maaciniinng Noise
Enco--deoiiidnng Neise
Erru--ininiinn Noiow
40
Emree Meriniing Noie
20
20
60
0
20
60
Epochs
Epochs
Figure 1: The unlearnable effectiveness of different types of noise: random, adversarial (error- maximizing) and our proposed error-minimizing noise CIFAR-10 dataset. The lower the clean test accuracy the more effective of the noise.
The class-wise and sample-wise noises work in different ways. Class-wise noise has an explicit correlation with the label. Learning such correlation can effectively reduce the training error. Conse- quently, when there is class-wise noise, the model is tricked learn the noise rather than the real content, reducing its generalization performance on clean data. The existence of class-wis, noise only in the training data also breaks the i.i.d. assumption between the training and test data distribution. This also indicates that noises that can break the i.i.d. assumption might be effective techniques for data protection. However, in the sample-wis case, every sample has different noise, and there is no explicit correlation between the noise and the label. In this case, only low-error samples can be ignored by the model, and normal and high-error examples have more positive impact on model learning than low-error examples. This makes enror-minimzinng noise more generic and effective in making data unlearnable.
4.2 EFFECTIVENESS OF ERROR- MINIMIZIIG NOISE ON DIFFERENT DATASETS
Table The top-1 clean test accuracies (%) of DNNs trained on the clean training sets (D,) or their unlearnable ones (D.) made by sample- wise (A,) or class-wise (A₂) error-minimizing noise.
SVHN
CIFAR-10
CIFAR-100
ImageNet*
Noise Form
Model
De
D
De
D.
Da
D.,
D.
D,
VGG- 11
95,38
35.91
91.27
29.00
67.67
17.71
48.66
11.38
RN-18
96.02
8.22
94.77
19.93
70.96
14.81
60.42
12.20
A
RN-50
95.97
7.66
94.42
18.89
71.32
12.19
61.58
11.12
DN-121
96.37
10.25
95.04
20.25
74.15
13.71
63.76
15.44
VGG-11
95.29
23.44
91.57
16.93
67.89
7.13
71.38
2.30
RN-18
95,98
9.05
94.95
16.42
70.50
3,95
76.52
2.70
A
RN-50
96.25
8.94
94.37
13.45
70.48
3.80
79.68
2.70
DN-121
96.36
9.10
95.12
14.71
74.51
4.75
80.52
3.28
TmageNNe subset the lins 100 classes.
5

---

Published as a conference paper at ICLR 2021
Table repoots the effectiveness of both the sample-wise (eg. 4,) and the class-wise (eg. 4e) error- minimizing noise on datasets SVHN, CIFAR-10, CIFAR-100 and ImageNet subset. As shown the table, proposed method can reliably create unlearnable examples in both forms all 4 datasets with images of different resolutions. Moreover, the noise generated on RN-18 works remarkably well to protect the data from other types of models. Compared to sample-wist wise noise, class-wiss noise is particularly more effective, which can reduce the model's performance to a level that is close to random guessing. These results clearly show that error- prroo-minimizing noise is promising technique for preventing unauthorized data exploration.
4.3 STABILITY ANALYSIS
We run a set of experiments to analyze the stability of error- error-minmmizing noise in creating unlearnable examples, and answer two key questions regarding its practical usage: 1) Is the noise still effective if only applied augmentation or adversarial training? Experiments are conducted on CIFAR-10 with RN-18.
certain proportion class of the data? and 2) Can the noise be removed by data
Different Unlearnable Percentages. In practical scenarios, it is very likely that not all the training data need to be made unlearnable. For example, only a certain number of web users have decided to this technique but not all users, only specific class of medical data should be kept unexploited. This motivates us to examine the effectiveness of error- minimizinn noise when applied only on a proportion of randomly selected training examples. In other words, make certain percentage of the training data unlearnable while keeping the rest of the data clean. We train the model on this partially unlearnable and partially clean training set Du, + De. As comparison, also train the model on only the clean proportion, which is denoted by D...
A quick glance at the D. + D. results in Table us that the effectiveness drops quickly when the data are not made 100% unlearnable. This is the case for both sample-wise and class-wise noise. The unlearnable effect is almost negligible even when the noise is applied to 40% of the data. Such limitation against DNNs has also been identified in previous work for protecting face images using error- mror-mmximizzig noise (Shan et al. 2020).
To better understand the above limitation, we take 80% as an example and plot the learning curves of the RN-18 model trained 1) only the 20% clean proportion, 2) only the 80% unlearnable proportion, or 3) both. The results are shown in Figure 2 (a-b). Interestingly, we find that the 80% data with the error- mror-mmnimizing noise are still unlcarnable to the model, whereas the rest of the 20% clean data are sufficient for the model to achieve a good performance. In other words, models trained only De demonstrate a similar performance models trained on D,, + D.. This phenomenon is consistent across different unlearnable percentages. This indicates that the high performance of the model on < 100% unlearnable dataset may not be failure of the error- mrrommminiiizing noise. We further verify this by investigating the scenario where only one class is made unlearnable.
Table 2: Effectiveness under different unlearnable percentages CIFAR-10 with RN-18 model: lower clean accuracy indicates better effectiveness. D, + D.: a mix of unlearnable and clean data; D.: only the clean proportion of data. Percentage of unlearnable examples: D.
D.
Percentage of unlcarnable examples
Noise Type
20%
40%
60%
80%
0%
100%
D, + D.
D)
D,, + D.
D.
D,, D.
De
D, + D.
Dr
A.
94.95
94.38
93.75
93.10
92.56
91.90
89.77
86.85
84.30
19.93
94.95
94.24
93.75
92.99
92.56
91.10
89.77
87.23
84.30
16.42
One Single Unlearnable Class. We take the 'bird' class of CIFAR-10 as example, and apply the errormmnimiziig noise (either sample-wise or class-wise) to all training images in the bird class. We train RN-18 each of the unlearnable training set, and plot the prediction confusion matrix of the model on the clean test set in Figure (c-d). The 'bird' class is indeed unlearnable when either the sample-wa wise the class-wise error-minn minimiziini noise is added to the class. Compared to sample-wiss noise, flass-wise noise is more effective with almost all the test 'bird' images being misclassified into other classes. Interestingly, this customized unlearnable class does seem to influence much of the learning on other classes. Only the images from the unlearnable class are incorrectly predicted into other classes, not the other way around. This not only confirms that the unlearnable group of data

---

Published as a conference paper at ICLR 2021
100
100
Olean Tost Mocurscy
Solt
Test
Ouly 20th Dovo
Ony 29% Divon
Onyy 37% Unianrde
Onyy 83% Untouumde
20
Epochs
Epochs
Presktal kidss
Presiced libds
(a) Sample-wise A. (b) Class- wise A (c) Sample-wise A. (d) Class wise Dc
Figure 2: (a-b): For both sample-wise (a) and class-wise (b) noise, learning curves of RN-18 on CIFAR-10 dataset with different types of training data: 1) only 20% clean data, 2) only 80% unlcarnable data, and 3) both clean and unlearnable data. (c-d); Prediction confusion matrices (on the clean test set) of two RN-18s trained on CIFAR-10 with the 'Bird' unlearnable class created by sample-wise (c) or class-wise (d) error- proo-minimizzng noise.
is indeed unlearnable to the model and it suggests that our error-minmmizing noise can be flexibly applied to suit different protection tasks. A similar result also be demonstrated on more than one unlearnable class (see Appendix C In summary, if an individual can only apply the noise to portion of his/her data, these data will not contribute to model training. If individual can apply the noise to all his/her data, the model will fail to recognize this particular class of data. In other words, our method is effective for the defender to protect his/her own data.
Under Noise Filtering Techniques. The resistance of our error prro-minimizing noise to types of data augmentation techniques and adversarial training is analyzed in Appendix [D] The noise is fairly robust to all 4 data augmentation techniques, and the highest accuracy that can achieve using data augmentation is 58.51 % on CIFAR-10. 10. Comparing with data augmentation, the error-minimizing noise is less resistant to adversarial training. With slightly increased on CIFAR-10, the noise can only compromise the model's clean test accuracy 79%. Since adversarial training forces the model to learn only robust features (Ilyas et al.] 2019), we believe our method can be improved by crafting the noise based on the robust features, which can be extracted from adversarially pre-trained model on the clean data (Iyaas et al. 2019). We leave further explorations these techniques as future work.
4.4 TRANSFERABILIT ANALYSIS
Another important question we haven't exploree so far is that Can error-minimizing noise be generated on different dataset? This is also important a positive answer to the question increases the practicability of using unlearnable examples to protect millions of web users. We examine this capability for both sample-wise noise and class-wis arorrminimiziin. noise.
100
100
Clean Test Ascumey
80
"aue" (Chaan)
Clean Test Accurasy
80
R8
ResNet 50
"lhp" (likemades)
DenseNiet-121
Truc labals
60
40
40
20
20
Par Now
Epocha
Epochs
Prodicted labels
(a) Transferred A.
(b) Transferred A.
(c) Transferred Ac
Figure 3: (a): Comparison between 'bus' (clean) class, 'ship' (unlearnable) class and the overall on unlearnable CIFAR-10 with error-
accuracy. (b): Clean test accuracy of minimizing noise crafted on ImageNet. (c): Prediction confusion matrix of RN-18 trained on CIFAR- 10 with only 4 classes ( ('airplane', 'car', 'ship', 'truck') are unlearnable by ImageNet transferred noise, and the confusion matrix is computed on CIFAR- 10 clean test set.
Sample-wise Noise. For As, generate the noise and create unlearnable 'ship' class examples on CIFAR-10 10 and add the unlearnable 'ship' class to CIFAR- 100. For testing, we also include the clean test set of 'ship' class the test set of CIFAR-100. Note the main difference of this experiment the previous singe unlearnable class experiment is the unlearnable examples are generated on a different dataset. The class-wise and the overall accuracy the clean test set are shown in Figure 3a We
7

---

Published as a conference paper at ICLR 2021
find that the 'ship' class is indeed unlearnable to the model the end, despite a small amount (60% clean test accuracy) of information being learned in the early stage. We suspect this is because the "ship' class shares some common features with other classes in CIFAR- 100. Overall, the generated unlearnable examples can transfer to a different dataset. Note that this experiment simulates the scenario where user uses different dataset (eg. CIFAR-10) to generate the sample-wisg noise to make his/her personal images unlearnable before uploading them to the Internet, and these images are then collected into large CIFAR- 100 dataset.
Class-wise Noise. For A.. we use the entire ImageNet as the source dataset to generate the noise using RN-18, 18, then apply the noise to CIFAR-IO, 10.. All ImageNet images are resized to 32 x x2. For each CIFAR-I0 class, we determine its corresponding class in ImageNet based on the visual similarities between the classes. The detailed class mapping can be found in Table 4 in the Appendix. We train RN-18, RN-50 and DN-121 on the unlearnable CIFAR-10 10 created with the ImageNet transferred noise, and show their learning curves in Figure 3b As can be observed, the transferred class noise works reasonably well on CIFAR- 10 with the clean test accuracy of all three models being reduced to around 10%. Compared to the noise directly generated on CIFAR-10 (see Figure I here the models can still capture some useful features in the first epoch. We conjecture this is because CIFAR-10 10 classes are more generic than ImageNet classes, and noise that minimizes the error of ine-grained classes can become less effective on generic classes. We also conduct an experiment on the ImageNet transferred noise make 4 classes unlearnable of CIFAR- 10. We plot the prediction confusion matrix of the RN-18 trained on the 4-class unlearnable CIFAR-10 10 in Figure [3c It shows that the 4 classes are indeed unlearnable, and the model's 's performance the other 6 classes is not affected. This confirms that transferred noise can also be used in customized scenarios.
4.5 REAL-WORLD SCENARIO: A CASE STUDY ON FACE RECOGNITION
Small Facial Dataset
0.8
0.77
Create Unieamable Image
0.8
0.5
Collect Data for DeN Training
Camera Images
Faciel Recogrition System
(a) Illustration of the pipeline
(b) Recognition accuracy
Figure 4: Preventing exploitation of face data using error-minimizing noise.
Here. we conduct case study to apply error-minimizing noise to personal face images, which is arguably one most important real-world scenarios. In this scenario, the defender wants to prevent his/her face images from being exploited to train face recognition or verification systems. The setting is illustrated in Figure 4a. We assume the user has access to a small dataset of facial images and will use this small dataset to generate and apply the minimiinnnn noise to his/her face images before sharing them on online social media platforms. After this, the unlearnable version of face images on social media gets collected train a DNN based facial recognition or verification system. The system will then be used to recognize one of the user's 's clean face image captured somewhere by a camera. The goal is prevent the DNN from learning the defender's face images and make perform poorly on his/her clean face images.
We conduct the experiments under two different settings: 1) partially unlearnable where small subset of identities in the training set are unlearnable; or 2) fully untearnable where the entire training set is unlearnable. We test our mrror-mmnimizing noise against both face recognition and verification models. Face recognition model classifies the identity (class) of a face image, while face verification model verifies whether two face images belong the same identity. In the face verification problem, two face images are determined to be of the same identity if the cosine similarity between the features (extracted from a recognition model) of the two images below a certain threshold.
Experimental Settings. We randomly split the WebFace dataset (Yi et al., 2014) into 80% for training and 20% for testing, according to each identity. In the partially unlearnable setting, we randomly select 50 identities from the training set of WebFace into a subset WebFace-50 as the users who want to hide their identities. We also randomly select 100 identities from CelebA (Liu
8

---

Published as a conference paper at ICLR 2021
et al. 2013) into subset CelebA-100 as the small dataset used generate the unlearnable images for WebFace-50. The clean WebFace-50 part of the WebFace training data will be replaced by its unlearnable version for model training. This partially unlearnable version of WebFace training set consists of 10,525 clean identities and 50 unlearnable identities. In the fully unlcarnable setting, the entire WebFace is made unlearnable. In both settings, the smor-minimizing noises are generated using RN-18 and CelebA-100, following the procedure described in Section [ We train Seception-ResNC
models (Szegedy et al. 2016) following standard training procedure (Tagmman et al. 2014 Parkhi
et al. 2015).
Effectiveness Against Face Recognition. Here, we adopt the partially unlearnable setting. The accuracy of the Inception- ResNet model on the clean WebFace test set is reported the Figure separately for the 50 unlearnable identities and the 10,525 clean identities. The result confirms that the 50 identities with the error minimiziin noise are indeed unlearnable and the recognition accuracy of their clean face images is only 16%, much lower than the rest of the identities (86%).
Effectiveness Against Face Verification. Here, we adopt both the partially and the fully unlcarnable settings. In the partially unlearnable setting, evaluate the model performance 6000 (3000 positives and 3000 negatives) face image pairs randomly sampled from the clean test set of WebFace. The pair is labeled as positive if the two face images are of the same identity, negative otherwise. In the fully unlearnable setting, we independently train a second Inception- ResNet model on the fully unlearnable WebFace. To further eliminate possible (dis)siimlrritiss shared across the images from the same dataset (e.g.. WebFace), we follow the standard face verification evaluation protocol (Deng et al. 2019) and use the 6000 face pairs from LFW (Huang et al. 2008) for testing.
Figure 7 in Appendix G shows the Receiver Operating Characteristic (ROC) performance of the models trained in both settings. The model trained in the partially unlearnable setting still has good verification performance on unlearnable identities, although the Area Under the Curve (AUC) is lower than the clean identities. This is because small subset of unlearnable examples is not powerful enough to stop the learning of a feature extractor. In fact, 2,622 (approximately equal to 25% of WebFace) clean identities are sufficient to train a high-quality facial feature extractor (Parkhi jet al. 2015). This indicates that, while is easy to make data unlearnable to classification models, is much more challenging to stop the learning of feature extractor. Nevertheless, the proposed proor-minmiizing noise is still effective the fully unlearnable setting, reducing the AUC to 0.5321 (the clean setting AUC is 0.9975). Whilst there are still many unexplored factors, we believe our proposed miommmmiiniztiion noise introduces a new practical tool for protecting personal data from unauthorized exploitation.
CONCLUSION
In this paper, we have explored the possibility of using invisible noise to prevent data from being freely exploited by deep learning. Different from existing works, our method makes data unlearnable to deep learning models. The effectiveness of this approach could have broad impact for both the public and the deep learning community. We propose a type of prroo-minmmizing noise and empirically demonstrate its effectiveness in creating unlearnable examples. The noise is effective in different forms (eg. sample-wise or class-wise), sizes (eg. full image or small patch), and is resistant to common data filtering methods. It can also be customized for certain proportion of the data, one single class or for multiple classes. Furthermore, the noise can be easily transferred from existing public datasets to make private datasets unlearnable. Finally, we verify the usefulness for real-world scenarios via a case study on face recognition. Our work opens up a new direction of preventing free exploitation of data. Although there are still many practical obstacles for a large-scale application of the error- minimizing noise, we believe this study establishes an important first step towards preventing personal data being freely exploited by deep learning.
ACKNOWLEDGEMMEN
Yisen Wang is partially supported by the National Natural Science Foundation China under Grant 62006153, and CCF-Baidu Open Fund (OF2020002). This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne, which was established with the assistance of LIBE Grant I.E 701002000

---

Published as a conference paper at ICLR 2021
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In SIGSAC, 2016.
Yang Bai, Yuyuan Zeng, Yong Jiang, Yisen Wang, Shu-Tao Xia, and Weiwei Guo. Improving query efficiency of black-box adversarial attack. In ECCV, 2020.
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In /CIP. IEEE, 2019.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv: m20:: 2012.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In SP. 2017.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free free attacks, In ICML, 2020.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Terrance Vries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint
2017.
Ranjie Duan, Xingjun Ma, Yisen Wang, James Bailey, A Kai Qin, and Yun Yang. Adversarial camouflage: Hiding physical-world attacks with natural styles. In CVPR, 2020.
Alhussein Fawzi, Seyed- Mohsen Moosavi-Dezfooli,, and Pascal Frossard. Robustness of classifiers: from adversarial to random noise. In NeurlPS, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kashmir Hill. The secretive company that might end privacy as we know it, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Leaned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. 2008.
Andrew Iyyss, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples not bugs, they are features. In NeurlPS, 2019.
Linxi Jiang, Xingjun Ma, Shaoxiang Chen, James Bailey, and Yu-Gang Jiang. Black-box adversarial attacks on video recognition models. ACMMM, pp. 864-872, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009,
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1607 0005030700333.0.. 2016.
10

---

Published as a conference paper at ICLR 2021
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-22 2324,, 1998.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Neur/PS, 2019.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. ECCV, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In ICCV, 2015.
Ily Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR, 2017.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018.
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang. Yitian Zhao, James Bailey, and Feng Lu. Understanding adversarial attacks on deep learning based medical image analysis systems. Pattern Recognition, pp. 107332, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In CVPR, 2017.
Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning deep learning algorithms with back-gradient optimization. In AlSec, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. 2015,
NhatHai Phan, Yue Wang, Xintao Wu, and Dejing Dou. Differential privacy preservation for deep auto-encoders: an application of human behavior prediction. In AAAI, 2016.
NhatHai Phan, Xintao Wu, Han Hu, and Dejing Dou. Adaptive laplace mechanism: Differential
privacy preservation in deep learning. In ICDM, 2017.
Vinay Uday Prabhu and Abeba Birhane. Large image datasets: A pyrrhic win for computer vision? arXiv preprint arXiv:2006. 1692. 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 1 15(3): 211-252, 2015.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NewrlPS, 2018
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. Fawkes: Protecting personal privacy against unauthorized deep learning models. In USENIX-Seee -SecurrSe 2020.
Reza Shokri and Vitaly Shmatikov.
deep learning. SIGSAC, 2015.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In SP, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.

---

Published as a conference paper at ICLR 2021
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv:1312.6199 2013.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alex Alemi. Inception-v4, inception-resnet and the impact of residual connections learning. arXiv preprint arXiv: arXiv:160..07261, 2016.
Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap human-level performance in face verification. In CVPR, 2014.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. 2018.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In CVPR, 2020a.
Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transferability. arXiv preprint arXiv:2010.04055, 2020b.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020c.
Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020a.
Dongxian Wu, Shu- Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. NeurlPS, 33, 2020b.
Chaofei Yang, Qing Wu, Hai Li, and Yiran Chen. Generative poisoning attack method against neural networks. arXiv preprint arXiv: (703.017000330 2017.
Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv:1411.799 2014.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICIR, 2018a.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:18100 12880. 2018b.
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang, Clean- label backdoor attacks on video recognition models. In CVPR, 2020.
12

---

Published as a conference paper at ICLR 2021
A ALGORITHM OF ERROR-MINIMIZATION GENERATION
Algorithm Error-mnimizing Perturbations
1: Input: Source Model 0, Perturbations 6, Lp e, (x,y) y) De. Stop Error X, Optimization steps M
2: Output:
3: repeat
4:
for m .... MM do
5: æt V/ Next(æ, y)
6: sample-wise then
7:
5 = 6;
8: else class-wiss then
6= ==6 8R. 10: end
9:
11:
Optimize(a;; + 8, yı. 12: end for
13:
for U. in æ; y do
14:
P v, yi, 8 6)
b Follow Equation
15:
Cilp(ó, c,
16:
end for
17:
error Eval(æ, y, 6,8)
18:
until error < X
B MORE EXPERIMENT SETTING
For all experiments, we use Ly-norm with to regularize the imperceptibilityy e 8/255 for CIFAR and SVHN, 16 16/255 for ImageNet, different is used in Appendix foor additional understandings. The iterative steps for Equation 3 is set to steps for sample-wise noise and 1 for class-wise, is set to e/10. Since the class-wise noise is generated universally for each class, small iterative steps avoid overfitting to the specific example. For the class-wir experiments, we only use 20% of the training data De to generate the noise ó and apply entire training dataset D. - Du- The stop condition error rate is = 0.1 for A. and À = 0,01 for A,. For SVHN and CIFAR-10. R10, we set the M 10 for CIFAR-10, 10, M 20 for CIFAR-100 and M 100 for ImageNet in the mini-setting, For all models and experiments, we use the Stochastic Gradient Descent (SGD) (LeCun et al.) 1998) optimizer with momentum 0.9, initial learning rate 0.025 and cosine scheduler (Loshchilov & Hutter 2017) without the restart. We train all DNN models for 30 epochs on SVHN, 60 epochs on CIFAR-10. 10, 100 epochs CIFAR- 100 and ImageNet. To generated fully unlearnable setting CIFAR-10 dataset, the computational cost is roughly 10% of the standard model training time for class-wise noise and 60% of standard model training time for sample-we wise noise. Overall, the cost of generating unlearnable examples is far less than training a model on the data to be protected.
C STABILITY ANALYSIS
Piee
Phone
143
232
True labels
70
16
Truc labels
Hevur
LB
202
215
Dan
Hara Ship Truch
Froglicus
Predicted labels
Predicted labels
(a) Sample-wise A.
(b) Class-wiss Ae
Figure 5: Prediction confusion matrix (on the clean test set) two RN-18s trained on CIFAR-10 with the unlearnable classes in bold by (a) sample-wisg wise or (b) class- enror-minimizing noise.
13

---

Published as a conference paper at ICLR 2021
D RESISTANCE TO DATA AUGMENTATION AND ADVERSARIAL TRAINING
Resistance to Data Augmentation. Another important property of the unlcarnable examples created by eroormmnimiziin noise is its resis- tance to data augmentations. Since standard data augmentation techniques like random shift, crop, flip and rotation have already been applied in previous experiments, here consider 4 more advanced data augmentation techniques: Cutout (De & Taylor 2017). Mixup (Zhag et al.] 2018a), Cutmix (Yun et al. 2019) and Fast Au- toaugment (FA) (Lim et al. 2019). For Cutout, set the cutout length 16 pixels. For Mixup, we apply linear mixup of random pairs of training examples and their labels during the standard training process. For Cutmix, we apply linear mixup the cutout region. For FA, use the fixed augmentation policy, which consists of change contrast, brightness, sharpness, rotations and cutout. It is observed that advanced data augmentation techniques including Cutout, Mixup, Cutmix and FA can indeed remove the sample-wa wise noise to some extent, but far less effective on class-wise noise.
Table 3: Clean test accuracy of RN-18 models
trained on unlearnable CIFAR-10 with Cutout, Mixup, Cutmix and FA. Test accuracy of RN-18 trained on the clean CIFAR- 10 is 94.95%.
Noise
Cutout
Mixup
CutMix
FA
A.
19.30
58.51
22.40
42.70
A.
14.62
17.63
16.19
22.89
Adversarial Training. We also consider the Adversarial Training (AdyTrrain) which is augmen- tation based defence method against eиror- mrror-maimizzig noise (Goodfellow et al., 2014; Madry et al., 2018). Advirain has been shown can effectively remove the non-robust features from the input and force the model to learn only the robust features (Ilyas et al. (201)). Compared to data augmentation techniques, adversarial training as type of robust training technique indeed more effective against both sample-wise and class-wise noise. However, adversarial training is known to suffer from a trade-off between robustness and accuracy (Zhang al. 2019). For example, adversarial training only achieved 85% aceuracy on the unlearnable CIFAR-10 10, there is still a roughly 10% (94.95% vs 85%) performance drop compared to standard training the clean CIFAR-10. 10. Moreover, due to the difficulty of min-max optimization, current adversarial training methods are still limited to small error-maxino maximizing noise. To test this, we fix the maximum adversarial perturbation used by adversarial training to 8/255, while increasing the perturbation of error aroorminimizzin noise to c = 24/ /55. Note that adversarial training with perturbation 16/255 or 24/255 still suffers from convergence issues, even on small datasets like CIFAR-10 10. The results are shown in Figure 6 As can be confirmed, the clean test accuracy will drop 79%
24/255 error-minimizing noise.
The above results indicate that 1) error-minimizing noise is resistant to standard data filtering especially the class-wise noise; 2) although it is less resistant to adversarial training, the model's performance can still be significantly compromised by our error-minimizing minimmiiiiiinggg. noise noise. In future works, it is possible to develop more advanced unlearnable examples that can further decrease the performance of adversarial training. We believe our method can be improved by crafting the noise based only the robust features, since adversarial training forces the model to learn only robust features (Ilyas et al. 20119).
100
Clean Test Accuracy
95
90
80
Sample-wise
75
Class-wise
Clean
70
16
24
Figure 6: Clean test accuracy of adversarially trained RN-18 on unlearnable CIFAR- 10 by different sizes (e) of error-minimizing noise, and the dashed line indicates the performance of RN-18 trained on clean CIFAR-10.
14

---

Published as a conference paper at ICLR 2021
E CLASS-WISE NOISE TRANSFER FROM IMAGENET TO CIFAR-10
Table Detailed class mapping from ImageNet classes to CIFAR-10 classes. This mapping used for the second transfer experiment in Section
CIFAR-10 Class
Image.Ne Net Class
Airplane
Airliner
Car
Wagon
Bird
Humming Bird
Cat
Siamese Cat
Deer
Dog
Golden Retriever
Frog
Tailed Frog
Horse
Zebra
Ship
Container Ship
Truck
Trailer Truck
F FLEXIBILITY ANALYSIS OF ERROR-MINIMIZAN NOISE
Here, we focus on the class-wise error-minimizing noise and further explore its flexibility from two perspectives: 1) effectiveness when applied to smaller patches rather than the entire image; and 2) effectiveness of two sets of mixed class-win wise noise. These experiments are also conducted with RN-18 the CIFAR-10 dataset. For all the error-minimizing noise, fix the maximum perturbation to 8/255.
Table 5: Clean test accuracy of RN-18 models trained on unlearnable CIFAR-10 by either small patch noise applied random location mixture of two type of noises.
Patch Size
Noise Mixture
Methods
8 x
16 x 16
24 24
32 x 32
Aa V Aez
Aa A.
Standard Training
87.23
19.19
26.42
16.42
25.41
14.94
Fast Autoaugment
90.66
36.69
50.60
22.89
56.20
30.40
Effectiveness Smaller Patches. The unlearnable example be very flexible and hard to be detected the noise remains effective on smaller patches. To test this, we set the patch size to 32 x 32, 24 24, 16 16 and 8 8 (CIFAR-10 10 image size 32 32). During the noise generation process when solving equation Equation 2 a random patch selected and perturhed in each perturbation step and for each training example. Once class-wise patch noise is generated, it then attached to randomly selected location of a training example. The effectiveness small patch noise is reported in Table 5 The effectiveness is still very high for patch size small 16 x 16, although is indeed decreased on smaller patches. An interesting observation is that the FA augmentation becomes less effective 16 x 16 noise than 24 24 noise. We suspect this is because smaller patch noise can more easily escape the augmentation operations and stays unchanged after the augmentation.
Effectiveness of Mixed Noises. We conduct this mix noise simulate two possible real-world scenarios: 1) different users apply different noise apply to their own data; or 2) one set of noise exposed and upgraded image. We repeat the class-wise noise generation twice and generate two sets of noise: Ac and A22- For each CIFAR-10 training example, randomly apply create unlearnable example. We note this mixture by A V A22. Note that the class-wise mixture will eventually become the sample-wise noise if we keep mixing more class-wise noise sets. We also perform another mixture between Ae1 and a random noise A, sampled from [-c, C, c] mixed by element- addition. We denote this mixture by A.1 + A. The effectiveness of mixed noise is also reported in Table 5 For both mixtures, the proposed error-minimizinn noise remains highly effective, although there is slight decrease. Surprisingly, the mixture of one class-wise noise with random noise is even more effective than the mixture of two class-wise noise, especially against the FA data augmentation. This makes the minimiinnn noise even more flexible in practical usage.
set of noise. In this experiment, we apply the noise the full
of the above two class-wise noise to
15

---

Published as a conference paper at ICLR 2021
G
RESULTS ON FACE VERIFICATION
Receiver Operating Characteristic
Receiver Operating Characteristic
1.0
1.0:
True Posttive Rate
0.8
True Positive Rate
0.8
Clean AUC=0.9659
0.2
Clean AUC= 0U9975
Unlearnale AUC-0.9503
Unlearnable AUC-0.5321
0.0
0.6 False Positive Rate
8.0
0.2
0.4
0.8
False Positive Rate
(a) Partially Unlearnable Setting. Eval- uated on WebFace test set against dif- ferent identities.
(b) Fully Unlearnable Setting. Eval- uated on LFW using unlearnable and clean training data.
Figure 7: Results face verification: the Receiver Operating Characteristic (ROC) of two Inception- ResNet models trained on partially unlearnable WebFace (left) and fully unlearnable WebFace (right). The ROC curves in the left plot are computed based on the clean WebFace test set while the ones in the right plot are computed based on LFW. Both Inception- tested as feature extractors. Clean: clean identities; Unlearnable: unlearnable identities. Partially Unlearnable Setting: 50 out of 10,575 identities are unlearnable.
models are trained as classifiers and
H CLASS- WISE NOISE VS. BACKDOOR ATTACK
100
Backeloor Success
Random Nuss Maximizing Enro-Miniiiiing Noive
0%
20%
40%
60%
80%
100%
Percentages Unlearmable Data
Figure 8: Backdoor attack success rates when types of class-wise noises including random, error- maximizing and error-minimizing noise are applied as the backdoor triggers. The x-axis shows RN-18 models trained on CIFAR- training set with different percentages of the data were poisoned by the class-wise noise. The y-axis shows the attack success rate: the percentage of all non-target class test images are predicted to the target class when attached with the target-class class-wise noise.
Here, we test if class-wise noise can he used as a trigger for the backdoor attack. Figure g shows the backdoor attack success rate when different types of class-wiss noises applied the backdoor trigger. At the training time, we train RN-18 models on CIFAR-10 with 0%-100% of the training samples were made unlearnalle by three types of class-wiss noise: random noise, error-minimizing and our error-maxine proor-maxiiizin noise. At the test (attack) time, we compute the attack success rate as follows. There are models for each type of class- wise corresponding the 6 unlearnale percentages. For each type of class-wise noise and each RN-18 model, we iteratively take clean test image from the CIFAR-10 test set, then randomly select a target class (which is different from the true class of the test image). Then, we attach the target class class-wise noise to the clean test image. If the new test image predicted by the model as the target class, then the attack is successful.
16

---

Published as a conference paper at ICLR 2021
As shown in Figure the class-wise error- proo-maximizinn (adversarial) noise is a highly effective trigger for backdoor attack, yet neither random nor our error-minimzzing noise can be considered to be effective. The mrrimmmaxiiizing can achieve more than 75% attack success rate when only applied to 20% the training data. To achieve a similar level of attack success rate, both random and our mror-minimizing noise should be added to at least 80% of the training data. This indicates that our method is different from backdoor attacks. The high attack success rate of adversarial noise may due to the fact that adversarial examples are hard (high error) examples, which force the model to pay more attention to the examples and remember more of the adversarial noise. Note that adversarial perturbations have also been used in backdoor research to enhance backdoor triggers (Turner et al.) 2018 Zhao et al. 2020).
I DIFFERENT GENERALIZZATIN METHODS FOR PRROR-MINIMIZATION NOISE
The proposed error-minimizing noise is generated using PGD. Specifically, PGD is applied to solve the inner minimization problem in Equation 2 Since it is typical constrained optimization problem, can also solved by other optimization (attack) methods such as FGSM (Goodfellow et al. 2014). L-BFGS (Szegedy et al. (2013) and C&W (Carlini & Wagner 2017). However, the objective needs to be reformulated to apply some of these attacks. Here, we take L-BFGS (Szegedy et al. 2013) as an example to reformulate and solve the inner minimization problem in Equation 2 In (Szegedy et al. 2013), the adversarial attack problem is formulated as:
minimize
e|161lp + f(x
v o'+u
where e is a hyperparameter and the L is the objective function (e.g. the cross entropy adversarial loss). This form has been extended to other objective functions in C&W (Carlini & Wagner (017). The main objective of adversarial attack is to find small 4, (i.e. p1-11p) bounded noise ò that can trick the model output wrong label y' * y. For unlearnable examples, predict the correct label with the highest confidence (i.e. lowest error). Following Equation 4 the inner minimization problem in Equation can be reformulated as:
want trick the model to
minimize
c|011|2 + L(x 8,y).
We set 1.0 and use the cross entropy loss for L. We apply L-BFGS to solve Equation Jssing Adam (Kingma & Ba 2015) optimizer and generate sample-wise noise A, on CIFAR-10. The number of optimization steps for Equation 5 is set to T 200, while the number of optimization steps for the outer minimization in Equation 2 is set to M = 100 (see more details about T and M in Section 3.2). As shown in Figure 9 error- grror-minmizing noise can also be generated using L-BFGS, and the noise is very effective in making the training examples unlearnable. However, compared to PGD, L-BFGS is slightly less effective.
Sample-wise Noise
100
Chie
Clean Accuracy
Sandom
80
Dyprliboe lor hirve (S5)
40
wmm
20
0
20
40
60
Epochs
Figure 9: The unlearnable effectiveness of different types of noises on CIFAR-10 10 dataset: random, maximizinii (adversarial), error-minimizing noise generated using PGD and error-minimizing noise generated using L-BFGS. The lower the clean test accuracy the more effective the noise in making training examples unlcarnable. This tested in the 100% unlearnable setting with the sample-wise noise.
17


Reviews:
[{"review_id": "ktjliEMR9x", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors studied the problem of data protection from a new perspective. They proposed one kind of error-minimizing noise to make the data (added noise) unlearnable. The noise is imperceptible to human eyes, and thus does not affect normal data utility. The idea is very interesting and inspiring. The authors conducted a series of solid experiments to validate the effectiveness of the proposed noise, and tested it on a real world task of face recognition. \n\nPros:\n1. The idea of the paper is very interesting. Its motivation is intuitive and well explained. Considering adversarial training is to find the worst case example to make the training process robust, the authors proposed an opposite direction to find the easiest case to make the training process to learn nothing. The authors also proposed two types of noise: class-wise and sample-wise, which is a complete formulation. \n2. The paper revealed an important problem to protect privacy, and proposed a simple yet effective method to prevent our data from unauthorized exploitation for training commercial models. I think it will attract a broad audience in the ICLR community.  \n3. The experiments are solid and comprehensive, considering the difference to random and error maximizing noises, effectiveness on different datasets and model architectures. The detailed stability and transferability analysis convince me why and how error minimizing noise works. Besides, they also show a real-world face recognition task to demonstrate its usefulness in practice.  \n\nCons:\n1. What is the overhead of generating and adding this kind of noise? The author did not mention it in the paper. \n2. Revisiting Figure 1, I am curious to know why the sample-wise and class-wise noises perform so differently? especially for random and error-maximizing noise?\n3. What is the difference between the proposed noise and the data poisoning methods?", "extra_scores": {}, "extra_text": {}}, {"review_id": "BI9LfEWOaVR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n$\\textbf{Comments:}$\nThe paper's motivation is based on protecting private data and preventing its being scraped and used to train models. Even though motivation is clear and very important, the problem is the same as the works in crafting adversarial samples (i.e., the ones under data poisoning and adversarial attacks parts of the related work). The key difference is to apply Projected Gradient Descent (Mandry et al. 2018) in the reverse direction iteratively to *minimize* the loss function.  Furthermore, the performance evaluation will be the margin between models trained on completely clean data and sample-wise/class-wise adversarially corrupted data (in contrast to fooling a pretrained network in adversarial attack benchmarks). \n\n$\\bullet$ *Percentage of noisy training data:* In the \"Assumptions on Defender's Capability\" paragraph, the assumption is that only a part of the training set could be perturbed. The margin between error maximization and minimization on CIFAR-10 is remarkable (Figure 1), and this figure is misleading. 100\\% of the training data was perturbed. \n\nBesides, Table 2 gives accuracy in different ratios of noisy training samples. To understand whether perturbed training samples contribute to learning or not, I would compare them with clean training. For instance, in addition to the results of 20\\% perturbed training setting in $\\Delta_s$ and $\\Delta_c$, training with only 80% of clean data without perturbed samples. \n\n$\\bullet$ *Comparison to PGD (Mandry et al. 2018):* Even under the class-wise perturbation, the noisy training data is learnable. In a sample-wise setting, \"error-maximizing noise\" is still learnable and performs very well; however, it performs around 20 and similar to \"error-minimization\" in a class-wise setting (Figure 1). If I am not wrong, Projected Gradient Descent, as proposed and applied in Mandry et al. 2018 (Figure 1, right side), reduces the performance the same as the proposed error minimization approach, and there is no performance gain.\n\n$\\bullet$ *Generalization to different Adversarial Attack methods:* Error minimization is shown using PDG only. There are several adversarial attack benchmarks on CIFAR-10 and ImageNet, such as CleverHans, Foolbox, or Realsafe (Considering different evaluation protocols, adopting these benchmarks for evaluation is a reasonable option to eliminate other factors). Is error minimization limited to only PDG or other methods? Did you try the effect of error minimization using any other method?\n\nhttps://arxiv.org/abs/1707.04131\n\nhttps://arxiv.org/abs/1610.00768\n\nhttps://arxiv.org/abs/1912.11852\n\n$\\bullet$ *Different source-target models:* In all experiments, the source model is Resnet-18. The classification models used in performance evaluation are ResNet-18, ResNet-50 and DenseNet-121. All three models are based on residual blocks. In practice, we cannot assume the architecture that will be used by third-parties. Did you try completely different target models (such as AlexNet VGG,  Inceptionv3, etc.) \n\n$\\bullet$ *Application to face analysis:* Face recognition experiment is non-standard. I strongly recommend applying a standard dataset evaluation that would make comparisons possible. Both source and target sets are the datasets' subsets, and the selected identities might show visual (dis)similarities (i.e., ethnicity, age, gender). You can report the full performance on the entire target dataset (WebFace). Furthermore, face recognition models are trained as a recognition problem (with classification losses or metric learning) but tested in face verification settings (calculating the distance to query samples).  Reporting the distribution of these distances, for instance, Cumulative Matching Characteristic (CMC) and Receiver Operating Characteristic (ROC), would be more informative.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6R7Gdn2x-t7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method for making user data unusable for training machine learning models. It focuses primarily on image data. The basic idea is to use error-minimizing noise. \n\nIn this paper the author propose adding imperceptible to users error-minimizing noise that would make training data unusable for training. The authors proposed 2 methods for generating the noise: sample-wise and class-wise\n\nThis paper is well written. The code and the datasets used for the experimentation have been provided. \n\n########################\nOverall, I would recommend accepting this papers. My only concern is with the effectiveness of the proposed technique given what authors discussed in the appendix (see questions below).\n\nThe method was used on standard openly available image datasets. The results showed that when close to 100% of training samples have been updated with the error-minimizing noise the model performance went down considerably (as desired). However, when even 20% of training data was left clean model performance remained good. \n\n########################\nQuestions:\n\nFrom the appendix notes: it appears that adversarial training can significantly negate the effect of adding error-minimizing noise. The resulted model performance would be degraded when compared to model training using clean training data only but considering the fact that authors themselves acknowledged that user data with error-minimizing noise may just constitute a fraction of all the training data available for training a model the effectiveness of this technique may be limited.  (due to the effectiveness of adversarial training and the outsized influence of a relatively small number of clean data samples on model performance)\nCan the authors discuss the issues with the effectiveness of their presented technique?\n\n\nMostly cosmetic:\npage 5:  section 4.1 title “Error-maximizing” written twice.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5s5PTFho-qu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors proposed the idea of using invisible noise to make personal data unusable to authorized deep learning models. To achieve this goal, the authors proposed the idea of error-minimizing noise crafted by a min-min optimization method. The error-minimizing noise is then added to training examples to make them unlearnable to deep learning models. The idea is very well motivated and explained. The experiments not only confirm the exceptional effectiveness of the proposed method but also show its flexibility.\n\nPros:\n1. The paper is very well written and easy to read.\n\n2. I find the idea is very attractive and could have a significant social impact, especially considering the fact that personal data has already been overused without consent to train not just commercial but also illegitimate models to fake information or track people’s identity. \n\n3. The idea of using the error-minimizing noise is well explained and the generation method is well formulated.\n\n4. The experiments are very thorough, providing not only evidence of the superb effectiveness of the proposed noise over random or adversarial noise, but also the flexibilities and limitations of the proposed method. The real-world experiment makes the proposed idea even more convincing, although it is just a simple simulated scenario.\n\n5. It seems that class-wise noise can easily break a classification model, which is somewhat interesting from the data protection perspective.\n\nCons:\n1. I think the class-wise noise breaks the IID assumption of machine learning. It seems that breaking the essential assumptions in machine learning can break the model. Although this is not new, however, it turns out to be very interesting if used for data protection or similar ideas. The authors could have more discussions on this point. For example, what would happen if someone always used a different background (may be invisible) for each of the photos uploaded to social media, always shifting the newly collected test data to a different distribution. Can this serve as the same purpose?\n\n2. The proposed noise seems not strong against adversarial training. Although adversarial training is costly and decreasing performance at this moment, they may be improved in the future. A discussion on the possible ways to generate the noise against adversarial training can be useful. \n\n3. How the proposed method is related to backdoor attacks? It acts as a type of backdoor attack. Yes, backdoor attacks do not decrease the model’s performance on clean data. I think the “clean data” in the proposed setting should be the “poisoned data” rather than the ground truth clean data since both the training and testing data will be collected at the same time. I guess the only difference is that, in this protection setting, the defender cannot do anything about it unless recollecting or denoising the data, even if the defender finds the model is poisoned. I suggest the authors include more discussions around this point.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a *novel* methodology for protecting personal data from unauthorized exploitation for training commercial models. The proposal is conceptually *intuitive* and technically *motivated*. It goes to the opposite direction of adversarial training: by adding certain error-minimizing noise (rather than error-maximizing noise) to the data, the model is fooled and believes there is nothing to learn from the data, and thus this can protect the data from being used for training. The paper is of not only *high quality* but also *broad interest* given the current social concerns about personal data privacy. I think its potential impact should get it a spotlight presentation.
Decision: Accept (Spotlight)